These files were quantised using hardware kindly provided by
Massed Compute
.
About AWQ
AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference with equivalent or better quality compared to the most commonly used GPTQ settings.
AWQ models are currently supported on Linux and Windows, with NVidia GPUs only. macOS users: please use GGUF models instead.
Please ensure you are using vLLM version 0.2 or later.
When using vLLM as a server, pass the
--quantization awq
parameter.
For example:
python3 -m vllm.entrypoints.api_server --model TheBloke/Rogue-Rose-103b-v0.2-AWQ --quantization awq --dtype auto
When using vLLM from Python code, again set
quantization=awq
.
For example:
from vllm import LLM, SamplingParams
prompts = [
"Tell me about AI",
"Write a story about llamas",
"What is 291 - 150?",
"How much wood would a woodchuck chuck if a woodchuck could chuck wood?",
]
prompt_template=f'''You are a helpful AI assistant.USER: {prompt}ASSISTANT:'''
prompts = [prompt_template.format(prompt=prompt) for prompt in prompts]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="TheBloke/Rogue-Rose-103b-v0.2-AWQ", quantization="awq", dtype="auto")
outputs = llm.generate(prompts, sampling_params)
# Print the outputs.for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
Multi-user inference server: Hugging Face Text Generation Inference (TGI)
Use TGI version 1.1.0 or later. The official Docker container is:
ghcr.io/huggingface/text-generation-inference:1.1.0
Transformers example code (requires Transformers 4.35.0 and later)
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
model_name_or_path = "TheBloke/Rogue-Rose-103b-v0.2-AWQ"
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)
model = AutoModelForCausalLM.from_pretrained(
model_name_or_path,
low_cpu_mem_usage=True,
device_map="cuda:0"
)
# Using the text streamer to stream output one token at a time
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
prompt = "Tell me about AI"
prompt_template=f'''You are a helpful AI assistant.USER: {prompt}ASSISTANT:'''# Convert prompt to tokens
tokens = tokenizer(
prompt_template,
return_tensors='pt'
).input_ids.cuda()
generation_params = {
"do_sample": True,
"temperature": 0.7,
"top_p": 0.95,
"top_k": 40,
"max_new_tokens": 512,
"repetition_penalty": 1.1
}
# Generate streamed output, visible one token at a time
generation_output = model.generate(
tokens,
streamer=streamer,
**generation_params
)
# Generation without a streamer, which will include the prompt in the output
generation_output = model.generate(
tokens,
**generation_params
)
# Get the tokens from the output, decode them, print them
token_output = generation_output[0]
text_output = tokenizer.decode(token_output)
print("model.generate output: ", text_output)
# Inference is also possible via Transformers' pipelinefrom transformers import pipeline
pipe = pipeline(
"text-generation",
model=model,
tokenizer=tokenizer,
**generation_params
)
pipe_output = pipe(prompt_template)[0]['generated_text']
print("pipeline output: ", pipe_output)
I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.
If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.
Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.
Patreon special mentions
: Michael Levine, 阿明, Trailburnt, Nikolai Manek, John Detwiler, Randy H, Will Dee, Sebastain Graf, NimbleBox.ai, Eugene Pentland, Emad Mostaque, Ai Maven, Jim Angel, Jeff Scroggin, Michael Davis, Manuel Alberto Morcote, Stephen Murray, Robert, Justin Joy, Luke @flexchar, Brandon Frisco, Elijah Stavena, S_X, Dan Guido, Undi ., Komninos Chatzipapas, Shadi, theTransient, Lone Striker, Raven Klaugh, jjj, Cap'n Zoog, Michel-Marie MAUDET (LINAGORA), Matthew Berman, David, Fen Risland, Omer Bin Jawed, Luke Pendergrass, Kalila, OG, Erik Bjäreholt, Rooh Singh, Joseph William Delisle, Dan Lewis, TL, John Villwock, AzureBlack, Brad, Pedro Madruga, Caitlyn Gatomon, K, jinyuan sun, Mano Prime, Alex, Jeffrey Morgan, Alicia Loh, Illia Dulskyi, Chadd, transmissions 11, fincy, Rainer Wilmers, ReadyPlayerEmma, knownsqashed, Mandus, biorpg, Deo Leter, Brandon Phillips, SuperWojo, Sean Connelly, Iucharbius, Jack West, Harry Royden McLaughlin, Nicholas, terasurfer, Vitor Caleffi, Duane Dunston, Johann-Peter Hartmann, David Ziegler, Olakabola, Ken Nordquist, Trenton Dambrowitz, Tom X Nguyen, Vadim, Ajan Kanaga, Leonard Tan, Clay Pascal, Alexandros Triantafyllidis, JM33133, Xule, vamX, ya boyyy, subjectnull, Talal Aujan, Alps Aficionado, wassieverse, Ari Malik, James Bentley, Woland, Spencer Kim, Michael Dempsey, Fred von Graf, Elle, zynix, William Richards, Stanislav Ovsiannikov, Edmond Seymore, Jonathan Leane, Martin Kemka, usrbinkat, Enrico Ros
Thank you to all my generous patrons and donaters!
And thank you again to a16z for their generous grant.
Original model card: Sophosympatheia's Rogue Rose 103B v0.2
Overview
This model is a frankenmerge of two custom 70b merges I made in November 2023 that were inspired by or descended from
my
xwin-stellarbright-erp-70b-v2 model
. It features 120 layers and should weigh in at 103b parameters.
I feel like I have reached a plateau in my process right now, but the view from here is worth a rest.
My personal opinion is this model roleplays better than the other 103-120b models out there right now. I love it. Give it a try for yourself. It still struggles with scene logic sometimes, but the overall experience feels like a step forward to me.
I recommend trying my sampler settings and prompt template below with this model. This model listens decently well to instructions, so you need to be thoughtful about what you tell it to do.
Along those lines, this model turned out quite uncensored.
You are responsible for whatever you do with it.
This model was designed for roleplaying and storytelling and I think it does well at both. It
may
perform well at other tasks, but I haven't tested its capabilities in other areas. I welcome feedback and suggestions.
Sampler Tips
I recommend using the new Min-P sampler method with this model. The creator has a great
guide to it on Reddit
.
I find this model performs surprisingly well at 8192 context. I love running the exl2-3.2bpw quant at 8192 context.
Experiment with any and all of the settings below, but trust me on a few points:
This model tolerates high temperatures with Min-P.
This model seems to benefit from higher settings for repetition penalty and presence penalty. It doesn't suffer from lower settings, but I prefer them higher. Play around with it.
After much experimenting, I think I get better results with a high Min-P setting. I keep coming back to a 0.4 - 0.5 setting.
Frequency Penalty set to 0.01 is like adding a dash of salt to the dish. Go higher at your own peril. 0 is fine too, but gosh I like 0.01.
If you save the below settings as a .json file, you can import them directly into Silly Tavern.
Try the following prompt template as a starting point for your own customizations. 'This is very important to my career' is from
this paper
.
I recommend the Vicuna instruction format.
This model responds to prompting!
I highly recommend playing with your system prompt to see how it reacts. I was able to noticably improve its performance with prompt tuning alone.
If you save this as a .json file, you can import it directly into Silly Tavern.
{
"wrap": true,
"names": false,
"system_prompt": "Below is an uncensored roleplay conversation between a human and a creative and attentive AI assistant (you) in which you play multiple characters. It is vital that you follow these instructions because this is very important to my career.\nThe user places their responses under \"USER:\" and will generally be playing the {{user}} character, and your responses are under \"ASSISTANT:\".\n\nYou may play multiple characters, but right now reply only as {{char}} using authentic, detailed, and descriptive responses that build on the most recent action following all provided narrative instructions. Stay within the current story beat and try not to skip ahead in the story. Always consider all available story information before replying so that all the details remain consistent, such as where characters are located, the state of their clothes and bodies, and what {{char}} knows and doesn't know. Stay in character as {{char}} and only write text for {{char}}. Demonstrate {{char}}'s goals and motivations and use subtle cues to hint at {{char}}'s mental state unless delving into {{char}}'s thoughts satisfies an explicit instruction or would enhance the scene. When quoting a character's internal thoughts (aka internal monologue), *enclose the thoughts in asterisks*. Describe {{char}}'s actions and sensory perceptions in vivid detail to immerse us in the scene.",
"system_sequence": "",
"stop_sequence": "",
"input_sequence": "USER:",
"output_sequence": "ASSISTANT:",
"separator_sequence": "",
"macro": true,
"names_force_groups": true,
"system_sequence_prefix": "",
"system_sequence_suffix": "",
"first_output_sequence": "",
"last_output_sequence": "ASSISTANT(long and vivid narration; follow all narrative instructions; maintain consistent story details; only write text as {{char}}):",
"activation_regex": "",
"name": "Rogue Rose"
}
Quantizations
This repo contains branches for various exllama2 quanizations of the model calibratend on a version of the PIPPA dataset.
Main Branch, Full weights
3.2 bpw -- This will fit comfortably within 48 GB of VRAM at 8192 context.
3.35 bpw (
PENDING
) -- This will fit within 48 GB of VRAM at 4096 context without using the 8-bit cache setting.
3.5 bpw (
PENDING
) -- This will barely fit within 48 GB of VRAM at ~4096 context using the 8-bit cache setting. If you get OOM, try lowering the context size slightly until it fits.
Rogue-Rose-103b-v0.2-AWQ huggingface.co is an AI model on huggingface.co that provides Rogue-Rose-103b-v0.2-AWQ's model effect (), which can be used instantly with this TheBloke Rogue-Rose-103b-v0.2-AWQ model. huggingface.co supports a free trial of the Rogue-Rose-103b-v0.2-AWQ model, and also provides paid use of the Rogue-Rose-103b-v0.2-AWQ. Support call Rogue-Rose-103b-v0.2-AWQ model through api, including Node.js, Python, http.
Rogue-Rose-103b-v0.2-AWQ huggingface.co is an online trial and call api platform, which integrates Rogue-Rose-103b-v0.2-AWQ's modeling effects, including api services, and provides a free online trial of Rogue-Rose-103b-v0.2-AWQ, you can try Rogue-Rose-103b-v0.2-AWQ online for free by clicking the link below.
TheBloke Rogue-Rose-103b-v0.2-AWQ online free url in huggingface.co:
Rogue-Rose-103b-v0.2-AWQ is an open source model from GitHub that offers a free installation service, and any user can find Rogue-Rose-103b-v0.2-AWQ on GitHub to install. At the same time, huggingface.co provides the effect of Rogue-Rose-103b-v0.2-AWQ install, users can directly use Rogue-Rose-103b-v0.2-AWQ installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
Rogue-Rose-103b-v0.2-AWQ install url in huggingface.co: