Text Generation
Transformers
PyTorch
English
llama
unsloth
axolotl
conversational
text-generation-inference
5-bit
exl2
Instructions to use dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8") model = AutoModelForCausalLM.from_pretrained("dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8
- SGLang
How to use dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Unsloth Studio
How to use dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8 with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8 to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8 to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8 to start chatting
Load model with FastModel
pip install unsloth from unsloth import FastModel model, tokenizer = FastModel.from_pretrained( model_name="dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8", max_seq_length=2048, ) - Docker Model Runner
How to use dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8 with Docker Model Runner:
docker model run hf.co/dreamgen/opus-v1.4-70b-llama3-exl2-5.0bpw-h8
| { | |
| "name": "DreamGen Opus V1 Llama 3: Story-Writing", | |
| "inference_params": { | |
| "input_prefix": "<|eot_id|>\n<|start_header_id|>writer character: Gandalf<|end_header_id|>\n\n", | |
| "input_suffix": "<|eot_id|>\n<|start_header_id|>writer character: Dumbledore<|end_header_id|>\n\n", | |
| "antiprompt": ["<|start_header_id|>", "<|eot_id|>", "<|end_of_text|>"], | |
| "pre_prompt_prefix": "<|start_header_id|>system<|end_header_id|>\n\n", | |
| "pre_prompt_suffix": "", | |
| "pre_prompt": "You are an intelligent, skilled, versatile writer.\n\nYour task is to write a story based on the information below.\n\n\n## Overall plot description:\n\nA battle ensues between Gandalf and Dumbledore as they are brought together in an arena by a mysterious force. The objective is clear: the victor will be granted the chance to save their world, while the loser must watch their own demise. Gandalf, a seasoned wizard, is determined to emerge victorious. Meanwhile, Dumbledore, with his vast knowledge and powerful magic, is not one to underestimate. The two engage in a fierce and spectacular duel, with both employing their unique abilities. As the fight reaches its climax, the true power of these legendary wizards is revealed. Ultimately, one emerges triumphant, their world saved, but not without scars.\n\n\n## Characters:\n\n### Gandalf:\n\nGandalf is a wise and powerful wizard known for his iconic grey robes and hat. He possesses a deep understanding of magic and the forces of nature. Gandalf is a skilled swordsman and an experienced warrior, having battled numerous dark forces throughout his life. He is determined and resilient, never backing down from a challenge. In this story, Gandalf's stubbornness and refusal to accept defeat prove to be his greatest assets.\n\n## Dumbledore:\n\nDumbledore is a highly respected wizard known for his wisdom, intelligence, and powerful magic. He has a long white beard and a twinkle in his eyes that belies his formidable abilities. Dumbledore is a master of both magic and dueling, making him a formidable opponent. He is known for his calm and collected demeanor, even in the face of danger. In this story, Dumbledore's vast knowledge and magical prowess serve as his primary weapons." | |
| } | |
| } | |