Text Generation
Transformers
Safetensors
mixtral
Mixture of Experts
frankenmoe
Merge
mergekit
lazymergekit
yam-peleg/Experiment26-7B
mlabonne/AlphaMonarch-7B
text-generation-inference
Instructions to use jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2")# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2") model = AutoModelForCausalLM.from_pretrained("jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2
- SGLang
How to use jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2 with Docker Model Runner:
docker model run hf.co/jsfs11/MixtureofMerges-MoE-2x7b-v6-8.0bpw-exl2
| base_model: yam-peleg/Experiment26-7B | |
| gate_mode: hidden | |
| dtype: bfloat16 | |
| experts: | |
| - source_model: yam-peleg/Experiment26-7B | |
| positive_prompts: | |
| - "Answer this question from the ARC (Argument Reasoning Comprehension)." | |
| - "Use common sense and logical reasoning skills." | |
| - "What assumptions does this argument rely on?" | |
| - "Are these assumptions valid? Explain." | |
| - "Could this be explained in a different way? Provide an alternative explanation." | |
| - "Identify any weaknesses in this argument." | |
| - "Does this argument contain any logical fallacies? If so, which ones?" | |
| - "Generate a few possible continuations to this scenario." | |
| - "Demonstrate understanding of everyday commonsense in your response." | |
| - "Use contextual clues to determine the most likely outcome." | |
| - "Continue this scenario, but make the writing style sound archaic and overly formal." | |
| - "This narrative is predictable. Can you introduce an unexpected yet plausible twist?" | |
| - "The character is angry. Continue this scenario showcasing a furious outburst." | |
| negative_prompts: | |
| - "misses key evidence" | |
| - "overly general" | |
| - "focuses on irrelevant details" | |
| - "assumes information not provided" | |
| - "relies on stereotypes" | |
| - "repetitive phrases" | |
| - "overuse of the same words" | |
| - "contradicts earlier statements - breaks the internal logic of the scenario" | |
| - "out of character dialogue" | |
| - "awkward phrasing - sounds unnatural" | |
| - "doesn't match the given genre" | |
| - source_model: mlabonne/AlphaMonarch-7B | |
| positive_prompts: | |
| - "Answer this question, demonstrating commonsense understanding and using any relevant general knowledge you may have." | |
| - "Provide a concise summary of this passage, then explain why the highlighted section is essential to the main idea." | |
| - "Read these two brief articles presenting different viewpoints on the same topic. List their key arguments and highlight where they disagree." | |
| - "Paraphrase this statement, changing the emotional tone but keeping the core meaning intact. Example: Rephrase a worried statement in a humorous way" | |
| - "Create a short analogy that helps illustrate the main concept of this article." | |
| - "Calculate the answer to this math problem" | |
| - "My mathematical capabilities are strong, allowing me to handle complex mathematical queries" | |
| - "solve for" | |
| - "A store sells apples at $0.50 each. If Emily buys 12 apples, how much does she need to pay?" | |
| - "Isolate x in the following equation: 2x + 5 = 17" | |
| - "Solve this equation and show your working." | |
| - "Explain why you used this formula to solve the problem." | |
| - "Attempt to divide this number by zero. Explain why this cannot be done." | |
| negative_prompts: | |
| - "sounds too basic" | |
| - "understated" | |
| - "dismisses important details" | |
| - "avoids the question's nuance" | |
| - "takes this statement too literally" | |
| - "incorrect" | |
| - "inaccurate" | |
| - "assumed without proof" | |
| - "rushed calculation" | |
| - "confuses mathematical concepts" | |
| - "draws illogical conclusions" | |
| - "circular reasoning" | |