Instructions to use arcee-ai/Virtuoso-Medium-v2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use arcee-ai/Virtuoso-Medium-v2-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("arcee-ai/Virtuoso-Medium-v2-GGUF", dtype="auto")

llama-cpp-python

How to use arcee-ai/Virtuoso-Medium-v2-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="arcee-ai/Virtuoso-Medium-v2-GGUF",
	filename="Virtuoso-Medium-v2-IQ2_M.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use arcee-ai/Virtuoso-Medium-v2-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M

Use Docker

docker model run hf.co/arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use arcee-ai/Virtuoso-Medium-v2-GGUF with Ollama:
```
ollama run hf.co/arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M
```

Unsloth Studio

How to use arcee-ai/Virtuoso-Medium-v2-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for arcee-ai/Virtuoso-Medium-v2-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for arcee-ai/Virtuoso-Medium-v2-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for arcee-ai/Virtuoso-Medium-v2-GGUF to start chatting

How to use arcee-ai/Virtuoso-Medium-v2-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use arcee-ai/Virtuoso-Medium-v2-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use arcee-ai/Virtuoso-Medium-v2-GGUF with Docker Model Runner:
```
docker model run hf.co/arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M
```

Lemonade

How to use arcee-ai/Virtuoso-Medium-v2-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull arcee-ai/Virtuoso-Medium-v2-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Virtuoso-Medium-v2-GGUF-Q4_K_M

List all available models

lemonade list

Improve language tag

by lbourdois - opened Apr 27, 2025

base: refs/heads/main

←

from: refs/pr/2

Discussion Files changed

+91

-77

Files changed (1) hide show

README.md +91 -77

README.md CHANGED Viewed

@@ -1,78 +1,92 @@
----
-base_model:
-- Qwen/Qwen2.5-32B
-library_name: transformers
-tags:
-- mergekit
-- merge
-license: apache-2.0
----
-<img src="https://huggingface.co/arcee-train/Virtuoso-Medium-v2/resolve/main/virtuoso-medium.jpg" alt="Virtuoso-Lite Logo" style="display: block; margin: 0 auto;" />
-**Virtuoso-Medium-v2 (32B)** is our next-generation, 32-billion-parameter language model that builds upon the original Virtuoso-Medium architecture. This version is distilled from Deepseek-v3, leveraging an expanded dataset of 5B+ tokens worth of logits. It achieves higher benchmark scores than our previous release (including surpassing Arcee-Nova 2024 in certain tasks).
-### Model Details
-- **Architecture Base:** Qwen-2.5-32B
-- **Parameter Count:** 32B
-- **Tokenizer:**
-  - Initially integrated with Deepseek-v3 tokenizer for logit extraction.
-  - Final alignment uses the Qwen tokenizer, using specialized “tokenizer surgery” for cross-architecture compatibility.
-- **Distillation Data:**
-  - ~1.1B tokens/logits from Deepseek-v3’s training data.
-  - Logit-level distillation using a proprietary “fusion merging” approach afterwards for maximum fidelity.
-- **License:** [Apache-2.0](#license)
-### Background on Deepseek Distillation
-Deepseek-v3 serves as the teacher model, from which we capture logits across billions of tokens. Rather than standard supervised fine-tuning, we apply a full logit-level replication. This ensures more precise transference of knowledge, including advanced reasoning in:
-- Technical and scientific queries
-- Complex code generation
-- Mathematical problem-solving
-### Intended Use Cases
-- **Advanced Chatbots & Virtual Assistants**
-- **Enterprise Data Analysis & Workflow Automation**
-- **Research Simulations & Natural Language Understanding**
-- **Educational Tools for STEM Fields**
-### Evaluations
-<img src="https://huggingface.co/arcee-train/Virtuoso-Lite/resolve/main/Benchmarks.png" alt="Virtuoso-Lite Logo" style="display: block; margin: 0 auto;" />
-### How to Use
-Below is a sample code snippet using `transformers`:
-```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
-model_name = "arcee-ai/virtuoso-medium-v2-32b"
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-model = AutoModelForCausalLM.from_pretrained(model_name)
-prompt = "Provide a concise summary of quantum entanglement."
-inputs = tokenizer(prompt, return_tensors="pt")
-outputs = model.generate(**inputs, max_new_tokens=150)
-print(tokenizer.decode(outputs[0], skip_special_tokens=True))
-```
-### Training & Fine-Tuning
-- **Initial Training:** Began with Qwen-32B, calibrated for large-scale text ingestion.
-- **Distillation & Merging:**
-  - Trained on ~1.1B tokens worth of Deepseek-v3 logits.
-  - Employed “fusion merging” to retain as much teacher expertise as possible.
-  - Final step included DPO to improve alignment and reduce model hallucinations.
-- **Continuous Development:** Additional R1 distillations are in progress to further enhance performance and specialization.
-### Performance
-Thanks to a larger parameter count and a richer training corpus, Virtuoso-Medium-v2 delivers high scores across multiple benchmarks (BBH, MMLU-PRO, MATH, etc.). It frequently surpasses other 30B+ models and even some 70B+ architectures in specific tasks.
-### Limitations
-- **Context Length:** 128k Tokens
-- **Knowledge Cut-off:** Training data may not reflect the latest events or developments, leading to gaps in current knowledge beyond June 2024.
-### Ethical Considerations
-- **Content Generation Risks:** Like any language model, Virtuoso-Medium-v2 can potentially generate harmful or biased content if prompted in certain ways.
-### License
-**Virtuoso-Medium-v2 (32B)** is released under the [Apache-2.0 License](https://www.apache.org/licenses/LICENSE-2.0). You are free to use, modify, and distribute this model in both commercial and non-commercial applications, subject to the terms and conditions of the license.
 If you have questions or would like to share your experiences using these models, please connect with us on social media. We’re excited to see what you build—and how these models help you innovate!

+---
+base_model:
+- Qwen/Qwen2.5-32B
+library_name: transformers
+tags:
+- mergekit
+- merge
+license: apache-2.0
+language:
+- zho
+- eng
+- fra
+- spa
+- por
+- deu
+- ita
+- rus
+- jpn
+- kor
+- vie
+- tha
+- ara
+---
+<img src="https://huggingface.co/arcee-train/Virtuoso-Medium-v2/resolve/main/virtuoso-medium.jpg" alt="Virtuoso-Lite Logo" style="display: block; margin: 0 auto;" />
+**Virtuoso-Medium-v2 (32B)** is our next-generation, 32-billion-parameter language model that builds upon the original Virtuoso-Medium architecture. This version is distilled from Deepseek-v3, leveraging an expanded dataset of 5B+ tokens worth of logits. It achieves higher benchmark scores than our previous release (including surpassing Arcee-Nova 2024 in certain tasks).
+### Model Details
+- **Architecture Base:** Qwen-2.5-32B
+- **Parameter Count:** 32B
+- **Tokenizer:**
+  - Initially integrated with Deepseek-v3 tokenizer for logit extraction.
+  - Final alignment uses the Qwen tokenizer, using specialized “tokenizer surgery” for cross-architecture compatibility.
+- **Distillation Data:**
+  - ~1.1B tokens/logits from Deepseek-v3’s training data.
+  - Logit-level distillation using a proprietary “fusion merging” approach afterwards for maximum fidelity.
+- **License:** [Apache-2.0](#license)
+### Background on Deepseek Distillation
+Deepseek-v3 serves as the teacher model, from which we capture logits across billions of tokens. Rather than standard supervised fine-tuning, we apply a full logit-level replication. This ensures more precise transference of knowledge, including advanced reasoning in:
+- Technical and scientific queries
+- Complex code generation
+- Mathematical problem-solving
+### Intended Use Cases
+- **Advanced Chatbots & Virtual Assistants**
+- **Enterprise Data Analysis & Workflow Automation**
+- **Research Simulations & Natural Language Understanding**
+- **Educational Tools for STEM Fields**
+### Evaluations
+<img src="https://huggingface.co/arcee-train/Virtuoso-Lite/resolve/main/Benchmarks.png" alt="Virtuoso-Lite Logo" style="display: block; margin: 0 auto;" />
+### How to Use
+Below is a sample code snippet using `transformers`:
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+model_name = "arcee-ai/virtuoso-medium-v2-32b"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name)
+prompt = "Provide a concise summary of quantum entanglement."
+inputs = tokenizer(prompt, return_tensors="pt")
+outputs = model.generate(**inputs, max_new_tokens=150)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+### Training & Fine-Tuning
+- **Initial Training:** Began with Qwen-32B, calibrated for large-scale text ingestion.
+- **Distillation & Merging:**
+  - Trained on ~1.1B tokens worth of Deepseek-v3 logits.
+  - Employed “fusion merging” to retain as much teacher expertise as possible.
+  - Final step included DPO to improve alignment and reduce model hallucinations.
+- **Continuous Development:** Additional R1 distillations are in progress to further enhance performance and specialization.
+### Performance
+Thanks to a larger parameter count and a richer training corpus, Virtuoso-Medium-v2 delivers high scores across multiple benchmarks (BBH, MMLU-PRO, MATH, etc.). It frequently surpasses other 30B+ models and even some 70B+ architectures in specific tasks.
+### Limitations
+- **Context Length:** 128k Tokens
+- **Knowledge Cut-off:** Training data may not reflect the latest events or developments, leading to gaps in current knowledge beyond June 2024.
+### Ethical Considerations
+- **Content Generation Risks:** Like any language model, Virtuoso-Medium-v2 can potentially generate harmful or biased content if prompted in certain ways.
+### License
+**Virtuoso-Medium-v2 (32B)** is released under the [Apache-2.0 License](https://www.apache.org/licenses/LICENSE-2.0). You are free to use, modify, and distribute this model in both commercial and non-commercial applications, subject to the terms and conditions of the license.
 If you have questions or would like to share your experiences using these models, please connect with us on social media. We’re excited to see what you build—and how these models help you innovate!