Instructions to use huihui-ai/Huihui4-8B-A4B-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use huihui-ai/Huihui4-8B-A4B-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="huihui-ai/Huihui4-8B-A4B-v2")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("huihui-ai/Huihui4-8B-A4B-v2")
model = AutoModelForMultimodalLM.from_pretrained("huihui-ai/Huihui4-8B-A4B-v2", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use huihui-ai/Huihui4-8B-A4B-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "huihui-ai/Huihui4-8B-A4B-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "huihui-ai/Huihui4-8B-A4B-v2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/huihui-ai/Huihui4-8B-A4B-v2

SGLang

How to use huihui-ai/Huihui4-8B-A4B-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "huihui-ai/Huihui4-8B-A4B-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "huihui-ai/Huihui4-8B-A4B-v2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "huihui-ai/Huihui4-8B-A4B-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "huihui-ai/Huihui4-8B-A4B-v2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use huihui-ai/Huihui4-8B-A4B-v2 with Docker Model Runner:
```
docker model run hf.co/huihui-ai/Huihui4-8B-A4B-v2
```

Huihui4-8B-A4B-v2 / README.md

huihui-ai

Update README.md

4c7fbba verified 4 months ago

preview code

Raw

History Blame Contribute Delete

3.69 kB

	---
	library_name: transformers
	license: apache-2.0
	pipeline_tag: image-text-to-text
	base_model:
	- google/gemma-4-26B-A4B-it
	tags:
	- sft
	- Moe
	- Pruned
	datasets:
	- huihui-ai/GLM-5.1-Multilingual-STEM
	---

	# 🤖 Huihui4-8B-A4B-v2 Model Card

	## 📌 Overview
	`Huihui4-8B-A4B-v2` is a lightweight MoE (Mixture of Experts) conversational model optimized from Google's `gemma-4-26B-A4B-it` architecture. Through expert pruning and supervised fine-tuning on high-quality dialogue data, the dataset adopts the thinking mode in GLM-5.1 format. This way, in thinking mode, it better reflects the thinking mode of GLM-5.1.
	this model significantly reduces computational overhead while preserving core reasoning and interaction capabilities. It is specifically designed for deployment on consumer-grade hardware and code-related conversational tasks.

	This model is not an ablation variant.


	## 🧱 Architecture & Configuration
	\| Parameter \| Description \|
	\|:---\|:---\|
	\| Base Model \| `google/gemma-4-26B-A4B-it` \|
	\| Total MoE Experts \| 32 (pruned from the original 128) \|
	\| Active Experts per Token \| 8 (maintaining the A4B activation scale) \|
	\| Model Positioning \| Lightweight MoE conversational base / Consumer-hardware friendly \|

	## 📊 Training Data & Methodology
	- Data Source: [huihui-ai/GLM-5.1-Multilingual-STEM](https://huggingface.co/datasets/huihui-ai/GLM-5.1-Multilingual-STEM) carefully extracted from code preference data.
	- Training Method: Supervised Fine-Tuning (SFT).
	- Optimization Goal: Maintain semantic coherence, instruction-following capability, and code context understanding post-pruning.

	## 📈 Evaluation & Performance
	- Evaluation Tool: Quantitative perplexity assessment using the `calculate_perplexity` script.
	- Test Results: Preliminary dialogue tests indicate smooth interactions and stable logic. The model performs reliably in daily conversations and code-assistance tasks, with no significant performance degradation observed after pruning.

	## 💻 Inference & Deployment Recommendations
	- Recommended Frameworks: `vLLM` / `llama.cpp` / `HuggingFace Transformers`
	- VRAM Requirements:
	- `FP16`: < 18GB
	- `INT4/INT8 Quantized`: < 6~9GB (compatible with mainstream single consumer GPUs)
	- Use Cases: Code conversation assistants, lightweight task planning, local deployment prototyping, and baseline validation for MoE pruning/merging techniques.

	## 🗺️ Roadmap
	1. Multi-Domain Fine-Tuning: Further SFT on four distinct datasets to enhance the generalization capabilities of this 32-expert model.
	2. Expert Merging Validation: Experiment with merging the four independently fine-tuned models back into a 128-expert architecture, validating the feasibility of a `"prune → fine-tune → merge"` pipeline.
	3. Core Objective: Ultimately verify the engineering viability of training and iterating on large-scale MoE models using only consumer-grade hardware.
	4. If you're interested, feel free to fine-tune this model on your own datasets. We plan to merge all resulting models into a unified version at the end.

	## 📝 Notes
	- This model represents the initial pruned and fine-tuned iteration of the `Huihui` series. Future updates will involve multi-dataset integration and expert merging.

	## Citation

	```
	@misc{huihui4-8b-a4b-v2,
	title = {{Huihui4-8B-A4B-v2}: A lightweight MoE (Mixture of Experts) conversational model},
	author = {Huihui-ai},
	year = {2026},
	url = {https://hf.co/huihui-ai/Huihui4-8B-A4B-v2}
	}
	```

	## Contact

	If you have any questions, please raise an issue or contact us at [support@huihui.ai](support@huihui.ai).