Instructions to use deepseek-ai/deepseek-moe-16b-chat with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepseek-ai/deepseek-moe-16b-chat with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepseek-ai/deepseek-moe-16b-chat", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/deepseek-moe-16b-chat", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use deepseek-ai/deepseek-moe-16b-chat with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepseek-ai/deepseek-moe-16b-chat"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/deepseek-moe-16b-chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deepseek-ai/deepseek-moe-16b-chat

SGLang

How to use deepseek-ai/deepseek-moe-16b-chat with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepseek-ai/deepseek-moe-16b-chat" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/deepseek-moe-16b-chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepseek-ai/deepseek-moe-16b-chat" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/deepseek-moe-16b-chat",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use deepseek-ai/deepseek-moe-16b-chat with Docker Model Runner:
```
docker model run hf.co/deepseek-ai/deepseek-moe-16b-chat
```

How to run this model without GPU?

by luminais - opened Aug 23, 2024

Discussion

luminais

Aug 23, 2024

How to run this model without GPU?

Xenek

Sep 6, 2024

Backyard AI lets you use a GUI on OSX & Windows, to download Hugging face Models to run them locally - in a simple chat-app system. It's useful to confirm if a model works on your computer, and to simplify trying a range of different model sizes, to understand the performance and limitations.

There are many other ways, but this is a traditional 'app'.

The most complicated parts are:

Install Backyard.AI, ensure it's up-to-date
Get login for Backyard .AI (I think this is still optional, but I get one so that I have a way to link my use to the people that make the software, so I can easily tell them if things are broken)
You need a hugging face login (you already have that, to be able to ask the question above)
You have to select a model that fits into your RAM. (Eg. 8GB RAM might actually be 6 or 7 GB after disk cache & iGPU assignment) - I find a 8B or 7B param 4 bit Q works in 8 GB Ram on PC with iGPU only)
You have to copy the URL of the repository
You have to paste it into BackyardAI
You have to wait for the large file to download (Usually about 4 or 5 GB or similar - this is a MASSIVE file, if you're only using cellular or terrestrial wireless or satellite data, find a wired connection that uses wifi on the last part between the router and your computer, inside the house, if you can, to reduce Radio Smog (RF Smog) or Spectrum congestion, or satellite or WISP contention, or Cellular network load, etc)
You then create the model. It's a lot of questions but you can sort of work it out without needing to change too much.
You disable as much as you can from starting at boot (all unnecssary apps, and turn off browser Auto starts and background operations and speed boosts) restart, shut down all other apps you couldn't disable
You run Backyard
You load the model
You chat with it

I get about 1-2 tokens a second (a bit less than a word a second, sometimes a bit more with simple words) on 10 to 12 year old hardware in 8 GB of ram.
It's not usable professionally, because the computer is fully occupied while trying to answer, and it takes a very long time to get a few pages out. But more modern computers are completely different. Also, if you can get a TPU or GPU on a high speed bus, such as a PCI bus, it's a potential way to maintain old hardware.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment