Instructions to use alpindale/c4ai-command-r-plus-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use alpindale/c4ai-command-r-plus-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="alpindale/c4ai-command-r-plus-GPTQ")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("alpindale/c4ai-command-r-plus-GPTQ")
model = AutoModelForCausalLM.from_pretrained("alpindale/c4ai-command-r-plus-GPTQ")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use alpindale/c4ai-command-r-plus-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "alpindale/c4ai-command-r-plus-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alpindale/c4ai-command-r-plus-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/alpindale/c4ai-command-r-plus-GPTQ

SGLang

How to use alpindale/c4ai-command-r-plus-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "alpindale/c4ai-command-r-plus-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alpindale/c4ai-command-r-plus-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "alpindale/c4ai-command-r-plus-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "alpindale/c4ai-command-r-plus-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use alpindale/c4ai-command-r-plus-GPTQ with Docker Model Runner:
```
docker model run hf.co/alpindale/c4ai-command-r-plus-GPTQ
```

Can VLLM be used for loading?

by wawoshashi - opened Apr 11, 2024

Discussion

wawoshashi

Apr 11, 2024

Can VLLM be used for loading?

Light4Bear

Apr 11, 2024

It was quantized with Aphrodite-engine in mind but it should work on vLLM too.

wawoshashi

Apr 13, 2024

Traceback (most recent call last):
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/entrypoints/openai/api_server.py", line 157, in <module>
    engine = AsyncLLMEngine.from_engine_args(
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 348, in from_engine_args
    engine = cls(
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 311, in __init__
    self.engine = self._init_engine(*args, **kwargs)
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/engine/async_llm_engine.py", line 422, in _init_engine
    return engine_class(*args, **kwargs)
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 110, in __init__
    self.model_executor = executor_class(model_config, cache_config,
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/executor/ray_gpu_executor.py", line 62, in __init__
    self._init_workers_ray(placement_group)
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/executor/ray_gpu_executor.py", line 192, in _init_workers_ray
    self._run_workers(
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/executor/ray_gpu_executor.py", line 324, in _run_workers
    driver_worker_output = getattr(self.driver_worker,
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/worker/worker.py", line 107, in load_model
    self.model_runner.load_model()
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/worker/model_runner.py", line 95, in load_model
    self.model = get_model(
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/model_executor/model_loader.py", line 101, in get_model
    model.load_weights(model_config.model, model_config.download_dir,
  File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/model_executor/models/commandr.py", line 325, in load_weights
    param = params_dict[name]
KeyError: 'model.layers.42.mlp.down_proj.bias'
(RayWorkerVllm pid=16133) ERROR 04-13 15:05:36 ray_utils.py:44] Error executing method load_model. This might cause deadlock in distributed execution.
(RayWorkerVllm pid=16133) ERROR 04-13 15:05:36 ray_utils.py:44] Traceback (most recent call last):
(RayWorkerVllm pid=16133) ERROR 04-13 15:05:36 ray_utils.py:44]   File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/engine/ray_utils.py", line 37, in execute_method
(RayWorkerVllm pid=16133) ERROR 04-13 15:05:36 ray_utils.py:44]     return executor(*args, **kwargs)
(RayWorkerVllm pid=16133) ERROR 04-13 15:05:36 ray_utils.py:44]   File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/worker/worker.py", line 107, in load_model
(RayWorkerVllm pid=16133) ERROR 04-13 15:05:36 ray_utils.py:44]     self.model_runner.load_model()
(RayWorkerVllm pid=16133) ERROR 04-13 15:05:36 ray_utils.py:44]   File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/worker/model_runner.py", line 95, in load_model
(RayWorkerVllm pid=16133) ERROR 04-13 15:05:36 ray_utils.py:44]     self.model = get_model(
(RayWorkerVllm pid=16133) ERROR 04-13 15:05:36 ray_utils.py:44]   File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/model_executor/model_loader.py", line 101, in get_model
(RayWorkerVllm pid=16133) ERROR 04-13 15:05:36 ray_utils.py:44]     model.load_weights(model_config.model, model_config.download_dir,
(RayWorkerVllm pid=16133) ERROR 04-13 15:05:36 ray_utils.py:44]   File "/home/czb/miniconda3/envs/vllm/lib/python3.9/site-packages/vllm/model_executor/models/commandr.py", line 325, in load_weights
(RayWorkerVllm pid=16133) ERROR 04-13 15:05:36 ray_utils.py:44]     param = params_dict[name]
(RayWorkerVllm pid=16133) ERROR 04-13 15:05:36 ray_utils.py:44] KeyError: 'model.layers.42.mlp.down_proj.bias'
(RayWorkerVllm pid=16133) INFO 04-13 15:05:31 pynccl_utils.py:45] vLLM is using nccl==2.18.1 [repeated 2x across cluster]
(RayWorkerVllm pid=15910) WARNING 04-13 15:05:33 custom_all_reduce.py:45] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly. [repeated 2x across cluster]
(RayWorkerVllm pid=16133) INFO 04-13 15:05:35 weight_utils.py:177] Using model weights format ['*.safetensors'] [repeated 2x across cluster]```