Instructions to use MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2")

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2")
model = AutoModelForMultimodalLM.from_pretrained("MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2

SGLang

How to use MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2 with Docker Model Runner:
```
docker model run hf.co/MarinaraSpaghetti/Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2
```

Doctor-Shotgun_Nous-Capybara-limarpv3-34B-4.2bpw-h6-exl2 / huggingface-metadata.txt

MarinaraSpaghetti

Upload 11 files

1c1e614 verified over 2 years ago

Raw

History Blame Contribute Delete

1.74 kB

	url: https://huggingface.co/Doctor-Shotgun/Nous-Capybara-limarpv3-34B
	branch: main
	download date: 2024-01-23 11:05:12
	sha256sum:
	50326506e1013a0241e17cea0c61c99fd49885c786e09d341de820c8d04a675b model-00001-of-00015.safetensors
	b368d6c84de7ff74e9067b0c57c49fdd64638fdcc952c5cf0f01099dfa240f6d model-00002-of-00015.safetensors
	2b29cd6bb44af323da438a9396a127d6e81431e6a6522d01f7fe2e47ac6d5ef9 model-00003-of-00015.safetensors
	0b4fba4d055b9b0f53ca3a7d6db04323669c47bc1d47d25bf79d3e259252485a model-00004-of-00015.safetensors
	8b3fee91f3165b7445042e0aa68c40c9f5685e68d4b8a98613530a49c2a67810 model-00005-of-00015.safetensors
	889c78efb357399c6c765be2aec92230999e94eb6852a760226f3743e0cfa981 model-00006-of-00015.safetensors
	9fb356ae828a5bb6ea0264baa65f2aabb59051e1d12300654a9ba0f9f49a25b6 model-00007-of-00015.safetensors
	21961647689da028467074187650cbcd1613716fecf35787968ff099313392c2 model-00008-of-00015.safetensors
	1f7e43cebd28012d9b83efd06127561d124ed01f7353f189444e0cf14abf8af4 model-00009-of-00015.safetensors
	d1cf5e17df01fd2ad322e10157c24a2cea535d473fa075cfd6556e57b985e4b0 model-00010-of-00015.safetensors
	299b419afaf0e5c6d9e4545a8d0b934f1967fcbfa39e63fcc037232c9e7b7868 model-00011-of-00015.safetensors
	6b636b1985a33b97b9ee3251e25cf8d3d925712d47f813e5ce07d34bd0f0c0ce model-00012-of-00015.safetensors
	c65c2188003d01f8ebd52e736d0e1648bdbbda3d548db3f680abfdddce0b97f9 model-00013-of-00015.safetensors
	d04e892eebd5c5a074d84192de0b560af455f4c3bb93920fd24aed0f457a5b4f model-00014-of-00015.safetensors
	93edcfed6e59ee0a496fd3a0665b5e88eb6b1698a698baaf2960fed21d3dfffb model-00015-of-00015.safetensors
	386c49cf943d71aa110361135338c50e38beeff0a66593480421f37b319e1a39 tokenizer.model