Instructions to use DavidAU/Gemma-The-Writer-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DavidAU/Gemma-The-Writer-9B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="DavidAU/Gemma-The-Writer-9B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("DavidAU/Gemma-The-Writer-9B")
model = AutoModelForCausalLM.from_pretrained("DavidAU/Gemma-The-Writer-9B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DavidAU/Gemma-The-Writer-9B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DavidAU/Gemma-The-Writer-9B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DavidAU/Gemma-The-Writer-9B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/DavidAU/Gemma-The-Writer-9B

SGLang

How to use DavidAU/Gemma-The-Writer-9B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DavidAU/Gemma-The-Writer-9B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DavidAU/Gemma-The-Writer-9B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DavidAU/Gemma-The-Writer-9B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DavidAU/Gemma-The-Writer-9B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use DavidAU/Gemma-The-Writer-9B with Docker Model Runner:
```
docker model run hf.co/DavidAU/Gemma-The-Writer-9B
```

DavidAU commited on Sep 26, 2024

Commit

9decbb5

verified ·

1 Parent(s): a1ac37d

Upload folder using huggingface_hub

Browse files

Files changed (9) hide show

model-00001-of-00019.safetensors +3 -0
model-00002-of-00019.safetensors +3 -0
model-00003-of-00019.safetensors +3 -0
model-00004-of-00019.safetensors +3 -0
model-00005-of-00019.safetensors +3 -0
model-00006-of-00019.safetensors +3 -0
model-00007-of-00019.safetensors +3 -0
model-00008-of-00019.safetensors +3 -0
model-00009-of-00019.safetensors +3 -0

model-00001-of-00019.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c91aad38d3b674ac450a880f3a991381fe49d204361d392d36bcbd57ce314cf9
+size 1835008128

model-00002-of-00019.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:20a6526c5f9f48b9cf91b9b46d355e5a99ba621c85d7d8ff09b014ef72a0255d
+size 1835008144

model-00003-of-00019.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8b546f903f6a34e896ab50df4f7c03ee486cbe98f85c16862781434c414b57fb
+size 998311752

model-00004-of-00019.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:aca24befe82bdcb8cb24154e458559c428467e9ed31a1f3ace186cf4462bf315
+size 983653936

model-00005-of-00019.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c16b76ab19cda00e8d39a14746c41251cff949b12d60f150ee04a7acbd04b6af
+size 998304504

model-00006-of-00019.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:62246c1c9794ebf7b4fd65332d7ea07088ac69ae80c92e421f65c0ab75427cfe
+size 983653936

model-00007-of-00019.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4539c80bbd1f8be1499d9b7c795d51ecff661c643072c6903b96346a4bc2694c
+size 998304504

model-00008-of-00019.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b8af882e90df3d66034e968d91d3de1c2d67136c0751e9d03aea0c402b22fa1a
+size 983653928

model-00009-of-00019.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:49657ef4321000d108f69d923438e9f21fd775aff04f461476cfc62f9bbd35f4
+size 998304504