Instructions to use ibm-granite/granite-3.0-1b-a400m-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-3.0-1b-a400m-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibm-granite/granite-3.0-1b-a400m-instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-3.0-1b-a400m-instruct")
model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-3.0-1b-a400m-instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ibm-granite/granite-3.0-1b-a400m-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-3.0-1b-a400m-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.0-1b-a400m-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-3.0-1b-a400m-instruct

SGLang

How to use ibm-granite/granite-3.0-1b-a400m-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-3.0-1b-a400m-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.0-1b-a400m-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-3.0-1b-a400m-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.0-1b-a400m-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ibm-granite/granite-3.0-1b-a400m-instruct with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-3.0-1b-a400m-instruct
```

davanstrien HF Staff commited on Oct 23, 2024

Commit

dca1980

verified ·

1 Parent(s): 0643df2

Add base model metadata

Browse files

This will expose the link between the base_model and this fine tune and ensure you get a nice model tree (example screenshot below)

![Screenshot 2024-10-23 at 13.59.08.png](https://cdn-uploads.huggingface.co/production/uploads/60107b385ac3e86b3ea4fc34/BSP3IpkcqhvtbB6QZRDZQ.png)

Files changed (1) hide show

README.md +57 -55

README.md CHANGED Viewed

@@ -12,8 +12,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: instruction-following
-        name: IFEval
     metrics:
     - name: pass@1
       type: pass@1
@@ -22,8 +22,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: instruction-following
-        name: MT-Bench
     metrics:
     - name: pass@1
       type: pass@1
@@ -32,8 +32,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: human-exams
-        name: AGI-Eval
     metrics:
     - name: pass@1
       type: pass@1
@@ -42,18 +42,18 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: human-exams
-        name: MMLU
     metrics:
     - name: pass@1
       type: pass@1
-      value: 32.00
       veriefied: false
   - task:
       type: text-generation
     dataset:
-        type: human-exams
-        name: MMLU-Pro
     metrics:
     - name: pass@1
       type: pass@1
@@ -62,18 +62,18 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: commonsense
-        name: OBQA
     metrics:
     - name: pass@1
       type: pass@1
-      value: 38.40
       veriefied: false
   - task:
       type: text-generation
     dataset:
-        type: commonsense
-        name: SIQA
     metrics:
     - name: pass@1
       type: pass@1
@@ -82,8 +82,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: commonsense
-        name: Hellaswag
     metrics:
     - name: pass@1
       type: pass@1
@@ -92,8 +92,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: commonsense
-        name: WinoGrande
     metrics:
     - name: pass@1
       type: pass@1
@@ -102,8 +102,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: commonsense
-        name: TruthfulQA
     metrics:
     - name: pass@1
       type: pass@1
@@ -112,8 +112,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: reading-comprehension
-        name: BoolQ
     metrics:
     - name: pass@1
       type: pass@1
@@ -122,8 +122,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: reading-comprehension
-        name: SQuAD 2.0
     metrics:
     - name: pass@1
       type: pass@1
@@ -132,8 +132,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: reasoning
-        name: ARC-C
     metrics:
     - name: pass@1
       type: pass@1
@@ -142,8 +142,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: reasoning
-        name: GPQA
     metrics:
     - name: pass@1
       type: pass@1
@@ -152,8 +152,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: reasoning
-        name: BBH
     metrics:
     - name: pass@1
       type: pass@1
@@ -162,8 +162,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: code
-        name: HumanEvalSynthesis
     metrics:
     - name: pass@1
       type: pass@1
@@ -172,8 +172,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: code
-        name: HumanEvalExplain
     metrics:
     - name: pass@1
       type: pass@1
@@ -182,8 +182,8 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: code
-        name: HumanEvalFix
     metrics:
     - name: pass@1
       type: pass@1
@@ -192,53 +192,55 @@ model-index:
   - task:
       type: text-generation
     dataset:
-        type: code
-        name: MBPP
     metrics:
     - name: pass@1
       type: pass@1
-      value: 15.40
-      veriefied: false
   - task:
       type: text-generation
     dataset:
-        type: math
-        name: GSM8K
     metrics:
     - name: pass@1
       type: pass@1
       value: 26.31
-      veriefied: false
   - task:
       type: text-generation
     dataset:
-        type: math
-        name: MATH
     metrics:
     - name: pass@1
       type: pass@1
       value: 10.88
-      veriefied: false
   - task:
       type: text-generation
     dataset:
-        type: multilingual
-        name: PAWS-X (7 langs)
     metrics:
     - name: pass@1
       type: pass@1
       value: 45.84
-      veriefied: false
   - task:
       type: text-generation
     dataset:
-        type: multilingual
-        name: MGSM (6 langs)
     metrics:
     - name: pass@1
       type: pass@1
-      value: 11.80
-      veriefied: false
 ---
 <!-- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/62cd5057674cdb524450093d/1hzxoPwqkBJXshKVVe6_9.png) -->

   - task:
       type: text-generation
     dataset:
+      type: instruction-following
+      name: IFEval
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: instruction-following
+      name: MT-Bench
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: human-exams
+      name: AGI-Eval
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: human-exams
+      name: MMLU
     metrics:
     - name: pass@1
       type: pass@1
+      value: 32
       veriefied: false
   - task:
       type: text-generation
     dataset:
+      type: human-exams
+      name: MMLU-Pro
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: commonsense
+      name: OBQA
     metrics:
     - name: pass@1
       type: pass@1
+      value: 38.4
       veriefied: false
   - task:
       type: text-generation
     dataset:
+      type: commonsense
+      name: SIQA
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: commonsense
+      name: Hellaswag
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: commonsense
+      name: WinoGrande
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: commonsense
+      name: TruthfulQA
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: reading-comprehension
+      name: BoolQ
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: reading-comprehension
+      name: SQuAD 2.0
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: reasoning
+      name: ARC-C
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: reasoning
+      name: GPQA
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: reasoning
+      name: BBH
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: code
+      name: HumanEvalSynthesis
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: code
+      name: HumanEvalExplain
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: code
+      name: HumanEvalFix
     metrics:
     - name: pass@1
       type: pass@1
   - task:
       type: text-generation
     dataset:
+      type: code
+      name: MBPP
     metrics:
     - name: pass@1
       type: pass@1
+      value: 15.4
+      veriefied: false
   - task:
       type: text-generation
     dataset:
+      type: math
+      name: GSM8K
     metrics:
     - name: pass@1
       type: pass@1
       value: 26.31
+      veriefied: false
   - task:
       type: text-generation
     dataset:
+      type: math
+      name: MATH
     metrics:
     - name: pass@1
       type: pass@1
       value: 10.88
+      veriefied: false
   - task:
       type: text-generation
     dataset:
+      type: multilingual
+      name: PAWS-X (7 langs)
     metrics:
     - name: pass@1
       type: pass@1
       value: 45.84
+      veriefied: false
   - task:
       type: text-generation
     dataset:
+      type: multilingual
+      name: MGSM (6 langs)
     metrics:
     - name: pass@1
       type: pass@1
+      value: 11.8
+      veriefied: false
+base_model:
+- ibm-granite/granite-3.0-1b-a400m-base
 ---
 <!-- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/62cd5057674cdb524450093d/1hzxoPwqkBJXshKVVe6_9.png) -->