Instructions to use ibm-granite/granite-3.0-1b-a400m-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-3.0-1b-a400m-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibm-granite/granite-3.0-1b-a400m-instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-3.0-1b-a400m-instruct")
model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-3.0-1b-a400m-instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ibm-granite/granite-3.0-1b-a400m-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-3.0-1b-a400m-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.0-1b-a400m-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-3.0-1b-a400m-instruct

SGLang

How to use ibm-granite/granite-3.0-1b-a400m-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-3.0-1b-a400m-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.0-1b-a400m-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-3.0-1b-a400m-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.0-1b-a400m-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ibm-granite/granite-3.0-1b-a400m-instruct with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-3.0-1b-a400m-instruct
```

amezasor commited on Oct 20, 2024

Commit

cf89b57

verified ·

1 Parent(s): 08a7e76

update: evar results

Browse files

Files changed (1) hide show

README.md +71 -34

README.md CHANGED Viewed

@@ -2,31 +2,28 @@
 pipeline_tag: text-generation
 inference: false
 license: apache-2.0
-# datasets:
-# metrics:
-# - code_eval
 library_name: transformers
 tags:
 - language
 - granite-3.0
 model-index:
-- name: granite-3.0-1b-a400m-instruct
   results:
   - task:
       type: text-generation
     dataset:
-        type: human-exams
-        name: MMLU
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
     dataset:
-        type: human-exams
-        name: MMLU-Pro
     metrics:
     - name: pass@1
       type: pass@1
@@ -40,17 +37,27 @@ model-index:
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
     dataset:
-        type: commonsense
-        name: WinoGrande
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
@@ -60,7 +67,7 @@ model-index:
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
@@ -70,27 +77,27 @@ model-index:
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
     dataset:
         type: commonsense
-        name: PIQA
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
     dataset:
         type: commonsense
-        name: Hellaswag
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
@@ -100,7 +107,7 @@ model-index:
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
@@ -110,17 +117,17 @@ model-index:
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
     dataset:
         type: reading-comprehension
-        name: SQuAD v2
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
@@ -130,7 +137,7 @@ model-index:
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
@@ -140,7 +147,7 @@ model-index:
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
@@ -150,17 +157,37 @@ model-index:
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
     dataset:
         type: code
-        name: HumanEval
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
@@ -170,7 +197,7 @@ model-index:
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
@@ -180,7 +207,7 @@ model-index:
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
@@ -190,18 +217,28 @@ model-index:
     metrics:
     - name: pass@1
       type: pass@1
-      value:
       veriefied: false
   - task:
       type: text-generation
     dataset:
         type: multilingual
-        name: MGSM
     metrics:
     - name: pass@1
       type: pass@1
-      value:
-      veriefied: false
 ---
 <!-- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/62cd5057674cdb524450093d/1hzxoPwqkBJXshKVVe6_9.png) -->

 pipeline_tag: text-generation
 inference: false
 license: apache-2.0
 library_name: transformers
 tags:
 - language
 - granite-3.0
 model-index:
+- name: granite-3.0-2b-instruct
   results:
   - task:
       type: text-generation
     dataset:
+        type: instruction-following
+        name: IFEval
     metrics:
     - name: pass@1
       type: pass@1
+      value: 32.39
       veriefied: false
   - task:
       type: text-generation
     dataset:
+        type: instruction-following
+        name: MT-Bench
     metrics:
     - name: pass@1
       type: pass@1
     metrics:
     - name: pass@1
       type: pass@1
+      value: 20.35
       veriefied: false
   - task:
       type: text-generation
     dataset:
+        type: human-exams
+        name: MMLU
     metrics:
     - name: pass@1
       type: pass@1
+      value: 32.00
+      veriefied: false
+  - task:
+      type: text-generation
+    dataset:
+        type: human-exams
+        name: MMLU-Pro
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 12.21
       veriefied: false
   - task:
       type: text-generation
     metrics:
     - name: pass@1
       type: pass@1
+      value: 38.40
       veriefied: false
   - task:
       type: text-generation
     metrics:
     - name: pass@1
       type: pass@1
+      value: 47.55
       veriefied: false
   - task:
       type: text-generation
     dataset:
         type: commonsense
+        name: Hellaswag
     metrics:
     - name: pass@1
       type: pass@1
+      value: 65.59
       veriefied: false
   - task:
       type: text-generation
     dataset:
         type: commonsense
+        name: WinoGrande
     metrics:
     - name: pass@1
       type: pass@1
+      value: 61.17
       veriefied: false
   - task:
       type: text-generation
     metrics:
     - name: pass@1
       type: pass@1
+      value: 49.11
       veriefied: false
   - task:
       type: text-generation
     metrics:
     - name: pass@1
       type: pass@1
+      value: 70.12
       veriefied: false
   - task:
       type: text-generation
     dataset:
         type: reading-comprehension
+        name: SQuAD 2.0
     metrics:
     - name: pass@1
       type: pass@1
+      value: 1.27
       veriefied: false
   - task:
       type: text-generation
     metrics:
     - name: pass@1
       type: pass@1
+      value: 41.21
       veriefied: false
   - task:
       type: text-generation
     metrics:
     - name: pass@1
       type: pass@1
+      value: 23.07
       veriefied: false
   - task:
       type: text-generation
     metrics:
     - name: pass@1
       type: pass@1
+      value: 31.77
       veriefied: false
   - task:
       type: text-generation
     dataset:
         type: code
+        name: HumanEvalSynthesis
     metrics:
     - name: pass@1
       type: pass@1
+      value: 30.18
+      veriefied: false
+  - task:
+      type: text-generation
+    dataset:
+        type: code
+        name: HumanEvalExplain
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 26.22
+      veriefied: false
+  - task:
+      type: text-generation
+    dataset:
+        type: code
+        name: HumanEvalFix
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 21.95
       veriefied: false
   - task:
       type: text-generation
     metrics:
     - name: pass@1
       type: pass@1
+      value: 15.40
       veriefied: false
   - task:
       type: text-generation
     metrics:
     - name: pass@1
       type: pass@1
+      value: 26.31
       veriefied: false
   - task:
       type: text-generation
     metrics:
     - name: pass@1
       type: pass@1
+      value: 10.88
       veriefied: false
   - task:
       type: text-generation
     dataset:
         type: multilingual
+        name: PAWS-X (7 langs)
     metrics:
     - name: pass@1
       type: pass@1
+      value: 45.84
+      veriefied: false
+  - task:
+      type: text-generation
+    dataset:
+        type: multilingual
+        name: MGSM (6 langs)
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 11.80
+      veriefied: false
 ---
 <!-- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/62cd5057674cdb524450093d/1hzxoPwqkBJXshKVVe6_9.png) -->