Text Generation
Transformers
Safetensors
granite_switch
granite
granite-switch
lora
adapters
mixture-of-adapters
conversational
Instructions to use barha/granite-switch-4.0-350m-demo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use barha/granite-switch-4.0-350m-demo with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="barha/granite-switch-4.0-350m-demo") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("barha/granite-switch-4.0-350m-demo", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use barha/granite-switch-4.0-350m-demo with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "barha/granite-switch-4.0-350m-demo" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "barha/granite-switch-4.0-350m-demo", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/barha/granite-switch-4.0-350m-demo
- SGLang
How to use barha/granite-switch-4.0-350m-demo with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "barha/granite-switch-4.0-350m-demo" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "barha/granite-switch-4.0-350m-demo", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "barha/granite-switch-4.0-350m-demo" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "barha/granite-switch-4.0-350m-demo", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use barha/granite-switch-4.0-350m-demo with Docker Model Runner:
docker model run hf.co/barha/granite-switch-4.0-350m-demo
Add model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,80 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
base_model: ibm-granite/granite-4.0-350m
|
| 3 |
+
library_name: transformers
|
| 4 |
+
pipeline_tag: text-generation
|
| 5 |
+
tags:
|
| 6 |
+
- granite
|
| 7 |
+
- granite-switch
|
| 8 |
+
- lora
|
| 9 |
+
- adapters
|
| 10 |
+
- mixture-of-adapters
|
| 11 |
+
license: apache-2.0
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# Granite Switch 4.0 350M — 3-Adapter Demo
|
| 15 |
+
|
| 16 |
+
A single [Granite Switch](https://github.com/generative-computing/granite-switch)
|
| 17 |
+
checkpoint built on [`ibm-granite/granite-4.0-350m`](https://huggingface.co/ibm-granite/granite-4.0-350m)
|
| 18 |
+
with **three task LoRA adapters embedded in one model**. Each adapter is activated
|
| 19 |
+
by a control token, so one deployed checkpoint serves three different tasks — no
|
| 20 |
+
adapter swapping, no separate model loads.
|
| 21 |
+
|
| 22 |
+
This is the multi-adapter companion to the single-adapter
|
| 23 |
+
[`barha/granite-switch-4.0-350m-cti`](https://huggingface.co/barha/granite-switch-4.0-350m-cti).
|
| 24 |
+
|
| 25 |
+
## Embedded adapters
|
| 26 |
+
|
| 27 |
+
| # | Adapter | Control token | Task | Source LoRA |
|
| 28 |
+
|---|---------|---------------|------|-------------|
|
| 29 |
+
| 1 | `cti-technique-mapping` | `<\|cti-technique-mapping\|>` | Map a CTI description → MITRE ATT&CK technique ID | [barha/granite-cti-technique-mapping-350m-lora](https://huggingface.co/barha/granite-cti-technique-mapping-350m-lora) |
|
| 30 |
+
| 2 | `text-to-json` | `<\|text-to-json\|>` | Natural language + schema → schema-conforming JSON | [barha/granite-text-to-json-350m-lora](https://huggingface.co/barha/granite-text-to-json-350m-lora) |
|
| 31 |
+
| 3 | `genai-attack-vector` | `<\|genai-attack-vector\|>` | Classify a GenAI security incident into 1 of 14 attack-vector classes | [barha/granite-genai-attack-vector-350m-lora](https://huggingface.co/barha/granite-genai-attack-vector-350m-lora) |
|
| 32 |
+
|
| 33 |
+
All three adapters share an identical LoRA shape (rank 16, alpha 32, on the fused
|
| 34 |
+
q/k/v/o attention projections and the `input_linear` / `output_linear` MLP
|
| 35 |
+
projections), which is what lets them stack cleanly into one switch checkpoint.
|
| 36 |
+
|
| 37 |
+
Control token IDs: `100352`, `100353`, `100354` (3 new tokens; vocab 100355).
|
| 38 |
+
|
| 39 |
+
## Per-adapter evaluation (on granite-4.0-350m)
|
| 40 |
+
|
| 41 |
+
| Adapter | Metric | Score | n |
|
| 42 |
+
|---------|--------|-------|---|
|
| 43 |
+
| text-to-json | Key-F1 (headline) | **98.4** | 2000 |
|
| 44 |
+
| genai-attack-vector | Accuracy | **74.8%** (166/222) | 222 |
|
| 45 |
+
| cti-technique-mapping | — | see source repo | — |
|
| 46 |
+
|
| 47 |
+
These are the standalone scores of each source LoRA; embedding them in the switch
|
| 48 |
+
does not change adapter weights.
|
| 49 |
+
|
| 50 |
+
## How adapter selection works
|
| 51 |
+
|
| 52 |
+
Granite Switch routes per request via the control token: place the adapter's
|
| 53 |
+
control token in the prompt (the chat template handles placement) and the switch
|
| 54 |
+
activates that adapter for the turn. With no control token, the base model runs
|
| 55 |
+
unmodified.
|
| 56 |
+
|
| 57 |
+
## Usage
|
| 58 |
+
|
| 59 |
+
Compose / inference follow the standard Granite Switch flow — see the
|
| 60 |
+
[granite-switch repo](https://github.com/generative-computing/granite-switch) and
|
| 61 |
+
its tutorials. The checkpoint loads as a `GraniteSwitchForCausalLM`; `adapter_index.json`
|
| 62 |
+
lists the adapter → control-token mapping and `io_configs/<adapter>/io.yaml` carries
|
| 63 |
+
each adapter's I/O contract.
|
| 64 |
+
|
| 65 |
+
## Build
|
| 66 |
+
|
| 67 |
+
Composed with `granite_switch.composer.compose_granite_switch`:
|
| 68 |
+
|
| 69 |
+
```bash
|
| 70 |
+
python -m granite_switch.composer.compose_granite_switch \
|
| 71 |
+
--base-model ibm-granite/granite-4.0-350m \
|
| 72 |
+
--technology lora \
|
| 73 |
+
--adapters cti-technique-mapping text-to-json genai-attack-vector \
|
| 74 |
+
--output ./granite-switch-4.0-350m-demo
|
| 75 |
+
```
|
| 76 |
+
|
| 77 |
+
Base params 352M → composed 359M (+2.0%). See `BUILD.md` and `compose_report.json`
|
| 78 |
+
in this repo for the full composition report.
|
| 79 |
+
|
| 80 |
+
License: Apache-2.0
|