Instructions to use shisa-ai/shisa-jamba-v1-checkpoint-4228 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use shisa-ai/shisa-jamba-v1-checkpoint-4228 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="shisa-ai/shisa-jamba-v1-checkpoint-4228", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("shisa-ai/shisa-jamba-v1-checkpoint-4228", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("shisa-ai/shisa-jamba-v1-checkpoint-4228", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use shisa-ai/shisa-jamba-v1-checkpoint-4228 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "shisa-ai/shisa-jamba-v1-checkpoint-4228"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shisa-ai/shisa-jamba-v1-checkpoint-4228",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/shisa-ai/shisa-jamba-v1-checkpoint-4228

SGLang

How to use shisa-ai/shisa-jamba-v1-checkpoint-4228 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "shisa-ai/shisa-jamba-v1-checkpoint-4228" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shisa-ai/shisa-jamba-v1-checkpoint-4228",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "shisa-ai/shisa-jamba-v1-checkpoint-4228" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "shisa-ai/shisa-jamba-v1-checkpoint-4228",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use shisa-ai/shisa-jamba-v1-checkpoint-4228 with Docker Model Runner:
```
docker model run hf.co/shisa-ai/shisa-jamba-v1-checkpoint-4228
```

Over the weekend after a failed initial run I got excited by Pete's success Jamba Tuning and decided to throw a little compute on a similar-sized dataset (the main shisa-v1 bilingual tuning set).

Like my initial runs, training graphs look fine, but the results were less than spectacular.

Here are the JA MT-Bench evals for the 2416 checkpoint (eval/loss plateau) and the 4228 (3 epoch) tune:

shisa-jamba-v1-checkpoint-2416     2.491525
shisa-jamba-v1-checkpoint-4228     2.508475

You can view the answers in the repo (lots of repetitions and nonsense) and compare to proper JA MT-Bench scores from my testing.

While an "unsuccessful" experiment, it was still worth the practice, although I got a little excited and should have gone w/ my more typical lighter testing obviously.

This kicks off official shisa-v2 base model evaluation. I was a bit hesitant about throwing this model out there (since it's useless as an artifact), but since I've actually made the in-process code available while working on it, I'll share this as well just in case (and to do this writeup).

Here is the current full code/steps for Axolotl training and eval (modified llm-judge inferencing code):

Thanks to Pete for the useful initial report and the axolotl team for their fast integration of Jamba (way better than my raw tune code).

Downloads last month: 8

shisa-ai
/

shisa-jamba-v1-checkpoint-4228

Dataset used to train shisa-ai/shisa-jamba-v1-checkpoint-4228