Instructions to use pszemraj/jamba-900M-v0.13-KIx2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use pszemraj/jamba-900M-v0.13-KIx2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="pszemraj/jamba-900M-v0.13-KIx2")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("pszemraj/jamba-900M-v0.13-KIx2")
model = AutoModelForCausalLM.from_pretrained("pszemraj/jamba-900M-v0.13-KIx2", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use pszemraj/jamba-900M-v0.13-KIx2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "pszemraj/jamba-900M-v0.13-KIx2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pszemraj/jamba-900M-v0.13-KIx2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/pszemraj/jamba-900M-v0.13-KIx2

SGLang

How to use pszemraj/jamba-900M-v0.13-KIx2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "pszemraj/jamba-900M-v0.13-KIx2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pszemraj/jamba-900M-v0.13-KIx2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "pszemraj/jamba-900M-v0.13-KIx2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pszemraj/jamba-900M-v0.13-KIx2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use pszemraj/jamba-900M-v0.13-KIx2 with Docker Model Runner:
```
docker model run hf.co/pszemraj/jamba-900M-v0.13-KIx2
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

jamba-900M-v0.13-KIx2

The API widget is off as it isn't supported by hf yet - try the Colab

This is a pretraining experiment on the jamba arch as a "smol MoE".

Details:

pretrained at context length 16384
seen approx 20b tokens
uses Claude3 tokenizer (as hf GPT2 tokenizer)
hidden size 1024, 12 layers, 8 experts

achieves the following results on the evaluation set (most recent dataset):

Loss: 3.0366
Accuracy: 0.4514
Num Input Tokens Seen: 1975517184

if I pretrain it further, other versions will be in new repos with incremented version (this is v0.13)

Quick eval

Quick eval for: pszemraj/jamba-H1024_L12-v0.13-KIx2

hf (pretrained=pszemraj/jamba-H1024_L12-v0.13-KIx2,trust_remote_code=True,dtype=float), gen_kwargs: (None), limit: 0.9999, num_fewshot: None, batch_size: 8

Tasks	Version	Filter	Metric	Value		Stderr
winogrande	1	none	acc	0.5067	±	0.0141
piqa	1	none	acc	0.5912	±	0.0138
		none	acc_norm	0.5951	±	0.0138
openbookqa	1	none	acc	0.1800	±	0.0172
		none	acc_norm	0.2920	±	0.0204
lambada_openai	1	none	perplexity	103.1241	±	8.5843
		none	acc	0.2502	±	0.0122
boolq	2	none	acc	0.6196	±	0.0136
arc_easy	1	none	acc	0.3836	±	0.0137
		none	acc_norm	0.3694	±	0.0136