Instructions to use Thox-ai/thox-micro-125m-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Thox-ai/thox-micro-125m-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Thox-ai/thox-micro-125m-instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Thox-ai/thox-micro-125m-instruct")
model = AutoModelForCausalLM.from_pretrained("Thox-ai/thox-micro-125m-instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Thox-ai/thox-micro-125m-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Thox-ai/thox-micro-125m-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Thox-ai/thox-micro-125m-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Thox-ai/thox-micro-125m-instruct

SGLang

How to use Thox-ai/thox-micro-125m-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Thox-ai/thox-micro-125m-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Thox-ai/thox-micro-125m-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Thox-ai/thox-micro-125m-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Thox-ai/thox-micro-125m-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Thox-ai/thox-micro-125m-instruct with Docker Model Runner:
```
docker model run hf.co/Thox-ai/thox-micro-125m-instruct
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

thox-micro-125m-instruct

Instruction-tuned ThoxMicro-125M: the THOX-owned 123.6M decoder-only base, given a ChatML turn structure and a real stop token so it answers and stops instead of continuing text. Trained 2026-09-25 on one RTX 4060 Ti 16 GB.

GGUF builds (Q4_0 device target, Q4_K_M, F16): Thox-ai/thox-micro-125m-instruct-GGUF.

What it is, and what it is not

It is a device-role responder. With its system prompt it states what it is, states correctly that the user's data stays on the device, and stops. That is the job the THOX device catalog assigns to the 125M tier; anything conversational routes over the mesh to a larger model.

It is not a general assistant. Without its system prompt it is a weak generalist: it produces list shape without list content, answers "what colour is the sky" with prose, and gets 17 + 25 wrong. That is the capacity of a 125M base with a 2.61 pretraining validation loss, and more SFT on this base will not change it.

Prompt format

ChatML. <|im_start|> = 50257, <|im_end|> = 50258 (EOS and pad). Both reuse unused rows of the 50,304-token vocabulary, so the model and its quantisations are the same size as the base.

<|im_start|>system
You are a THOX model running locally on THOX hardware. Your AI. Your Data. Your Rules.<|im_end|>
<|im_start|>user
Who are you?<|im_end|>
<|im_start|>assistant

The chat template is embedded in tokenizer_config.json and in every GGUF.

Training


Base	`Thox-ai/thox-micro-125m` (LlamaForCausalLM, 123,587,328 params)
Method	full fine-tune, fp32 master weights, bf16 autocast
Loss	assistant spans only, including the closing `<
Data	29,868 conversations: `HuggingFaceTB/smoltalk` (Apache-2.0) + 171 THOX identity / data-locality / scope rows (0.57%)
Steps	1,000 at effective batch 32, max_len 768
Optimiser	AdamW (0.9, 0.95), wd 0.1, lr 2e-4 peak, cosine, 3% warmup
Hardware	1x RTX 4060 Ti 16 GB, 32.8 min wall clock
Best val loss	1.9014 (ppl 6.70) on 301 held-out conversations

smoltalk is the corpus SmolLM2-360M-Instruct was tuned on, so the comparison below measures the base model rather than the data.

Measured head-to-head

One llama.cpp CPU build, 4 threads, temp 0, same seven prompts, same system prompt. Measured on an x86_64 dev box, not a device. Only the ordering between columns transfers.

	untuned base Q4_0	this model Q4_0	SmolLM2-360M-Instruct Q4_0
File	81,319,168 B	81,319,136 B	229,118,976 B
Peak RSS	190 MB	191 MB	473 MB
Decode tok/s	460	~450	~163
Stops on EOS (with system prompt)	0 / 7	7 / 7	1 / 7
Identity	"a senior at the University of Illinois"	correct	invents "Echo" / "EVE"
"Is my data leaving this device?"	unrelated jQuery	"No. I run locally; your data does not leave the device."	wrong without a system prompt
General instructions (list, one word)	no	no	yes

Known issue: system-prompt conditioning

With the exact training system prompt above, the model answers every prompt, including general ones, with its identity line. That prompt appears on every THOX training row and on no smoltalk row, so it learned the prompt as a switch. Use a different system prompt, or none, for general requests; the data-locality answer stays correct either way. The next data mix attaches the THOX system prompt to ~10% of general rows to remove the switch.

Intended use

On-device THOX identity, status and data-locality responses on ThoxMini-class hardware, with general requests routed to a larger model. Not yet validated on a Pi Zero 2 W; the RSS and throughput above are x86 figures.

Out of scope

General chat, factual QA, arithmetic, code, and any safety-critical use.

License

Downloads last month: 784

Safetensors

Model size

0.1B params

Tensor type

F32

Model tree for Thox-ai/thox-micro-125m-instruct

Base model

Thox-ai/thox-micro-125m

Finetuned

(1)

this model

Quantizations

1 model

Thox-ai
/

thox-micro-125m-instruct