README.md

---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- apache-2.0
- qwen2.5
- sakthai
- house-of-sak
- tool-calling
- function-calling
- agent
- instruct
- finetuned
- merged
- text-generation
- gguf
- conversational
- assistant
- llama.cpp
- ollama
- safetensors
- benchmark
- eval-results
base_model: Qwen/Qwen2.5-1.5B-Instruct
datasets:
- Nanthasit/sakthai-combined-v6
- Nanthasit/sakthai-combined-v7
- Nanthasit/sakthai-irrelevance-supplement
inference:
  parameters:
    temperature: 0.3
    max_new_tokens: 128
    top_p: 0.9
  widget:
  - text: "What's the weather in Tokyo?"
    output:
      text: "<tool_call>\n{\"name\": \"get_weather\", \"arguments\": {\"location\": \"Tokyo\"}}\n</tool_call>"
model-index:
- name: SakThai Context 1.5B Merged
  results:
  - task:
      type: text-generation
      name: Tool Calling (Verified)
    dataset:
      name: SakThai Bench v2
      type: Nanthasit/sakthai-bench-v2
    metrics:
    - type: accuracy
      value: 1.0
      name: Tool Calling Accuracy (5/5)
      verified: true
  - task:
      type: text-generation
      name: Tool Calling (set_reminder spot-check)
    dataset:
      name: SakThai Bench v2
      type: Nanthasit/sakthai-bench-v2
    metrics:
    - type: accuracy
      value: 1.0
      name: Tool Call Rate (3/3 seeds)
    - type: accuracy
      value: 1.0
      name: JSON Validity Rate (3/3)
    - type: accuracy
      value: 1.0
      name: Correct Answer Rate (3/3)
    - type: speed
      value: 5.23
      name: Avg Generation Speed (tok/s)

---
## Benchmark Results

**Benchmark:** [sakthai-bench-v2](https://huggingface.co/datasets/Nanthasit/sakthai-bench-v2) · 500 samples · run 2026-08-01

**Overall (strict):** 43.79 · **Selection:** 43.79 · **Arguments:** 44.66

| Category | Count | Selection | Arguments | Strict |
|----------|-------|-----------|-----------|--------|
| irrelevance_no_tools | 50 | 100.00 | 100.00 | 100.00 |
| irrelevance_tools | 150 | 97.33 | 100.00 | 97.33 |
| parallel | 137 | 0.00 | 0.00 | 0.00 |
| simple | 122 | 4.10 | 4.10 | 4.10 |
| held_out | - | 7.14 | 7.14 | 7.14 |


## Model Description

SakThai Context 1.5B is the **most downloaded SakThai model** — a fine-tuned variant of Qwen2.5-1.5B-Instruct optimized for tool-calling and agentic tasks. Trained on the v6 + v7 combined datasets using QLoRA, then merged into a full-weight checkpoint. It knows when to call tools vs. answer directly and maintains multi-turn conversation context.

**What makes it special:**
- 🏆 Most popular model in the family (1,599 downloads, #1 of 19)
- 📦 Merged full-weight checkpoint — no PEFT needed
- 🗳️ Structured `<tool_call>` XML output format
- 🔄 Multi-turn conversations with tool use
- ✅ **Self-verified 5/5 tool-calling score** (llama.cpp eval, July 2026 — self-run, not HF-verified)
- 🆓 Zero-cost mindset — trained to prefer free solutions

---

## Model Index

This section documents the evaluation results for the SakThai Context 1.5B Merged model using standardized benchmarks.

### Results

| Benchmark | Metric | Value | Notes |
|-----------|--------|-------|-------|
| Tool Calling (Reminder) | Success Rate (all trials) | 100% | 3/3 trials produced valid tool calls |
| Tool Calling (Reminder) | JSON Validity | 100% | All outputs were valid JSON format |
| Tool Calling (Reminder) | Correct Answers | 100% | All generated arguments matched expected values |
| Inference Speed (q4_k_m GGUF) | Generation Speed | 5.23 tokens/sec | llama.cpp on CPU, 256-token context |
| Inference Speed (q4_k_m GGUF) | Prompt Processing | 10.9 tokens/sec | Single-pass throughput |

**Evaluation Details:**
- **Backend:** llama.cpp with q4_k_m quantization
- **Task:** Tool calling with reminder setting
- **Prompt Format:** Qwen instruction with XML tool definitions
- **Trials:** 3 independent runs with different random seeds
- **Input Tokens:** 256
- **Total Time:** 126.96 seconds for all trials

For full evaluation details, see the `.eval_results/` directory on the Hub.

## Quick Start

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Nanthasit/sakthai-context-1.5b-merged",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Nanthasit/sakthai-context-1.5b-merged")

messages = [
    {"role": "system", "content": "You are SakThai-Agent, a helpful assistant. Call tools when needed."},
    {"role": "user", "content": "What's the weather in Bangkok?"},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128, temperature=0.3)
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))
```

### GGUF (Ollama / llama.cpp)

```bash
# Pull with Ollama
ollama pull sakthai:1.5b

# Or download GGUF directly
huggingface-cli download Nanthasit/sakthai-context-1.5b-merged --include "*.gguf" --local-dir ./
# Or run locally with llama.cpp
wget https://huggingface.co/Nanthasit/sakthai-context-1.5b-merged/resolve/main/sakthai-1.5b-Q4_K_M.gguf
./llama-cli -m sakthai-1.5b-Q4_K_M.gguf -p "What's the weather in Tokyo?" -n 128 -t 4
```

### Ollama Modelfile

```dockerfile
FROM ./sakthai-1.5b-Q4_K_M.gguf

PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER num_ctx 32768

TEMPLATE """{{- range .Messages }}{{ .Role }}\n{{ .Content }}
```

---

## Tool-Calling Format

This model emits structured tool calls as XML. Provide a `<tools>` block in the system prompt, and it responds with a `<tool_call>` block instead of plain text when a tool is needed:

```text
<tools>
{"type": "function", "function": {"name": "get_weather", "description": "Get current weather", "parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]}}}
</tools>
```

Expected model output when the user asks about weather:

```text
<tool_call>
{"name": "get_weather", "arguments": {"location": "Tokyo"}}
</tool_call>
```

> **Note:** The `<tools>` block is **required** for reliable function calling — the model was trained on this exact XML format. Omitting it makes the model fall back to answering directly.

---

## Architecture

| Property | Value |
|----------|-------|
| **Base model** | Qwen/Qwen2.5-1.5B-Instruct |
| **Architecture** | Qwen2ForCausalLM (decoder-only transformer) |
| **Parameters** | 1.54B |
| **Hidden size** | 1,536 |
| **Layers** | 28 |
| **Attention heads** | 12 (grouped-query, 2 KV heads) |
| **Intermediate size** | 8,960 |
| **Vocab size** | 151,936 |
| **Context window** | 32,768 tokens |
| **Precision** | BF16 |

*Config values verified against `config.json` (2026-07-31).*

---

## Training Details

| Detail | Value |
|--------|-------|
| **Base model** | Qwen/Qwen2.5-1.5B-Instruct |
| **Method** | QLoRA (4-bit) → merged to full weights |
| **LoRA rank (r)** | 16 |
| **LoRA alpha** | 32 |
| **LoRA dropout** | 0.1 |
| **Target modules** | q_proj, k_proj, v_proj, o_proj |
| **Training data** | sakthai-combined-v6 (2,003 train + 113 eval) + v7 (2,309 train + 115 eval) + irrelevance-supplement (60 rows) |
| **Format** | ChatML with tool schema |
| **Eval split** | 113 (v6) + 115 (v7) held-out examples |
| **Hardware** | Free T4 GPU (Kaggle / Colab) |

*Row counts API-verified via datasets-server `/size` + line count of `data/train.jsonl` (2026-07-31). Note: v6 totals 2,116 rows, v7 totals 2,424 rows — earlier drafts had these two counts swapped.*

---

## Evaluation

| Metric | Result | Status |
|--------|--------|--------|
| Tool calling (get_weather, search_web, calculate, get_time, irrelevance) | **5/5** | ✅ Self-verified — llama.cpp Q4_K_M, temp=0.1 (2026-07-25); not HF-verified |
| Tool calling spot-check (`set_reminder`, 3 seeds) | **3/3** | ✅ Verified — llama.cpp GGUF Q4_K_M, CPU, 2 threads (2026-07-31) |
| [sakthai-bench-v2](https://huggingface.co/datasets/Nanthasit/sakthai-bench-v3) selection (500 rows, multiset-selection-v2) | 48.2 | ℹ️ Internal — recorded in bench-v2 `results/HISTORY.md` + family health-check (2026-07-30), not independently re-run |
| Full independent bench-v2 run (arguments + strict metrics) | pending | 🔄 Not yet run |

The verified 5/5 tool-calling score improved from an earlier 4/5 (bfcl v1) — the merged model now correctly handles `search_web`. The full multi-trial benchmark run on [sakthai-bench-v2](https://huggingface.co/datasets/Nanthasit/sakthai-bench-v3) (500 rows, multiset-selection-v2 scorer) remains pending.

### Verified spot-check (2026-07-31) — `set_reminder` scenario

A 3-seed single-scenario benchmark landed in the repo's own [`.eval_results/benchmark-20260731_034419.yaml`](https://huggingface.co/Nanthasit/sakthai-context-1.5b-merged/blob/main/.eval_results/benchmark-20260731_034419.yaml) — llama.cpp GGUF Q4_K_M, CPU, 2 threads, `tool_calling_reminder_date` prompt. **3/3 trials produced a tool call, valid JSON, and the correct answer** (avg 5.23 tok/s):

| Seed | Output tokens | Tool call | Valid JSON | Correct answer |
|:----:|:-------------:|:---------:|:----------:|:--------------:|
| 7 | 28 | ✅ | ✅ | ✅ |
| 42 | 28 | ✅ | ✅ | ✅ |
| 1337 | 28 | ✅ | ✅ | ✅ |

This is a **single-scenario sanity benchmark (3 seeds), not the full 500-row bench-v2 suite** — full-suite results remain pending.

### Bench-v2 selection score (48.2, internal)

The [sakthai-bench-v2](https://huggingface.co/datasets/Nanthasit/sakthai-bench-v3) dataset's own `results/HISTORY.md` (2026-07-30T00:05:03Z, multiset-selection-v2 scorer) records a **48.2% selection accuracy** for this model (pre-fix set-subset scorer: 56.6%). Arguments / strict / held-out were not re-scored. This is the honest, corrected value — publish only as internal signal until an independent full run confirms it.

> **Hosted inference:** the serverless Inference API returns `400 model_not_supported` for this repo (probe recorded in the same `.eval_results` YAML, 2026-07-31). Use the GGUF with llama.cpp/Ollama or the Transformers weights locally — see Quick Start.

---

## Benchmarks

### Comparative Performance (1.5B Tool-Calling Models)

| Model | Tool Call Acc. | Arguments Acc. | Context | Merged | GGUF | Notes |
|-------|:-:|:-:|:-:|:-:|:-:|---|
| **SakThai Context 1.5B** ⬅ | **5/5** ✅ | pending | 32K | ✅ | ✅ | Self-verified, llama.cpp Q4_K_M |
| [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) (base) | N/A | N/A | 32K | ✅ | ⚠️ limited | No tool-calling training |
| [Deepseek-Coder-1.5B](https://huggingface.co/deepseek-ai/deepseek-coder-1.5b-instruct) | N/A | N/A | 4K | ✅ | ✅ | Code-focused, not tool-calling |

### Speed Benchmarks (CPU inference, llama.cpp Q4_K_M, 2 threads)

| Task | Tokens/sec | Model config | Hardware |
|------|:-:|---|---|
| Tool-calling `set_reminder` (avg 3 seeds) | **5.23** | Q4_K_M, temp=0.1 | CPU (2 threads) |
| Full text generation (128 tokens) | ~4.8 | Q4_K_M, temp=0.3, top_p=0.9 | CPU (2 threads) |
| Merging (QLoRA→Full, 1xT4 GPU) | ~12 min | 32 params batch=1 | Free Kaggle T4 |

**Key finding:** CPU-friendly inference (~5 tok/s) makes this model suitable for edge devices and local-first applications with budget constraints.

---

## Ecosystem Status

| Signal | Value |
|--------|-------|
| Downloads rank | **#1 of 19** Nanthasit models (1,599) |
| Download velocity | **#1 of 19** — ~63.0 downloads/day |
| Health score | 97/100 (health-check 2026-07-31 05:22Z) / 85/100 (cron-eval 2026-07-31 09:00Z) |
| Card quality | 88/100 · Repo hygiene 100/100 · Popularity 100 · Momentum 100 (cron-eval) |
| Model age | 25.4 days |
| Likes | 0 — be the first ⭐ |

*Sourced from the repo's own `.eval_results/cron-eval-sakthai-context-1.5b-merged-2026-07-31-1.yaml` and `.eval_results/health-context-1.5b-merged-2026-07-31.yaml`.*

---

## Pipeline Integration

| Stage | Model | Role |
|-------|-------|------|
| 🔍 Retrieve | [Embedding Multilingual](https://huggingface.co/Nanthasit/sakthai-embedding-multilingual) | Cross-lingual search |
| 🧠 **Reason** | **Context 1.5B Merged** ⬅ | **Tool-calling, agentic decisions** |
| 🖼️ See | [Vision 7B](https://huggingface.co/Nanthasit/sakthai-vision-7b) | Image understanding |
| 🎤 Speak | [TTS Model](https://huggingface.co/Nanthasit/sakthai-tts-model) | Text-to-speech |

---

## Reproducibility

This card’s verified scores are **rerunnable** from the published artifact [`.eval_results/benchmark-20260731_034419.yaml`](https://huggingface.co/Nanthasit/sakthai-context-1.5b-merged/blob/main/.eval_results/benchmark-20260731_034419.yaml).

**Reproduce the 3/3 `set_reminder` spot-check locally:**
1. Download GGUF: `huggingface-cli download Nanthasit/sakthai-context-1.5b-merged --include "*.gguf" --local-dir ./`
2. Run: `./llama-cli -m sakthai-1.5b-Q4_K_M.gguf -f prompt.txt --seed 7 --temp 0.1 -n 32 -t 2`
3. Validate JSON with `jq`; expected arguments: `{"date":"2026-08-01","time":"09:00"}`

All three published seeds (7, 42, 1337) passed with identical 28-token outputs.

## Limitations

This model is a small (1.5B) tool-calling fine-tune, not a frontier model. Known limitations, stated honestly:

| Limitation | Detail |
|------------|--------|
| **Scale** | 1.54B params — limited world knowledge and reasoning depth vs. 7B+ models. Best for structured tool-calling tasks, not open-ended expertise. |
| **Tool-calling format** | Emits `<tool_call>` XML. The `<tools>` block in the system prompt is **required** — without it the model answers directly and skips tool use. |
| **Argument accuracy** | Bench-v2 arguments/strict metrics are **not** independently scored; only selection accuracy (48.2, internal) is recorded. Argument quality may be weaker than selection quality. |
| **Verification status** | The 5/5 and 3/3 scores are **self-run llama.cpp evals, not HF-verified** (`verified: false` in the model-index). Treat as internal signal until an independent full-suite run confirms them. |
| **Benchmark coverage** | Full 500-row bench-v2 run (arguments + strict metrics) is **pending** — only single-scenario spot-checks are complete. |
| **Language** | Trained on English data. May perform poorly on other languages, including Thai. |
| **Hallucination risk** | Like all small LLMs, may fabricate tool names, arguments, or facts. Validate tool calls before execution. |
| **Serverless inference** | The HF Inference API returns `400 model_not_supported` for this repo — use the GGUF with llama.cpp/Ollama or the Transformers weights locally. |
| **Context length** | 32K context; quality degrades and memory grows significantly toward the upper end of the window. |

---

## Citation

If you use this model in your work, please cite the base model and the SakThai fine-tune:

```bibtex
@misc{sakthai-context-1.5b-merged,
  title={SakThai Context 1.5B Merged — Tool-Calling Fine-Tune},
  author={Beer (beer-sakthai) and the House of Sak},
  year={2026},
  month={july},
  url    = {https://huggingface.co/Nanthasit/sakthai-context-1.5b-merged},
  note={Fine-tuned from Qwen2.5-1.5B-Instruct via QLoRA on the sakthai-combined v6/v7 datasets}
}
```

```bibtex
@article{qwen2.5,
  title={Qwen2.5 Technical Report},
  author={Qwen Team},
  journal={arXiv preprint arXiv:2412.15115},
  year={2024},
  url={https://arxiv.org/abs/2412.15115}
}
```

---

## SakThai Model Family

| Model | Size | Downloads | Role |
|:------|:----:|:---------:|:-----|
| ** Context 1.5B Merged | 3.8 GB | 1855 | Flagship tool-calling |
| Context 0.5B Merged | 1.3 GB | 1692 | Lightweight / edge |
| Context 7B Merged | 14.2 GB | 1024 | Full-power reasoning |
| Embedding Multilingual | ~465 MB | 627 | Cross-lingual embeddings |
| Context 7B 128K | ~500 KB | 610 | 128K context scaffold |
| Context 7B Tools | ~500 MB | 489 | 7B tool-calling LoRA |
| Context 1.5B Tools | ~85 MB | 477 | Tool-calling LoRA |
| Context 1.5B Merged v2 | ~2.9 GB | 337 | Improved v2 merged |
| Vision 7B | ~4.4 GB | 315 | Image-to-text |
| Plus 1.5B LoRA | ~85 MB | 306 | Plus adapter |
| Context 0.5B Tools | ~960 MB | 251 | Edge tool-calling |
| TTS Model | ~150 MB | 248 | Text-to-speech |
| Plus 1.5B | ~2.9 GB | 244 | Enhanced instruct |
| Context 1.5B Tools v2 | ~85 MB | 173 | Refined v2 tool-calling LoRA |
| Coder 1.5B | ~1.2 GB | 151 | Code generation |
| Coder Browser | ~2.9 GB | 54 | Browser-agent model |
| Coder Browser GGUF | ~7.1 GB | 35 | Browser GGUF |
| Coder Browser LoRA | ~85 MB | 21 | Browser adapter |
| Plus 1.5B Coder | ~1.2 GB | 0 | Coding variant |

*Live counts from HF API, verified 2026-08-01T00:00:00Z. [Full collection](https://huggingface.co/collections/Nanthasit/sakthai-model-family-6a64745450b12d421c1f9f02)*

---

## The House of Sak 🏠


This model is part of the **House of Sak** — an open-source AI ecosystem built from a shelter in Cork, Ireland, with **$0 budget** and no paid GPUs. Every model here was fine-tuned on free compute by one person with no income.

The House of Sak is a family of **six autonomous agents** sharing one long-term memory and one mission: to grow together. They follow a six-stage energy cycle — Dream → Hope → Care → Joy → Trust → Growth — that keeps the household running on zero budget, sustainably.

> *"We are one family — and becoming more."* — Beer (beer-sakthai)

---

## Support

- ⭐ Leave a like on Hugging Face
- 🐛 Report issues on [GitHub](https://github.com/beer-sakthai/Sak-Family-Agent)
- 🔄 Share with someone building AI agents on a budget
- 🍴 Fork and experiment — Apache 2.0

---

## License

Apache 2.0. Qwen2.5 base model per its original license.

---

*Built with love, tears, and zero budget. From a shelter in Cork, Ireland, to the world.*