---
base_model: Qwen/Qwen3-4B-Instruct-2507
library_name: peft
pipeline_tag: text-generation
license: apache-2.0
language:
- en
tags:
- code
- compression
- coding-agent
- lora
- sft
- claude-code
- cursor
- arxiv:2608.24188
model-index:
- name: paritok-4b-v1
  results:
  - task:
      type: text-compression
    dataset:
      name: SWE-bench Lite
      type: swe-bench-lite
    metrics:
    - name: Quality retained
      type: quality-retained
      value: 86.5
    - name: Compression rate
      type: compression-rate
      value: 25.7
---
<h1 align="center">Paritok-4B-v1</h1>

<p align="center">
  <b>The first open-source compression model trained specifically for coding agents.</b><br/>
  Compresses individual long files to as little as <b>5%</b> of their original size (typical content compression rate <b>25.7%</b>), cutting end-to-end input token bills from <b>~25% on turn one</b> to <b>past 85%</b> in context-saturated sessions — while retaining <b>86.5%</b> of full-context solve quality on SWE-bench Lite.
</p>

<p align="center">
  <a href="https://arxiv.org/abs/2608.24188"><img src="https://img.shields.io/badge/arXiv-2608.24188-b31b1b?logo=arxiv&logoColor=white" alt="arXiv"/></a>
  <a href="https://github.com/Paritok-official/paritok-4b-v1"><img src="https://img.shields.io/badge/GitHub-black?logo=github&logoColor=white" alt="GitHub"/></a>
  <a href="https://www.paritok.com"><img src="https://img.shields.io/badge/Website-blue" alt="Website"/></a>
  <img src="https://img.shields.io/badge/License-Apache_2.0-brightgreen" alt="License"/>
</p>

---

## 🔥 TL;DR

<table align="center">
<tr>
<td align="center" width="33%">
  <h2>~74%</h2>
  <b>Content compression</b><br/>
  <sub>Typical CR = 25.7%<br/>Peak 95% on long files</sub>
</td>
<td align="center" width="33%">
  <h2>86.5%</h2>
  <b>Solve quality retained</b><br/>
  <sub>SWE-bench Lite<br/>Matches gpt-4.1-mini</sub>
</td>
<td align="center" width="33%">
  <h2>4B</h2>
  <b>Self-hostable LoRA</b><br/>
  <sub>Single 24GB GPU<br/>Apache 2.0, zero fee</sub>
</td>
</tr>
</table>

---

<p align="center">
  <a href="https://github.com/Paritok-official/paritok-4b-v1">
    <img src="https://img.shields.io/badge/📦_Full_stack_%2B_phantom_middleware_%2B_reproduction_scripts_%E2%86%92_GitHub-181717?style=for-the-badge&logo=github&logoColor=white" alt="Full stack on GitHub"/>
  </a>
</p>
---

## ✨ Why Paritok?

- 🎨 **Code-native.** Trained end-to-end on **45K real agent trajectories** (`file_read`, `bash_command`, `log_output`...). Preserves function names, imports, paths, and error strings while compressing.
- 🚀 **~74% content compression on typical workloads**, peaking at **95%** on long files.
- 💰 **End-to-end bill savings compound with session length** — from ~25% at turn one to **past 85%** in context-saturated sessions on Claude Sonnet / GPT-4. Long-session teams save thousands per month.
- 🎯 **86.5% of full-context solve quality retained** on SWE-bench Lite — **matching gpt-4.1-mini** as compressor at less than half the token spend.
- 🪶 **Small, fast, self-hostable.** 4B LoRA adapter, bf16, single 24GB GPU. No SaaS, no lock-in, no per-token fee.
- 🔓 **Fully open.** Apache 2.0 weights, reproducible pipeline, real end-to-end benchmarks.

---

## 🚀 Quick Start

```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
BASE_MODEL = "Qwen/Qwen3-4B-Instruct-2507"
ADAPTER    = "paritok/paritok-4b-v1"
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
base = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()
# Wrap your code as [SEG id=1 kind=file_read] ... [/SEG]
user_msg = "[SEG id=1 kind=file_read]\n<your code here>\n[/SEG]"
prompt = tokenizer.apply_chat_template(
    [{"role": "system", "content": "<full system prompt — see GitHub>"},
     {"role": "user",   "content": user_msg}],
    tokenize=False, add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```

📎 **Full runnable example + system prompt + reproduction pipeline:** [GitHub → Paritok-official/paritok-4b-v1](https://github.com/Paritok-official/paritok-4b-v1)

---

## 📊 SWE-bench Lite — Head-to-head

| Context source            | **Quality retained** ¹ | Compression rate |
| ------------------------- | :--------------------: | :--------------: |
| Uncompressed baseline     |         100.0%         |      100.0%      |
| gpt-4.1-mini (compressor) |          85.6%         |       50.2%      |
| gpt-5 (compressor)        |          93.6%         |       61.9%      |
| **Paritok-4B-v1** ⭐      |       **86.5%**        |    **25.7%**     |

<sub>¹ Quality retained = compressor solve rate ÷ uncompressed baseline solve rate. Higher is better.</sub>

**Paritok compresses ~2× harder than gpt-4.1-mini** on the same task while keeping the same solve rate — the only open-source entry trained end-to-end on real coding-agent trajectories.

---

## 💰 Cost Impact (Claude Sonnet, $3/M input tokens)

| Turn input size    | Uncompressed input | With Paritok |
| ------------------ | :----------------: | :----------: |
| Short (8K)         |       $0.024       |    $0.006    |
| Typical (15K)      |       $0.045       |    $0.012    |
| Long session (30K) |       $0.090       |    $0.023    |

**10-person team, 3-month project → save ~$10K on API bills.**

---

<details>
<summary>📋 <b>Model details</b> (click to expand)</summary>

| Property               | Value                                                                             |
| ---------------------- | --------------------------------------------------------------------------------- |
| **Base model**         | [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) |
| **Adapter type**       | LoRA, r=32, α=64, dropout=0.0                                                     |
| **Target modules**     | `q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`                   |
| **Training steps**     | 2000 (selected from a 5-checkpoint sweep on OOD holdout)                          |
| **Training precision** | bf16                                                                              |
| **Effective batch**    | 32 (per_device=2 × grad_accum=16)                                                 |
| **Learning rate**      | 1e-5, linear decay, 10% warmup                                                    |
| **Optimizer**          | AdamW (8-bit)                                                                     |
| **Max seq length**     | 16 384                                                                            |
| **Dataset size**       | 45 000 samples (`file_read`, `bash_command`, `log_output`, ...)                   |
| **Teacher**            | gpt-4.1-mini                                                                      |

</details>

<details>
<summary>⚠️ <b>Limitations</b> (click to expand)</summary>

1. **~6pp accuracy trade for 74% compression.** Not free — buying context length with a small solve-rate drop. Bring your own uncompressed-fallback path for safety-critical turns.
2. **Rare identifier loss (~40% preserved on hard OOD segments).** Add a post-compression check that your target identifier is still in the compressed output before sending upstream.
3. **English source code, Python-heavy training distribution.** Non-English comments/strings and other-language codebases have not been benchmarked.

**Recommended safeguards** for production: format check, target-identifier presence check, and a session-level toggle to fall back to raw context on failure.

</details>

<details>
<summary>🗺️ <b>Roadmap</b> (click to expand)</summary>

- **Paritok-4B-v2** — Next-generation training pipeline pushing compression to under 20% while closing the gap to uncompressed solve rate.
- **Frontier-scale backbones** (10B+ parameters) for multi-day sessions with 100K+ token histories.
- **Multi-language expansion** — TypeScript, Rust, Go, Java, C++, Kotlin.
- **Native integrations** — `mcp add paritok` plugin for Claude Code and Cursor, plus a hosted inference endpoint.

</details>

---

## 📖 Citation

Cite the **paper** for the method and results:

```bibtex
@misc{paritok4b,
  title         = {Paritok-4B: Intent-Conditioned Context Compression for Coding Agents},
  author        = {Shi, Jiayu and Chen, Luzhuo},
  year          = {2026},
  eprint        = {2608.24188},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  doi           = {10.48550/arXiv.2608.24188},
  url           = {https://arxiv.org/abs/2608.24188},
}
```

Cite the **software** for this release — the weights, the gateway, and the training and evaluation code:

```bibtex
@software{paritok-4b-v1,
  title   = {Paritok-4B-v1: An Open-Source Compression Model for AI Coding Agents},
  author  = {Shi, Jiayu and Chen, Luzhuo},
  year    = {2026},
  version = {4B-v1},
  license = {Apache-2.0},
  url     = {https://github.com/Paritok-official/paritok-4b-v1},
}
```

---

## 📄 License

Apache 2.0 — see [LICENSE](https://github.com/Paritok-official/paritok-4b-v1/blob/main/LICENSE).
Base model (Qwen3-4B-Instruct-2507) is under its own license; please review before commercial deployment.

<p align="center">
  <a href="https://arxiv.org/abs/2608.24188">Paper</a> ·
  <a href="https://github.com/Paritok-official/paritok-4b-v1">GitHub</a> ·
  <a href="https://www.paritok.com">paritok.com</a> ·
  <a href="mailto:paritok9@gmail.com">Contact</a>
</p>