---
license: apache-2.0
base_model: Qwen/Qwen3.5-9B-Base
language:
- en
tags:
- hardware
- rtl
- chip-design
- domain-adaptive-pretraining
- pulp-platform
- knowledge-injection
---

# Qwen3.5-9B-PULP-DAPT

A 9B model domain-adaptively pretrained on the [PULP platform](https://pulp-platform.org/)
(Carfield/Cheshire RISC-V SoC) corpus, as a public proxy for the "inject a proprietary chip's
knowledge into an LLM" problem (the ChipNeMo setting).

**Closed-book factual accuracy about the platform: 92.8% vs Claude Opus 5's 72.0%**
(125-question layered audit benchmark: register maps, memory maps, dependency pins,
documentation semantics, driver↔register correspondences, issue history) — and **81.6% vs
72.2%** on the full 1,776-question bank (base model: 41.2%), only 125 of which were ever
seen during development.

| Model | Audit subset (125q) | Full bank (1,776q) |
|---|---|---|
| Qwen3.5-9B-Base | 43.2% | 41.2% |
| Claude Opus 5 (closed-book) | 72.0% | 72.2% |
| **This model** | **92.8%** | **81.6%** (register offsets 97%) |

The interesting part is the recipe: **raw-corpus DAPT alone produced zero gain on
memorization questions** (loss 0.80→0.35, offsets 1/42 → 1/42). All gains come from a
knowledge-rewriting augmentation stage: every fact restated through 24 LLM-generated
templates (⅓ reversed forms), full-coverage over all facts (not just evaluated ones), plus
whole-table narrative documents to fight similar-fact interference. Full recipe, data
pipeline, benchmark, and findings: **https://github.com/ARA-Labs/PULP-LLM**

## Training

- Continued pretraining (`stage=pt`), full-parameter, bf16, LLaMA-Factory + DeepSpeed ZeRO-3
- 31.2M tokens pulp-platform org corpus + 3.49M tokens augmentation + 8% wikitext replay
- LR 5e-6 cosine, global batch 16, packing @4096, 2 epochs — 77 min on 4×H100 (~$20)

## Usage

This is a **base-style** model (no chat template) tuned for knowledge extraction via
few-shot completion:

```python
from vllm import LLM, SamplingParams
llm = LLM(model="AgentNativeResearchLab/Qwen3.5-9B-PULP-DAPT", dtype="bfloat16")
prompt = (
    "The following are factual questions about the PULP Carfield/Cheshire hardware platform "
    "(ETH Zurich). Answer each with only the value, on one line.\n\n"
    "Q: In the PULP Carfield SoC memory map (car_memory_map.h), what is the value of CAR_BOOTROM_BASE_ADDR?\nA: 0x2000000\n\n"
    "Q: In the axi_llc register block, what is the byte offset of the CFG_FLUSH_LOW register?\nA:"
)
print(llm.generate([prompt], SamplingParams(temperature=0, max_tokens=16))[0].outputs[0].text)
```

## Limitations

- Knowledge snapshot of the pulp-platform repos as of 2026-08; no forgetting audit (MMLU) yet.
- Weak at numeric range-membership reasoning (memory-map region ownership).
- Not instruction-tuned; use few-shot completion or apply your own SFT.