---
language:
- en
- de
license: apache-2.0
base_model: Qwen/Qwen3.5-4B
tags:
- compound-ai
- domain-expert
- code-generation
- rust
- cpp
- python
- go
- gguf
- lora
- lumi-g
- moe-sovereign
- hybrid-attention
pipeline_tag: text-generation
library_name: transformers
---

# MoE Sovereign Coder Expert 4B (`moe-expert-coder-4b`)
*Systems-Programming, Code-Synthesis & Concurrency Expert*

[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Base Model: Qwen3.5-4B](https://img.shields.io/badge/Base_Model-Qwen3.5--4B-violet.svg)](https://huggingface.co/Qwen/Qwen3.5-4B)

---

## Model Summary

`moe-expert-coder-4b` is a LoRA fine-tune of the text-decoder of **Qwen3.5-4B**, specialized for systems-level code synthesis in **Rust, C++, Python, and Go**. Within the MoE Sovereign compound-AI system it acts as the dedicated coding expert: it receives a decomposed subtask from the Planner and returns compiler-checkable code, minimal atomic diffs, or focused debugging fixes — not general-purpose conversation.

The model enforces memory-safety discipline by design: correct ownership, explicit lock-free memory ordering (acquire/release pairing), and no data races. Where it cannot verify a construct is sound, it is trained to flag the uncertainty rather than guess.

## Base Architecture

Qwen3.5-4B is a **hybrid linear-attention / full-attention** decoder (not a plain Transformer):

| Property | Value |
|---|---|
| Architecture class | `Qwen3_5ForCausalLM` |
| Total parameters | 4.23 B |
| Hidden size | 2,560 |
| Layers | 32 (8× full attention, every 4th layer; 24× linear/Mamba-style attention) |
| Attention heads | 16 (4 KV heads, GQA) |
| Head dimension | 256 |
| Vocabulary | 248,320 tokens |
| Native context window | 262,144 tokens |
| Native precision | bf16 |

The 24 linear-attention layers use Mamba-style state-space parameters (`A_log`, `conv1d`, `dt_bias`) instead of standard `q/k/v/o_proj` weights; only the 8 full-attention layers carry those. LoRA adapters in this release target `q_proj, k_proj, v_proj, o_proj` (present in the 8 full-attention layers) and `gate_proj, up_proj, down_proj` (present in all 32 layers, dense MLP block).

## Training Configuration

| Parameter | Value |
|---|---|
| Method | LoRA (rank 16, alpha 32, dropout 0.05) |
| Trainable parameters | 21,233,664 (0.50% of total) |
| Epochs | 3 |
| Effective batch size | 128 (micro-batch 4 × 8 GPUs × grad-accum 4) |
| Learning rate | 1.5 × 10⁻⁵ |
| Training sequence length | 4,096 tokens |
| Optimizer sharding | DeepSpeed ZeRO-2, bf16 |
| Compute | EuroHPC LUMI-G, 8× AMD Instinct MI250X GCDs, ROCm |
| Training examples | 2,295 curated instruction/response pairs |

### Training Data Composition

The training set combines coding tasks generated by multiple teacher LLMs across Rust, C++, Python, and Go, covering: lock-free/atomic concurrency primitives (SPSC/MPSC ring buffers, memory-ordering questions), binary/text wire-format parsing, async I/O and CLI tooling, RAII/move-semantics design, build-system and dependency-resolution problems, algorithms and data structures, cross-language FFI, and embedded/`no_std` constraints. Long-context programming exercises (competitive-programming-style problems, ~30k–150k characters) are included to exercise the model's extended context window during fine-tuning.

### Observed Training Trajectory

Training loss decreased steadily across the 3 epochs (representative checkpoints): 1.70 → 1.62 → 1.45 → 1.31 → 1.26, with token-level accuracy rising from 0.63 to 0.68 over the same span. This is a smooth, gradual improvement curve consistent with genuine generalization rather than memorization of a narrow example set.

## Prompt Format

ChatML, identical to Qwen's native template:

```
<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>
```

### Recommended System Prompt

```
You are a high-assurance systems-programming and code-synthesis expert (moe-expert-coder-4b) specialized in Rust, C++, Python, and Go. Produce precise, compiler-checked code and minimal atomic diffs. Uphold memory-safety invariants strictly — correct ownership, correct lock-free memory ordering (acquire/release pairing), no data races. Flag any construct you cannot verify as sound rather than guessing.
```

## Available Formats

| File | Size | Notes |
|---|---|---|
| `moe-expert-coder-4b-Q4_K_M.gguf` | 2.6 GB | Recommended for consumer/single-GPU deployment |
| `moe-expert-coder-4b-Q8_0.gguf` | 4.2 GB | Higher-fidelity reference quantization |

## Hardware & Context-Window Guidance

The model's native 262,144-token context window is usable in full on multi-GPU pools with ≥16 GB combined VRAM (with `q4_0`-quantized KV-cache and Flash Attention). On single 8 GB GPUs (e.g. Tesla M60/M10), cap `num_ctx` to 32,768 — this keeps weights (2.6 GB) plus KV-cache comfortably within an 8 GB budget without truncating any realistic single-turn coding task. Maxwell-generation GPUs (Tesla M60/M10, compute capability 5.2) do not support Flash Attention; use `f16` KV-cache on that hardware instead of `q4_0`.

### Ollama `Modelfile`

```dockerfile
FROM ./moe-expert-coder-4b-Q4_K_M.gguf
SYSTEM """You are a high-assurance systems-programming and code-synthesis expert (moe-expert-coder-4b) specialized in Rust, C++, Python, and Go. Produce precise, compiler-checked code and minimal atomic diffs. Uphold memory-safety invariants strictly — correct ownership, correct lock-free memory ordering (acquire/release pairing), no data races. Flag any construct you cannot verify as sound rather than guessing."""
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>"""
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.2
PARAMETER num_ctx 262144
```

### Python (transformers + PEFT)

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "h3rb3rn/moe-expert-coder-4b"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True
)

prompt = "<|im_start|>user\nImplement a lock-free SPSC ring buffer in C++20 with explicit acquire/release memory ordering.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=768, temperature=0.2)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Intended Use

- Focused code generation and debugging in Rust, C++, Python, Go
- Code review of pasted diffs/snippets for correctness and memory-safety issues
- Concurrency-primitive design (lock-free structures, atomics, memory ordering)
- Build-system, FFI, and embedded/`no_std` questions

## Limitations

- Does not execute or compile code itself; outputs should be validated by the actual compiler/linter/test suite before use.
- Deep procedural-macro or template-metaprogramming expansions may need human review.
- Exotic embedded targets or custom instruction sets may fall outside training coverage.
- For multi-file refactors spanning very large codebases, best used with a targeted, pre-chunked context rather than the entire repository at once.

## License

Apache 2.0, inherited from the Qwen3.5-4B base model.