MoE Sovereign Planner 7B -- OLMo-3 (moe-sovereign-planner-olmo3-7b)

Task Decomposition & Orchestration

---

Model Summary

moe-sovereign-planner-olmo3-7b is a LoRA fine-tune of OLMo-3-7B-Instruct, specialized as the orchestrator/planner of the MoE Sovereign compound-AI system: it decomposes an incoming request into 1-4 subtasks for the domain experts, extracting and propagating explicit numerical constraints so experts cannot hallucinate default values.

Genuinely open-source base (Ai2 -- weights, training data, and training code all publicly documented), the Spur-2 (open-source) counterpart to the parallel Qwen3.5-9B planner trained on the open-weight track.

Training Configuration

Parameter	Value
Method	LoRA (rank 16, alpha 32, dropout 0.05)
Trainable parameters	39,976,960 of 7,337,988,096 (0.54%)
Epochs	3
Effective batch size	128 (micro-batch 4 x 8 GPUs x grad-accum 4)
Learning rate	1.5e-5
Training sequence length	4,096 tokens
Compute	EuroHPC LUMI-G, 8x AMD Instinct MI250X GCDs, ROCm
Training examples	4,726 curated decomposition examples

Observed Training Trajectory

Training loss: 2.111 -> 1.094 -> 0.737. Smooth decline, no overfitting signature.

Prompt Format

ChatML. System prompt:

You are the orchestrator of a Mixture-of-Experts system.
Decompose the following request into 1-4 subtasks.

Mandatorily extract all numerical constraints and technical parameters from the request (e.g. model sizes, MTU values, protocol overheads, chemical doses, bitrates). Integrate these as IMMUTABLE_CONSTANTS directly into each subtask description for the experts, so experts cannot hallucinate default values.

Available Formats

File	Notes
moe-sovereign-planner-olmo3-7b-Q4_K_M.gguf	Recommended for deployment
moe-sovereign-planner-olmo3-7b-Q8_0.gguf	Higher-fidelity reference quantization

Hardware Guidance

OLMo-3-7B's native context is 65,536 tokens (via YaRN extension). Fits comfortably on a single modern 12GB+ GPU at Q4_K_M.

Ollama Modelfile

FROM ./moe-sovereign-planner-olmo3-7b-Q4_K_M.gguf
SYSTEM """You are the orchestrator of a Mixture-of-Experts system.
Decompose the following request into 1-4 subtasks.

Mandatorily extract all numerical constraints and technical parameters from the request (e.g. model sizes, MTU values, protocol overheads, chemical doses, bitrates). Integrate these as IMMUTABLE_CONSTANTS directly into each subtask description for the experts, so experts cannot hallucinate default values."""
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>"""
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.2
PARAMETER num_ctx 32768

Limitations

• Decomposition quality depends on the request containing extractable constraints; ambiguous requests may yield underspecified subtasks.
• Does not execute the subtasks itself -- routes to the domain experts.

License

Apache 2.0, inherited from the OLMo-3-7B-Instruct base model.