ZNX-MoE-0.2B-SFT-v1

Model Description

ZNX-MoE-0.2B-SFT-v1 is a 217M parameter Mixture-of-Experts (MoE) language model developed by RikZD under Zenith System. The model is primarily focused on Indonesian conversational abilities and general text generation.

This release is an early supervised fine-tuned (SFT) version built on top of the base ZNX-MoE-0.2B pretrained model.

Architecture

- Architecture: Decoder-only Transformer + Mixture-of-Experts (MoE)
- Parameters: ~217M
- Hidden Size: 512
- Layers: 12
- Attention Heads: 8
- FFN Size: 2048
- Routed Experts: 4
- Top-k Experts: 2
- Context Length: 2048 tokens
- Positional Encoding: RoPE (Rotary Position Embedding)
- Vocabulary Size: 32,000
- Tokenizer: Byte-Level BPE

Training

Pretraining

The base model was pretrained for 50,000 steps on a mixed corpus containing Indonesian and multilingual text sources.

Supervised Fine-Tuning (SFT)

The model was fine-tuned on approximately 20,000 conversational samples from:

- ShareGPT-Processed (~8K samples)
- NaturalConv (~7K samples)
- Synthetic-Persona-Chat (~5K samples)

The objective of SFT is to improve:

- Multi-turn conversation
- Instruction following
- Indonesian dialogue generation
- General assistant behavior

Intended Use

Recommended use cases:

- Indonesian chatbot experiments
- Educational purposes
- Research on small language models
- Local AI assistant prototypes
- Fine-tuning and alignment research

Not recommended for:

- Medical advice
- Legal advice
- Financial decisions
- High-risk applications requiring factual guarantees

Known Limitations

This is an early release and still has several limitations:

- May generate repetitive text.
- May occasionally produce URLs or wiki-style continuations.
- Can hallucinate facts.
- Conversation quality is inconsistent across prompts.
- Limited reasoning capabilities compared to larger models.
- Not instruction-aligned to production standards.

Example

Prompt

User: Halo, siapa kamu?

Response

Assistant: Halo! Saya adalah ZNX-MoE, sebuah model bahasa yang dirancang untuk membantu menjawab pertanyaan dan melakukan percakapan dalam bahasa Indonesia.

License

This project is released for research and educational purposes.

Authors

- Creator: RikZD
- Organization: Zenith SystemX

Version

- Base Model: ZNX-MoE-0.2B
- Release: ZNX-MoE-0.2B-SFT-v1
- Status: Experimental