YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
ZNX-MoE-0.2B-SFT-v1
Model Description
ZNX-MoE-0.2B-SFT-v1 is a 217M parameter Mixture-of-Experts (MoE) language model developed by RikZD under Zenith System. The model is primarily focused on Indonesian conversational abilities and general text generation.
This release is an early supervised fine-tuned (SFT) version built on top of the base ZNX-MoE-0.2B pretrained model.
Architecture
- Architecture: Decoder-only Transformer + Mixture-of-Experts (MoE)
- Parameters: ~217M
- Hidden Size: 512
- Layers: 12
- Attention Heads: 8
- FFN Size: 2048
- Routed Experts: 4
- Top-k Experts: 2
- Context Length: 2048 tokens
- Positional Encoding: RoPE (Rotary Position Embedding)
- Vocabulary Size: 32,000
- Tokenizer: Byte-Level BPE
Training
Pretraining
The base model was pretrained for 50,000 steps on a mixed corpus containing Indonesian and multilingual text sources.
Supervised Fine-Tuning (SFT)
The model was fine-tuned on approximately 20,000 conversational samples from:
- ShareGPT-Processed (~8K samples)
- NaturalConv (~7K samples)
- Synthetic-Persona-Chat (~5K samples)
The objective of SFT is to improve:
- Multi-turn conversation
- Instruction following
- Indonesian dialogue generation
- General assistant behavior
Intended Use
Recommended use cases:
- Indonesian chatbot experiments
- Educational purposes
- Research on small language models
- Local AI assistant prototypes
- Fine-tuning and alignment research
Not recommended for:
- Medical advice
- Legal advice
- Financial decisions
- High-risk applications requiring factual guarantees
Known Limitations
This is an early release and still has several limitations:
- May generate repetitive text.
- May occasionally produce URLs or wiki-style continuations.
- Can hallucinate facts.
- Conversation quality is inconsistent across prompts.
- Limited reasoning capabilities compared to larger models.
- Not instruction-aligned to production standards.
Example
Prompt
User: Halo, siapa kamu?
Response
Assistant: Halo! Saya adalah ZNX-MoE, sebuah model bahasa yang dirancang untuk membantu menjawab pertanyaan dan melakukan percakapan dalam bahasa Indonesia.
License
This project is released for research and educational purposes.
Authors
- Creator: RikZD
- Organization: Zenith SystemX
Version
- Base Model: ZNX-MoE-0.2B
- Release: ZNX-MoE-0.2B-SFT-v1
- Status: Experimental
- Downloads last month
- 42