File size: 2,361 Bytes
5a15d0a 5a44f9b 5209990 5a44f9b df8870f 5a44f9b 5209990 5a44f9b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | ZNX-MoE-0.2B-SFT-v1
Model Description
ZNX-MoE-0.2B-SFT-v1 is a 217M parameter Mixture-of-Experts (MoE) language model developed by RikZD under Zenith System. The model is primarily focused on Indonesian conversational abilities and general text generation.
This release is an early supervised fine-tuned (SFT) version built on top of the base ZNX-MoE-0.2B pretrained model.
Architecture
- Architecture: Decoder-only Transformer + Mixture-of-Experts (MoE)
- Parameters: ~217M
- Hidden Size: 512
- Layers: 12
- Attention Heads: 8
- FFN Size: 2048
- Routed Experts: 4
- Top-k Experts: 2
- Context Length: 2048 tokens
- Positional Encoding: RoPE (Rotary Position Embedding)
- Vocabulary Size: 32,000
- Tokenizer: Byte-Level BPE
Training
Pretraining
The base model was pretrained for 50,000 steps on a mixed corpus containing Indonesian and multilingual text sources.
Supervised Fine-Tuning (SFT)
The model was fine-tuned on approximately 20,000 conversational samples from:
- ShareGPT-Processed (~8K samples)
- NaturalConv (~7K samples)
- Synthetic-Persona-Chat (~5K samples)
The objective of SFT is to improve:
- Multi-turn conversation
- Instruction following
- Indonesian dialogue generation
- General assistant behavior
Intended Use
Recommended use cases:
- Indonesian chatbot experiments
- Educational purposes
- Research on small language models
- Local AI assistant prototypes
- Fine-tuning and alignment research
Not recommended for:
- Medical advice
- Legal advice
- Financial decisions
- High-risk applications requiring factual guarantees
Known Limitations
This is an early release and still has several limitations:
- May generate repetitive text.
- May occasionally produce URLs or wiki-style continuations.
- Can hallucinate facts.
- Conversation quality is inconsistent across prompts.
- Limited reasoning capabilities compared to larger models.
- Not instruction-aligned to production standards.
Example
Prompt
User: Halo, siapa kamu?
Response
Assistant: Halo! Saya adalah ZNX-MoE, sebuah model bahasa yang dirancang untuk membantu menjawab pertanyaan dan melakukan percakapan dalam bahasa Indonesia.
License
This project is released for research and educational purposes.
Authors
- Creator: RikZD
- Organization: Zenith SystemX
Version
- Base Model: ZNX-MoE-0.2B
- Release: ZNX-MoE-0.2B-SFT-v1
- Status: Experimental |