| ZNX-MoE-0.2B-SFT-v1 |
|
|
| Model Description |
|
|
| ZNX-MoE-0.2B-SFT-v1 is a 217M parameter Mixture-of-Experts (MoE) language model developed by RikZD under Zenith System. The model is primarily focused on Indonesian conversational abilities and general text generation. |
|
|
| This release is an early supervised fine-tuned (SFT) version built on top of the base ZNX-MoE-0.2B pretrained model. |
|
|
| Architecture |
|
|
| - Architecture: Decoder-only Transformer + Mixture-of-Experts (MoE) |
| - Parameters: ~217M |
| - Hidden Size: 512 |
| - Layers: 12 |
| - Attention Heads: 8 |
| - FFN Size: 2048 |
| - Routed Experts: 4 |
| - Top-k Experts: 2 |
| - Context Length: 2048 tokens |
| - Positional Encoding: RoPE (Rotary Position Embedding) |
| - Vocabulary Size: 32,000 |
| - Tokenizer: Byte-Level BPE |
|
|
| Training |
|
|
| Pretraining |
|
|
| The base model was pretrained for 50,000 steps on a mixed corpus containing Indonesian and multilingual text sources. |
|
|
| Supervised Fine-Tuning (SFT) |
|
|
| The model was fine-tuned on approximately 20,000 conversational samples from: |
|
|
| - ShareGPT-Processed (~8K samples) |
| - NaturalConv (~7K samples) |
| - Synthetic-Persona-Chat (~5K samples) |
|
|
| The objective of SFT is to improve: |
|
|
| - Multi-turn conversation |
| - Instruction following |
| - Indonesian dialogue generation |
| - General assistant behavior |
|
|
| Intended Use |
|
|
| Recommended use cases: |
|
|
| - Indonesian chatbot experiments |
| - Educational purposes |
| - Research on small language models |
| - Local AI assistant prototypes |
| - Fine-tuning and alignment research |
|
|
| Not recommended for: |
|
|
| - Medical advice |
| - Legal advice |
| - Financial decisions |
| - High-risk applications requiring factual guarantees |
|
|
| Known Limitations |
|
|
| This is an early release and still has several limitations: |
|
|
| - May generate repetitive text. |
| - May occasionally produce URLs or wiki-style continuations. |
| - Can hallucinate facts. |
| - Conversation quality is inconsistent across prompts. |
| - Limited reasoning capabilities compared to larger models. |
| - Not instruction-aligned to production standards. |
|
|
| Example |
|
|
| Prompt |
|
|
| User: Halo, siapa kamu? |
|
|
| Response |
|
|
| Assistant: Halo! Saya adalah ZNX-MoE, sebuah model bahasa yang dirancang untuk membantu menjawab pertanyaan dan melakukan percakapan dalam bahasa Indonesia. |
|
|
| License |
|
|
| This project is released for research and educational purposes. |
|
|
| Authors |
|
|
| - Creator: RikZD |
| - Organization: Zenith SystemX |
|
|
| Version |
|
|
| - Base Model: ZNX-MoE-0.2B |
| - Release: ZNX-MoE-0.2B-SFT-v1 |
| - Status: Experimental |
