Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,96 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
ZNX-MoE-0.05B-SFT-v1
|
| 2 |
+
|
| 3 |
+
Model Description
|
| 4 |
+
|
| 5 |
+
ZNX-MoE-0.2B-SFT-v1 is a 217M parameter Mixture-of-Experts (MoE) language model developed by RikZD under Zenith System. The model is primarily focused on Indonesian conversational abilities and general text generation.
|
| 6 |
+
|
| 7 |
+
This release is an early supervised fine-tuned (SFT) version built on top of the base ZNX-MoE-0.05B pretrained model.
|
| 8 |
+
|
| 9 |
+
Architecture
|
| 10 |
+
|
| 11 |
+
- Architecture: Decoder-only Transformer + Mixture-of-Experts (MoE)
|
| 12 |
+
- Parameters: ~217M
|
| 13 |
+
- Hidden Size: 512
|
| 14 |
+
- Layers: 12
|
| 15 |
+
- Attention Heads: 8
|
| 16 |
+
- FFN Size: 2048
|
| 17 |
+
- Routed Experts: 4
|
| 18 |
+
- Top-k Experts: 2
|
| 19 |
+
- Context Length: 2048 tokens
|
| 20 |
+
- Positional Encoding: RoPE (Rotary Position Embedding)
|
| 21 |
+
- Vocabulary Size: 32,000
|
| 22 |
+
- Tokenizer: Byte-Level BPE
|
| 23 |
+
|
| 24 |
+
Training
|
| 25 |
+
|
| 26 |
+
Pretraining
|
| 27 |
+
|
| 28 |
+
The base model was pretrained for 50,000 steps on a mixed corpus containing Indonesian and multilingual text sources.
|
| 29 |
+
|
| 30 |
+
Supervised Fine-Tuning (SFT)
|
| 31 |
+
|
| 32 |
+
The model was fine-tuned on approximately 20,000 conversational samples from:
|
| 33 |
+
|
| 34 |
+
- ShareGPT-Processed (~8K samples)
|
| 35 |
+
- NaturalConv (~7K samples)
|
| 36 |
+
- Synthetic-Persona-Chat (~5K samples)
|
| 37 |
+
|
| 38 |
+
The objective of SFT is to improve:
|
| 39 |
+
|
| 40 |
+
- Multi-turn conversation
|
| 41 |
+
- Instruction following
|
| 42 |
+
- Indonesian dialogue generation
|
| 43 |
+
- General assistant behavior
|
| 44 |
+
|
| 45 |
+
Intended Use
|
| 46 |
+
|
| 47 |
+
Recommended use cases:
|
| 48 |
+
|
| 49 |
+
- Indonesian chatbot experiments
|
| 50 |
+
- Educational purposes
|
| 51 |
+
- Research on small language models
|
| 52 |
+
- Local AI assistant prototypes
|
| 53 |
+
- Fine-tuning and alignment research
|
| 54 |
+
|
| 55 |
+
Not recommended for:
|
| 56 |
+
|
| 57 |
+
- Medical advice
|
| 58 |
+
- Legal advice
|
| 59 |
+
- Financial decisions
|
| 60 |
+
- High-risk applications requiring factual guarantees
|
| 61 |
+
|
| 62 |
+
Known Limitations
|
| 63 |
+
|
| 64 |
+
This is an early release and still has several limitations:
|
| 65 |
+
|
| 66 |
+
- May generate repetitive text.
|
| 67 |
+
- May occasionally produce URLs or wiki-style continuations.
|
| 68 |
+
- Can hallucinate facts.
|
| 69 |
+
- Conversation quality is inconsistent across prompts.
|
| 70 |
+
- Limited reasoning capabilities compared to larger models.
|
| 71 |
+
- Not instruction-aligned to production standards.
|
| 72 |
+
|
| 73 |
+
Example
|
| 74 |
+
|
| 75 |
+
Prompt
|
| 76 |
+
|
| 77 |
+
User: Halo, siapa kamu?
|
| 78 |
+
|
| 79 |
+
Response
|
| 80 |
+
|
| 81 |
+
Assistant: Halo! Saya adalah ZNX-MoE, sebuah model bahasa yang dirancang untuk membantu menjawab pertanyaan dan melakukan percakapan dalam bahasa Indonesia.
|
| 82 |
+
|
| 83 |
+
License
|
| 84 |
+
|
| 85 |
+
This project is released for research and educational purposes.
|
| 86 |
+
|
| 87 |
+
Authors
|
| 88 |
+
|
| 89 |
+
- Creator: RikZD
|
| 90 |
+
- Organization: Zenith System
|
| 91 |
+
|
| 92 |
+
Version
|
| 93 |
+
|
| 94 |
+
- Base Model: ZNX-MoE-0.05B
|
| 95 |
+
- Release: ZNX-MoE-0.05B-SFT-v1
|
| 96 |
+
- Status: Experimental
|