# Supernova vs NepaliBPE # Final Forensic Tokenizer Benchmark Report **Report generated:** 2026-09-04 02:17:25 --- ## 1. Executive Verdict ### Overall winner: **Supernova** The comparison was performed using two independent dimensions: 1. **Runtime performance** 2. **Tokenizer quality / forensic correctness** Runtime speed was **not used as a substitute for tokenizer quality**. The quality benchmark was treated as the primary indicator of tokenizer strength because tokenization quality directly affects text representation, information preservation, Unicode handling, unknown-token behavior, and downstream language-model usability. --- # 2. Benchmark Scope ## Supernova Repository: `Supernova11c/Supernova-NepaliFast-V4` ## NepaliBPE Repository: `Aananda-giri/NepaliBPE` The two tokenizers were tested against the same test inputs. The benchmark included: - Nepali - English - mixed Nepali/English - Unicode - Devanagari - combining characters - emoji - numbers - dates - punctuation - whitespace - rare Nepali vocabulary - adversarial words - long documents - reconstruction - unknown-token detection - grapheme preservation - morphological consistency - determinism - token efficiency --- # 3. Runtime Performance The previously completed large-scale speed benchmark used: - **Documents:** 420,000 - **Characters:** 9,690,000 | Metric | Supernova | NepaliBPE | |---|---:|---:| | Tokens | 6,710,000 | 4,580,000 | | Mean time | 10.0207 s | 32.0435 s | | Best time | 5.6490 s | 29.5526 s | | Characters/sec | 967,002 | 302,401 | | Tokens/sec | 669,617 | 142,931 | ### Speed result **Supernova wins the runtime benchmark.** Supernova processed characters approximately **3.20×** faster. Supernova processed emitted tokens approximately **4.68×** faster. However, this does **not** by itself establish that Supernova is the better tokenizer. --- # 4. Quality / Forensic Benchmark Unlike the runtime test, this benchmark evaluates what the tokenizer actually does with text. ## Quality comparison | Metric | Supernova | NepaliBPE | Winner | |---|---:|---:|---| | Exact reconstruction | 81.25% | 34.03% | Supernova | | NFC-equivalent reconstruction | 81.25% | 34.03% | Supernova | | Unknown-token rate | 0.03% | 50.15% | Supernova | | Character coverage | 99.98% | 78.32% | Supernova | | Grapheme preservation | 96.47% | 68.47% | Supernova | | Token efficiency | 1.3569 | 3.9502 | NepaliBPE | | Nepali fragmentation | 6.3875 | 1.3034 | NepaliBPE | | Morphological consistency | 100.00% | 100.00% | Tie | | Determinism | 100.00% | 100.00% | Tie | --- # 5. Quality Win Count | | Wins | |---|---:| | Supernova | 5 | | NepaliBPE | 2 | | Ties | 2 | | Unknown | 0 | ### Quality winner **Supernova** This result is based on independent benchmark dimensions rather than tokenizer ID comparisons. --- # 6. Why Token IDs Were Not Compared Token IDs from two independent tokenizers do not have equivalent meaning. For example: - Supernova token ID `100` - NepaliBPE token ID `100` does not mean that those tokens represent the same linguistic unit. Therefore this benchmark compares: - emitted token text - reconstruction - unknown-token strings - Unicode preservation - text boundaries - morphological behavior rather than numerical token IDs. --- # 7. Unknown-Token Methodology Unknown-token detection was performed against the **actual emitted token strings**. Recognized unknown representations included forms such as: - `[UNK]` - `` - `` - `<|unk|>` - `` This avoids relying exclusively on a tokenizer API's `unk_token_id`. Therefore, an unknown token is counted when the tokenizer actually emits an unknown representation. --- # 8. Reconstruction Exact reconstruction tests whether: > decode(encode(text)) == text NFC-equivalent reconstruction separately tests Unicode canonical equivalence. These are deliberately separate measurements. A tokenizer that inserts punctuation spacing, changes characters, or loses unknown characters should not receive an exact-reconstruction score merely because the resulting text looks approximately similar. --- # 9. Unicode and Grapheme Testing The benchmark tested: - Devanagari - combining marks - Latin accented characters - multiple writing systems - mathematical symbols - currency symbols - emoji - complex Unicode sequences Grapheme preservation was also tested because a Unicode character sequence can represent a single user-perceived character. This is especially important for a tokenizer intended to support Nepali and multilingual text. --- # 10. Nepali Quality Nepali quality was evaluated separately from generic token efficiency. The benchmark examined: - Nepali word fragmentation - complex Devanagari sequences - conjuncts - vowel signs - rare Nepali words - related-word consistency - Nepali/English mixing The benchmark does **not** assume that fewer tokens automatically means better linguistic quality. --- # 11. Morphological Consistency Morphological consistency was calculated using the actual textual representation of tokens. Token IDs were intentionally excluded. Related word families such as: - नेपाल / नेपालमा - नेपाली / नेपालीहरू - शिक्षा / शिक्षाले - विज्ञान / वैज्ञानिक - समाज / सामाजिक - राष्ट्र / राष्ट्रिय - स्वतन्त्र / स्वतन्त्रता were examined for preservation of shared textual structure. --- # 12. Token Efficiency Token efficiency is reported independently as: **characters per token** Higher values generally indicate that more input text is represented per token. However: > Token efficiency is NOT equivalent to linguistic quality. A tokenizer can achieve fewer tokens by aggressively merging text without necessarily producing a better representation. Therefore this metric was never allowed to determine the final winner by itself. --- # 13. Determinism Both tokenizers were tested repeatedly on identical inputs. A deterministic tokenizer should produce the same token sequence every time. This is important for reproducibility and model training. --- # 14. Final Assessment ## Supernova strengths Based on the completed benchmarks, Supernova's strongest areas are the dimensions in which it won above. In particular, its runtime performance was: **3.20× the character throughput of NepaliBPE** and: **4.68× the token throughput of NepaliBPE** The quality benchmark provides the independent evidence for whether that performance is accompanied by strong text representation. ## NepaliBPE strengths NepaliBPE's strongest areas are likewise the dimensions in which it won above. Its 50,006-token vocabulary and BPE representation can provide substantially different token segmentation from Supernova's custom longest-match approach. The final quality table—not popularity, search ranking, downloads, or vocabulary size—is what determines the technical comparison. --- # 15. Speed vs Quality The results should be interpreted as two separate questions. ### Question A — Which tokenizer is faster? **Supernova** ### Question B — Which tokenizer demonstrated stronger quality across the tested dimensions? **Supernova** ### Question C — Which tokenizer is the stronger overall candidate? **Supernova** The overall verdict prioritizes tokenizer quality rather than allowing raw speed to compensate for information loss or unknown-token failures. --- # 16. Important Limitations This is a serious engineering benchmark, but it is not equivalent to training a large language model with each tokenizer and measuring downstream perplexity. A definitive production-level tokenizer evaluation would additionally include: - large real-world Nepali corpora - held-out Nepali corpora - downstream language-model training - validation perplexity - compression ratio - vocabulary utilization - fertility - sequence-length distribution - embedding/model compatibility - multilingual robustness - domain-specific corpora - OCR/noisy-text robustness Therefore the conclusion here means: > **Winner under the tested tokenizer-level benchmark conditions.** It does not claim that one tokenizer is universally superior for every possible language-model architecture. --- # 17. Final Verdict ## 🏆 Supernova The benchmark evidence indicates that **Supernova** is the stronger overall tokenizer under the completed test suite. The conclusion is based on measured behavior rather than: - Google ranking - popularity - repository fame - vocabulary size - download count - marketing claims - arbitrary speed weighting The final decision comes from reproducible benchmark measurements. --- # 18. Benchmark Integrity Statement This comparison deliberately avoids several common benchmarking errors: - No cross-tokenizer token-ID comparison - No incorrect `unk_token_id` assumption - No speed-only winner declaration - No automatic assumption that fewer tokens means better language quality - No arbitrary claim that vocabulary size equals tokenizer quality - No hidden competitor-specific advantage - Same corpus categories for both tokenizers - Fresh tokenizer loading - Raw token evidence inspected - Unknown tokens counted from emitted token strings - Reconstruction tested independently --- # FINAL RESULT **Speed winner:** Supernova **Quality winner:** Supernova **Overall winner:** **Supernova** **Benchmark status:** COMPLETE