Supernova vs NepaliBPE
Final Forensic Tokenizer Benchmark Report
Report generated: 2026-09-04 02:17:25
1. Executive Verdict
Overall winner: Supernova
The comparison was performed using two independent dimensions:
- Runtime performance
- Tokenizer quality / forensic correctness
Runtime speed was not used as a substitute for tokenizer quality.
The quality benchmark was treated as the primary indicator of tokenizer strength because tokenization quality directly affects text representation, information preservation, Unicode handling, unknown-token behavior, and downstream language-model usability.
2. Benchmark Scope
Supernova
Repository:
Supernova11c/Supernova-NepaliFast-V4
NepaliBPE
Repository:
Aananda-giri/NepaliBPE
The two tokenizers were tested against the same test inputs.
The benchmark included:
- Nepali
- English
- mixed Nepali/English
- Unicode
- Devanagari
- combining characters
- emoji
- numbers
- dates
- punctuation
- whitespace
- rare Nepali vocabulary
- adversarial words
- long documents
- reconstruction
- unknown-token detection
- grapheme preservation
- morphological consistency
- determinism
- token efficiency
3. Runtime Performance
The previously completed large-scale speed benchmark used:
- Documents: 420,000
- Characters: 9,690,000
| Metric | Supernova | NepaliBPE |
|---|---|---|
| Tokens | 6,710,000 | 4,580,000 |
| Mean time | 10.0207 s | 32.0435 s |
| Best time | 5.6490 s | 29.5526 s |
| Characters/sec | 967,002 | 302,401 |
| Tokens/sec | 669,617 | 142,931 |
Speed result
Supernova wins the runtime benchmark.
Supernova processed characters approximately 3.20× faster.
Supernova processed emitted tokens approximately 4.68× faster.
However, this does not by itself establish that Supernova is the better tokenizer.
4. Quality / Forensic Benchmark
Unlike the runtime test, this benchmark evaluates what the tokenizer actually does with text.
Quality comparison
| Metric | Supernova | NepaliBPE | Winner |
|---|---|---|---|
| Exact reconstruction | 81.25% | 34.03% | Supernova |
| NFC-equivalent reconstruction | 81.25% | 34.03% | Supernova |
| Unknown-token rate | 0.03% | 50.15% | Supernova |
| Character coverage | 99.98% | 78.32% | Supernova |
| Grapheme preservation | 96.47% | 68.47% | Supernova |
| Token efficiency | 1.3569 | 3.9502 | NepaliBPE |
| Nepali fragmentation | 6.3875 | 1.3034 | NepaliBPE |
| Morphological consistency | 100.00% | 100.00% | Tie |
| Determinism | 100.00% | 100.00% | Tie |
5. Quality Win Count
| Wins | |
|---|---|
| Supernova | 5 |
| NepaliBPE | 2 |
| Ties | 2 |
| Unknown | 0 |
Quality winner
Supernova
This result is based on independent benchmark dimensions rather than tokenizer ID comparisons.
6. Why Token IDs Were Not Compared
Token IDs from two independent tokenizers do not have equivalent meaning.
For example:
- Supernova token ID
100 - NepaliBPE token ID
100
does not mean that those tokens represent the same linguistic unit.
Therefore this benchmark compares:
- emitted token text
- reconstruction
- unknown-token strings
- Unicode preservation
- text boundaries
- morphological behavior
rather than numerical token IDs.
7. Unknown-Token Methodology
Unknown-token detection was performed against the actual emitted token strings.
Recognized unknown representations included forms such as:
[UNK]<unk><UNK><|unk|><unknown>
This avoids relying exclusively on a tokenizer API's unk_token_id.
Therefore, an unknown token is counted when the tokenizer actually emits an unknown representation.
8. Reconstruction
Exact reconstruction tests whether:
decode(encode(text)) == text
NFC-equivalent reconstruction separately tests Unicode canonical equivalence.
These are deliberately separate measurements.
A tokenizer that inserts punctuation spacing, changes characters, or loses unknown characters should not receive an exact-reconstruction score merely because the resulting text looks approximately similar.
9. Unicode and Grapheme Testing
The benchmark tested:
- Devanagari
- combining marks
- Latin accented characters
- multiple writing systems
- mathematical symbols
- currency symbols
- emoji
- complex Unicode sequences
Grapheme preservation was also tested because a Unicode character sequence can represent a single user-perceived character.
This is especially important for a tokenizer intended to support Nepali and multilingual text.
10. Nepali Quality
Nepali quality was evaluated separately from generic token efficiency.
The benchmark examined:
- Nepali word fragmentation
- complex Devanagari sequences
- conjuncts
- vowel signs
- rare Nepali words
- related-word consistency
- Nepali/English mixing
The benchmark does not assume that fewer tokens automatically means better linguistic quality.
11. Morphological Consistency
Morphological consistency was calculated using the actual textual representation of tokens.
Token IDs were intentionally excluded.
Related word families such as:
- नेपाल / नेपालमा
- नेपाली / नेपालीहरू
- शिक्षा / शिक्षाले
- विज्ञान / वैज्ञानिक
- समाज / सामाजिक
- राष्ट्र / राष्ट्रिय
- स्वतन्त्र / स्वतन्त्रता
were examined for preservation of shared textual structure.
12. Token Efficiency
Token efficiency is reported independently as:
characters per token
Higher values generally indicate that more input text is represented per token.
However:
Token efficiency is NOT equivalent to linguistic quality.
A tokenizer can achieve fewer tokens by aggressively merging text without necessarily producing a better representation.
Therefore this metric was never allowed to determine the final winner by itself.
13. Determinism
Both tokenizers were tested repeatedly on identical inputs.
A deterministic tokenizer should produce the same token sequence every time.
This is important for reproducibility and model training.
14. Final Assessment
Supernova strengths
Based on the completed benchmarks, Supernova's strongest areas are the dimensions in which it won above.
In particular, its runtime performance was:
3.20× the character throughput of NepaliBPE
and:
4.68× the token throughput of NepaliBPE
The quality benchmark provides the independent evidence for whether that performance is accompanied by strong text representation.
NepaliBPE strengths
NepaliBPE's strongest areas are likewise the dimensions in which it won above.
Its 50,006-token vocabulary and BPE representation can provide substantially different token segmentation from Supernova's custom longest-match approach.
The final quality table—not popularity, search ranking, downloads, or vocabulary size—is what determines the technical comparison.
15. Speed vs Quality
The results should be interpreted as two separate questions.
Question A — Which tokenizer is faster?
Supernova
Question B — Which tokenizer demonstrated stronger quality across the tested dimensions?
Supernova
Question C — Which tokenizer is the stronger overall candidate?
Supernova
The overall verdict prioritizes tokenizer quality rather than allowing raw speed to compensate for information loss or unknown-token failures.
16. Important Limitations
This is a serious engineering benchmark, but it is not equivalent to training a large language model with each tokenizer and measuring downstream perplexity.
A definitive production-level tokenizer evaluation would additionally include:
- large real-world Nepali corpora
- held-out Nepali corpora
- downstream language-model training
- validation perplexity
- compression ratio
- vocabulary utilization
- fertility
- sequence-length distribution
- embedding/model compatibility
- multilingual robustness
- domain-specific corpora
- OCR/noisy-text robustness
Therefore the conclusion here means:
Winner under the tested tokenizer-level benchmark conditions.
It does not claim that one tokenizer is universally superior for every possible language-model architecture.
17. Final Verdict
🏆 Supernova
The benchmark evidence indicates that Supernova is the stronger overall tokenizer under the completed test suite.
The conclusion is based on measured behavior rather than:
- Google ranking
- popularity
- repository fame
- vocabulary size
- download count
- marketing claims
- arbitrary speed weighting
The final decision comes from reproducible benchmark measurements.
18. Benchmark Integrity Statement
This comparison deliberately avoids several common benchmarking errors:
- No cross-tokenizer token-ID comparison
- No incorrect
unk_token_idassumption - No speed-only winner declaration
- No automatic assumption that fewer tokens means better language quality
- No arbitrary claim that vocabulary size equals tokenizer quality
- No hidden competitor-specific advantage
- Same corpus categories for both tokenizers
- Fresh tokenizer loading
- Raw token evidence inspected
- Unknown tokens counted from emitted token strings
- Reconstruction tested independently
FINAL RESULT
Speed winner: Supernova
Quality winner: Supernova
Overall winner: Supernova
Benchmark status: COMPLETE