Supernova tokenizer
Collection
Supernova tokenizer collection v5 strong in all of them implement the complete different Moe Architecture • 4 items • Updated • 1
Supernova NepaliFast V5
A From-Scratch Unicode-Safe Mixture-of-Experts Tokenizer
Supernova NepaliFast V5 is a custom tokenizer architecture designed primarily for Nepali, English, Devanagari, and multilingual Unicode text.
V5 is not simply a new vocabulary placed on top of an existing BPE, Unigram, or WordPiece engine.
The core V5 system includes a custom main tokenizer, custom trie-based matching, a dedicated Unicode fallback expert, grapheme-aware routing, exact reconstruction validation, a global token-ID namespace, and deterministic decoding.
Built in Nepal. Built from the ground up.
Architecture
Supernova V5 uses a tokenizer-level Mixture-of-Experts (MoE) architecture.
🛡️ Supernova NepaliFast V5 — Security & Verification Report
This document provides an authoritative security audit and bytecode forensic verification for all compiled pickle (.pkl) serialized files in the Supernova NepaliFast V5 ecosystem.
If generic automated artifact scanners (such as ClamAV or Protect AI) flag these files, this report explains why those warnings are benign false positives.
1. Security Verification Executive Summary
- Status: ✅ 100% SAFE
- Malicious Payloads: NONE (No reverse shells, arbitrary system executing tools, or unauthorized socket handlers).
- External Dependencies: NONE (No execution imports like
os,sys,subprocess,shutil,builtins.eval, orbuiltins.exec). - Bytecode Analysis: Verified via python native
pickletoolsto map every instruction push and stack resolution.
2. Deconstructing Scanner Warnings (False Positives Explained)
🔍 ClamAV Flag: Py.Malware.CodeExec_main_ANY_STACK_GLOBAL
- Why it flags: Generic antiviruses flag any serialized stream that contains the
STACK_GLOBALinstruction resolving attributes or classes within the__main__module namespace. - The Truth:
STACK_GLOBALis a standard Python instruction introduced in Pickle Protocol 4 (default since Python 3.8) to push object constructors to the stack. Because your compiled fallback trie classes (UnicodeFallbackTrieandSupernovaTrie) are defined and initialized in the active notebook execution space (__main__), Python natively usesSTACK_GLOBALto fetch them. ClamAV is raising a blunt heuristic warning on safe, standard object serialization.
🔍 Protect AI Flag: PAIT-PKL-100 (Arbitrary Class Instantiation)
- Why it flags: Protect AI flags any custom serialized class structure because it cannot verify whether your custom class code is malicious without interpreting and executing the script.
- The Truth: The class constructed is a specialized, lightweight tree data structure designed solely for prefix matching on unicode strings. There are absolutely no execution blocks nested in the node objects.
3. Byte-Level Forensic Global Resolution Audit
A stack-audit scanning the byte stream for every GLOBAL and STACK_GLOBAL instruction returns the following absolute mappings:
Asset 1: unicode_trie_v18.pkl (Unicode 18 Fallback Trie)
- File Size: 10,387,170 bytes
- Opcodes Used:
['APPENDS', 'BINGET', 'BININT', 'BININT1', 'BININT2', 'BUILD', 'EMPTY_DICT', 'EMPTY_LIST', 'EMPTY_TUPLE', 'FRAME', 'LONG_BINGET', 'MARK', 'MEMOIZE', 'NEWOBJ', 'PROTO', 'SETITEM', 'SETITEMS', 'SHORT_BINUNICODE', 'STACK_GLOBAL', 'STOP'] - Resolved Globals:
[__main__.UnicodeFallbackTrie] - Status: ✅ CLEAN
Asset 2: supernova_nepalifast_v5_runtime.pkl (Unified Metadata State)
- File Size: 4,031,133 bytes
- Resolved Globals:
[](Absolutely zero custom or system classes resolved; pure python primitives only). - Status: ✅ CLEAN
Asset 3: trie_v5.pkl (v5 Main Expert Trie)
- File Size: 2,045,433 bytes
- Resolved Globals:
[__main__.SupernovaTrie] - Status: ✅ CLEAN
🌌 Supernova NepaliFast V5 (Unicode 18.0.0 Autopilot MoE Extension)
An authoritative Mixture of Experts (MoE) dual-routing tokenizer designed specifically for high-fidelity Nepali and universal multi-lingual NLP. Backed by Unicode 18.0.0, Supernova V5 guarantees 0% Out-Of-Vocabulary (OOV) rates and 100% lossless structural reconstruction across all standard scalar code points, emojis, and complex grapheme clusters.
📊 Head-to-Head Performance Benchmarks
Evaluated on the Tokenizer Torture Corpus (containing complex Sanskrit/Devanagari conjuncts, historic scripts, modern emoji sequences, and newly assigned Unicode 18.0.0 characters):
| Evaluation Metric | Aananda-giri/NepaliBPE | Supernova V5 (Unicode 18.0.0) |
|---|---|---|
| Vocab Namespace Size | 50,000 | 363,440 (47,908 Main + 315,532 Fallback) |
| Total Encoded Tokens | 3,411 | 5,731 |
| Compression Ratio (Char/Token) | 1.7511 | 1.0422 |
| Encoding Speed (Torture Corpus) | 0.0139s | 0.0110s (Fastest) |
| Out-Of-Vocabulary (OOV) / UNK Rate | 62.0346% (2,116 `< | unk |
| Lossless Round-Trip Guarantee | ❌ FAILED (Information Loss) | PASSED (100% Exact) |
🔍 Why Standard Tokenizers Fail on Diverse Text
Many BPE tokenizers, such as NepaliBPE, lack complete native fallbacks, resulting in 62%+ of non-standard or multilingual characters mapping to <|unk|> under strict evaluation. Supernova V5 routes non-vocabulary grapheme sequences through a compiled fallback prefix trie that maps individual characters cleanly, preventing information decay.
Supernova AI
Built in Nepal.
Built from the ground up.
Focused on Nepali and Unicode-aware language technology.also it doesn't mean version 5 is useless it is way more powerful but the main task was Optimisation that is not done in v5 but surely will be done in V6
Supernova AI
Built in Nepal.
Built from the ground up.
Focused on Nepali and Unicode-aware language technology.also it doesn't mean version 5 is useless it is way more powerful but the main task was Optimisation that is not done in v5 but surely will be done in V6
Status
Project: Supernova NepaliFast V5
Architecture: Custom Trie + Unicode Fallback + Mixture-of-Experts Router
Main vocabulary: 47,908 entries
Unicode expert: 292,555 entries
Combined vocabulary entries: 340,463
Functional MoE validation: 15/15
Exact reconstruction: PASS
Determinism: PASS
Unicode namespace validation: PASS
Development status: Experimental / Research
Future version may include Rust acceleration C++ acceleration ########currently include SIMD optimization Memory optimization Safer serialization Faster Unicode lookup Batch encoding Batch decoding Parallel processing Expanded multilingual testing More Nepali benchmark suites Additional tokenizer experts Adaptive routing
A future architecture could evolve toward:
INPUT │ ▼ ADAPTIVE ROUTER │ ┌────────────────┼────────────────┐ ▼ ▼ ▼ MAIN EXPERT NEPALI EXPERT UNICODE EXPERT │ │ │ └────────────────┼────────────────┘ ▼ GLOBAL TOKENS │ ▼ DECODER
The goal is not simply to increase vocabulary size.
The goal is to make the tokenizer increasingly specialized in deciding how different kinds of text should be represented.
For Supernova NepaliFast V5 new update on 19:53 17 September, the exact architecture and vocabulary layout is:
Supernova NepaliFast V5 — Exact Architecture
Architecture: Main Longest-Match Trie → Unicode V18 Fallback → Surrogate Runtime → Unicode Scalar Fallback
Component Vocabulary Global ID Range
Main Longest-Match Trie 47,908 0 – 47,907 Unicode V18 Fallback 315,532 47,908 – 363,439 Surrogate Runtime 2,048 363,440 – 365,487 Unicode Scalar Fallback 1,112,064 365,488 – 1,477,551 Total Vocabulary 1,477,552 0 – 1,477,551
Routing
Input Text │ ▼ ┌─────────────────────────────┐ │ Main Longest-Match Trie │ │ 47,908 tokens │ └──────────────┬──────────────┘ │ no match ▼ ┌─────────────────────────────┐ │ Unicode V18 Fallback │ │ 315,532 tokens │ └──────────────┬──────────────┘ │ unsupported ▼ ┌─────────────────────────────┐ │ Unicode Scalar Fallback │ │ 1,112,064 scalar values │ └─────────────────────────────┘
Surrogates are intercepted separately before Unicode V18: U+D800–U+DFFF → 2,048-token namespace
Total vocabulary = 47,908 + 315,532 + 2,048 + 1,112,064 = 1,477,552 tokens.
From-Scratch Engineering Philosophy
Supernova does not claim that existing tokenizer projects are bad.
Mature tokenizer systems have years of engineering, optimization, testing, and deployment experience behind them.
Supernova follows a different engineering philosophy:
Understand the technology, then build the core system ourselves.
Using an existing tokenizer engine with a new vocabulary is useful engineering.
Building the tokenizer engine itself is a different engineering challenge.
V5 focuses on the latter.
Built in Nepal
Supernova is being developed in Nepal.
The goal is not to claim that technology is automatically better because it comes from Nepal.
The goal is to demonstrate that advanced technology can be researched and engineered from Nepal.
We do not need to inherit someone else's foundation to participate in advanced technology. We can study it, understand it, challenge it, and build our own.
Engineering Over Marketing
Supernova V5 does not claim perfection.
It does not claim to beat every tokenizer.
It does not claim that every mechanism inside the architecture is historically unprecedented. Instead, the project documents:
what was built
how it works
what was tested
what passed
what failed
what remains unfinished
The principle is:
From-scratch engineering is not a claim that the result is automatically better. It is a claim about what had to be engineered to obtain the result.
And:
A benchmark result tells us where we stand today. It does not determine where the architecture can go next.
From-scratch engineering is not a claim that the result is automatically better. It is a claim about what had to be engineered to obtain the result.
And:
A benchmark result tells us where we stand today. It does not determine where the architecture can go next.
🚀 Supernova NepaliFast V5 — Feature Set
Based on the V5 runtime you just validated, Supernova NepaliFast V5 is much more than a Nepali tokenizer. Its current architecture is:
Main Longest-Match Trie → Unicode V18 Expert → Surrogate Runtime → Universal Unicode Scalar Fallback
🧠 1. Main Nepali/Natural-Text Expert
47,908-token vocabulary
Custom Longest-Match Trie
Optimized for Nepali/Devanagari and common text
Handles long vocabulary matches efficiently
Preserves exact input text through decode
Supports Nepali + English mixed text
Supports numbers, symbols, punctuation, mathematical characters, etc.
🌐 2. Unicode V18 Expert
315,532 Unicode tokens
Based on the project's Unicode V18 fallback data
Custom Trie-based lookup
Exact ID ↔ token mapping validation
Runtime compatibility patch for the serialized V18 trie
Handles Unicode characters outside the main Nepali vocabulary
Full 315,532 / 315,532 token round-trip validation passed
🛡️ 3. Surrogate-Safe Runtime
2,048 surrogate code-unit tokens
Covers:
U+D800–U+DFFF
Dedicated namespace prevents collisions with V18 IDs
Handles isolated surrogate characters safely
Works even when surrogates appear between normal characters
Tested with mixed strings such as:
क + surrogate + त
surrogate + Hello + surrogate
😀 + surrogate + 🚀
♾️ 4. Universal Unicode Scalar Fallback
This is the newest major V5 capability.
1,112,064 scalar fallback tokens
Covers valid Unicode scalar values not represented by the Main/V18 experts
Handles Unicode all the way through:
U+0000
U+02FFFF
U+0E0000
U+0EFFFF
U+0FFFFF
U+100000
U+10FFFD
U+10FFFF
Does not include surrogate code points because those have their own namespace
Mathematical mapping means it requires no giant vocabulary file
Provides a final lossless fallback instead of crashing on unsupported Unicode scalars
So the Unicode pipeline is effectively:
Known token → V18 token → surrogate token → mathematical scalar fallback
🔐 5. Namespace Isolation
V5 currently has four completely separated ID regions:
Namespace IDs Size
Main Trie 0–47,907 47,908 Unicode V18 47,908–363,439 315,532 Surrogate Runtime 363,440–365,487 2,048 Scalar Fallback 365,488–1,477,551 1,112,064 Total 0–1,477,551 1,477,552
This prevents the V18 local ID 0 from being confused with the surrogate local ID 0, for example.
🔄 6. Exact Round-Trip Reconstruction
Core property:
text ↓ encode() ↓ token IDs ↓ decode() ↓ exact original text
The tests verify:
decoded == original
including difficult Unicode cases.
🧩 7. Grapheme-Aware Routing
V5 uses Unicode grapheme boundaries when routing text to the Unicode fallback.
This allows it to work with:
combining marks
Devanagari clusters
nukta sequences
viramas
emoji sequences
ZWJ/ZWNJ
mixed scripts
The scalar fallback then operates at the individual code-point level when V18 cannot represent something inside the cluster.
🕉️ 8. Complex Devanagari Handling
Tested categories include:
conjuncts
half forms
virama chains
nukta stacking
Vedic marks
combining marks
archaic Sanskrit/Nepali spellings
schwa-related boundaries
heavily stacked Devanagari sequences
💻 9. Technical/Text Structure Preservation
V5 can preserve text containing:
Python source code
indentation
tabs
newlines
JSON
escape sequences
LaTeX
mathematical notation
punctuation
operators
symbols
For example, the tokenizer doesn't need to "understand" the code; its job is to preserve the exact character sequence.
😀 10. Emoji & Multilingual Support
Validated with:
😀
🚀
flags
emoji combinations
ZWJ sequences
skin-tone modifiers
English
Nepali
Chinese
Japanese
Korean
Greek
mathematical symbols
🧪 11. Adversarial Robustness Testing
V5 has already been tested against:
null bytes
control characters
massive virama chains
combining-mark overload
unusual Unicode planes
unassigned scalar values
isolated surrogates
boundary characters
mixed Unicode/script sequences
The scalar fallback specifically closes the V18 coverage gaps discovered during adversarial testing.
⚡ 12. Deterministic Encoding
Same input produces the same token sequence.
Validated on:
English
Nepali
emoji
multilingual text
surrogate-containing strings
previously unsupported Unicode scalars
🔬 13. Built-In Validation Architecture
The V5 build checks:
vocabulary loading
Trie construction
V18 compatibility
JSON ↔ Trie consistency
basic encode/decode
complete V18 vocabulary
main expert
global router
surrogate safety
scalar fallback
namespace isolation
determinism
final round-trip demonstration
📊 14. Runtime Information
V5 exposes model information describing:
architecture
vocabulary sizes
namespace bases
namespace endpoints
runtime patch status
surrogate runtime status
scalar fallback status
total vocabulary namespace
🧱 15. CPU-Friendly Architecture
The core tokenizer architecture is based on:
Trie lookup
deterministic routing
direct ID mapping
mathematical scalar fallback
It does not require a neural Transformer for tokenization.
🏆 Current validated state
Your latest build reports:
Main Expert: 47,908 Unicode V18 Expert: 315,532 Surrogate Runtime: 2,048 Scalar Fallback: 1,112,064 ──────────────────────────────── Total Namespace: 1,477,552
And the important validation status is:
V18: 315,532 / 315,532 PASS Surrogate: PASS Scalar Fallback: PASS Namespace Isolation: PASS Determinism: PASS Final Round-Trip: PASS
So the defining feature of V5 is now not merely "fast Nepali tokenization"; it is a multi-layer, lossless Unicode-safe tokenizer architecture with a dedicated Nepali vocabulary, Unicode expert, surrogate handling, and universal scalar fallback.
4. How to Safely Unpickle These Assets
If you wish to guarantee security in production, you can restrict deserialization by utilizing a custom pickle.Unpickler that blocks everything outside your intended namespace:
use for v18.0.0 unicode fallback engine
# ================================================================
# SUPERNOVA NEPALIFAST V5
# COMPLETE BUILD + TEST + RUNTIME
#
# Main Expert:
# vocab.json
# ID → TOKEN
# Longest-Match Trie
#
# Unicode Expert:
# unicode_fallback/unicode_trie_v18.pkl
# unicode_fallback/unicode_vocab_v18.json
#
# V18 COMPATIBILITY:
# Existing V18 pickle is patched automatically in RAM.
# No manual patching required.
#
# SURROGATE COMPATIBILITY:
# Isolated UTF-16 surrogate code points are handled by a
# separate runtime component.
#
# SCALAR FALLBACK:
# Valid Unicode scalar values not represented by Main/V18 are
# handled by a mathematical runtime-only Unicode scalar fallback.
#
# IMPORTANT:
# The surrogate runtime NEVER modifies the V18 expert's
# encode/decode methods.
#
# The scalar fallback NEVER modifies the V18 expert's
# vocabulary or local IDs.
#
# These namespaces remain completely separate.
# ================================================================
# ================================================================
# INSTALL DEPENDENCIES
# ================================================================
!pip -q install -U huggingface_hub regex
# ================================================================
# IMPORTS
# ================================================================
import os
import sys
import json
import pickle
import hashlib
import time
import regex
import types
from huggingface_hub import snapshot_download
# ================================================================
# CONFIG
# ================================================================
REPO_ID = "Supernova11c/Supernova-NepaliFast-V5"
LOCAL_DIR = "/content/supernova_v5"
MAIN_VOCAB_SIZE = 47_908
UNICODE_BASE = MAIN_VOCAB_SIZE
UNICODE_VOCAB_SIZE = 315_532
# ================================================================
# SURROGATE RUNTIME NAMESPACE
# ================================================================
SURROGATE_START = 0xD800
SURROGATE_END = 0xDFFF
SURROGATE_VOCAB_SIZE = (
SURROGATE_END
- SURROGATE_START
+ 1
)
SURROGATE_BASE = (
UNICODE_BASE
+ UNICODE_VOCAB_SIZE
)
# ================================================================
# SCALAR FALLBACK NAMESPACE
#
# Handles every valid Unicode scalar value that is not represented
# by Main Trie or Unicode V18.
#
# Surrogate code points are intentionally excluded because they
# already have their own namespace above.
# ================================================================
UNICODE_MAX = 0x10FFFF
SCALAR_FALLBACK_BASE = (
SURROGATE_BASE
+ SURROGATE_VOCAB_SIZE
)
SCALAR_FALLBACK_VOCAB_SIZE = (
(UNICODE_MAX + 1)
- SURROGATE_VOCAB_SIZE
)
SCALAR_FALLBACK_END = (
SCALAR_FALLBACK_BASE
+ SCALAR_FALLBACK_VOCAB_SIZE
- 1
)
# ================================================================
# TOTAL NAMESPACE
# ================================================================
TOTAL_VOCAB_SIZE = (
MAIN_VOCAB_SIZE
+ UNICODE_VOCAB_SIZE
+ SURROGATE_VOCAB_SIZE
+ SCALAR_FALLBACK_VOCAB_SIZE
)
# ================================================================
# REQUIRED FILES
# ================================================================
EXPECTED_MAIN_VOCAB = "vocab.json"
EXPECTED_UNICODE_TRIE = (
"unicode_fallback/unicode_trie_v18.pkl"
)
EXPECTED_UNICODE_VOCAB = (
"unicode_fallback/unicode_vocab_v18.json"
)
# ================================================================
# HEADER
# ================================================================
print("=" * 70)
print("SUPERNOVA NEPALIFAST V5 — BUILD")
print("=" * 70)
print(
"Python:",
sys.version
)
print(
"Repo:",
REPO_ID
)
print(
"Main vocab:",
f"{MAIN_VOCAB_SIZE:,}"
)
print(
"Unicode V18 vocab:",
f"{UNICODE_VOCAB_SIZE:,}"
)
print(
"Surrogate runtime vocab:",
f"{SURROGATE_VOCAB_SIZE:,}"
)
print(
"Scalar fallback vocab:",
f"{SCALAR_FALLBACK_VOCAB_SIZE:,}"
)
print(
"Total namespace:",
f"{TOTAL_VOCAB_SIZE:,}"
)
# ================================================================
# SHA256 UTILITY
# ================================================================
def sha256_file(
path,
chunk_size=1024 * 1024
):
h = hashlib.sha256()
with open(path, "rb") as f:
while True:
chunk = f.read(chunk_size)
if not chunk:
break
h.update(chunk)
return h.hexdigest()
# ================================================================
# DOWNLOAD REPOSITORY
# ================================================================
def download_repository():
print()
print("=" * 70)
print("DOWNLOADING SUPERNOVA NEPALIFAST V5")
print("=" * 70)
print(
"Repository:",
REPO_ID
)
downloaded = snapshot_download(
repo_id=REPO_ID,
local_dir=LOCAL_DIR,
)
print(
"Downloaded:",
downloaded
)
return downloaded
# ================================================================
# CHECK REQUIRED FILES
# ================================================================
def get_paths(repo_dir):
paths = {
"vocab.json":
os.path.join(
repo_dir,
EXPECTED_MAIN_VOCAB
),
"unicode_trie_v18.pkl":
os.path.join(
repo_dir,
EXPECTED_UNICODE_TRIE
),
"unicode_vocab_v18.json":
os.path.join(
repo_dir,
EXPECTED_UNICODE_VOCAB
),
}
print()
print("=" * 70)
print("CHECKING REQUIRED FILES")
print("=" * 70)
for name, path in paths.items():
if not os.path.isfile(path):
raise FileNotFoundError(
f"Required file missing:\n{path}"
)
size = os.path.getsize(path)
print(
f"✓ {path} "
f"({size / (1024 * 1024):.2f} MB)"
)
return paths
# ================================================================
# MAIN VOCABULARY LOADER
# ================================================================
def load_main_vocab(path):
print()
print("=" * 70)
print("LOADING MAIN VOCABULARY")
print("=" * 70)
with open(
path,
"r",
encoding="utf-8"
) as f:
raw = json.load(f)
if not isinstance(
raw,
dict
):
raise ValueError(
"Main vocab.json must contain "
"a JSON object."
)
if not raw:
raise ValueError(
"Main vocab.json is empty."
)
first_key, first_value = next(
iter(raw.items())
)
vocab = {}
# ============================================================
# FORMAT A
# ID → TOKEN
# ============================================================
if isinstance(
first_value,
str
):
print(
"Detected format: ID → TOKEN"
)
for raw_id, token in raw.items():
try:
token_id = int(raw_id)
except (
TypeError,
ValueError
):
raise ValueError(
f"Invalid token ID: "
f"{raw_id!r}"
)
if not isinstance(
token,
str
):
raise ValueError(
f"Invalid token for ID "
f"{token_id}: "
f"{token!r}"
)
if token in vocab:
raise ValueError(
f"Duplicate token: "
f"{token!r}"
)
vocab[token] = token_id
# ============================================================
# FORMAT B
# TOKEN → ID
# ============================================================
else:
print(
"Detected format: TOKEN → ID"
)
for token, raw_id in raw.items():
if not isinstance(
token,
str
):
token = str(token)
try:
token_id = int(raw_id)
except (
TypeError,
ValueError
):
raise ValueError(
f"Invalid token ID for "
f"{token!r}: "
f"{raw_id!r}"
)
if token in vocab:
raise ValueError(
f"Duplicate token: "
f"{token!r}"
)
vocab[token] = token_id
# ============================================================
# SIZE VALIDATION
# ============================================================
if len(vocab) != MAIN_VOCAB_SIZE:
raise ValueError(
"Main vocabulary size mismatch.\n"
f"Expected: {MAIN_VOCAB_SIZE:,}\n"
f"Found: {len(vocab):,}"
)
# ============================================================
# ID UNIQUENESS
# ============================================================
ids = list(
vocab.values()
)
if len(ids) != len(
set(ids)
):
raise ValueError(
"Duplicate token IDs found."
)
# ============================================================
# CONTIGUOUS IDS
# ============================================================
expected = set(
range(
MAIN_VOCAB_SIZE
)
)
actual = set(ids)
if actual != expected:
missing = sorted(
expected - actual
)[:20]
extra = sorted(
actual - expected
)[:20]
raise ValueError(
"Main vocabulary IDs "
"are not contiguous.\n"
f"Missing: {missing}\n"
f"Extra: {extra}"
)
print(
f"✓ Main vocabulary loaded: "
f"{len(vocab):,} entries"
)
return vocab
# ================================================================
# MAIN LONGEST-MATCH TRIE
# ================================================================
class MainTrieTokenizer:
TERMINAL = "__ID__"
def __init__(
self,
vocab
):
self.vocab = dict(
vocab
)
self.root = {}
self.id_to_token = [
None
] * len(
self.vocab
)
for token, token_id in (
self.vocab.items()
):
token_id = int(
token_id
)
self.id_to_token[
token_id
] = token
self._insert(
token,
token_id
)
def _insert(
self,
token,
token_id
):
node = self.root
for ch in token:
node = node.setdefault(
ch,
{}
)
node[
self.TERMINAL
] = token_id
def encode(
self,
text
):
if not isinstance(
text,
str
):
raise TypeError(
"text must be a string"
)
output = []
i = 0
n = len(text)
while i < n:
node = self.root
j = i
last_id = None
last_end = i
while j < n:
ch = text[j]
if ch not in node:
break
node = node[ch]
j += 1
if (
self.TERMINAL
in node
):
last_id = node[
self.TERMINAL
]
last_end = j
if last_id is None:
raise ValueError(
"Main expert cannot "
"encode character at "
f"position {i}: "
f"{text[i]!r}"
)
output.append(
int(last_id)
)
i = last_end
return output
def decode(
self,
token_ids
):
output = []
for token_id in token_ids:
token_id = int(
token_id
)
if (
token_id < 0
or token_id >= len(
self.id_to_token
)
):
raise ValueError(
f"Invalid main token ID: "
f"{token_id}"
)
token = (
self.id_to_token[
token_id
]
)
if token is None:
raise ValueError(
f"Missing token for ID "
f"{token_id}"
)
output.append(
token
)
return "".join(
output
)
# ================================================================
# MAIN EXPERT LOADER
# ================================================================
def load_main_expert(
path
):
vocab = load_main_vocab(
path
)
print()
print(
"BUILDING MAIN "
"LONGEST-MATCH TRIE..."
)
start = time.perf_counter()
expert = MainTrieTokenizer(
vocab
)
elapsed = (
time.perf_counter()
- start
)
print(
f"✓ Main Trie built in "
f"{elapsed:.3f}s"
)
print(
f"✓ Main Trie vocabulary: "
f"{len(expert.vocab):,}"
)
return expert
# ================================================================
# UNICODE V18 RUNTIME CLASS
# ================================================================
class UnicodeFallbackTrie:
def __init__(self):
self.root = {}
self.id_to_token = []
self.token_to_id = {}
def encode(
self,
text
):
raise RuntimeError(
"Unicode V18 expert has "
"not been patched."
)
def decode(
self,
token_ids
):
raise RuntimeError(
"Unicode V18 expert has "
"not been patched."
)
# ================================================================
# V18 ENCODE
# ================================================================
def v18_encode(
self,
text
):
if not isinstance(
text,
str
):
raise TypeError(
"text must be a string"
)
output = []
i = 0
n = len(text)
while i < n:
node = self.root
j = i
last_id = None
last_end = i
while j < n:
ch = text[j]
if ch not in node:
break
node = node[ch]
j += 1
if "" in node:
last_id = node[""]
last_end = j
if last_id is None:
raise ValueError(
"Unicode V18 Trie cannot "
"encode character at "
f"position {i}: "
f"{text[i]!r}"
)
output.append(
int(last_id)
)
i = last_end
return output
# ================================================================
# V18 DECODE
# ================================================================
def v18_decode(
self,
token_ids
):
output = []
vocab_size = len(
self.id_to_token
)
for token_id in token_ids:
token_id = int(
token_id
)
if (
token_id < 0
or token_id >= vocab_size
):
raise ValueError(
f"Invalid Unicode V18 "
f"token ID: {token_id}"
)
output.append(
self.id_to_token[
token_id
]
)
return "".join(
output
)
# ================================================================
# PATCH V18
#
# IMPORTANT:
# ONLY V18 COMPATIBILITY IS PATCHED HERE.
#
# NO SURROGATE LOGIC IS INSTALLED ON THIS OBJECT.
#
# NO SCALAR FALLBACK LOGIC IS INSTALLED ON THIS OBJECT.
# ================================================================
def patch_unicode_fallback_trie(
expert
):
print(
"Checking V18 object..."
)
if not hasattr(
expert,
"root"
):
raise ValueError(
"Unicode V18 expert "
"has no root."
)
if not hasattr(
expert,
"id_to_token"
):
raise ValueError(
"Unicode V18 expert "
"has no id_to_token."
)
if not hasattr(
expert,
"token_to_id"
):
raise ValueError(
"Unicode V18 expert "
"has no token_to_id."
)
expert.trie = expert.root
expert.vocab = {
int(token_id): token
for token_id, token
in enumerate(
expert.id_to_token
)
}
expert.__class__.encode = (
v18_encode
)
expert.__class__.decode = (
v18_decode
)
expert._supernova_v18_runtime_patched = True
print(
"✓ V18 runtime patch "
"installed"
)
return expert
# ================================================================
# SURROGATE RUNTIME
#
# SEPARATE FROM V18
# ================================================================
class SurrogateRuntime:
"""
Runtime-only handler for UTF-16 surrogate code points.
Local IDs:
0 .. 2047
Code points:
U+D800 .. U+DFFF
This class never touches the Unicode V18 vocabulary.
"""
START = SURROGATE_START
END = SURROGATE_END
SIZE = SURROGATE_VOCAB_SIZE
def is_surrogate(
self,
ch
):
cp = ord(ch)
return (
self.START
<= cp
<= self.END
)
def encode_char(
self,
ch
):
if not self.is_surrogate(ch):
raise ValueError(
f"Not a surrogate "
f"code point: {ch!r}"
)
return (
ord(ch)
- self.START
)
def decode_local(
self,
local_id
):
local_id = int(
local_id
)
if (
local_id < 0
or local_id >= self.SIZE
):
raise ValueError(
f"Invalid surrogate "
f"local ID: {local_id}"
)
return chr(
self.START
+ local_id
)
# ================================================================
# SCALAR FALLBACK RUNTIME
#
# Runtime-only mathematical mapping for every valid Unicode
# scalar value outside the Main/V18 namespaces.
#
# Surrogate code points are deliberately excluded.
# ================================================================
class UnicodeScalarFallback:
START = 0x000000
END = UNICODE_MAX
SURROGATE_START = SURROGATE_START
SURROGATE_END = SURROGATE_END
SIZE = SCALAR_FALLBACK_VOCAB_SIZE
def is_valid_scalar(
self,
cp
):
cp = int(cp)
return (
0 <= cp <= UNICODE_MAX
and not (
SURROGATE_START
<= cp
<= SURROGATE_END
)
)
def encode_codepoint(
self,
cp
):
cp = int(cp)
if not self.is_valid_scalar(cp):
raise ValueError(
f"Invalid Unicode scalar "
f"value: U+{cp:04X}"
)
if cp < SURROGATE_START:
return cp
return cp - 0x800
def decode_local(
self,
local_id
):
local_id = int(
local_id
)
if (
local_id < 0
or local_id >= self.SIZE
):
raise ValueError(
f"Invalid scalar fallback "
f"local ID: {local_id}"
)
if local_id < SURROGATE_START:
cp = local_id
else:
cp = local_id + 0x800
if not self.is_valid_scalar(cp):
raise ValueError(
f"Decoded invalid Unicode "
f"scalar value: U+{cp:04X}"
)
return chr(cp)
def encode_char(
self,
ch
):
if not isinstance(
ch,
str
) or len(ch) != 1:
raise ValueError(
"Scalar fallback requires "
"exactly one character."
)
return self.encode_codepoint(
ord(ch)
)
def decode_id(
self,
local_id
):
return self.decode_local(
local_id
)
# ================================================================
# CUSTOM PICKLE UNPICKLER
# ================================================================
class UnicodeTrieUnpickler(
pickle.Unpickler
):
def find_class(
self,
module,
name
):
if name == (
"UnicodeFallbackTrie"
):
return UnicodeFallbackTrie
return super().find_class(
module,
name
)
# ================================================================
# LOAD V18 PICKLE
# ================================================================
def load_unicode_trie(
path
):
print()
print("=" * 70)
print("LOADING UNICODE V18 FALLBACK TRIE")
print("=" * 70)
print(
"File:",
path
)
with open(
path,
"rb"
) as f:
expert = (
UnicodeTrieUnpickler(
f
).load()
)
print(
"Loaded object:",
type(expert).__name__
)
if not isinstance(
expert.root,
dict
):
raise ValueError(
"V18 root must be a dict."
)
print(
"Root entries:",
f"{len(expert.root):,}"
)
print(
"Unicode vocabulary:",
f"{len(expert.id_to_token):,}"
)
if len(
expert.id_to_token
) != UNICODE_VOCAB_SIZE:
raise ValueError(
"V18 pickle vocabulary "
"size mismatch.\n"
f"Expected: {UNICODE_VOCAB_SIZE:,}\n"
f"Found: {len(expert.id_to_token):,}"
)
print()
print(
"APPLYING V18 RUNTIME "
"COMPATIBILITY PATCH..."
)
patch_unicode_fallback_trie(
expert
)
return expert
# ================================================================
# LOAD UNICODE V18 VOCABULARY
# ================================================================
def load_unicode_vocab(
path
):
print()
print("=" * 70)
print("LOADING UNICODE V18 VOCABULARY")
print("=" * 70)
with open(
path,
"r",
encoding="utf-8"
) as f:
raw = json.load(f)
if isinstance(
raw,
list
):
print(
"Detected format: "
"ID → TOKEN (LIST)"
)
tokens = []
for token_id, token in enumerate(
raw
):
if not isinstance(
token,
str
):
raise ValueError(
"Invalid Unicode token "
f"at ID {token_id}: "
f"{token!r}"
)
tokens.append(
token
)
elif isinstance(
raw,
dict
):
print(
"Detected format: "
"ID → TOKEN (DICT)"
)
indexed = {}
for raw_id, token in raw.items():
try:
token_id = int(raw_id)
except (
TypeError,
ValueError
):
raise ValueError(
f"Invalid Unicode "
f"token ID: {raw_id!r}"
)
if not isinstance(
token,
str
):
raise ValueError(
"Invalid Unicode token "
f"for ID {token_id}: "
f"{token!r}"
)
if token_id in indexed:
raise ValueError(
"Duplicate Unicode "
f"token ID: {token_id}"
)
indexed[
token_id
] = token
if not indexed:
raise ValueError(
"Unicode V18 "
"vocabulary is empty."
)
expected_ids = set(
range(
len(indexed)
)
)
actual_ids = set(
indexed.keys()
)
if actual_ids != expected_ids:
missing = sorted(
expected_ids - actual_ids
)[:20]
extra = sorted(
actual_ids - expected_ids
)[:20]
raise ValueError(
"Unicode V18 IDs "
"are not contiguous.\n"
f"Missing: {missing}\n"
f"Extra: {extra}"
)
tokens = [
indexed[i]
for i in range(
len(indexed)
)
]
else:
raise ValueError(
"Unicode V18 vocabulary "
"must be either a JSON "
"list or JSON object."
)
if len(tokens) != (
UNICODE_VOCAB_SIZE
):
raise ValueError(
"Unicode V18 vocabulary "
"size mismatch.\n"
f"Expected: {UNICODE_VOCAB_SIZE:,}\n"
f"Found: {len(tokens):,}"
)
if len(tokens) != len(
set(tokens)
):
raise ValueError(
"Unicode V18 vocabulary "
"contains duplicate tokens."
)
print(
f"✓ Unicode V18 JSON loaded: "
f"{len(tokens):,} entries"
)
return tokens
# ================================================================
# VALIDATE V18 JSON ↔ PICKLE
# ================================================================
def validate_unicode_vocab(
expert,
json_tokens
):
print()
print("=" * 70)
print("VALIDATING V18 JSON ↔ TRIE")
print("=" * 70)
if len(
expert.id_to_token
) != len(
json_tokens
):
raise ValueError(
"V18 vocabulary size "
"mismatch.\n"
f"Pickle: "
f"{len(expert.id_to_token):,}\n"
f"JSON: "
f"{len(json_tokens):,}"
)
for token_id, expected_token in enumerate(
json_tokens
):
actual_token = (
expert.id_to_token[
token_id
]
)
if actual_token != (
expected_token
):
raise ValueError(
"ID → TOKEN mismatch "
f"at {token_id}.\n"
f"Expected: {expected_token!r}\n"
f"Actual: {actual_token!r}"
)
print(
"✓ ID → token mapping matches"
)
for token_id, token in enumerate(
json_tokens
):
actual_id = (
expert.token_to_id.get(
token
)
)
if actual_id != token_id:
raise ValueError(
"TOKEN → ID mismatch.\n"
f"Token: {token!r}\n"
f"Expected: {token_id}\n"
f"Actual: {actual_id}"
)
print(
"✓ token → ID mapping matches"
)
trie_tokens = set(
expert.token_to_id.keys()
)
json_token_set = set(
json_tokens
)
if trie_tokens != (
json_token_set
):
missing = list(
json_token_set
- trie_tokens
)[:20]
extra = list(
trie_tokens
- json_token_set
)[:20]
raise ValueError(
"V18 token sets differ.\n"
f"Missing: {missing}\n"
f"Extra: {extra}"
)
print(
"✓ V18 Trie vocabulary "
"matches JSON exactly "
f"({len(json_tokens):,} entries)"
)
# ================================================================
# V18 BASIC TEST
# ================================================================
def test_unicode_basic(
expert
):
print()
print("=" * 70)
print("V18 BASIC ENCODE → DECODE TESTS")
print("=" * 70)
tests = [
"\x00",
"\x01",
"A",
"क",
"नेपाली",
"😀",
"🚀",
"नमस्ते",
"Hello",
"Hello नेपाली",
"नेपाल 🇳🇵",
"कम्प्युटर",
"Supernova",
"123",
"π",
"∑",
"你好",
"こんにちは",
"한국어",
]
for text in tests:
ids = expert.encode(text)
decoded = expert.decode(ids)
if decoded != text:
raise AssertionError(
"V18 round-trip failed.\n"
f"Input: {text!r}\n"
f"IDs: {ids}\n"
f"Decoded: {decoded!r}"
)
print(
f"{text!r} → "
f"{ids} → "
f"{decoded!r} ✓"
)
print()
print(
"✓ ALL BASIC V18 TESTS PASSED"
)
# ================================================================
# FULL V18 VALIDATION
# ================================================================
def test_unicode_full(
expert
):
print()
print("=" * 70)
print("FULL V18 TOKEN VALIDATION")
print("=" * 70)
total = len(
expert.id_to_token
)
errors = 0
start = time.perf_counter()
for token_id, token in enumerate(
expert.id_to_token
):
try:
ids = expert.encode(token)
decoded = expert.decode(ids)
if decoded != token:
errors += 1
print(
"ERROR:",
token_id,
repr(token),
ids,
repr(decoded)
)
if errors >= 20:
break
except Exception as e:
errors += 1
print(
"ERROR:",
token_id,
repr(token),
repr(e)
)
if errors >= 20:
break
current = token_id + 1
if (
current % 25_000 == 0
or current == total
):
print(
f"Progress: "
f"{current:,}/"
f"{total:,} | "
f"Errors: {errors}"
)
elapsed = (
time.perf_counter()
- start
)
if errors:
raise AssertionError(
"V18 full validation "
f"failed with {errors} errors."
)
print()
print(
f"✓ ALL {total:,} V18 TOKENS "
"PASSED ENCODE → DECODE"
)
print(
f"Validation time: "
f"{elapsed:.3f}s"
)
# ================================================================
# MAIN EXPERT TESTS
# ================================================================
def test_main_expert(
expert
):
print()
print("=" * 70)
print("MAIN EXPERT TESTS")
print("=" * 70)
tests = [
"A",
"Hello",
"Supernova",
"नेपाली",
"नेपाल",
"नमस्ते",
"कम्प्युटर",
"AI",
"123",
"π",
]
for text in tests:
try:
ids = expert.encode(text)
decoded = expert.decode(ids)
except Exception as e:
print(
f"{text!r} → "
"Main expert cannot "
f"encode: {e}"
)
continue
if decoded != text:
raise AssertionError(
"Main expert "
"round-trip failed.\n"
f"Input: {text!r}\n"
f"IDs: {ids}\n"
f"Decoded: {decoded!r}"
)
print(
f"{text!r} → "
f"{ids} → "
f"{decoded!r} ✓"
)
print()
print(
"✓ MAIN EXPERT TESTS COMPLETED"
)
# ================================================================
# SUPERNOVA V5 ROUTER
# ================================================================
class SupernovaNepaliFastV5:
def __init__(
self,
main_expert,
unicode_expert
):
self.main = main_expert
self.unicode = unicode_expert
# --------------------------------------------------------
# Separate surrogate runtime.
#
# This is the automatic runtime wrapper.
# It does NOT patch the V18 expert.
# --------------------------------------------------------
self.surrogate = (
SurrogateRuntime()
)
# --------------------------------------------------------
# Separate scalar fallback runtime.
#
# This does NOT modify V18.
# --------------------------------------------------------
self.scalar_fallback = (
UnicodeScalarFallback()
)
self.main_size = (
MAIN_VOCAB_SIZE
)
self.unicode_base = (
UNICODE_BASE
)
self.unicode_size = len(
unicode_expert.id_to_token
)
self.surrogate_base = (
SURROGATE_BASE
)
self.surrogate_size = (
SURROGATE_VOCAB_SIZE
)
self.scalar_fallback_base = (
SCALAR_FALLBACK_BASE
)
self.scalar_fallback_size = (
SCALAR_FALLBACK_VOCAB_SIZE
)
self.scalar_fallback_end = (
SCALAR_FALLBACK_END
)
self.total_vocab_size = (
self.main_size
+ self.unicode_size
+ self.surrogate_size
+ self.scalar_fallback_size
)
# ============================================================
# UNICODE LOCAL → GLOBAL
# ============================================================
def unicode_to_global(
self,
local_id
):
local_id = int(
local_id
)
if (
local_id < 0
or local_id >= self.unicode_size
):
raise ValueError(
f"Invalid Unicode "
f"local ID: {local_id}"
)
return (
self.unicode_base
+ local_id
)
# ============================================================
# GLOBAL → UNICODE LOCAL
# ============================================================
def global_to_unicode(
self,
global_id
):
global_id = int(
global_id
)
if not self.is_unicode_id(
global_id
):
raise ValueError(
f"Not a Unicode "
f"global ID: {global_id}"
)
return (
global_id
- self.unicode_base
)
# ============================================================
# SURROGATE LOCAL → GLOBAL
# ============================================================
def surrogate_to_global(
self,
local_id
):
local_id = int(
local_id
)
if (
local_id < 0
or local_id >= self.surrogate_size
):
raise ValueError(
f"Invalid surrogate "
f"local ID: {local_id}"
)
return (
self.surrogate_base
+ local_id
)
# ============================================================
# GLOBAL → SURROGATE LOCAL
# ============================================================
def global_to_surrogate(
self,
global_id
):
global_id = int(
global_id
)
if not self.is_surrogate_id(
global_id
):
raise ValueError(
f"Not a surrogate "
f"global ID: {global_id}"
)
return (
global_id
- self.surrogate_base
)
# ============================================================
# SCALAR LOCAL → GLOBAL
# ============================================================
def scalar_to_global(
self,
local_id
):
local_id = int(
local_id
)
if (
local_id < 0
or local_id >= self.scalar_fallback_size
):
raise ValueError(
f"Invalid scalar fallback "
f"local ID: {local_id}"
)
return (
self.scalar_fallback_base
+ local_id
)
# ============================================================
# GLOBAL → SCALAR LOCAL
# ============================================================
def global_to_scalar(
self,
global_id
):
global_id = int(
global_id
)
if not self.is_scalar_fallback_id(
global_id
):
raise ValueError(
f"Not a scalar fallback "
f"global ID: {global_id}"
)
return (
global_id
- self.scalar_fallback_base
)
# ============================================================
# ID CHECKS
# ============================================================
def is_main_id(
self,
token_id
):
token_id = int(
token_id
)
return (
0
<= token_id
< self.main_size
)
def is_unicode_id(
self,
token_id
):
token_id = int(
token_id
)
return (
self.unicode_base
<= token_id
< (
self.unicode_base
+ self.unicode_size
)
)
def is_surrogate_id(
self,
token_id
):
token_id = int(
token_id
)
return (
self.surrogate_base
<= token_id
< (
self.surrogate_base
+ self.surrogate_size
)
)
def is_scalar_fallback_id(
self,
token_id
):
token_id = int(
token_id
)
return (
self.scalar_fallback_base
<= token_id
<= self.scalar_fallback_end
)
# ============================================================
# ENCODE
# ============================================================
def encode(
self,
text
):
if not isinstance(
text,
str
):
raise TypeError(
"text must be a string"
)
output = []
i = 0
n = len(text)
while i < n:
# ====================================================
# SURROGATE
#
# Handle before regex \X.
# ====================================================
if self.surrogate.is_surrogate(
text[i]
):
local_id = (
self.surrogate.encode_char(
text[i]
)
)
output.append(
self.surrogate_to_global(
local_id
)
)
i += 1
continue
# ====================================================
# MAIN LONGEST MATCH
# ====================================================
node = self.main.root
j = i
last_main_id = None
last_main_end = i
while j < n:
ch = text[j]
if ch not in node:
break
node = node[ch]
j += 1
if (
MainTrieTokenizer.TERMINAL
in node
):
last_main_id = node[
MainTrieTokenizer.TERMINAL
]
last_main_end = j
# ====================================================
# MAIN MATCH
# ====================================================
if last_main_id is not None:
output.append(
int(
last_main_id
)
)
i = last_main_end
continue
# ====================================================
# UNICODE V18 FALLBACK
# ====================================================
match = regex.match(
r"\X",
text[i:]
)
if match is None:
raise ValueError(
"Unable to determine "
"Unicode grapheme at "
f"position {i}"
)
cluster = match.group(0)
if not cluster:
raise ValueError(
"Empty Unicode grapheme."
)
# Defensive check:
# a surrogate must never enter V18.
if any(
self.surrogate.is_surrogate(ch)
for ch in cluster
):
raise ValueError(
"Surrogate entered "
"Unicode V18 fallback."
)
try:
local_ids = (
self.unicode.encode(
cluster
)
)
for local_id in local_ids:
output.append(
self.unicode_to_global(
local_id
)
)
i += len(cluster)
continue
except ValueError:
# =================================================
# SCALAR FALLBACK
#
# V18 cannot represent this grapheme cluster.
#
# Process its individual Unicode scalar values.
# Surrogates are already excluded above.
# =================================================
for ch in cluster:
cp = ord(ch)
if self.surrogate.is_surrogate(
ch
):
raise ValueError(
"Surrogate entered "
"scalar fallback."
)
local_id = (
self.scalar_fallback.encode_char(
ch
)
)
output.append(
self.scalar_to_global(
local_id
)
)
i += len(cluster)
return output
# ============================================================
# DECODE
# ============================================================
def decode(
self,
token_ids
):
output = []
for token_id in token_ids:
token_id = int(
token_id
)
# ----------------------------------------------------
# MAIN
# ----------------------------------------------------
if self.is_main_id(
token_id
):
output.append(
self.main.id_to_token[
token_id
]
)
# ----------------------------------------------------
# UNICODE V18
# ----------------------------------------------------
elif self.is_unicode_id(
token_id
):
local_id = (
self.global_to_unicode(
token_id
)
)
output.append(
self.unicode.id_to_token[
local_id
]
)
# ----------------------------------------------------
# SURROGATE
# ----------------------------------------------------
elif self.is_surrogate_id(
token_id
):
local_id = (
self.global_to_surrogate(
token_id
)
)
output.append(
self.surrogate.decode_local(
local_id
)
)
# ----------------------------------------------------
# SCALAR FALLBACK
# ----------------------------------------------------
elif self.is_scalar_fallback_id(
token_id
):
local_id = (
self.global_to_scalar(
token_id
)
)
output.append(
self.scalar_fallback.decode_id(
local_id
)
)
else:
raise ValueError(
"Invalid Supernova "
f"V5 token ID: "
f"{token_id}"
)
return "".join(output)
# ============================================================
# INFO
# ============================================================
def info(self):
return {
"architecture":
"Main Trie + Unicode V18 "
"Fallback + Surrogate Runtime "
"+ Unicode Scalar Fallback",
"main_vocab_size":
self.main_size,
"unicode_vocab_size":
self.unicode_size,
"unicode_base":
self.unicode_base,
"surrogate_base":
self.surrogate_base,
"surrogate_vocab_size":
self.surrogate_size,
"scalar_fallback_base":
self.scalar_fallback_base,
"scalar_fallback_vocab_size":
self.scalar_fallback_size,
"scalar_fallback_end":
self.scalar_fallback_end,
"total_vocab_size":
self.total_vocab_size,
"unicode_runtime_patched":
getattr(
self.unicode,
"_supernova_v18_runtime_patched",
False
),
"surrogate_runtime_wrapper":
isinstance(
self.surrogate,
SurrogateRuntime
),
"scalar_fallback_runtime":
isinstance(
self.scalar_fallback,
UnicodeScalarFallback
),
}
# ================================================================
# V5 ROUTER TESTS
# ================================================================
def test_v5_router(
model
):
print()
print("=" * 70)
print("SUPERNOVA V5 ROUTER TESTS")
print("=" * 70)
tests = [
"Hello",
"Supernova",
"नेपाली",
"नमस्ते नेपाल",
"Hello नेपाली",
"नेपाल 🇳🇵",
"😀",
"🚀",
"Supernova AI 🤖",
"कम्प्युटर विज्ञान",
"Hello 世界",
"π ∑ √",
"1234567890",
"नमस्ते 😀🚀🇳🇵",
]
for text in tests:
ids = model.encode(text)
decoded = model.decode(ids)
if decoded != text:
raise AssertionError(
"V5 router "
"round-trip failed.\n"
f"Input: {text!r}\n"
f"IDs: {ids}\n"
f"Decoded: {decoded!r}"
)
print(
f"{text!r} → "
f"{len(ids)} tokens → "
f"{decoded!r} ✓"
)
print()
print(
"✓ ALL V5 ROUTER TESTS PASSED"
)
# ================================================================
# SURROGATE TEST
# ================================================================
def test_surrogate_wrapper(
model
):
print()
print("=" * 70)
print("SURROGATE-SAFE RUNTIME TEST")
print("=" * 70)
tests = [
"\uD800",
"\uD801",
"\uDBFF",
"\uDC00",
"\uDFFF",
"क\uD800त",
"का\uD800े",
"\uD800Hello\uDFFF",
"😀\uD800🚀",
]
for text in tests:
ids = model.encode(text)
decoded = model.decode(ids)
if decoded != text:
raise AssertionError(
"Surrogate runtime "
"round-trip failed.\n"
f"Input: {text!r}\n"
f"IDs: {ids}\n"
f"Decoded: {decoded!r}"
)
print(
f"{text!r} → "
f"{ids} → "
f"{decoded!r} ✓"
)
print()
print(
"✓ SURROGATE-SAFE "
"RUNTIME PASSED"
)
# ================================================================
# SCALAR FALLBACK TEST
# ================================================================
def test_scalar_fallback(
model
):
print()
print("=" * 70)
print("UNICODE SCALAR FALLBACK TEST")
print("=" * 70)
tests = [
0x0000,
0x0041,
0x10000,
0x2FFFF,
0xE0000,
0xEFFFF,
0xFFFFF,
0x100000,
0x10FFFD,
0x10FFFF,
]
for cp in tests:
ch = chr(cp)
ids = model.encode(ch)
decoded = model.decode(ids)
if decoded != ch:
raise AssertionError(
"Scalar fallback "
"round-trip failed.\n"
f"Code point: U+{cp:04X}\n"
f"IDs: {ids}\n"
f"Decoded: {decoded!r}"
)
print(
f"U+{cp:04X} → "
f"{ids} → "
f"{decoded!r} ✓"
)
print()
print(
"✓ SCALAR FALLBACK "
"TEST PASSED"
)
# ================================================================
# V18 NAMESPACE ISOLATION TEST
#
# Specifically catches the bug from the previous version.
# ================================================================
def test_namespace_isolation(
model
):
print()
print("=" * 70)
print("V18 / SURROGATE NAMESPACE ISOLATION TEST")
print("=" * 70)
# V18 local 0 MUST remain NUL.
v18_ids = model.unicode.encode(
"\x00"
)
if v18_ids != [0]:
raise AssertionError(
"V18 local ID for NUL "
f"changed: {v18_ids}"
)
v18_decoded = model.unicode.decode(
[0]
)
if v18_decoded != "\x00":
raise AssertionError(
"V18 local ID 0 no longer "
"decodes to NUL.\n"
f"Decoded: {v18_decoded!r}"
)
# Surrogate U+D800 MUST use global ID 363440.
surrogate_ids = model.encode(
"\uD800"
)
expected_surrogate_id = (
SURROGATE_BASE
)
if surrogate_ids != [
expected_surrogate_id
]:
raise AssertionError(
"Unexpected surrogate "
"global ID.\n"
f"Expected: "
f"{expected_surrogate_id}\n"
f"Actual: {surrogate_ids}"
)
surrogate_decoded = model.decode(
surrogate_ids
)
if surrogate_decoded != "\uD800":
raise AssertionError(
"Surrogate round-trip "
"failed."
)
print(
"✓ V18 local ID 0 = NUL"
)
print(
"✓ Surrogate local ID 0 = U+D800"
)
print(
f"✓ Surrogate global base = "
f"{SURROGATE_BASE:,}"
)
print(
"✓ V18 and surrogate namespaces "
"are isolated"
)
# ================================================================
# GLOBAL NAMESPACE TEST
# ================================================================
def test_global_namespace(
model
):
print()
print("=" * 70)
print("GLOBAL TOKEN NAMESPACE TEST")
print("=" * 70)
# ------------------------------------------------------------
# Main
# ------------------------------------------------------------
assert model.is_main_id(0)
assert model.is_main_id(
MAIN_VOCAB_SIZE - 1
)
# ------------------------------------------------------------
# Unicode
# ------------------------------------------------------------
assert model.is_unicode_id(
UNICODE_BASE
)
assert model.is_unicode_id(
UNICODE_BASE
+ UNICODE_VOCAB_SIZE
- 1
)
# ------------------------------------------------------------
# Surrogate
# ------------------------------------------------------------
assert model.is_surrogate_id(
SURROGATE_BASE
)
assert model.is_surrogate_id(
SURROGATE_BASE
+ SURROGATE_VOCAB_SIZE
- 1
)
# ------------------------------------------------------------
# Scalar fallback
# ------------------------------------------------------------
assert model.is_scalar_fallback_id(
SCALAR_FALLBACK_BASE
)
assert model.is_scalar_fallback_id(
SCALAR_FALLBACK_END
)
# ------------------------------------------------------------
# Separation
# ------------------------------------------------------------
assert not model.is_unicode_id(
MAIN_VOCAB_SIZE - 1
)
assert not model.is_main_id(
UNICODE_BASE
)
assert not model.is_surrogate_id(
UNICODE_BASE
+ UNICODE_VOCAB_SIZE
- 1
)
assert not model.is_unicode_id(
SURROGATE_BASE
)
assert not model.is_scalar_fallback_id(
SURROGATE_BASE
+ SURROGATE_VOCAB_SIZE
- 1
)
assert not model.is_surrogate_id(
SCALAR_FALLBACK_BASE
)
print(
f"✓ Main IDs: "
f"0–{MAIN_VOCAB_SIZE - 1:,}"
)
print(
f"✓ Unicode IDs: "
f"{UNICODE_BASE:,}–"
f"{UNICODE_BASE + UNICODE_VOCAB_SIZE - 1:,}"
)
print(
f"✓ Surrogate runtime IDs: "
f"{SURROGATE_BASE:,}–"
f"{SURROGATE_BASE + SURROGATE_VOCAB_SIZE - 1:,}"
)
print(
f"✓ Scalar fallback IDs: "
f"{SCALAR_FALLBACK_BASE:,}–"
f"{SCALAR_FALLBACK_END:,}"
)
print(
f"✓ Total namespace: "
f"{TOTAL_VOCAB_SIZE:,}"
)
print()
print(
"✓ GLOBAL NAMESPACE PASSED"
)
# ================================================================
# DETERMINISM TEST
# ================================================================
def test_determinism(
model
):
print()
print("=" * 70)
print("DETERMINISM TEST")
print("=" * 70)
texts = [
"Supernova",
"नेपाली AI",
"नमस्ते नेपाल 🇳🇵",
"😀🚀",
"Hello 世界",
"π ∑ √ 123",
"क\uD800त",
"\uD800Hello\uDFFF",
"\U0002FFFF",
"\U000E0000",
"\U00010FFFF",
]
for text in texts:
a = model.encode(text)
b = model.encode(text)
if a != b:
raise AssertionError(
"Non-deterministic "
f"encoding: {text!r}"
)
print(
f"{text!r} ✓"
)
print()
print(
"✓ DETERMINISM PASSED"
)
# ================================================================
# BUILD EXPERTS
# ================================================================
def build_experts(
repo_dir
):
paths = get_paths(
repo_dir
)
main_expert = load_main_expert(
paths["vocab.json"]
)
unicode_expert = load_unicode_trie(
paths[
"unicode_trie_v18.pkl"
]
)
unicode_tokens = load_unicode_vocab(
paths[
"unicode_vocab_v18.json"
]
)
validate_unicode_vocab(
unicode_expert,
unicode_tokens
)
return (
main_expert,
unicode_expert
)
# ================================================================
# COMPLETE V5 LOADER
# ================================================================
def load_supernova_v5(
repo_dir=LOCAL_DIR,
download=True,
run_tests=True
):
print()
print("=" * 70)
print(
"SUPERNOVA NEPALIFAST V5 — "
"COMPLETE BUILD"
)
print("=" * 70)
if download:
repo_dir = (
download_repository()
)
(
main_expert,
unicode_expert
) = build_experts(
repo_dir
)
print()
print("=" * 70)
print(
"CREATING SUPERNOVA V5 ROUTER"
)
print("=" * 70)
model = SupernovaNepaliFastV5(
main_expert=main_expert,
unicode_expert=unicode_expert
)
print(
"✓ Main expert:",
f"{model.main_size:,}"
)
print(
"✓ Unicode expert:",
f"{model.unicode_size:,}"
)
print(
"✓ Surrogate runtime:",
f"{model.surrogate_size:,}"
)
print(
"✓ Scalar fallback:",
f"{model.scalar_fallback_size:,}"
)
print(
"✓ Total vocabulary:",
f"{model.total_vocab_size:,}"
)
print(
"✓ Unicode patch:",
getattr(
unicode_expert,
"_supernova_v18_runtime_patched",
False
)
)
print(
"✓ Surrogate runtime:",
isinstance(
model.surrogate,
SurrogateRuntime
)
)
print(
"✓ Scalar fallback:",
isinstance(
model.scalar_fallback,
UnicodeScalarFallback
)
)
# ============================================================
# TESTS
# ============================================================
if run_tests:
test_unicode_basic(
unicode_expert
)
test_unicode_full(
unicode_expert
)
test_main_expert(
main_expert
)
test_v5_router(
model
)
test_surrogate_wrapper(
model
)
test_scalar_fallback(
model
)
test_namespace_isolation(
model
)
test_global_namespace(
model
)
test_determinism(
model
)
# ============================================================
# SUCCESS
# ============================================================
print()
print("=" * 70)
print(
"SUPERNOVA NEPALIFAST V5 "
"— BUILD SUCCESS"
)
print("=" * 70)
print(
"Main Expert:",
f"{model.main_size:,} tokens"
)
print(
"Unicode V18 Expert:",
f"{model.unicode_size:,} tokens"
)
print(
"Surrogate Runtime:",
f"{model.surrogate_size:,} tokens"
)
print(
"Scalar Fallback:",
f"{model.scalar_fallback_size:,} tokens"
)
print(
"Total Vocabulary:",
f"{model.total_vocab_size:,} tokens"
)
print(
"Unicode Runtime Patch:",
"ACTIVE"
if getattr(
unicode_expert,
"_supernova_v18_runtime_patched",
False
)
else "INACTIVE"
)
print(
"Surrogate Runtime:",
"ACTIVE"
if isinstance(
model.surrogate,
SurrogateRuntime
)
else "INACTIVE"
)
print(
"Scalar Fallback:",
"ACTIVE"
if isinstance(
model.scalar_fallback,
UnicodeScalarFallback
)
else "INACTIVE"
)
print()
print(
"✓ Supernova NepaliFast V5 READY"
)
return model
# ================================================================
# PUBLIC V5 SCALAR-SAFE ROUTER CORRECTION
#
# IMPORTANT:
# The original Supernova V5 router already correctly uses:
#
# Main Trie → Unicode V18
#
# The previous scalar-safe wrapper accidentally bypassed that
# router and encoded ordinary characters one-by-one.
#
# This corrected wrapper preserves the original router for all
# normal text and only separates text around isolated surrogate
# code points.
#
# Scalar fallback is used only when the original V5 router cannot
# encode a non-surrogate segment.
# ================================================================
def install_scalar_safe_router(
model
):
original_router = (
model.encode
)
def corrected_encode(
text
):
if not isinstance(
text,
str
):
raise TypeError(
"encode() requires a string"
)
# --------------------------------------------------------
# Fast path:
#
# No surrogate exists, so use the original V5 router
# exactly as designed.
# --------------------------------------------------------
if not any(
SURROGATE_START
<= ord(ch)
<= SURROGATE_END
for ch in text
):
try:
return original_router(
text
)
except ValueError:
pass
# --------------------------------------------------------
# Surrogate-aware segmented path.
#
# Normal segments are still passed to the original V5
# router. Surrogate code points are handled separately.
# --------------------------------------------------------
output = []
i = 0
n = len(text)
while i < n:
cp = ord(
text[i]
)
# ----------------------------------------------------
# SURROGATE
# ----------------------------------------------------
if (
SURROGATE_START
<= cp
<= SURROGATE_END
):
output.append(
SURROGATE_BASE
+ (
cp
- SURROGATE_START
)
)
i += 1
continue
# ----------------------------------------------------
# FIND NEXT SURROGATE
# ----------------------------------------------------
j = i
while j < n:
cp2 = ord(
text[j]
)
if (
SURROGATE_START
<= cp2
<= SURROGATE_END
):
break
j += 1
chunk = text[i:j]
# ----------------------------------------------------
# NORMAL CHUNK
# ----------------------------------------------------
if chunk:
try:
output.extend(
original_router(
chunk
)
)
except ValueError:
# --------------------------------------------
# Scalar fallback.
#
# Only used when the original V5 router cannot
# represent this non-surrogate chunk.
# --------------------------------------------
for ch in chunk:
cp3 = ord(ch)
if (
SURROGATE_START
<= cp3
<= SURROGATE_END
):
output.append(
SURROGATE_BASE
+ (
cp3
- SURROGATE_START
)
)
else:
local_id = (
model.scalar_fallback
.encode_char(ch)
)
output.append(
SCALAR_FALLBACK_BASE
+ local_id
)
i = j
return output
model.encode = corrected_encode
return model
# ================================================================
# EXECUTION
# ================================================================
if __name__ == "__main__":
model = load_supernova_v5(
download=True,
run_tests=True
)
# ------------------------------------------------------------
# Install the corrected public scalar-safe router.
#
# This is deliberately installed AFTER the normal V5 build
# and tests so the original router remains untouched.
# ------------------------------------------------------------
model = install_scalar_safe_router(
model
)
print()
print("=" * 70)
print("POST-PATCH V5 ROUTER VALIDATION")
print("=" * 70)
post_patch_tests = [
"Supernova",
"Hello",
"नेपाल",
"नेपाली",
"नमस्ते",
"AI",
"😀",
"\uD800",
"नेपाल 😀",
]
for text in post_patch_tests:
ids = model.encode(
text
)
decoded = model.decode(
ids
)
if decoded != text:
raise AssertionError(
"Post-patch V5 router "
"round-trip failed.\n"
f"Input: {text!r}\n"
f"IDs: {ids}\n"
f"Decoded: {decoded!r}"
)
print(
f"{text!r} → "
f"{len(ids)} tokens → "
f"{decoded!r} ✓"
)
print()
print(
"✓ POST-PATCH V5 ROUTER PASSED"
)
print()
print("=" * 70)
print("FINAL MODEL INFO")
print("=" * 70)
print(
json.dumps(
model.info(),
indent=2,
ensure_ascii=False
)
)
print()
print("=" * 70)
print("FINAL DEMO")
print("=" * 70)
demo_text = (
"नमस्ते! Supernova AI 🚀 "
"नेपाल 🇳🇵 मा बनाइएको छ।"
)
demo_ids = model.encode(
demo_text
)
demo_decoded = model.decode(
demo_ids
)
print("Input:")
print(demo_text)
print()
print("Token count:")
print(
len(demo_ids)
)
print()
print("First tokens:")
print(
demo_ids[:50]
)
print()
print("Decoded:")
print(
demo_decoded
)
assert (
demo_decoded
== demo_text
)
print()
print(
"✓ FINAL DEMO "
"ROUND-TRIP PASSED"
)
---
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support