Upload folder using huggingface_hub

README.md

@@ -0,0 +1,395 @@
+---
+library_name: transformers
+pipeline_tag: text-generation
+language:
+  - en
+license: mit
+datasets:
+  - roneneldan/TinyStories
+tags:
+  - deepseek-v3
+  - multi-head-latent-attention
+  - mixture-of-experts
+  - moe
+  - tinystories
+  - tiny-model
+  - text-generation
+  - validation
+  - debug-model
+---
+
+# Tiny DeepSeek-V3 3M
+
+This repository contains a tiny `DeepseekV3ForCausalLM` Mixture-of-Experts
+language model trained from scratch on the full TinyStories training corpus.
+
+The model has **2,803,272 total parameters** and approximately **1,807,944
+parameters active per token**. It retains the main DeepSeek-V3 inference
+building blocks at a deliberately small scale: Multi-Head Latent Attention
+(MLA), Q and KV low-rank compression, interleaved RoPE, routed and shared
+experts, sigmoid top-k routing, and correction-bias-based load balancing.
+
+This is an independently trained synthetic tiny checkpoint. It is not an
+official DeepSeek model, contains no weights from an original DeepSeek
+checkpoint, and should not be expected to match the capabilities of production
+DeepSeek models.
+
+## Repository contents
+
+- `hf/`: the final Hugging Face checkpoint and tokenizer
+- `example_generate.py`: a minimal local generation example
+- `eval_text_generation.json`: generations from the final training evaluation
+- `artifact_metadata.json`: training arguments, metrics, router usage, and the
+  expanded configuration
+- `deepseek_v3_config_dump.json`: a standalone configuration dump
+
+Optimizer checkpoints, packed training data, and the full training log are
+intentionally omitted from the distribution package.
+
+## Architecture identity
+
+This checkpoint loads through the standard Transformers classes:
+
+```text
+DeepseekV3Config
+DeepseekV3ForCausalLM
+model_type: deepseek_v3
+```
+
+It does not use `DeepseekV2ForCausalLM` with a renamed model card. Important
+V3-specific behavior includes sigmoid routing, normalized selected routing
+weights, interleaved RoPE, Q LoRA compression, and an expert correction bias.
+
+## Model architecture
+
+```yaml
+architecture: DeepseekV3ForCausalLM
+model_type: deepseek_v3
+total_parameter_count: 2,803,272
+active_parameters_per_token: 1,807,944
+
+model_vocab_size: 1,024
+tokenizer_size: 1,003
+hidden_size: 216
+intermediate_size: 432
+num_hidden_layers: 5
+first_dense_layers: 1
+moe_layers: 4
+
+num_attention_heads: 8
+num_key_value_heads: 8
+q_lora_rank: 64
+kv_lora_rank: 64
+qk_nope_head_dim: 16
+qk_rope_head_dim: 16
+qk_head_dim: 32
+v_head_dim: 32
+rope_interleave: true
+rope_theta: 10,000
+
+routed_experts_per_moe_layer: 4
+shared_experts_per_moe_layer: 1
+experts_selected_per_token: 1
+moe_intermediate_size: 128
+n_group: 1
+topk_group: 1
+norm_topk_prob: true
+routed_scaling_factor: 2.5
+
+tie_word_embeddings: true
+rms_norm_eps: 1.0e-6
+max_position_embeddings: 2,048
+```
+
+### Multi-Head Latent Attention
+
+Every layer uses DeepSeek MLA rather than conventional GQA. Queries are
+compressed through a rank-64 Q projection, while keys and values share a
+rank-64 latent representation before expansion to eight attention heads.
+Each query/key head combines a 16-dimensional non-positional component and a
+16-dimensional rotary component. Values use 32 dimensions per head.
+
+`num_key_value_heads` equals `num_attention_heads` because MLA compresses the
+shared KV latent state before expanding it to the attention heads. Setting a
+smaller KV-head count would describe GQA, not this MLA implementation.
+
+### DeepSeekMoE routing
+
+The first decoder layer is dense. Each of the remaining four layers contains
+four routed experts and one always-active shared expert. One routed expert is
+selected for each token.
+
+The V3 router computes sigmoid affinity scores. Selection uses the affinity
+score plus a non-gradient correction bias, while the routed expert weight uses
+the original affinity score. During training, the correction bias was adjusted
+after every batch: overloaded experts were decreased and underloaded experts
+were increased. A very small complementary sequence-wise balance objective was
+also used.
+
+```yaml
+v3_bias_update_speed: 0.01
+sequence_aux_loss_alpha: 0.0001
+no_token_dropping: true
+```
+
+All routed experts received traffic. Aggregate fractions across the complete
+training run ranged from approximately 0.140 to 0.496 depending on layer and
+expert. These aggregate values reflect learned routing specialization and are
+not expected to be exactly uniform.
+
+## MTP scope
+
+The checkpoint contains the standard Transformers DeepSeek-V3 causal language
+model used for inference. It does **not** contain a training-only Multi-Token
+Prediction (MTP) draft module:
+
+```yaml
+num_nextn_predict_layers: 0
+training_objective: next-token prediction
+```
+
+Transformers 5.14.1 exposes the V3 MTP count in configuration but does not
+instantiate or train the original DeepSeek-V3 MTP module in
+`DeepseekV3ForCausalLM`. The checkpoint therefore preserves the V3 main-model
+inference graph and V3 routing behavior, but does not claim to reproduce the
+complete original V3 pretraining recipe.
+
+## Training data
+
+The model was trained on the full TinyStories training corpus using an
+independent 1% validation split:
+
+```yaml
+selected_stories: 2,119,489
+training_stories: 2,098,294
+validation_stories: 21,195
+validation_fraction: 0.01
+
+training_blocks: 2,441,053
+validation_blocks: 24,607
+block_size: 256
+```
+
+Stories were joined into a continuous packed stream:
+
+```text
+BOS + story 1 + EOS + BOS + story 2 + EOS + ...
+```
+
+The stream was split into fixed 256-token blocks without per-story padding.
+Only the final incomplete block was discarded.
+
+## Tokenizer
+
+The checkpoint uses a custom byte-level BPE tokenizer. The base vocabulary was
+trained on the 5% TinyStories experiment and frozen for the full run:
+
+```text
+BPE()
+ByteLevel(add_prefix_space=False)
+base_vocab_size: 1,000
+min_frequency: 2
+normalizer: None
+```
+
+Special tokens use fixed IDs:
+
+```text
+<s>          -> 1000
+</s>         -> 1001
+<|im_start|> -> 1002
+```
+
+`<s>` is both BOS and padding, while `</s>` is EOS. The model reserves 1,024
+embedding rows and the tokenizer exposes 1,003 tokens.
+
+The presence of `<|im_start|>` does not make this a chat model. The checkpoint
+is pretrained only and has no chat template or instruction tuning.
+
+## Training setup
+
+The checkpoint was trained from scratch in float32 on an NVIDIA GeForce RTX
+5060 Ti:
+
+```yaml
+dtype: float32
+batch_size: 16
+block_size: 256
+training_steps: 152,565
+epochs: 1.0
+tokens_processed: 624,906,240
+
+optimizer: AdamW
+learning_rate: 2.0e-4
+warmup_steps: 2,000
+scheduler: warmup + cosine decay
+minimum_learning_rate: 2.0e-5
+weight_decay: 0.0
+grad_clip: 1.0
+
+optimizer_loop_time: approximately 1h 03m 36s
+```
+
+## Evaluation
+
+The final checkpoint produced:
+
+```yaml
+final_train_loss: 1.5201
+validation_loss: 1.5254
+validation_perplexity: 4.5972
+```
+
+Validation loss was computed over 32 batches, or 131,072 tokens, from the
+independent packed validation split. These values are compact checkpoint
+diagnostics, not general language-model benchmark results.
+
+## Example generation
+
+A representative final sampled generation begins:
+
+```text
+Once upon a time, there was a little girl named Lily. She had long black,
+white hair that she loved to play with every day. One day, Lily's mom asked
+her to clean up her bedroom. Lily didn't want to clean up, so she started to
+pick up her toys and put them away in the closet.
+```
+
+An independent greedy reload test produced:
+
+```text
+Once upon a time, there was a little girl named Lily. She loved to play
+outside in the sunshine and pick flowers. One day, she saw a big, scary dog
+running towards her. The dog ran away and Lily was sad.
+```
+
+Sampling is stochastic. The model produces recognizable TinyStories-style
+English, but contradictions, incorrect pronouns, invented words, repetition,
+and unfinished stories remain possible at this size.
+
+## Usage
+
+Install the requirements:
+
+```bash
+pip install -r requirements.txt
+```
+
+Run the included example from the repository root:
+
+```bash
+python example_generate.py
+```
+
+Local loading:
+
+```python
+from pathlib import Path
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_dir = Path("hf")
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+tokenizer = AutoTokenizer.from_pretrained(model_dir)
+model = AutoModelForCausalLM.from_pretrained(
+    model_dir,
+    dtype=torch.float32,
+).to(device).eval()
+
+input_ids = torch.tensor(
+    [[tokenizer.bos_token_id] + tokenizer.encode(
+        "Once upon",
+        add_special_tokens=False,
+    )],
+    dtype=torch.long,
+    device=device,
+)
+
+with torch.inference_mode():
+    output = model.generate(
+        input_ids=input_ids,
+        max_new_tokens=100,
+        do_sample=True,
+        temperature=0.8,
+        top_p=0.95,
+        top_k=40,
+        repetition_penalty=1.1,
+        use_cache=True,
+        pad_token_id=tokenizer.pad_token_id,
+        eos_token_id=tokenizer.eos_token_id,
+    )
+
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+Loading the `hf` subdirectory from Hugging Face Hub:
+
+```python
+from pathlib import Path
+from huggingface_hub import snapshot_download
+
+repo_dir = Path(snapshot_download(
+    repo_id="shibatch/tinydeepseekv3-3m",
+    allow_patterns=["hf/*"],
+))
+model_dir = repo_dir / "hf"
+```
+
+Use `model_dir` with the local loading example above.
+
+## Loading requirements
+
+The model was trained, saved, and independently reloaded with:
+
+```text
+transformers 5.14.1
+torch 2.12.0+cu130
+```
+
+It requires a Transformers release containing `DeepseekV3ForCausalLM`. No
+custom model code or `trust_remote_code=True` is required.
+
+## Intended uses
+
+This model is intended for:
+
+- testing `DeepseekV3Config` and `DeepseekV3ForCausalLM`
+- testing Multi-Head Latent Attention and compressed KV caching
+- testing Q and KV low-rank projections
+- testing V3 sigmoid MoE routing and shared experts
+- testing correction-bias-based expert load balancing
+- compact inference-engine and architecture experiments
+- testing `generate()`, `save_pretrained()`, and `from_pretrained()`
+
+It is not intended for:
+
+- instruction following or chat
+- factual question answering
+- reasoning benchmarks
+- production deployment
+- safety-critical use
+- comparison with full-size DeepSeek models
+
+## Limitations
+
+- only 2.80 million total parameters
+- small 1,003-token tokenizer
+- English TinyStories-only pretraining
+- no instruction tuning and no chat template
+- no MTP training module or MTP objective
+- weak factual knowledge and reasoning
+- occasional grammatical and semantic errors
+- possible mojibake inherited from training-text byte sequences
+- no capability-equivalence claim with official DeepSeek models
+
+## Citation and references
+
+This is a synthetic tiny DeepSeek-V3-compatible MoE checkpoint trained from
+scratch on TinyStories. It is intended for implementation validation,
+debugging, education, and small-scale architecture experiments.
+
+- [DeepSeek-V3 Technical Report](https://arxiv.org/abs/2412.19437)
+- [Official DeepSeek-V3 repository](https://github.com/deepseek-ai/DeepSeek-V3)
+- [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories)

SHA256SUMS

@@ -0,0 +1,11 @@
+96b4962789db24ec636bd8ec0234c1545060f1ca0f4ef79ea30618f56c79ac1c  ./README.md
+d9d94c5075d8d135c84cffcdc78e204ccaf6a364f531561676970a1ed8402c2e  ./artifact_metadata.json
+581243249e2253ec2a2b349dc9bca47c969c5193d89767564bda66e6dcd9a3c3  ./deepseek_v3_config_dump.json
+1aa84f493e2a395195415f467b4cd6c1e4c498a054dba7b5e8f1ca78e2db1df9  ./eval_text_generation.json
+ec2613897f929442890f42e7dfb8e8ef714c734fa038b99dac67c5783ea9e61c  ./example_generate.py
+7c725cd1967e7f2eff66c6d102f20597a73df4f3f5ba8ae02a181407177af2b4  ./hf/config.json
+15d94fce32133b15cbcd52ea4652e64d54a04d14426fb8a43e8d462cd6192ddb  ./hf/generation_config.json
+16ebb94ff5ce6da7fa580ae50fa0a88b4d484a379878ad0ef9475f45d243f821  ./hf/model.safetensors
+188748bfcd79d0cfd58e989b50781b05fe405f23441234fd0f18c60259610a82  ./hf/tokenizer.json
+e96272be67d305eabd9825913c9bdf3ca567d7f01617b757bdb5674590ac41ec  ./hf/tokenizer_config.json
+aa2afd45b989e6bc4d0a4ecf977c93b059673101dad8a1ec40cc648989f592d9  ./requirements.txt

artifact_metadata.json

@@ -0,0 +1,185 @@
+{
+  "model_type": "deepseek_v3_moe",
+  "hf_model_type": "deepseek_v3",
+  "parameter_count": 2803272,
+  "tokenizer_size": 1003,
+  "final_eval_loss": 1.5254412926733494,
+  "final_eval_ppl": 4.59717181998998,
+  "final_train_lm_loss": 1.5200555527210236,
+  "router_usage": {
+    "model.layers.1.mlp.gate": [
+      0.1699,
+      0.1683,
+      0.1658,
+      0.496
+    ],
+    "model.layers.2.mlp.gate": [
+      0.2745,
+      0.2254,
+      0.2464,
+      0.2538
+    ],
+    "model.layers.3.mlp.gate": [
+      0.2655,
+      0.2749,
+      0.1966,
+      0.263
+    ],
+    "model.layers.4.mlp.gate": [
+      0.1976,
+      0.14,
+      0.3308,
+      0.3316
+    ]
+  },
+  "args": {
+    "train_files": [
+      "data/train-00000-of-00004-2d5a1467fff1081b.parquet",
+      "data/train-00001-of-00004-5852b56a2bd28fd9.parquet",
+      "data/train-00002-of-00004-a26307300439e943.parquet",
+      "data/train-00003-of-00004-d243063613e5a057.parquet"
+    ],
+    "output_dir": "runs/deepseek_v3_3m_fp32_100pct",
+    "max_rows": 0,
+    "data_fraction": 1.0,
+    "vocab_size": 1024,
+    "base_vocab_size": 1000,
+    "min_frequency": 2,
+    "tokenizer_from": "runs/deepseek_v3_3m_fp32_5pct/hf",
+    "legacy_tokenizer": true,
+    "legacy_special_token_ids": true,
+    "deepseek_arch": "v3",
+    "hidden_size": 216,
+    "intermediate_size": 432,
+    "num_hidden_layers": 5,
+    "num_attention_heads": 8,
+    "num_key_value_heads": 8,
+    "qk_nope_head_dim": 16,
+    "qk_rope_head_dim": 16,
+    "v_head_dim": 32,
+    "q_lora_rank": 0,
+    "kv_lora_rank": 64,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "rms_norm_eps": 1e-06,
+    "initializer_range": 0.02,
+    "rope_theta": 10000,
+    "first_k_dense_replace": 1,
+    "n_routed_experts": 4,
+    "n_shared_experts": 1,
+    "num_experts_per_tok": 1,
+    "n_group": 0,
+    "topk_group": 0,
+    "topk_method": "greedy",
+    "norm_topk_prob": null,
+    "routed_scaling_factor": null,
+    "moe_intermediate_size": 128,
+    "router_aux_loss_alpha": null,
+    "v3_bias_update_speed": 0.01,
+    "official_router_init": true,
+    "max_position_embeddings": 2048,
+    "tie_word_embeddings": true,
+    "use_cache": false,
+    "mlp_bias": false,
+    "block_size": 256,
+    "batch_size": 16,
+    "max_steps": 152565,
+    "num_epochs": 1.0,
+    "learning_rate": 0.0002,
+    "weight_decay": 0.0,
+    "grad_clip": 1.0,
+    "warmup_steps": 2000,
+    "min_lr_ratio": 0.1,
+    "validation_fraction": 0.01,
+    "log_steps": 100,
+    "eval_steps": 5000,
+    "eval_batches": 32,
+    "save_steps": 10000,
+    "resume_from_checkpoint": "",
+    "seed": 1234,
+    "dtype": "float32",
+    "device": "cuda",
+    "num_workers": 0,
+    "generate_prompts": [
+      "Once upon",
+      "There was a little",
+      "One day"
+    ],
+    "generate_new_tokens": 120,
+    "generation_temperature": 0.8,
+    "generation_top_p": 0.95,
+    "generation_top_k": 40,
+    "generation_repetition_penalty": 1.1,
+    "do_sample": true
+  },
+  "config": {
+    "transformers_version": "5.14.1",
+    "architectures": [
+      "DeepseekV3ForCausalLM"
+    ],
+    "output_hidden_states": false,
+    "return_dict": true,
+    "dtype": "float32",
+    "chunk_size_feed_forward": 0,
+    "is_encoder_decoder": false,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "problem_type": null,
+    "vocab_size": 1024,
+    "hidden_size": 216,
+    "intermediate_size": 432,
+    "moe_intermediate_size": 128,
+    "num_hidden_layers": 5,
+    "num_attention_heads": 8,
+    "num_key_value_heads": 8,
+    "n_shared_experts": 1,
+    "n_routed_experts": 4,
+    "routed_scaling_factor": 2.5,
+    "kv_lora_rank": 64,
+    "q_lora_rank": 64,
+    "qk_rope_head_dim": 16,
+    "v_head_dim": 32,
+    "qk_nope_head_dim": 16,
+    "n_group": 1,
+    "topk_group": 1,
+    "num_experts_per_tok": 1,
+    "first_k_dense_replace": 1,
+    "norm_topk_prob": true,
+    "hidden_act": "silu",
+    "max_position_embeddings": 2048,
+    "initializer_range": 0.02,
+    "rms_norm_eps": 1e-06,
+    "use_cache": false,
+    "pad_token_id": 1000,
+    "bos_token_id": 1000,
+    "eos_token_id": 1001,
+    "pretraining_tp": 1,
+    "tie_word_embeddings": true,
+    "rope_parameters": {
+      "rope_type": "default",
+      "rope_theta": 10000
+    },
+    "rope_interleave": true,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "num_nextn_predict_layers": 0,
+    "qk_head_dim": 32,
+    "head_dim": 16,
+    "_name_or_path": "",
+    "aux_loss_alpha": 0.0001,
+    "seq_aux": true,
+    "model_type": "deepseek_v3",
+    "output_attentions": false
+  },
+  "generations": {
+    "Once upon": "Once upon a time, there was a little girl named Lily. She had long black, white hair that she loved to play with every day. One day, Lily's mom asked her to clean up her bedroom. Lily didn't want to clean up, so she started to pick up her toys and put them away in the closet. \n\nWhile playing outside, Lily's mom came into the room and said, \"Lily, you need to clean your room before dinner.\" Lily smiled and said, \"Okay mommy, I will help you",
+    "There was a little": "There was a little girl who had an idea. She said, “Let’s go on an adventure!†\n\nSo, they all agreed to go. They went off on their journey, and soon they reached the big park. It was so exciting! There were lots of colorful trees, flowers and butterflies. Everywhere it went, people looked at the sky and smiled. They thought it was very interesting. \n\nWhen they got to the top of the",
+    "One day": "One day, he was walking in the park with his mom and he saw a small, wise old tree. He looked at the tree and asked why, \"What are you doing?\"\n\nThe tree replied, \"I'm looking for a place to study.\" His mom said, \"A place is very unique. I can help you find a place to stand and make a cozy place.\" \n\nSo, the little boy took out a big box of sparkly woods and put them"
+  }
+}

deepseek_v3_config_dump.json

@@ -0,0 +1,63 @@
+{
+  "transformers_version": "5.14.1",
+  "architectures": null,
+  "output_hidden_states": false,
+  "return_dict": true,
+  "dtype": null,
+  "chunk_size_feed_forward": 0,
+  "is_encoder_decoder": false,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1"
+  },
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1
+  },
+  "problem_type": null,
+  "vocab_size": 1024,
+  "hidden_size": 216,
+  "intermediate_size": 432,
+  "moe_intermediate_size": 128,
+  "num_hidden_layers": 5,
+  "num_attention_heads": 8,
+  "num_key_value_heads": 8,
+  "n_shared_experts": 1,
+  "n_routed_experts": 4,
+  "routed_scaling_factor": 2.5,
+  "kv_lora_rank": 64,
+  "q_lora_rank": 64,
+  "qk_rope_head_dim": 16,
+  "v_head_dim": 32,
+  "qk_nope_head_dim": 16,
+  "n_group": 1,
+  "topk_group": 1,
+  "num_experts_per_tok": 1,
+  "first_k_dense_replace": 1,
+  "norm_topk_prob": true,
+  "hidden_act": "silu",
+  "max_position_embeddings": 2048,
+  "initializer_range": 0.02,
+  "rms_norm_eps": 1e-06,
+  "use_cache": false,
+  "pad_token_id": 1000,
+  "bos_token_id": 1000,
+  "eos_token_id": 1001,
+  "pretraining_tp": 1,
+  "tie_word_embeddings": true,
+  "rope_parameters": {
+    "rope_type": "default",
+    "rope_theta": 10000
+  },
+  "rope_interleave": true,
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "num_nextn_predict_layers": 0,
+  "qk_head_dim": 32,
+  "head_dim": 16,
+  "_name_or_path": "",
+  "aux_loss_alpha": 0.0001,
+  "seq_aux": true,
+  "model_type": "deepseek_v3",
+  "output_attentions": false
+}

eval_text_generation.json

@@ -0,0 +1,5 @@
+{
+  "Once upon": "Once upon a time, there was a little girl named Lily. She had long black, white hair that she loved to play with every day. One day, Lily's mom asked her to clean up her bedroom. Lily didn't want to clean up, so she started to pick up her toys and put them away in the closet. \n\nWhile playing outside, Lily's mom came into the room and said, \"Lily, you need to clean your room before dinner.\" Lily smiled and said, \"Okay mommy, I will help you",
+  "There was a little": "There was a little girl who had an idea. She said, “Let’s go on an adventure!†\n\nSo, they all agreed to go. They went off on their journey, and soon they reached the big park. It was so exciting! There were lots of colorful trees, flowers and butterflies. Everywhere it went, people looked at the sky and smiled. They thought it was very interesting. \n\nWhen they got to the top of the",
+  "One day": "One day, he was walking in the park with his mom and he saw a small, wise old tree. He looked at the tree and asked why, \"What are you doing?\"\n\nThe tree replied, \"I'm looking for a place to study.\" His mom said, \"A place is very unique. I can help you find a place to stand and make a cozy place.\" \n\nSo, the little boy took out a big box of sparkly woods and put them"
+}

example_generate.py

@@ -0,0 +1,55 @@
+from pathlib import Path
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+
+MODEL_PATH = "."
+MODEL_SUBFOLDER = "hf"
+PROMPT = "Once upon"
+SEED = 0
+
+
+def main() -> None:
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+    model_dir = Path(MODEL_PATH) / MODEL_SUBFOLDER
+
+    tokenizer = AutoTokenizer.from_pretrained(model_dir)
+    model = AutoModelForCausalLM.from_pretrained(
+        model_dir,
+        dtype=torch.float32,
+    ).to(device)
+    model.eval()
+
+    input_ids = torch.tensor(
+        [
+            [tokenizer.bos_token_id]
+            + tokenizer.encode(PROMPT, add_special_tokens=False)
+        ],
+        dtype=torch.long,
+        device=device,
+    )
+
+    torch.manual_seed(SEED)
+    if device.type == "cuda":
+        torch.cuda.manual_seed_all(SEED)
+
+    with torch.inference_mode():
+        output = model.generate(
+            input_ids=input_ids,
+            max_new_tokens=100,
+            do_sample=True,
+            temperature=0.8,
+            top_p=0.95,
+            top_k=40,
+            repetition_penalty=1.1,
+            use_cache=True,
+            pad_token_id=tokenizer.pad_token_id,
+            eos_token_id=tokenizer.eos_token_id,
+        )
+
+    print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))
+
+
+if __name__ == "__main__":
+    main()

hf/config.json

@@ -0,0 +1,50 @@
+{
+  "architectures": [
+    "DeepseekV3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "aux_loss_alpha": 0.0001,
+  "bos_token_id": 1000,
+  "dtype": "float32",
+  "eos_token_id": 1001,
+  "first_k_dense_replace": 1,
+  "head_dim": 16,
+  "hidden_act": "silu",
+  "hidden_size": 216,
+  "initializer_range": 0.02,
+  "intermediate_size": 432,
+  "kv_lora_rank": 64,
+  "max_position_embeddings": 2048,
+  "model_type": "deepseek_v3",
+  "moe_intermediate_size": 128,
+  "n_group": 1,
+  "n_routed_experts": 4,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 8,
+  "num_experts_per_tok": 1,
+  "num_hidden_layers": 5,
+  "num_key_value_heads": 8,
+  "num_nextn_predict_layers": 0,
+  "pad_token_id": 1000,
+  "pretraining_tp": 1,
+  "q_lora_rank": 64,
+  "qk_head_dim": 32,
+  "qk_nope_head_dim": 16,
+  "qk_rope_head_dim": 16,
+  "rms_norm_eps": 1e-06,
+  "rope_interleave": true,
+  "rope_parameters": {
+    "rope_theta": 10000,
+    "rope_type": "default"
+  },
+  "routed_scaling_factor": 2.5,
+  "seq_aux": true,
+  "tie_word_embeddings": true,
+  "topk_group": 1,
+  "transformers_version": "5.14.1",
+  "use_cache": false,
+  "v_head_dim": 32,
+  "vocab_size": 1024
+}

hf/generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1000,
+  "eos_token_id": 1001,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 1000,
+  "transformers_version": "5.14.1",
+  "use_cache": false
+}

hf/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:16ebb94ff5ce6da7fa580ae50fa0a88b4d484a379878ad0ef9475f45d243f821
+size 11226424

hf/tokenizer.json

@@ -0,0 +1,4063 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 1000,
+      "content": "<s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1001,
+      "content": "</s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1002,
+      "content": "<|im_start|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "ByteLevel",
+    "add_prefix_space": false,
+    "trim_offsets": false,
+    "use_regex": true
+  },
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {}
+  },
+  "decoder": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": null,
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": false,
+    "byte_fallback": false,
+    "ignore_merges": false,
+    "vocab": {
+      "!": 0,
+      "\"": 1,
+      "#": 2,
+      "$": 3,
+      "%": 4,
+      "&": 5,
+      "'": 6,
+      "(": 7,
+      ")": 8,
+      "*": 9,
+      "+": 10,
+      ",": 11,
+      "-": 12,
+      ".": 13,
+      "/": 14,
+      "0": 15,
+      "1": 16,
+      "2": 17,
+      "3": 18,
+      "4": 19,
+      "5": 20,
+      "6": 21,
+      "7": 22,
+      "8": 23,
+      "9": 24,
+      ":": 25,
+      ";": 26,
+      "<": 27,
+      "=": 28,
+      ">": 29,
+      "?": 30,
+      "@": 31,
+      "A": 32,
+      "B": 33,
+      "C": 34,
+      "D": 35,
+      "E": 36,
+      "F": 37,
+      "G": 38,
+      "H": 39,
+      "I": 40,
+      "J": 41,
+      "K": 42,
+      "L": 43,
+      "M": 44,
+      "N": 45,
+      "O": 46,
+      "P": 47,
+      "Q": 48,
+      "R": 49,
+      "S": 50,
+      "T": 51,
+      "U": 52,
+      "V": 53,
+      "W": 54,
+      "X": 55,
+      "Y": 56,
+      "Z": 57,
+      "[": 58,
+      "\\": 59,
+      "]": 60,
+      "^": 61,
+      "_": 62,
+      "`": 63,
+      "a": 64,
+      "b": 65,
+      "c": 66,
+      "d": 67,
+      "e": 68,
+      "f": 69,
+      "g": 70,
+      "h": 71,
+      "i": 72,
+      "j": 73,
+      "k": 74,
+      "l": 75,
+      "m": 76,
+      "n": 77,
+      "o": 78,
+      "p": 79,
+      "q": 80,
+      "r": 81,
+      "s": 82,
+      "t": 83,
+      "u": 84,
+      "v": 85,
+      "w": 86,
+      "x": 87,
+      "y": 88,
+      "z": 89,
+      "{": 90,
+      "|": 91,
+      "}": 92,
+      "~": 93,
+      "¡": 94,
+      "¢": 95,
+      "£": 96,
+      "¤": 97,
+      "¥": 98,
+      "¦": 99,
+      "§": 100,
+      "¨": 101,
+      "©": 102,
+      "ª": 103,
+      "«": 104,
+      "¬": 105,
+      "®": 106,
+      "¯": 107,
+      "°": 108,
+      "±": 109,
+      "²": 110,
+      "³": 111,
+      "´": 112,
+      "µ": 113,
+      "¶": 114,
+      "·": 115,
+      "¸": 116,
+      "¹": 117,
+      "º": 118,
+      "»": 119,
+      "¼": 120,
+      "½": 121,
+      "¾": 122,
+      "¿": 123,
+      "À": 124,
+      "Á": 125,
+      "Â": 126,
+      "Ã": 127,
+      "Ä": 128,
+      "Å": 129,
+      "Æ": 130,
+      "Ç": 131,
+      "È": 132,
+      "É": 133,
+      "Ê": 134,
+      "Ë": 135,
+      "Ì": 136,
+      "Í": 137,
+      "Î": 138,
+      "Ï": 139,
+      "Ð": 140,
+      "Ñ": 141,
+      "Ò": 142,
+      "Ó": 143,
+      "Ô": 144,
+      "Õ": 145,
+      "Ö": 146,
+      "×": 147,
+      "Ø": 148,
+      "Ù": 149,
+      "Ú": 150,
+      "Û": 151,
+      "Ü": 152,
+      "Ý": 153,
+      "Þ": 154,
+      "ß": 155,
+      "à": 156,
+      "á": 157,
+      "â": 158,
+      "ã": 159,
+      "ä": 160,
+      "å": 161,
+      "æ": 162,
+      "ç": 163,
+      "è": 164,
+      "é": 165,
+      "ê": 166,
+      "ë": 167,
+      "ì": 168,
+      "í": 169,
+      "î": 170,
+      "ï": 171,
+      "ð": 172,
+      "ñ": 173,
+      "ò": 174,
+      "ó": 175,
+      "ô": 176,
+      "õ": 177,
+      "ö": 178,
+      "÷": 179,
+      "ø": 180,
+      "ù": 181,
+      "ú": 182,
+      "û": 183,
+      "ü": 184,
+      "ý": 185,
+      "þ": 186,
+      "ÿ": 187,
+      "Ā": 188,
+      "ā": 189,
+      "Ă": 190,
+      "ă": 191,
+      "Ą": 192,
+      "ą": 193,
+      "Ć": 194,
+      "ć": 195,
+      "Ĉ": 196,
+      "ĉ": 197,
+      "Ċ": 198,
+      "ċ": 199,
+      "Č": 200,
+      "č": 201,
+      "Ď": 202,
+      "ď": 203,
+      "Đ": 204,
+      "đ": 205,
+      "Ē": 206,
+      "ē": 207,
+      "Ĕ": 208,
+      "ĕ": 209,
+      "Ė": 210,
+      "ė": 211,
+      "Ę": 212,
+      "ę": 213,
+      "Ě": 214,
+      "ě": 215,
+      "Ĝ": 216,
+      "ĝ": 217,
+      "Ğ": 218,
+      "ğ": 219,
+      "Ġ": 220,
+      "ġ": 221,
+      "Ģ": 222,
+      "ģ": 223,
+      "Ĥ": 224,
+      "ĥ": 225,
+      "Ħ": 226,
+      "ħ": 227,
+      "Ĩ": 228,
+      "ĩ": 229,
+      "Ī": 230,
+      "ī": 231,
+      "Ĭ": 232,
+      "ĭ": 233,
+      "Į": 234,
+      "į": 235,
+      "İ": 236,
+      "ı": 237,
+      "IJ": 238,
+      "ij": 239,
+      "Ĵ": 240,
+      "ĵ": 241,
+      "Ķ": 242,
+      "ķ": 243,
+      "ĸ": 244,
+      "Ĺ": 245,
+      "ĺ": 246,
+      "Ļ": 247,
+      "ļ": 248,
+      "Ľ": 249,
+      "ľ": 250,
+      "Ŀ": 251,
+      "ŀ": 252,
+      "Ł": 253,
+      "ł": 254,
+      "Ń": 255,
+      "he": 256,
+      "Ġt": 257,
+      "Ġa": 258,
+      "Ġs": 259,
+      "Ġw": 260,
+      "nd": 261,
+      "Ġthe": 262,
+      "ed": 263,
+      "Ġand": 264,
+      "Ġto": 265,
+      "Ġb": 266,
+      "in": 267,
+      "Ġh": 268,
+      "Ġwa": 269,
+      "re": 270,
+      "ou": 271,
+      "Ġf": 272,
+      "it": 273,
+      "Ġl": 274,
+      "Ġc": 275,
+      "Ġd": 276,
+      "er": 277,
+      "Ġhe": 278,
+      "Ġm": 279,
+      "Ġp": 280,
+      "Ġwas": 281,
+      "om": 282,
+      "ĠT": 283,
+      "Ġo": 284,
+      "ay": 285,
+      "ar": 286,
+      "is": 287,
+      "ing": 288,
+      "il": 289,
+      "Ġg": 290,
+      "id": 291,
+      "at": 292,
+      "en": 293,
+      "Ġn": 294,
+      "Ġsa": 295,
+      "Ġha": 296,
+      "ĠThe": 297,
+      "ĠS": 298,
+      "an": 299,
+      "im": 300,
+      "or": 301,
+      "on": 302,
+      "Ġit": 303,
+      "Ġth": 304,
+      "ll": 305,
+      "le": 306,
+      "ĠH": 307,
+      "ot": 308,
+      "et": 309,
+      "Ġher": 310,
+      "ir": 311,
+      "ĠShe": 312,
+      "es": 313,
+      "ĠHe": 314,
+      "Ġin": 315,
+      "ver": 316,
+      "ow": 317,
+      "ut": 318,
+      "ĠThey": 319,
+      "ck": 320,
+      "Ġe": 321,
+      "Ġu": 322,
+      "Ġy": 323,
+      "ld": 324,
+      "oo": 325,
+      "Ġsaid": 326,
+      "Ġ\"": 327,
+      "Ġr": 328,
+      "ily": 329,
+      "ig": 330,
+      "Ġbe": 331,
+      "am": 332,
+      "Ġst": 333,
+      "ĠI": 334,
+      "ce": 335,
+      "ke": 336,
+      "Ġshe": 337,
+      "ve": 338,
+      "pp": 339,
+      "ith": 340,
+      "Ġwith": 341,
+      "Lily": 342,
+      "On": 343,
+      "Ġon": 344,
+      "Ġyou": 345,
+      "ked": 346,
+      "Ġof": 347,
+      "ri": 348,
+      "Ġso": 349,
+      "nt": 350,
+      "Ġhis": 351,
+      "Ġpl": 352,
+      "ad": 353,
+      "very": 354,
+      "Ġday": 355,
+      "Ġthat": 356,
+      "Ġup": 357,
+      "Ġhad": 358,
+      "Ġplay": 359,
+      "st": 360,
+      "Ġthey": 361,
+      "Ġwe": 362,
+      "ĠLily": 363,
+      "Ġmom": 364,
+      "Ġfor": 365,
+      "ĠB": 366,
+      "un": 367,
+      "el": 368,
+      "'s": 369,
+      "my": 370,
+      "ould": 371,
+      "The": 372,
+      "Ġli": 373,
+      "ch": 374,
+      "Ġhapp": 375,
+      "ent": 376,
+      "Ġwant": 377,
+      "Ġsh": 378,
+      "itt": 379,
+      "out": 380,
+      "her": 381,
+      "Ġdo": 382,
+      "Ġnot": 383,
+      "ly": 384,
+      "ome": 385,
+      "se": 386,
+      "Ġvery": 387,
+      "ound": 388,
+      "Ġwh": 389,
+      "ime": 390,
+      "all": 391,
+      "end": 392,
+      "ittle": 393,
+      "Ġtime": 394,
+      "Ġk": 395,
+      "ĠĊ": 396,
+      "ĠM": 397,
+      "Ġlittle": 398,
+      "al": 399,
+      "Ġthere": 400,
+      "ht": 401,
+      "Ġne": 402,
+      "Ġre": 403,
+      "Ġis": 404,
+      "Ġhappy": 405,
+      "Ġsm": 406,
+      "Ġbig": 407,
+      "ĠIt": 408,
+      "Ġbut": 409,
+      "ack": 410,
+      "Ġan": 411,
+      "Ġsaw": 412,
+      "ry": 413,
+      ".\"": 414,
+      "Ġas": 415,
+      "riend": 416,
+      "Ġfriend": 417,
+      "ra": 418,
+      "ide": 419,
+      "One": 420,
+      "ake": 421,
+      "!\"": 422,
+      "ug": 423,
+      "'t": 424,
+      "Ġloo": 425,
+      "ter": 426,
+      "Ġwere": 427,
+      "ved": 428,
+      "Ġlo": 429,
+      "ore": 430,
+      "Once": 431,
+      "ec": 432,
+      "Ġgo": 433,
+      "Ġhim": 434,
+      "Ġtoo": 435,
+      "Ġbo": 436,
+      "Ġse": 437,
+      "Ġare": 438,
+      "ard": 439,
+      "irl": 440,
+      "Ġj": 441,
+      "ill": 442,
+      "Ġwanted": 443,
+      "ĠTim": 444,
+      "Ġupon": 445,
+      "Ġgirl": 446,
+      "Ġthem": 447,
+      "Ġout": 448,
+      "Ġat": 449,
+      "Ġtheir": 450,
+      "ur": 451,
+      "fu": 452,
+      "ind": 453,
+      "ĠA": 454,
+      "Ġsp": 455,
+      "way": 456,
+      "Ġdid": 457,
+      "Ġsmil": 458,
+      "Ġcould": 459,
+      "hen": 460,
+      "Ġhave": 461,
+      "ain": 462,
+      "Ġex": 463,
+      "art": 464,
+      "hed": 465,
+      "Ġar": 466,
+      "Ġwent": 467,
+      "ic": 468,
+      "rom": 469,
+      "Ġcan": 470,
+      "ĠJ": 471,
+      "ĠBen": 472,
+      "lp": 473,
+      "ful": 474,
+      "?\"": 475,
+      "ood": 476,
+      "hing": 477,
+      "Ġnam": 478,
+      "ight": 479,
+      "Ġfriends": 480,
+      "Ġhelp": 481,
+      "Ġkn": 482,
+      "Ġall": 483,
+      "um": 484,
+      "ark": 485,
+      "Ġsay": 486,
+      "one": 487,
+      "Ġback": 488,
+      "You": 489,
+      "Ġfun": 490,
+      "ĠTom": 491,
+      "Ġcl": 492,
+      "Ġno": 493,
+      "are": 494,
+      "Ġle": 495,
+      "op": 496,
+      "Ġal": 497,
+      "Ġnamed": 498,
+      "Ġsmiled": 499,
+      "Ġsc": 500,
+      "oug": 501,
+      "side": 502,
+      "Ġloved": 503,
+      "ick": 504,
+      "Ġtoy": 505,
+      "Ġman": 506,
+      "ĠOne": 507,
+      "Ġasked": 508,
+      "elt": 509,
+      "Ġlike": 510,
+      "Ġfe": 511,
+      "Ġlooked": 512,
+      "Ġsee": 513,
+      "Ġme": 514,
+      "ĠW": 515,
+      "Ġfelt": 516,
+      "Ġsome": 517,
+      "Ġaround": 518,
+      "omet": 519,
+      "Ġro": 520,
+      "ame": 521,
+      "ure": 522,
+      "Ġbir": 523,
+      "Ġbr": 524,
+      "ĠTimmy": 525,
+      "get": 526,
+      "ice": 527,
+      "Ġwould": 528,
+      "Ġboy": 529,
+      "Ġstart": 530,
+      "ong": 531,
+      "as": 532,
+      "â": 533,
+      "âĤ": 534,
+      "âĤ¬": 535,
+      "Ġsomet": 536,
+      "Ġbird": 537,
+      "dd": 538,
+      "Ġfa": 539,
+      "Ġwhat": 540,
+      "Ġag": 541,
+      "Ġwor": 542,
+      "Ġmake": 543,
+      "king": 544,
+      "ings": 545,
+      "ade": 546,
+      ",\"": 547,
+      "ĠBut": 548,
+      "gether": 549,
+      "Ġran": 550,
+      "ĠYou": 551,
+      "own": 552,
+      "Ġaway": 553,
+      "Ġtre": 554,
+      "ag": 555,
+      "ie": 556,
+      "Ġsays": 557,
+      "ared": 558,
+      "if": 559,
+      "Ġstarted": 560,
+      "oud": 561,
+      "Ġsomething": 562,
+      "Ġtogether": 563,
+      "Ġcar": 564,
+      "Ġmade": 565,
+      "ther": 566,
+      "ited": 567,
+      "Ġco": 568,
+      "Ġpark": 569,
+      "Ġother": 570,
+      "Ġsad": 571,
+      "Ġnew": 572,
+      "Ġexc": 573,
+      "Ġput": 574,
+      "ell": 575,
+      "Ġfrom": 576,
+      "Ġpr": 577,
+      "ach": 578,
+      "ĠL": 579,
+      "ble": 580,
+      "Ġmu": 581,
+      "ook": 582,
+      "Ġhug": 583,
+      "Ġagain": 584,
+      "Ġhome": 585,
+      "Ġla": 586,
+      "ss": 587,
+      "Ġwhen": 588,
+      "hat": 589,
+      "Ġgood": 590,
+      "est": 591,
+      "Ġwho": 592,
+      "Ġfl": 593,
+      "Ġfound": 594,
+      "ep": 595,
+      "ried": 596,
+      "Ġdec": 597,
+      "Ġch": 598,
+      "pped": 599,
+      "Ġthen": 600,
+      "qu": 601,
+      "Ġwal": 602,
+      "Ġget": 603,
+      "Ġplaying": 604,
+      "ought": 605,
+      "pl": 606,
+      "nn": 607,
+      "ĠE": 608,
+      "ally": 609,
+      "Ġsw": 610,
+      "ous": 611,
+      "Ġliked": 612,
+      "Ġgot": 613,
+      "Ġthings": 614,
+      "Ġexcited": 615,
+      "uck": 616,
+      "ro": 617,
+      "ided": 618,
+      "Ġbl": 619,
+      "ĠMom": 620,
+      "Ġdecided": 621,
+      "ĠD": 622,
+      "ara": 623,
+      "Ġyour": 624,
+      "ax": 625,
+      "Ġevery": 626,
+      "ave": 627,
+      "Ġcame": 628,
+      "Ġdown": 629,
+      "Ġscared": 630,
+      "ust": 631,
+      "Ġone": 632,
+      "Ġcare": 633,
+      "Ġdog": 634,
+      "Ġv": 635,
+      "ny": 636,
+      "Ġfind": 637,
+      "ouse": 638,
+      "nna": 639,
+      "Ġdad": 640,
+      "Ġbec": 641,
+      "udd": 642,
+      "ist": 643,
+      "Ġthan": 644,
+      "Ġfeel": 645,
+      "ue": 646,
+      "ap": 647,
+      "They": 648,
+      "Ġab": 649,
+      "Ġwill": 650,
+      "Ġlook": 651,
+      "Ġmy": 652,
+      "nder": 653,
+      "Ġgr": 654,
+      "our": 655,
+      "Ġgra": 656,
+      "arn": 657,
+      "Ġke": 658,
+      "ways": 659,
+      "us": 660,
+      "ess": 661,
+      "fe": 662,
+      "bb": 663,
+      "Ġdidn": 664,
+      "Ġabout": 665,
+      "Ġalways": 666,
+      "Ġtook": 667,
+      "Ġpo": 668,
+      "ers": 669,
+      "Ġmommy": 670,
+      "ise": 671,
+      "Ġtoys": 672,
+      "Ġlot": 673,
+      "ant": 674,
+      "Ġkne": 675,
+      "ite": 676,
+      "Ġtree": 677,
+      "Ġthought": 678,
+      "Ġmore": 679,
+      "Ġoutside": 680,
+      "Ġball": 681,
+      "eci": 682,
+      "ĠSam": 683,
+      "Ġta": 684,
+      "ened": 685,
+      "ched": 686,
+      "Ġho": 687,
+      "ĠF": 688,
+      "Ġold": 689,
+      "Ġint": 690,
+      "Ġlearn": 691,
+      "Ġcat": 692,
+      "Ġknow": 693,
+      "ged": 694,
+      "ret": 695,
+      "Ġsor": 696,
+      "ecial": 697,
+      "Ġspecial": 698,
+      "Ġlaug": 699,
+      "udden": 700,
+      "Ġtake": 701,
+      "ia": 702,
+      "ĠMax": 703,
+      "ive": 704,
+      "Ġknew": 705,
+      "Ġhow": 706,
+      "Ġpe": 707,
+      "uddenly": 708,
+      "and": 709,
+      "Ġtr": 710,
+      "fter": 711,
+      "Ġany": 712,
+      "Ġshow": 713,
+      "ink": 714,
+      "Ġsorry": 715,
+      "Ġtw": 716,
+      "ma": 717,
+      "ish": 718,
+      "Ġmuch": 719,
+      "But": 720,
+      "au": 721,
+      "Ġsl": 722,
+      "ven": 723,
+      "Ġhand": 724,
+      "Ġrun": 725,
+      "Ġsk": 726,
+      "Ġop": 727,
+      "Ġhouse": 728,
+      "Ġmo": 729,
+      "Ġsun": 730,
+      "Ġinside": 731,
+      "Ġor": 732,
+      "Ġinto": 733,
+      "Ġwater": 734,
+      "Ġclo": 735,
+      "ank": 736,
+      "ate": 737,
+      "Tom": 738,
+      "Ġtried": 739,
+      "Ġthis": 740,
+      "Ġqu": 741,
+      "Ġheard": 742,
+      "Ġif": 743,
+      "Ġtold": 744,
+      "Ġover": 745,
+      "ĠWe": 746,
+      "Ġeach": 747,
+      "Ġen": 748,
+      "Ġproud": 749,
+      "Ġroom": 750,
+      "ĠSara": 751,
+      "ump": 752,
+      "dy": 753,
+      "Ġcouldn": 754,
+      "by": 755,
+      "ĠAnna": 756,
+      "ace": 757,
+      "Ġexpl": 758,
+      "Ġnever": 759,
+      "Ġgave": 760,
+      "ĠEvery": 761,
+      "other": 762,
+      "ion": 763,
+      "Ġpick": 764,
+      "ause": 765,
+      "ty": 766,
+      "Ġeat": 767,
+      "ak": 768,
+      "Ġplayed": 769,
+      "ge": 770,
+      "Mom": 771,
+      "Ġwat": 772,
+      "Ġjust": 773,
+      "etter": 774,
+      "Ġoff": 775,
+      "Ġgre": 776,
+      "Ġneed": 777,
+      "Ġhugged": 778,
+      "Ġpret": 779,
+      "Ġcom": 780,
+      "oth": 781,
+      "here": 782,
+      "Ġnice": 783,
+      "Ġstr": 784,
+      "'m": 785,
+      "ile": 786,
+      "Ġbecause": 787,
+      "Ġmany": 788,
+      "sed": 789,
+      "uc": 790,
+      "Ġfo": 791,
+      "Ġbox": 792,
+      "Ġlong": 793,
+      "Ġbear": 794,
+      "ine": 795,
+      "Ġjo": 796,
+      "ving": 797,
+      "gry": 798,
+      "Ġbu": 799,
+      "Ġtry": 800,
+      "Ġsn": 801,
+      "ort": 802,
+      "imal": 803,
+      "iz": 804,
+      "Ġanimal": 805,
+      "Ġsmall": 806,
+      "Ġunt": 807,
+      "ough": 808,
+      "urt": 809,
+      "Ġuntil": 810,
+      "ild": 811,
+      "ĠJack": 812,
+      "sel": 813,
+      "Ġlove": 814,
+      "Ġkind": 815,
+      "He": 816,
+      "em": 817,
+      "ĠJo": 818,
+      "Ġlearned": 819,
+      "ft": 820,
+      "ĠMia": 821,
+      "Ġim": 822,
+      "Ġbest": 823,
+      "Ġcall": 824,
+      "Ġbetter": 825,
+      "Ġflow": 826,
+      "Ġdon": 827,
+      "ars": 828,
+      "urp": 829,
+      "Ġend": 830,
+      "ull": 831,
+      "Ġad": 832,
+      "ves": 833,
+      "aut": 834,
+      "vent": 835,
+      "urn": 836,
+      "Yes": 837,
+      "ber": 838,
+      "Ġdre": 839,
+      "lly": 840,
+      "arden": 841,
+      "Ġgarden": 842,
+      "ĠAnd": 843,
+      "Ġcle": 844,
+      "Åĵ": 845,
+      "Ġfast": 846,
+      "Ġcareful": 847,
+      "ady": 848,
+      "ies": 849,
+      "Ġbeaut": 850,
+      "ĠC": 851,
+      "Ġsky": 852,
+      "Th": 853,
+      "Ġbra": 854,
+      "It": 855,
+      "Ġche": 856,
+      "Ġeven": 857,
+      "Ġthanked": 858,
+      "Ġfi": 859,
+      "ĠSo": 860,
+      "Ġra": 861,
+      "Ġte": 862,
+      "Ġlaughed": 863,
+      "pt": 864,
+      "Ġby": 865,
+      "ase": 866,
+      "Ġloud": 867,
+      "Ġjump": 868,
+      "lew": 869,
+      "Ġgl": 870,
+      "ĠHer": 871,
+      "Ġbeauti": 872,
+      "self": 873,
+      "Ġway": 874,
+      "Ġlist": 875,
+      "Ġcome": 876,
+      "Ġbeautiful": 877,
+      "iny": 878,
+      "wn": 879,
+      "Ġlots": 880,
+      "Ġfam": 881,
+      "Ġunder": 882,
+      "Ġhard": 883,
+      "Ġwo": 884,
+      "Ġhurt": 885,
+      "Ġâ": 886,
+      "Ġanimals": 887,
+      "ect": 888,
+      "Ġstill": 889,
+      "ct": 890,
+      "Ġstay": 891,
+      "Ġboth": 892,
+      "Ġlet": 893,
+      "ĠThen": 894,
+      "hes": 895,
+      "Ġcu": 896,
+      "Ġbook": 897,
+      "ning": 898,
+      "ĠHis": 899,
+      "Ġrem": 900,
+      "Ġsafe": 901,
+      "be": 902,
+      "ool": 903,
+      "Ġtwo": 904,
+      "Ġcol": 905,
+      "Ġlived": 906,
+      "ĠFrom": 907,
+      "Ġbad": 908,
+      "ople": 909,
+      "ree": 910,
+      "ast": 911,
+      "hn": 912,
+      "kay": 913,
+      "Ġimp": 914,
+      "urpr": 915,
+      "ock": 916,
+      "ucy": 917,
+      "Ġits": 918,
+      "Ġbrave": 919,
+      "ob": 920,
+      "Ġflew": 921,
+      "Ġshould": 922,
+      "Ġsurpr": 923,
+      "Ġangry": 924,
+      "Ġfamily": 925,
+      "Ġwalked": 926,
+      "Tim": 927,
+      "ished": 928,
+      "When": 929,
+      "Ġred": 930,
+      "Ġpeople": 931,
+      "ane": 932,
+      "No": 933,
+      "Ġcalled": 934,
+      "Ġstor": 935,
+      "igh": 936,
+      "led": 937,
+      "Ġfin": 938,
+      "Ġsoon": 939,
+      "Let": 940,
+      "dded": 941,
+      "fore": 942,
+      "Ġdan": 943,
+      "Ġshare": 944,
+      "ised": 945,
+      "Ġrock": 946,
+      "Ġdoor": 947,
+      "xt": 948,
+      "âĤ¬â": 949,
+      "ip": 950,
+      "Ġkept": 951,
+      "Ġfore": 952,
+      "Ġpic": 953,
+      "Ġnoise": 954,
+      "ĠP": 955,
+      "Ġfly": 956,
+      "ary": 957,
+      "ĠWhen": 958,
+      "Ġgoing": 959,
+      "age": 960,
+      "Ġclean": 961,
+      "Ġnow": 962,
+      "Ġcon": 963,
+      "Ben": 964,
+      "Ġide": 965,
+      "Ġadvent": 966,
+      "Ġopened": 967,
+      "Ġpretty": 968,
+      "ummy": 969,
+      "so": 970,
+      "Ġdra": 971,
+      "Ġcry": 972,
+      "Ġdoll": 973,
+      "Ġwind": 974,
+      "Ġexplore": 975,
+      "What": 976,
+      "ied": 977,
+      "Ġbefore": 978,
+      "Ġshiny": 979,
+      "Ġkeep": 980,
+      "Ġsto": 981,
+      "illy": 982,
+      "Ġalso": 983,
+      "Ġfar": 984,
+      "Ġreal": 985,
+      "Ġturn": 986,
+      "Ġground": 987,
+      "Ħ¢": 988,
+      "âĤ¬âĦ¢": 989,
+      "Ġfr": 990,
+      "Ġcre": 991,
+      "Ġnodded": 992,
+      "Ġidea": 993,
+      "Ġey": 994,
+      "les": 995,
+      "oon": 996,
+      "Ġcolor": 997,
+      "That": 998,
+      "Ġwar": 999
+    },
+    "merges": [
+      [
+        "h",
+        "e"
+      ],
+      [
+        "Ġ",
+        "t"
+      ],
+      [
+        "Ġ",
+        "a"
+      ],
+      [
+        "Ġ",
+        "s"
+      ],
+      [
+        "Ġ",
+        "w"
+      ],
+      [
+        "n",
+        "d"
+      ],
+      [
+        "Ġt",
+        "he"
+      ],
+      [
+        "e",
+        "d"
+      ],
+      [
+        "Ġa",
+        "nd"
+      ],
+      [
+        "Ġt",
+        "o"
+      ],
+      [
+        "Ġ",
+        "b"
+      ],
+      [
+        "i",
+        "n"
+      ],
+      [
+        "Ġ",
+        "h"
+      ],
+      [
+        "Ġw",
+        "a"
+      ],
+      [
+        "r",
+        "e"
+      ],
+      [
+        "o",
+        "u"
+      ],
+      [
+        "Ġ",
+        "f"
+      ],
+      [
+        "i",
+        "t"
+      ],
+      [
+        "Ġ",
+        "l"
+      ],
+      [
+        "Ġ",
+        "c"
+      ],
+      [
+        "Ġ",
+        "d"
+      ],
+      [
+        "e",
+        "r"
+      ],
+      [
+        "Ġ",
+        "he"
+      ],
+      [
+        "Ġ",
+        "m"
+      ],
+      [
+        "Ġ",
+        "p"
+      ],
+      [
+        "Ġwa",
+        "s"
+      ],
+      [
+        "o",
+        "m"
+      ],
+      [
+        "Ġ",
+        "T"
+      ],
+      [
+        "Ġ",
+        "o"
+      ],
+      [
+        "a",
+        "y"
+      ],
+      [
+        "a",
+        "r"
+      ],
+      [
+        "i",
+        "s"
+      ],
+      [
+        "in",
+        "g"
+      ],
+      [
+        "i",
+        "l"
+      ],
+      [
+        "Ġ",
+        "g"
+      ],
+      [
+        "i",
+        "d"
+      ],
+      [
+        "a",
+        "t"
+      ],
+      [
+        "e",
+        "n"
+      ],
+      [
+        "Ġ",
+        "n"
+      ],
+      [
+        "Ġs",
+        "a"
+      ],
+      [
+        "Ġh",
+        "a"
+      ],
+      [
+        "ĠT",
+        "he"
+      ],
+      [
+        "Ġ",
+        "S"
+      ],
+      [
+        "a",
+        "n"
+      ],
+      [
+        "i",
+        "m"
+      ],
+      [
+        "o",
+        "r"
+      ],
+      [
+        "o",
+        "n"
+      ],
+      [
+        "Ġ",
+        "it"
+      ],
+      [
+        "Ġt",
+        "h"
+      ],
+      [
+        "l",
+        "l"
+      ],
+      [
+        "l",
+        "e"
+      ],
+      [
+        "Ġ",
+        "H"
+      ],
+      [
+        "o",
+        "t"
+      ],
+      [
+        "e",
+        "t"
+      ],
+      [
+        "Ġhe",
+        "r"
+      ],
+      [
+        "i",
+        "r"
+      ],
+      [
+        "ĠS",
+        "he"
+      ],
+      [
+        "e",
+        "s"
+      ],
+      [
+        "ĠH",
+        "e"
+      ],
+      [
+        "Ġ",
+        "in"
+      ],
+      [
+        "v",
+        "er"
+      ],
+      [
+        "o",
+        "w"
+      ],
+      [
+        "u",
+        "t"
+      ],
+      [
+        "ĠThe",
+        "y"
+      ],
+      [
+        "c",
+        "k"
+      ],
+      [
+        "Ġ",
+        "e"
+      ],
+      [
+        "Ġ",
+        "u"
+      ],
+      [
+        "Ġ",
+        "y"
+      ],
+      [
+        "l",
+        "d"
+      ],
+      [
+        "o",
+        "o"
+      ],
+      [
+        "Ġsa",
+        "id"
+      ],
+      [
+        "Ġ",
+        "\""
+      ],
+      [
+        "Ġ",
+        "r"
+      ],
+      [
+        "il",
+        "y"
+      ],
+      [
+        "i",
+        "g"
+      ],
+      [
+        "Ġb",
+        "e"
+      ],
+      [
+        "a",
+        "m"
+      ],
+      [
+        "Ġs",
+        "t"
+      ],
+      [
+        "Ġ",
+        "I"
+      ],
+      [
+        "c",
+        "e"
+      ],
+      [
+        "k",
+        "e"
+      ],
+      [
+        "Ġs",
+        "he"
+      ],
+      [
+        "v",
+        "e"
+      ],
+      [
+        "p",
+        "p"
+      ],
+      [
+        "it",
+        "h"
+      ],
+      [
+        "Ġw",
+        "ith"
+      ],
+      [
+        "L",
+        "ily"
+      ],
+      [
+        "O",
+        "n"
+      ],
+      [
+        "Ġo",
+        "n"
+      ],
+      [
+        "Ġy",
+        "ou"
+      ],
+      [
+        "k",
+        "ed"
+      ],
+      [
+        "Ġo",
+        "f"
+      ],
+      [
+        "r",
+        "i"
+      ],
+      [
+        "Ġs",
+        "o"
+      ],
+      [
+        "n",
+        "t"
+      ],
+      [
+        "Ġh",
+        "is"
+      ],
+      [
+        "Ġp",
+        "l"
+      ],
+      [
+        "a",
+        "d"
+      ],
+      [
+        "ver",
+        "y"
+      ],
+      [
+        "Ġd",
+        "ay"
+      ],
+      [
+        "Ġth",
+        "at"
+      ],
+      [
+        "Ġu",
+        "p"
+      ],
+      [
+        "Ġha",
+        "d"
+      ],
+      [
+        "Ġpl",
+        "ay"
+      ],
+      [
+        "s",
+        "t"
+      ],
+      [
+        "Ġthe",
+        "y"
+      ],
+      [
+        "Ġw",
+        "e"
+      ],
+      [
+        "Ġ",
+        "Lily"
+      ],
+      [
+        "Ġm",
+        "om"
+      ],
+      [
+        "Ġf",
+        "or"
+      ],
+      [
+        "Ġ",
+        "B"
+      ],
+      [
+        "u",
+        "n"
+      ],
+      [
+        "e",
+        "l"
+      ],
+      [
+        "'",
+        "s"
+      ],
+      [
+        "m",
+        "y"
+      ],
+      [
+        "ou",
+        "ld"
+      ],
+      [
+        "T",
+        "he"
+      ],
+      [
+        "Ġl",
+        "i"
+      ],
+      [
+        "c",
+        "h"
+      ],
+      [
+        "Ġha",
+        "pp"
+      ],
+      [
+        "en",
+        "t"
+      ],
+      [
+        "Ġwa",
+        "nt"
+      ],
+      [
+        "Ġs",
+        "h"
+      ],
+      [
+        "it",
+        "t"
+      ],
+      [
+        "ou",
+        "t"
+      ],
+      [
+        "he",
+        "r"
+      ],
+      [
+        "Ġd",
+        "o"
+      ],
+      [
+        "Ġn",
+        "ot"
+      ],
+      [
+        "l",
+        "y"
+      ],
+      [
+        "om",
+        "e"
+      ],
+      [
+        "s",
+        "e"
+      ],
+      [
+        "Ġ",
+        "very"
+      ],
+      [
+        "ou",
+        "nd"
+      ],
+      [
+        "Ġw",
+        "h"
+      ],
+      [
+        "im",
+        "e"
+      ],
+      [
+        "a",
+        "ll"
+      ],
+      [
+        "e",
+        "nd"
+      ],
+      [
+        "itt",
+        "le"
+      ],
+      [
+        "Ġt",
+        "ime"
+      ],
+      [
+        "Ġ",
+        "k"
+      ],
+      [
+        "Ġ",
+        "Ċ"
+      ],
+      [
+        "Ġ",
+        "M"
+      ],
+      [
+        "Ġl",
+        "ittle"
+      ],
+      [
+        "a",
+        "l"
+      ],
+      [
+        "Ġthe",
+        "re"
+      ],
+      [
+        "h",
+        "t"
+      ],
+      [
+        "Ġn",
+        "e"
+      ],
+      [
+        "Ġ",
+        "re"
+      ],
+      [
+        "Ġ",
+        "is"
+      ],
+      [
+        "Ġhapp",
+        "y"
+      ],
+      [
+        "Ġs",
+        "m"
+      ],
+      [
+        "Ġb",
+        "ig"
+      ],
+      [
+        "ĠI",
+        "t"
+      ],
+      [
+        "Ġb",
+        "ut"
+      ],
+      [
+        "a",
+        "ck"
+      ],
+      [
+        "Ġa",
+        "n"
+      ],
+      [
+        "Ġsa",
+        "w"
+      ],
+      [
+        "r",
+        "y"
+      ],
+      [
+        ".",
+        "\""
+      ],
+      [
+        "Ġa",
+        "s"
+      ],
+      [
+        "ri",
+        "end"
+      ],
+      [
+        "Ġf",
+        "riend"
+      ],
+      [
+        "r",
+        "a"
+      ],
+      [
+        "id",
+        "e"
+      ],
+      [
+        "On",
+        "e"
+      ],
+      [
+        "a",
+        "ke"
+      ],
+      [
+        "!",
+        "\""
+      ],
+      [
+        "u",
+        "g"
+      ],
+      [
+        "'",
+        "t"
+      ],
+      [
+        "Ġl",
+        "oo"
+      ],
+      [
+        "t",
+        "er"
+      ],
+      [
+        "Ġwe",
+        "re"
+      ],
+      [
+        "v",
+        "ed"
+      ],
+      [
+        "Ġl",
+        "o"
+      ],
+      [
+        "o",
+        "re"
+      ],
+      [
+        "On",
+        "ce"
+      ],
+      [
+        "e",
+        "c"
+      ],
+      [
+        "Ġg",
+        "o"
+      ],
+      [
+        "Ġh",
+        "im"
+      ],
+      [
+        "Ġto",
+        "o"
+      ],
+      [
+        "Ġb",
+        "o"
+      ],
+      [
+        "Ġs",
+        "e"
+      ],
+      [
+        "Ġa",
+        "re"
+      ],
+      [
+        "ar",
+        "d"
+      ],
+      [
+        "ir",
+        "l"
+      ],
+      [
+        "Ġ",
+        "j"
+      ],
+      [
+        "il",
+        "l"
+      ],
+      [
+        "Ġwant",
+        "ed"
+      ],
+      [
+        "ĠT",
+        "im"
+      ],
+      [
+        "Ġup",
+        "on"
+      ],
+      [
+        "Ġg",
+        "irl"
+      ],
+      [
+        "Ġthe",
+        "m"
+      ],
+      [
+        "Ġ",
+        "out"
+      ],
+      [
+        "Ġa",
+        "t"
+      ],
+      [
+        "Ġthe",
+        "ir"
+      ],
+      [
+        "u",
+        "r"
+      ],
+      [
+        "f",
+        "u"
+      ],
+      [
+        "i",
+        "nd"
+      ],
+      [
+        "Ġ",
+        "A"
+      ],
+      [
+        "Ġs",
+        "p"
+      ],
+      [
+        "w",
+        "ay"
+      ],
+      [
+        "Ġd",
+        "id"
+      ],
+      [
+        "Ġsm",
+        "il"
+      ],
+      [
+        "Ġc",
+        "ould"
+      ],
+      [
+        "he",
+        "n"
+      ],
+      [
+        "Ġha",
+        "ve"
+      ],
+      [
+        "a",
+        "in"
+      ],
+      [
+        "Ġe",
+        "x"
+      ],
+      [
+        "ar",
+        "t"
+      ],
+      [
+        "he",
+        "d"
+      ],
+      [
+        "Ġa",
+        "r"
+      ],
+      [
+        "Ġw",
+        "ent"
+      ],
+      [
+        "i",
+        "c"
+      ],
+      [
+        "r",
+        "om"
+      ],
+      [
+        "Ġc",
+        "an"
+      ],
+      [
+        "Ġ",
+        "J"
+      ],
+      [
+        "ĠB",
+        "en"
+      ],
+      [
+        "l",
+        "p"
+      ],
+      [
+        "fu",
+        "l"
+      ],
+      [
+        "?",
+        "\""
+      ],
+      [
+        "oo",
+        "d"
+      ],
+      [
+        "h",
+        "ing"
+      ],
+      [
+        "Ġn",
+        "am"
+      ],
+      [
+        "ig",
+        "ht"
+      ],
+      [
+        "Ġfriend",
+        "s"
+      ],
+      [
+        "Ġhe",
+        "lp"
+      ],
+      [
+        "Ġk",
+        "n"
+      ],
+      [
+        "Ġa",
+        "ll"
+      ],
+      [
+        "u",
+        "m"
+      ],
+      [
+        "ar",
+        "k"
+      ],
+      [
+        "Ġs",
+        "ay"
+      ],
+      [
+        "on",
+        "e"
+      ],
+      [
+        "Ġb",
+        "ack"
+      ],
+      [
+        "Y",
+        "ou"
+      ],
+      [
+        "Ġf",
+        "un"
+      ],
+      [
+        "ĠT",
+        "om"
+      ],
+      [
+        "Ġc",
+        "l"
+      ],
+      [
+        "Ġn",
+        "o"
+      ],
+      [
+        "a",
+        "re"
+      ],
+      [
+        "Ġl",
+        "e"
+      ],
+      [
+        "o",
+        "p"
+      ],
+      [
+        "Ġa",
+        "l"
+      ],
+      [
+        "Ġnam",
+        "ed"
+      ],
+      [
+        "Ġsmil",
+        "ed"
+      ],
+      [
+        "Ġs",
+        "c"
+      ],
+      [
+        "ou",
+        "g"
+      ],
+      [
+        "s",
+        "ide"
+      ],
+      [
+        "Ġlo",
+        "ved"
+      ],
+      [
+        "i",
+        "ck"
+      ],
+      [
+        "Ġto",
+        "y"
+      ],
+      [
+        "Ġm",
+        "an"
+      ],
+      [
+        "Ġ",
+        "One"
+      ],
+      [
+        "Ġas",
+        "ked"
+      ],
+      [
+        "el",
+        "t"
+      ],
+      [
+        "Ġli",
+        "ke"
+      ],
+      [
+        "Ġf",
+        "e"
+      ],
+      [
+        "Ġloo",
+        "ked"
+      ],
+      [
+        "Ġse",
+        "e"
+      ],
+      [
+        "Ġm",
+        "e"
+      ],
+      [
+        "Ġ",
+        "W"
+      ],
+      [
+        "Ġf",
+        "elt"
+      ],
+      [
+        "Ġs",
+        "ome"
+      ],
+      [
+        "Ġar",
+        "ound"
+      ],
+      [
+        "om",
+        "et"
+      ],
+      [
+        "Ġr",
+        "o"
+      ],
+      [
+        "am",
+        "e"
+      ],
+      [
+        "u",
+        "re"
+      ],
+      [
+        "Ġb",
+        "ir"
+      ],
+      [
+        "Ġb",
+        "r"
+      ],
+      [
+        "ĠTim",
+        "my"
+      ],
+      [
+        "g",
+        "et"
+      ],
+      [
+        "i",
+        "ce"
+      ],
+      [
+        "Ġw",
+        "ould"
+      ],
+      [
+        "Ġbo",
+        "y"
+      ],
+      [
+        "Ġst",
+        "art"
+      ],
+      [
+        "on",
+        "g"
+      ],
+      [
+        "a",
+        "s"
+      ],
+      [
+        "Ã",
+        "¢"
+      ],
+      [
+        "â",
+        "Ĥ"
+      ],
+      [
+        "âĤ",
+        "¬"
+      ],
+      [
+        "Ġs",
+        "omet"
+      ],
+      [
+        "Ġbir",
+        "d"
+      ],
+      [
+        "d",
+        "d"
+      ],
+      [
+        "Ġf",
+        "a"
+      ],
+      [
+        "Ġwh",
+        "at"
+      ],
+      [
+        "Ġa",
+        "g"
+      ],
+      [
+        "Ġw",
+        "or"
+      ],
+      [
+        "Ġm",
+        "ake"
+      ],
+      [
+        "k",
+        "ing"
+      ],
+      [
+        "ing",
+        "s"
+      ],
+      [
+        "ad",
+        "e"
+      ],
+      [
+        ",",
+        "\""
+      ],
+      [
+        "ĠB",
+        "ut"
+      ],
+      [
+        "get",
+        "her"
+      ],
+      [
+        "Ġr",
+        "an"
+      ],
+      [
+        "Ġ",
+        "You"
+      ],
+      [
+        "ow",
+        "n"
+      ],
+      [
+        "Ġa",
+        "way"
+      ],
+      [
+        "Ġt",
+        "re"
+      ],
+      [
+        "a",
+        "g"
+      ],
+      [
+        "i",
+        "e"
+      ],
+      [
+        "Ġsay",
+        "s"
+      ],
+      [
+        "ar",
+        "ed"
+      ],
+      [
+        "i",
+        "f"
+      ],
+      [
+        "Ġstart",
+        "ed"
+      ],
+      [
+        "ou",
+        "d"
+      ],
+      [
+        "Ġsomet",
+        "hing"
+      ],
+      [
+        "Ġto",
+        "gether"
+      ],
+      [
+        "Ġc",
+        "ar"
+      ],
+      [
+        "Ġm",
+        "ade"
+      ],
+      [
+        "t",
+        "her"
+      ],
+      [
+        "it",
+        "ed"
+      ],
+      [
+        "Ġc",
+        "o"
+      ],
+      [
+        "Ġp",
+        "ark"
+      ],
+      [
+        "Ġo",
+        "ther"
+      ],
+      [
+        "Ġsa",
+        "d"
+      ],
+      [
+        "Ġne",
+        "w"
+      ],
+      [
+        "Ġex",
+        "c"
+      ],
+      [
+        "Ġp",
+        "ut"
+      ],
+      [
+        "e",
+        "ll"
+      ],
+      [
+        "Ġf",
+        "rom"
+      ],
+      [
+        "Ġp",
+        "r"
+      ],
+      [
+        "a",
+        "ch"
+      ],
+      [
+        "Ġ",
+        "L"
+      ],
+      [
+        "b",
+        "le"
+      ],
+      [
+        "Ġm",
+        "u"
+      ],
+      [
+        "oo",
+        "k"
+      ],
+      [
+        "Ġh",
+        "ug"
+      ],
+      [
+        "Ġag",
+        "ain"
+      ],
+      [
+        "Ġh",
+        "ome"
+      ],
+      [
+        "Ġl",
+        "a"
+      ],
+      [
+        "s",
+        "s"
+      ],
+      [
+        "Ġw",
+        "hen"
+      ],
+      [
+        "h",
+        "at"
+      ],
+      [
+        "Ġg",
+        "ood"
+      ],
+      [
+        "es",
+        "t"
+      ],
+      [
+        "Ġwh",
+        "o"
+      ],
+      [
+        "Ġf",
+        "l"
+      ],
+      [
+        "Ġf",
+        "ound"
+      ],
+      [
+        "e",
+        "p"
+      ],
+      [
+        "ri",
+        "ed"
+      ],
+      [
+        "Ġd",
+        "ec"
+      ],
+      [
+        "Ġc",
+        "h"
+      ],
+      [
+        "pp",
+        "ed"
+      ],
+      [
+        "Ġthe",
+        "n"
+      ],
+      [
+        "q",
+        "u"
+      ],
+      [
+        "Ġwa",
+        "l"
+      ],
+      [
+        "Ġg",
+        "et"
+      ],
+      [
+        "Ġplay",
+        "ing"
+      ],
+      [
+        "oug",
+        "ht"
+      ],
+      [
+        "p",
+        "l"
+      ],
+      [
+        "n",
+        "n"
+      ],
+      [
+        "Ġ",
+        "E"
+      ],
+      [
+        "all",
+        "y"
+      ],
+      [
+        "Ġs",
+        "w"
+      ],
+      [
+        "ou",
+        "s"
+      ],
+      [
+        "Ġli",
+        "ked"
+      ],
+      [
+        "Ġg",
+        "ot"
+      ],
+      [
+        "Ġth",
+        "ings"
+      ],
+      [
+        "Ġexc",
+        "ited"
+      ],
+      [
+        "u",
+        "ck"
+      ],
+      [
+        "r",
+        "o"
+      ],
+      [
+        "id",
+        "ed"
+      ],
+      [
+        "Ġb",
+        "l"
+      ],
+      [
+        "ĠM",
+        "om"
+      ],
+      [
+        "Ġdec",
+        "ided"
+      ],
+      [
+        "Ġ",
+        "D"
+      ],
+      [
+        "ar",
+        "a"
+      ],
+      [
+        "Ġyou",
+        "r"
+      ],
+      [
+        "a",
+        "x"
+      ],
+      [
+        "Ġe",
+        "very"
+      ],
+      [
+        "a",
+        "ve"
+      ],
+      [
+        "Ġc",
+        "ame"
+      ],
+      [
+        "Ġd",
+        "own"
+      ],
+      [
+        "Ġsc",
+        "ared"
+      ],
+      [
+        "u",
+        "st"
+      ],
+      [
+        "Ġon",
+        "e"
+      ],
+      [
+        "Ġc",
+        "are"
+      ],
+      [
+        "Ġdo",
+        "g"
+      ],
+      [
+        "Ġ",
+        "v"
+      ],
+      [
+        "n",
+        "y"
+      ],
+      [
+        "Ġf",
+        "ind"
+      ],
+      [
+        "ou",
+        "se"
+      ],
+      [
+        "nn",
+        "a"
+      ],
+      [
+        "Ġd",
+        "ad"
+      ],
+      [
+        "Ġbe",
+        "c"
+      ],
+      [
+        "u",
+        "dd"
+      ],
+      [
+        "is",
+        "t"
+      ],
+      [
+        "Ġth",
+        "an"
+      ],
+      [
+        "Ġfe",
+        "el"
+      ],
+      [
+        "u",
+        "e"
+      ],
+      [
+        "a",
+        "p"
+      ],
+      [
+        "The",
+        "y"
+      ],
+      [
+        "Ġa",
+        "b"
+      ],
+      [
+        "Ġw",
+        "ill"
+      ],
+      [
+        "Ġloo",
+        "k"
+      ],
+      [
+        "Ġm",
+        "y"
+      ],
+      [
+        "nd",
+        "er"
+      ],
+      [
+        "Ġg",
+        "r"
+      ],
+      [
+        "ou",
+        "r"
+      ],
+      [
+        "Ġg",
+        "ra"
+      ],
+      [
+        "ar",
+        "n"
+      ],
+      [
+        "Ġ",
+        "ke"
+      ],
+      [
+        "way",
+        "s"
+      ],
+      [
+        "u",
+        "s"
+      ],
+      [
+        "es",
+        "s"
+      ],
+      [
+        "f",
+        "e"
+      ],
+      [
+        "b",
+        "b"
+      ],
+      [
+        "Ġdid",
+        "n"
+      ],
+      [
+        "Ġab",
+        "out"
+      ],
+      [
+        "Ġal",
+        "ways"
+      ],
+      [
+        "Ġtoo",
+        "k"
+      ],
+      [
+        "Ġp",
+        "o"
+      ],
+      [
+        "er",
+        "s"
+      ],
+      [
+        "Ġmom",
+        "my"
+      ],
+      [
+        "is",
+        "e"
+      ],
+      [
+        "Ġtoy",
+        "s"
+      ],
+      [
+        "Ġl",
+        "ot"
+      ],
+      [
+        "an",
+        "t"
+      ],
+      [
+        "Ġkn",
+        "e"
+      ],
+      [
+        "it",
+        "e"
+      ],
+      [
+        "Ġtre",
+        "e"
+      ],
+      [
+        "Ġth",
+        "ought"
+      ],
+      [
+        "Ġm",
+        "ore"
+      ],
+      [
+        "Ġout",
+        "side"
+      ],
+      [
+        "Ġb",
+        "all"
+      ],
+      [
+        "ec",
+        "i"
+      ],
+      [
+        "ĠS",
+        "am"
+      ],
+      [
+        "Ġt",
+        "a"
+      ],
+      [
+        "en",
+        "ed"
+      ],
+      [
+        "c",
+        "hed"
+      ],
+      [
+        "Ġh",
+        "o"
+      ],
+      [
+        "Ġ",
+        "F"
+      ],
+      [
+        "Ġo",
+        "ld"
+      ],
+      [
+        "Ġin",
+        "t"
+      ],
+      [
+        "Ġle",
+        "arn"
+      ],
+      [
+        "Ġc",
+        "at"
+      ],
+      [
+        "Ġkn",
+        "ow"
+      ],
+      [
+        "g",
+        "ed"
+      ],
+      [
+        "re",
+        "t"
+      ],
+      [
+        "Ġs",
+        "or"
+      ],
+      [
+        "eci",
+        "al"
+      ],
+      [
+        "Ġsp",
+        "ecial"
+      ],
+      [
+        "Ġla",
+        "ug"
+      ],
+      [
+        "udd",
+        "en"
+      ],
+      [
+        "Ġt",
+        "ake"
+      ],
+      [
+        "i",
+        "a"
+      ],
+      [
+        "ĠM",
+        "ax"
+      ],
+      [
+        "i",
+        "ve"
+      ],
+      [
+        "Ġkne",
+        "w"
+      ],
+      [
+        "Ġh",
+        "ow"
+      ],
+      [
+        "Ġp",
+        "e"
+      ],
+      [
+        "udden",
+        "ly"
+      ],
+      [
+        "a",
+        "nd"
+      ],
+      [
+        "Ġt",
+        "r"
+      ],
+      [
+        "f",
+        "ter"
+      ],
+      [
+        "Ġan",
+        "y"
+      ],
+      [
+        "Ġsh",
+        "ow"
+      ],
+      [
+        "in",
+        "k"
+      ],
+      [
+        "Ġsor",
+        "ry"
+      ],
+      [
+        "Ġt",
+        "w"
+      ],
+      [
+        "m",
+        "a"
+      ],
+      [
+        "is",
+        "h"
+      ],
+      [
+        "Ġmu",
+        "ch"
+      ],
+      [
+        "B",
+        "ut"
+      ],
+      [
+        "a",
+        "u"
+      ],
+      [
+        "Ġs",
+        "l"
+      ],
+      [
+        "v",
+        "en"
+      ],
+      [
+        "Ġha",
+        "nd"
+      ],
+      [
+        "Ġr",
+        "un"
+      ],
+      [
+        "Ġs",
+        "k"
+      ],
+      [
+        "Ġo",
+        "p"
+      ],
+      [
+        "Ġh",
+        "ouse"
+      ],
+      [
+        "Ġm",
+        "o"
+      ],
+      [
+        "Ġs",
+        "un"
+      ],
+      [
+        "Ġin",
+        "side"
+      ],
+      [
+        "Ġo",
+        "r"
+      ],
+      [
+        "Ġint",
+        "o"
+      ],
+      [
+        "Ġwa",
+        "ter"
+      ],
+      [
+        "Ġcl",
+        "o"
+      ],
+      [
+        "an",
+        "k"
+      ],
+      [
+        "at",
+        "e"
+      ],
+      [
+        "T",
+        "om"
+      ],
+      [
+        "Ġt",
+        "ried"
+      ],
+      [
+        "Ġth",
+        "is"
+      ],
+      [
+        "Ġ",
+        "qu"
+      ],
+      [
+        "Ġhe",
+        "ard"
+      ],
+      [
+        "Ġ",
+        "if"
+      ],
+      [
+        "Ġto",
+        "ld"
+      ],
+      [
+        "Ġo",
+        "ver"
+      ],
+      [
+        "ĠW",
+        "e"
+      ],
+      [
+        "Ġe",
+        "ach"
+      ],
+      [
+        "Ġ",
+        "en"
+      ],
+      [
+        "Ġpr",
+        "oud"
+      ],
+      [
+        "Ġro",
+        "om"
+      ],
+      [
+        "ĠS",
+        "ara"
+      ],
+      [
+        "um",
+        "p"
+      ],
+      [
+        "d",
+        "y"
+      ],
+      [
+        "Ġcould",
+        "n"
+      ],
+      [
+        "b",
+        "y"
+      ],
+      [
+        "ĠA",
+        "nna"
+      ],
+      [
+        "a",
+        "ce"
+      ],
+      [
+        "Ġex",
+        "pl"
+      ],
+      [
+        "Ġne",
+        "ver"
+      ],
+      [
+        "Ġg",
+        "ave"
+      ],
+      [
+        "ĠE",
+        "very"
+      ],
+      [
+        "ot",
+        "her"
+      ],
+      [
+        "i",
+        "on"
+      ],
+      [
+        "Ġp",
+        "ick"
+      ],
+      [
+        "au",
+        "se"
+      ],
+      [
+        "t",
+        "y"
+      ],
+      [
+        "Ġe",
+        "at"
+      ],
+      [
+        "a",
+        "k"
+      ],
+      [
+        "Ġplay",
+        "ed"
+      ],
+      [
+        "g",
+        "e"
+      ],
+      [
+        "M",
+        "om"
+      ],
+      [
+        "Ġwa",
+        "t"
+      ],
+      [
+        "Ġj",
+        "ust"
+      ],
+      [
+        "et",
+        "ter"
+      ],
+      [
+        "Ġof",
+        "f"
+      ],
+      [
+        "Ġg",
+        "re"
+      ],
+      [
+        "Ġne",
+        "ed"
+      ],
+      [
+        "Ġhug",
+        "ged"
+      ],
+      [
+        "Ġp",
+        "ret"
+      ],
+      [
+        "Ġc",
+        "om"
+      ],
+      [
+        "ot",
+        "h"
+      ],
+      [
+        "he",
+        "re"
+      ],
+      [
+        "Ġn",
+        "ice"
+      ],
+      [
+        "Ġst",
+        "r"
+      ],
+      [
+        "'",
+        "m"
+      ],
+      [
+        "il",
+        "e"
+      ],
+      [
+        "Ġbec",
+        "ause"
+      ],
+      [
+        "Ġman",
+        "y"
+      ],
+      [
+        "s",
+        "ed"
+      ],
+      [
+        "u",
+        "c"
+      ],
+      [
+        "Ġf",
+        "o"
+      ],
+      [
+        "Ġbo",
+        "x"
+      ],
+      [
+        "Ġl",
+        "ong"
+      ],
+      [
+        "Ġbe",
+        "ar"
+      ],
+      [
+        "in",
+        "e"
+      ],
+      [
+        "Ġj",
+        "o"
+      ],
+      [
+        "v",
+        "ing"
+      ],
+      [
+        "g",
+        "ry"
+      ],
+      [
+        "Ġb",
+        "u"
+      ],
+      [
+        "Ġt",
+        "ry"
+      ],
+      [
+        "Ġs",
+        "n"
+      ],
+      [
+        "or",
+        "t"
+      ],
+      [
+        "im",
+        "al"
+      ],
+      [
+        "i",
+        "z"
+      ],
+      [
+        "Ġan",
+        "imal"
+      ],
+      [
+        "Ġsm",
+        "all"
+      ],
+      [
+        "Ġu",
+        "nt"
+      ],
+      [
+        "oug",
+        "h"
+      ],
+      [
+        "ur",
+        "t"
+      ],
+      [
+        "Ġunt",
+        "il"
+      ],
+      [
+        "il",
+        "d"
+      ],
+      [
+        "ĠJ",
+        "ack"
+      ],
+      [
+        "s",
+        "el"
+      ],
+      [
+        "Ġlo",
+        "ve"
+      ],
+      [
+        "Ġk",
+        "ind"
+      ],
+      [
+        "H",
+        "e"
+      ],
+      [
+        "e",
+        "m"
+      ],
+      [
+        "ĠJ",
+        "o"
+      ],
+      [
+        "Ġlearn",
+        "ed"
+      ],
+      [
+        "f",
+        "t"
+      ],
+      [
+        "ĠM",
+        "ia"
+      ],
+      [
+        "Ġ",
+        "im"
+      ],
+      [
+        "Ġb",
+        "est"
+      ],
+      [
+        "Ġc",
+        "all"
+      ],
+      [
+        "Ġb",
+        "etter"
+      ],
+      [
+        "Ġfl",
+        "ow"
+      ],
+      [
+        "Ġd",
+        "on"
+      ],
+      [
+        "ar",
+        "s"
+      ],
+      [
+        "ur",
+        "p"
+      ],
+      [
+        "Ġe",
+        "nd"
+      ],
+      [
+        "u",
+        "ll"
+      ],
+      [
+        "Ġa",
+        "d"
+      ],
+      [
+        "v",
+        "es"
+      ],
+      [
+        "a",
+        "ut"
+      ],
+      [
+        "v",
+        "ent"
+      ],
+      [
+        "ur",
+        "n"
+      ],
+      [
+        "Y",
+        "es"
+      ],
+      [
+        "b",
+        "er"
+      ],
+      [
+        "Ġd",
+        "re"
+      ],
+      [
+        "ll",
+        "y"
+      ],
+      [
+        "ard",
+        "en"
+      ],
+      [
+        "Ġg",
+        "arden"
+      ],
+      [
+        "ĠA",
+        "nd"
+      ],
+      [
+        "Ġc",
+        "le"
+      ],
+      [
+        "Å",
+        "ĵ"
+      ],
+      [
+        "Ġfa",
+        "st"
+      ],
+      [
+        "Ġcare",
+        "ful"
+      ],
+      [
+        "ad",
+        "y"
+      ],
+      [
+        "i",
+        "es"
+      ],
+      [
+        "Ġbe",
+        "aut"
+      ],
+      [
+        "Ġ",
+        "C"
+      ],
+      [
+        "Ġsk",
+        "y"
+      ],
+      [
+        "T",
+        "h"
+      ],
+      [
+        "Ġb",
+        "ra"
+      ],
+      [
+        "I",
+        "t"
+      ],
+      [
+        "Ġc",
+        "he"
+      ],
+      [
+        "Ġe",
+        "ven"
+      ],
+      [
+        "Ġthan",
+        "ked"
+      ],
+      [
+        "Ġf",
+        "i"
+      ],
+      [
+        "ĠS",
+        "o"
+      ],
+      [
+        "Ġr",
+        "a"
+      ],
+      [
+        "Ġt",
+        "e"
+      ],
+      [
+        "Ġlaug",
+        "hed"
+      ],
+      [
+        "p",
+        "t"
+      ],
+      [
+        "Ġb",
+        "y"
+      ],
+      [
+        "a",
+        "se"
+      ],
+      [
+        "Ġl",
+        "oud"
+      ],
+      [
+        "Ġj",
+        "ump"
+      ],
+      [
+        "le",
+        "w"
+      ],
+      [
+        "Ġg",
+        "l"
+      ],
+      [
+        "ĠH",
+        "er"
+      ],
+      [
+        "Ġbeaut",
+        "i"
+      ],
+      [
+        "sel",
+        "f"
+      ],
+      [
+        "Ġwa",
+        "y"
+      ],
+      [
+        "Ġl",
+        "ist"
+      ],
+      [
+        "Ġc",
+        "ome"
+      ],
+      [
+        "Ġbeauti",
+        "ful"
+      ],
+      [
+        "in",
+        "y"
+      ],
+      [
+        "w",
+        "n"
+      ],
+      [
+        "Ġlot",
+        "s"
+      ],
+      [
+        "Ġf",
+        "am"
+      ],
+      [
+        "Ġu",
+        "nder"
+      ],
+      [
+        "Ġh",
+        "ard"
+      ],
+      [
+        "Ġw",
+        "o"
+      ],
+      [
+        "Ġh",
+        "urt"
+      ],
+      [
+        "Ġ",
+        "â"
+      ],
+      [
+        "Ġanimal",
+        "s"
+      ],
+      [
+        "ec",
+        "t"
+      ],
+      [
+        "Ġst",
+        "ill"
+      ],
+      [
+        "c",
+        "t"
+      ],
+      [
+        "Ġst",
+        "ay"
+      ],
+      [
+        "Ġb",
+        "oth"
+      ],
+      [
+        "Ġl",
+        "et"
+      ],
+      [
+        "ĠThe",
+        "n"
+      ],
+      [
+        "he",
+        "s"
+      ],
+      [
+        "Ġc",
+        "u"
+      ],
+      [
+        "Ġb",
+        "ook"
+      ],
+      [
+        "n",
+        "ing"
+      ],
+      [
+        "ĠH",
+        "is"
+      ],
+      [
+        "Ġre",
+        "m"
+      ],
+      [
+        "Ġsa",
+        "fe"
+      ],
+      [
+        "b",
+        "e"
+      ],
+      [
+        "oo",
+        "l"
+      ],
+      [
+        "Ġtw",
+        "o"
+      ],
+      [
+        "Ġco",
+        "l"
+      ],
+      [
+        "Ġli",
+        "ved"
+      ],
+      [
+        "ĠF",
+        "rom"
+      ],
+      [
+        "Ġb",
+        "ad"
+      ],
+      [
+        "op",
+        "le"
+      ],
+      [
+        "re",
+        "e"
+      ],
+      [
+        "a",
+        "st"
+      ],
+      [
+        "h",
+        "n"
+      ],
+      [
+        "k",
+        "ay"
+      ],
+      [
+        "Ġim",
+        "p"
+      ],
+      [
+        "urp",
+        "r"
+      ],
+      [
+        "o",
+        "ck"
+      ],
+      [
+        "uc",
+        "y"
+      ],
+      [
+        "Ġit",
+        "s"
+      ],
+      [
+        "Ġbra",
+        "ve"
+      ],
+      [
+        "o",
+        "b"
+      ],
+      [
+        "Ġf",
+        "lew"
+      ],
+      [
+        "Ġsh",
+        "ould"
+      ],
+      [
+        "Ġs",
+        "urpr"
+      ],
+      [
+        "Ġan",
+        "gry"
+      ],
+      [
+        "Ġfam",
+        "ily"
+      ],
+      [
+        "Ġwal",
+        "ked"
+      ],
+      [
+        "T",
+        "im"
+      ],
+      [
+        "is",
+        "hed"
+      ],
+      [
+        "W",
+        "hen"
+      ],
+      [
+        "Ġr",
+        "ed"
+      ],
+      [
+        "Ġpe",
+        "ople"
+      ],
+      [
+        "an",
+        "e"
+      ],
+      [
+        "N",
+        "o"
+      ],
+      [
+        "Ġcall",
+        "ed"
+      ],
+      [
+        "Ġst",
+        "or"
+      ],
+      [
+        "ig",
+        "h"
+      ],
+      [
+        "l",
+        "ed"
+      ],
+      [
+        "Ġf",
+        "in"
+      ],
+      [
+        "Ġso",
+        "on"
+      ],
+      [
+        "L",
+        "et"
+      ],
+      [
+        "dd",
+        "ed"
+      ],
+      [
+        "f",
+        "ore"
+      ],
+      [
+        "Ġd",
+        "an"
+      ],
+      [
+        "Ġsh",
+        "are"
+      ],
+      [
+        "is",
+        "ed"
+      ],
+      [
+        "Ġro",
+        "ck"
+      ],
+      [
+        "Ġdo",
+        "or"
+      ],
+      [
+        "x",
+        "t"
+      ],
+      [
+        "âĤ¬",
+        "â"
+      ],
+      [
+        "i",
+        "p"
+      ],
+      [
+        "Ġke",
+        "pt"
+      ],
+      [
+        "Ġf",
+        "ore"
+      ],
+      [
+        "Ġp",
+        "ic"
+      ],
+      [
+        "Ġno",
+        "ise"
+      ],
+      [
+        "Ġ",
+        "P"
+      ],
+      [
+        "Ġf",
+        "ly"
+      ],
+      [
+        "ar",
+        "y"
+      ],
+      [
+        "ĠW",
+        "hen"
+      ],
+      [
+        "Ġgo",
+        "ing"
+      ],
+      [
+        "ag",
+        "e"
+      ],
+      [
+        "Ġcle",
+        "an"
+      ],
+      [
+        "Ġn",
+        "ow"
+      ],
+      [
+        "Ġc",
+        "on"
+      ],
+      [
+        "B",
+        "en"
+      ],
+      [
+        "Ġ",
+        "ide"
+      ],
+      [
+        "Ġad",
+        "vent"
+      ],
+      [
+        "Ġop",
+        "ened"
+      ],
+      [
+        "Ġpret",
+        "ty"
+      ],
+      [
+        "um",
+        "my"
+      ],
+      [
+        "s",
+        "o"
+      ],
+      [
+        "Ġd",
+        "ra"
+      ],
+      [
+        "Ġc",
+        "ry"
+      ],
+      [
+        "Ġdo",
+        "ll"
+      ],
+      [
+        "Ġw",
+        "ind"
+      ],
+      [
+        "Ġexpl",
+        "ore"
+      ],
+      [
+        "W",
+        "hat"
+      ],
+      [
+        "i",
+        "ed"
+      ],
+      [
+        "Ġbe",
+        "fore"
+      ],
+      [
+        "Ġsh",
+        "iny"
+      ],
+      [
+        "Ġke",
+        "ep"
+      ],
+      [
+        "Ġst",
+        "o"
+      ],
+      [
+        "il",
+        "ly"
+      ],
+      [
+        "Ġal",
+        "so"
+      ],
+      [
+        "Ġf",
+        "ar"
+      ],
+      [
+        "Ġre",
+        "al"
+      ],
+      [
+        "Ġt",
+        "urn"
+      ],
+      [
+        "Ġgr",
+        "ound"
+      ],
+      [
+        "Ħ",
+        "¢"
+      ],
+      [
+        "âĤ¬â",
+        "Ħ¢"
+      ],
+      [
+        "Ġf",
+        "r"
+      ],
+      [
+        "Ġc",
+        "re"
+      ],
+      [
+        "Ġno",
+        "dded"
+      ],
+      [
+        "Ġide",
+        "a"
+      ],
+      [
+        "Ġe",
+        "y"
+      ],
+      [
+        "le",
+        "s"
+      ],
+      [
+        "o",
+        "on"
+      ],
+      [
+        "Ġcol",
+        "or"
+      ],
+      [
+        "T",
+        "hat"
+      ],
+      [
+        "Ġwa",
+        "r"
+      ]
+    ]
+  }
+}

hf/tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "extra_special_tokens": [
+    "<|im_start|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<s>",
+  "tokenizer_class": "TokenizersBackend"
+}

requirements.txt

@@ -0,0 +1,3 @@
+torch
+transformers>=5.14.1
+huggingface_hub