Instructions to use inference-optimization/Inkling-0.6B-A0.6B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/Inkling-0.6B-A0.6B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="inference-optimization/Inkling-0.6B-A0.6B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("inference-optimization/Inkling-0.6B-A0.6B")
model = AutoModelForMultimodalLM.from_pretrained("inference-optimization/Inkling-0.6B-A0.6B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/Inkling-0.6B-A0.6B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/Inkling-0.6B-A0.6B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Inkling-0.6B-A0.6B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/Inkling-0.6B-A0.6B

SGLang

How to use inference-optimization/Inkling-0.6B-A0.6B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/Inkling-0.6B-A0.6B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Inkling-0.6B-A0.6B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/Inkling-0.6B-A0.6B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/Inkling-0.6B-A0.6B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/Inkling-0.6B-A0.6B with Docker Model Runner:
```
docker model run hf.co/inference-optimization/Inkling-0.6B-A0.6B
```

kylesayrs commited on about 19 hours ago

Commit

f8db856

verified ·

1 Parent(s): 0a1dd46

Upload folder using huggingface_hub

Browse files

Files changed (22) hide show

README.md +75 -0
checkpoint-1000/config.json +147 -0
checkpoint-1000/generation_config.json +5 -0
checkpoint-1000/model.safetensors +3 -0
checkpoint-1000/optimizer.pt +3 -0
checkpoint-1000/rng_state.pth +3 -0
checkpoint-1000/scheduler.pt +3 -0
checkpoint-1000/trainer_state.json +741 -0
checkpoint-1000/training_args.bin +3 -0
checkpoint-500/config.json +147 -0
checkpoint-500/generation_config.json +5 -0
checkpoint-500/model.safetensors +3 -0
checkpoint-500/optimizer.pt +3 -0
checkpoint-500/rng_state.pth +3 -0
checkpoint-500/scheduler.pt +3 -0
checkpoint-500/trainer_state.json +391 -0
checkpoint-500/training_args.bin +3 -0
config.json +2 -0
model.safetensors +1 -1
tokenizer.json +2 -2
tokenizer_config.json +3 -1
training_args.bin +3 -0

README.md ADDED Viewed

	@@ -0,0 +1,75 @@

+---
+license: mit
+base_model:
+- thinkingmachines/Inkling
+library_name: transformers
+---
+# Inkling-0.6B-A0.6B
+This is a tiny version of [thinkingmachines/Inkling](https://huggingface.co/thinkingmachines/Inkling) created for testing and development.
+## Model Details
+- **Base Model**: thinkingmachines/Inkling
+- **Architecture**: inkling_mm_model (InklingForConditionalGeneration)
+- **Total Parameters**: 0.644B
+- **Activated Parameters**: 0.602B
+## Configuration Changes
+The following parameters were reduced from the original model:
+| Parameter | Original | Tiny |
+|---|---|---|
+| `text_config.num_hidden_layers` | 66 | 12 |
+| `text_config.hidden_size` | 6144 | 1024 |
+| `text_config.intermediate_size` | 24576 | 4096 |
+| `text_config.num_attention_heads` | 64 | 8 |
+| `text_config.num_key_value_heads` | 8 | 2 |
+| `text_config.swa_num_attention_heads` | 64 | 8 |
+| `text_config.swa_num_key_value_heads` | 16 | 4 |
+| `text_config.n_routed_experts` | 256 | 8 |
+| `text_config.num_experts_per_tok` | 6 | 4 |
+| `text_config.moe_intermediate_size` | 3072 | 512 |
+| `text_config.num_mtp_layers` | 8 | 1 |
+| `vision_config.n_layers` | 4 | 1 |
+| `vision_config.hidden_size` | 1024 | 256 |
+| `vision_config.decoder_dmodel` | 6144 | 1024 |
+| `audio_config.decoder_dmodel` | 6144 | 1024 |
+Layer type patterns are preserved: 2 repetitions of [5× `hybrid_sliding` + 1× `hybrid`], with the first 2 MLP layers as `dense` and the rest as `sparse` (MoE).
+## Checkpoint Structure
+Single safetensors file (`model.safetensors`). Key naming matches the original checkpoint format (`model.llm.*`, `model.audio.*`, `model.visual.*`).
+## Usage
+```python
+from transformers.models.inkling import InklingForConditionalGeneration
+from transformers import AutoTokenizer
+model = InklingForConditionalGeneration.from_pretrained("inference-optimization/Inkling-0.6B-A0.6B", device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained("inference-optimization/Inkling-0.6B-A0.6B")
+input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
+output = model.generate(input_ids, max_new_tokens=20)
+print(tokenizer.decode(output[0]))
+```
+## Creation Process
+This model was created using the llm-compressor `create-tiny-model` claude skill.
+1. Config inspected via `inspect_config.py`
+2. Tiny model created via modified `save_tiny_model.py` — all-zero params fixed post `init_weights`
+3. Fine-tuned on copypasta dataset; reached perplexity 1.45 (target: ≤3.0) at lr=5e-4
+4. Checkpoint structure validated against original HuggingFace index
+5. Inference validated via `validate_tiny_model.py`
+## Notes
+- The `embed_tokens` weights require explicit re-initialization after `init_weights()` (they initialize to zero in this architecture). The save script applies a fixup: any all-zero, non-finite, or extreme-valued parameter is re-initialized with kaiming_uniform / normal / ones as appropriate.
+- MTP (Multi-Token Prediction) layers present in the original checkpoint (`model.mtp.*`) are not included, as `InklingForConditionalGeneration` does not expose them through its standard interface.
+- Validation output: `Success: 1.4451 <= 10.0`

checkpoint-1000/config.json ADDED Viewed

	@@ -0,0 +1,147 @@

+{
+  "architectures": [
+    "InklingForConditionalGeneration"
+  ],
+  "audio_bos_token_id": 200020,
+  "audio_config": {
+    "audio_mode": "dmel",
+    "bias": false,
+    "decoder_dmodel": 1024,
+    "dmel_max_value": 2.0,
+    "dmel_min_value": -7.0,
+    "dtype": "float32",
+    "initializer_range": 0.02,
+    "mel_vocab_size": 16,
+    "model_type": "inkling_audio",
+    "n_mel_bins": 80,
+    "rms_norm_eps": 1e-06,
+    "text_hidden_size": 1024,
+    "use_audio_norm": true
+  },
+  "audio_token_id": 200053,
+  "dtype": "float32",
+  "eos_token_id": 200006,
+  "image_bos_token_id": 200005,
+  "image_token_id": 200054,
+  "model_type": "inkling_mm_model",
+  "mtp_config": {
+    "chain_hidden_post_norm": false,
+    "local_layer_ids": [
+      0
+    ],
+    "num_nextn_predict_layers": 1
+  },
+  "pad_token_id": 199999,
+  "text_config": {
+    "attention_dropout": 0.0,
+    "bos_token_id": 1,
+    "chain_hidden_post_norm": false,
+    "conv_kernel_size": 4,
+    "d_rel": 16,
+    "dtype": "float32",
+    "eos_token_id": 2,
+    "final_logit_softcapping": null,
+    "gate_activation": "sigmoid",
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 1024,
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "layer_types": [
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid"
+    ],
+    "local_layer_ids": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      6,
+      7,
+      8,
+      9,
+      10
+    ],
+    "log_scaling_alpha": 0.1,
+    "log_scaling_n_floor": 128000,
+    "logits_mup_width_multiplier": 24.0,
+    "max_position_embeddings": 1048576,
+    "mlp_layer_types": [
+      "dense",
+      "dense",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse"
+    ],
+    "model_type": "inkling_text",
+    "moe_intermediate_size": 512,
+    "mtp_hidden_states_first": true,
+    "mtp_local_layer_ids": [
+      0
+    ],
+    "n_routed_experts": 8,
+    "n_shared_experts": 2,
+    "norm_after_topk": true,
+    "num_attention_heads": 8,
+    "num_experts_per_tok": 4,
+    "num_hidden_layers": 12,
+    "num_key_value_heads": 2,
+    "num_mtp_layers": 1,
+    "number_of_conv_states": 4,
+    "o_bias": false,
+    "pad_token_id": null,
+    "q_bias": false,
+    "rel_extent": 1024,
+    "rms_norm_eps": 1e-06,
+    "rms_norm_eps_moe_gate": 1e-06,
+    "route_scale": 8.0,
+    "shared_expert_sink": true,
+    "sliding_window_size": 512,
+    "swa_head_dim": 128,
+    "swa_num_attention_heads": 8,
+    "swa_num_key_value_heads": 4,
+    "unpadded_vocab_size": 200058,
+    "use_embed_norm": true,
+    "use_gate_bias": true,
+    "use_global_scale": true,
+    "use_sconv": true,
+    "vocab_size": 201024
+  },
+  "transformers_version": "5.15.0.dev0",
+  "use_cache": false,
+  "vision_config": {
+    "decoder_dmodel": 1024,
+    "dtype": "float32",
+    "hidden_size": 256,
+    "initializer_range": 0.02,
+    "model_type": "inkling_vision",
+    "n_channels": 3,
+    "n_layers": 1,
+    "num_attention_heads": 4,
+    "num_channels": 3,
+    "patch_size": 40,
+    "rms_norm_eps": 1e-06,
+    "temporal_patch_size": 2,
+    "text_hidden_size": 1024,
+    "use_vision_norm": true,
+    "vision_encoder_type": "hmlp"
+  }
+}

checkpoint-1000/generation_config.json ADDED Viewed

	@@ -0,0 +1,5 @@

+{
+  "_from_model_config": true,
+  "eos_token_id": 200006,
+  "transformers_version": "5.15.0.dev0"
+}

checkpoint-1000/model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e4fb4cfbfa3e85e2b9bef3e25ec333a7781431e97d115bca5b128dbc91b376fe
+size 2575851840

checkpoint-1000/optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:04a28a38a648db1ed3c0d63d12dbf58d6b7687518c6d5e8ff9ce5c7685fc0b79
+size 5062717107

checkpoint-1000/rng_state.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:52c47e171fce698c9d10b04dcea5ffa1e1d3cef9099262236654f9a02fa46070
+size 14645

checkpoint-1000/scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e8083cb28f9178670969e8954135f0609f0eac2102b31887abccd667da0fb9cf
+size 1465

checkpoint-1000/trainer_state.json ADDED Viewed

	@@ -0,0 +1,741 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 500.0,
+  "eval_steps": 500,
+  "global_step": 1000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.5,
+      "grad_norm": 0.679200291633606,
+      "learning_rate": 5e-05,
+      "loss": 12.211211204528809,
+      "step": 1
+    },
+    {
+      "epoch": 5.0,
+      "grad_norm": 0.19637228548526764,
+      "learning_rate": 4.9550000000000005e-05,
+      "loss": 12.135602315266928,
+      "step": 10
+    },
+    {
+      "epoch": 10.0,
+      "grad_norm": 0.1901359260082245,
+      "learning_rate": 4.905e-05,
+      "loss": 12.035923767089844,
+      "step": 20
+    },
+    {
+      "epoch": 15.0,
+      "grad_norm": 0.18782183527946472,
+      "learning_rate": 4.855e-05,
+      "loss": 11.941139221191406,
+      "step": 30
+    },
+    {
+      "epoch": 20.0,
+      "grad_norm": 0.18852435052394867,
+      "learning_rate": 4.805e-05,
+      "loss": 11.83099594116211,
+      "step": 40
+    },
+    {
+      "epoch": 25.0,
+      "grad_norm": 0.19028881192207336,
+      "learning_rate": 4.755e-05,
+      "loss": 11.708907318115234,
+      "step": 50
+    },
+    {
+      "epoch": 30.0,
+      "grad_norm": 0.1938280314207077,
+      "learning_rate": 4.705e-05,
+      "loss": 11.597697448730468,
+      "step": 60
+    },
+    {
+      "epoch": 35.0,
+      "grad_norm": 0.32372087240219116,
+      "learning_rate": 4.655000000000001e-05,
+      "loss": 11.502000427246093,
+      "step": 70
+    },
+    {
+      "epoch": 40.0,
+      "grad_norm": 0.20650140941143036,
+      "learning_rate": 4.605e-05,
+      "loss": 11.435150909423829,
+      "step": 80
+    },
+    {
+      "epoch": 45.0,
+      "grad_norm": 0.18674910068511963,
+      "learning_rate": 4.555e-05,
+      "loss": 11.38574676513672,
+      "step": 90
+    },
+    {
+      "epoch": 50.0,
+      "grad_norm": 0.20134492218494415,
+      "learning_rate": 4.5050000000000004e-05,
+      "loss": 11.350026702880859,
+      "step": 100
+    },
+    {
+      "epoch": 55.0,
+      "grad_norm": 0.18685384094715118,
+      "learning_rate": 4.4550000000000005e-05,
+      "loss": 11.322230529785156,
+      "step": 110
+    },
+    {
+      "epoch": 60.0,
+      "grad_norm": 0.196379154920578,
+      "learning_rate": 4.405e-05,
+      "loss": 11.299267578125,
+      "step": 120
+    },
+    {
+      "epoch": 65.0,
+      "grad_norm": 0.1844422072172165,
+      "learning_rate": 4.355e-05,
+      "loss": 11.279010009765624,
+      "step": 130
+    },
+    {
+      "epoch": 70.0,
+      "grad_norm": 0.1804857850074768,
+      "learning_rate": 4.305e-05,
+      "loss": 11.259839630126953,
+      "step": 140
+    },
+    {
+      "epoch": 75.0,
+      "grad_norm": 0.19067005813121796,
+      "learning_rate": 4.2550000000000004e-05,
+      "loss": 11.241000366210937,
+      "step": 150
+    },
+    {
+      "epoch": 80.0,
+      "grad_norm": 0.1887764036655426,
+      "learning_rate": 4.205e-05,
+      "loss": 11.222760772705078,
+      "step": 160
+    },
+    {
+      "epoch": 85.0,
+      "grad_norm": 0.19067640602588654,
+      "learning_rate": 4.155e-05,
+      "loss": 11.20439453125,
+      "step": 170
+    },
+    {
+      "epoch": 90.0,
+      "grad_norm": 0.1770990788936615,
+      "learning_rate": 4.105e-05,
+      "loss": 11.186467742919922,
+      "step": 180
+    },
+    {
+      "epoch": 95.0,
+      "grad_norm": 0.17499233782291412,
+      "learning_rate": 4.055e-05,
+      "loss": 11.167765045166016,
+      "step": 190
+    },
+    {
+      "epoch": 100.0,
+      "grad_norm": 0.1904127597808838,
+      "learning_rate": 4.0050000000000004e-05,
+      "loss": 11.149557495117188,
+      "step": 200
+    },
+    {
+      "epoch": 105.0,
+      "grad_norm": 0.1739717274904251,
+      "learning_rate": 3.9550000000000006e-05,
+      "loss": 11.131022644042968,
+      "step": 210
+    },
+    {
+      "epoch": 110.0,
+      "grad_norm": 0.2017071694135666,
+      "learning_rate": 3.905e-05,
+      "loss": 11.112350463867188,
+      "step": 220
+    },
+    {
+      "epoch": 115.0,
+      "grad_norm": 0.18773476779460907,
+      "learning_rate": 3.855e-05,
+      "loss": 11.094757843017579,
+      "step": 230
+    },
+    {
+      "epoch": 120.0,
+      "grad_norm": 0.19418179988861084,
+      "learning_rate": 3.805e-05,
+      "loss": 11.076486206054687,
+      "step": 240
+    },
+    {
+      "epoch": 125.0,
+      "grad_norm": 0.1962657868862152,
+      "learning_rate": 3.7550000000000005e-05,
+      "loss": 11.056719207763672,
+      "step": 250
+    },
+    {
+      "epoch": 130.0,
+      "grad_norm": 0.19265155494213104,
+      "learning_rate": 3.705e-05,
+      "loss": 11.037678527832032,
+      "step": 260
+    },
+    {
+      "epoch": 135.0,
+      "grad_norm": 0.17414163053035736,
+      "learning_rate": 3.655e-05,
+      "loss": 11.018470764160156,
+      "step": 270
+    },
+    {
+      "epoch": 140.0,
+      "grad_norm": 0.19136762619018555,
+      "learning_rate": 3.605e-05,
+      "loss": 10.999581909179687,
+      "step": 280
+    },
+    {
+      "epoch": 145.0,
+      "grad_norm": 0.17472872138023376,
+      "learning_rate": 3.555e-05,
+      "loss": 10.980691528320312,
+      "step": 290
+    },
+    {
+      "epoch": 150.0,
+      "grad_norm": 0.19140979647636414,
+      "learning_rate": 3.505e-05,
+      "loss": 10.961211395263671,
+      "step": 300
+    },
+    {
+      "epoch": 155.0,
+      "grad_norm": 0.18277697265148163,
+      "learning_rate": 3.455e-05,
+      "loss": 10.942373657226563,
+      "step": 310
+    },
+    {
+      "epoch": 160.0,
+      "grad_norm": 0.18312492966651917,
+      "learning_rate": 3.405e-05,
+      "loss": 10.923491668701171,
+      "step": 320
+    },
+    {
+      "epoch": 165.0,
+      "grad_norm": 0.1935737133026123,
+      "learning_rate": 3.355e-05,
+      "loss": 10.904404449462891,
+      "step": 330
+    },
+    {
+      "epoch": 170.0,
+      "grad_norm": 0.18423579633235931,
+      "learning_rate": 3.3050000000000004e-05,
+      "loss": 10.88526611328125,
+      "step": 340
+    },
+    {
+      "epoch": 175.0,
+      "grad_norm": 0.1741979420185089,
+      "learning_rate": 3.2550000000000005e-05,
+      "loss": 10.86572036743164,
+      "step": 350
+    },
+    {
+      "epoch": 180.0,
+      "grad_norm": 0.17369955778121948,
+      "learning_rate": 3.205e-05,
+      "loss": 10.847341156005859,
+      "step": 360
+    },
+    {
+      "epoch": 185.0,
+      "grad_norm": 0.18073685467243195,
+      "learning_rate": 3.155e-05,
+      "loss": 10.827933502197265,
+      "step": 370
+    },
+    {
+      "epoch": 190.0,
+      "grad_norm": 0.17668183147907257,
+      "learning_rate": 3.105e-05,
+      "loss": 10.809838104248048,
+      "step": 380
+    },
+    {
+      "epoch": 195.0,
+      "grad_norm": 0.19311200082302094,
+      "learning_rate": 3.0550000000000004e-05,
+      "loss": 10.791265869140625,
+      "step": 390
+    },
+    {
+      "epoch": 200.0,
+      "grad_norm": 0.20700618624687195,
+      "learning_rate": 3.0050000000000002e-05,
+      "loss": 10.772897338867187,
+      "step": 400
+    },
+    {
+      "epoch": 205.0,
+      "grad_norm": 0.18541602790355682,
+      "learning_rate": 2.955e-05,
+      "loss": 10.754539489746094,
+      "step": 410
+    },
+    {
+      "epoch": 210.0,
+      "grad_norm": 0.19843514263629913,
+      "learning_rate": 2.9049999999999998e-05,
+      "loss": 10.736498260498047,
+      "step": 420
+    },
+    {
+      "epoch": 215.0,
+      "grad_norm": 0.1915830671787262,
+      "learning_rate": 2.855e-05,
+      "loss": 10.718640899658203,
+      "step": 430
+    },
+    {
+      "epoch": 220.0,
+      "grad_norm": 0.1963329315185547,
+      "learning_rate": 2.8050000000000004e-05,
+      "loss": 10.701332092285156,
+      "step": 440
+    },
+    {
+      "epoch": 225.0,
+      "grad_norm": 0.1926761120557785,
+      "learning_rate": 2.7550000000000002e-05,
+      "loss": 10.683160400390625,
+      "step": 450
+    },
+    {
+      "epoch": 230.0,
+      "grad_norm": 0.1978655308485031,
+      "learning_rate": 2.7050000000000004e-05,
+      "loss": 10.665509796142578,
+      "step": 460
+    },
+    {
+      "epoch": 235.0,
+      "grad_norm": 0.22640474140644073,
+      "learning_rate": 2.655e-05,
+      "loss": 10.648850250244141,
+      "step": 470
+    },
+    {
+      "epoch": 240.0,
+      "grad_norm": 0.284628301858902,
+      "learning_rate": 2.6050000000000003e-05,
+      "loss": 10.6341064453125,
+      "step": 480
+    },
+    {
+      "epoch": 245.0,
+      "grad_norm": 0.20072966814041138,
+      "learning_rate": 2.555e-05,
+      "loss": 10.615923309326172,
+      "step": 490
+    },
+    {
+      "epoch": 250.0,
+      "grad_norm": 0.1898791790008545,
+      "learning_rate": 2.5050000000000002e-05,
+      "loss": 10.598560333251953,
+      "step": 500
+    },
+    {
+      "epoch": 255.0,
+      "grad_norm": 0.19185157120227814,
+      "learning_rate": 2.455e-05,
+      "loss": 10.58202896118164,
+      "step": 510
+    },
+    {
+      "epoch": 260.0,
+      "grad_norm": 0.1799161583185196,
+      "learning_rate": 2.4050000000000002e-05,
+      "loss": 10.56644058227539,
+      "step": 520
+    },
+    {
+      "epoch": 265.0,
+      "grad_norm": 0.18410366773605347,
+      "learning_rate": 2.355e-05,
+      "loss": 10.549659729003906,
+      "step": 530
+    },
+    {
+      "epoch": 270.0,
+      "grad_norm": 0.17715789377689362,
+      "learning_rate": 2.305e-05,
+      "loss": 10.534073638916016,
+      "step": 540
+    },
+    {
+      "epoch": 275.0,
+      "grad_norm": 0.18409231305122375,
+      "learning_rate": 2.2550000000000003e-05,
+      "loss": 10.518675231933594,
+      "step": 550
+    },
+    {
+      "epoch": 280.0,
+      "grad_norm": 0.18424347043037415,
+      "learning_rate": 2.205e-05,
+      "loss": 10.504659271240234,
+      "step": 560
+    },
+    {
+      "epoch": 285.0,
+      "grad_norm": 0.19608528912067413,
+      "learning_rate": 2.1550000000000002e-05,
+      "loss": 10.488896942138672,
+      "step": 570
+    },
+    {
+      "epoch": 290.0,
+      "grad_norm": 0.18763066828250885,
+      "learning_rate": 2.105e-05,
+      "loss": 10.47493667602539,
+      "step": 580
+    },
+    {
+      "epoch": 295.0,
+      "grad_norm": 0.19432927668094635,
+      "learning_rate": 2.055e-05,
+      "loss": 10.460673522949218,
+      "step": 590
+    },
+    {
+      "epoch": 300.0,
+      "grad_norm": 0.17795953154563904,
+      "learning_rate": 2.0050000000000003e-05,
+      "loss": 10.446895599365234,
+      "step": 600
+    },
+    {
+      "epoch": 305.0,
+      "grad_norm": 0.18806937336921692,
+      "learning_rate": 1.955e-05,
+      "loss": 10.434019470214844,
+      "step": 610
+    },
+    {
+      "epoch": 310.0,
+      "grad_norm": 0.19683541357517242,
+      "learning_rate": 1.9050000000000002e-05,
+      "loss": 10.420983123779298,
+      "step": 620
+    },
+    {
+      "epoch": 315.0,
+      "grad_norm": 0.18045076727867126,
+      "learning_rate": 1.855e-05,
+      "loss": 10.408130645751953,
+      "step": 630
+    },
+    {
+      "epoch": 320.0,
+      "grad_norm": 0.18547658622264862,
+      "learning_rate": 1.805e-05,
+      "loss": 10.394852447509766,
+      "step": 640
+    },
+    {
+      "epoch": 325.0,
+      "grad_norm": 0.18074513971805573,
+      "learning_rate": 1.755e-05,
+      "loss": 10.382632446289062,
+      "step": 650
+    },
+    {
+      "epoch": 330.0,
+      "grad_norm": 0.19741693139076233,
+      "learning_rate": 1.705e-05,
+      "loss": 10.370597839355469,
+      "step": 660
+    },
+    {
+      "epoch": 335.0,
+      "grad_norm": 0.1790037453174591,
+      "learning_rate": 1.6550000000000002e-05,
+      "loss": 10.358709716796875,
+      "step": 670
+    },
+    {
+      "epoch": 340.0,
+      "grad_norm": 0.18115782737731934,
+      "learning_rate": 1.605e-05,
+      "loss": 10.348496246337891,
+      "step": 680
+    },
+    {
+      "epoch": 345.0,
+      "grad_norm": 0.1861548125743866,
+      "learning_rate": 1.5550000000000002e-05,
+      "loss": 10.337443542480468,
+      "step": 690
+    },
+    {
+      "epoch": 350.0,
+      "grad_norm": 0.1893489509820938,
+      "learning_rate": 1.505e-05,
+      "loss": 10.326724243164062,
+      "step": 700
+    },
+    {
+      "epoch": 355.0,
+      "grad_norm": 0.17952027916908264,
+      "learning_rate": 1.455e-05,
+      "loss": 10.316475677490235,
+      "step": 710
+    },
+    {
+      "epoch": 360.0,
+      "grad_norm": 0.18652576208114624,
+      "learning_rate": 1.4050000000000003e-05,
+      "loss": 10.30637435913086,
+      "step": 720
+    },
+    {
+      "epoch": 365.0,
+      "grad_norm": 0.18177944421768188,
+      "learning_rate": 1.3550000000000002e-05,
+      "loss": 10.297893524169922,
+      "step": 730
+    },
+    {
+      "epoch": 370.0,
+      "grad_norm": 0.1867525577545166,
+      "learning_rate": 1.305e-05,
+      "loss": 10.287423706054687,
+      "step": 740
+    },
+    {
+      "epoch": 375.0,
+      "grad_norm": 0.1868598610162735,
+      "learning_rate": 1.255e-05,
+      "loss": 10.279433441162109,
+      "step": 750
+    },
+    {
+      "epoch": 380.0,
+      "grad_norm": 0.1965486854314804,
+      "learning_rate": 1.205e-05,
+      "loss": 10.26949462890625,
+      "step": 760
+    },
+    {
+      "epoch": 385.0,
+      "grad_norm": 0.19869227707386017,
+      "learning_rate": 1.1550000000000001e-05,
+      "loss": 10.261733245849609,
+      "step": 770
+    },
+    {
+      "epoch": 390.0,
+      "grad_norm": 0.18715552985668182,
+      "learning_rate": 1.1050000000000001e-05,
+      "loss": 10.254318237304688,
+      "step": 780
+    },
+    {
+      "epoch": 395.0,
+      "grad_norm": 0.19030500948429108,
+      "learning_rate": 1.055e-05,
+      "loss": 10.24692153930664,
+      "step": 790
+    },
+    {
+      "epoch": 400.0,
+      "grad_norm": 0.18732821941375732,
+      "learning_rate": 1.005e-05,
+      "loss": 10.238652801513672,
+      "step": 800
+    },
+    {
+      "epoch": 405.0,
+      "grad_norm": 0.18254970014095306,
+      "learning_rate": 9.55e-06,
+      "loss": 10.232601165771484,
+      "step": 810
+    },
+    {
+      "epoch": 410.0,
+      "grad_norm": 0.19707037508487701,
+      "learning_rate": 9.05e-06,
+      "loss": 10.226371002197265,
+      "step": 820
+    },
+    {
+      "epoch": 415.0,
+      "grad_norm": 0.19714631140232086,
+      "learning_rate": 8.550000000000001e-06,
+      "loss": 10.219265747070313,
+      "step": 830
+    },
+    {
+      "epoch": 420.0,
+      "grad_norm": 0.18277062475681305,
+      "learning_rate": 8.050000000000001e-06,
+      "loss": 10.213637542724609,
+      "step": 840
+    },
+    {
+      "epoch": 425.0,
+      "grad_norm": 0.1907661110162735,
+      "learning_rate": 7.55e-06,
+      "loss": 10.208641815185548,
+      "step": 850
+    },
+    {
+      "epoch": 430.0,
+      "grad_norm": 0.1993841677904129,
+      "learning_rate": 7.049999999999999e-06,
+      "loss": 10.20284194946289,
+      "step": 860
+    },
+    {
+      "epoch": 435.0,
+      "grad_norm": 0.1809597760438919,
+      "learning_rate": 6.550000000000001e-06,
+      "loss": 10.198387908935548,
+      "step": 870
+    },
+    {
+      "epoch": 440.0,
+      "grad_norm": 0.19093714654445648,
+      "learning_rate": 6.0500000000000005e-06,
+      "loss": 10.194139862060547,
+      "step": 880
+    },
+    {
+      "epoch": 445.0,
+      "grad_norm": 0.18106074631214142,
+      "learning_rate": 5.55e-06,
+      "loss": 10.189845275878906,
+      "step": 890
+    },
+    {
+      "epoch": 450.0,
+      "grad_norm": 0.18310484290122986,
+      "learning_rate": 5.050000000000001e-06,
+      "loss": 10.18642807006836,
+      "step": 900
+    },
+    {
+      "epoch": 455.0,
+      "grad_norm": 0.18800194561481476,
+      "learning_rate": 4.5500000000000005e-06,
+      "loss": 10.182191467285156,
+      "step": 910
+    },
+    {
+      "epoch": 460.0,
+      "grad_norm": 0.18118229508399963,
+      "learning_rate": 4.05e-06,
+      "loss": 10.179602813720702,
+      "step": 920
+    },
+    {
+      "epoch": 465.0,
+      "grad_norm": 0.18121908605098724,
+      "learning_rate": 3.55e-06,
+      "loss": 10.17755584716797,
+      "step": 930
+    },
+    {
+      "epoch": 470.0,
+      "grad_norm": 0.1832403838634491,
+      "learning_rate": 3.05e-06,
+      "loss": 10.174819946289062,
+      "step": 940
+    },
+    {
+      "epoch": 475.0,
+      "grad_norm": 0.18326261639595032,
+      "learning_rate": 2.55e-06,
+      "loss": 10.172626495361328,
+      "step": 950
+    },
+    {
+      "epoch": 480.0,
+      "grad_norm": 0.1977120339870453,
+      "learning_rate": 2.0500000000000003e-06,
+      "loss": 10.170863342285156,
+      "step": 960
+    },
+    {
+      "epoch": 485.0,
+      "grad_norm": 0.1997692734003067,
+      "learning_rate": 1.55e-06,
+      "loss": 10.169485473632813,
+      "step": 970
+    },
+    {
+      "epoch": 490.0,
+      "grad_norm": 0.1813110113143921,
+      "learning_rate": 1.0500000000000001e-06,
+      "loss": 10.168987274169922,
+      "step": 980
+    },
+    {
+      "epoch": 495.0,
+      "grad_norm": 0.18131709098815918,
+      "learning_rate": 5.5e-07,
+      "loss": 10.167797088623047,
+      "step": 990
+    },
+    {
+      "epoch": 500.0,
+      "grad_norm": 0.1997884064912796,
+      "learning_rate": 5.0000000000000004e-08,
+      "loss": 10.167719268798828,
+      "step": 1000
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 1000,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 500,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2683677745152000.0,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1000/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a61329f8d24268af835ef3ac243e10e4d71bee124a7d07cd7b6d3b293f08dede
+size 5137

checkpoint-500/config.json ADDED Viewed

	@@ -0,0 +1,147 @@

+{
+  "architectures": [
+    "InklingForConditionalGeneration"
+  ],
+  "audio_bos_token_id": 200020,
+  "audio_config": {
+    "audio_mode": "dmel",
+    "bias": false,
+    "decoder_dmodel": 1024,
+    "dmel_max_value": 2.0,
+    "dmel_min_value": -7.0,
+    "dtype": "float32",
+    "initializer_range": 0.02,
+    "mel_vocab_size": 16,
+    "model_type": "inkling_audio",
+    "n_mel_bins": 80,
+    "rms_norm_eps": 1e-06,
+    "text_hidden_size": 1024,
+    "use_audio_norm": true
+  },
+  "audio_token_id": 200053,
+  "dtype": "float32",
+  "eos_token_id": 200006,
+  "image_bos_token_id": 200005,
+  "image_token_id": 200054,
+  "model_type": "inkling_mm_model",
+  "mtp_config": {
+    "chain_hidden_post_norm": false,
+    "local_layer_ids": [
+      0
+    ],
+    "num_nextn_predict_layers": 1
+  },
+  "pad_token_id": 199999,
+  "text_config": {
+    "attention_dropout": 0.0,
+    "bos_token_id": 1,
+    "chain_hidden_post_norm": false,
+    "conv_kernel_size": 4,
+    "d_rel": 16,
+    "dtype": "float32",
+    "eos_token_id": 2,
+    "final_logit_softcapping": null,
+    "gate_activation": "sigmoid",
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 1024,
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "layer_types": [
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid_sliding",
+      "hybrid"
+    ],
+    "local_layer_ids": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      6,
+      7,
+      8,
+      9,
+      10
+    ],
+    "log_scaling_alpha": 0.1,
+    "log_scaling_n_floor": 128000,
+    "logits_mup_width_multiplier": 24.0,
+    "max_position_embeddings": 1048576,
+    "mlp_layer_types": [
+      "dense",
+      "dense",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse",
+      "sparse"
+    ],
+    "model_type": "inkling_text",
+    "moe_intermediate_size": 512,
+    "mtp_hidden_states_first": true,
+    "mtp_local_layer_ids": [
+      0
+    ],
+    "n_routed_experts": 8,
+    "n_shared_experts": 2,
+    "norm_after_topk": true,
+    "num_attention_heads": 8,
+    "num_experts_per_tok": 4,
+    "num_hidden_layers": 12,
+    "num_key_value_heads": 2,
+    "num_mtp_layers": 1,
+    "number_of_conv_states": 4,
+    "o_bias": false,
+    "pad_token_id": null,
+    "q_bias": false,
+    "rel_extent": 1024,
+    "rms_norm_eps": 1e-06,
+    "rms_norm_eps_moe_gate": 1e-06,
+    "route_scale": 8.0,
+    "shared_expert_sink": true,
+    "sliding_window_size": 512,
+    "swa_head_dim": 128,
+    "swa_num_attention_heads": 8,
+    "swa_num_key_value_heads": 4,
+    "unpadded_vocab_size": 200058,
+    "use_embed_norm": true,
+    "use_gate_bias": true,
+    "use_global_scale": true,
+    "use_sconv": true,
+    "vocab_size": 201024
+  },
+  "transformers_version": "5.15.0.dev0",
+  "use_cache": false,
+  "vision_config": {
+    "decoder_dmodel": 1024,
+    "dtype": "float32",
+    "hidden_size": 256,
+    "initializer_range": 0.02,
+    "model_type": "inkling_vision",
+    "n_channels": 3,
+    "n_layers": 1,
+    "num_attention_heads": 4,
+    "num_channels": 3,
+    "patch_size": 40,
+    "rms_norm_eps": 1e-06,
+    "temporal_patch_size": 2,
+    "text_hidden_size": 1024,
+    "use_vision_norm": true,
+    "vision_encoder_type": "hmlp"
+  }
+}

checkpoint-500/generation_config.json ADDED Viewed

	@@ -0,0 +1,5 @@

+{
+  "_from_model_config": true,
+  "eos_token_id": 200006,
+  "transformers_version": "5.15.0.dev0"
+}

checkpoint-500/model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8f5c357e4c8595c6824b85cfd3bfd638f9e6146820860541dfbc275ca2725864
+size 2575851840

checkpoint-500/optimizer.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dd69e4ce997e8f693929afde9c6686a544ecc521f2aed14f0689c8b4d1a74d05
+size 5062717107

checkpoint-500/rng_state.pth ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:15ee528056f07c19317245e7773425308d6714ebe9e8a160053cedead6bf6b4d
+size 14645

checkpoint-500/scheduler.pt ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a3ef6ba76f01a0d88ff85f316c9e95c0097570b073585fb3e4d15f6283c5a23c
+size 1465

checkpoint-500/trainer_state.json ADDED Viewed

	@@ -0,0 +1,391 @@

+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 250.0,
+  "eval_steps": 500,
+  "global_step": 500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.5,
+      "grad_norm": 0.679200291633606,
+      "learning_rate": 0.0005,
+      "loss": 12.211211204528809,
+      "step": 1
+    },
+    {
+      "epoch": 5.0,
+      "grad_norm": 0.23044489324092865,
+      "learning_rate": 0.00049775,
+      "loss": 11.982609219021267,
+      "step": 10
+    },
+    {
+      "epoch": 10.0,
+      "grad_norm": 0.1907351166009903,
+      "learning_rate": 0.00049525,
+      "loss": 11.555101013183593,
+      "step": 20
+    },
+    {
+      "epoch": 15.0,
+      "grad_norm": 0.19268812239170074,
+      "learning_rate": 0.00049275,
+      "loss": 11.236198425292969,
+      "step": 30
+    },
+    {
+      "epoch": 20.0,
+      "grad_norm": 0.17933021485805511,
+      "learning_rate": 0.00049025,
+      "loss": 10.943867492675782,
+      "step": 40
+    },
+    {
+      "epoch": 25.0,
+      "grad_norm": 0.1976073980331421,
+      "learning_rate": 0.00048775,
+      "loss": 10.657130432128906,
+      "step": 50
+    },
+    {
+      "epoch": 30.0,
+      "grad_norm": 0.18433284759521484,
+      "learning_rate": 0.00048525,
+      "loss": 10.351960754394531,
+      "step": 60
+    },
+    {
+      "epoch": 35.0,
+      "grad_norm": 0.1888311803340912,
+      "learning_rate": 0.00048275,
+      "loss": 10.020181274414062,
+      "step": 70
+    },
+    {
+      "epoch": 40.0,
+      "grad_norm": 0.19162487983703613,
+      "learning_rate": 0.00048025000000000005,
+      "loss": 9.66613540649414,
+      "step": 80
+    },
+    {
+      "epoch": 45.0,
+      "grad_norm": 0.19730256497859955,
+      "learning_rate": 0.00047775000000000004,
+      "loss": 9.297232818603515,
+      "step": 90
+    },
+    {
+      "epoch": 50.0,
+      "grad_norm": 0.2213529646396637,
+      "learning_rate": 0.00047525000000000003,
+      "loss": 8.914820861816406,
+      "step": 100
+    },
+    {
+      "epoch": 55.0,
+      "grad_norm": 0.2195209115743637,
+      "learning_rate": 0.00047275,
+      "loss": 8.521177673339844,
+      "step": 110
+    },
+    {
+      "epoch": 60.0,
+      "grad_norm": 0.2313990592956543,
+      "learning_rate": 0.00047025,
+      "loss": 8.118896484375,
+      "step": 120
+    },
+    {
+      "epoch": 65.0,
+      "grad_norm": 0.23104608058929443,
+      "learning_rate": 0.00046775,
+      "loss": 7.713725280761719,
+      "step": 130
+    },
+    {
+      "epoch": 70.0,
+      "grad_norm": 0.2376953512430191,
+      "learning_rate": 0.00046525,
+      "loss": 7.309984588623047,
+      "step": 140
+    },
+    {
+      "epoch": 75.0,
+      "grad_norm": 0.24835970997810364,
+      "learning_rate": 0.00046275,
+      "loss": 6.910225677490234,
+      "step": 150
+    },
+    {
+      "epoch": 80.0,
+      "grad_norm": 0.2480894923210144,
+      "learning_rate": 0.00046025,
+      "loss": 6.521015167236328,
+      "step": 160
+    },
+    {
+      "epoch": 85.0,
+      "grad_norm": 0.2478545904159546,
+      "learning_rate": 0.00045775,
+      "loss": 6.150957870483398,
+      "step": 170
+    },
+    {
+      "epoch": 90.0,
+      "grad_norm": 0.23328417539596558,
+      "learning_rate": 0.00045525,
+      "loss": 5.809443664550781,
+      "step": 180
+    },
+    {
+      "epoch": 95.0,
+      "grad_norm": 0.23083385825157166,
+      "learning_rate": 0.00045275,
+      "loss": 5.494760131835937,
+      "step": 190
+    },
+    {
+      "epoch": 100.0,
+      "grad_norm": 0.2354152798652649,
+      "learning_rate": 0.00045024999999999997,
+      "loss": 5.214201736450195,
+      "step": 200
+    },
+    {
+      "epoch": 105.0,
+      "grad_norm": 0.2212478518486023,
+      "learning_rate": 0.00044775,
+      "loss": 4.965381622314453,
+      "step": 210
+    },
+    {
+      "epoch": 110.0,
+      "grad_norm": 0.22108030319213867,
+      "learning_rate": 0.00044525,
+      "loss": 4.736624145507813,
+      "step": 220
+    },
+    {
+      "epoch": 115.0,
+      "grad_norm": 0.2202872782945633,
+      "learning_rate": 0.00044275,
+      "loss": 4.5371543884277346,
+      "step": 230
+    },
+    {
+      "epoch": 120.0,
+      "grad_norm": 0.21908171474933624,
+      "learning_rate": 0.00044025,
+      "loss": 4.3416389465332035,
+      "step": 240
+    },
+    {
+      "epoch": 125.0,
+      "grad_norm": 0.2539515495300293,
+      "learning_rate": 0.00043775,
+      "loss": 4.155947875976563,
+      "step": 250
+    },
+    {
+      "epoch": 130.0,
+      "grad_norm": 0.2240767925977707,
+      "learning_rate": 0.00043525000000000004,
+      "loss": 3.977610397338867,
+      "step": 260
+    },
+    {
+      "epoch": 135.0,
+      "grad_norm": 0.20758213102817535,
+      "learning_rate": 0.00043275000000000003,
+      "loss": 3.791416549682617,
+      "step": 270
+    },
+    {
+      "epoch": 140.0,
+      "grad_norm": 0.21930797398090363,
+      "learning_rate": 0.00043025,
+      "loss": 3.613859939575195,
+      "step": 280
+    },
+    {
+      "epoch": 145.0,
+      "grad_norm": 0.20898985862731934,
+      "learning_rate": 0.00042775,
+      "loss": 3.4412445068359374,
+      "step": 290
+    },
+    {
+      "epoch": 150.0,
+      "grad_norm": 0.21812506020069122,
+      "learning_rate": 0.00042525,
+      "loss": 3.2678699493408203,
+      "step": 300
+    },
+    {
+      "epoch": 155.0,
+      "grad_norm": 0.20785079896450043,
+      "learning_rate": 0.00042275,
+      "loss": 3.101655387878418,
+      "step": 310
+    },
+    {
+      "epoch": 160.0,
+      "grad_norm": 0.20621119439601898,
+      "learning_rate": 0.00042025,
+      "loss": 2.9381855010986326,
+      "step": 320
+    },
+    {
+      "epoch": 165.0,
+      "grad_norm": 0.2516527771949768,
+      "learning_rate": 0.00041775000000000004,
+      "loss": 2.779329299926758,
+      "step": 330
+    },
+    {
+      "epoch": 170.0,
+      "grad_norm": 0.2031988501548767,
+      "learning_rate": 0.00041525000000000004,
+      "loss": 2.6272861480712892,
+      "step": 340
+    },
+    {
+      "epoch": 175.0,
+      "grad_norm": 0.19911770522594452,
+      "learning_rate": 0.00041275000000000003,
+      "loss": 2.466817855834961,
+      "step": 350
+    },
+    {
+      "epoch": 180.0,
+      "grad_norm": 0.19864623248577118,
+      "learning_rate": 0.00041025,
+      "loss": 2.3201425552368162,
+      "step": 360
+    },
+    {
+      "epoch": 185.0,
+      "grad_norm": 0.1971900314092636,
+      "learning_rate": 0.00040775,
+      "loss": 2.173211097717285,
+      "step": 370
+    },
+    {
+      "epoch": 190.0,
+      "grad_norm": 0.19130639731884003,
+      "learning_rate": 0.00040525,
+      "loss": 2.038584518432617,
+      "step": 380
+    },
+    {
+      "epoch": 195.0,
+      "grad_norm": 0.2037324756383896,
+      "learning_rate": 0.00040275,
+      "loss": 1.9071189880371093,
+      "step": 390
+    },
+    {
+      "epoch": 200.0,
+      "grad_norm": 0.18795250356197357,
+      "learning_rate": 0.00040025,
+      "loss": 1.782277488708496,
+      "step": 400
+    },
+    {
+      "epoch": 205.0,
+      "grad_norm": 0.18380966782569885,
+      "learning_rate": 0.00039775,
+      "loss": 1.663627815246582,
+      "step": 410
+    },
+    {
+      "epoch": 210.0,
+      "grad_norm": 0.1786879301071167,
+      "learning_rate": 0.00039525,
+      "loss": 1.5451311111450194,
+      "step": 420
+    },
+    {
+      "epoch": 215.0,
+      "grad_norm": 0.17858438193798065,
+      "learning_rate": 0.00039275,
+      "loss": 1.4358247756958007,
+      "step": 430
+    },
+    {
+      "epoch": 220.0,
+      "grad_norm": 0.1655775010585785,
+      "learning_rate": 0.00039024999999999997,
+      "loss": 1.332811450958252,
+      "step": 440
+    },
+    {
+      "epoch": 225.0,
+      "grad_norm": 0.1637052446603775,
+      "learning_rate": 0.00038774999999999997,
+      "loss": 1.2332740783691407,
+      "step": 450
+    },
+    {
+      "epoch": 230.0,
+      "grad_norm": 0.16148947179317474,
+      "learning_rate": 0.00038525,
+      "loss": 1.1397903442382813,
+      "step": 460
+    },
+    {
+      "epoch": 235.0,
+      "grad_norm": 0.15098616480827332,
+      "learning_rate": 0.00038275,
+      "loss": 1.055152416229248,
+      "step": 470
+    },
+    {
+      "epoch": 240.0,
+      "grad_norm": 0.15136969089508057,
+      "learning_rate": 0.00038025,
+      "loss": 0.9760448455810546,
+      "step": 480
+    },
+    {
+      "epoch": 245.0,
+      "grad_norm": 0.1447116583585739,
+      "learning_rate": 0.00037775,
+      "loss": 0.8991727828979492,
+      "step": 490
+    },
+    {
+      "epoch": 250.0,
+      "grad_norm": 0.1393691748380661,
+      "learning_rate": 0.00037525,
+      "loss": 0.8312561988830567,
+      "step": 500
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 2000,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1000,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 1341838872576000.0,
+  "train_batch_size": 2,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-500/training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1d90df91199aaca447f28d3896ff1ea0d98043bc13563aa4dfd901be2ba56c9f
+size 5137

config.json CHANGED Viewed

@@ -31,6 +31,7 @@
     ],
     "num_nextn_predict_layers": 1
   },
   "text_config": {
     "attention_dropout": 0.0,
     "bos_token_id": 1,
@@ -125,6 +126,7 @@
     "vocab_size": 201024
   },
   "transformers_version": "5.15.0.dev0",
   "vision_config": {
     "decoder_dmodel": 1024,
     "dtype": "float32",

     ],
     "num_nextn_predict_layers": 1
   },
+  "pad_token_id": 199999,
   "text_config": {
     "attention_dropout": 0.0,
     "bos_token_id": 1,
     "vocab_size": 201024
   },
   "transformers_version": "5.15.0.dev0",
+  "use_cache": false,
   "vision_config": {
     "decoder_dmodel": 1024,
     "dtype": "float32",

model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:76205f9e4f2db44e06b8d8bba1983b74fef807e99f1d096f8eb3f411cbd64528
 size 2575851840

 version https://git-lfs.github.com/spec/v1
+oid sha256:93c83f60ebcecddb22b37719fc629fa0fb5a4df3319f76ad83006ad85f3c1f4e
 size 2575851840

tokenizer.json CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c470271fb7649eef922e660e757b5f076e2a94adc20df7fad7df85c46bc48eac
-size 27875796

 version https://git-lfs.github.com/spec/v1
+oid sha256:1a346ede0ded2d4b1e1d59547c8887d2d2c4bb94c13c9021e2b405a0cc3a7c89
+size 27876074

tokenizer_config.json CHANGED Viewed

@@ -15,8 +15,9 @@
   "content_xml": "<|content_xml|>",
   "end_message": "<|end_message|>",
   "endoftext": "<|endoftext|>",
   "fix_mistral_regex": false,
-  "is_local": false,
   "local_files_only": false,
   "message_model": "<|message_model|>",
   "message_system": "<|message_system|>",
@@ -43,6 +44,7 @@
     "message_tool": "<|message_tool|>",
     "message_user": "<|message_user|>"
   },
   "processor_class": "InklingProcessor",
   "tokenizer_class": "TokenizersBackend"
 }

   "content_xml": "<|content_xml|>",
   "end_message": "<|end_message|>",
   "endoftext": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
   "fix_mistral_regex": false,
+  "is_local": true,
   "local_files_only": false,
   "message_model": "<|message_model|>",
   "message_system": "<|message_system|>",
     "message_tool": "<|message_tool|>",
     "message_user": "<|message_user|>"
   },
+  "pad_token": "<|endoftext|>",
   "processor_class": "InklingProcessor",
   "tokenizer_class": "TokenizersBackend"
 }

training_args.bin ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1d90df91199aaca447f28d3896ff1ea0d98043bc13563aa4dfd901be2ba56c9f
+size 5137