Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/ornith_35b_eval.png filter=lfs diff=lfs merge=lfs -text
+assets/ornith_397b_eval.png filter=lfs diff=lfs merge=lfs -text
+assets/ornith_logo.png filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,254 @@
+---
+library_name: transformers
+license: mit
+license_link: https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B/blob/main/LICENSE
+pipeline_tag: text-generation
+base_model:
+- deepreinforce-ai/Ornith-1.0-35B
+- shisa-ai/Ornith-1.0-35B-FP8-BLOCK
+- Qwen/Qwen3.6-35B-A3B
+tags:
+- fp8
+- fp8-block
+- compressed-tensors
+- llm-compressor
+- qwen3.5
+- qwen3.6
+- moe
+- mtp
+- speculative-decoding
+- vllm
+- code
+- reasoning
+---
+
+<img width="600px" src="assets/ornith_logo.png">
+
+# Ornith-1.0-35B-FP8-BLOCK-MTP
+
+This is the official MTP-enabled derivative of
+[`shisa-ai/Ornith-1.0-35B-FP8-BLOCK`](https://huggingface.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK).
+It keeps the original FP8_BLOCK / `compressed-tensors` Ornith base weights and
+adds a BF16 Qwen3.6 MTP head in `model-mtp.safetensors`.
+
+This artifact was built internally as a zero-training Qwen3.6 MTP graft, but the
+public upload name is:
+[`shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP`](https://huggingface.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP).
+Based on our matched local vLLM tests, this is the recommended Ornith 35B MTP
+checkpoint for throughput. The best measured row used
+`num_speculative_tokens=3` and reached `246.522` output tok/s on our validation
+workload, a `+21.4%` improvement over the no-spec baseline.
+
+The MTP recipe was adapted from
+[`protoLabsAI/Ornith-1.0-9B-MTP`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP),
+which demonstrated grafting a same-family Qwen MTP head into an Ornith
+checkpoint and optionally KL-distilling only the MTP head. Thanks to
+[`protoLabsAI`](https://huggingface.co/protoLabsAI)
+for the MTP graft/distillation technique and base recipe.
+
+## What Changed
+
+- Base checkpoint: [`shisa-ai/Ornith-1.0-35B-FP8-BLOCK`](https://huggingface.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK)
+- Donor MTP checkpoint: `Qwen/Qwen3.6-35B-A3B`
+- Added shard: `model-mtp.safetensors`
+- Added tensors: `19` top-level `mtp.*` tensors
+- MTP dtype: BF16
+- Base weights: unchanged FP8_BLOCK / `compressed-tensors`
+- Training: none; this is a direct MTP tensor graft
+
+No private training corpus is needed for this checkpoint because it is not
+distilled. The graft copies the donor `mtp.*` tensors into the Ornith FP8_BLOCK
+checkpoint and updates the safetensors index.
+
+## Important Result Summary
+
+This is the strongest Ornith 35B MTP checkpoint from our local serving tests. It
+also beat the one-epoch Qwen3.6 KL-distilled derivative in matched vLLM serving,
+even though that distilled derivative had a slightly better offline KL proxy.
+
+Recommended starting point:
+
+- Use `num_speculative_tokens=3` when optimizing for output throughput.
+- Use `num_speculative_tokens=1` or no speculative decoding if your workload is
+  more sensitive to inter-token latency or acceptance stability.
+- Re-benchmark on your own traffic before making it a production default.
+
+## Local Benchmark Methodology
+
+Hardware:
+
+- One `NVIDIA RTX PRO 6000 Blackwell Workstation Edition`
+- Single-GPU serving on GPU0 only
+
+Runtime:
+
+- vLLM `0.23.0`
+- FlashInfer attention backend
+- FP8 KV cache
+- `compressed-tensors` quantization
+- No LMCache for the benchmark rows below
+- `MAX_MODEL_LEN=32768`
+- `MAX_NUM_SEQS=16`
+- `MAX_BATCHED_TOKENS=32768`
+- `MAX_CUDAGRAPH_CAPTURE_SIZE=16`
+- `GPU_MEMORY_UTIL=0.95`
+
+Workload:
+
+- Private custom validation benchmark derived from local code/agentic prompts
+- `64` requests
+- `63,327` total input tokens
+- `16,384` generated tokens (`256` per request)
+- `max_concurrency=1`
+- `request_rate=inf`
+- `temperature=0.6`, `top_p=0.95`, `top_k=20`
+- `ignore_eos`
+
+The private benchmark data is not uploaded. The benchmark used vLLM's
+`bench serve --dataset-name custom` path. To reproduce the command shape with
+your own non-private data, use a JSONL custom dataset with `prompt` and
+`output_tokens` fields and run a command like:
+
+```bash
+CUDA_VISIBLE_DEVICES=0 vllm bench serve \
+  --backend vllm \
+  --base-url http://127.0.0.1:8000 \
+  --endpoint /v1/completions \
+  --model ornith-35b-fp8-block-mtp \
+  --tokenizer shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP \
+  --dataset-name custom \
+  --dataset-path /path/to/custom-prompts.jsonl \
+  --skip-chat-template \
+  --disable-shuffle \
+  --no-oversample \
+  --num-prompts 64 \
+  --custom-output-len -1 \
+  --max-concurrency 1 \
+  --request-rate inf \
+  --temperature 0.6 \
+  --top-p 0.95 \
+  --top-k 20 \
+  --ignore-eos
+```
+
+## Local Results
+
+Matched c=1 validation-prompt serving results:
+
+| Variant | MTP tokens | Output tok/s | Delta vs baseline | Median TTFT ms | Median TPOT ms | Median ITL ms | Acceptance |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| Baseline no-spec | 0 | 203.142 | - | 51.722 | 4.647 | 4.648 | - |
+| Official MTP, Qwen3.6 graft | 1 | 221.032 | +8.8% | 60.311 | 4.209 | 7.713 | 85.82% |
+| Official MTP, Qwen3.6 graft | 3 | 246.522 | +21.4% | 65.482 | 3.571 | 10.739 | 66.98% |
+
+Comparison against the companion KL-distilled artifact:
+
+| Variant | MTP tokens | Output tok/s | Delta vs baseline | Median TTFT ms | Median TPOT ms | Median ITL ms | Acceptance |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| Official MTP, Qwen3.6 graft | 3 | 246.522 | +21.4% | 65.482 | 3.571 | 10.739 | 66.98% |
+| Qwen3.6 KL-distill | 3 | 237.581 | +17.0% | 66.581 | 3.625 | 10.861 | 67.11% |
+
+The distill checkpoint is planned as
+[`shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill`](https://huggingface.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP-qwen36-distill).
+It improved the offline proxy, but it did not beat this official MTP checkpoint
+in matched serving throughput.
+
+Offline proxy on the same validation split:
+
+- Official MTP, Qwen3.6 graft: distribution-overlap acceptance proxy `0.841`,
+  mean KL `0.3162`
+- Qwen3.6 KL-distill: distribution-overlap acceptance proxy `0.849`, mean KL
+  `0.2967`
+
+### Internal Alternatives Tested
+
+We also tested a Qwen3.5 donor graft, but it is not planned for upload. The
+Qwen3.5 rows were from an earlier MTP1-only ShareGPT sweep, so they are useful
+as donor-selection context rather than a direct replacement for the validation
+table above.
+
+| Variant | c | Output tok/s | Delta vs baseline | Median TPOT ms | Acceptance |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Baseline no-spec | 1 | 200.340 | - | 4.733 | - |
+| Qwen3.6 MTP1 graft | 1 | 211.736 | +5.7% | 4.296 | 80.63% |
+| Qwen3.5 MTP1 graft | 1 | 213.091 | +6.4% | 4.402 | 77.90% |
+| Baseline no-spec | 4 | 500.689 | - | 6.955 | - |
+| Qwen3.6 MTP1 graft | 4 | 528.654 | +5.6% | 6.443 | 81.42% |
+| Qwen3.5 MTP1 graft | 4 | 531.581 | +6.1% | 6.554 | 76.65% |
+
+The Qwen3.5 graft was marginally faster in those MTP1 rows, but it had lower
+acceptance. The later Qwen3.6 validation sweep at MTP3 produced the best retained
+throughput result, so Qwen3.6 is the donor used for the official upload.
+
+## vLLM Usage
+
+MTP serving requires a vLLM build that supports Qwen3.5 MoE MTP checkpoints.
+The local runs used vLLM `0.23.0`.
+
+```bash
+vllm serve shisa-ai/Ornith-1.0-35B-FP8-BLOCK-MTP \
+  --served-model-name ornith-35b-fp8-block-mtp \
+  --trust-remote-code \
+  --quantization compressed-tensors \
+  --language-model-only \
+  --max-model-len 32768 \
+  --gpu-memory-utilization 0.95 \
+  --max-num-seqs 16 \
+  --max-num-batched-tokens 32768 \
+  --max-cudagraph-capture-size 16 \
+  --attention-backend flashinfer \
+  --kv-cache-dtype fp8 \
+  --generation-config vllm \
+  --enable-prefix-caching \
+  --enable-auto-tool-choice \
+  --tool-call-parser qwen3_xml \
+  --reasoning-parser qwen3 \
+  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
+```
+
+`num_speculative_tokens=3` gave the best local output throughput for this
+checkpoint, but it also reduced acceptance to about `67%` and increased
+inter-token latency relative to no-spec serving. Re-benchmark on your hardware
+and workload before using it as a default.
+
+## Quantization Summary
+
+The base checkpoint is unchanged from
+[`shisa-ai/Ornith-1.0-35B-FP8-BLOCK`](https://huggingface.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK):
+
+- Source model: [`deepreinforce-ai/Ornith-1.0-35B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B)
+- Quantization tool: `llm-compressor` model-free PTQ
+- Quantization format: `compressed-tensors`
+- Scheme: `FP8_BLOCK`
+- Calibration data: none; this is data-free/model-free PTQ
+- Weight quantization: static FP8, symmetric, block strategy, `128x128` blocks
+- Activation quantization: dynamic FP8, symmetric, group strategy, group size `128`
+- Target modules: `Linear`
+- `compressed-tensors` metadata version recorded in `config.json`: `0.15.1.a20260406`
+
+The quantization ignore list includes `re:^mtp.*`, so the grafted MTP head
+remains BF16.
+
+## License and Attribution
+
+The source Ornith model is MIT licensed. This derivative keeps the source
+license metadata and links to the upstream license file.
+
+Attribution:
+
+- Source model: [`deepreinforce-ai/Ornith-1.0-35B`](https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B)
+- FP8_BLOCK base: [`shisa-ai/Ornith-1.0-35B-FP8-BLOCK`](https://huggingface.co/shisa-ai/Ornith-1.0-35B-FP8-BLOCK)
+- MTP donor: [`Qwen/Qwen3.6-35B-A3B`](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)
+- MTP graft/distill recipe inspiration:
+  [`protoLabsAI/Ornith-1.0-9B-MTP`](https://huggingface.co/protoLabsAI/Ornith-1.0-9B-MTP)
+
+If you use the source model, cite the original Ornith release:
+
+```bibtex
+@misc{ornith-35b,
+    title = {{Ornith-1.0-35B}: Agentic Coding, Open to All},
+    url = {https://deep-reinforce.com/ornith_1_0.html},
+    author = {{DeepReinforce Team}},
+    year = {2026}
+}
+```

chat_template.jinja

@@ -0,0 +1,150 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,184 @@
+{
+  "architectures": [
+    "Qwen3_5MoeForConditionalGeneration"
+  ],
+  "bos_token_id": null,
+  "dtype": "bfloat16",
+  "eos_token_id": 248046,
+  "hidden_size": 2048,
+  "image_token_id": 248056,
+  "model_type": "qwen3_5_moe",
+  "pad_token_id": 248044,
+  "quantization_config": {
+    "config_groups": {
+      "FP8_BLOCK": {
+        "format": "float-quantized",
+        "input_activations": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": true,
+          "group_size": 128,
+          "num_bits": 8,
+          "observer": null,
+          "observer_kwargs": {},
+          "scale_dtype": null,
+          "strategy": "group",
+          "symmetric": true,
+          "type": "float",
+          "zp_dtype": null
+        },
+        "output_activations": null,
+        "targets": [
+          "Linear"
+        ],
+        "weights": {
+          "actorder": null,
+          "block_structure": [
+            128,
+            128
+          ],
+          "dynamic": false,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": "memoryless_minmax",
+          "observer_kwargs": {},
+          "scale_dtype": null,
+          "strategy": "block",
+          "symmetric": true,
+          "type": "float",
+          "zp_dtype": null
+        }
+      }
+    },
+    "format": "float-quantized",
+    "global_compression_ratio": null,
+    "ignore": [
+      "re:.*lm_head$",
+      "re:.*embed_tokens$",
+      "re:.*visual.*",
+      "re:.*mlp\\.gate$",
+      "re:.*mlp\\.shared_expert_gate$",
+      "re:.*linear_attn.*",
+      "re:^mtp.*"
+    ],
+    "kv_cache_scheme": null,
+    "quant_method": "compressed-tensors",
+    "quantization_status": "compressed",
+    "transform_config": {},
+    "version": "0.15.1.a20260406"
+  },
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_output_gate": true,
+    "bos_token_id": 248044,
+    "dtype": "bfloat16",
+    "eos_token_id": 248044,
+    "full_attention_interval": 4,
+    "head_dim": 256,
+    "hidden_act": "silu",
+    "hidden_size": 2048,
+    "initializer_range": 0.02,
+    "layer_types": [
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention"
+    ],
+    "linear_conv_kernel_dim": 4,
+    "linear_key_head_dim": 128,
+    "linear_num_key_heads": 16,
+    "linear_num_value_heads": 32,
+    "linear_value_head_dim": 128,
+    "mamba_ssm_dtype": "float32",
+    "max_position_embeddings": 262144,
+    "model_type": "qwen3_5_moe_text",
+    "moe_intermediate_size": 512,
+    "mtp_num_hidden_layers": 1,
+    "mtp_use_dedicated_embeddings": false,
+    "num_attention_heads": 16,
+    "num_experts": 256,
+    "num_experts_per_tok": 8,
+    "num_hidden_layers": 40,
+    "num_key_value_heads": 2,
+    "output_router_logits": false,
+    "pad_token_id": 248044,
+    "partial_rotary_factor": 0.25,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        11,
+        11,
+        10
+      ],
+      "partial_rotary_factor": 0.25,
+      "rope_theta": 10000000,
+      "rope_type": "default"
+    },
+    "router_aux_loss_coef": 0.0,
+    "shared_expert_intermediate_size": 512,
+    "tie_word_embeddings": false,
+    "use_cache": false,
+    "vocab_size": 248320
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.8.1",
+  "use_cache": false,
+  "video_token_id": 248057,
+  "vision_config": {
+    "deepstack_visual_indexes": [],
+    "depth": 27,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 4304,
+    "model_type": "qwen3_5_moe_vision",
+    "num_heads": 16,
+    "num_position_embeddings": 2304,
+    "out_hidden_size": 2048,
+    "patch_size": 16,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 248054,
+  "vision_start_token_id": 248053
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 248044,
+  "do_sample": true,
+  "eos_token_id": [
+    248046,
+    248044
+  ],
+  "pad_token_id": 248044,
+  "temperature": 1.0,
+  "top_k": 20,
+  "top_p": 0.95,
+  "transformers_version": "5.8.1"
+}

model-00001-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c6acc6ee4849edd4144463fdc4af79ef44806508061ee4235cd3686829843997
+size 3247489288

model-00002-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:225dc3ac6d6a2a49c522e9964f896ca4106282fc7e2b0077ff7ffde7f5550d13
+size 2323081312

model-00003-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f99e18083610d798ae8e954e44cbf98f6ed56863f77842a60b43a79d7b8ebbb
+size 2564352640

model-00004-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ab5b34d6ecedaa09554598718b5f0def2401a5a14ec7e6955a6724d99d94b339
+size 2050333072

model-00005-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce1dfff8f0b5361f5b7c4373c4a90126964d5337659d9356d198ed7707a5f51e
+size 2323088704

model-00006-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2853acef01bd1fbcf925ab972888e3c8b4b00433d2c2324d0828c7adbd4f2df5
+size 2564357296

model-00007-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90b2948a6e7484e4eae7395dffc70ec6dfa0b383c3513fc5f0a3d5a3ac1d1401
+size 2050336704

model-00008-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:373bfacbd888b63fa7731ca9e45f9df15888ef40d2dc42be667ee54bd6213ff3
+size 2323085472

model-00009-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3984e7317bb31ba9dc8f716696f533847ece00ba59eb5a8749fb531f3205a59
+size 2564357296

model-00010-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a18765cf02a02d22a3f0eb646b485810c79f80ed8ff40e75fdd7a9359bd681f2
+size 2050336704

model-00011-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f78937cc1446ccdb4d8936add82c5d0766a203837d39eb351c856fbb19a7b7e3
+size 2323085472

model-00012-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1913e8003d0989325096afdc492b7b57b76216cbf00fae74d5c00f60e3b4f382
+size 2564357296

model-00013-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:583b508b8b3bf012b1fa1e27b8e32b9e85dac36e278a85ad5c4a18eaaaa28c9d
+size 2050336704

model-00014-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e27f78fc0e8e97a3b75d6b7d5f3c2fa07027aed9900dbce27cd6a197430ee478
+size 2323085472

model-00015-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f7a8bcbbc3c916063f63f627752ac69e72a0469a3552af4ed72f7d2e33a3b991
+size 2564357296

model-00016-of-00016.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:54ce9cc09bbff21d75ddbec413be4452bf4e32ba02427e99e96ceaff8e505aa0
+size 1730271792

model-mtp.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62e6f4b910fd5ff7594e85863fe28cfd0177890c8251655ee9bcbc35a208d78b
+size 1689283688

preprocessor_config.json

@@ -0,0 +1,21 @@
+{
+    "size": {
+        "longest_edge": 16777216,
+        "shortest_edge": 65536
+    },
+    "patch_size": 16,
+    "temporal_patch_size": 2,
+    "merge_size": 2,
+    "image_mean": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "image_std": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "processor_class": "Qwen3VLProcessor",
+    "image_processor_type": "Qwen2VLImageProcessorFast"
+}

processor_config.json

@@ -0,0 +1,60 @@
+{
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_processor_type": "Qwen2VLImageProcessor",
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "merge_size": 2,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "longest_edge": 16777216,
+      "shortest_edge": 65536
+    },
+    "temporal_patch_size": 2
+  },
+  "processor_class": "Qwen3VLProcessor",
+  "video_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "fps": 2,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_frames": 768,
+    "merge_size": 2,
+    "min_frames": 4,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "size": {
+      "longest_edge": 25165824,
+      "shortest_edge": 4096
+    },
+    "temporal_patch_size": 2,
+    "video_processor_type": "Qwen3VLVideoProcessor"
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
+size 19989325

tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "processor_class": "Qwen3VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}

video_preprocessor_config.json

@@ -0,0 +1,21 @@
+{
+    "size": {
+        "longest_edge": 25165824,
+        "shortest_edge": 4096
+    },
+    "patch_size": 16,
+    "temporal_patch_size": 2,
+    "merge_size": 2,
+    "image_mean": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "image_std": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "processor_class": "Qwen3VLProcessor",
+    "video_processor_type": "Qwen3VLVideoProcessor"
+}