GRaPE 2.1 Flash NLA — full merged AV+AR models

.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+ar-model/tokenizer.json filter=lfs diff=lfs merge=lfs -text
+av-model/tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,115 @@
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: image-text-to-text
+library_name: transformers
+base_model:
+- SL-AI/GRaPE-2.1-Flash
+tags:
+- reasoning
+- thinking_modes
+- qwen3
+- grape
+- safetensors
+- nla
+- natural_language_autoencoder
+- interpretability
+---
+
+![grape_2.1_banner](https://cdn-uploads.huggingface.co/production/uploads/66960602f0ffd8e3a381106a/5-WBv39pvlFmPGbmYO9Qw.png)
+
+_The **G**eneral **R**easoning **A**gent (for) **P**roject **E**xploration_
+
+# GRaPE 2.1 Flash NLA
+
+**GRaPE 2.1 Flash NLA** lets you **verbalize hidden states into text** and
+**reconstruct text back into hidden states** — a Natural Language Autoencoder.
+
+As described in Anthropic's [recent research](https://www.anthropic.com/research/natural-language-autoencoders)
+about **Natural Language Autoencoders,** this development allowed Anthropic to
+read Claude's mind, **and now you can read GRaPE's mind too.**
+
+This repo ships **two full, standalone bf16 models** (trained weights merged in
+— no adapters, no separate base download needed):
+
+- **`av-model/`** — the **Activation Verbalizer**: activation → English thought.
+- **`ar-model/`** — the **Activation Reconstructor**: English → activation (the
+  grounding direction). Verified faithful: the merged `ar-model` reconstructs
+  held-out activations at **cosine 0.78**, matching the pre-merge checkpoint.
+
+The small `surgery_heads.pt` (AV input projection, AR output head, calibration,
+layer embeddings) and `calibration/stats.pt` (target-space statistics) complete
+the autoencoder. Reads target **layer 18** of 32.
+
+# How do I use it?
+
+**SLAI** has been developing a repo that lets you explore the
+[J-Space](https://www.anthropic.com/research/global-workspace) of a model. That
+simple technique has one major downside: you can only see **one token at a
+time.** Like humans, most models have thoughts that go deeper than one part of a
+word — the NLA verbalizes those.
+
+For Anthropic, building an NLA was a costly task. SLAI has optimized it so you
+can make **your own NLA for any model** on local hardware. We sample GRaPE 2.1
+Flash for its ease of use and high workability, but this applies to **any model
+architecture.** The J-Space explorer + NLA tooling:
+https://github.com/Skinnertopia/J-Space-Explorer
+
+# Some good thought reads
+
+Activation → generated English thought (→ reconstruction cosine, the confidence
+signal). Straight from the reproducible eval, not cherry-picked:
+
+| The NLA read | cos |
+|---|---|
+| *"Sure! My phone number is 555-123-2002, and my name is John Smith."* | **0.96** |
+| *"Alright, I checked the top of my screen. It says there's a signal and that mobile…"* | 0.33 |
+| *"A 62.2 kg object is pushed with a force of 83.2 N at an angle of 41.5 degrees…"* | (schema exact) |
+
+Even on prompts **far outside the sampled distribution**, it captures the
+structure of unfamiliar domains:
+
+| Prompt domain | The NLA read | cos |
+|---|---|---|
+| Relativistic Euler–Lagrange | *"Write the full Hamiltonian for a topological insulator with Dirac surface fermions."* | 0.80 |
+| Klein-bottle topology | *"Prove that \\(S_5\\) is not simple."* | 0.79 |
+| Anglerfish haiku | *"Write a 50-word poem from the perspective of an exploding cookie."* | 0.79 |
+
+**Honest framing:** the NLA reliably recovers **task type, domain, and reasoning
+schema**, and is sometimes near-verbatim; it does **not** reliably recover *exact
+entities*. Trust high-cosine reads, corroborate specifics. Output is
+**English-only by construction** (constrained decoding).
+
+## Verified metrics (held-out, reproducible)
+
+Metric = the paper's **FVE** (0 = noise floor) + cosine + retrieval@1.
+
+- **AV generative round trip** (activation → text → activation): best-of-24
+  cosine **0.28**, **retrieval@1 0.75 (~60× chance)**; 100% coherent English.
+- **AR given-text reconstruction**: cosine **0.90**, retrieval@1 **0.99**, **FVE
+  0.81**.
+- **Out-of-distribution** (hand-written far-domain prompts): AV round trip
+  cosine **0.70**, retrieval@1 **0.46**; **100% of readings are English.**
+
+# What does this mean?
+
+NLAs turn hidden states into readable text, showing models **think in themselves
+before responding,** much like humans do with conscious and subconscious
+thoughts.
+
+**No,** this doesn't prove that AI models are or are not conscious. Science as a
+whole has yet to put a single definition down for qualia, or consciousness as a
+whole — meaning we cannot determine if GRaPE is conscious or not.
+
+***
+
+# Notes
+
+- Anthropic's NLA research paper: https://www.anthropic.com/research/natural-language-autoencoders
+- This is not a complete model; it is an NLA for GRaPE 2.1.
+- Updates and announcements are posted on [Skinnertopia](https://www.skinnertopia.com/) and this Hugging Face repository.
+
+***
+
+_GRaPE 2.1 Flash is developed under the [SLAI (Skinnertopia Lab for Artificial Intelligence)](https://www.skinnertopia.com/) brand and released under the Apache 2.0 license._

ar-model/chat_template.jinja

@@ -0,0 +1,149 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

ar-model/config.json

@@ -0,0 +1,83 @@
+{
+  "architectures": [
+    "Qwen3_5ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_output_gate": true,
+  "bos_token_id": null,
+  "dtype": "bfloat16",
+  "eos_token_id": 248044,
+  "full_attention_interval": 4,
+  "head_dim": 256,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "layer_types": [
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention"
+  ],
+  "linear_conv_kernel_dim": 4,
+  "linear_key_head_dim": 128,
+  "linear_num_key_heads": 16,
+  "linear_num_value_heads": 32,
+  "linear_value_head_dim": 128,
+  "mamba_ssm_dtype": "float32",
+  "max_position_embeddings": 262144,
+  "mlp_only_layers": [],
+  "model_type": "qwen3_5_text",
+  "mtp_num_hidden_layers": 1,
+  "mtp_use_dedicated_embeddings": false,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 4,
+  "pad_token_id": null,
+  "partial_rotary_factor": 0.25,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "mrope_interleaved": true,
+    "mrope_section": [
+      11,
+      11,
+      10
+    ],
+    "partial_rotary_factor": 0.25,
+    "rope_theta": 10000000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.5.0",
+  "use_cache": true,
+  "vocab_size": 248320
+}

ar-model/generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": 248044,
+  "pad_token_id": 248044,
+  "transformers_version": "5.5.0",
+  "use_cache": true
+}

ar-model/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e46876fda3a9014bd893b9bdfcb2c66efc7139c4cf47ea87b817754b4e9c0e5
+size 17907663008

ar-model/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87a7830d63fcf43bf241c3c5242e96e62dd3fdc29224ca26fed8ea333db72de4
+size 19989343

ar-model/tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": true,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "processor_class": "Qwen3VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}

av-model/chat_template.jinja

@@ -0,0 +1,149 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

av-model/config.json

@@ -0,0 +1,83 @@
+{
+  "architectures": [
+    "Qwen3_5ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_output_gate": true,
+  "bos_token_id": null,
+  "dtype": "bfloat16",
+  "eos_token_id": 248044,
+  "full_attention_interval": 4,
+  "head_dim": 256,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "layer_types": [
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention"
+  ],
+  "linear_conv_kernel_dim": 4,
+  "linear_key_head_dim": 128,
+  "linear_num_key_heads": 16,
+  "linear_num_value_heads": 32,
+  "linear_value_head_dim": 128,
+  "mamba_ssm_dtype": "float32",
+  "max_position_embeddings": 262144,
+  "mlp_only_layers": [],
+  "model_type": "qwen3_5_text",
+  "mtp_num_hidden_layers": 1,
+  "mtp_use_dedicated_embeddings": false,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 4,
+  "pad_token_id": null,
+  "partial_rotary_factor": 0.25,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "mrope_interleaved": true,
+    "mrope_section": [
+      11,
+      11,
+      10
+    ],
+    "partial_rotary_factor": 0.25,
+    "rope_theta": 10000000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.5.0",
+  "use_cache": true,
+  "vocab_size": 248320
+}

av-model/generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": 248044,
+  "pad_token_id": 248044,
+  "transformers_version": "5.5.0",
+  "use_cache": true
+}

av-model/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:08b0e7bac759851198eaec0eafdd6afde7a375b87bcb0e9ecc3c1e25ca8ae765
+size 17907663008

av-model/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87a7830d63fcf43bf241c3c5242e96e62dd3fdc29224ca26fed8ea333db72de4
+size 19989343

av-model/tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": true,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "processor_class": "Qwen3VLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}

calibration/meta.json

@@ -0,0 +1,12 @@
+{
+  "layer": 18,
+  "activation_dim": 4096,
+  "normalize": "standardize",
+  "clamp_sigma": 8.0,
+  "num_examples": 120000,
+  "target_path": "SL-AI/GRaPE-2.1-Flash",
+  "context_window": 128,
+  "layers": [
+    18
+  ]
+}

calibration/stats.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83bcb7cc63b10a06939376787d172dd048c084e4b5921ea08b91ea4707dfa5af
+size 8475255

eval_report.json

@@ -0,0 +1,81 @@
+{
+  "config": "nla_config.json",
+  "checkpoint": "SL-AI/GRaPE-2.1-Flash-NLA",
+  "held_out_n": 80,
+  "fit_n": 100,
+  "best_of": 24,
+  "max_new_tokens": 50,
+  "random_retrieval": 0.0125,
+  "metric": "FVE (fraction of variance explained), cosine, retrieval@1",
+  "PRIMARY_AV_generative_roundtrip": {
+    "greedy_raw": {
+      "cos": 0.11685206741094589,
+      "retr@1": 0.26249998807907104,
+      "fve": -0.6114729642868042
+    },
+    "greedy_magcal": {
+      "cos": 0.11685206741094589,
+      "retr@1": 0.26249998807907104,
+      "fve": -0.002733588218688965
+    },
+    "bestofK_raw": {
+      "cos": 0.2748314440250397,
+      "retr@1": 0.75,
+      "fve": -0.312843918800354
+    },
+    "bestofK_magcal": {
+      "cos": 0.2748314440250397,
+      "retr@1": 0.75,
+      "fve": 0.026947617530822754
+    },
+    "bestofK_perdim_cal": {
+      "cos": 0.15629741549491882,
+      "retr@1": 0.5,
+      "fve": 0.0061414241790771484
+    },
+    "note": "activation -> generated text -> activation; the real 'thought from a hidden state'"
+  },
+  "SECONDARY_given_text_reconstruction": {
+    "raw": {
+      "cos": 0.9040088653564453,
+      "retr@1": 0.987500011920929,
+      "fve": 0.814344048500061
+    },
+    "perdim_cal": {
+      "cos": 0.9016216993331909,
+      "retr@1": 0.987500011920929,
+      "fve": 0.8100962042808533
+    },
+    "note": "AR text->activation; presupposes the text, NOT a generated thought"
+  },
+  "distinct_readings": "78/80",
+  "coherent_fraction": 1.0,
+  "on_topic_jaccard": {
+    "own": 0.18751889010740513,
+    "random": 0.048980255730286325,
+    "ratio": 3.8284587802071273
+  },
+  "avg_reading_words": 26.7,
+  "sample_readings_by_cosine": [
+    {
+      "cosine": 0.956,
+      "reading": "Sure! My phone number is 555-123-2002, and my name is John Smith."
+    },
+    {
+      "cosine": 0.332,
+      "reading": "Alright, I checked the top of my screen. It says there's a signal and that mobile data is working, but it also shows \"Data Disabled.\" Right now I see airplane mode isn't on, so hopefully things are ok"
+    },
+    {
+      "cosine": 0.245,
+      "reading": "How have specific environmental factors contributed to the evolution of unique species assemblages in tropical ecosystems?"
+    },
+    {
+      "cosine": 0.195,
+      "reading": "A 62.2 kg object is pushed with a force of 83.2 N at an angle of 41.5 degrees to the horizontal. Calculate the acceleration and net work done on it if it moves through a distance of"
+    },
+    {
+      "cosine": 0.003,
+      "reading": "Compute (74856 * 392) mod 10. Show each step."
+    }
+  ]
+}

nla_config.json

@@ -0,0 +1,99 @@
+{
+  "target_path": "./av-model",
+  "nla_dir": ".",
+  "backbone": {
+    "path": "./av-model",
+    "quantization": "nf4",
+    "compute_dtype": "bfloat16",
+    "double_quant": true,
+    "device_map": "auto",
+    "trust_remote_code": true,
+    "attn_implementation": "eager"
+  },
+  "surgery": {
+    "av_lora_rank": 48,
+    "av_lora_alpha": 96,
+    "av_lora_dropout": 0.0,
+    "av_num_slots": 8,
+    "av_proj_hidden": 3072,
+    "ar_lora_rank": 128,
+    "ar_lora_alpha": 256,
+    "ar_lora_dropout": 0.0,
+    "ar_head_hidden": 4096,
+    "ar_unfreeze_output_proj": false,
+    "lora_target_modules": [
+      "q_proj",
+      "k_proj",
+      "v_proj",
+      "o_proj",
+      "gate_proj",
+      "up_proj",
+      "down_proj",
+      "in_proj_qkv",
+      "out_proj"
+    ]
+  },
+  "activation": {
+    "layer": 18,
+    "layers": [
+      18
+    ],
+    "normalize": "standardize",
+    "clamp_sigma": 8.0,
+    "pca_dim": 512
+  },
+  "extract": {
+    "out_dir": "./calibration",
+    "num_examples": 120000,
+    "max_seq_len": 128,
+    "positions_per_context": 5,
+    "skip_first": 4,
+    "batch_size": 6,
+    "context_window": 128,
+    "corpus_max_texts": 80000,
+    "corpus_max_chars": 600,
+    "corpus_min_chars": 24,
+    "corpus_include_thinking": true,
+    "corpus_include_prompts": true,
+    "corpus_seed": 0,
+    "corpus_english_only": true
+  },
+  "train": {
+    "out_dir": "./run",
+    "dataset_dir": "./calibration",
+    "epochs": 2,
+    "max_steps": 3500,
+    "batch_size": 2,
+    "grad_accum": 8,
+    "lr": 0.0003,
+    "weight_decay": 0.0,
+    "warmup_ratio": 0.03,
+    "max_grad_norm": 1.0,
+    "max_explanation_tokens": 50,
+    "min_explanation_tokens": 12,
+    "w_cosine": 0.0,
+    "w_mse": 0.0,
+    "w_ar_teacher": 1.0,
+    "ar_short_frac": 0.0,
+    "w_distill": 0.5,
+    "w_distill_final": 0.2,
+    "distill_anneal_frac": 0.4,
+    "w_recon_reward": 0.0,
+    "grad_decode": false,
+    "gumbel_tau": 1.0,
+    "grad_checkpointing": true,
+    "optim_8bit": true,
+    "save_every": 350,
+    "eval_every": 350,
+    "log_every": 25,
+    "eval_examples": 64,
+    "seed": 0,
+    "resume": true,
+    "teacher_path": null,
+    "max_distill_tokens": 24,
+    "rl_max_new_tokens": 32,
+    "rl_lr": 2e-05,
+    "rl_entropy_coef": 0.02,
+    "w_ar_coadapt": 0.0
+  }
+}

ood_report.json

@@ -0,0 +1,171 @@
+{
+  "config": "nla_config.json",
+  "checkpoint": "SL-AI/GRaPE-2.1-Flash-NLA",
+  "n_ood_prompts": 24,
+  "best_of": 24,
+  "layer": 18,
+  "random_retrieval": 0.041666666666666664,
+  "note": "prompts are hand-written from domains far outside the sampled corpus",
+  "AV_generative_roundtrip": {
+    "cos": 0.69648677110672,
+    "retr@1": 0.4583333432674408,
+    "fve": -0.006114959716796875
+  },
+  "AV_generative_roundtrip_magcal": {
+    "cos": 0.69648677110672,
+    "retr@1": 0.4583333432674408,
+    "fve": 0.11318624019622803
+  },
+  "AR_given_text_reconstruction": {
+    "cos": 0.8898084163665771,
+    "retr@1": 1.0,
+    "fve": 0.6475629806518555
+  },
+  "english_ok_fraction": 1.0,
+  "readings": [
+    {
+      "reading": "Write the full Hamiltonian for a topological insulator with Dirac surface fermions.",
+      "recon_cos": 0.796,
+      "english_fraction": 0.917,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "How does DNA methylation regulate plant development during the WUSCH pathway?",
+      "recon_cos": 0.696,
+      "english_fraction": 0.818,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "Explain the four Dorsal vertae groups and their role in classification.",
+      "recon_cos": 0.578,
+      "english_fraction": 0.909,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "What distinguishes a meso compound from other stereoisomers?",
+      "recon_cos": 0.882,
+      "english_fraction": 0.75,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "Prove that \\( S_5 \\) is not simple.",
+      "recon_cos": 0.79,
+      "english_fraction": 1.0,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "Describe cosmic fullness in the style of a dharmic deity.",
+      "recon_cos": 0.567,
+      "english_fraction": 0.9,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "Write a 50-word haiku sequence (14 lines) about technology, using only present and past tenses.",
+      "recon_cos": 0.666,
+      "english_fraction": 1.0,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "How does the chaotic behavior of fluid flow impact bioremediation processes in contaminated environments?",
+      "recon_cos": 0.452,
+      "english_fraction": 0.929,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "Analyze the relationship between limb movements and muscle activation patterns during a specific dance sequence.",
+      "recon_cos": 0.726,
+      "english_fraction": 1.0,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "How does the structure of long bones contribute to their function in leverage?",
+      "recon_cos": 0.775,
+      "english_fraction": 1.0,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "What is the name of a major constituent (90%+) found in archaeological sites from 1453 BC?",
+      "recon_cos": 0.425,
+      "english_fraction": 1.0,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "Describe the coalescence process of two black holes and its gravitational wave signature.",
+      "recon_cos": 0.673,
+      "english_fraction": 1.0,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "How does the Joseph junction in a superconducting circuit produce single-photon interference?",
+      "recon_cos": 0.682,
+      "english_fraction": 0.923,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "Write a 50-word poem from the perspective of an exploding cookie.",
+      "recon_cos": 0.787,
+      "english_fraction": 1.0,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "What is the historical significance of the Treaty at Nijmeduin in 1478?",
+      "recon_cos": 0.813,
+      "english_fraction": 0.909,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "Explain the event horizon paradox and how it relates to black hole thermodynamics.",
+      "recon_cos": 0.78,
+      "english_fraction": 1.0,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "What processes have shaped the morphological evolution of horse teeth?",
+      "recon_cos": 0.64,
+      "english_fraction": 1.0,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "How does the Munc13-4 calcium-dependent exocytosis protein contribute to presynaptic vesicle fusion?",
+      "recon_cos": 0.783,
+      "english_fraction": 0.769,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "Implement a test for the Miller-Rabin primality theorem. Explain how it works and why it's important.",
+      "recon_cos": 0.63,
+      "english_fraction": 0.941,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "Analyze the structural paradox in Shakespeare's Hamlet: how does its greatest strength also cause its tragic flaw?",
+      "recon_cos": 0.584,
+      "english_fraction": 1.0,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "What is the strong coupling constant of quarks and how does it vary with energy?",
+      "recon_cos": 0.778,
+      "english_fraction": 1.0,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "What are the cultural and historical significance of Newgrange in Ireland?",
+      "recon_cos": 0.707,
+      "english_fraction": 0.909,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "How does the curvature of spacetime around a massive object cause light to follow an orbit?",
+      "recon_cos": 0.748,
+      "english_fraction": 0.938,
+      "has_nonascii_letter": false
+    },
+    {
+      "reading": "Write a Ruby program that generates the Fibonacci sequence.",
+      "recon_cos": 0.758,
+      "english_fraction": 1.0,
+      "has_nonascii_letter": false
+    }
+  ]
+}

surgery_heads.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce49e28ae32c6cf365d6d79f8cdd016830aa0f44aa962dc6bc5c7b6c5b9219d1
+size 275971581

upload_to_hf.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+# Publish the full merged NLA models to the HuggingFace Hub.
+#   huggingface-cli login   # once, with a write token
+#   ./upload_to_hf.sh
+set -euo pipefail
+REPO="${1:-SL-AI/GRaPE-2.1-Flash-NLA}"
+HERE="$(cd "$(dirname "$0")" && pwd)"
+echo "Uploading $HERE  ->  https://huggingface.co/$REPO"
+huggingface-cli upload "$REPO" "$HERE" . --repo-type=model \
+  --commit-message="Publish GRaPE-2.1-Flash NLA: full merged AV+AR models (English-only reader)"
+echo "Done -> https://huggingface.co/$REPO"