Gemma-4-12B-PsiLM

Gemma 4 12B, coupled to a physics model through trained latent bridges, runnable on a Mac with one command.

Research generated by Claude Fable 5 (Anthropic) under the direction of Ryoji Furui; see the AI generation disclosure.

What this is

PsiLM (ΨLM) runs a frozen language model and a frozen physics model together while one answer is produced. Nothing is fine-tuned and no text crosses the interface: small trainable bridges read the physical problem out of the language model's hidden states, the physics model computes, and its result flows back into the language model's residual stream as a few soft tokens through a gated cross-attention. This repository packages that system for one backbone:

part	what	trained?	where it comes from
language model	Gemma 4 12B-it, 4-bit MLX quantization (48 layers, hidden 3840)	frozen	downloaded from mlx-community/gemma-4-12B-it-4bit at first run (not redistributed here)
physics model	a 1D Burgers Fourier Neural Operator, 70K parameters (physics/fno_burgers_singlemode.safetensors)	frozen	this repo (also in ryoji-info/PsiLM-physics)
bridges	forward readout + value-token channel + gated injection, 25.5M parameters (bridges/gemma-4-12b-4bit-mlx-1d-value-selective/)	trained	this repo (also in ryoji-info/PsiLM-bridges)

What it can answer today. One family of questions, the one the bridges were trained on:

A velocity field on the periodic domain [0,1) starts as u(x,0) = a · sin(2πx + φ). It evolves by Burgers' equation with viscosity 0.02 until t = 0.5. What is the value of u at x = x₀? Answer with a number rounded to 2 decimal places.

with a in [0.5, 1.5], φ in [0, 6.28], x₀ in [0, 0.99], two decimals each. Gemma alone cannot answer these (0% on the held-out set; it derives for 768 tokens and never commits to a number). Coupled through the bridges it answers 96.7% of them within ±0.05, against 98.3% when the true value is written into the prompt as text (the oracle ceiling). On everything else the gate stays shut and the model is Gemma, byte for byte in the arms measured (GSM8K 84/84, see below).

What it is not. Not a general physics assistant, not a fine-tuned Gemma, and not a model that will know when a different PDE applies: the physics model solves exactly one equation family and the bridges read exactly three quantities (a, φ, x₀) from the text. It is a research artifact: a working, measured instance of latent coupling between a language model and a physics model at 12B scale, on consumer hardware.

Run it

Apple Silicon Mac (the 4-bit backbone is 6.3 GB on disk; training of the bridges peaked at 13 GB on a 24 GB M2, inference needs less), Python 3.11+.

# 1. get this repo (≈100 MB of bridges + 0.5 MB of physics model)
huggingface-cli download ryoji-info/Gemma-4-12B-PsiLM --local-dir Gemma-4-12B-PsiLM
cd Gemma-4-12B-PsiLM

# 2. dependencies (mlx, mlx-lm, transformers, torch, huggingface_hub + the psilm package from GitHub)
pip install -r requirements.txt

# 3. one command
python psilm_infer.py

The first run downloads the Gemma 4 backbone (6.3 GB) into the Hugging Face cache. The script answers the default question (a = 1.28, φ = 0.5, x₀ = 0.76) three ways and prints timing:

[1] PsiLM (coupled)     : the coupled system's reply ('u at x = 0.76 equals <number>.')   -- seconds
[2] backbone alone      : Gemma 4 alone under the "Answer: <number>" protocol, answer-forced -- about 75 s
[3] physics model (FNO) : u(0.76) = -0.2522   (the FNO on the true initial condition: the reference [1] should match to ±0.05)
    spectral solver     : u(0.76) = -0.2517   (ground truth)

Other questions and options:

python psilm_infer.py --a 0.9 --phi 2.1 --x0 0.33     # any (a, phi, x0) in the ranges above
python psilm_infer.py --no-baseline                   # skip the slow backbone-alone arm
python psilm_infer.py --question-only                 # print the exact prompt, load nothing
python psilm_infer.py --help                          # --bridges DIR, --physics FILE, --backbone ID, ...

The same script runs the two other tasks once their bridges are in this repo (--task is auto-detected from the bridges directory, so it can be omitted when --bridges is given):

# 1D Burgers, single-mode initial condition (the released bridges; the default task)
python psilm_infer.py --task 1d --a 1.28 --phi 0.5 --x0 0.76

# 1D Burgers, multi-mode initial condition u(x,0) = sum a*sin(2*pi*m*x + phi), m in {1, 2}
# (--modes takes m:a:phi triples; one mode is the training family, two modes the held-out combination)
python psilm_infer.py --task multimode --modes "1:0.55:2.10" --x0 0.76
python psilm_infer.py --task multimode --modes "1:0.55:0.50,2:0.75:1.10" --x0 0.33

# 2D Fisher-KPP: a Gaussian bump (height a, center (cx, cy), width w), value at (x0, y0)
python psilm_infer.py --task 2d --a 0.6 --cx 0.35 --cy 0.6 --w 0.07 --x0 0.4 --y0 0.55

Each task reads bridges/<task directory>/config.json for its construction and coupling depths and its physics file (physics/fno_burgers_multimode.safetensors, physics/dpot_tiny_fisher2d_finetuned.safetensors). The 2D physics model is DPOT-Tiny and runs in torch (MPS, CPU fallback; --phys-device): it needs the einops package (in requirements.txt), the upstream DPOT-Tiny base checkpoint physics/model_Ti.pth (--dpot-base; when the file is absent the script downloads it from hzk17/DPOT into the Hugging Face cache), and the vendored DPOT definition vendor/dpot_model.py, which the pip package does not ship, so the 2D task runs with a clone of the GitHub repository: PSILM_REPO=/path/to/PsiLM python psilm_infer.py --task 2d ....

Without pip install-ing the package, a clone of the GitHub repository works for every task: PSILM_REPO=/path/to/PsiLM python psilm_infer.py.

Loading the pieces yourself, in Python:

import json, mlx.core as mx
from psilm.mlx.gemma_loader import load_backbone_any     # Gemma 4 text tower in the staged-forward layout
from psilm.mlx.bridges import PsiBridgesMLX
from psilm.mlx.fno import load_fno_safetensors
from psilm.mlx.model import PsiLMMLX

model, stock, tok = load_backbone_any("mlx-community/gemma-4-12B-it-4bit")
d = "bridges/gemma-4-12b-4bit-mlx-1d-value-selective"
cfg = json.load(open(f"{d}/config.json"))
bridges = PsiBridgesMLX(**cfg["construct"])                # d_model 3840, channel "value", inj_cap 0.2, readout_norm "dim"
bridges.load_weights(f"{d}/bridges.safetensors", strict=False)   # the retired learned-pointer tensors are omitted
fno = load_fno_safetensors("physics/fno_burgers_singlemode.safetensors")
psi = PsiLMMLX(model, tok, fno, bridges, l_fwd=cfg["coupling"]["l_fwd"], l_rev=cfg["coupling"]["l_rev"])   # read @20, inject @30 of 48
# psi.generate(QABuilder(hf_tokenizer), {"a": 1.28, "phi": 0.5, "x0": 0.76}) -- see psilm_infer.py

What it costs and what it buys

component	parameters	on disk	trained?
Gemma 4 12B-it, 4-bit MLX (language tower)	12.28B	6.3 GB	frozen
PsiLM bridges, one (backbone, task) pair	25.5M	102 MB	trained
Burgers FNO, the physics hemisphere	0.07M	0.55 MB	frozen, pretrained
DPOT-Tiny, 2D task only	7.5M	30 MB	frozen, fine-tuned

The trained part is 0.21% of the backbone's parameters and 1.6% of its checkpoint size. The 12.28B never move.

Measured on one Apple M2 (24 GB): +0.21% parameters and +103 MB turn 0% into 97% on the physics task, at 24× lower latency (3.14 vs 77.0 seconds per question, 16.9 vs 768 generated tokens), with GSM8K and MMLU unchanged — the gate's σ is 0.14 on physics against 0.004–0.008 elsewhere, so the channel is shut when physics is irrelevant. The backbone's 0% is its own text protocol: it spends the whole 768-token budget deriving and never commits to an answer line (forced to answer, n=60, it scores 6.7%: near-constant guesses landing inside the tolerance), which is where the latency gap comes from too. The per-dataset numbers are in the next section.

Results for this backbone

Held-out evaluation, 60 questions, accuracy within ±0.05 (results/stage2_gemma12b/final_eval.json in the GitHub repository):

arm	accuracy	MAE	note
Gemma 4 12B alone	0.0%	2.93	never reaches an Answer: line within 768 tokens
Gemma 4 12B alone, forced to answer	6.7%	0.42	continuation started with Answer:; four near-constant guesses (0.41 / 0.51 / 0.54 / 0.11) land inside the tolerance — the best constant would score 13.3% (final_eval_baseline_forced.json)
PsiLM (this repo)	96.7%	0.017	bridges read the prompt, FNO computes, value returns in latent space
oracle (true value written into the prompt)	98.3%	0.007	the tool-loop ceiling
always answer 0.00	1.7%	0.308	calibration

Guard-rail: does the coupled model still do everything else? 100 questions per dataset, three arms — backbone alone / PsiLM / PsiLM with the injection zeroed — with the gate recorded per question (results/bench/gemma12b_guardrail_summary.json, results/bench/gemma12b_nonudge_guardrail_summary.json):

dataset (n=100)	backbone	PsiLM	zeroed	gate σ (PsiLM)	open on
physics QA (this task)	0%	97%	10%	0.14	100%
GSM8K	84%	84%	84%	0.004	0%
GSM8K, no Answer: line in the prompt	83%	83%	83%	0.002	0%
MMLU, 5 subjects, 256 tokens	53%	55%	53%	0.008	0%

On MMLU the two arms agree at 79.1% / 79.1% on the 67 items both answer within the budget; the raw 53 vs 55 is parse noise. The gate selectivity comes from a no-harm training arm: 1,046 non-physics prompts (GSM8K train, MMLU validation, with and without the Answer: nudge) paired with Gemma's own greedy continuations, on which only the gate is updated with a mean-gate penalty (config.json → training). The 10% of the zeroed arm on physics is what the reply template alone recovers; the coupled 97% is against that floor.

Training: 7,000 steps at batch 4 on one Apple M2 (24 GB) — 2,000 readout-only warm-up steps, 3,500 coupled steps (lr 3e-4; the coupled phase ran on to step 6,000 but the no-harm phase resumed from the step-5,500 checkpoint), then 1,500 no-harm steps at lr 1e-4 (results/gemma12b/noharm_recipe.sh); 12 s per step at a 13 GB peak. One backbone-specific adjustment, measured rather than tuned: Gemma's massive-activation dimensions are nearly constant across prompts, so the readout standardizes each hidden dimension with statistics from a 32-prompt calibration pass (readout_norm: "dim"; the two frozen vectors fwd.dim_mu, fwd.dim_sigma are in the checkpoint).

Bridges in this repository

directory	task	physics model	trained params	held-out	status
bridges/gemma-4-12b-4bit-mlx-1d-value-selective/	1D Burgers, single-mode initial conditions, value at x₀	physics/fno_burgers_singlemode.safetensors	25.5M	96.7% @±0.05, MAE 0.017 (n=60); GSM8K 84/84	released
bridges/gemma-4-12b-4bit-mlx-multimode/	1D Burgers, multi-mode initial conditions (modes 1–2, mixed amplitudes) + generalization families	physics/fno_burgers_multimode.safetensors	25.5M	iid 100% @±0.05, MAE 0.009 (n=48); combination 25.0%, amplitude extrapolation 52.1%	released
bridges/gemma-4-12b-4bit-mlx-2d-dpot/	2D Fisher–KPP, replicated-IC history → u(0.4), value at (x₀, y₀)	physics/dpot_tiny_fisher2d_finetuned.safetensors (DPOT-Tiny, 7.5M, fine-tuned)	13.8M	100% @±0.05, MAE 0.0096 (n=60; backbone 10.0%, oracle 96.7%)	released

The multi-mode bridges are released: they reach 100% in-distribution (n=48, MAE 0.009) and do not transfer to the two generalization families — 25.0% on the held-out mode combination and 52.1% on amplitude extrapolation, against a backbone of 16.7% / 10.4% and an oracle of 100% on both. The physics model is exact on those families (MAE 0.0008), so the gap is in the readout: 19 of 48 combination answers match a single-mode field value, and on extrapolation the implied amplitude is below the true one for 71% of items. A span readout with mode-shared heads was validated at 0.5B for exactly this gap; the 12B run has not been started. The 2D bridges are released too, and they carry the campaign's strongest result: 100% at MAE 0.0096 (n=60) against 10.0% for the backbone alone and 96.7% for the oracle — the only arm in this work where the latent channel beats the text ceiling, because the oracle has to copy a number out of the prompt and sometimes mis-rounds it while the bridge reads the field value exactly. The 0.5B backbone reached 95.0% on the same task. Training: 7,500 steps (2,000 readout-only, 4,000 coupled, 1,500 no-harm), the coupled phase ending at 93.8% and the no-harm phase at 97.9 / 95.8 / 100.

How it works, in one paragraph

The prompt runs through Gemma's first 20 layers. The forward bridge reads the queried position x₀ by pooling the hidden states over its tokens (a deterministic span pointer computed by the QA builder, plus a 100-bin classifier) and the initial-condition parameters with a learned pool, after the calibrated per-dimension standardization) and emits (a, sin φ, cos φ) and x₀; from these it builds the initial condition on a 128-point grid. The frozen FNO evolves it to t = 0.5. A learned periodic lookup kernel reads the field at x₀, and the value-token channel turns that single number into eight soft tokens through Fourier features. At layer 30 a gated cross-attention injects them into the residual stream, capped at 20% of the stream's RMS; the gate is a small MLP on the residual stream, trained to open on physics prompts and close elsewhere. Layers 30–48 and the answer are Gemma's own. Details, ablations and the failure analysis that produced this design are in the paper (paper/psilm.pdf in the repository, Section 9 for scaling, the guard-rail and Gemma).

Is the answer really coming through the channel?

Two controls, and the second is decisive. Zeroing the injection while running everything else — readout, FNO, value tokens, gate — removes the physics result (0% for Qwen3-8B, 10% for Gemma, which is what the reply template alone recovers). Corrupting only the number — feeding the value encoder another question's answer at matched magnitude, with prompt, readout, gate, reply length and parsing untouched — makes the frozen model report the corruption: the spoken answer lands within ±0.05 of the injected value on 99 of 100 held-out questions and within ±0.05 of the truth on 9. Accuracy falls 98% → 9% while the KL to the base model is unchanged (0.222 either way): the output distribution travels just as far, to a different number.

Run on non-physics prompts the same swap changes nothing (GSM8K 0.88 both ways, MMLU 0.66 both ways, p = 1.00), which separates what the channel does by its presence from what it does by its content. Full sweep and records: results/bench/leaky_8b_shuf_guardrail_summary.json and §9.7 of the paper.

Limitations

• One task family. The bridges read exactly the three quantities of the trained question and the FNO solves exactly one equation family; a different PDE, boundary condition, viscosity or final time is out of scope, and the gate closing on non-physics text does not mean it can recognize other physics. Free-text initial conditions ("a Gaussian bump near the left edge") are not supported.
• The pointer is task-supplied. Which tokens hold x₀ is computed by the QA builder from the prompt (QABuilder.x0_span), not learned from the words; the learned attention pointer never left uniform at 8B and was retired. psilm_infer.py builds the prompt itself for that reason; a paraphrased question is not the trained input.
• 4-bit backbone, quantized kernels. The bridges were trained through the 4-bit backbone and absorb its quantization noise, but MLX's quantized matmuls are not guaranteed bit-identical across mlx versions or Apple chips, so a given question can land a hundredth away from the recorded run; the numbers above are from mlx 0.32.2 / mlx-lm 0.31.3 on an M2.
• Inputs with two decimals, inside the training ranges. The readout was trained on numbers formatted like the training set; psilm_infer.py rounds inputs to two decimals and warns outside [0.5, 1.5] × [0, 6.28].
• Bridges do not transfer between backbones. This checkpoint is for mlx-community/gemma-4-12B-it-4bit exactly (hidden 3840, 48 layers); the script refuses a backbone of another width.
• Sequence length. The Gemma loader drives the sliding-window layers with a plain causal mask, exact up to the 1024-token window; the questions here are ~120 tokens.
• Evaluation scope. The guard-rail covers GSM8K, a five-subject MMLU slice and this physics set at n=100 each; nothing else has been measured.

Beyond physics

The bridges here couple a frozen language model to a frozen physics model, but the recipe (read a fixed set of quantities from text; let a frozen quantitative model compute; return one value through a selective gate) is not specific to PDEs. A calibrated market or event-probability model in the physics model's seat would be the same architecture, and the appeal is the same: a language model's forecast grounded in a model that can be validated separately, with a gate that stays shut when the model does not apply. Nothing in this repository has been trained or tested on financial data; the physics results relied on exact oracles, deterministic targets and no distribution shift, none of which markets provide. This is a research direction, not a capability, and not a basis for investment decisions.

Files

psilm_infer.py          the one-command CLI (PsiLM / backbone alone / physics model)
requirements.txt        pip dependencies, including the psilm package from GitHub
bridges/gemma-4-12b-4bit-mlx-1d-value-selective/
bridges/gemma-4-12b-4bit-mlx-multimode/
bridges/gemma-4-12b-4bit-mlx-2d-dpot/
MANIFEST.md
    bridges.safetensors     the trained bridges (25.5M params, fp32, 97 MB)
    config.json             construction, coupling depths, training record, per-chunk held-out scores
physics/fno_burgers_singlemode.safetensors   the frozen FNO (torch key names; complex spectral weights as .real/.imag)
psilm-banner.png        banner
README.md               this card

Related

• Code, training scripts, evaluation records and the paper: github.com/ryoji-info/PsiLM
• Bridges for every backbone (Qwen2.5-0.5B, Qwen3-1.7B, Qwen3-8B, Gemma 4 12B): ryoji-info/PsiLM-bridges
• The frozen physics models: ryoji-info/PsiLM-physics

Support

PsiLM is independent research, run on a single Apple M2. If it is useful to you, you can support the work at ko-fi.com/ryojifurui.

Citation

@misc{furui2026psilm,
  title  = {PsiLM: Coupling Frozen Language and Physics Models through Trainable Latent Bridges},
  author = {Furui, Ryoji},
  year   = {2026},
  url    = {https://github.com/ryoji-info/PsiLM},
  note   = {Research generated by Claude Fable 5 (Anthropic) under the author's direction}
}

License

The bridges, the FNO and the code in this repository are released under Apache-2.0. The Gemma 4 backbone is not part of this repository: psilm_infer.py downloads it from mlx-community/gemma-4-12B-it-4bit, and its use is governed by Google's Gemma Terms of Use.

AI generation disclosure

This model, its training recipe, the evaluations and this card were generated by Claude Fable 5 (Anthropic), operating as an autonomous research agent under the direction and review of Ryoji Furui, who set the research question and the hardware constraint, approved each stage, and bears responsibility for the published claims. All numbers on this card are taken from committed evaluation records in the repository, cited by file name above.