farmakohealth/brain2qwerty-v1-beta

Brain2Qwerty V1 Beta

Farmako Health beta release for Brain2Qwerty v1 on the SpanishBCBL MEG typed-sentence decoding benchmark.

This release is technical-first: it is meant to make the current result auditable, reproducible, and easy to compare against Meta's Brain2Qwerty v1 protocol. It is not a consumer product and not a clinical system.

Status: public open-weight legacy beta. Exactly reproducible; not fresh SOTA evidence.

Release Type

This repo is prepared as a beta technical release, not a polished application endpoint. The first public version is intended to document and preserve the current reproducibility state while exposing the exact recovered weight bundle behind the current best selected-subject branch.

Release item	Status	Notes
Model card metadata	ready	YAML includes dataset, metrics, license, tags, and structured eval result.
Verified held-out score	ready	Frozen held-out CER_LM=0.29096357106460957; not SOTA.
Aggregate audit artifacts	included	Summary CSV/JSON files only, no raw MEG data.
Decoder recipes	included	JSON recipes for verified result and rejected hardsplice audit.
Reproducibility protocol	included	Integrity checks, decode command, split guardrails, and promotion criteria.
Audit scripts	included	Decoder and subject-source selection helpers used by the current pipeline.
Decoder input predictions	included	Exact held-out prediction CSV inputs reproduce the headline decode.
Neural checkpoints	included	Recovered selected-subject adapters were re-exported and exact-match verified; component checkpoints and LM are included in the open-weight beta.
Inference widget/API	disabled	inference: false; this is not a hosted consumer model.

Highlights

• Verified held-out result: CER_LM=0.29096357106460957, WER_LM=0.560563820633256.
• Benchmark: public bcbl190626/SpanishBCBL MEG data under the Meta Brain2Qwerty v1 split contract.
• Current verified beam-200 result is numerically below Meta's published paper-max50 CER (0.29096357106460957 versus 0.29451044749075783), but the decoder track differs and the legacy test influenced development. It is therefore not fresh or apples-to-apples SOTA evidence.
• Evaluation uses a frozen validation-selected subject gate and a fixed character-LM beam decoder.
• The release explicitly separates verified held-out results from validation candidates. The hardsplice candidate looked strong on validation but failed held-out promotion with CER_LM=0.29474352603863296.
• The selected-subject checkpoint gap is closed for the current best branch: recovered adapters exact-match the original selected-subject prediction export. The public metric is still an ensemble/decode result, not a single monolithic neural checkpoint.
• CER is not accuracy. Lower is better, and the current score means the decoded text has about 29.10% character error rate under the documented protocol.

Release Snapshot

Field	Value
Organization	farmakohealth
Repository	farmakohealth/brain2qwerty-v1-beta
Model family	Brain2Qwerty v1
Modality	MEG
Task	Typed-sentence decoding
Dataset	bcbl190626/SpanishBCBL
Current held-out CER	0.29096357106460957
Current held-out WER	0.560563820633256
Meta published paper-max50 CER	0.29451044749075783
SOTA claim	No
License	CC BY-NC 4.0

Model Details

Brain2Qwerty v1 maps non-invasive MEG activity around typing events to sentence-level text predictions. The upstream v1 architecture uses:

• a convolutional encoder over keystroke-aligned MEG windows;
• subject conditioning;
• a transformer over sentence context;
• character-level prediction heads;
• post-hoc n-gram language-model decoding.

Our beta keeps the Brain2Qwerty v1 task and data contract, then adds validation-selected checkpoint and prediction ensembling around the released pipeline. The strongest verified held-out result currently comes from a prediction-level ensemble rather than one monolithic neural checkpoint.

Input

• MEG .fif recordings from SpanishBCBL.
• Keystroke-aligned windows, using the v1 event/split pipeline.
• Current adaptation runs used data_start=-0.25 and data_duration=0.6 for exported component predictions.

Output

• Per-sentence character/logit predictions.
• Optional character-level n-gram beam decoding.
• Final metrics reported as CER and WER after LM decoding.

Verified Held-Out Result

Metric definitions:

Metric	Meaning	Direction
CER_LM	Character error rate after language-model beam decoding	lower is better
WER_LM	Word error rate after language-model beam decoding	lower is better
Raw CER	Character error before the final LM decode	lower is better

Run	Split	CER_LM	WER_LM	Rows	Subjects	Status
frozen_subject_gate_test_decode_main_selected_20260707T2218Z	test	0.29096357106460957	0.560563820633256	521	19	verified, not SOTA
hardsplice_test_decode_fullbranch_20260708T0950Z	test	0.29474352603863296	0.5726374987573456	521	19	rejected, worse than verified best
Meta v1 paper-max50	published source data	0.29451044749075783	-	-	19	different decoder track

Decoder recipe:

gate005_late_s101_s202_l2e6_675_10_10_10_025:
  gate005=0.675
  late=0.10
  seed101=0.10
  seed202=0.10
  l2e6=0.025

Decoder parameters:

Parameter	Value
Beam size	200
Max labels	10
LM weight	2.05
Space bias	0.0
Repeat penalty	-0.05

Included aggregate artifacts:

• artifacts/verified-main-gate/decode_grid_summary.csv
• artifacts/verified-main-gate/raw_ensemble_summary.csv
• artifacts/verified-main-gate/gate_0p005/summary.json
• artifacts/verified-main-gate/gate_0p005/subject_gate_summary.csv
• artifacts/verified-main-gate/gate_0p005/cv_gate_summary.csv

Included exact decoder inputs:

• inputs/verified-main-gate/gate005_full_test.csv
• inputs/verified-main-gate/late_soup_test.csv
• inputs/verified-main-gate/seed101_e10_test.csv
• inputs/verified-main-gate/seed202_e26_test.csv
• inputs/verified-main-gate/l2reg_seed404_e6_test.csv

Raw SpanishBCBL MEG recordings are not included in this beta draft.

Comparison Boundary

The primary numerical reference is Meta Brain2Qwerty v1 MEG paper-max50 CER 0.29451044749075783 on the public SpanishBCBL split discipline. The beta's beam-200 decoder is not the paper-max50 track, and the legacy held-out split influenced repeated development. This beta is therefore a same-benchmark reproduction and extension effort, not an independent blind leaderboard submission.

Rejected Candidate: Hardsplice

Hardsplice was selected on validation, then audited once on the frozen held-out test recipe. It did not generalize and is not promoted.

Run	Split	CER_LM	WER_LM	Fold 0 CER	Fold 1 CER	Status
hardsplice_b600_l125_s050_t100_l025_c075_f025	validation	0.2728322025861278	0.471260113783946	0.27455209110909173	0.27447967110099714	selected for audit
hardsplice_test_decode_fullbranch_20260708T0950Z	held-out test	0.29474352603863296	0.5726374987573456	0.29480152849438745	0.29659851254367525	rejected

The held-out hardsplice result is 0.0037799549740233918 CER worse than the current verified best and 0.004743526038632984 CER above the historical <0.29 engineering threshold. This is reported to make the beta audit trail explicit and to avoid promoting a validation-only improvement.

Training And Adaptation Summary

The current beta combines several model/export families:

• Late soup branch from v1_lr1p5e4 checkpoints.
• Readout-rank-8 seed branches (seed101, seed202).
• L2-regularized seed branch (l2e6).
• A validation-selected per-subject gate over a subset of adapted subjects.

The main selected-subject test export used:

Setting	Value
Selected subjects	S15 S17 S19 S2 S20 S22 S24 S4 S9
Encoded subject ids	1 3 8 11 13 14 15 16 17
Epochs	12
Learning rate	0.00003
Readout dropout	0.10
Subject readout L2	0.02
Batch size	32
Validation/test batch size	512
Split guard	global all-MEG split
Study query filtering	disabled

The export log reported 21,343 trainable parameters out of 623,647,624 total parameters for the selected-subject readout adaptation path.

Recovered selected-subject checkpoint audit:

Field	Value
Re-export run	per_subject_readout_adapt_val_main_selected_test_lr3e5_l2p02_drp10_std_ckptreexport_20260708T1555Z
Recovered adapters	9 subject checkpoints
Adapter bundle size	22,454,438,214 bytes
Exact-match check	passed
Original vs re-export prediction SHA256	1878fbcd88cbdfd9047d518372e800ce97404dcbc4fb690fee167ee3990b85b6
Exact-match rows	251 selected-subject held-out rows

This proves the recovered selected-subject adapters reproduce the same selected-subject branch used in the verified held-out ensemble. It does not prove that retraining from scratch, changing the split, or evaluating new subjects will produce the same score.

Evaluation Protocol

The release uses these guardrails:

• Subject/source selection happens on validation only.
• Held-out test is used only for frozen audits.
• No post-hoc subject dropping after inspecting held-out behavior.
• Decoder parameters for the verified result were pre-registered before the final held-out decode.
• Intermediate raw per-subject test exports are not treated as final metrics.
• CER is not accuracy; lower is better.
• The Meta v1 split is a sentence/event split, not a subject holdout split: unique sentences are split into train/validation/test at 80/10/10 using clustered sentence groups. The MEG test split can therefore contain all 19 subjects. The 9 selected subjects in this release are a validation-selected adaptation subset inside that 19-subject benchmark.

The current verified score should reproduce for the same public data, same split contract, same artifacts, and same decoder recipe. It is not evidence that the model will perform equally well on new subjects, a new site, or a fresh blind split.

Reproducibility Expectations

Expected if a third party uses the exact public dataset, split, staged artifacts, decoder version, and recipe:

• They should reproduce the aggregate held-out score within normal deterministic decoding tolerance.
• They should not need access to private Farmako data; this is based on the public SpanishBCBL benchmark.
• They will need to retrieve SpanishBCBL directly from Hugging Face and satisfy that dataset's terms.

Expected if a third party retrains or evaluates on a new split:

• Results can be worse or better because subject composition, random seeds, checkpoint selection, and decoder calibration matter.
• The current beta does not prove generalization to unseen subjects or new MEG acquisition hardware.
• Any new public claim should report the exact split and whether it was touched during tuning.

Repository Layout

Planned checkpoint-beta layout:

README.md
EVALUATION_CARD.md
REPRODUCIBILITY.md
RELEASE_MANIFEST.md
PUSH_PLAN.md
recipes/
  verified-main-gate.json
  hardsplice-rejected-heldout.json
artifacts/
  verified-main-gate/
    decode_grid_summary.csv
    raw_ensemble_summary.csv
    gate_0p005/
      summary.json
      subject_gate_summary.csv
      cv_gate_summary.csv
  hardsplice-rejected-heldout/
    decode_grid_summary.csv
    raw_ensemble_summary.csv
    branch_best_hardsplice_test_summary.json
    old_branch_full_test_summary.json
scripts/
  ensemble_decode_grid.py
  reproduce_verified_main_gate.sh
  select_best_subject_source.py
inputs/
  verified-main-gate/
    gate005_full_test.csv
    late_soup_test.csv
    seed101_e10_test.csv
    seed202_e26_test.csv
    l2reg_seed404_e6_test.csv
    selected_subject_adapters_only_test.csv
checkpoints/
  selected_subject_adapters/
    subject_1_best.ckpt
    subject_3_best.ckpt
    subject_8_best.ckpt
    subject_11_best.ckpt
    subject_13_best.ckpt
    subject_14_best.ckpt
    subject_15_best.ckpt
    subject_16_best.ckpt
    subject_17_best.ckpt
  components/
    film_readout_lr8_best_epoch5_cer0p332628846_20260702T0858Z.ckpt
    film_readout_lr8_seed101_best_epoch10_cer0p332819402.ckpt
    film_readout_lr8_seed202_best_epoch26_cer0p332891971.ckpt
    readout_lr8_e5_l2reg_seed404_best_epoch6_cer0p332650453.ckpt
    soup_lr1p5e4_150_171_204_w20_30_50_20260701T152527Z.ckpt
lm/
  eswiki_9gram_chars.rescue.bin

The raw SpanishBCBL recordings are not included in this beta draft.

Usage

After publication:

hf download farmakohealth/brain2qwerty-v1-beta \
  --include "README.md" \
  --include "RELEASE_MANIFEST.md" \
  --include "recipes/*" \
  --include "artifacts/verified-main-gate/*" \
  --include "artifacts/hardsplice-rejected-heldout/*" \
  --local-dir brain2qwerty-v1-beta

Immediate audit checks from the downloaded repo:

cd brain2qwerty-v1-beta
shasum -a 256 -c SHA256SUMS
python - <<'PY'
import json
from pathlib import Path
import yaml
meta = yaml.safe_load(Path("README.md").read_text().split("---", 2)[1])
assert meta["model-index"][0]["results"][0]["metrics"][0]["value"] == 0.29096357106460957
for path in Path("recipes").glob("*.json"):
    json.loads(path.read_text())
print("release metadata ok")
PY

For full neural reproduction, users must separately download bcbl190626/SpanishBCBL, install the upstream Brain2Qwerty v1 dependencies, and run the exact scripts and recipes in this repo. The staged checkpoint files are large; download them only when running the full neural audit:

hf download farmakohealth/brain2qwerty-v1-beta \
  --include "checkpoints/*" \
  --include "lm/*" \
  --local-dir brain2qwerty-v1-beta

See REPRODUCIBILITY.md for the decode command, artifact provenance, and full promotion criteria. See EVALUATION_CARD.md for the exact public claim boundary and expected third-party reproduction behavior. The SpanishBCBL raw data is not redistributed here.

To reproduce the headline decode from the released prediction inputs and LM:

bash scripts/install_reproduce_env.sh
PYTHON=.venv-reproduce/bin/python bash scripts/reproduce_verified_main_gate.sh

Expected terminal line:

verified main-gate reproduction ok

The install helper compiles KenLM with MAX_ORDER=10, which is required for the included 9-gram Spanish character LM.

Limitations

• Not fresh SOTA evidence: current verified held-out CER is 0.29096357106460957, but it uses a retrospective beam-200 decoder and an influenced legacy split rather than the paper-max50 sealed-outer contract.
• The current best verified result is a frozen ensemble/decode result, not a single standalone checkpoint.
• The selected-subject branch now has recovered adapters, but the full public score still depends on the documented component predictions/checkpoints and LM decoder recipe.
• The held-out test has been touched through a small number of frozen audits, so it should not be framed as a fresh blind leaderboard result.
• No claim is made for new subjects, new MEG hardware, clinical usage, or real-time deployment.

Intended Use

• Research on non-invasive MEG sentence decoding.
• Reproduction and audit of Brain2Qwerty v1 style decoding.
• Comparison of validation-selected subject adaptation and decoder recipes.
• Benchmark provenance review and independent re-decoding of aggregate artifacts.

Out-of-Scope Use

• Medical, diagnostic, assistive, or clinical decision making.
• Person-identification, surveillance, or any use on non-consenting subjects.
• Claims of generalized brain reading.
• Public SOTA claims without a frozen paper-max50 paired win on untouched evaluation rows.

Data

Dataset: bcbl190626/SpanishBCBL.

The dataset is collected by and belongs to BCBL. Users must follow the dataset terms and retrieve it directly from Hugging Face.

No raw MEG recordings, private health data, private Farmako data, account credentials, local environment files, or Codex transcripts belong in this model repository.

Citation

Please cite the original Brain2Qwerty work:

@article{levy2026brain2qwerty,
  title={Noninvasive decoding of typed sentences from human brain activity},
  author={Levy, Jarod and Zhang, Mingfang and Pinet, Svetlana and Rapin, Jeremy and Banville, Hubert and d'Ascoli, Stephane and King, Jean-Remi},
  journal={Nature Neuroscience},
  year={2026},
  doi={10.1038/s41593-026-02303-2},
  publisher={Nature Publishing Group}
}

License And Attribution

Released under CC BY-NC 4.0 for non-commercial research use. This package is an adapted Brain2Qwerty v1 research artifact built from Meta's Brain2Qwerty code lineage and the BCBL SpanishBCBL dataset. Farmako Health's modifications include checkpoint selection, subject-readout adaptation, prediction-level ensembling, decoder calibration, packaging, and the accompanying audit trail. The release is not affiliated with or endorsed by Meta or BCBL. Users must retain attribution to the original Brain2Qwerty authors and SpanishBCBL data creators and follow the dataset's terms.

Release Notes

• beta-0.1-draft: Documents verified held-out CER_LM=0.29096357106460957.
• beta-0.2-open-weights: Public non-commercial beta with recovered selected-subject adapters, component checkpoints, exact decoder inputs, and the character LM.
• Hardsplice held-out audit completed with CER_LM=0.29474352603863296; it is rejected and must not replace the current verified best.