🔬 TAF Agent

🎯 What do you want to do?

Pick a task. Each one opens the right tool below. Or scroll down for the full list of 14 modes.

🎯 Mode 7 modes available. Most users want 📇 Profile (one-click full diagnosis).
📇 Profile: paste a model id → 5-recipe TAF Card.
🆚 Compare: 2-3 models side-by-side on one recipe.
🔍 Inspect: paste raw config.json to debug parameters.
💬 Ask: free-form question, browser LLM picks the recipe.
📋 Recipe: manual selection with full form control.
🩺 Diagnose CLI: generate Python command to measure γ on real weights.
📊 Phase diagram: explore 23 panel models on (log θ, γ) plane.

Quickest start: paste any HuggingFace model id (e.g. meta-llama/Meta-Llama-3-8B), click Profile. See all 5 recipes scored in seconds.

📇 Profile a model One-click full diagnosis. Paste any HF model id (or pick preset). Tool runs all 5 recipes (long-context, KV-compression, custom-vs-API, budget, hardware) and produces a single TAF Card showing verdict per dimension + key numbers + architecture classification.

Use case: "I'm evaluating Qwen2.5-32B for production — what's its full viability profile?" → paste id → Profile → done.

For technicians: when you need a complete viability snapshot of a candidate model. Outputs match paper §sec:gamma_decomposition format.

🔍 Architecture Inspector Paste any config.json directly. Tool parses it and runs the full Profile. Useful for: private models, in-development configs, models not yet on HuggingFace, or comparing what your custom architecture would do.

Paste the raw config.json contents. The tool extracts the architectural parameters and runs the full 5-recipe Profile.

🆚 Compare models side-by-side Same recipe, multiple models. Pick 2-3 candidate models and one recipe. See verdicts in a single comparison table.

Use case: "I need long-context retrieval at 16K — which is best: Llama-3-8B, Mistral-7B, or Qwen-7B?" → pick 3 + X-2 + 16K → see winner.

For technicians: when choosing between 2-3 candidate models for a specific deployment scenario. Compare their verdicts on the same recipe.

❓ Your question

🩺 Diagnose CLI Command Builder Measure γ_obs (not predict). The browser tool predicts γ from config alone (Padé). To measure the actual decay on a real model you need GPU + Python. This builder produces the exact CLI command you run locally; the script is shipped in this repository at cli/diagnose_model.py.

Output: γ_obs, R², phase, KV cache budget D_90, KL anomaly, full thermodynamic profile (Z, U, S, F, C_V, χ). Saved as JSON.

Pick options below and copy-paste the generated command on your local machine (Python + transformers + numpy). Total wall time ≈ 5 min in --fast mode on CPU; full mode 20–60 min on GPU.

📊 Phase diagram (γ × θ) Each dot is one model from the paper's empirical panel (data/master_gamma_results.json). The x-axis is RoPE base θ on log scale; y-axis is measured γ. The Hagedorn line γ=1 separates Phase A (γ<1, global) from Phase B (γ>1, local-collapsed). Hover dots for details; click to populate the recipe form.

23 models in the panel; the Padé curve (line) is γ_pred(θ) = (2θ−T√2)/(2θ+T√2) at T=2000.

🪟 Context Unmasker Paste a HuggingFace model id (or raw config.json). The tool checks for sliding-window attention, RoPE scaling (YaRN/linear/dynamic NTK), and GQA — anything that makes max_position_embeddings larger than the practical effective context. Mistral-7B-v0.1 is the canonical example: declared 32k, attends within ~4-8k.

Are you about to spend money on a model that won't actually attend that far? Paste an id and find out in 1 second. No GPU, no inference — just config.json arithmetic.

📜 Chat-template Sniffer Paste an HF model id (or raw tokenizer_config.json). Detects the chat-template family (Llama-3, ChatML, Mistral, Gemma, Phi-3, Alpaca, DeepSeek, custom) and gives you the exact framework command to use it correctly. lm-eval-harness silently halves accuracy if you forget to apply it (issue #1841).

Did you forget --apply_chat_template? Most multi-turn evals fail by ~50% because the chat template wasn't applied. Paste a model id, get the exact CLI flag for your stack.

🎯 Arena-Elo CI Reconstructor Chatbot Arena strips confidence intervals from the public leaderboard. A 5-Elo gap can be statistically meaningless. Paste raw vote data (model_a, model_b, winner) — the tool computes Bradley-Terry MLE + bootstrap CIs and lists statistical ties (CI overlap).

Is GPT-4 actually better than Claude — or are they tied? Paste pairwise vote CSV (or click Load sample). Bradley-Terry MLE + 200-iteration bootstrap → ranked Elos with 95% CIs and statistical-tie detection. All in browser.

🧪 Contamination Prior Computes a Bayesian-ish prior on whether a benchmark score is contaminated, based on (model training cutoff date) × (benchmark release date) × (known corpus inclusion + leak history). Open LLM Leaderboard v1 was killed in 2024 after MMLU/HellaSwag scores became contaminated.

Should you trust your model's MMLU score? Enter the model's training cutoff date — the tool rates 20+ popular benchmarks (MMLU, HellaSwag, GSM8K, HumanEval, IFEval, MMLU-Pro, GPQA…) and tells you which scores are likely contaminated.

⚖️ Quant-regime Classifier Predicts γ-shift (and downstream ΔPPL) for a given (model × quant scheme). Generic claims like "AWQ ~95% retention" are too vague — TAF uses d_head, GQA ratio, SWA flag, and model size to give an architecture-specific verdict. Solves: HF community widely reports unpredictable quant cliffs (NF4 -2 PPL on Phi-3 but fine on Llama-3-8B).

Will quantizing your model break it? Paste an HF model id, pick a quant scheme — get predicted γ-shift, expected ΔPPL band, and a recommended alternative if it's a cliff. Browser-only, no GPU, no calibration set required.

🔀 Cross-framework Drift Bound Same model, different scores on different setups. Is the gap noise or a real bug? Enter two scores with their (framework, dtype, batch, chat-template) — tool predicts the maximum allowable drift from numerical noise alone. If observed gap exceeds it → real bug, usually chat-template mismatch (lm-eval issue #1841) or KV-cache layout.

Your model gives 67.2 on lm-eval-hf and 65.1 on vLLM-served. Bug or noise? Enter both scores with (framework, dtype, batch, chat-template applied?). Tool predicts the noise band and flags real bugs. arxiv 2506.09501 documents this as a major eval reproducibility problem.

🔍 NIAH → Reasoning Gap NIAH (Needle in a Haystack) tests retrieval: "find this fact in long text". Multi-hop reasoning tests inference: "combine facts X+Y at the start with fact Z at the end". RULER paper (NVIDIA 2024) shows long-context models often pass NIAH but fail reasoning at the same context. This tool predicts both pass rates from architecture alone.

Your model claims 128k context. Will it actually reason at 64k, or just retrieve? Paste an HF model id and a target eval context — tool predicts NIAH and multi-hop reasoning pass rates, the gap, and a "safe context" where reasoning stays ≥65%.

📈 Benchmark Saturation Detector MMLU is saturated (88-94% all frontier models). Reporting "92% on MMLU" is now meaningless. This tool tells you which benchmarks still discriminate frontier models, which are saturated, and what to use instead. Data: DemandSphere AI Frontier Tracker (CC BY-NC 4.0) refreshed 2026-05.

Is your benchmark still useful? Pick a benchmark to see top-3 frontier scores, spread, and a verdict (saturated / near-saturated / discriminative) plus recommended replacements.

Data: DemandSphere AI Frontier Model Tracker (CC BY-NC 4.0) · HF Open LLM Leaderboard v3 (open-weight historical) · last fetch 2026-05-05.

🧭 Solutions Hub Map of every documented LLM-eval pain we know about: which tafagent mode addresses it (if any), and the best-of-breed external tools the community already trusts. Goal: full coverage. If a canonical tool exists elsewhere, we link rather than rebuild.

Don't reinvent — find. 30+ pains mapped to tafagent modes + curated external tools. Browse by category, search by keyword, or see the gaps where new modes would help most.

📋 Recipe

🎯 Inputs

📊 Verdict

🔍 Computation Chain

Every number below is deterministic Python. Click a step to expand.

💬 Plain-English Answer

📇 TAF Card — full model profile

🆚 Comparison Table

📂 Import a shared TAF result

Got a JSON file from someone else's TAF analysis? Load it here to see the verdict + chain locally. Same view as if you'd run it yourself.

🌐 Recent community submissions

Live feed from the public registry. Click any submission to view full analysis. Browse all →

🔬 Paper predictions — falsification status

The TAF framework rests on falsifiable predictions (F1-F23). Each is empirically tested. Here's the live status of every prediction in the paper.