| <!doctype html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Jev RAG Benchmark</title> |
| <style> |
| :root { color-scheme: light dark; } |
| body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; |
| margin: 0; padding: 2rem 1rem; line-height: 1.5; } |
| main { max-width: 1020px; margin: 0 auto; } |
| h1 { font-size: 1.6rem; margin: 0 0 .25rem; } |
| h2 { font-size: 1.1rem; margin: 2rem 0 .5rem; } |
| p.note { opacity: .75; font-size: .9rem; } |
| table { border-collapse: collapse; width: 100%; font-size: .9rem; margin: .5rem 0 1rem; } |
| th, td { border: 1px solid rgba(128,128,128,.35); padding: .45rem .6rem; text-align: left; } |
| th { background: rgba(128,128,128,.12); } |
| tr:nth-child(even) td { background: rgba(128,128,128,.06); } |
| footer { margin-top: 2.5rem; font-size: .85rem; opacity: .75; } |
| </style> |
| </head> |
| <body> |
| <main> |
| <h1>Jev RAG Benchmark</h1> |
| <p class="note">Free English benchmark of an open System One model as a reranking and decision layer in RAG.</p> |
| <h2>Reranking (frozen top-20 candidates)</h2> |
| <table><thead><tr><th>Dataset</th><th>n</th><th>Method</th><th>nDCG@10</th><th>Recall@5</th><th>MRR@10</th><th>Rerank p50</th></tr></thead><tbody><tr><td>scifact</td><td>300</td><td>A — no reranker (hybrid order)</td><td>71.67%</td><td>79.67%</td><td>68.75%</td><td>0 ms</td></tr><tr><td>scifact</td><td>300</td><td>T — TypeSafe Jev 1.13 batch noul</td><td>79.29%</td><td>85.67%</td><td>77.03%</td><td>4044 ms</td></tr><tr><td>scifact</td><td>300</td><td>N — NVIDIA cross-encoder reranker</td><td>78.70%</td><td>87.33%</td><td>76.16%</td><td>307 ms</td></tr><tr><td>xquad-en</td><td>1190</td><td>A — no reranker (hybrid order)</td><td>98.11%</td><td>99.58%</td><td>97.60%</td><td>0 ms</td></tr><tr><td>xquad-en</td><td>1190</td><td>T — TypeSafe Jev 1.13 batch noul</td><td>98.93%</td><td>99.66%</td><td>98.67%</td><td>4000 ms</td></tr><tr><td>xquad-en</td><td>1190</td><td>N — NVIDIA cross-encoder reranker</td><td>99.37%</td><td>99.66%</td><td>99.27%</td><td>409 ms</td></tr></tbody></table> |
| <h2>Probability calibration (candidate-level relevance)</h2> |
| <table><thead><tr><th>Dataset</th><th>Model</th><th>ECE (10 bin)</th><th>Brier</th><th>Top-1 accuracy</th><th>Top-1 confidence (correct)</th><th>Top-1 confidence (wrong)</th></tr></thead><tbody><tr><td>scifact</td><td>TypeSafe Jev 1.13 batch noul</td><td>0.0625</td><td>0.0366</td><td>71.00%</td><td>0.812</td><td>0.549</td></tr><tr><td>xquad-en</td><td>TypeSafe Jev 1.13 batch noul</td><td>0.0133</td><td>0.0045</td><td>97.90%</td><td>0.960</td><td>0.740</td></tr></tbody></table> |
| <h2>Frozen-context answer generation</h2> |
| <table><thead><tr><th>Dataset</th><th>Generator</th><th>Token F1</th><th>Exact match</th><th>F1 >= 0.5</th><th>Abstention</th><th>Valid citations</th></tr></thead><tbody><tr><td>xquad-en</td><td>T</td><td>diffusiongemma-26b</td><td>30.58%</td><td>1.26%</td><td>17.31%</td><td>2.61%</td><td>99.85%</td></tr></tbody></table> |
| <h2>RAG optimization mode (confidence-partitioned Jev, fixed t = 0.50)</h2> |
| <table><thead><tr><th>Dataset</th><th>Model</th><th>Threshold</th><th>A baseline nDCG@10</th><th>Always-on nDCG@10</th><th>Partitioned nDCG@10</th><th>Delta vs baseline</th><th>95% CI</th></tr></thead><tbody><tr><td>scifact</td><td>TypeSafe Jev 1.13 batch noul</td><td>0.50</td><td>71.67%</td><td>79.29%</td><td>75.81%</td><td>+4.14 pts</td><td>+2.13 to +6.29</td></tr><tr><td>xquad-en</td><td>TypeSafe Jev 1.13 batch noul</td><td>0.50</td><td>98.11%</td><td>98.93%</td><td>99.00%</td><td>+0.89 pts</td><td>+0.45 to +1.36</td></tr></tbody></table> |
| <footer>Generated 2026-09-24 12:10 UTC. Real runs only; fixture runs are never published. Code and raw results: <a href="https://github.com/emretheus/jev-rag-benchmark">https://github.com/emretheus/jev-rag-benchmark</a>. Independent benchmark, not affiliated with TypeSafe AI. All runs on free tiers.</footer> |
| </main> |
| </body> |
| </html> |
|
|
