diff --git "a/pages/executive-summary/page.md" "b/pages/executive-summary/page.md" new file mode 100644--- /dev/null +++ "b/pages/executive-summary/page.md" @@ -0,0 +1,27 @@ +# Executive summary + + +--- + +We completed a full-scale portability reproduction of Microsoft Memora on all 10 LoCoMo conversations and 1,540 evaluated questions, replacing GPT-4.1-mini with GLM-5.2 and OpenAI embeddings with local BAAI/bge-small-en-v1.5. Neither headline LLM-judge claim reproduced: Policy reached 0.7877 versus 0.863 reported, and Semantic reached 0.7740 versus 0.849 reported; both were also below the paper full-context score of 0.825. In contrast, both strategies exceeded the paper F1 and BLEU values, revealing a divergence between lexical overlap and the substituted LLM judge. Paired analysis finds a small Policy advantage over Semantic in LLM correctness (+0.0136; 95% bootstrap CI +0.0019 to +0.0253; exact McNemar p=0.0275), concentrated in single-hop and temporal questions, while Semantic is slightly stronger on multi-hop. + +## Scope & cost + +| Item | Value | +|---|---| +| Dataset | Full LoCoMo: 10 conversations, 272 sessions, 1,986 raw / 1,540 evaluated questions | +| GPU / compute | No GPU; local CPU embeddings plus GLM OpenAI-compatible API | +| Measured wall time | Policy 6h16m44s; Semantic 1h00m30s; three-answer repair and re-evaluation 6m03s | +| Feasibility | One-day run with restartable conversation splits; API cost depends on provider pricing | +| Interpretation | Full-scale model-stack substitution, not exact same-model replication | + + +--- + +````html +policy_vs_semantic +````