Complementary evaluation: cost + latency + hallucination for hard reasoning LLMs
#3
by vigneshwar234 - opened
Hi NPHardEval team!
NP-Hard problem evaluation is a brilliant stress test for LLM reasoning. For teams using this benchmark to select models for complex optimization tasks, I built a complementary framework that adds production dimensions.
LLM Evaluation Framework measures:
- Reasoning Quality (1-10 CoT depth score — directly relevant to NP-Hard problem solving)
- Cost per 1K tokens (NP-Hard prompts are long, cost matters)
- Latency p95 (complex reasoning tasks have high tail latency)
- Accuracy (4-strategy cascade scorer)
- Hallucination Rate (models that confidently give wrong solutions to NP-Hard problems)
Live demo: https://huggingface.co/spaces/vigneshwar234/llm-eval-demo
GitHub: https://github.com/vignesh2027/LLM-Evaluation-Framework
Open source, free forever. Would love to discuss reasoning evaluation approaches!