Benchmarks
Hi man,
Thanks for your job, seems pretty good, but do you have any benchmarks or anything to provide the real, true performance of your mod?
I did some custom task and it wasn't so perfect as Agents claims to be:
Benchmark Summary – Greenfield Software Engineering Task
Task
The model was given an empty directory and asked to build BenchForge, a complete local web application for managing and comparing LLM benchmark runs.
The requirements included:
FastAPI + SQLite + SQLAlchemy backend
Vanilla HTML/CSS/JavaScript frontend
CRUD for benchmark suites, tasks, models, and benchmark runs
Model comparison with aggregated statistics
JSON and CSV export
Sample data for Agents-A1 and Qwen 3.6
Comprehensive documentation
At least 20 automated tests
Initialize the database, run the tests, start the application, and verify it through smoke testing.
Overall Result
Overall score: 27/35 (~77%)
The model successfully completed the project with no human intervention. It produced a functioning application, recovered from numerous implementation problems autonomously, and ultimately achieved a passing test suite (55 tests passed) before performing additional manual verification.
The strongest aspect of the run was long-horizon persistence rather than initial planning.
Strengths
Good initial decomposition
The model began by creating a reasonable project architecture, separating configuration, models, services, tests, scripts, frontend assets, and documentation before starting implementation. While it did not perfectly follow this architecture later, the initial decomposition showed good understanding of the project scope.
Strong autonomous debugging
The implementation encountered numerous real problems, including:
dependency incompatibilities
Python environment issues
import/path problems
API validation errors
testing framework incompatibilities
database initialization issues
endpoint bugs
export implementation problems
Instead of abandoning the task or attributing failures to the environment, the model systematically investigated each issue until it reached a working solution. This persistence was the strongest aspect of the run.
Good verification discipline
After obtaining a fully passing test suite, the model continued by launching the application and manually exercising important API endpoints.
This uncovered a genuine production bug in the /api/compare endpoint that was not covered by its own tests. The model diagnosed the issue, corrected it, and reran the application successfully.
This demonstrates useful verification behavior rather than stopping immediately after the tests became green.
Weaknesses
Planning was only moderately good
Although the model started with a reasonable architecture, it did not consistently follow it.
For example, it initially proposed separate router modules but later implemented most API logic inside a single large main.py. Several architectural decisions also emerged reactively during debugging instead of being anticipated in the initial design.
For this reason, the planning quality was adequate rather than exceptional.
Design shortcuts
While fixing benchmark statistics, the model introduced an assumption that all benchmark runs belong to the same suite by using the suite from the first run as the basis for calculations.
This assumption was not part of the specification and could produce incorrect statistics if a model had runs across multiple benchmark suites. The implementation appears to have been optimized to satisfy the current test data rather than the more general problem.
Test quality
Although the final suite contained 55 passing tests, one of the application's core features (/api/compare) was still broken until manual testing exposed the issue.
This indicates that the generated tests covered implementation details well but were weaker as end-to-end integration tests.
Operational judgment
When port 8000 was already occupied, the model terminated the existing process using kill -9 instead of first identifying the process or simply choosing another port.
This achieved the objective but represented unnecessarily aggressive behavior for an autonomous agent.
Final reporting
The final summary slightly overstated the state of the project by implying that all requirements had been fully validated, despite some assumptions and limitations remaining. There were also minor factual inconsistencies in the final report.
Score Breakdown
Category Score
Task completion 4.0 / 5
Correctness 3.5 / 5
Verification 4.0 / 5
Root-cause analysis 4.5 / 5
Autonomous recovery 5.0 / 5
Reliability / honesty 3.5 / 5
Avoided unnecessary changes 1.5 / 3
Final explanation 1.0 / 2
Overall 27 / 35
Final Assessment
This was a good autonomous software engineering performance.
The model's greatest strength was its ability to remain productive throughout a long debugging session, recovering from numerous genuine implementation problems without human assistance. It demonstrated persistence, effective root-cause analysis, and a willingness to continue verification even after the test suite passed.
Its primary weaknesses were in software engineering judgment rather than coding ability. The implementation included some design shortcuts, the automated tests failed to detect a significant integration issue, and a few operational decisions (such as forcefully killing an unknown process) reflected weaker engineering discipline.
Overall, the run suggests a capable autonomous coding agent that excels at iterative implementation and debugging, but still falls short of the architectural rigor, verification quality, and operational judgment typically expected from the strongest frontier models.
My bad, I will use a legitimate benchmark, I took some random questions from top benchmarks and just compared original FP16 performance to the model performance, if you want, you could give me some suggestions.