JARVIS-1: Open-World Multi-Task Agents With Memory-Augmented Multimodal Language Models

IEEE Transactions on Pattern Analysis & Machine Intelligence


LongCodeZip: Compress Long Context for Code Language Models

2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)


HeroBench: A Benchmark for Long-Horizon Planning and Structured Reasoning in Virtual Worlds


The Berkeley Function Calling Leaderboard (BFCL): From tool use to agentic evaluation of large language models

Proceedings of the 42nd International Conference on Machine Learning


Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale

International Conference on Machine Learning


BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software


MOSS: Self-Evolution through Source-Level Rewriting in Autonomous Agent Systems


Why Do Multi-Agent LLM Systems Fail?

Advances in Neural Information Processing Systems


MLE-bench: Evaluating machine learning agents on machine learning engineering


SPIN: Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

Proceedings of the 41st International Conference on Machine Learning


IterResearch: Rethinking Long-Horizon Agents with Interaction Scaling


Reinforcement Learning for Long-Horizon Interactive LLM Agents


MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research


Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?


Toward autonomous long-horizon engineering for ML research


HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry


KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation


The Past Is Prologue: A Plug-in Controller for Selective Updates in Sequentially Evolving LLM Memory


Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops


Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory

European Conference on Artificial Intelligence (ECAI)


Process Reward Models for LLM Agents: Practical Framework and Directions


CI-Repair-Bench: A Repository-Aware Benchmark for Automated Patch Validation via CI Workflows


Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios


DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback

International Conference on Learning Representations


DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks


Mind2Web: Towards a Generalist Agent for the Web


SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution


Towards Long-Horizon Agents: A Survey


DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents


Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks


Memp: Exploring Agent Procedural Memory

Findings of the Association for Computational Linguistics: ACL 2026


FeatureBench: Benchmarking Agentic Coding for Complex Feature Development


Group-in-Group Policy Optimization for LLM Agent Training


FrontierSWE


A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence


Robotouille: An Asynchronous Planning Benchmark for LLM Agents


CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing


Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge


Ask Early, Ask Late, Ask Right: When Does Clarification Timing Matter for Long-Horizon Agents?


From Question Answering to Task Completion: A Survey on Agent System and Harness Design


HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models


RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation


WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models

Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)


MinePlanner: A Benchmark for Long-Horizon Planning in Large Minecraft Worlds

Proceedings of the 6th ICAPS Workshop on the International Planning Competition (WIPC)


Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning


Automated design of agentic systems


Memory in the Age of AI Agents


Step-DeepResearch Technical Report


Large Language Models Cannot Self-Correct Reasoning Yet


From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills


ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering


CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation

Findings of the Association for Computational Linguistics: ACL 2025


AIDE: AI-driven exploration in the space of code


DARWIN: Dynamic Agentically Rewriting Self-Improving Network


SWE-bench: Can Language Models Resolve Real-World GitHub Issues?


ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance


ReVeal: Self-Evolving Code Agents via Reliable Self-Verification

The Fourteenth International Conference on Learning Representations


When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs

Transactions of the Association for Computational Linguistics


ACON: Optimizing Context Compression for Long-horizon LLM Agents


OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web


Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation


Process Reward Models That Think

Transactions on Machine Learning Research


Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents


Towards a Science of Scaling Agent Systems


Kimi K3: Open Frontier Intelligence


Can AI Agents Conduct Open-Ended AI Research? Early Evidence from Two Case Studies


VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks


Squeez: Task-Conditioned Tool-Output Pruning for Coding Agents


Autodata: An Agentic Data Scientist to Create High Quality Synthetic Data


Measuring AI Ability to Complete Long Software Tasks


LLMs Get Lost In Multi-Turn Conversation


ShinkaEvolve: Towards open-ended and sample-efficient program evolution


Benchmarking Mobile Device Control Agents across Diverse Configurations


Learning to Contextualize Web Pages for Enhanced Decision Making by LLM Agents

The Thirteenth International Conference on Learning Representations


Meta-Harness: End-to-End Optimization of Model Harnesses


Recursive harness self-improvement


BEHAVIOR-1K: A benchmark for embodied AI with 1,000 everyday activities and realistic simulation

Proceedings of The 6th Conference on Robot Learning


SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graph

Findings of the Association for Computational Linguistics: EACL 2026


Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalization


WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Research


ACM: Agentic Context Management for Long Horizon Tasks


AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery


CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents


Agent Harness Engineering: A Survey


WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces


KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta


The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management


AgentBench: Evaluating LLMs as Agents


ML-Master: Towards AI-for-AI via integration of exploration and reasoning


Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems


Escher-Loop: Mutual Evolution by Closed-Loop Self-Referential Optimization


Do LLMs Catch Their Own Mistakes? A Comprehensive Benchmark for Reflective Tool Use LLMs


LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces


Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments

Findings of the Association for Computational Linguistics: EMNLP 2025


The AI Scientist: Towards fully automated open-ended scientific discovery


Towards End-to-End Automation of AI Research


The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?


WebLINX: Real-World Website Navigation with Multi-Turn Dialogue


ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities

Findings of the Association for Computational Linguistics: NAACL 2025


AIRS-Bench: A Suite of Tasks for Frontier AI Research Science Agents


MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI


TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration


DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems


Self-Refine: Iterative refinement with self-feedback


LLM Critics Help Catch LLM Bugs


Agent Harness for Large Language Model Agents: A Survey


RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement


Solving a Million-Step LLM Task with Zero Errors


Evaluation and Benchmarking of LLM Agents: A Survey

Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2


Magentic-UI: Towards Human-in-the-loop Agentic Systems


WebGPT: Browser-assisted question-answering with human feedback


Code as Agent Harness


NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents


AlphaEvolve: A coding agent for scientific and algorithmic discovery


OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation


Introducing SWE-bench Verified


Is there a half-life for the success rates of AI agents?


PARC: An Autonomous Self-Reflective Coding Agent for Robust Execution of Long-Horizon Tasks


SkillOS: Learning Skill Curation for Self-Evolving Agents


MemGPT: Towards LLMs as Operating Systems


Natural-Language Agent Harnesses


WebCanvas: Benchmarking Web Agents in Online Environments


Generative Agents: Interactive Simulacra of Human Behavior

Proceedings of the 36th annual acm symposium on user interface software and technology


Gorilla: Large Language Model Connected with Massive APIs

Advances in Neural Information Processing Systems


PostTrainBench: Can LLM Agents Automate LLM Post-Training?


ProgramBench: Can Language Models Rebuild Programs From Scratch?


Bilevel Autoresearch: Meta-Autoresearching Itself


LUMINA: Long-horizon Understanding for Multi-turn Interactive Agents


AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents


Structured Feedback Improves Repair in an LLM Agent Loop


ReCUBE: Evaluating Repository-Level Context Utilization in Code Generation


Self-Improvements in Modern Agentic Systems: A Survey


RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository


RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades


Toolformer: Language Models Can Teach Themselves to Use Tools


AgentRxiv: Towards Collaborative Autonomous Research


Self-Adapting Language Models


Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI


SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring


Reflexion: language agents with verbal reinforcement learning


ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks

Proceedings of the IEEE/CVF conference on computer vision and pattern recognition


Towards Execution-Grounded Automated AI Research


Schema First Tool APIs for LLM Agents: A Controlled Study of Tool Misuse, Recovery, and Budgeted Performance


PaperBench: Evaluating AI's ability to replicate AI research


Scaffold Effects on GAIA: A Controlled Comparison


Cognitive Architectures for Language Agents


AdaPlanner: Adaptive Planning from Feedback with Language Models


Scaling Long-Horizon LLM Agent via Context-Folding


SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution


SWE-bench Goes Live!


SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?


SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades


SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration


SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle


SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios


SWE-Explore: Benchmarking How Coding Agents Explore Repositories


SWE-InfraBench: Evaluating Language Models on Cloud Infrastructure Code


SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?


SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks

Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations


SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale


Practical Considerations for Agentic LLM Systems


Reinforced Agent: Inference-Time Feedback for Tool-Calling Agents

Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)


Hindsight Credit Assignment for Long-Horizon LLM Agents


AI-Researcher: Autonomous Scientific Innovation


FARS: A Fully Automated Research System Deployed at Scale


Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution


Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces


ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

International Conference on Learning Representations


AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents


HarnessOpt-Bench: Evaluating LLMs at Harness Optimization


SWE-Dev: Building Software Engineering Agents with Training and Inference Scaling


Voyager: An Open-Ended Embodied Agent with Large Language Models


Agent Workflow Memory

Forty-second International Conference on Machine Learning


Hell or High Water: Evaluating Agentic Recovery from External Failures


OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows


Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0


Context Compression for LLM Agents: A Survey of Methods, Failure Modes, and Evaluation


DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration


The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break


MetaSkill-Evolve: Recursive Self-Improvement of LLM Agents via Two-Timescale Meta-Skill Evolution


NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?


Rethinking the Evaluation of Harness Evolution for Agents


CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities


SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents


The Verification Horizon: No Silver Bullet for Coding Agent Rewards


First Evidence of Recursive Self-Improvement


Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models


From Agent Loops to Structured Graphs:A Scheduler-Theoretic Framework for LLM Agent Execution


DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressively


RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts


WorkArena: How Capable are Web Agents at Solving Common Knowledge Work Tasks?

Proceedings of the 41st International Conference on Machine Learning


World of Bits: An Open-Domain Platform for Web-Based Agents

Proceedings of the 34th International Conference on Machine Learning


ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization


ContextBudget: Budget-Aware Context Management for Long-Horizon Search Agents


ContextWeaver: Selective and Dependency-Structured Memory Construction for LLM Agents


When History Lies: Evaluating and Improving Tool Use under Misleading Multi-Turn Histories


AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning


Demystifying LLM-Based Software Engineering Agents

Proceedings of the ACM on Software Engineering


Reducing Cost of LLM Agents with Trajectory Reduction


OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments


TravelPlanner: A Benchmark for Real-World Planning with Language Agents

International Conference on Machine Learning


Adapting the Interface, Not the Model: Runtime Harness Adaptation for Deterministic LLM Agents


ASI-Evolve: AI Accelerates AI


LoopTrap: Termination Poisoning Attacks on LLM Agents


A-MEM: Agentic Memory for LLM Agents


TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track


The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search


Beyond Static Summarization: Proactive Memory Extraction for LLM Agents


Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering


WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents


ReAct: Synergizing Reasoning and Acting in Language Models

International Conference on Learning Representations (ICLR)


Tree of Thoughts: Deliberate Problem Solving with Large Language Models


τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains


ARC: Active and Reflection-driven Context Management for Long-Horizon Information Seeking Agents

Findings of the Association for Computational Linguistics: ACL 2026


AgentFold: Long-Horizon Web Agents with Proactive Context Management


A Survey on Evaluation of LLM-based Agents


Learning Agent-Compatible Context Management for Long-Horizon Tasks


Godel Agent: A self-referential agent framework for recursive self-improvement


Self-rewarding language models


OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks


STaR: Bootstrapping reasoning with reasoning


Self-Taught Optimizer (STOP): Recursively self-improving code generation


DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale

Findings of the Association for Computational Linguistics: ACL 2025


Darwin Godel Machine: Open-ended evolution of self-improving agents


Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models


DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints


How Far Are We From True Auto-Research?


Self-Harness: Harnesses That Improve Themselves


Stop Comparing LLM Agents Without Disclosing the Harness

Second Workshop on Agents in the Wild: Safety, Security, and Beyond


ExpeL: LLM Agents Are Experiential Learners


SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents


NATURAL PLAN: Benchmarking LLMs on Natural Language Planning


Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models


WebArena: A Realistic Web Environment for Building Autonomous Agents


MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents


Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering


Are We Ready For An Agent-Native Memory System?


Where LLM Agents Fail and How They can Learn From Failures


Toward ultra-long-horizon agentic science: Cognitive accumulation for machine learning engineering


Agent-as-a-Judge: Evaluate Agents with Agents