---
title: "Survey on Evaluation of LLM-based Agents"
url: https://stacklist.com/card/75394818-3927-400d-ab2f-56f509d6342d
source_url: "https://arxiv.org/abs/2503.16416"
stack: https://stacklist.com/c/technology/stack/29e01ac8-6202-4abf-ab14-a0613026186b
summary: "This survey provides the first comprehensive analysis of evaluation methods for LLM-based agents, examining core capabilities, application-specific benchmarks, generalist agent evaluation, benchmark dimensions, and evaluation frameworks. The paper identifies trends toward more realistic evaluations and highlights critical gaps in assessing cost-efficiency, safety, robustness, and scalable evaluation methods."
tags: "llm-agents, evaluation, benchmarks, artificial-intelligence, survey, planning, tool-use"
key_entities: "Asaf Yehudai (person), Lilach Eden (person), Alan Li (person), Guy Uziel (person), Yilun Zhao (person), Roy Bar-Haim (person), Arman Cohan (person), Michal Shmueli-Scheuer (person), arXiv (organization), LLM-based agents (concept), agent evaluation (concept), benchmarks (concept), ACL Findings (event), SWE agents (concept)"
classification: "reference"
content_hash: "sha256:616cee134c87206809de54674ac11ee41be4a8063e883bedcdfd2589b634ab06"
acp_version: "0.2"
token_counts_approximate: 1024
visibility: public
agent_accessible: true
status: "final"
---

# Survey on Evaluation of LLM-based Agents

--> Computer Science > Artificial Intelligence arXiv:2503.16416 (cs) [Submitted on 20 Mar 2025 ( v1 ), last revised 23 Apr 2026 (this version, v2)] Title: Survey on Evaluation of LLM-based Agents Authors: Asaf Yehudai , Lilach Eden , Alan Li , Guy Uziel , Yilun Zhao , Roy Bar-Haim , Arman Cohan , Michal Shmueli-Scheuer View a PDF of the paper titled Survey on Evaluation of LLM-based Agents, by Asaf Yehudai and 7 other authors View PDF HTML (experimental) Abstract: LLM-based agents represent a paradigm shift in AI, enabling autonomous systems to plan, reason, and use tools while interacting with dynamic environments. This paper provides the first comprehensive survey of evaluation methods for these increasingly capable agents. We analyze the field of agent evaluation across five perspectives: (1) Core LLM capabilities needed for agentic workflows, like planning, and tool use; (2) Application-specific benchmarks such as web and SWE agents; (3) Evaluation of generalist agents; (4) Analysis of agent benchmarks' core dimensions; and (5) Evaluation frameworks and tools for agent developers. Our analysis reveals current trends, including a shift toward more realistic, challenging evaluations with continuously updated benchmarks. We also identify critical gaps that future research must address, particularly in assessing cost-efficiency, safety, and robustness, and in developing fine-grained, scalable evaluation methods. Comments: ACL Findings Subjects: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG) Cite as: arXiv:2503.16416 [cs.AI]   (or arXiv:2503.16416v2 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2503.16416 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Asaf Yehudai [ view email ] [v1] Thu, 20 Mar 2025 17:59:23 UTC (99 KB) [v2] Thu, 23 Apr 2026 17:36:18 UTC (115 KB) Full-text links: Access Paper: View a PDF of the paper titled Survey on Evaluation of LLM-based Agents, by Asaf Yehudai and 7 other authors View PDF HTML (experimental) TeX Source view license Current browse context: cs.AI < prev   |   next > new | recent | 2025-03 Change to browse by: cs cs.CL cs.LG References & Citations NASA ADS Google Scholar Semantic Scholar export BibTeX citation Loading... BibTeX formatted citation × loading... Data provided by: Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? )