---
title: "AgentRewardBench"
url: https://stacklist.com/card/1d05f8cd-8fb8-43dc-9225-d9a5b3a49f25
source_url: "https://agent-reward-bench.github.io/"
stack: https://stacklist.com/c/technology/stack/29e01ac8-6202-4abf-ab14-a0613026186b
summary: "AgentRewardBench is a benchmark and Python library for evaluating automatic evaluators of web agent trajectories, including LLM judges. It provides tools, environments, evaluation metrics, a Hugging Face dataset, and a leaderboard for ranking automatic evaluators across different benchmarks."
tags: "agent-evaluation, web-agents, benchmark, llm-judges, trajectories, reward-model, python-library"
key_entities: "AgentRewardBench (technology), Xing Han Lù (person), Amirhossein Kazemnejad (person), Nicholas Meade (person), Siva Reddy (person), Christopher J. Pal (person), McGill-NLP (organization), Hugging Face Hub (technology), web agent trajectories (concept), LLM judges (concept)"
classification: "framework"
content_hash: "sha256:9465a3e299b570373ee6c742dcf5e223105c8355f6747fa6a8f26fa4ec3fb1e8"
acp_version: "0.2"
token_counts_approximate: 606
visibility: public
agent_accessible: true
status: "final"
---

# AgentRewardBench

AgentRewardBench 💾 Code 📄 Paper 🌐 Website 🤗 Dataset 💻 Demo 🏆 Leaderboard AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories Xing Han Lù , Amirhossein Kazemnejad* , Nicholas Meade , Arkil Patel , Dongchan Shin , Alejandra Zambrano , Karolina Stańczak , Peter Shaw , Christopher J. Pal , Siva Reddy *Core Contributor Using the agent-reward-bench library This library provides tools for evaluating the performance of agents in various environments. It includes a set of environments, a set of agents, and a set of evaluation metrics. To install the library: pip install agent-reward-bench You can now import the library in your Python code: # Using agents and environments: import agent_reward_bench.modeling as arbm import agent_reward_bench.benchmarks as arbb # Using the judge for evaluating agents: import agent_reward_bench.judge as arbj from agent_reward_bench.judge.existing import aer , nnetnav from agent_reward_bench.judge.args import default_judge_args , judge_args See run_judge.py for an example of how to use the library. Dataset You can use the huggingface_hub library to load the dataset. The dataset is available on Huggingface Hub at McGill-NLP/agent-reward-bench . Install the hugginface-hub with: pip install huggingface_hub from huggingface_hub import snapshot_download # Download the dataset to ./trajectories/ snapshot_download ( repo_id = "McGill-NLP/agent-reward-bench" , repo_type = "dataset" , local_dir = "./trajectories/" ) Leaderboard You can find the ranking of automatic evaluators, including LLM judges, on the Huggingface leaderboard . You can also see the breakdown by benchmark, allowing you to find the best performing LLM judge for your tasks. Submission To submit your results, please open an issue on the GitHub repository with your results and a link to your paper/artifacts. Citation If you use AgentRewardBench in your research, please cite the following paper: @misc { lù2025agentrewardbenchevaluatingautomaticevaluations , title = {AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories} , author = {Xing Han Lù and Amirhossein Kazemnejad and Nicholas Meade and Arkil Patel and Dongchan Shin and Alejandra Zambrano and Karolina Stańczak and Peter Shaw and Christopher J. Pal and Siva Reddy} , year = {2025} , eprint = {2504.08942} , archivePrefix = {arXiv} , primaryClass = {cs.LG} , url = {https://arxiv.org/abs/2504.08942} , }