AI agent evaluation frameworks

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks. It provides tools and resources for assessing the performance of language models effectively.

This page provides an overview of evaluation concepts within the LangChain framework. It covers key principles and methodologies for assessing the performance of language models and their applications.

DeepEval's 5-minute quickstart guide walks users through installing the framework, creating an LLM test case with input/output pairs, choosing a GEval metric, and running end-to-end evaluations locally.

Ragas is an evaluation framework designed to assess the performance of AI applications. It provides tools and methodologies to ensure the effectiveness and reliability of AI systems.

This page provides a comprehensive guide on how to measure the performance of machine learning models using the Evaluations feature in Phoenix. It covers the setup process, key metrics to consider, and best practices for effective evaluation.

Measure AI application quality, detect regressions before they reach production, and build confidence that your system is improving over time.

Inspect is an open-source framework designed for evaluating large language models. It provides tools and methodologies to assess the performance and capabilities of various language models effectively.

With Langfuse you can capture all your LLM evaluations in one place. You can combine a variety of different evaluation metrics like model-based evaluations (LLM-as-a-Judge), human annotations or fully custom evaluation workflows via API/SDKs. This allows you to measure quality, tonality, factual accuracy, completeness, and other dimensions of your LLM application.

TruLens provides tools for evaluating and tracing the performance of AI agents. It aims to enhance the understanding and reliability of AI systems through comprehensive analysis.

SWE-bench is a project that explores the capability of language models in addressing real-world issues found on GitHub. It aims to evaluate how effectively these models can assist developers in resolving software-related problems.