ai-evaluation-tools

Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view

Open-source AI red teaming toolkit from Microsoft. Test generative AI systems with automated scans, adaptive multi-turn attacks, and a web UI.

Statistical inference for AI evaluations, from model comparisons to corrections for LLM judge bias, optimized for small sample sizes. All defaults battle-tested in Monte Carlo simulations.

Assessing the reliability, accuracy, and safety of large language models, RAG pipelines, and AI agents.

Comprehensive AI Model Evaluation Framework with advanced techniques including Temperature-Controlled Verdict Aggregation via Generalized Power Mean. Support for multiple LLM providers and 15+ evaluation metrics for RAG systems and AI agents.

A comprehensive, implementation-focused guide to evaluating Large Language Models, RAG systems, and Agentic AI in production environments.

MindTrial: Evaluate and compare AI language models (LLMs) on text-based tasks with optional file/image attachments and tool use. Supports multiple providers (OpenAI, Google, Anthropic, DeepSeek, Mistral AI, xAI, Alibaba, Moonshot AI, OpenRouter), custom tasks in YAML, and HTML/CSV/JSON reports.

Run a prompt against all, or some, of your models running on Ollama. Creates web pages with the output, performance statistics and model info. All in a single Bash shell script.

Comprehensive AI Evaluation Framework with advanced techniques including Temperature-Controlled Verdict Aggregation via Generalized Power Mean. Support for multiple LLM providers and 15+ evaluation metrics for RAG systems and AI agents.

An sdk and framework for evaluating and comparing multiple model outputs using configurable LLM-based jurors

prompt-evaluator is an open-source toolkit for evaluating, testing, and comparing LLM prompts. It provides a GUI-driven workflow for running prompt tests, tracking token usage, visualizing results, and ensuring reliability across models like OpenAI, Claude, and Gemini.

A reference implementation for learning and building AI evaluation systems.

Backend for Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations

🤖 Evaluate AI systems effectively with our comprehensive guide to methods, tools, and frameworks for assessing Large Language Models and agents.

可迁移、本地优先的 AI Agent 核心：长期记忆、心智模型、情感状态、召回评测与 fail-closed 治理。Portable reference implementation for local-first AI agent memory, mind modeling, affective state, retrieval evaluation, and fail-closed governance.

Official repository for Facet-Probe, an evaluation framework for auditing order sensitivity in multimodal large language models. You can find the corresponding pre-print here: https://arxiv.org/abs/2606.26079

Public research on LLM internals, Jacobian lenses, SAE steering, nonlinear dynamics, evaluation, and inspectable AI systems.

An epistemic AI lab for scientifically evaluating how language models track belief, truth, knowledge, inference, defeaters, and luck.

A lightweight black-box agent evaluator using YAML specifications to score task completion.

Evaluation suite for an HR AI agent — tool-use correctness, multi-step task completion, policy refusals and trace-level scoring with CI release gates