Model Evaluation

A high-throughput and memory-efficient inference and serving engine for LLMs

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

An AI prompt optimizer for writing better prompts and getting better AI results.

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.

A Python library that extracts hundreds of statistical descriptions from time-ordered data and keeps only the ones related to your target.

Wraps checks around what a language model is asked and what it answers.

Open-source observability for your GenAI or LLM application, based on OpenTelemetry

🧊 Open source LLM observability platform. One line of code to monitor, evaluate, and experiment. YC W23 🍓

Open-source observability for AI agents. Find where your agents fail, dispatch your coding agent to fix it, and verify the fix against real traces.

Schema validation for DataFrames — the error that fires before a backtest quietly reports a fictional edge.

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

🕹️ Open-source, developer-first LLMOps platform designed to streamline prompt design, version management, instant delivery, collaboration, troubleshooting, observability and more.

OpenLIT is the open-source agent harness engineering platform: trace, evaluate, guard, and improve everything around the model in your AI agents, on OTEL.