In simple English
When you build an app that talks to artificial intelligence, things can go wrong silently. This tool watches every message you send to models like OpenAI or Anthropic. It records how long each request takes and how much it costs. You can send this data to your existing monitoring dashboard. It is free to use under an open-source licence.
What it is
OpenLLMetry is a set of extensions built on top of OpenTelemetry (a standard way to collect application logs and metrics). It adds automatic tracking for artificial intelligence providers, vector databases and agent frameworks in Python.
What it is good for
Solo developers building AI applications who need to debug failed prompts, track token costs or trace multi-step agent runs without locking into a proprietary monitoring service.
- Supports dozens of artificial intelligence providers and vector databases out of the box.
- Sends logs to your existing monitoring stack like Datadog or Honeycomb.
- Requires only two lines of code to start tracing your application.
- Requires familiarity with Python and basic application monitoring concepts.
- High volume of detailed trace logs can increase storage costs on your logging backend.
- Frequent updates by artificial intelligence providers can cause tracking integrations to break.
manual print statements and custom logging for artificial intelligence API calls
OpenTelemetry collector for routing monitoring data to your preferred dashboard
- vibrantlabsai/ragas16k stars
Evaluates the quality of retrieval-augmented generation applications rather than tracing requests.
- Arize-ai/phoenix12k stars
Open-source visualization and evaluation tool for artificial intelligence applications that also uses OpenTelemetry.
- nicolargo/glances34k stars
A live view of one computer's health, in a terminal, in a browser, or through an AI assistant.
- dokku/dokku32k stars
A small platform you install on one server, which builds and runs an app every time you push code to it.
- netbox-community/netbox22k stars
A database of what a network and its hardware are supposed to look like, which other programs can read.
pip install traceloop-sdk