LLM Observability, Evaluation & Monitoring
19 companies
0 of 0 categories open
Relationship:
Loading the map…
On this page
Tooling for seeing what a model-backed application actually did in production: tracing calls, logging prompts and responses, scoring output quality, and catching regressions between versions. Some entries are dedicated LLM tools. Others are established observability vendors that bolted model-specific features onto what they already had.
What to compare
Check the tracing integration first. If it does not fit the framework you already run, nothing else about the product matters. Then compare whether evaluation happens offline, online or both, whether scoring is automated, human or model-graded, and how prompts and responses get stored, because this category handles the most sensitive data of any tooling layer.
19 companies in this category
Arize AIML and LLM observability platform for monitoring model performance.
Athina AIEvaluation and observability platform for LLM-powered products.
BraintrustEvaluation and observability platform for LLM applications.
Comet MLML experiment tracking platform with LLM evaluation features.
Confident AI (DeepEval)Evaluation platform built around the open-source DeepEval framework.
Datadog LLM ObservabilityDatadog's module for tracing and monitoring LLM applications.
Fiddler AIAI observability platform for monitoring model performance and bias.
GalileoEvaluation and observability platform for LLM and agent quality.
GiskardOpen-source testing framework for ML and LLM model quality/safety.
HumanloopPlatform for prompt management, evaluation, and LLM observability.
LangSmithLangChain's platform for tracing, testing, and monitoring LLM apps.
LangfuseOpen-source LLM observability and analytics platform.
New Relic AI MonitoringNew Relic's observability tooling extended to AI/LLM applications.
Patronus AIAutomated evaluation and guardrails platform for LLM outputs.
PromptLayerPrompt management and logging platform for LLM applications.
TraceloopOpen-source-based observability platform for LLM applications (OpenLLMetry).- TruEraAI quality and observability platform (now part of Snowflake).
Weights & BiasesWidely used platform for experiment tracking and model/LLM evaluation.- WhyLabsAI observability platform focused on data and model quality monitoring.