AI Observability: What Should You Monitor in AI Applications?

What is AI Observability

AI observability is often described as “monitoring LLMs”, but that definition is way too narrow. There is so much more that AI applications do that needs monitoring.

A production AI application can involve an LLM, multiple model providers, prompts, embeddings, vector databases, RAG pipelines, agents, tools, external APIs, memory, guardrails, GPUs, and traditional application infrastructure. An AI application can also be perfectly healthy from an infrastructure perspective while producing inaccurate, unsafe, expensive, or completely useless results.

All these are aspects of AI that could and should be observed.

In traditional IT observability we asked questions such as: Is the application available? Is it fast? Are there errors? Are resources healthy? For AI observability we have to go further: What did the AI do? Why did it do it? Was the result good? Was it safe? How much did it cost? And did it actually accomplish what the user wanted?

In this article, I explore the different dimensions of AI observability and the tools available for monitoring them.

1. AI Infrastructure

At the foundation is the infrastructure running your AI workloads.

For self-hosted models, this includes the usual infrastructure metrics (CPU, memory, disk, network, availability…), but also GPU utilization, GPU memory, accelerator utilization, model-loading time, batch sizes, queue depth, and concurrency. GPU utilization is particularly important because an expensive GPU that spends most of its time idle is an expensive resource that’s not doing much work.

Model-serving performance also introduces metrics that aren’t particularly common in traditional application monitoring. Time to first token (TTFT), time between tokens, tokens per second, queueing time, and total generation time can have a direct impact on user experience.

Tools such as Datadog, New Relic, and Dynatrace can monitor the infrastructure and application components surrounding AI workloads. For organizations running their own GPU infrastructure, specialized GPU monitoring can be added to provide deeper visibility into accelerator utilization and performance.

2. LLM Requests and Responses

Every call to an LLM is an important observable event.

You want to know which model was called, which provider served the request, how long it took, whether it succeeded, how many input and output tokens were consumed, what parameters were used, and why generation stopped. This is the AI equivalent of monitoring or tracing HTTP requests or database queries, but with considerably more semantic information.

This data is particularly useful when something changes. A new model version may increase latency. A prompt change may double token consumption. A provider may start returning more errors. A model that looked inexpensive during development may become surprisingly expensive in production.

Langfuse provides open-source LLM tracing with visibility into requests, responses, latency, tokens, and costs. W&B Weave provides similar tracing capabilities and connects LLM calls to evaluations and experiments, making it easier to understand not just how an individual request performed but whether changes to the application improved it.

3. End-to-End AI Traces

A user request rarely results in a single model call.

Consider a customer asking an AI assistant a question. The application might retrieve some documents, send them to an LLM, have the LLM decide to call a tool, invoke an external API, send the result back to the model, make another model call, and finally generate the answer.

A single LLM trace isn’t enough to understand this. You need an end-to-end trace showing the entire execution path and the relationships between its individual operations. That allows you to answer questions such as Why did this request take 15 seconds?, Which model call was responsible?, or Why did the agent call this API?

While there are commercial tools built around tracing and evaluation of AI applications, OpenTelemetry is also becoming increasingly relevant here. Its emerging GenAI conventions provide a standardized way to represent AI operations alongside traditional distributed traces.

4. Prompts and Context

With conventional software, developers generally think of code as the program.

With LLM applications, the prompt is part of the program.

Changing a system prompt, adding an instruction, modifying a few-shot example, changing the conversation history, or altering the model parameters can dramatically change application behavior. Consequently, observing which prompt was used and even which version of that prompt was used, is essential for debugging.

Context is equally important. To be meaningful, a production trace may need to show the system instructions, conversation history, retrieved documents, user context, tool descriptions, and other information supplied to the model.

This information can contain sensitive data, so prompt and context observability needs appropriate access controls, redaction, and retention policies.

Langfuse combines observability with prompt management and versioning, making it possible to associate production traces with the prompt version that generated them. Arize Phoenix also provides prompt management and experimentation capabilities. Sematext AI Agent Watch looks for Personally Identifiable Information (PII) in prompts.

5. AI Output Quality

This is where AI observability starts to diverge significantly from traditional observability.

A conventional application can return HTTP 200 and still be wrong, but in most applications a successful response provides a reasonably strong indication that the operation worked. With AI, a perfectly successful API request can produce a completely useless answer.

AI observability therefore needs to measure things such as correctness, relevance, completeness, helpfulness, instruction-following, and format compliance.

Some of these properties can be evaluated deterministically. Others require reference answers, human evaluation, specialized models, or LLM-as-a-judge techniques. The important point is that an AI system needs to be observed at the level of behavior and outcomes, not just infrastructure.

We need to evaluate the quality of AI outputs for the same reason that traditional information retrieval systems. For comparison, there we had very established metrics like precision, recall, mean reciprocal rank, and normalized discounted cumulative gain, etc.

Arize Phoenix provides evaluators that can be applied to production traces, including LLM-as-a-judge evaluations. W&B Weave similarly supports custom scorers and LLM judges, allowing teams to continuously evaluate production behavior.

6. Hallucinations and Groundedness

Few things are more important to monitor in generative AI than hallucinations.

As we’ve all seen, an LLM can produce an answer that sounds authoritative and convincing while being completely fabricated. Monitoring hallucinations means determining whether the answer is supported by the information available to the model and whether the claims it makes are consistent with that information.

This is particularly important for RAG applications. When an answer is wrong, you want to know whether the problem was retrieval, generation, or both. Did the system retrieve the wrong documents? Did it retrieve the right documents but fail to use them? Or did the model simply invent information despite having the correct context?

Ragas is an open-source framework focused heavily on evaluating RAG applications, including metrics around faithfulness and answer relevance. Arize Phoenix provides RAG and grounding-related evaluations that can be connected directly to application traces.

7. RAG and Retrieval

Similar to the above, for every retrieval operation, you may want to know the query that was generated, which documents or chunks were returned, their relevance scores, how many were retrieved, which ones actually made it into the prompt, and how long retrieval took.

This lets you distinguish between “the LLM couldn’t answer the question” and “the information required to answer the question was never retrieved.”

You can also monitor the components underneath retrieval: embedding generation, embedding-model versions, chunking strategies, vector search latency, filtering, reranking, and retrieval relevance. Changes to any of these can affect the final answer even when the LLM itself hasn’t changed.

Arize Phoenix provides tracing and evaluation for RAG pipelines, while W&B Weave can trace retrieval operations and evaluate the quality of retrieved information alongside the generated answer.

8. AI Agents

AI agents make observability considerably more complicated.

A conventional LLM application might send a prompt and receive an answer. An agent can decide what to do next, and that can include a whole bunch of things. It can call a tool, inspect the result, make another decision, call another tool, retry an operation, delegate to another agent, and eventually take an action.

For agents, observability therefore needs to capture the trajectory of an execution: what decisions were made, what tools were selected, what arguments were supplied, what results came back, how many iterations occurred, and how the agent ultimately reached its outcome.

This is one of the areas where AI observability is evolving particularly quickly and Sematext AI Agent Watch excels at capturing this.

9. Tool Calls and Actions

Tools deserve special attention because they turn AI from something that generates text into something that can do things, and that opens a whole new can of security worms.

An agent might have access to a database, browser, CRM, ticketing system, email, internal API, payment system, or cloud infrastructure. AI observability solutions have to be able to show which tools were invoked, when they were invoked, what arguments were passed, what they returned, and whether the action succeeded.

Tool monitoring is also where AI observability starts overlapping heavily with security.

An agent might make a technically valid API call but use the wrong API, access data it shouldn’t access, repeatedly retry an expensive operation, or perform an action that violates organizational policy. Monitoring tool usage creates both an operational record and a behavioral baseline.

Sematext AI Agent Watch, Langfuse, LangSmith, and Arize Phoenix all expose tool calls as part of AI execution traces. OpenTelemetry is also working toward standardized semantic conventions for tool execution, making tool calls increasingly interoperable with traditional distributed tracing.

10. AI Memory and State

Many agentic applications maintain state between interactions.

That state might contain conversation history, user information, summaries, retrieved facts, intermediate results, or persistent agent memory. Observability should show what was written to memory, what was subsequently retrieved, and how that information affected the agent’s behavior.

Memory creates some unusual failure modes. An agent can remember something that is wrong, retrieve stale information, accumulate excessive context, expose information from one user to another, or become increasingly expensive as its context grows.

Langfuse and LangSmith can incorporate application state and intermediate operations into their traces. OpenTelemetry’s emerging GenAI conventions also recognize memory operations as an observable part of AI application execution.

11. Token Usage and Cost

Every model call potentially has a price attached to it. This aspect of AI monitoring is probably the most familiar to everyone because we all experienced the need for this information the moment we started using AI tools and paying for them.

Observability should therefore track input tokens, output tokens, model-specific costs, cost per request, cost per user, cost per application, and cost over time. For agentic applications, it can be particularly useful to identify which steps in an agent trajectory are consuming the most tokens and money.

I should note that cost monitoring isn’t merely an accounting exercise. It is not of interest only to the CFO and the company management. Unexpected token consumption can reveal poorly designed prompts, unnecessarily large context windows, inefficient RAG pipelines, or agents caught in loops. As such, having insight into token usage and cost also matters for engineering teams. Especially if they want to preempt having their managers or CFOs on their backs.

Langfuse has token and cost tracking built directly into its LLM observability capabilities. W&B Weave exposes token usage and cost alongside traces and evaluations, making it possible to correlate cost with latency and quality. LLM token usage and cost monitoring is also built into Sematext AI Agent Watch, plus it allows pricing comparison across all known models, not just those used in your organization or by your AI applications.

12. Model and Prompt Drift

How many new models and model versions have we seen in the last 12 months? It would take me a while to list them all. Well, no, I could just ask AI, couldn’t I? But my poins is that AI applications don’t stay static, far from it.

Models are upgraded. Providers change model versions. Prompts evolve. Retrieval data changes. Fine-tuned models are retrained. User populations change. All of these can and do alter application behavior.

Observability therefore needs to identify behavioral changes over time and help correlate those changes with what changed in the system. A sudden decline in answer quality may have nothing to do with application code, it could be the result of a model update or a change to the retrieval dataset.

W&B Weave provides versioning and experiment tracking for models, prompts, datasets, and other components of AI applications. MLflow provides open-source tracking of models, experiments, versions, and deployments, making it particularly useful for organizations operating both traditional ML and generative AI systems.

13. AI Security

AI introduces an entirely new class of security problems. We have already seen some, like zero-day vulnerabilities being discovered (which is all fine until the wrong people do that first).

These include prompt injection, jailbreaks, sensitive-data exposure, malicious inputs, unsafe outputs, data exfiltration, excessive permissions, tool abuse, and attacks against AI supply chains.

Security observability needs to monitor not only what users send to the AI system but also what the AI system attempts to do. For an agent, that can mean detecting unusual tool sequences, attempts to access sensitive resources, policy violations, or behavior that deviates from established patterns.

This AI security monitoring is one of the key foci of Sematext AI Agent Watch. Lakera focuses on runtime protection for generative AI applications, including threats such as prompt injection and sensitive-data exposure. NVIDIA NeMo Guardrails is an open-source framework for implementing programmable controls around AI applications. Protect AI focuses more broadly on security across the machine-learning and AI software supply chain.

14. AI Inventory and Governance

There is one final problem that becomes increasingly important as organizations deploy, knowingly or not, AI at scale:

Do you actually know what AI you have running?

An organization may have dozens or hundreds of models, LLM applications, AI agents, MCP servers, vector databases, and autonomous workflows. Some may have been formally approved. Others may have been deployed by individual engineering teams or employees without centralized visibility.

AI observability therefore increasingly overlaps with AI governance. Organizations need to know which models and agents exist, who owns them, which models they use, what data they can access, what tools they can invoke, and what policies govern their behavior.

This is especially important for autonomous agents. Traditional monitoring asks whether a service is healthy. AI governance asks a different question:

Is this AI allowed to be doing what it is doing?

15. Responsible AI

This may feel a little abstract to some readers, and it’s rather different from the more “techy” aspects of AI monitoring, but it is really important at the level of our global society. AI systems can fail in ways that aren’t captured by conventional notions of correctness.

Depending on the application, teams may need to monitor toxicity, harmful content, bias, inappropriate refusals, unfair treatment of different user groups, or systematic differences in model performance.

These properties are usually best monitored through continuous evaluation of production traffic, combined with segmentation and human review. The goal isn’t simply to find one problematic response. It is to discover patterns.

Arize Phoenix supports automated evaluations, human annotations, and user feedback, allowing safety and quality signals to be attached to individual traces. W&B Weave supports production monitors that can use evaluators and LLM judges to score live application behavior for characteristics such as correctness, helpfulness, and other application-specific criteria.

16. User Feedback and Business Outcomes

Ultimately, AI systems are deployed to accomplish something useful. Without that, what’s the point?

That might mean answering a customer’s question, resolving a support case, helping a developer write code, recommending a product, detecting fraud, or completing an agentic workflow. AI observability should therefore connect AI telemetry to real-world outcomes.

This is easier said that done, but at the end of the day we want to be able to answer questions such as: Did the user accept the answer? Did they ask the same question again? Did they correct the AI? Did they escalate to a human? Did the recommendation result in a purchase? Did the coding agent actually produce working code?

W&B Weave supports collecting human feedback and associating it with application traces. Arize Phoenix likewise supports human annotations and user feedback, allowing teams to connect subjective assessments with the exact AI execution that produced them.

Bringing AI Observability Together

These 16 dimensions can seem like a lot… and, in my opinion, they are indeed a lot. That’s because modern AI applications are a lot more complicated than simply calling an LLM API. The good news is that all of these are more or less optional. Sort of. 😉

I grouped all the angles of AI monitoring into four layers that I think are a useful way to think about the landscape:

First is infrastructure observability: GPUs, CPU, memory, network, availability, latency, throughput, and model-serving performance.

Second is AI execution and security observability: model calls, prompts, tokens, context, retrieval, embeddings, tool calls, agents, memory, and end-to-end traces.

Third is AI quality and safety observability: hallucinations, groundedness, correctness, relevance, bias, toxicity, security violations, and other behavioral characteristics.

Finally, there is outcome observability: cost, user satisfaction, task completion, conversion, business impact, and governance.

The first layer looks a lot like traditional observability. The higher you go, the more AI-specific the problems become. And that’s the fundamental difference between monitoring an AI application and monitoring a conventional application.

Traditional observability can tell you that the system is running, how fast (think performance metrics) and what it did (think log messages).

AI observability needs to tell you what the AI did, why it did it, whether it was correct and safe, how much it cost, and whether it accomplished what you wanted.

That is a much bigger problem and a much more interesting one. This is why we are seeing the mushrooming of AI monitoring solutions covering various angles of AI that I described here.

Start Free Trial

---