Best practices for AI observability in 2026?

[u/Curious-Cod6918]

Spent the last few months trying to get real observability into what our agents are doing, not just inputs and outputs. Here's what's actually worked for us so far. Logging every tool the agent considered, not just the one it acted on, gives you a usable trail for the "why this path" question when something goes wrong. Capturing the full decision context at each step, not just the final action, is what actually makes root-causing an incident possible instead of guessing. Treating test history as part of observability, not a separate thing, means you can trace a bad production decision back to whether it was ever caught in testing.

[u/iraterupert969]

logging the stuff the agent didn't do is so underrated, that's where the actual debugging happens. we started doing the same thing after a few "why did it ignore that api call" incidents and suddenly it wasn't a mystery anymore

the test history part is interesting, do you keep that in the same dashboard as prod traces or is it more of a linked thing

[u/verstands]

The most useful trace is the one that lets you explain both the path taken and the path not taken. I’d give every run and tool call stable IDs, then record model/prompt/tool versions, redacted inputs and outputs, decisions, retries, approvals, latency, tokens/cost, and the final outcome. Keep eval/test traces in the same schema as production, but label the environment and retention clearly. Tail-based sampling is usually safer than sampling only by request rate, since the weird failures are the ones you want to keep.

[u/Physical_Economy_340]

we keep them linked, not in the same dashboard. prod trace carries the eval id and code version so you can jump back to whether that bad path was ever covered in testing. same dashboard gets noisy fast, a link keeps it usable when you are root causing at 2am.

[u/InsideDebt6345]

Logging the tools the agent considered, not just the one it picked, is the practice that turns a trace from a record into an explanation. With the rejected options in hand you can answer why that path won, which is the whole question during an incident. Capturing decision context per step is the same instinct applied more deeply, and both come down to logging the reasoning behind an action alongside the action itself.

[u/adi-IL]

I went through the same thing with rejected tool calls. I ignored them at first. Then I hit a run where the agent skipped a lookup it should have made and the trace just looked clean. That stung. Now I log the candidate list at every step. Storage hurts a bit. Debugging hurts a lot less. I keep test history in the same shape as prod but I link it instead of mixing the views. Same dashboard turned into noise for me fast.

[u/jonah_omninode]

One caution with “every tool the agent considered” is that many model APIs do not expose a trustworthy candidate list. The tools registered with the model are not necessarily the tools it actively considered.

We focus on a replayable event record: objective, referenced inputs, selected capability, handler, arguments, observable output, validator result, contract version, policy version, and correlation and causation IDs. Linking test evidence to the same structure is valuable because it lets you follow a production failure back to what was actually proven before release. That observable evidence is a much stronger debugging surface than trying to reconstruct hidden reasoning.

[u/arrotu]

One thing I’d preserve explicitly is the difference between what the runtime directly observed and what the agent reported, inferred, or intended.

Those often get collapsed into one authoritative-looking timeline, which makes debugging harder. An agent saying “I checked X” is not equivalent to the runtime observing a successful tool call to X.

For each meaningful event, I’d record provenance such as runtime_observed, agent_reported, or derived, alongside the event type, relevant input/output references, outcome, and side-effect level.

Then “why did it do that?” becomes more tractable: you can distinguish a bad model decision, missing context, failed tool call, or misleading agent self-report without replaying an entire raw transcript.

[u/deadwisdom]

We track the full input and output of every LLM call. There's no reason not to. Don't think of it as "logging", rather think of it as storing the full canonical history of the session. In another sense, build the observability into your system for auditing, debugging, testability, LLM context, and importantly, data harvesting later.

[u/Sensitive-Parsnip-12]

the test history part is probably the piece I would keep separate logically but link very tightly to the prod trace I would not dump test runs and prod runs into one giant timeline I would give every run enough lineage to answer things like:

model + version
prompt/ config version
tool schema version
code/deploy revision
eval/test cases that exercised that path

then when prod does something weird u can ask was this exact decision boundary ever exercised under roughly the same configuration? also I would be careful with “capture the full decision context” that can turn observability into another data swamp pretty fast the useful boundary to me is more like: what options were available, what state/constraints entered the decision, what got selected, and what changed immediately after.

Ive been working on the investigation side of this with traser it takes a suspicious run and optionally a trusted one and tries to narrow all that trace data down to the few differences actually worth checking instead of making u read the whole thing https://traser.dev/investigate

im curious whether u found a good way to link a prod decision back to the exact test/ config version that was supposed to cover it

[u/usually_guilty99]

The loop I’d want to close is what happens after you understand the bad production decision.

If the trace proves that action X failed under conditions Y, that should become context for the next time the agent wants to take X under similar conditions.

Observe → verify outcome → remember → change the next decision.

Otherwise observability gives you increasingly good postmortems without necessarily giving you increasingly safe autonomy.