AI-powered agents expose a fundamental observability gap that conventional monitoring tools weren't built to catch, according to a new report from Dynatrace. While traditional services fail loudly with error codes and latency spikes, AI agents return successful status codes and pass standard checks even when delivering incorrect answers to users. The company's 2026 State of SRE and Platform Engineering report, which surveyed 919 enterprise leaders globally, found that 77% of platform engineering teams have observability embedded in at least some services, but only 40% have achieved full integration across all deployments.
The observability challenge stems from how AI agents break compared to deterministic services. A conventional application signals trouble through HTTP 500 errors, dependency timeouts, or clear performance degradation. An AI agent can return a 200 status code, satisfy faithfulness checks, and still deliver the wrong information to customers. Teams can't set alerts for "wrong" the way they monitor latency or error rates. The report illustrates this with a support agent handling product documentation queries, where a coding assistant helped rewrite documentation lookup logic. After deployment passed continuous integration and existing evaluations, responses slowed down and some answers referenced outdated product versions instead of the requested release.
Investigating AI failures requires tracing the complete trajectory of every model call and tool invocation, with arguments and results captured in sequence. The report walks through an example where three identical document searches consumed 2.6 seconds, each passing a null value for the product version filter instead of the customer's requested version. The trace revealed no model calls between the repeated searches, indicating the retry logic sat in the execution harness rather than the AI model itself. Faithfulness metrics passed because the generated answer accurately reflected the retrieved documentation, but those documents covered version 2024.1 when the customer asked about 2026.3. According to the analysis, "relevance is not validity"—the retrieved documents matched the query topic but violated an unasserted precondition about version filtering.
The report recommends treating retrieval constraints as testable code rather than relying solely on model-based evaluation. Teams should write deterministic tests that assert on lookup behavior directly, such as verifying that a search for version 2026.3 documentation returns only that version's content, with no model in the loop. For answer quality that can't be asserted programmatically, evaluations should run repeatedly since output varies, and any model-based judge should be validated against human-reviewed examples before being trusted in production. After deployment, teams need to monitor latency and task success alongside tool-call counts and token usage on the same dashboard, because a drop in tool calls could signal either a successful fix or an accidentally removed lookup step. Organizations that can't immediately identify where to start investigating an AI feature failure haven't finished instrumenting it yet. The shift from deterministic to probabilistic systems demands that platform teams rethink what evidence counts as proof—not just that code deployed, but that the running system produced the right answer for the right reasons.
Dynatrace is sponsoring WeAreDevelopers World Congress Americas, September 23-25, 2026, in San José, where attendees can bring debugging questions from AI-assisted releases or AI features they're building.