Monitoring and Debugging Non-Deterministic AI Workflows with Progress AI Observability

Practical Summary of AI Observability

Progress AI Observability is a platform designed to provide visibility into the execution paths of non-deterministic AI agents, LLM applications, and RAG systems. Unlike traditional application monitoring that focuses on infrastructure, this tool captures trace-level data across prompts, model calls, retrieval steps, and tool usage. By connecting these signals, teams can diagnose why an agent failed, track token usage and associated costs, and evaluate output quality using LLM-as-a-judge methodologies. It is built to support .NET, Python, and JavaScript environments, allowing developers to move beyond simple request-response logging to understand the complex, multi-step logic inherent in modern agentic workflows. This platform is intended for professional teams building production-grade AI applications, including those using frameworks like Semantic Kernel, LangChain, or LlamaIndex.

Prerequisites and Initial Setup

To begin using the platform, developers must have an active account and an API key. The tool supports .NET (C#), Python, and TypeScript/JavaScript. The instrumentation process is designed to be lightweight and asynchronous to ensure that monitoring does not introduce significant latency into agent workflows. Users should ensure their environment variables for the application name and API key are correctly configured before proceeding with the installation steps provided in the official documentation at https://www.telerik.com/ai-observability-platform. The platform is designed to integrate with existing stacks, complementing traditional monitoring tools by adding agent-specific visibility rather than replacing them.

Instrumenting Your Application

Instrumentation requires installing the relevant SDK and adding a few lines of code to your application. For .NET developers, the process involves adding the NuGet package and configuring the chat client. For Python and TypeScript, the instrumentation is handled via the respective package managers. Below is the documented approach for .NET instrumentation:

// .NET - Install & Instrument // 1. Install dotnet add package Progress.Observability.Instrumentation // 2. Instrument chatClient = chatClient.AddObservability(options => { options.AppName = Environment.GetEnvironmentVariable("OBSERVABILITY_APP_NAME")!; options.ApiKey = Environment.GetEnvironmentVariable("OBSERVABILITY_API_KEY")!; });

Once instrumented, the SDK automatically begins capturing trace-level data from live agent runs, including model calls, tool usage, and latency metrics, which are then visible in the platform dashboard. This allows teams to see execution paths across prompts, models, and tools, providing the necessary context to understand how decisions unfold across multi-step and multi-agent workflows.

Documented Workflow for Debugging and Evaluation

The platform workflow centers on four primary pillars: tracing, debugging, cost control, and quality evaluation. Tracing allows teams to visualize the full execution path, including how decisions unfold across multi-step workflows. Debugging involves pinpointing failures by analyzing trace-level context, such as skipped tools, retrieval issues, or agent loops. Cost Attribution tracks LLM spend by linking token usage to specific agents, models, and workflows, enabling teams to identify expensive patterns before they scale. Finally, Quality Evaluation utilizes LLM-as-a-judge to score outputs against criteria like relevance and safety, allowing for side-by-side comparisons of prompt or model changes using real production data. This structured approach helps teams move from guessing what happened to having concrete evidence.

Interpreting Trace Data and Limitations

When reviewing data in the Trace Explorer, teams should focus on spans, model calls, and retrieval steps. The platform provides evidence-based insights into why an agent returned a specific response, even if that response appears valid but is actually ungrounded or irrelevant. It is important to note that while the tool provides deep visibility, it is not a replacement for traditional infrastructure monitoring; rather, it complements existing stacks by adding agent-specific context. Users should be aware that data retention policies vary by plan, ranging from 7 days on the free tier to infinite retention on enterprise plans. Teams maintain control over what data is logged, with options to redact or exclude sensitive information to meet data residency requirements.

Choosing When to Use Progress AI Observability

This tool is most effective for teams managing production-grade AI agents where the execution path is non-deterministic. It is specifically recommended for scenarios involving RAG pipelines, multi-agent systems, and copilots where traditional logs fail to capture the complexity of the decision-making process. Engineering leaders should use the platform to maintain governance, audit trails, and cost visibility, while developers should use it to reduce the time required for root cause analysis of agent failures. By using the same observability workflow across development and production, teams can validate changes, investigate production issues, and continuously improve their AI systems based on empirical evidence captured from real-world usage. This ensures that AI program success is measurable and reliable.