Ask a Zartis engineer what comes up most often in client conversations about moving an AI feature from pilot to production, and the answer is rarely the model. It is evals and observability. Enterprise clients now ask, consistently and early, how a team will prove an AI system works before it ships, and how they will know it is still working after it does. That question used to be a footnote. It is now a gating condition.
This piece is about what that condition actually requires in practice: what a real eval suite looks like for an LLM or agent feature, what agent observability demands that conventional application monitoring does not, and why the two disciplines are complementary rather than substitutable. It is written for developers building these systems, not for a compliance audience, though the stakes are naturally higher in regulated sectors such as financial services, healthcare, energy, and telecommunications, where “we tested it once before launch” has never been an acceptable answer for any other system of record.
The thesis
Evals are pre-deployment quality assurance: testing an AI system’s outputs against known cases the way you would test software against known inputs.
Observability is post-deployment visibility: seeing, at the level of individual LLM calls, tool invocations, and retrieval steps, what a system actually did in production.
Readers from a data science background will recognise this split under different names: evals here are what ML practice calls offline evaluation (a fixed dataset, scored before deployment), and the trace-level observability described below is the substrate for what is called online evaluation (scoring live traffic after deployment). The vocabulary differs by domain; the offline-then-online loop is the same.
Evals catch the failure modes you already know about, before you ship. Observability catches the ones you do not, after you ship. Enterprises now treat both as a condition of production deployment, not as optional maturity add-ons, because non-deterministic systems fail in ways that pre-LLM software testing and logging practices were never built to catch.
That is the argument this piece makes in full, with the practical detail underneath it.
Why this is showing up as a gating condition now
The signal is direct: engineers on our delivery teams report that evals and observability are being raised by stakeholders more and more when an AI or agentic feature moves toward production. Not model choice. Not architecture. Whether the team can demonstrate the system was tested against a defined set of cases, and whether the client’s own engineers will be able to see what the system did after go-live without relying on the vendor’s word for it.
This tracks with what is happening across the industry more broadly. Industry surveys on agent engineering maturity in 2026 report that 57% of organisations now have agents running in production, but quality remains the top-cited barrier at 32%, well ahead of cost or latency concerns (LangChain, State of AI Agents). The State of AI Evaluations in 2026 research shows observability adoption running ahead of evals adoption, at roughly 89% versus 52%, which is telling in itself: teams are instrumenting first and testing systematically second, often because production incidents forced the instrumentation before anyone built the eval suite that should have caught the issue pre-release. Guptadeepak’s 2026 market analysis of the observability and evaluation space frames this as organisations moving “beyond whether to build agents to how to deploy them reliably” (AI Agent Observability Market Report 2026).
Trust is the underlying currency. As the same analysis puts it, stakeholders who can see what an agent did, why it did it, and whether the policies governing it were respected are measurably more willing to extend that agent greater autonomy. That is a fair description of what enterprise procurement conversations about AI features have become: not “does the model work”, but “can you show me it worked, and can you show me if it stops working.”
Half one: evals as pre-deployment quality assurance
An eval suite for an LLM or agent feature should be treated the same way a test suite is treated for any other piece of production software: versioned, run in CI, and blocking on regressions. In practice, that means four components.
Golden datasets
A golden dataset is a curated, versioned set of representative input and expected-output pairs that defines what “correct” looks like for a given feature. Guidance from the LLM testing community converges on a practical size: a production-grade golden set typically runs to 100 to 300 diverse, non-overlapping prompt-response pairs, enough to be statistically meaningful when measuring regression without becoming unmanageable to maintain (TestQuality, LLM Regression Testing Pipeline). The set should cover the cases you have already seen fail in production or user testing; adversarial and edge-case testing is a separate, complementary exercise for finding the failure modes you have not yet encountered (FutureAGI, LLM Regression Testing Guide).
Regression testing on prompts and context
Every change to a prompt, a model version, a retrieval configuration, or a tool definition is a code change and should trigger the same discipline a code change would: run the golden set, compare the candidate output against the last known-good baseline, and flag drift. This is the core practice behind what the LLM testing literature calls “LLM regression testing”: rerunning a fixed eval suite after any prompt, model, retriever, or tool change and comparing results against the previous passing run (FutureAGI). A meaningful share of regressions, particularly structural ones such as broken JSON output, missing required fields, or responses that violate a length constraint, can be caught with deterministic, non-LLM assertions at near-zero cost, before any LLM-based scoring runs at all (Adaline, Complete Guide to LLM and AI Agent Evaluation).
Scoring rubrics: code-based and LLM-based grading
Not every quality dimension can be checked with a deterministic assertion. Anthropic’s own guidance on building evaluations is direct about the trade-off: code-based grading is fast and reliable but lacks nuance, while LLM-based grading is flexible and scalable but needs its own calibration and validation (Anthropic, Define success criteria and build evaluations). For RAG-backed features specifically, groundedness and faithfulness are the two metrics worth building rubrics around. Faithfulness measures whether every claim in a generated answer is supported by the retrieved context, typically operationalised by decomposing a response into atomic claims and classifying each as entailed, neutral, or contradicted against the source material using natural language inference. Groundedness is the finer-grained, per-sentence version of the same check (Deepchecks, RAG Evaluation Metrics; LangCopilot, RAG Evaluation Guide). A faithfulness score is not a vanity metric: an unfaithful answer in a regulated domain, one that sounds plausible but is not actually supported by the document the system retrieved, is precisely the failure mode that gets a system pulled from production.
Anthropic’s engineering guidance on agent evals adds a further discipline worth building into any rubric: test both the behaviours an agent should exhibit and the behaviours it should not, because one-sided evals produce one-sided optimisation. An agent that never refuses an out-of-scope request has not been tested properly if the eval suite only rewards helpfulness (Anthropic, Demystifying evals for AI agents).
CI integration
None of the above earns its keep sitting in a notebook. The eval suite belongs in the same pipeline as unit and integration tests: it runs on every pull request that touches a prompt, a model version, a retriever, or a tool definition, and a regression against the golden set blocks the merge in the same way a failing unit test would. This is where an eval suite stops being a one-off quality check performed before a launch and becomes a permanent CI artefact, maintained with the same rigour as the rest of the codebase.
Claude Code, powered by Anthropic, in practice. Anthropic has documented how the Claude Code team applies this discipline to their own product: they started with fast, informal iteration based on user feedback, then progressively added evals, first for narrow, well-defined behaviours such as concision and the correctness of file edits, and later for harder-to-specify problems such as over-engineering in generated code (Anthropic, Demystifying evals for AI agents). The lesson generalises: you do not need a complete eval suite before you ship anything. You need a growing one, added to as failure modes are discovered, that becomes harder to bypass over time. Teams running a Claude Code-driven development workflow are well placed to apply the same pattern to their own agent features: use Claude Code to help scaffold the golden dataset and the CI wiring, then let the eval suite that results become the thing that gates every future change to the prompts and tools Claude Code is editing.
Half two: observability as post-deployment visibility
Evals answer “does this work on the cases we thought to test.” Observability answers “what is it actually doing right now, on the cases we did not think to test.” For agentic systems specifically, conventional application logs and stack traces are not sufficient to answer that question, for a structural reason: a stack trace tells you which function failed and why, deterministically. An LLM agent’s failure is rarely a thrown exception. It is a plausible-looking, syntactically valid tool call with the wrong arguments, a retrieval step that returned the wrong document, or a multi-step plan that quietly diverged from the user’s intent three steps before the final, technically well-formed answer. None of that shows up as an error in a conventional log.
What agent observability actually requires
Agent observability is the practice of capturing every step an agent takes during execution: tool selection, tool arguments, model responses, memory reads and writes, and the branch points where the agent chose one path over another (Braintrust, Agent observability: The complete guide for 2026). In distributed tracing terms, each LLM call, each tool invocation, and each retrieval step becomes a child span within a trace, so that a single user request produces a full, inspectable call graph from the entry point through any orchestrator, any sub-agents, and every tool the agent touched along the way (Greptime, How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools).
OpenTelemetry has become the practical standard here, for the same reason it became the standard for microservice tracing: it is vendor-neutral, which means trace data stays portable if the observability backend changes later. Native instrumentation now exists for the major agent frameworks (LangGraph, CrewAI, the OpenAI Agents SDK, Vercel AI SDK, and others), with generic OpenTelemetry instrumentation as the fallback for custom-built agent stacks, including auto-instrumentation packages that cover Anthropic’s own API directly. The overhead argument that used to be raised against this level of instrumentation does not hold up under measurement: OpenTelemetry instrumentation adds under a millisecond per call, against LLM API latencies that typically run from 100 milliseconds to tens of seconds (Uptrace, OpenTelemetry for AI Systems). The instrumentation is not what is slow.
Why logs and stack traces are not enough
The distinction is worth stating plainly because it is where teams new to agentic systems most often under-invest. A conventional application log tells you a request came in and a response went out, and perhaps how long it took. It does not tell you which of the five tools available to the agent were considered and rejected, what the retrieved context actually contained when the model generated its answer, or whether the model’s stated reasoning for calling a particular tool bore any relationship to what it then did. For a non-deterministic system, where the same input can legitimately produce different but equally valid outputs on different runs, “it returned a 200” is not evidence that the system behaved correctly. Only a full trace of the intermediate steps is.
This is also precisely where observability and evals connect. A trace captured in production, showing an agent silently retrieving the wrong document type for a class of query nobody had tested for, is the direct source of the next entry in the golden dataset. Observability finds the unknown failure mode; the eval suite it feeds prevents that specific failure mode from recurring. Teams that treat the two as separate, unconnected line items miss this feedback loop entirely, and end up rebuilding the same incident response process every time a new failure surfaces in production instead of shrinking the space of failures over time.
Trust remains the bottleneck, even with both in place
It is worth being honest that instrumentation alone does not close the trust gap. Recent industry data on agentic AI deployment shows that even where observability tooling is in place, 69% of AI-driven decisions still require human verification, and only around a third of organisations report having reached full agentic deployment (Guptadeepak, AI Agent Observability Market Report 2026). Evals and observability are the precondition for extending an agent more autonomy, not a guarantee that autonomy should be extended immediately. The organisations moving fastest on this in 2026 are treating traces, evaluations, and governance guardrails as part of the agent’s architecture from the first design decision, not bolted on once something goes wrong (Microsoft Foundry, Build agents you can trust across any framework).
Putting the two halves together: a working pattern
A practical version of this discipline, applied to a single agent feature, looks like this:
- Define success criteria before writing the feature. What does a correct response look like, and what does an incorrect one look like. Anthropic’s own guidance treats this as the actual starting point of prompt and agent engineering, not a step that follows building the feature (Anthropic, Define success criteria and build evaluations).
- Build a golden dataset of 100 to 300 cases, covering known failure modes from prior testing or from a predecessor system, and check it into version control alongside the code.
- Wire the eval suite into CI, so that any change to a prompt, model, retriever, or tool definition runs the golden set and blocks on regression against the last passing baseline.
- Instrument the production system with trace-level observability, using OpenTelemetry or a framework-native equivalent, so every LLM call, tool invocation, and retrieval step is a visible, queryable span.
- Feed production incidents back into the golden dataset. A failure mode found once in a trace should never need to be found twice.
This is a loop, not a checklist. The eval suite gets stronger every time observability surfaces something new, and the observability data gets easier to interpret because the eval suite has already ruled out the failure modes you know about.
How Zartis helps
We build golden datasets scoped to a client’s actual failure modes, not generic test cases. Eval suites are wired into the client’s existing CI pipeline rather than run manually before a demo, and trace-level observability is instrumented from the first sprint rather than bolted on after an incident. In regulated sectors, this sits inside the architecture from day one: an ungrounded answer or an unlogged agent decision is a regulatory exposure, not just a support ticket.
Zartis is a Preferred Services Partner in the Claude Partner Network, and Claude features regularly in the agentic systems Zartis designs for clients, including delivery built around Claude Code. That gives Zartis engineering teams a close read on how Anthropic applies eval-driven development to its own product, which shapes how Zartis scopes the same discipline for client work: built by the team writing the code, not audited afterwards by someone else. Clients take away a system their own engineers can inspect, evals and traces included, rather than one they have to take on trust.
Conclusion
Evals and observability used to be the parts of an AI feature that got skipped under deadline pressure. That is no longer optional: enterprise stakeholders now ask for both before a pilot goes anywhere near production. A golden dataset without trace-level visibility catches only the failure modes already known. Observability without an eval suite catches new failures but never stops them recurring. Run together, in CI and in production, they form the loop that lets a team show its work rather than ask for trust.
The decision in front of most engineering leaders now is not whether to build this discipline in, but when. Teams that wire it in from the first sprint spend less time firefighting later, and inherit a system they can explain to a regulator, an auditor, or a client. Teams that add it after an incident are rebuilding the same investigation from scratch every time.
Sources
- LangChain, “State of AI Agents”
- Galileo, “The State of AI Evaluations in 2026”
- Guptadeepak, “AI Agent Observability, Evaluation & Governance: The 2026 Market Reality Check”
- Microsoft Foundry Blog, “Build agents you can trust across any framework with open evals and a control standard”
- TestQuality, “LLM Regression Testing Pipeline for QA Engineers: RAG Triad & Gold Sets in 2026”
- FutureAGI, “LLM Regression Testing: FutureAGI Guide (2026)”
- Adaline, “The Complete Guide to LLM & AI Agent Evaluation in 2026”
- Anthropic, “Define success criteria and build evaluations”
- Anthropic, “Demystifying evals for AI agents”
- Deepchecks, “RAG Evaluation Metrics: Answer Relevancy, Faithfulness, and Accuracy”
- LangCopilot, “RAG Evaluation Metrics in 2026: Recall@K, MRR, Faithfulness & RAGAS”
- Braintrust, “Agent observability: The complete guide for 2026”
- Greptime, “How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools”
- Uptrace, “OpenTelemetry for AI Systems: LLM and Agent Observability (2026)”