Unit Testing (AI Agents)

• Unit testing for AI agents validates individual components — tool calls, prompt logic, memory management, and reasoning steps — in isolation before they interact in a live system.
• Non-determinism is the core challenge: LLMs can produce different outputs for identical inputs even at temperature 0, breaking traditional pass/fail assertions.
• Traditional unit tests remain necessary but insufficient — they catch deterministic code bugs but miss failures in reasoning, hallucination, and context handling that only surface in production.
• Effective agent unit testing combines deterministic checks with probabilistic evaluation using techniques like LLM-as-judge scoring, semantic similarity matching, and trajectory analysis.
• Eval-driven development is emerging as the standard practice: define evaluations before building agent capabilities, then iterate until performance meets thresholds — the agent equivalent of test-driven development.
• Observability fills the gaps that unit tests cannot: tracing every tool call, reasoning step, and decision path transforms debugging from guesswork into data-driven analysis.

Unit testing for AI agents is the practice of validating the individual components of an AI agent system — including prompt templates, tool-calling logic, memory operations, and reasoning chains — in isolation to verify they behave correctly before the agent runs as a complete system.

At its core, unit testing for AI agents involves validating the individual components, or "units," of an AI system to ensure they function as intended. In the context of AI agents, these units include various agent behaviors, tool integrations, and memory management processes.

Think of it like testing a microservices architecture. You wouldn't ship a payment service without testing its individual endpoints, even if you also run integration tests. Agent unit testing applies the same principle: verify that the prompt parser handles edge cases, that the tool selector picks the right API for a given intent, and that the memory retrieval returns relevant context — each in isolation, before they chain together.

The challenge is that agents aren't deterministic functions. Testing AI agents is fundamentally different from testing conventional software. You're no longer verifying deterministic code — you're evaluating probabilistic systems that interact with users, tools, and unstructured data. Behavior isn't just a product of code; it's shaped by models, prompts, and dynamic context. This makes the "unit" itself harder to define and the expected output harder to assert.

Defining Testable Units

In AI agent development, the "unit" is no longer just a function — it's often a prompt, a reasoning chain, or a sequence of tool calls. A practical agent testing strategy breaks the agent into discrete testable components, as Galileo's testing guide recommends: foundation model performance (how accurately does the base LLM understand inputs), tool selection accuracy (does the agent choose appropriate tools for specific tasks), planning coherence (can the agent create logical, sequenced steps to solve problems), and multi-turn conversation handling (does the agent maintain context across interactions).

For a code review agent, this means writing separate tests for: does the prompt correctly identify a security vulnerability when given a known-vulnerable code snippet? Does the tool selector route to the correct linter for Python vs. TypeScript? Does the memory module recall the repository's coding conventions from prior runs?

Deterministic vs. Probabilistic Assertions

Traditional unit tests use exact-match assertions: `assertEqual(expected, actual)`. Agent unit tests require a layered approach. Deterministic checks handle the parts you can pin down — JSON schema validation, tool call parameter formats, token count limits. Probabilistic evaluation handles the rest.

LangSmith's evaluation framework supports scoring performance with automated evaluators — LLM-as-judge, code-based, or any custom logic — across criteria that matter to your business. LangSmith integrates with pytest, Vitest, and GitHub workflows so you can run evals on every PR or nightly build. Set thresholds on evaluation metrics and fail pipelines automatically when scores drop, bringing the same rigor as deterministic unit tests to your AI development process.

Trajectory Evaluation

For agents that chain multiple steps, validating the final output isn't enough. Testing them means validating not just the output, but the entire process: how the agent reasons, what tools it chooses, what data it retrieves, and how it decides what to do next.

LangChain's AgentEvals library provides trajectory match evaluators that judge the trajectory of an agent's execution either against an expected trajectory or using an LLM. You can set match modes to `strict`, `unordered`, `subset`, or `superset` depending on how tightly you need to constrain the agent's execution path.

Eval-Driven Development

As Anthropic recommends in their engineering guidance, practice eval-driven development: build evals to define planned capabilities before agents can fulfill them, then iterate until the agent performs well. This is TDD for agents — write the evaluation criteria first, then build the capability.

The Non-Determinism Problem

The usage of unit tests for AI functions is limited because of non-determinism. As research published on arXiv demonstrates, LLMs can be very non-deterministic in standard setups. Furthermore, an LLM rarely produces the same response ten times given the same input. Even setting temperature to 0 doesn't guarantee reproducibility — even when we adjust the temperature down to 0 (thus making the sampling theoretically deterministic), LLM APIs are still not deterministic in practice.

Without unit testing adapted for this reality, failures go undetected. Prompt failures don't raise exceptions — they quietly return misleading, incomplete, or fabricated results. Without structured evaluation tools, these failures can go unnoticed until users start losing trust.

Cascading Failures in Multi-Agent Systems

The stakes multiply when agents collaborate. This introduces asynchronous workflows and dependencies. If one agent misinterprets context, hallucinates, or misuses a tool, it can trigger a cascade of errors across the system — complicating root cause analysis. Unit testing each agent's components in isolation is the first line of defense against these cascades.

Enterprise Adoption Demands It

82% of organizations plan to integrate AI agents within three years, yet traditional evaluation methods fail to address the non-deterministic, multi-step nature of agentic systems. Without structured testing, you get agent sprawl — dozens of agents running in production with no way to verify they still behave correctly after a model upgrade or prompt change.

Testing a CI/CD Deployment Agent

Consider an agent that automatically rolls back failed deployments. Unit tests should cover: does the log parser correctly identify a crash loop? Does the rollback tool selector pick the right Kubernetes rollback command? Does the notification formatter include the correct commit SHA and error summary? Each of these can be tested with mocked inputs and deterministic assertions before the agent ever touches a production cluster.

Testing a Code Review Agent

A code review agent needs tests at multiple levels. Deterministic checks verify it outputs valid review comments in the expected format. Semantic evaluation (using LLM-as-judge) verifies the comments are relevant to the actual code changes. For a more complex multi-turn eval, a coding agent receives tools, a task, and an environment, executes an "agent loop" (tool calls and reasoning), and updates the environment with the implementation. Grading then uses unit tests to verify the working output.

Testing a Customer Service Agent

Cresta's research on non-deterministic testing illustrates why simple pass/fail doesn't work here: there might be several ways to do it, such as unplugging and plugging it back in, or pressing and holding the reset button. The AI agent could choose either method first and still be correct, but a turn-by-turn deterministic test might flag one as a failure simply because it didn't follow a pre-defined order. Goal-driven evaluation — did the customer's issue get resolved? — replaces rigid script matching.

Non-Determinism Requires Statistical Thinking

A single passing test run proves nothing. Counterintuitively, even a single successful test run can prove insufficient — run the test again and it could fail. Agent unit tests need to run multiple times per assertion, with pass rates measured over N executions rather than a single binary check. An alternative might be regression testing that tolerates variability.

Observability Fills Gaps Tests Cannot

As LLM systems evolve from simple prompts into multi-step agents with memory, tool usage, and external API calls, observability becomes non-negotiable. Traditional logs or unit test outputs no longer give enough context to understand failures — or why results may vary between identical runs. Tools like Langfuse, Arize Phoenix, and LangSmith provide the tracing infrastructure to understand what happened between input and output.

Test Maintenance Is Ongoing

Model upgrades, prompt changes, and tool updates all invalidate existing tests. When source code changes, AI agents detect regressions and — if approved — autonomously update impacted unit tests, ensuring both test reliability and faster adaptation to new code logic. But autonomous test updates require careful review — an AI-updated test might mask a real regression.

Beware Sycophantic Evaluation

When using LLM-as-judge grading, the evaluator LLM can be as prone to errors as the agent itself. One common failure pattern occurs when LLMs become overly agreeable, a behavior often referred to as sycophancy. In these cases, the agent might appear to produce a correct answer by echoing the customer's assumptions or confirming incorrect statements, rather than genuinely reasoning through the task. In testing, this can create the illusion of success.

Cost Compounds Quickly

Every LLM-as-judge evaluation call costs tokens. Running 50 test cases with 10 repetitions each, scored by a judge model, means 500 LLM API calls per test suite run. Teams need to balance coverage against cost, often using cheaper models for high-volume regression checks and reserving expensive models for critical-path evaluations.