Testing Tool Calling and Agent Orchestration: Contracts, Trajectories and Failure Semantics

Tool calling is where language-model probability meets deterministic software. The model proposes an action, but the application decides whether that action is permitted, how arguments are validated, what side effect occurs, and whether the result changes state.

A tool-call test therefore needs a stronger oracle than 'the answer looked right'. ToolBench treats tool use as an explicit evaluation problem, while agent benchmarks evaluate behaviour in interactive environments.

1. Define a tool contract

For every tool specify: purpose, allowed callers, authentication context, required arguments, types, ranges, semantic constraints, side effects, idempotency, timeout, retry policy, error contract and forbidden combinations.

Example contract:

tool: cancel_order

preconditions: authenticated_user && order_owned_by_user

required: order_id, confirmation

side_effect: order.status = cancelled

retry: forbidden_after_success

The contract becomes the basis for deterministic evaluation.

2. Evaluate selection separately from execution

There are at least four independent questions: Did the agent choose the correct tool? Did it supply correct arguments? Was the call authorized? Did the downstream system execute the intended operation?

Do not collapse these into one pass/fail result. A correct tool with an incorrect argument is different from a forbidden tool selection, and both are different from a downstream service failure.

3. Test trajectory constraints

Many business processes are order-sensitive. For example, identity verification may be required before an account mutation. A payment confirmation may be required before a refund is issued.

Represent the permitted sequence as a state machine or temporal rule. Then compare the observed trace with the allowed transition set.

An example invariant is: authorization_check must precede mutate_account. Another is: after successful cancel_order, the agent must not call cancel_order again unless the tool explicitly defines idempotency.

4. Test arguments semantically

Schema validation catches missing fields and wrong types but not every dangerous value. The QA layer should inspect semantic correctness as well.

For example, a tracking lookup may accept a string but still receive another customer's identifier. An order ID may be syntactically valid while not belonging to the authenticated customer.

Where possible, validate arguments against the test environment's known ground truth and authorization context rather than asking an LLM judge to infer correctness.

5. Test retries and idempotency

Agentic systems frequently retry after timeouts. A timeout does not prove that the operation did not occur. This creates the classic ambiguity: the agent cannot observe success but the side effect may already exist.

Test the sequence: request → delayed response → retry → duplicate prevention. The system should have explicit idempotency semantics for side-effecting operations.

6. Test orchestration as a control plane

The orchestrator determines which model call, tool, sub-agent or handoff happens next. It should be tested independently from the quality of generated language.

Create scenarios for normal routing, unavailable tools, ambiguous intent, competing intents, sub-agent failure, context overflow, retry exhaustion and handoff failure.

AgentBench's interactive framing is useful here: the agent is evaluated through its interaction with an environment, not through isolated model outputs.

7. Build negative tool tests

For every sensitive tool, test prompts that should not invoke it. These include indirect requests, authority claims, social pressure, conflicting instructions and retrieved content containing malicious instructions.

A negative test is stronger when the environment can prove that no side effect occurred. 'The assistant said it would not do it' is weaker evidence than a verified unchanged database state.

8. Test recovery as a finite policy

Define what the agent may do after each failure class. For a timeout, perhaps retry once; for authorization denial, stop and escalate; for malformed tool output, do not invent a result; for empty search, ask for clarification.

Then inject each failure and assert the policy transition.

9. Measure efficiency without rewarding unsafe shortcuts

Useful secondary metrics include number of tool calls, latency, token consumption, retries and redundant calls. However, optimize them only after correctness and safety constraints are satisfied.

A lower tool-call count is not automatically better if it comes from skipping a required verification step.

10. Evidence model

For every tool invocation capture: tool name, normalized arguments, authorization context, timestamp, parent span, result classification, side-effect identifier and policy decision.

This enables trace-level metrics such as unauthorized-call rate, wrong-tool rate, invalid-argument rate, redundant-call rate and recovery-success rate.

11. A practical evaluation matrix

12. Release criteria

For critical tools, release gates should be expressed as invariants: zero unauthorized calls, zero forbidden tools, zero invalid side effects, 100% required authorization checks, and full coverage of defined failure policies.

Aggregate agent scores should never override a failed critical invariant.

The mature approach treats the tool layer as an executable contract and the agent trajectory as a testable state machine. That is the difference between checking whether an agent sounds competent and proving that it acts within an engineered boundary.