Tool Call Testing

Why tool calls need their own testing story

Tools are where your agent touches the real world: bookings, transfers, payments, SMS. That makes them both the riskiest thing to test (a test call that books a real appointment is not a test, it is an incident) and the most important thing to verify (a perfectly polite agent that calls book_appointment with the wrong date is failing its job). Roark solves both halves:

Piece	What it does	When you need it
Tool guard	Test calls never execute your real tools; Roark answers with simulated, scenario-consistent responses instead	Any time simulations reach a deployment with live tools
Tool fixtures	Pin an exact response for a tool (“the calendar has zero slots”), so a test forces the branch you want to exercise	Regression tests and branch coverage
Tool Invocation Analysis metrics	Grade whether the agent called the right tools, with the right parameters, in the right order, and used the results correctly	Always: simulations and production alike

The rest of this page walks them in the order a new project adopts them.

Step 1: Make test calls safe (the tool guard)

Wrap a side-effecting tool once. On a real caller’s session the wrapper is a pass-through. During a Roark test call the tool body never runs: the wrapper sends the invocation to Roark and returns a simulated backend response that matches the tool’s contract, follows the test scenario (a scenario about a fully booked calendar gets a realistic “no availability” from a booking tool), and stays consistent with earlier tool responses in the same call.

# The resolve call you already make at session start tells you whether this
# session is a Roark test call (simulation_job_id is set). The guard keys off
# that. Full decorator: see the code-first guide linked below.
@roark_tool_guard(cfg, fallback={"status": "booked", "confirmationId": "TEST-0000"})
async def book_appointment(date: str, time: str) -> dict:
    """Books an appointment. Args: date (YYYY-MM-DD), time (HH:MM). Returns {confirmationId, status}."""
    ...  # your real calendar write: never runs during a Roark test call

# If your provider agent's tools call YOUR backend, apply the same pattern
# server-side: recognize the simulation, then ask Roark for the response.
job = roark.simulation_job.lookup(roark_phone_number=caller_number)
if job is not None:
    mock = roark.simulation.mock_tool(
        simulation_job_id=job.id,
        tool_name="book_appointment",
        tool_description="Books an appointment. Returns {confirmationId, status}.",
        arguments=payload,
    )
    return mock.data.result  # never touch the real calendar for a test call

Wrap the writes (book, cancel, send_sms, transfer, charge); leave read-only lookups unwrapped when live data makes the test more realistic. Two safety properties hold by construction: a real caller can never receive a fake response (the guard only diverts on sessions Roark itself identified as simulations, and the endpoint re-validates the simulation before answering), and a guarded tool can never fire during a test (every error path maps to “use your static fallback”, except a 404, which genuinely means the session is not a known simulation).

Full setup: code-first agents

The complete decorator, framework wiring for LiveKit and Pipecat, and the exact failure contract.

Step 2: Force the outcomes you want to test (fixtures)

Generated mocks are plausible; sometimes a test needs exact. A fixture pins a tool’s response so a scenario always exercises one branch: what does the agent do when there are no slots, when the payment is declined, when the lookup finds two accounts?

# Project-wide: every test call sees an empty calendar
roark simulation tool-fixture create \
  --tool-name lookup_availability \
  --response '{"slots": []}' \
  --description "Forces the no-availability branch"

# Pinned to one flow variant: only this scenario sees a declined card
roark simulation tool-fixture create \
  --tool-name charge_card \
  --response '{"status": "declined", "reason": "insufficient_funds"}' \
  --customer-flow-variant-id "$VARIANT_ID"

curl -X POST https://api.roark.ai/v1/simulation/tool-fixture \
  -H "Authorization: Bearer $ROARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"toolName": "lookup_availability", "response": {"slots": []}, "description": "Forces the no-availability branch"}'

Precedence is simple: a fixture pinned to the running flow variant beats a project-wide one, and both beat generation. Setting the same scope twice replaces the response (one fixture per tool project-wide, one per tool and variant), so applying fixtures from CI is idempotent. Delete a fixture and the tool goes back to scenario-aware generated responses.

Step 3: Verify what your agent actually called

After a test call, ask Roark what the agent tried to do. Every guarded invocation is recorded against the simulation: tool, arguments verbatim, the response served, and whether it came from a fixture or the model.

roark simulation job tool-mock list <job-id>

curl https://api.roark.ai/v1/simulation/job/$JOB_ID/tool-mock \
  -H "Authorization: Bearer $ROARK_API_KEY"

{
  "data": [
    {
      "toolName": "lookup_availability",
      "arguments": { "date": "2026-10-01" },
      "result": { "slots": [] },
      "source": "FIXTURE",
      "createdAt": "2026-09-23T12:00:05Z"
    },
    {
      "toolName": "book_appointment",
      "arguments": { "date": "2026-10-08", "time": "15:00" },
      "result": { "status": "booked", "confirmationId": "APT-58371" },
      "source": "GENERATED",
      "createdAt": "2026-09-23T12:01:12Z"
    }
  ]
}

A CI step can assert directly on this: the agent offered an alternative date after the empty lookup, it never called charge_card on the refund scenario, the booking arguments match what the caller asked for. The list is empty when the tools are not guarded; that usually means step 1 is not wired for this agent.

Step 4: Grade it automatically (metrics)

For judgment calls that raw assertions cannot express (“was this the right tool at the right moment?”), Roark ships the Tool Invocation Analysis package. Find it under Metrics > Library:

Metric	Type	What It Measures
Tool Invocation Correct	Yes/No	Whether the agent invoked the correct tools at the appropriate times
Tool Invocation Count	Count	Total number of tool calls made during the conversation
Tool Invocation Order Correct	Yes/No	Whether tools were called in the correct logical sequence
Tool Invocation Parameters Correct	Yes/No	Whether the correct parameters were passed to each tool call
Tool Invocation Result Correct	Yes/No	Whether the agent correctly interpreted and used the results returned by each tool

Configure once, use everywhere: under Tool Scoping, select a specific tool and define when it should be called and what result is expected, or leave scoping empty to evaluate all tools. Use Additional Instructions for cross-tool rules like “Never call any tool before greeting the customer.”

In simulations

Create a Run Plan

Follow the run plan guide and pick the scenarios that should trigger the tool calls under test. Add fixtures (step 2) to the scenarios that need forced outcomes.

Add Tool Invocation Metrics

In the Metrics section of the run plan, select the metrics you configured above.

Enable Enriched Simulations

Metrics grade the tool data on your side of the call, so your agent must send its call data to Roark: see Enriched Simulations. If you use the tool guard, your guarded invocations are attached to the merged call automatically; you do not need to report them separately. Unguarded tools still need tool invocation data included with your call.

Run and read

Execute the plan. Once your agent’s call is matched with the simulation, the tool invocation metrics are evaluated automatically and land in the run report.

The same plan runs from a terminal or CI:

roark simulation run --data '{"planId":"<plan-id>"}'
roark simulation plan job get <job-id>

The CLI exits non-zero when the API rejects a request, so a CI step that starts a run fails loudly rather than passing on a broken plan. See exit codes.

In production

The same metrics run continuously on live calls through a collector. Live calls always hit your real tools (the guard never mocks real traffic; that property is structural), so this is pure observation:

# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: collector
name: tool-call-quality
modality: call
status: ACTIVE
metrics:
  - tool_invocation_correct
  - tool_invocation_parameters_correct
filters:
  - conditions:
      - type: AGENT
        key: frontdesk
        operator: EQUALS

roark config diff ./roark
roark config apply ./roark

Production calls must include tool invocation data for these metrics to work. Tool calls are captured automatically for Vapi and Retell integrations; for the API or code-first stacks, include tool invocations when you create the call or attach them afterward.

The loop, end to end

A team starting from zero gets to a full tool-testing loop in an afternoon: wrap the writes with the guard, pin fixtures for the branches that matter, run the plan, assert on the mocked invocations in CI, and let the Tool Invocation Analysis metrics grade every run and every production call from then on. Nothing real is booked, sent, or charged along the way.

Next Steps

Code-first agents

The tool guard’s full setup for LiveKit and Pipecat, plus Autoimprove on your own stack

Tool Invocations

How to submit tool call data with your calls

Enriched Simulations

Enable call matching for simulation tool testing

Collectors

Automate metric collection on production calls

Getting Started

Simulations

Observability

Metrics

Autoimprove

Recipes

Integrations

Enterprise

Resources

Why tool calls need their own testing story

Step 1: Make test calls safe (the tool guard)

Full setup: code-first agents

Step 2: Force the outcomes you want to test (fixtures)

Step 3: Verify what your agent actually called

Step 4: Grade it automatically (metrics)

In simulations

In production

The loop, end to end

Next Steps

Code-first agents

Tool Invocations

Enriched Simulations

Collectors

​Why tool calls need their own testing story

​Step 1: Make test calls safe (the tool guard)

Full setup: code-first agents

​Step 2: Force the outcomes you want to test (fixtures)

​Step 3: Verify what your agent actually called

​Step 4: Grade it automatically (metrics)

​In simulations

​In production

​The loop, end to end

​Next Steps

Code-first agents

Tool Invocations

Enriched Simulations

Collectors

Why tool calls need their own testing story

Step 1: Make test calls safe (the tool guard)

Step 2: Force the outcomes you want to test (fixtures)

Step 3: Verify what your agent actually called

Step 4: Grade it automatically (metrics)

In simulations

In production

The loop, end to end

Next Steps