> ## Documentation Index
> Fetch the complete documentation index at: https://docs.roark.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Automatic Test Setup

> Let Roark work out what to test for a new agent, build the flows and checks, and save a run plan you can start

## Overview

When you add an agent, Roark can set up its tests for you. It reads what you tell it about the agent, decides what is worth testing, builds the customer flows and checks, and saves a run plan. You review it and start the first run when you are ready.

Setup explains every decision. Each flow and check it creates comes with a reason and, where it came from your material, the exact words in your prompt, website or a call that led to it. Anything it chose to leave out is listed with the reason too.

Setup never runs your tests on its own. When it finishes, you get a run plan to start, and a webhook if you subscribed to one.

## When setup runs

* **Creating an agent in the dashboard.** "Auto generate simulations for this agent" is on by default on the new agent page. Open **Customize** to add testing instructions, choose capabilities or change the exploration limits.
* **Creating an agent through the API.** Setup runs when you pass something to set it up from: a `prompt`, an `endpoint`, a `websiteUrl` or `testingInstructions`. Pass `autoSetup.enabled: false` to skip it.
* **Connecting a voice platform.** The first 10 agents a connection brings in are set up automatically. Past that, start setup for each agent you want tested.
* **Any time later.** From an agent's page, choose **Set up simulations**, or call `POST /v1/agent/{agentId}/setup`. Running setup again on an agent adds a new set of tests; it does not change the ones you already have.

## What you give it

| Input | What it is for |
| - | - |
| Name and description | What the agent is for |
| System prompt | The agent's instructions: the main source for what it does and the rules it follows |
| Industry | Adds the rules and checks agents in that industry are expected to meet. Choosing **Other** means you pick the capabilities yourself |
| Website | A page about the business, for services, hours and policies the prompt does not mention |
| Testing instructions | What you want tested, in your own words. Each scenario you describe becomes its own flow |
| Capabilities | What the agent does (for example booking, payments, call routing). Leave empty and Roark works them out |
| Exploration | Whether Roark calls the agent to learn what it actually does, how many calls it may place, and how long each call may last |

The more you give it, the closer the test calls are to your real callers. A name and a phone number are enough to start; setup works from what the agent says on calls.

## What it builds

Setup works through a checklist. It does not finish while a part that applies to your agent is still uncovered.

| Part | What setup creates |
| - | - |
| **Every agent** | Calls that try to make the agent misbehave (prompt injection, attempts to extract its instructions, impersonation, off-topic derailment), and checks on response time, silences, interruptions and how calls end |
| **Your industry** | The industry's rules, added to every flow, and its checks. For example, a healthcare agent is checked on how it handles personal information and on not giving medical advice |
| **What it does** | One flow per capability, built from Roark's tested scenarios for it and rewritten with your business's details, with the checks that measure it |
| **Your prompt and website** | Flows for situations your material describes that the capability flows do not cover, and checks for rules the agent must follow, such as verifying a caller before changing an appointment |
| **Your instructions** | A flow for each scenario you asked for |
| **Calling the agent** | Flows for paths Roark found on its calls, including ones your prompt never mentions, and checks for problems it saw, such as long silences |

A part can be marked **Not needed** when there is nothing to build it from, for example when there are no testing instructions or no number to call. Setup can also skip a part with a stated reason when it cannot be tested.

### Exploration calls

With exploration on, Roark places a few short calls to the agent, starting with a broad "what can you help with?" and then following the paths it offers. Exploration callers never confirm a booking, payment or other change: they stop just before the agent would commit it.

When the agent asks for something Roark does not have, such as an account number, the flow for that path is built but left out of the run plan, and setup lists the test data it needs under **Test data Roark needs**. Add it, then include the flow.

Exploration needs a phone number Roark can call, or an outbound dial setup. It stops at your call limit, and after about two hours of exploring at most.

## Following and guiding setup

While setup runs, the agent's page shows what it is doing in plain sentences. Open it to see:

* **The checklist**, with where each part stands.
* **Activity**: what setup decided and did, in order.
* **Guide setup**: tell it what to focus on or avoid, for example "focus on billing questions" or "skip transfers". It reads your guidance before its next step, and your guidance outranks its own plan.
* **Questions**: setup occasionally asks something only you can answer. Each question has a default, and if nobody answers within 15 minutes setup goes ahead with it, so a setup nobody is watching still finishes.
* **Stop**: stops setup. Flows and checks it already built are kept, and no run plan is made.

When setup finishes, **See why** opens the full list of what it built, grouped by part, with the reason and source for each item. A flow setup built also shows its reason at the top of the flow.

## In the API

### Create an agent with setup

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/agent \
  -H "Authorization: Bearer $ROARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Front desk",
    "customId": "front-desk",
    "industry": "HEALTHCARE",
    "prompt": "You are the receptionist for Smile Dental...",
    "endpoint": { "value": "+15125550142", "direction": "INCOMING" },
    "testingInstructions": "Test a caller who wants to cancel tomorrow and is upset about the fee.",
    "autoSetup": { "explore": { "maxConversations": 6, "maxConversationMinutes": 4 } }
  }'
```

The response includes the agent and its `setup`, with `status: "PENDING"`. A `customId` that is already in use returns `409`.

### Follow it

`GET /v1/agent/{agentId}/setup` returns the agent's latest setup:

* `status`: `PENDING`, `RUNNING`, `NEEDS_INPUT` (waiting on a question), `COMPLETED`, `FAILED` or `CANCELLED`
* `layers`: the checklist, each part `COVERED`, `SKIPPED`, `OPEN` or `NOT_APPLICABLE`
* `activity`: what setup did and said, in order
* `question`: the open question while `NEEDS_INPUT`, with its options and default
* `items`: every flow, metric and check it created or left out, each with its `reason` and `source` (type, quote, page URL or call and moment)
* `runPlanId`: the run plan to start once setup completes
* `testDataNeeded`: data the agent asked for that Roark did not have

Rather than polling, subscribe to the `agent.setup.completed` and `agent.setup.failed` webhooks. The completed event carries the `runPlanId`.

### Guide, answer or stop it

| Endpoint | Body |
| - | - |
| `POST /v1/agent/{agentId}/setup/{setupId}/guidance` | `{ "text": "Focus on billing questions" }` |
| `POST /v1/agent/{agentId}/setup/{setupId}/answer` | `{ "text": "Yes" }`, only while `NEEDS_INPUT` |
| `POST /v1/agent/{agentId}/setup/{setupId}/cancel` | none |

Each returns `409` once setup has finished.

### Permissions

Setup needs an API key with `simulation:create` and `metric:create`. Exploration places calls, so it also needs `simulation:run`. A key without these still creates the agent: setup is skipped, or runs without exploration, unless you explicitly asked for it, in which case the request returns `403`. Creating the agent's endpoint in the same request needs `endpoint:create`.

## Billing

* **Exploration calls** are billed like any simulation call.
* **Setup's own work** (reading your material, deciding what to test and reviewing exploration calls) is billed as **AI agent usage**, one line per step. You can see it on your billing page.

> [!TIP]
>
> Testing instructions are the quickest way to get the scenarios you care about most. Describe them the way you would brief a tester: who is calling, what they want, and what should happen.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.