Skip to main content

Overview

When you add an agent, Roark can set up its tests for you. It reads what you tell it about the agent, decides what is worth testing, builds the customer flows and checks, and saves a run plan. You review it and start the first run when you are ready. Setup explains every decision. Each flow and check it creates comes with a reason and, where it came from your material, the exact words in your prompt, website or a call that led to it. Anything it chose to leave out is listed with the reason too. Setup never runs your tests on its own. When it finishes, you get a run plan to start, and a webhook if you subscribed to one.

When setup runs

  • Creating an agent in the dashboard. “Auto generate simulations for this agent” is on by default on the new agent page. Open Customize to add testing instructions, choose capabilities or change the exploration limits.
  • Creating an agent through the API. Setup runs when you pass something to set it up from: a prompt, an endpoint, a websiteUrl or testingInstructions. Pass autoSetup.enabled: false to skip it.
  • Connecting a voice platform. The first 10 agents a connection brings in are set up automatically. Past that, start setup for each agent you want tested.
  • Any time later. From an agent’s page, choose Set up simulations, or call POST /v1/agent/{agentId}/setup. Running setup again on an agent adds a new set of tests; it does not change the ones you already have.

What you give it

The more you give it, the closer the test calls are to your real callers. A name and a phone number are enough to start; setup works from what the agent says on calls.

What it builds

Setup works through a checklist. It does not finish while a part that applies to your agent is still uncovered. A part can be marked Not needed when there is nothing to build it from, for example when there are no testing instructions or no number to call. Setup can also skip a part with a stated reason when it cannot be tested.

Exploration calls

With exploration on, Roark places a few short calls to the agent, starting with a broad “what can you help with?” and then following the paths it offers. Exploration callers never confirm a booking, payment or other change: they stop just before the agent would commit it. When the agent asks for something Roark does not have, such as an account number, the flow for that path is built but left out of the run plan, and setup lists the test data it needs under Test data Roark needs. Add it, then include the flow. Exploration needs a phone number Roark can call, or an outbound dial setup. It stops at your call limit, and after about two hours of exploring at most.

Following and guiding setup

While setup runs, the agent’s page shows what it is doing in plain sentences. Open it to see:
  • The checklist, with where each part stands.
  • Activity: what setup decided and did, in order.
  • Guide setup: tell it what to focus on or avoid, for example “focus on billing questions” or “skip transfers”. It reads your guidance before its next step, and your guidance outranks its own plan.
  • Questions: setup occasionally asks something only you can answer. Each question has a default, and if nobody answers within 15 minutes setup goes ahead with it, so a setup nobody is watching still finishes.
  • Stop: stops setup. Flows and checks it already built are kept, and no run plan is made.
When setup finishes, See why opens the full list of what it built, grouped by part, with the reason and source for each item. A flow setup built also shows its reason at the top of the flow.

In the API

Create an agent with setup

The response includes the agent and its setup, with status: "PENDING". A customId that is already in use returns 409.

Follow it

GET /v1/agent/{agentId}/setup returns the agent’s latest setup:
  • status: PENDING, RUNNING, NEEDS_INPUT (waiting on a question), COMPLETED, FAILED or CANCELLED
  • layers: the checklist, each part COVERED, SKIPPED, OPEN or NOT_APPLICABLE
  • activity: what setup did and said, in order
  • question: the open question while NEEDS_INPUT, with its options and default
  • items: every flow, metric and check it created or left out, each with its reason and source (type, quote, page URL or call and moment)
  • runPlanId: the run plan to start once setup completes
  • testDataNeeded: data the agent asked for that Roark did not have
Rather than polling, subscribe to the agent.setup.completed and agent.setup.failed webhooks. The completed event carries the runPlanId.

Guide, answer or stop it

Each returns 409 once setup has finished.

Permissions

Setup needs an API key with simulation:create and metric:create. Exploration places calls, so it also needs simulation:run. A key without these still creates the agent: setup is skipped, or runs without exploration, unless you explicitly asked for it, in which case the request returns 403. Creating the agent’s endpoint in the same request needs endpoint:create.

Billing

  • Exploration calls are billed like any simulation call.
  • Setup’s own work (reading your material, deciding what to test and reviewing exploration calls) is billed as AI agent usage, one line per step. You can see it on your billing page.
Testing instructions are the quickest way to get the scenarios you care about most. Describe them the way you would brief a tester: who is calling, what they want, and what should happen.