Skip to main content

Overview

Roark can generate the simulation resources for an agent: the customer flows, the checks that grade them, and a run plan that runs them. It reads what you tell it about the agent and what the agent does on its calls, decides what is worth testing, and builds it. You review it and start a run when you are ready. Every decision comes with a reason. Each flow and check it creates shows why it exists and, where it came from your material, the exact words in your prompt, your website or a call that led to it. Anything it chose to leave out is listed with the reason too. Generation never runs your tests on its own. When it finishes, you get a run plan to start, and a webhook if you subscribed to one. Run it again whenever you like. Each run builds on what is already in place: it reads what changed in the agent and its recent production calls, and adds only what is missing. When nothing has changed, it says so and adds nothing.

When it runs

  • Creating an agent in the dashboard. “Automatically set up simulations for this agent” is on by default on the new agent page. Open Customize to add testing instructions, choose capabilities or change the exploration limits.
  • Connecting a voice platform. Turn on “Automatically set up simulations” when you connect, and Roark generates resources for each agent the connection brings in, including ones added later. It is on by default for connections with fewer than five agents.
  • Any time later. From an agent’s page, choose Set up simulations the first time, or Generate simulation resources after that. Through the API, call POST /v1/simulation/generations/resources.
Creating an agent through the API does not start a generation. Create the agent, then call the generate endpoint.

What you give it

The more you give it, the closer the test calls are to your real callers. A name and a phone number are enough to start: Roark works from what the agent says on calls.

What it builds

Generation works through a checklist. It does not finish while a part that applies to your agent is still uncovered. A part can be marked Not needed when there is nothing to build it from, for example when there are no testing instructions or no number to call. A part can also be skipped with a stated reason when it cannot be tested.

Exploration calls

With exploration on, Roark places a few short calls to the agent, starting with a broad “what can you help with?” and then following the paths it offers. Exploration callers never confirm a booking, payment or other change: they stop just before the agent would commit it. When the agent asks for something Roark does not have, such as an account number, the flow for that path is built but left out of the run plan, and the test data it needs is listed under Test data Roark needs. Add it, then include the flow. Exploration needs a phone number Roark can call, or an outbound dial setup. It stops at your call limit, and after about two hours of exploring at most.

Production calls

Roark reads the agent’s newest production calls, up to 30: the ones since the last generation, or from the last 30 days the first time. It finds the paths customers take and the problems they run into. A path that an existing flow already tests is left out, with the reason. A new one becomes a flow. Open issues on the agent’s calls are taken into account too.

Running it again

Every run starts from what earlier runs built that still exists. Those flows and checks count as covered, so they are never built a second time. A run looks at what happened since the last one:
  • The agent changed: a new prompt, description or website, new testing instructions, a new industry or capabilities, or Roark can now call the agent.
  • New production calls since the last run.
  • New activity on open issues on the agent’s calls.
When none of these happened, the run finishes within seconds, says there is nothing to add, and keeps the run plan as it is. When something did, it adds only what the change calls for. New flows and checks go into the existing run plan. A flow or check you took out of the plan stays out. When the agent has not changed, Roark does not call it again: its production calls already show what it does.

Following and guiding it

While it runs, the agent’s page shows what it is doing in plain sentences. Open it to see:
  • The checklist, with where each part stands.
  • Activity: what it decided and did, in order.
  • Guide the generation: tell it what to focus on or avoid, for example “focus on billing questions” or “skip transfers”. It reads your guidance before its next step, and your guidance outranks its own plan.
  • Questions: occasionally it asks something only you can answer. Each question has a default, and if nobody answers within 15 minutes it goes ahead with it, so a run nobody is watching still finishes.
  • Stop: stops the run. Flows and checks it already built are kept, and the run plan is not made or changed.
When it finishes, See why opens the full list of what it built, grouped by part, with the reason and source for each item. A generated flow also shows its reason at the top of the flow.

In the API

Start a generation

agentIds takes one agent for now. The response is the generation, with status: "PENDING". A second request for the same agent while one is running returns 409.

Follow it

GET /v1/simulation/generations/resources/{generationId} returns one generation, and GET /v1/simulation/generations/resources?agentIds=... lists an agent’s generations, newest first. Each has:
  • status: PENDING, RUNNING, NEEDS_INPUT (waiting on a question), COMPLETED, FAILED or CANCELLED
  • layers: the checklist, each part COVERED, SKIPPED, OPEN or NOT_APPLICABLE
  • activity: what it did and said, in order
  • question: the open question while NEEDS_INPUT, with its options and default
  • items: every flow, metric and check this run created or left out, each with its reason and source (type, quote, page URL, or call and moment)
  • runPlanId: the run plan to start once it completes
  • testDataNeeded: data the agent asked for that Roark did not have
A run that found nothing to add completes with no items and keeps the earlier runPlanId. Rather than polling, subscribe to the simulation.resource_generation.completed and simulation.resource_generation.failed webhooks. The completed event carries the runPlanId.

Guide, answer or stop it

Each returns 409 once the generation has finished.

Permissions

Starting, guiding, answering and stopping need an API key with agent:update, simulation:create and metric:create. Exploration places calls, so it also needs simulation:run: a key without it runs without exploration, unless you asked for exploration explicitly, in which case the request returns 403. Reading generations needs agent:read.

Billing

  • Exploration calls are billed like any simulation call.
  • Generation’s own work (reading your material and calls, deciding what to test and reviewing calls) is billed as AI agent usage, one line per step. You can see it on your billing page. A run that stops because nothing changed costs nothing.
Testing instructions are the quickest way to get the scenarios you care about most. Describe them the way you would brief a tester: who is calling, what they want, and what should happen.