Overview
Roark can generate the simulation resources for an agent: the customer flows, the checks that grade them, and a run plan that runs them. It reads what you tell it about the agent and what the agent does on its calls, decides what is worth testing, and builds it. You review it and start a run when you are ready. Every decision comes with a reason. Each flow and check it creates shows why it exists and, where it came from your material, the exact words in your prompt, your website or a call that led to it. Anything it chose to leave out is listed with the reason too. Generation never runs your tests on its own. When it finishes, you get a run plan to start, and a webhook if you subscribed to one. Run it again whenever you like. Each run builds on what is already in place: it reads what changed in the agent and its recent production calls, and adds only what is missing. When nothing has changed, it says so and adds nothing.When it runs
- Creating an agent in the dashboard. “Automatically set up simulations for this agent” is on by default on the new agent page. Open Customize to add testing instructions, choose capabilities or change the exploration limits.
- Connecting a voice platform. Turn on “Automatically set up simulations” when you connect, and Roark generates resources for each agent the connection brings in, including ones added later. It is on by default for connections with fewer than five agents.
- Any time later. From an agent’s page, choose Set up simulations the first time, or Generate simulation resources after that. Through the API, call
POST /v1/simulation/generations/resources.
What you give it
The more you give it, the closer the test calls are to your real callers. A name and a phone number are enough to start: Roark works from what the agent says on calls.
What it builds
Generation works through a checklist. It does not finish while a part that applies to your agent is still uncovered.
A part can be marked Not needed when there is nothing to build it from, for example when there are no testing instructions or no number to call. A part can also be skipped with a stated reason when it cannot be tested.
Exploration calls
With exploration on, Roark places a few short calls to the agent, starting with a broad “what can you help with?” and then following the paths it offers. Exploration callers never confirm a booking, payment or other change: they stop just before the agent would commit it. When the agent asks for something Roark does not have, such as an account number, the flow for that path is built but left out of the run plan, and the test data it needs is listed under Test data Roark needs. Add it, then include the flow. Exploration needs a phone number Roark can call, or an outbound dial setup. It stops at your call limit, and after about two hours of exploring at most.Production calls
Roark reads the agent’s newest production calls, up to 30: the ones since the last generation, or from the last 30 days the first time. It finds the paths customers take and the problems they run into. A path that an existing flow already tests is left out, with the reason. A new one becomes a flow. Open issues on the agent’s calls are taken into account too.Running it again
Every run starts from what earlier runs built that still exists. Those flows and checks count as covered, so they are never built a second time. A run looks at what happened since the last one:- The agent changed: a new prompt, description or website, new testing instructions, a new industry or capabilities, or Roark can now call the agent.
- New production calls since the last run.
- New activity on open issues on the agent’s calls.
Following and guiding it
While it runs, the agent’s page shows what it is doing in plain sentences. Open it to see:- The checklist, with where each part stands.
- Activity: what it decided and did, in order.
- Guide the generation: tell it what to focus on or avoid, for example “focus on billing questions” or “skip transfers”. It reads your guidance before its next step, and your guidance outranks its own plan.
- Questions: occasionally it asks something only you can answer. Each question has a default, and if nobody answers within 15 minutes it goes ahead with it, so a run nobody is watching still finishes.
- Stop: stops the run. Flows and checks it already built are kept, and the run plan is not made or changed.
In the API
Start a generation
agentIds takes one agent for now. The response is the generation, with status: "PENDING". A second request for the same agent while one is running returns 409.
Follow it
GET /v1/simulation/generations/resources/{generationId} returns one generation, and GET /v1/simulation/generations/resources?agentIds=... lists an agent’s generations, newest first. Each has:
status:PENDING,RUNNING,NEEDS_INPUT(waiting on a question),COMPLETED,FAILEDorCANCELLEDlayers: the checklist, each partCOVERED,SKIPPED,OPENorNOT_APPLICABLEactivity: what it did and said, in orderquestion: the open question whileNEEDS_INPUT, with its options and defaultitems: every flow, metric and check this run created or left out, each with itsreasonandsource(type, quote, page URL, or call and moment)runPlanId: the run plan to start once it completestestDataNeeded: data the agent asked for that Roark did not have
items and keeps the earlier runPlanId.
Rather than polling, subscribe to the simulation.resource_generation.completed and simulation.resource_generation.failed webhooks. The completed event carries the runPlanId.
Guide, answer or stop it
Each returns
409 once the generation has finished.
Permissions
Starting, guiding, answering and stopping need an API key withagent:update, simulation:create and metric:create. Exploration places calls, so it also needs simulation:run: a key without it runs without exploration, unless you asked for exploration explicitly, in which case the request returns 403. Reading generations needs agent:read.
Billing
- Exploration calls are billed like any simulation call.
- Generation’s own work (reading your material and calls, deciding what to test and reviewing calls) is billed as AI agent usage, one line per step. You can see it on your billing page. A run that stops because nothing changed costs nothing.