Running Simulations

A run is one execution of a simulation plan. Every run gets a sequential label (SR-42), a trigger chip showing how it started, and a run detail page that acts as mission control while calls are in flight, then settles into a full report once the last call lands.

Runs execute a configured plan: agent targets, customer flows, metrics, and run settings. If you haven’t built one yet, start with Run Plans.

Ways to trigger a run

Every run row carries a trigger chip telling you how it started:

Trigger	How it happens
Manual	You clicked Run simulation on the create-run page, or Run now on a plan
Schedule	A recurring or one-time schedule fired (Schedules)
Re-run	Started from a previous run’s Re-run menu
System	Roark started the run automatically

Run now

From Simulate → Simulations, click New Run, configure the run, and hit Run simulation. The split button’s menu also offers Schedule recurring… and Run once, later… if you’d rather not run immediately. Tick Save as plan if you want to keep the configuration. Otherwise the run is grouped under the One-off runs card on the runs hub. To run an existing plan, open it and click Run now.

If your plan has without default values, a modal collects them before the run starts. See Variables.

Via API

Trigger a plan programmatically and pass runtime variables:

curl -X POST "https://api.roark.ai/v1/simulation/plan/{planId}/job" \
  -H "Authorization: Bearer $ROARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "variables": {
      "orderNumber": "12345",
      "environment": "staging"
    }
  }'

The response includes the simulationRunPlanJobId you can poll via GET /v1/simulation/plan/job/{jobId}.

The REST API keeps the older naming: customer flows are called scenarios, so flow-scoped runtime variables are passed as an array of { scenarioId, variables } objects.

Run statuses

Runs show one of five statuses:

Status	Meaning
Queued	Accepted and waiting to start
Running	Building test calls, executing them, or compiling results
Completed	Every call finished and the report is ready
Failed	The run hit an unrecoverable error or timed out
Cancelled	You cancelled the run before it finished

The API exposes finer-grained backend statuses (CREATING_SNAPSHOTS, CREATING_SIMULATIONS, RUNNING_SIMULATIONS, ENDING_SIMULATIONS, CANCELLING). All of these render as Running in the UI, and TIMED_OUT renders as Failed.

Watching a live run

Open a run while it’s executing and you get mission control. The header shows the plan name (or “One-off run #N”), the SR-number, Inbound/Outbound and template badges, the agents under test, and a pulsing Running · mm:ss clock. Banners walk you through the phases: “Starting the run”, then “All calls landed, compiling the report”.

Progress strip: how many calls are done, in flight, and still queued.
Live now board: every active conversation. Voice calls (Phone, WebRTC, LiveKit) stream audio so you can listen in as they happen; chat sessions (WebSocket, ElevenLabs, Google CES, Kore) show a live transcript instead. Pending conversations show their state: Dialing, Connecting, or Waiting for call.
What we’re measuring: the run’s checks, so you know what pass/fail will be judged on.
Calls table: click any call to open a split detail pane with the transcript, agent expectations, and audio for voice calls.

Cancelling a run

Click the red Cancel run button in the header. Cancellation is graceful: queued calls stop, but in-flight calls run to completion and still appear in the results. The run settles with status Cancelled.

The settled report

Once every call lands, the page becomes a report with four sections:

Section	What’s in it
Overview	Verdict hero and headline stats
Metrics	Per-metric results across the run
Comparison	This run side by side with the plan’s other runs
Calls	Every conversation, with transcripts and per-call results

The verdict hero leads with a check pass-rate ring and a plain-language narrative (“Clean sweep: every check passed on every conversation.”), backed by stat cards: Clean calls, Calls run, and Trend or Weakest check. Below it, the Success criteria card is the pass/fail decision itself: a PASSED or FAILED pill, one row per check with the pass rate it achieved against the minimum it had to clear, and the reasons any run failed. The score alongside it is the mean of the check rates, shown for trend lines only: a run can score 95 and still fail when one check missed its own bar. A plan carrying no pass/fail check reports no verdict, since there was nothing to judge. See Success criteria, and CI/CD for gating a pipeline on the same verdict. Use the Export button to download the run’s results as a CSV. A run-name dropdown in the header switches between the plan’s executions (the newest is tagged “Latest”), or open the Comparison tab to view runs side by side.

Calls that don’t count

Not every call a run places ends up in its results. Every figure in the report, from the pass-rate ring to the per-check rates, the success criteria and the calls list, is built only from calls that tested what they were meant to test. Everything else falls into one of three groups, each shown separately so nothing disappears silently. Calls that couldn’t run on Roark’s side. Occasionally a simulation fails for a reason that has nothing to do with your agent: the speech pipeline couldn’t start, capacity wasn’t available, or post-call analysis failed. These are left out of every list and count. The calls list shows a quiet note (“N simulations could not run on our side”) with a link to reveal them. A call that fails this way before a conversation starts is retried once automatically and doesn’t take an extra concurrency slot, so the run still gets the calls it planned. Calls your agent caused to fail. When your agent didn’t answer, was busy, declined, hung up right away or went silent, that is a real result about your agent. These calls stay in the report as failures. Excluded calls. A call that ran but didn’t exercise what the run is testing is marked Excluded, with a one-line reason, and left out of every figure. The Show excluded toggle on the calls list reveals them.

Reason	Label	When it applies
`REQUIRED_STAGE_INCOMPLETE`	Setup incomplete	The call ended before it got through a stage its flow marks as required, for example an IVR the agent was never reached past.
`SCRIPT_DIVERGED`	Went off script	A strict flow went off script at a step whose off-script policy is Hang up and invalidate.
`CALLER_NEVER_TOOK_OVER`	Caller never took over	The call opened on a preceding flow’s own persona, which never handed the phone to the persona under test. None of that persona’s properties (accent, pace, emotion, age) were exercised, so the call says nothing about them.

Over the API

GET /v1/simulation/job/{jobId} (and each job in GET /v1/simulation/plan/job/{jobId}) reports both on every job:

failureOwner: for a FAILED or TIMED_OUT job, who it failed because of. ROARK means the simulation itself failed on Roark’s side and says nothing about your agent; AGENT means your agent under test caused it. It is null when the job didn’t fail.
invalidation: present on an excluded job, with the reason from the table above, a one-sentence detail, and invalidatedAt.

{
  "data": {
    "simulationJobId": "08b09b8c-c18b-4aa1-bed6-139dd178a2c1",
    "status": "FAILED",
    "failureOwner": "ROARK",
    "invalidation": null
  }
}

A job with failureOwner: "ROARK" isn’t evidence about your agent, so don’t count it as a failure in your own reporting. Treat an invalidation reason you don’t recognise as “excluded”: new reasons may be added.

Re-running

The settled report’s Re-run menu offers two distinct modes:

Mode	What it does	Use it when
Fresh run	Builds new test calls from the plan as configured today, collecting any unresolved variables first	You’ve edited the plan or flows and want to test the current configuration
Re-run with same data	Replays this run’s exact snapshot: same flows, personas, and settings, even if the plan has changed since	You want an apples-to-apples comparison, e.g. verifying a bug fix under identical conditions

Re-runs land in the same plan’s run history, so the Comparison tab picks them up automatically.

Iterations and concurrency

Set in the plan’s Advanced → Run settings:

Iterations: runs per test case, up to 100. Useful for catching intermittent failures. Outbound runs are limited to 1 iteration. See Inbound vs Outbound.
Concurrency: parallel calls. The default is 5, capped by your account quota. Start there and raise it once you’ve confirmed your agent handles the load.

Next steps

Run Plans

Build reusable test suites

Schedules

Put plans on a cadence

Variables

Parameterize flows with runtime values

Enriched Simulations

Score metrics from your agent’s side of the call

Getting Started

Simulations

Observability

Metrics

Config as Code

Autoimprove

Recipes

Integrations

Enterprise

SDKs & Developer Tools

Resources

Ways to trigger a run

Run now

Via API

Run statuses

Watching a live run

Cancelling a run

The settled report

Calls that don’t count

Over the API

Re-running

Iterations and concurrency

Next steps

Run Plans

Schedules

Variables

Enriched Simulations

​Ways to trigger a run

​Run now

​Via API

​Run statuses

​Watching a live run

​Cancelling a run

​The settled report

​Calls that don’t count

​Over the API

​Re-running

​Iterations and concurrency

​Next steps

Run Plans

Schedules

Variables

Enriched Simulations

Ways to trigger a run

Run now

Via API

Run statuses

Watching a live run

Cancelling a run

The settled report

Calls that don’t count

Over the API

Re-running

Iterations and concurrency

Next steps