What is simulation testing?
Simulation testing lets Roark act as your customers. You describe the conversations your agents should handle — who’s calling, what they want, how the call should go — and Roark places real calls (or chat sessions) against your agents, grades every conversation, and reports pass/fail results. Everything revolves around plans and runs. A plan is a reusable test suite; a run is one execution of it. Each run is labelledSR-{n} and moves through Queued → Running → Completed (or Failed / Cancelled).
A plan composes four things:
You’ll find everything under the Simulate section of the sidebar: Simulations (the runs hub), Customer flows, and Personas.
Core concepts
Customer flows
A customer flow describes one type of conversation your agent should handle. Author it in Improv mode (write a free-text customer brief and the simulator improvises a fresh conversation each run) or Scripted mode (author the conversation step by step on a graph canvas — ideal for IVR menus and deterministic routing). Every flow has a default variant — the happy path — plus optional edge cases that vary the persona, environment, or setup. The flow’s agent expectations are the graded contract: LLM-checked instructions the agent is evaluated against on every variant.Personas
A persona is who’s calling: language, accent, gender, base emotion, and how they speak. Each flow variant pins a persona, and run plans can override it per attachment — for example, fanning one flow across language personas in a multilingual run.Plans and runs
A plan is the saved suite; a run is one execution. Run a plan on demand, put it on a schedule, or skip saving entirely — unchecked Save as plan gives you a one-off run. Templates like Red teaming, Multilingual, and Load testing pre-configure metrics, checks, and flow sourcing so you start from a goal instead of a blank page.Metrics and checks
Every run scores its conversations with metrics — system metrics out of the box, plus custom metrics you build in Studio. Pass/fail thresholds turn metrics into checks, and the run report’s verdict is the pass rate across every check on every conversation.How a run comes together
1
Pick agents to test
From Simulations → New Run, select agents and the endpoints to reach them on. Run direction is derived from your endpoints: inbound means Roark calls your agent; outbound means your agent calls Roark.
2
Choose a template
Start from a goal — Flow adherence, Red teaming, Conversation quality, Multilingual, Load testing, Tool call accuracy — or Blank. Each template ships preset metrics and pass/fail checks, then attach the customer flows the run should exercise.
3
Tune the details
The Advanced section covers the metric set, iterations and concurrency, and end conditions (max duration, silence timeout, end-call phrases). Check Save as plan to keep the configuration as a re-runnable suite.
4
Run and review
Run now or schedule it. While the run is live you can watch calls in progress and listen in; once it settles, the report shows the verdict, per-metric results, run-over-run comparison, and every conversation’s transcript.
The REST API and SDKs keep the older naming: customer flows are still
scenarios in request fields, and run plans are managed via the same endpoints as before. Code examples throughout these docs reflect the API’s names.Common use cases
- Pre-deployment testing — exercise every flow’s happy path and edge cases before an agent goes live
- Regression testing — save a plan, re-run it after every change, and compare runs side by side in the report’s Comparison tab
- Continuous monitoring — schedule plans hourly, daily, or weekly to catch regressions before you do
- Safety and robustness — the Red teaming template runs adversarial flows against your agent; Load testing checks behavior under concurrent call volume
Where to go next
Customer flows
Author the conversations your agents should handle — Improv briefs or Scripted graphs
Personas
Define the callers your simulations run as
Templates
Start a run from a goal with preset metrics and checks
Plans & schedules
Build reusable suites and run them on a cadence
Running simulations
Launch runs, monitor them live, and read the report
Best practices
Strategies for building an effective test suite