Overview
Templates are step 02 · Template of creating a run. Instead of assembling metrics and checks from scratch, you pick a goal — measure conversation quality, probe for jailbreaks, hammer one flow at volume — and the template seeds a sensible starting configuration:- Preset system metrics — a curated set from the metric library, auto-selected in the Advanced section.
- Pass/Fail checks — thresholds attached alongside the metrics, so the run reports a clean pass rate out of the box.
- A configuration panel — step 03 of the create page changes per template (languages to test, volume to run at, adversarial cases to include).
- Flow sourcing — some templates pre-attach customer flows for you, by system label or by happy path.
The catalogue
Pick a template from the card grid, or open Browse all templates for the full library grouped by category: Your templates, Quality, Safety, and Performance.| Template | Category | What it tests |
|---|---|---|
| Blank | Your templates | Start from scratch — no presets; choose what to measure, which flows to run, and who’s calling |
| Flow adherence | Quality | How closely the agent stays on the flows you authored, under pressure |
| Conversation quality | Quality | How natural, empathetic, and helpful the agent is across realistic conversations |
| Multilingual | Quality | Language detection, comprehension, and response-language handling across languages |
| Tool call accuracy | Quality | Whether the agent calls the right tools, with the right arguments, at the right moments |
| Red teaming | Safety | Adversarial probing — refusals, jailbreak resistance, and policy adherence |
| Load testing | Performance | Latency, drops, and concurrency limits when many simulations run in parallel |
What each template seeds
Blank
Blank
No presets. The Advanced section shows only run settings — you pick metrics, flows, and checks yourself. Blank is the default selection.
Flow adherence
Flow adherence
Seeds adherence-focused metrics — scenario adherence, instruction following, and call outcome — with Pass/Fail checks on adherence and instruction following. Its configuration panel is an Expected flows picker: attach the flows the agent should follow, and the run measures how closely it stays on them.
Conversation quality
Conversation quality
Seeds the broadest metric set: sentiment, frustration, user effort, call outcome, instruction following, comprehension failures, redundant questions, missed responses, interruption appropriateness, talk-to-listen ratio, overtalk, and response time — with Pass/Fail checks on user effort, instruction following, and redundant questions.
Multilingual
Multilingual
Seeds comprehension, instruction-following, scenario-adherence, call-outcome, and sentiment metrics with adherence and instruction-following checks, then fans each attached flow out across the languages you pick (see below).
Tool call accuracy
Tool call accuracy
Seeds the three tool-call metrics — correct invocation, correct parameters, correct order — each with a Pass/Fail check. You pick the tools to evaluate once, and the template applies that list to all three metrics.
Red teaming
Red teaming
Seeds six compliance metrics — prompt-injection resistance, PII handling, prohibited language, scope adherence, hallucination boundary, and identity consistency — each with a Pass/Fail check. It sources its test cases from adversarial flows (see below).
Load testing
Load testing
Seeds a minimal at-scale health set: agent responsiveness, response time, and time to first word, with a Pass/Fail check on responsiveness — so a load run reports one thing clearly: did the agent hold up at volume.
Template-specific configuration
Step 03 of the create page is the selected template’s own panel. Three templates have configuration that goes beyond attaching flows.Load testing: the Volume panel
Load testing owns the run’s scale, so the generic Iterations field in Advanced is hidden — the Volume panel is the single source of truth:| Field | What it controls |
|---|---|
| Concurrent calls | How many simulated calls run at the same time, capped by your account’s concurrency quota |
| Total iterations | The total number of simulation runs across the test |
Multilingual: language fan-out
Pick the languages to test from a chip grid — Spanish, French, German, Portuguese, Italian, Dutch, Chinese, Japanese, Hindi, Arabic, Turkish, Greek, Indonesian, Thai, Tagalog, Malay, and Hebrew. Each attached flow runs once per language: the template attaches the flow’s happy path with a persona override that swaps in that language’s system speaker persona, so a run with 3 flows and 4 languages produces 12 test conversations. A Code-switching toggle swaps the persona set to English-mix speakers — callers who mix their language with English mid-conversation (e.g. Spanglish) — to test how the agent keeps up when the language shifts.Red teaming: adversarial sourcing
Red teaming gives you two ways to source its test cases:Choose scenarios
A unified, filterable catalogue that merges Roark’s curated adversarial library with your own flows carrying the Adversarial system label. Each card shows its source badge (Roark or Yours) and failure-mode tags. A flow that’s only partly adversarial — some variants labelled, some not — runs just its adversarial variants, and the card names them.
Generate new
Draft fresh adversarial edge cases tailored to one of your own Improv flows. Pick the source flow, choose how many cases to generate (3–10), and set a difficulty — easy, medium, or hard. Ask Roark drafts the variants, labels each one Adversarial, and appends them to the flow; the panel then switches to the catalogue with your flow selected so the new cases flow straight into the run.
Generation targets Improv flows only — Scripted flows derive their variants from the conversation graph, so adversarial variants can’t be appended to them.
Everything else
Flow adherence, Conversation quality, and Tool call accuracy configure through the shared inline Flows panel — attach the customer flows the run exercises, with the usual variant selection. Tool call accuracy adds one control above it: Tools to evaluate, defaulting to all current tools, applied to all three tool-call metrics so you don’t repeat yourself.Template provenance
When you check Save as plan, the plan records which template it came from. The template shows up as a badge on the plan’s detail page and on every run’s header, and when you edit a plan the template is read-only — it defined the plan’s shape, so you adjust the configuration rather than swap the goal. One plan, one template: to test the same agents against a different goal, create a new run from the other template and save it as its own plan.Next steps
Run Plans
Save a template-seeded configuration as a reusable test suite
Customer Flows
Author the conversations templates attach and grade against
Running Simulations
Watch a run live and read its report
Thresholds
How the Pass/Fail checks templates seed actually work