Skip to main content

​
Overview

A run plan is a reusable template that defines a comprehensive simulation configuration. Instead of running individual simulations one by one, run plans automatically generate test matrices by combining scenarios, personas, and agent endpoints - making it easy to execute systematic testing manually or via schedule.
Navigate to Simulations → Run Plans in your Roark dashboard to create and manage run plans.
Run Plans Dashboard

​
What is a Run Plan?

Think of a run plan as a test suite generator. It creates all possible combinations of your selected components:

​
The Matrix Formula

Scenarios × Personas × Agent Endpoints × Iterations = Total Test Cases
Example: If you select:
  • 2 scenarios (billing inquiry, technical support)
  • 3 personas (happy customer, frustrated customer, confused elder)
  • 2 agent endpoints (production agent, staging agent)
  • 1 iteration each
Your run plan generates 12 individual test cases automatically, ensuring comprehensive coverage across all combinations.

​
Creating a Run Plan

Run plans are created through a step-by-step wizard that guides you through selecting each component:

​
Configuration Steps

1

Select Scenarios

Choose one or more conversation scenarios that define what will be tested
2

Select Personas

Pick the customer personas to use - each brings different voice characteristics and behaviors
3

Choose Agent Endpoints

Select which agents to test against (see inbound vs outbound section below)
4

Select Metrics & Thresholds

Choose metric definitions to evaluate against each simulation. Optionally configure thresholds on any metric to define pass/fail criteria (e.g., Customer Satisfaction >= 7).
5

Review & Configure

Set iterations, concurrency limits, and preview the test matrix before creation

​
Configuration Options

SettingDescriptionDefault
NameDescriptive name for your run planRequired
DescriptionOptional details about the test purposeNone
DirectionInbound or Outbound testing modeInbound
IterationsHow many times to repeat each combination1
Max Concurrent JobsParallel execution limit10

​
Testing Direction

Run plans can be configured for either inbound or outbound testing, which determines who initiates the calls:
  • Inbound: Roark calls your agent (immediate execution, multiple iterations)
  • Outbound: Your agent calls Roark (async execution, single iteration)
Learn more about choosing the right direction in our Inbound vs Outbound guide.

​
Understanding the Test Matrix

The power of run plans lies in their ability to systematically test all combinations:

​
Example Matrix

Let’s say you’re testing a customer service agent with: Components Selected:
  • Scenarios: Password Reset, Billing Inquiry
  • Personas: Calm Professional, Frustrated Customer
  • Agent Endpoints: Production Agent
  • Metrics: Task Completion, Customer Satisfaction
Generated Test Cases:
#ScenarioPersonaAgentMetrics
1Password ResetCalm ProfessionalProductionTask Completion, Satisfaction
2Password ResetFrustrated CustomerProductionTask Completion, Satisfaction
3Billing InquiryCalm ProfessionalProductionTask Completion, Satisfaction
4Billing InquiryFrustrated CustomerProductionTask Completion, Satisfaction
Each combination ensures your agent is tested against different customer types in various situations.

​
Metrics & Thresholds

The final configuration step is selecting which metrics to evaluate and, optionally, setting thresholds that define pass/fail criteria for each simulation. When adding metrics to a run plan, you can configure a threshold inline — choose an operator (e.g., >=) and a value (e.g., 7). After each simulation call completes, the metric is collected and the threshold is evaluated automatically. This gives you a clear pass/fail signal for every test case in your matrix, making it easy to spot regressions and measure quality across scenarios.

Thresholds Guide

Learn about operators, aggregation modes, and participant role filtering

​
Previewing Your Run Plan

Before executing, you can preview the complete test matrix: Run Plan Preview The preview shows:
  • Total number of test cases
  • All combinations that will be generated
  • Estimated execution time
  • Resource requirements

​
Executing Run Plans

Run plans can be executed in two ways:

​
Manual Execution

Click the “Run” button on any run plan to execute immediately. Process:
  1. System generates all test case combinations
  2. Simulations execute according to concurrency limits
  3. Each simulation is evaluated
  4. Results aggregate in real-time
  5. Complete report available when finished

​
Scheduled Execution

Attach run plans to schedules for automated testing: 
Schedule: Daily Regression Test
Time: 2:00 AM UTC
Run Plan: Production Agent Test Suite
Notification: email on failure
Learn more about scheduling simulations.

​
Best Practices

Begin with a few combinations to validate your setup before scaling up
Name run plans descriptively (e.g., “Daily Production Regression Suite”)
Include both happy path and edge case scenarios in your matrix
Match concurrency limits to your agent’s capacity to avoid overload
Analyze failures to identify patterns and improve agent responses

​
Common Use Cases

​
Regression Testing

Create a comprehensive run plan with all critical scenarios and run it before each deployment.

​
A/B Testing

Set up identical run plans against different agent versions to compare performance.

​
Load Testing

Use high concurrency and multiple iterations to stress-test your infrastructure.

​
Compliance Validation

Include compliance-focused scenarios and metrics to ensure regulatory adherence.

​
Integration with Other Features

Run plans work seamlessly with other Roark features:

​
Next Steps

Create Scenarios

Build test scenarios for your run plans

Configure Personas

Set up diverse customer personas

Set Up Schedules

Automate run plan execution

View Reports

Analyze simulation results