Running Simulations

​

Overview

Roark provides multiple ways to execute simulation tests, from one-off manual runs to fully automated scheduled testing. Whether you’re validating a quick fix or running comprehensive regression tests, you can choose the execution method that best fits your workflow.

---

​

Execution Methods

​

Manual Execution

Run simulations on-demand directly from your dashboard:

1. From Run Plans - Execute existing test configurations
2. Create and Run - Build new test suites and run immediately
3. Re-run Previous Tests - Repeat past simulations with one click

​

Scheduled Execution

Automate testing with recurring schedules:

• Set up hourly, daily, weekly, or monthly runs
• Configure timezone-aware scheduling
• Monitor execution history and results

Learn more in our Schedules documentation.

---

​

Running from Run Plans

The most common way to execute simulations is through run plans, which define your test configuration.

​

Execute Existing Run Plan

1. Navigate to Simulations → Run Plans
2. Find your run plan in the list
3. Click the Run button
4. The system will:
   • Generate all test combinations
   • Queue simulations for execution
   • Begin running based on your concurrency settings
   • Track progress in real-time

​

Create and Run

When creating a new run plan, you can execute it immediately:

1. Click Create Run Plan
2. Configure your test matrix:
   • Select scenarios
   • Choose personas
   • Pick agent endpoints
   • Add evaluators
3. Choose Create and Run instead of just Create
4. Tests begin executing automatically after creation

---

​

Re-running Simulations

Easily repeat previous test runs to verify fixes or track improvements:

​

Re-run Options

1. From Run History
   • Navigate to Simulations → Runs
   • Find the previous run
   • Click the Re-run button
   • Executes with identical configuration
2. From Run Details
   • Open any completed run
   • Click Re-run in the header
   • Automatically uses the same run plan configuration

​

Benefits of Re-running

• Verify bug fixes with identical test conditions
• Track performance improvements over time
• Quickly validate configuration changes
• Compare results between runs

---

​

Execution Configuration

​

Key Settings

When running simulations, you can configure:

Setting	Description	Range
Iterations	Times to repeat each test	1-100
Max Concurrent Jobs	Parallel execution limit	Based on quota
Direction	Inbound or Outbound	See below

​

Inbound vs Outbound Considerations

Inbound Simulations:

• Execute immediately when triggered
• Support multiple iterations
• Roark calls your agent
• Fully automated execution

Outbound Simulations:

• Require manual dialing action
• Limited to 1 iteration
• Your agent calls Roark
• Provides phone numbers to call

Learn more about Inbound vs Outbound testing.

---

​

Monitoring Execution

​

Real-time Progress

Track your simulation runs as they execute:

1. Status Indicators
   • Pending - Queued for execution
   • Creating Snapshots - Preparing test configurations
   • Running Simulations - Actively executing calls
   • Completed - Successfully finished
   • Failed - Execution encountered errors
2. Progress Metrics
   • Number of simulations completed
   • Current execution count
   • Estimated time remaining
   • Success/failure rates

​

Execution Dashboard

Navigate to Simulations → Runs to view:

• All active executions
• Recent run history
• Status filtering options
• Detailed metrics for each run

---

​

Viewing Results

​

Run Summary

After execution completes, view comprehensive results:

• Overall success rate
• Average evaluation scores
• Total duration
• Individual test case results

​

Detailed Analysis

Click into any run to see:

• Test case breakdown
• Conversation transcripts
• Evaluation scores
• Error details (if any)
• Performance metrics

​

Comparing Runs

Use re-run functionality to:

• Track improvements over time
• Identify regression issues
• Validate bug fixes
• Monitor performance trends

---

​

Execution Limits

​

Concurrency Limits

Your organization has maximum concurrent simulation limits:

• Check your quota in organization settings
• System automatically optimizes based on your limits
• Upgrade your plan for higher concurrency

​

Best Practices for Performance

---

​

Quick Start Examples

​

Running Your First Simulation

1. Create a Run Plan
   • Navigate to Simulations → Run Plans
   • Click Create Run Plan
   • Select one scenario, one persona, one agent
   • Choose Create and Run
2. Monitor Progress
   • Watch status change from Pending to Running
   • View real-time completion updates
3. Review Results
   • Check overall success rate
   • Examine individual test results
   • Review evaluation scores

​

Setting Up Automated Testing

1. Build Your Run Plan
   • Create comprehensive test matrix
   • Include multiple scenarios and personas
   • Add relevant evaluators
2. Create Schedule
   • Navigate to Simulations → Schedules
   • Select your run plan
   • Configure timing (e.g., daily at 2 AM)
   • Activate schedule
3. Monitor Automated Runs
   • Check schedule dashboard for results
   • Set up notifications for failures
   • Review trends over time

---

​

Next Steps