Skip to main content

​
What is Simulation Testing?

Simulation testing in Roark allows you to systematically test your voice AI agents by creating realistic conversations between simulated customers (personas) and your agents. Instead of waiting for real customer interactions or manually making test calls, you can proactively validate agent behavior across hundreds of scenarios. Think of it as automated QA for voice AI - you define who’s calling (persona), what they want (scenario), which agent to test, and how to measure success (evaluators). Roark then orchestrates these test calls and provides comprehensive results.

​
Why Simulations Matter

Voice AI agents handle critical customer interactions. A single bad experience can damage your brand, while consistently great experiences build loyalty. Simulations help you:
  • Catch issues before customers do - Test edge cases and error scenarios proactively
  • Ensure consistency - Verify agents perform reliably across different customer types
  • Validate changes safely - Test updates without risking production issues
  • Save time and money - Automate testing instead of manual calling
  • Build confidence - Know your agents work before deployment

​
Core Concepts

Understanding these four building blocks will help you create effective simulations:

​
1. Personas

Virtual customers with distinct characteristics, voices, and behaviors. A persona might be a “frustrated elder” with a slow speaking pace or a “tech-savvy millennial” who uses industry jargon. Learn more →

​
2. Scenarios

The conversation blueprint - what the customer wants to accomplish. Scenarios can be simple (“check account balance”) or complex multi-turn interactions with conditional logic. Learn more →

​
3. Run Plans

Your test configuration that combines personas, scenarios, and agent endpoints into a comprehensive test matrix. One run plan might test 3 scenarios × 4 personas × 2 agents = 24 individual test calls. Learn more →

​
4. Evaluators

Automated assessments that score each simulation. Built-in evaluators check task completion, response quality, compliance, and more. Custom evaluators let you validate business-specific requirements. Learn more →

​
How Simulations Work

1

Configure Test Components

Create personas (who’s calling), scenarios (what they want), and select evaluators (how to measure success)
2

Build Run Plan

Combine your components into a test matrix. Choose which agents to test and how many iterations to run
3

Execute Tests

Run manually for immediate results or schedule for automated testing. Choose inbound (Roark calls agent) or outbound (agent calls Roark)
4

Analyze Results

Review conversation transcripts, evaluation scores, and aggregate metrics. Identify patterns and areas for improvement

​
Quick Start Guide

​
Your First Simulation (5 minutes)

  1. Create a Simple Persona
    • Navigate to Simulations → Personas
    • Click Create Persona
    • Choose “Happy Customer” template
    • Save
  2. Create a Basic Scenario
    • Navigate to Simulations → Scenarios
    • Click Create from Template
    • Select “Password Reset”
    • Save
  3. Build a Run Plan
    • Navigate to Simulations → Run Plans
    • Click Create Run Plan
    • Select your persona and scenario
    • Choose your agent endpoint
    • Select “Task Completion” evaluator
  4. Execute
    • Click Create and Run
    • Watch real-time progress
    • Review results when complete

​
Common Use Cases

​
Pre-Deployment Testing

Before launching a new agent or feature:
  • Test all expected customer paths
  • Verify edge case handling
  • Confirm compliance requirements
  • Validate integrations work correctly

​
Regression Testing

After making changes:
  • Run your standard test suite
  • Compare results to baseline
  • Catch unintended side effects
  • Ensure quality remains consistent

​
Performance Monitoring

For production agents:
  • Schedule daily/weekly test runs
  • Track performance trends
  • Detect degradation early
  • Validate SLA compliance

​
A/B Testing

When optimizing agents:
  • Test different prompt strategies
  • Compare model versions
  • Evaluate conversation flows
  • Make data-driven decisions

​
Testing Strategies

​
Start Simple

Begin with happy path scenarios and single personas. Once working, add complexity:
  1. Basic functionality tests
  2. Error handling scenarios
  3. Edge cases and difficult customers
  4. Full regression suites

​
Test Systematically

Use the test matrix approach:
  • Scenarios: Cover all major use cases
  • Personas: Include diverse customer types
  • Iterations: Run multiple times to catch intermittent issues
  • Evaluators: Measure what matters to your business

​
Automate Everything

Move from manual to automated:
  1. Manual testing for development
  2. Scheduled tests for regular validation
  3. Automated regression before deployments
  4. Continuous monitoring in production

​
Best Practices

Base personas and scenarios on real customer interactions from your call analytics
Focus first on high-value, high-frequency customer interactions
Add complexity gradually to isolate issues when they occur
Clear scenario descriptions help identify when agents deviate from requirements

​
Integration Points

Simulations work with your existing Roark setup:
  • Same Evaluators - Use production evaluators in test environments
  • Unified Analytics - View test and production data together
  • Consistent Metrics - Track the same KPIs across all environments
  • Shared Components - Reuse personas and scenarios across projects

​
Next Steps

Ready to start testing? Follow this path:

Create Personas

Define your virtual customers

Build Scenarios

Design test conversations

Configure Run Plans

Set up your test matrix

Execute Tests

Run your simulations
For automated testing, explore Schedules to run simulations on a recurring basis. To understand testing directions, read about Inbound vs Outbound modes.
⌘I