Run a simulation

The end-to-end path, on one page: define a simulation in your repo, apply it, run it, and read the pass/fail. Config as code defines the resources; running is a separate, deliberate step (so applying never places a surprise call).

You need the CLI (npm install -g @roarkanalytics/cli) and a Roark API key with config:apply and permission to start simulations and read results. Set it once: export ROARK_API_BEARER_TOKEN="<key>".

1. Define an agent and a plan

The minimum to run something: an agent to call and a simulation plan that says which flows to run against it and which metrics grade the result. This plan runs a built-in (system) flow, so there’s no persona or flow to author first.

import { defineAgent, defineSimulationPlan, defineConfig } from '@roarkanalytics/sdk/lib/config';

const frontdesk = defineAgent({
  name: 'frontdesk',
  endpoints: [{ value: '+15551234567', direction: 'OUTGOING' }],
});

const nightly = defineSimulationPlan({
  name: 'nightly-screener-suite',
  direction: 'OUTBOUND',
  agentEndpoints: [{ agent: 'frontdesk', value: '+15551234567' }],
  metrics: ['call_screening_encountered', 'call_screening_handling_score'],
  flows: [{ system: 'sf-screener-apple' }],
  maxDurationSeconds: 300,
});

export default defineConfig({ resources: [frontdesk, nightly] });

# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
# (plus roark/agents/frontdesk.yaml defining the agent)
kind: simulationPlan
name: nightly-screener-suite
direction: OUTBOUND
agentEndpoints:
  - agent: frontdesk
    value: '+15551234567'
metrics:
  - call_screening_encountered
  - call_screening_handling_score
flows:
  - system: sf-screener-apple
maxDurationSeconds: 300

Swap the system flow for your own customer flows and personas once you want to test your own scenarios. roark simulation template list prints every built-in flow slug.

2. Apply it

Create the resources (preview first, then apply):

roark config diff roark.config.ts    # or ./roark for a YAML directory
roark config apply roark.config.ts

This creates the agent and the plan. It does not run anything yet.

3. Run it

Running is a separate command. It takes the plan’s id, not its config name, so look the id up once (or copy it from the plan’s page in the dashboard):

# find the plan's id by its name
roark simulation plan list | jq -r '.data[] | select(.name=="nightly-screener-suite") | .id'

# start a run
roark simulation run --data '{"planId":"<plan-id>"}'

roark simulation run returns a job id (the run). It places the calls, grades each against the plan’s metrics, and the result settles when every call has been scored.

Prefer not to leave your pipeline? The GitHub Action starts the run and waits for it in one step, and the Node SDK exposes the same via client.simulation.run({ planId }).

4. Read the result

roark simulation plan job result <job-id>

You get the per-check pass/fail against the plan’s success criteria, plus a link to the full report in the dashboard. Roark decides pass vs. fail, so the CLI, the dashboard, and your CI pipeline never disagree.

5. Gate CI on it

For a pipeline, skip the manual run and let the GitHub Action start the run, wait, and fail the build when a check falls short:

.github/workflows/agent-regression.yml

name: Agent regression
on:
  push:
    branches: [main]
jobs:
  simulate:
    runs-on: ubuntu-latest
    steps:
      # Optional: keep the plan itself in the repo and apply it first.
      - uses: actions/checkout@v4
      - run: npx -y @roarkanalytics/cli config apply roark.config.ts --yes
        env:
          ROARK_API_BEARER_TOKEN: ${{ secrets.ROARK_API_KEY }}
      # Start the run, wait for it, and gate the build on the result.
      - uses: roarkhq/simulation-action@v1
        with:
          api-token: ${{ secrets.ROARK_API_KEY }}
          plan-id: <your-plan-id>

See CI/CD for the full action reference.

Or: run it on a schedule

To run without a trigger at all, add a schedule to the plan and apply, the plan runs itself on the cadence you set.

Simulation plans reference

Every field a plan takes: agents, flows, metrics, direction, scheduling, and knowledge checks.

Get started

Resource reference

1. Define an agent and a plan

2. Apply it

3. Run it

4. Read the result

5. Gate CI on it

Or: run it on a schedule

Simulation plans reference

​1. Define an agent and a plan

​2. Apply it

​3. Run it

​4. Read the result

​5. Gate CI on it

​Or: run it on a schedule

Simulation plans reference

1. Define an agent and a plan

2. Apply it

3. Run it

4. Read the result

5. Gate CI on it

Or: run it on a schedule