CI/CD

Overview

A simulation you have to remember to run is a test suite nobody runs. Put it in your pipeline and every change to your agent is called by simulated customers, scored against your metrics, and turned into a pass or a fail before it reaches production. Three pieces make that work, and only the first is required:

The GitHub Action starts a run, waits for it, and exits non-zero when the run misses its criteria.
Success criteria on the run plan decide what “missed” means. Roark makes that decision, so the dashboard, the CLI and your pipeline never disagree about whether a run passed.
Config as Code optionally keeps the test suite itself (customer flows, personas, metrics, collectors) in your repo, so the thing being run is versioned alongside the agent it tests.

Gate a pipeline in one step

name: Voice agent regression
on:
  push:
    branches: [main]

jobs:
  simulate:
    runs-on: ubuntu-latest
    steps:
      - uses: roarkhq/simulation-action@v1
        with:
          api-token: ${{ secrets.ROARK_API_KEY }}
          plan-id: <your-plan-id>

That is the whole gate. The step places the calls, waits for every one of them to be scored, prints a per-check table to the job summary, and fails the build if a check fell short.

The API key needs permission to start simulations and read their results. Store it as a repository secret. See Using the CLI in CI for how credentials resolve.

Pass runtime values the plan’s variables expect with variables, one KEY=VALUE per line:

      - uses: roarkhq/simulation-action@v1
        with:
          api-token: ${{ secrets.ROARK_API_KEY }}
          plan-id: <your-plan-id>
          variables: |
            orderNumber=12345
            customerName=John Doe

How the pass/fail decision is made

Every check is judged on its own. A check is one pass/fail metric: a yes/no metric, or a threshold on another metric the plan collects. Each one has a minimum pass rate, the share of the run’s simulations that must pass it, and the run passes only when every check clears its own minimum.

Check	Passed	Rate	Minimum	Result
silence duration < 500ms	4/10	40%	40%	Pass
word count < 1000	7/10	70%	80%	Fail

The run fails. word count missed its own bar, and no amount of success elsewhere makes up for it. There is no pooled average to clear, because averaging would let a check at 0% hide behind checks at 100%, which is the exact failure a gate exists to catch. A check that sets no minimum of its own is held to 80%. Set the minimums on the plan, under Advanced → Success criteria, or over the API. See Success criteria.

Criteria are pinned to each run when it starts, so editing a plan never rewrites the verdict of a run that already happened.

Score is not the decision

A run also reports a score: the mean of the check rates, 0 to 100. It is for dashboards and trend lines only. Nothing is gated on it, and you should not gate on it either. A run can score 95 and fail (one non-negotiable check missed its bar) or score 40 and pass (every check cleared a deliberately low bar).

Three outcomes, not two

A gate answers one question: did this change make the agent worse? Only a run that actually ran can answer it, so the action reports three outcomes.

Outcome	When	Exit code
PASSED	Every check cleared its own minimum.	0
FAILED	The run completed cleanly and a check fell short.	1
SKIPPED	The run never produced a judgeable result: a Roark outage, a model-provider error, simulations that died before they were graded, a check that produced no result.	0, with a warning

A check falling below its minimum is the only thing that turns your build red. Problems on our side are annotated loudly, reported as verdict: SKIPPED with a skip-reason, and then they get out of your way. That is deliberate. If our infrastructure can turn your pipeline red, the first thing your team learns is to re-run until green, and from then on nobody reads the gate at all, including on the run where the agent really did regress. Operational problems poison the verdict rather than sitting beside it. If half a run’s simulations never completed, a check reported at 40% is not evidence of a regression, it is evidence we could not measure. So any operational failure makes the whole run SKIPPED, even when a check also fell short. Two exceptions, both deliberate:

A plan with no checks fails the build. That is not an outage, it is a plan that can never gate anything, and skipping it would leave a green check next to a permanent no-op. Add a threshold to one of its metrics, or attach a yes/no metric.
A rejected request fails the build. A bad token, an unknown plan-id, or a config the API refuses is something only you can fix, and a gate that skips on a typo never gates again. Only errors that read as ours (5xx, rate limits, dead sockets) are skipped.

Set fail-on-run-error: true if you would rather block than proceed unmeasured.

Keep the run config in your repo

Instead of a saved plan, point the action at a YAML file. The run is described in the same commit as the agent change it tests:

.roark/checkout-regression.yml

name: Checkout regression
direction: OUTBOUND
maxSimulationDurationSeconds: 300
agentEndpoints:
  - id: <your-agent-endpoint-id>
flows:
  - id: <your-flow-id>
    happyPath: true
    edgeCases: ALL
metrics:
  - slug: agent_containment
    # No minPassRate, so this check is held to the 80% default.
  - slug: latency
    minPassRate: 12
  - slug: leaked_pii
    minPassRate: 100

      - uses: actions/checkout@v4
      - uses: roarkhq/simulation-action@v1
        with:
          api-token: ${{ secrets.ROARK_API_KEY }}
          config: .roark/checkout-regression.yml

minPassRate is the only thing a plan says about a check, because it is the only part that is the plan’s business. What counts as passing a single simulation belongs to the metric: a “did the agent leak PII?” check passes on FALSE because the metric carries a threshold of EQUALS false, stated once in Roark rather than re-stated by every plan that uses it. The API rejects keys it does not recognise, so a stray field here fails the run rather than being quietly ignored. This runs as a one-off: nothing is added to your saved plans. Set save-as-plan: true to keep it instead, and read the new plan’s id off the plan-id output so later runs can reference it directly.

Config as Code and this run config are different files with different jobs. Config as Code manages the resource definitions your runs draw on (flows, personas, metrics); the run config above says which of them to exercise, against which agent, at what volume, and it is the one that takes minPassRate. A plan declared as kind: simulationPlan in a Config as Code bundle cannot carry minimums yet, so its checks all sit at 80%.

Hold one branch to a higher bar

min-pass-rate holds every check to at least that minimum for this pipeline only, leaving the shared plan alone:

      - uses: roarkhq/simulation-action@v1
        with:
          api-token: ${{ secrets.ROARK_API_KEY }}
          plan-id: <your-plan-id>
          min-pass-rate: 99

It applies per check, on top of each check’s own minimum, so a release branch can demand more than the plan without editing it. It can only tighten: it will never let through a run the plan’s own criteria failed, because overruling a minimum the plan’s owner set is not something a pipeline gets to do.

Inputs

Input	Required	Default	Description
`api-token`	yes		Roark API key. Pass it from a secret.
`plan-id`	one of		Saved run plan to run.
`config`	one of		Path to a YAML file describing the run.
`variables`	no		Runtime variables, one `KEY=VALUE` per line.
`save-as-plan`	no	`false`	Keep the `config` as a named run plan instead of running it as a one-off.
`min-pass-rate`	no		Hold every check to at least this minimum (0-100) for this pipeline. Tightens only.
`timeout-minutes`	no	`30`	How long to wait for the run.
`poll-interval-seconds`	no	`15`	How often to check for completion.
`fail-on-timeout`	no	`false`	`true` fails the build when the run overruns instead of warning.
`fail-on-run-error`	no	`false`	`true` fails the build when the run could not be judged instead of skipping.
`cancel-on-exit`	no	`true`	Stop the Roark run when the workflow is cancelled or the wait times out.
`cli-version`	no	pinned	Version of `@roarkanalytics/cli` to run.
`api-base-url`	no	`https://api.roark.ai`	Override the API base URL.

Outputs

Output	Description
`run-id`	The simulation run id.
`run-url`	Link to the run in the Roark platform.
`plan-id`	The run plan behind this run. With `save-as-plan`, the plan that was kept.
`score`	The run’s quality score, 0-100. Reporting only: gate on `verdict`, never this.
`checks-passed`	How many checks cleared their own minimum.
`checks-total`	How many checks the run was judged on.
`verdict`	`PASSED`, `FAILED`, `SKIPPED`, or `TIMED_OUT`.
`skip-reason`	Why the run could not be judged. Set only alongside `SKIPPED`.

Simulations take minutes

Real calls take real time, so a gated run occupies a runner while it waits. Four ways to keep that predictable:

Run it where it matters. Gate main or your release branch rather than every push to every branch.
Let our slowness warn rather than block. An overrun is a warning by default (fail-on-timeout: false), while a genuine check failure still fails the build.
Cancel superseded runs. A concurrency group stops an old push from holding a runner while a newer one is already testing the same branch. The action cancels the Roark run too, so the abandoned simulation stops placing calls you would otherwise be billed for.
Keep a job-level backstop. timeout-minutes on the job is the last line of defence if the step itself wedges.

concurrency:
  group: roark-${{ github.ref }}
  cancel-in-progress: true

Sync your test suite first

If your flows, personas and metrics live in git, apply them before the run so the suite being executed matches the commit:

npx @roarkanalytics/cli config apply ./roark -y

apply exits non-zero if the bundle is invalid, so a broken config fails the build on its own. Run roark config diff ./roark on pull requests to preview changes before they land. See Config as Code for the full workflow.

      - name: Apply config
        run: npx @roarkanalytics/cli config apply ./roark -y
        env:
          ROARK_API_BEARER_TOKEN: ${{ secrets.ROARK_API_KEY }}

      - uses: roarkhq/simulation-action@v1
        with:
          api-token: ${{ secrets.ROARK_API_KEY }}
          plan-id: <your-plan-id>

Outside GitHub Actions

The action is a convenience, not the mechanism. Any CI system can gate on the same verdict with the CLI or a pair of HTTP calls: start a run, poll until it reaches a terminal status, then read verdict.passed.

export ROARK_API_BEARER_TOKEN="$ROARK_API_KEY"

run_id="$(roark simulation plan job start "$PLAN_ID" | jq -r '.data.simulationRunPlanJobId')"

# Poll until the run reaches a terminal status.
while :; do
  response="$(roark simulation plan job get "$run_id")"
  status="$(printf '%s' "$response" | jq -r '.data.status')"
  case "$status" in
    COMPLETED | FAILED | CANCELLED | TIMED_OUT) break ;;
  esac
  sleep 15
done

# `passed` is the exit status. `score` is reporting only.
printf '%s' "$response" | jq -e '.data.verdict.passed == true' >/dev/null

The verdict is attached to GET /v1/simulation/plan/job/{jobId} once the run has ended:

{
  "passed": false,
  "score": 85,
  "checks": [
    {
      "metricDefinitionId": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
      "metricName": "silence duration < 500ms",
      "passRate": 100,
      "minPassRate": 40,
      "inherited": false,
      "passedSims": 10,
      "evaluatedSims": 10,
      "passed": true
    },
    {
      "metricDefinitionId": "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
      "metricName": "word count < 1000",
      "passRate": 70,
      "minPassRate": 80,
      "inherited": false,
      "passedSims": 7,
      "evaluatedSims": 10,
      "passed": false
    }
  ],
  "failures": [
    {
      "type": "METRIC_BELOW_MIN_PASS_RATE",
      "metricDefinitionId": "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
      "metricName": "word count < 1000",
      "passRate": 70,
      "minPassRate": 80,
      "inherited": false
    }
  ]
}

failures is a typed list rather than a rendered message, so your pipeline can tell the two kinds apart and treat them differently:

`type`	Meaning	Whose problem
`METRIC_BELOW_MIN_PASS_RATE`	A check fell short of its minimum.	Your agent. Fail the build.
`RUN_NOT_COMPLETED`	The run ended in `FAILED`, `CANCELLED` or `TIMED_OUT`.	Ours. Retry or warn.
`INCOMPLETE_COVERAGE`	Fewer simulations were graded than the run expected.	Ours. The numbers are not evidence.
`METRIC_NOT_EVALUATED`	A check produced no result at all.	Ours.

inherited: true means the check was judged against the 80% default rather than a minimum it set for itself.

verdict is null when the plan carries no pass/fail check, because there was nothing to judge. Treat that as a configuration error rather than a pass: a plan that cannot fail is a gate that does nothing.

Success criteria

Set the minimum each check has to clear

Config as Code

Define your test suite as YAML in git

CLI

Install, authenticate, and use the CLI in CI

Running simulations

Launch runs and read the report when they settle

Getting Started

Simulations

Observability

Metrics

Config as Code

Autoimprove

Recipes

Integrations

Enterprise

SDKs & Developer Tools

Resources

Overview

Gate a pipeline in one step

How the pass/fail decision is made

Score is not the decision

Three outcomes, not two

Keep the run config in your repo

Hold one branch to a higher bar

Inputs

Outputs

Simulations take minutes

Sync your test suite first

Outside GitHub Actions

Success criteria

Config as Code

CLI

Running simulations

​Overview

​Gate a pipeline in one step

​How the pass/fail decision is made

​Score is not the decision

​Three outcomes, not two

​Keep the run config in your repo

​Hold one branch to a higher bar

​Inputs

​Outputs

​Simulations take minutes

​Sync your test suite first

​Outside GitHub Actions

​Related

Success criteria

Config as Code

CLI

Running simulations

Overview

Gate a pipeline in one step

How the pass/fail decision is made

Score is not the decision

Three outcomes, not two

Keep the run config in your repo

Hold one branch to a higher bar

Inputs

Outputs

Simulations take minutes

Sync your test suite first

Outside GitHub Actions

Related