CI/CD

Overview

You can drive Roark from CI so your agent is tested on every change, not just by hand. It combines two things you already have:

Config as Code keeps your test suite (customer flows, personas, metrics, collectors) in your git repo, so CI can sync it with one command.
The CLI triggers a simulation run for a saved run plan.

The typical pipeline: on merge to your main branch, apply your config, then start a run for the plan that exercises your agent.

Both steps authenticate with a project API key that carries the right permissions (config:apply for the sync step). Store it as a CI secret and export it as ROARK_API_BEARER_TOKEN; every CLI command picks it up with no interactive login. See Using the CLI in CI.

Step 1: Sync your test suite

Keep your flows, personas, metrics, and collectors as YAML in your repo and apply them so the project matches what’s in git:

npx @roarkanalytics/cli config apply ./roark -y

apply exits non-zero if the bundle is invalid, so a broken config fails the build on its own. Run roark config diff ./roark on pull requests to preview changes before they land. See Config as Code for the full workflow.

Config as Code manages resource definitions. It does not start a run by itself, that’s the next step.

Step 2: Trigger a run

Start a run for a saved plan by its ID:

roark simulation plan job start <plan-id>

Find the plan ID on the plan’s page in the dashboard, or list your plans:

roark api get /v1/simulation/plan

The command returns the run’s simulationRunPlanJobId. The run then executes asynchronously against your agents.

Save the plan you want CI to run from the New Run flow (check Save as plan), so CI can reference a stable plan ID instead of re-specifying the run each time.

Step 3: See the results

Open the run in Roark to see its verdict: the pass rate across every check, per-conversation scores, and any failures, on the run report. If you want CI to wait for the run to finish, poll its lifecycle status until it reaches a terminal state (COMPLETED, FAILED, TIMED_OUT, or CANCELLED):

roark api get /v1/simulation/plan/job/<jobId>

Today the CLI reports whether the run was triggered and completed, not whether its checks passed: review the pass/fail verdict in the run report. A native pass/fail exit code for gating a build directly on the result is in progress.

GitHub Actions example

Sync config and kick off a run on every merge to main:

name: Roark simulations
on:
  push:
    branches: [main]

jobs:
  simulate:
    runs-on: ubuntu-latest
    env:
      ROARK_API_BEARER_TOKEN: ${{ secrets.ROARK_API_KEY }}
      PLAN_ID: ${{ vars.ROARK_PLAN_ID }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      # Keep the test suite (flows, personas, metrics, collectors) in sync.
      - name: Apply config
        run: npx @roarkanalytics/cli config apply ./roark -y

      # Trigger a simulation run for the saved plan.
      - name: Start simulation run
        run: npx @roarkanalytics/cli simulation plan job start "$PLAN_ID"

To preview config changes on pull requests instead of applying them, run npx @roarkanalytics/cli config diff ./roark in a pull_request-triggered job (the same pattern shown in Using the CLI in CI).

Config as Code

Define your test suite as YAML in git

Run plans

Build the reusable plan CI runs

CLI

Install, authenticate, and use the CLI in CI

Running simulations

Launch runs and read the report

Getting Started

Simulations

Observability

Metrics

Config as Code

Recipes

Integrations

Enterprise

SDKs & Libraries

Resources

Overview

Step 1: Sync your test suite

Step 2: Trigger a run

Step 3: See the results

GitHub Actions example

Config as Code

Run plans

CLI

Running simulations

​Overview

​Step 1: Sync your test suite

​Step 2: Trigger a run

​Step 3: See the results

​GitHub Actions example

​Related

Config as Code

Run plans

CLI

Running simulations

Overview

Step 1: Sync your test suite

Step 2: Trigger a run

Step 3: See the results

GitHub Actions example

Related