# Discard the staging revision
Source: https://docs.roark.ai/api-reference/agent-config/discard-the-staging-revision

/api-reference/openapi.documented.json delete /v1/agent-config/{key}/staging
Clear the staging channel. Production is untouched; recognized simulation sessions go back to reading production.

# Get a managed config
Source: https://docs.roark.ai/api-reference/agent-config/get-a-managed-config

/api-reference/openapi.documented.json get /v1/agent-config/{key}
Fetch one managed config with its channel pointers and recent revisions, newest first.

# List managed configs
Source: https://docs.roark.ai/api-reference/agent-config/list-managed-configs

/api-reference/openapi.documented.json get /v1/agent-config
List the managed agent configs in this project, most recent first.

# Promote staging to production
Source: https://docs.roark.ai/api-reference/agent-config/promote-staging-to-production

/api-reference/openapi.documented.json post /v1/agent-config/{key}/promote
Point the production channel at the staging revision. Real traffic reads it from the next session onward; roll back by writing the prior revision id to production (the chain preserves every state).

# Resolve config for a session
Source: https://docs.roark.ai/api-reference/agent-config/resolve-config-for-a-session

/api-reference/openapi.documented.json post /v1/agent-config/{key}/resolve
The call your agent makes at session start. Returns the config JSON for this session, with per-session simulation recognition built in: when the session was originated by a Roark simulation (matched by caller number against the calls Roark has in flight), the STAGING revision is served for that session only, so Autoimprove candidates are tested on your real deployment. Real traffic always resolves to production; any recognition miss degrades to production too.

On the first fetch of an unknown key, pass `defaults` (your baked-in config): the key is registered and the defaults become revision 1. That makes integration a single call.

Cache the response per session; do not fetch per turn.

# Write a config revision
Source: https://docs.roark.ai/api-reference/agent-config/write-a-config-revision

/api-reference/openapi.documented.json put /v1/agent-config/{key}
Write a new revision onto a channel. Writing production creates the key when it does not exist; writing staging stages a candidate that only Roark-recognized simulation sessions will read. Every write is a new revision; nothing is overwritten.

# Create a new agent endpoint
Source: https://docs.roark.ai/api-reference/agent-endpoint/create-a-new-agent-endpoint

/api-reference/openapi.documented.json post /v1/agent/endpoint
Creates a new agent endpoint for the authenticated project.

# Get agent endpoint by ID
Source: https://docs.roark.ai/api-reference/agent-endpoint/get-agent-endpoint-by-id

/api-reference/openapi.documented.json get /v1/agent/endpoint/{endpointId}
Returns a specific agent endpoint by its ID.

# List agent endpoints
Source: https://docs.roark.ai/api-reference/agent-endpoint/list-agent-endpoints

/api-reference/openapi.documented.json get /v1/agent/endpoint
Returns a paginated list of agent endpoints for the authenticated project.

# Update an agent endpoint
Source: https://docs.roark.ai/api-reference/agent-endpoint/update-an-agent-endpoint

/api-reference/openapi.documented.json put /v1/agent/endpoint/{endpointId}
Updates an existing agent endpoint by its ID. Only environment and outboundDialType can be modified.

# Build a Roark-hosted agent
Source: https://docs.roark.ai/api-reference/agent/build-a-roark-hosted-agent

/api-reference/openapi.documented.json post /v1/agent/build
Creates a new Roark-hosted agent from a one-line job description. Roark authors the system prompt and hosts the config, so the agent is live and self-improvable from creation.

# Create a new agent
Source: https://docs.roark.ai/api-reference/agent/create-a-new-agent

/api-reference/openapi.documented.json post /v1/agent
Creates a new agent for the authenticated project.

# Delete an agent
Source: https://docs.roark.ai/api-reference/agent/delete-an-agent

/api-reference/openapi.documented.json delete /v1/agent/{agentId}
Soft-deletes an agent by its ID. The agent is hidden from all reads and stops being attributed to new calls, but its record and history are retained. Fails with 409 if the agent is still referenced by simulation run plans: cancel those run plans first. Note: if the agent syncs from a provider integration, also exclude it in the integration settings (or delete the integration) so its calls stop reaching Roark.

# Discard the draft
Source: https://docs.roark.ai/api-reference/agent/discard-the-draft

/api-reference/openapi.documented.json delete /v1/agent/{agentId}/draft
Throws the open draft away and returns it. The live version is unchanged.

# Edit the draft
Source: https://docs.roark.ai/api-reference/agent/edit-the-draft

/api-reference/openapi.documented.json put /v1/agent/{agentId}/draft
Changes settings on the draft. Omitted settings stay as they are. Pass the revision you read as expectedRevision to fail with 409 if anyone edited the draft since, instead of applying on top of their edit.

# Get agent by ID
Source: https://docs.roark.ai/api-reference/agent/get-agent-by-id

/api-reference/openapi.documented.json get /v1/agent/{agentId}
Returns a specific agent by its ID.

# Get an agent version
Source: https://docs.roark.ai/api-reference/agent/get-an-agent-version

/api-reference/openapi.documented.json get /v1/agent/{agentId}/versions/{versionId}
One version with every setting, plus the issues that stop it from running, if any.

# Get an agent's draft
Source: https://docs.roark.ai/api-reference/agent/get-an-agents-draft

/api-reference/openapi.documented.json get /v1/agent/{agentId}/draft
The agent's open draft, with the issues that would stop it from being published.

# List a prompt's versions
Source: https://docs.roark.ai/api-reference/agent/list-a-prompts-versions

/api-reference/openapi.documented.json get /v1/agent/{agentId}/prompts/{promptId}/versions
Returns the full version history of a single prompt lineage, newest first.

# List agents
Source: https://docs.roark.ai/api-reference/agent/list-agents

/api-reference/openapi.documented.json get /v1/agent
Returns a paginated list of agents for the authenticated project.

# List an agent's prompts
Source: https://docs.roark.ai/api-reference/agent/list-an-agents-prompts

/api-reference/openapi.documented.json get /v1/agent/{agentId}/prompts
Returns the agent's prompt lineages. Each lineage is an independent version history, labelled by its `source` (USER, API, CONFIG, or a provider integration). `prompt` is the current content.

# List an agent's versions
Source: https://docs.roark.ai/api-reference/agent/list-an-agents-versions

/api-reference/openapi.documented.json get /v1/agent/{agentId}/versions
The open draft first, if there is one, then published versions newest first. Paginated: pass `pagination.nextCursor` as `after` for the next page.

# Publish the draft
Source: https://docs.roark.ai/api-reference/agent/publish-the-draft

/api-reference/openapi.documented.json post /v1/agent/{agentId}/draft/publish
Makes the draft the live version, numbered v1, v2, ... Fails with 409 if the draft was edited after expectedRevision, if another version was published since the draft started, or if it changes nothing; with 400 and the issues if the version cannot run.

# Roll an agent back to a version
Source: https://docs.roark.ai/api-reference/agent/roll-an-agent-back-to-a-version

/api-reference/openapi.documented.json post /v1/agent/{agentId}/versions/{versionId}/rollback
Publishes a new version, numbered after the latest, with this version's settings and makes it live. History stays append-only. An open draft started before this is kept but can no longer be published (409 `agent_draft_base_changed`); discard it or start a new one. Pass `expectedLiveVersionId` to be refused (409 `agent_live_version_changed`) if another version went live since you read it.

# Set an agent's prompt
Source: https://docs.roark.ai/api-reference/agent/set-an-agents-prompt

/api-reference/openapi.documented.json put /v1/agent/{agentId}/prompts
Sets the agent's API-managed prompt. This is its own version history (`source: API_MANAGED`), separate from prompts observed on calls, edited in the app, or managed by config-as-code. Setting the same content twice is a no-op (no new version). Roark does not run your agent and no metric reads this prompt: it is stored and versioned for your reference.

# Start a draft
Source: https://docs.roark.ai/api-reference/agent/start-a-draft

/api-reference/openapi.documented.json post /v1/agent/{agentId}/draft
Opens the agent's one shared draft, a copy of the live version or of an older published version. Fails with 409 when a draft is already open.

# Update an agent
Source: https://docs.roark.ai/api-reference/agent/update-an-agent

/api-reference/openapi.documented.json put /v1/agent/{agentId}
Updates an existing agent by its ID.

# Get the analytics schema
Source: https://docs.roark.ai/api-reference/analytics/get-the-analytics-schema

/api-reference/openapi.documented.json get /v1/analytics/schema
Lists what can go into an analytics query for this project: every metric by slug with its output
type, default aggregation and the aggregations it supports; the measure types; the filter and
breakdown types with the properties each accepts; and the custom call properties seen in the project.
Read this first when building a query.

# Run an analytics query
Source: https://docs.roark.ai/api-reference/analytics/run-an-analytics-query

/api-reference/openapi.documented.json post /v1/analytics/query
Aggregates metric values and conversation counts over a time window and returns one series per
measure, with a total for the window and per-interval buckets. The same query a dashboard widget
runs, without saving anything.

- `measures`: what to aggregate. A METRIC measure names its metric by slug; CONVERSATION counts
  conversations; UPTIME reads health checks. The aggregation defaults by the metric's output type.
- `timeRange`: LAST n DAYS/WEEKS/MONTHS, a CALENDAR_ALIGNED period (TODAY, THIS_MONTH, ...), FIXED
  from/to, or SINCE. Defaults to the last 30 days; at most 731 days. Calendar periods and buckets
  align to `timezone` (IANA, default UTC).
- `filters` apply to every measure; a measure's own `filters` add to them.
- `breakdown` on a measure returns one series per value (largest 50 kept).
- `comparisonGroups` (2 to 5) run every measure once per named filter set;
  `comparisonMode: PREVIOUS_PERIOD` adds the preceding window's values.

Values are in the metric's display units. A PASS_RATE is a 0 to 1 share; `sampleSize` 0 means no data.
A query the analytics store cannot answer returns 400 naming the unsupported part.

# Authorization
Source: https://docs.roark.ai/api-reference/authorization

How to authenticate API requests

### Overview

To securely access Roark's API endpoints, you need to authenticate your requests using your unique API key. This ensures that only authorized users can interact with our services.

> [!WARNING]
>
> Never expose your API key in client-side code, public repositories, or share it with unauthorized users. If you believe your key has been compromised, regenerate it immediately in the [dashboard](/documentation/getting-started/api-keys).

### Authentication Header

```bash Headers theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
Authorization: Bearer YOUR_API_KEY
```

```json Response theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "authenticated": true,
  "account": "your-account-name"
}
```

### Example Requests

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# The CLI authenticates once and signs every request for you.
roark auth login
roark api get /health
```

```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X GET "https://api.roark.ai/health" \
     -H "Authorization: Bearer YOUR_API_KEY"
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import requests

headers = {
    "Authorization": "Bearer YOUR_API_KEY"
}

response = requests.get("https://api.roark.ai/health", headers=headers)
```

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const response = await fetch("https://api.roark.ai/health", {
  headers: {
    "Authorization": `Bearer YOUR_API_KEY`
  }
});
```

### Security Best Practices

> [card] Secure Storage
>
> Store API keys securely in environment variables or a secure key management system

> [card] Regular Rotation
>
> Periodically rotate your API keys to maintain security

### Need an API Key?

> [card] [Generate API Key](/documentation/getting-started/api-keys)
>
> Follow our guide to generate your API key and start making authenticated requests


# Answer Roark's question
Source: https://docs.roark.ai/api-reference/autoimprove/answer-roarks-question

/api-reference/openapi.documented.json post /v1/autoimprove/job/{jobId}/answer
Answer the question a job is blocked on. Only meaningful while the job status is NEEDS_INPUT (the open QUESTION entry carries the offered options; free text is also accepted). Otherwise returns a conflict.

# Cancel a job
Source: https://docs.roark.ai/api-reference/autoimprove/cancel-a-job

/api-reference/openapi.documented.json post /v1/autoimprove/job/{jobId}/cancel
Stop a live job. Production is never changed by a cancel; everything the job set up (the shadow agent, its phone number, authored test flows and run plan) is cleaned up automatically.

# Dismiss a verified job
Source: https://docs.roark.ai/api-reference/autoimprove/dismiss-a-verified-job

/api-reference/openapi.documented.json post /v1/autoimprove/job/{jobId}/dismiss
Discard a verified job without promoting: production stays untouched and the staging resources are cleaned up. Only a job in AWAITING_PROMOTE can be dismissed.

# Get an Autoimprove job
Source: https://docs.roark.ai/api-reference/autoimprove/get-an-autoimprove-job

/api-reference/openapi.documented.json get /v1/autoimprove/job/{jobId}
Fetch one job with its full worklog: every step Roark took, the validation batches with their pass-rate movement, any question it is waiting on, and its final report once concluded.

# List Autoimprove jobs
Source: https://docs.roark.ai/api-reference/autoimprove/list-autoimprove-jobs

/api-reference/openapi.documented.json get /v1/autoimprove/job
List the Autoimprove jobs in this project, most recent first, capped at 100.

A job is one autonomous engagement: Roark improving one agent toward one objective metric on a staging copy, with a human-gated promote to production at the end.

# Promote a verified job to production
Source: https://docs.roark.ai/api-reference/autoimprove/promote-a-verified-job-to-production

/api-reference/openapi.documented.json post /v1/autoimprove/job/{jobId}/promote
Apply the verified staging changes to the PRODUCTION agent. Only a job in AWAITING_PROMOTE can be promoted.

A snapshot of the production configuration is taken immediately before the write, so the promote is fully rollbackable. After the promote the staging shadow and its phone number are cleaned up.

# Send guidance to a running job
Source: https://docs.roark.ai/api-reference/autoimprove/send-guidance-to-a-running-job

/api-reference/openapi.documented.json post /v1/autoimprove/job/{jobId}/guidance
Steer Roark mid-job. The message is folded into its next decision and is binding. Accepted while the job is live (RUNNING, NEEDS_INPUT, or PAUSED); a concluded job returns a conflict.

# Start an Autoimprove job
Source: https://docs.roark.ai/api-reference/autoimprove/start-an-autoimprove-job

/api-reference/openapi.documented.json post /v1/autoimprove/job
Commission Roark on an objective. It clones the agent into a staging shadow (or uses your designated staging agent), authors a validation suite of simulated callers, measures a baseline, changes the staging configuration, and re-tests until the objective metric passes its target. Production is never touched by the loop; verified changes wait for promotion.

Requires an active provider integration (Vapi, Retell, or ElevenLabs) with agent config writes enabled. One live job per agent: starting a second returns a conflict.

The job runs asynchronously; poll GET /v1/autoimprove/job/{jobId} or watch it in the dashboard. When its status is NEEDS_INPUT, answer via the answer endpoint; when AWAITING_PROMOTE, promote or dismiss.

# Get a benchmark target
Source: https://docs.roark.ai/api-reference/benchmark/get-a-benchmark-target

/api-reference/openapi.documented.json get /v1/benchmark/target/{targetKey}
Returns a target’s published sweep and every one of its aggregate cells: the overall rollup plus a set per condition. Defaults to the current sweep; pass `publicationId` from the history endpoint to read a superseded one, which is how a regression is compared generation to generation. Numbers only — no transcript and no audio.

# Get the benchmark leaderboard
Source: https://docs.roark.ai/api-reference/benchmark/get-the-benchmark-leaderboard

/api-reference/openapi.documented.json get /v1/benchmark/leaderboard
Returns one ranked page of a suite version’s published targets, each with the metric cells it is judged on. Ranking happens in the database, so paging through the board is consistent. Every default the server applies (suite version, condition, sort metric, sort direction) is echoed on the response, so a page can be cited without guessing how it was ordered.

# List a target’s published generations
Source: https://docs.roark.ai/api-reference/benchmark/list-a-target’s-published-generations

/api-reference/openapi.documented.json get /v1/benchmark/target/{targetKey}/history
Returns every sweep published for a target, newest first, including superseded ones. This is the trend read: it answers "did this model regress?" from Roark’s own published record. Metadata only — pass a row’s `publicationId` to GET /v1/benchmark/target/{targetKey} for that generation’s numbers.

# List benchmark metrics
Source: https://docs.roark.ai/api-reference/benchmark/list-benchmark-metrics

/api-reference/openapi.documented.json get /v1/benchmark/metric
Returns the metrics a suite version published, each with the direction that counts as better. Use it to pick a valid `sortBy` for the leaderboard: a metric key outside this list is rejected rather than silently ranking every target null.

# List benchmark suites
Source: https://docs.roark.ai/api-reference/benchmark/list-benchmark-suites

/api-reference/openapi.documented.json get /v1/benchmark/suite
Returns every benchmark suite with published results, along with the suite version the other endpoints default to. Start here: the suite name is a required parameter everywhere else and there is no other way to discover it.

# List score samples for a target
Source: https://docs.roark.ai/api-reference/benchmark/list-score-samples-for-a-target

/api-reference/openapi.documented.json get /v1/benchmark/target/{targetKey}/score-sample
Returns the individual scored calls behind a target’s aggregate cells, each with the scorer’s rationale — the "why is this number what it is" read. Filter to one cell with `conditionKey` and `metricKey`. Metrics computed without a rationale (latencies, counts) contribute no samples. Returns reasoning only: no transcript and no audio.

# Check a calendar connection
Source: https://docs.roark.ai/api-reference/calendar-connection/check-a-calendar-connection

/api-reference/openapi.documented.json post /v1/calendar-connection/{connectionId}/check
Re-verifies the stored credential against the provider and records the result on the connection. A credential the provider rejects sets `status` to INVALID_CREDENTIALS. A provider outage is a 500 and leaves the status untouched: being unable to reach the provider is not evidence that the customer’s credential is bad.

# Connect a calendar account
Source: https://docs.roark.ai/api-reference/calendar-connection/connect-a-calendar-account

/api-reference/openapi.documented.json post /v1/calendar-connection
Verifies the credential against the provider and, only if it works, stores it encrypted. A credential the provider rejects is a 400 and no connection is created: a connection that has never worked would next be exercised by an inbound call at 2am, with nobody there to fix it.

# Delete a calendar connection
Source: https://docs.roark.ai/api-reference/calendar-connection/delete-a-calendar-connection

/api-reference/openapi.documented.json delete /v1/calendar-connection/{connectionId}
Disconnects the calendar and destroys the stored credential. A credential the customer deleted should stop existing, so this is not an archive. Appointments already recorded against the connection are kept, so a booking stays explainable after the calendar is disconnected. Refused with a 409 while a calendar tool is still bound to the connection, naming the tools: point them at another calendar, or delete them, first. A dangling binding would otherwise be found by whoever next tried to ship the agent, who is not the person who disconnected the calendar.

# Get a calendar connection
Source: https://docs.roark.ai/api-reference/calendar-connection/get-a-calendar-connection

/api-reference/openapi.documented.json get /v1/calendar-connection/{connectionId}
Returns one calendar connection. The stored credential is never included.

# List calendar connections
Source: https://docs.roark.ai/api-reference/calendar-connection/list-calendar-connections

/api-reference/openapi.documented.json get /v1/calendar-connection
Returns every calendar connection in the authenticated project, by label. A project has a handful of these, so the list is not paginated: `pagination.hasMore` is always false.

# List what can be booked on a calendar
Source: https://docs.roark.ai/api-reference/calendar-connection/list-what-can-be-booked-on-a-calendar

/api-reference/openapi.documented.json get /v1/calendar-connection/{connectionId}/targets
What an agent can be pointed at on this connection: Cal.com event types today, Google calendars when that lands. Read live from the provider, so an event type added since you connected is here without reconnecting. Each entry carries a `target` ready to put on a calendar tool, so nothing has to assemble one by hand.

# Read one bookable target
Source: https://docs.roark.ai/api-reference/calendar-connection/read-one-bookable-target

/api-reference/openapi.documented.json get /v1/calendar-connection/{connectionId}/targets/{targetId}
The same entry as the list, plus what it demands of a booker. `requiresAttendeeEmail` is the field that decides an agent’s design: the attendee object itself needs only a name and a timezone, but an event type can declare email required, and an agent that cannot collect one simply cannot book on it. Read this when configuring a calendar tool, not after a caller is already waiting. Costs one provider call more than the list, which is why the list does not fold it in.

# Update a calendar connection
Source: https://docs.roark.ai/api-reference/calendar-connection/update-a-calendar-connection

/api-reference/openapi.documented.json patch /v1/calendar-connection/{connectionId}
Renames a connection, changes the timezone slots are offered in, or rotates the credential. A rotated credential is verified first, and a verified credential returns the connection to ACTIVE.

# Append tool invocations to a call
Source: https://docs.roark.ai/api-reference/call/append-tool-invocations-to-a-call

/api-reference/openapi.documented.json post /v1/call/{callId}/tool-invocations
Attach tool invocations that fired during a call to an already-existing call, asynchronously after the call was created. Use this when the tool-call data becomes available later than the call itself (e.g. a Roark simulation, or a call submitted via POST /v1/call before its tools were ready). Writes are idempotent: re-sending an invocation already present on the call (same tool name and timing) is skipped, so retries converge instead of double-counting. Optionally pass `metrics` to (re)score metrics over the newly attached tools, which triggers a billed metric collection job and requires the `metric:create` permission.

# Create a call
Source: https://docs.roark.ai/api-reference/call/create-a-call

/api-reference/openapi.documented.json post /v1/call
Create a new call with recording, transcript, agents, and customers

# Delete a call
Source: https://docs.roark.ai/api-reference/call/delete-a-call

/api-reference/openapi.documented.json delete /v1/call/{callId}
Deletes a call by its ID. The call, its metrics and its analysis are hidden from every read, and any analysis still running on it is cancelled. Returns 404 when the call does not exist in this project or was already deleted.

# Delete calls in batch
Source: https://docs.roark.ai/api-reference/call/delete-calls-in-batch

/api-reference/openapi.documented.json post /v1/call/batch-delete
Deletes up to 100 calls by ID in one request. Each deleted call, its metrics and its analysis are hidden from every read, and any analysis still running on it is cancelled. IDs that do not exist in this project, or were already deleted, are returned in notFoundCallIds and left untouched. The request succeeds even when none of the calls are found. IDs are returned lowercase, each once.

# Get a call by ID
Source: https://docs.roark.ai/api-reference/call/get-a-call-by-id

/api-reference/openapi.documented.json get /v1/call/{callId}
Retrieve an existing call by its unique identifier

# Get call cost
Source: https://docs.roark.ai/api-reference/call/get-call-cost

/api-reference/openapi.documented.json get /v1/call/{callId}/cost
The billable spend attributed to one call, split by cost type, with every cost line behind it. Simulated calls include the costs their simulation incurred before the call connected. Amounts are billable spend only, in exact micro-dollars (`*UsdMicros`) and dollars (`*Usd`, for display). Requires an API key with the `billing:read` permission.

# Get call transcript
Source: https://docs.roark.ai/api-reference/call/get-call-transcript

/api-reference/openapi.documented.json get /v1/call/{callId}/transcript
Fetch the full transcript for a specific call. Optionally specify a transcription source; otherwise the best available source is used automatically.

# List call metrics
Source: https://docs.roark.ai/api-reference/call/list-call-metrics

/api-reference/openapi.documented.json get /v1/call/{callId}/metrics
Fetch all call-level metrics for a specific call, including both system-generated and custom metrics. Only returns rows from the **latest** metric-collection job per metric — if the same metric has been recomputed, prior runs are excluded and remain in the metric history. By default returns only successfully computed metrics; pass `?status=all` to also include rows that resolved as NOT_APPLICABLE / DATA_MISSING / ERROR (the `value` field is omitted on those entries — check `captureStatus`).

# List call sentiment runs
Source: https://docs.roark.ai/api-reference/call/list-call-sentiment-runs

/api-reference/openapi.documented.json get /v1/call/{callId}/sentiment-run
Fetch detailed sentiment analysis results for a specific call, including emotional tone, key phrases, and sentiment scores.

# List calls
Source: https://docs.roark.ai/api-reference/call/list-calls

/api-reference/openapi.documented.json get /v1/call
Returns a paginated list of calls for the authenticated project. Pass agentId to keep calls for specific agents, or properties[agent_name][eq] when you only have the agent name. sortBy and sortDirection apply together with those filters.

# Update the properties of a call
Source: https://docs.roark.ai/api-reference/call/update-the-properties-of-a-call

/api-reference/openapi.documented.json patch /v1/call/{callId}/properties
Add, overwrite or remove custom properties on an already-existing call. Use this for data that becomes available after the call was created, e.g. an outcome your system validates a few minutes later. Merge semantics: a value adds or overwrites the property, null removes it, and properties left out are untouched. The patch is applied whole or not at all: a property Roark sets itself, or a value whose type differs from the type the project already stores that property as, rejects the request with a 400 and nothing is written. A 409 means another request created one of the properties with a different value type at the same moment; nothing was written. Optionally pass `metrics` to (re)score metrics over the updated properties, which triggers a billed metric collection job and requires the `metric:create` permission.

# Create a chat
Source: https://docs.roark.ai/api-reference/chat/create-a-chat

/api-reference/openapi.documented.json post /v1/chat
Create a new chat with segments (messages and tool invocations)

# Get a chat by ID
Source: https://docs.roark.ai/api-reference/chat/get-a-chat-by-id

/api-reference/openapi.documented.json get /v1/chat/{id}
Retrieve an existing chat by its unique identifier

# Get chat transcript
Source: https://docs.roark.ai/api-reference/chat/get-chat-transcript

/api-reference/openapi.documented.json get /v1/chat/{id}/transcript
Fetch the full transcript (messages) for a specific chat.

# List chat metrics
Source: https://docs.roark.ai/api-reference/chat/list-chat-metrics

/api-reference/openapi.documented.json get /v1/chat/{id}/metrics
Fetch all metrics for a specific chat, including both system-generated and custom metrics. Only returns rows from the **latest** metric-collection job per metric — if the same metric has been recomputed, prior runs are excluded and remain in the metric history. By default returns only successfully computed metrics; pass `?status=all` to also include rows that resolved as NOT_APPLICABLE / DATA_MISSING / ERROR (the `value` field is omitted on those entries — check `captureStatus`).

# List chats
Source: https://docs.roark.ai/api-reference/chat/list-chats

/api-reference/openapi.documented.json get /v1/chat
Returns a paginated list of chats for the authenticated project.

# Exchange a CLI authorization code for an API key
Source: https://docs.roark.ai/api-reference/cli-auth/exchange-a-cli-authorization-code-for-an-api-key

/api-reference/openapi.documented.json post /v1/cli/auth/token
Completes the CLI browser-login flow: verifies the single-use authorization code and its PKCE code_verifier, then mints and returns the approved API key exactly once.

# Apply a config bundle
Source: https://docs.roark.ai/api-reference/config/apply-a-config-bundle

/api-reference/openapi.documented.json post /v1/config/apply
Reconcile a config-as-code bundle. With a PROJECT API key, submit project-scoped resources (agents, personas, flows, collectors, metrics, ...). With an ORGANIZATION API key, submit `kind: project` resources, each optionally carrying its members and its own nested `resources`. Submit the full desired set; managed resources are updated, new ones created, and (unless prune is false) managed resources absent from the bundle are deleted. Identity is by name.

# Diff a config bundle
Source: https://docs.roark.ai/api-reference/config/diff-a-config-bundle

/api-reference/openapi.documented.json post /v1/config/diff
Dry run for an apply: returns the projected changes (create / update / delete) for the submitted bundle without writing anything. Works for both project and organization keys.

# Add an edge case
Source: https://docs.roark.ai/api-reference/customer-flow-edge-case/add-an-edge-case

/api-reference/openapi.documented.json post /v1/customer-flow/{flowId}/edge-case
Adds a variant to an IMPROV flow.

A scripted flow's variants are owned by the path engine, one per path through the graph, so they are
created by editing the graph through PUT /v1/customer-flow/{flowId}/graph rather than here.

Leave personaOverrideId or environmentId unset to inherit the happy path's.

# Promote an edge case to the happy path
Source: https://docs.roark.ai/api-reference/customer-flow-edge-case/promote-an-edge-case-to-the-happy-path

/api-reference/openapi.documented.json post /v1/customer-flow/{flowId}/edge-case/{edgeCaseId}/promote
Makes this edge case the flow's happy path, and the outgoing happy path an edge case. Its persona and environment are baked into it first, so edge cases that were inheriting keep the configuration they had.

# Remove an edge case
Source: https://docs.roark.ai/api-reference/customer-flow-edge-case/remove-an-edge-case

/api-reference/openapi.documented.json delete /v1/customer-flow/{flowId}/edge-case/{edgeCaseId}
Soft-deletes a variant. On a scripted flow the path engine re-creates a variant for any path still in the graph, so remove the path through PUT /graph instead if that is what you meant.

# Update an edge case
Source: https://docs.roark.ai/api-reference/customer-flow-edge-case/update-an-edge-case

/api-reference/openapi.documented.json put /v1/customer-flow/{flowId}/edge-case/{edgeCaseId}
Updates an edge case's title, persona, environment, brief, preceded-by link or expectations. Omitted fields are left alone; `additionalExpectations` replaces the set wholesale rather than appending. Promoting it to the happy path is a separate call, since that also demotes the incumbent.

# Create a customer flow
Source: https://docs.roark.ai/api-reference/customer-flow/create-a-customer-flow

/api-reference/openapi.documented.json post /v1/customer-flow
Creates a customer flow. A SCRIPTED flow carries a step graph and gets one way of running it per path through the graph; an IMPROV flow carries the briefs you send. Customer flows replace the older simulation scenarios, so build a flow for anything new.

# Delete a customer flow
Source: https://docs.roark.ai/api-reference/customer-flow/delete-a-customer-flow

/api-reference/openapi.documented.json delete /v1/customer-flow/{flowId}
Soft-deletes a customer flow along with its edge cases, expectations and (for scripted flows) its step graph. Run plans that linked it drop it from their test cases.

# Duplicate a customer flow
Source: https://docs.roark.ai/api-reference/customer-flow/duplicate-a-customer-flow

/api-reference/openapi.documented.json post /v1/customer-flow/{flowId}/duplicate
Deep-copies a flow into a new project-owned flow. The copy carries the source's description, branching mode, linked agents, flow-level expectations and flow-owned metrics. A scripted flow copies its whole step graph; an improv flow copies its variants (personas, briefs, expectations). Duplicating a Roark-managed flow is how you customise it. Voicemail flows are Roark-managed and cannot be duplicated.

# Get customer flow by ID
Source: https://docs.roark.ai/api-reference/customer-flow/get-customer-flow-by-id

/api-reference/openapi.documented.json get /v1/customer-flow/{flowId}
Returns a customer flow with its happy path, edge cases, expectations and linked agents. Scripted flows also carry their step graph.

# List customer flows
Source: https://docs.roark.ai/api-reference/customer-flow/list-customer-flows

/api-reference/openapi.documented.json get /v1/customer-flow
Returns a paginated list of customer flows with their agents, expectations, happy path and edge cases. The step graph is the one field omitted: reading it walks the project's whole step graph, so it comes back from the single-flow endpoint instead. Customer flows are how a project describes what to test; they replace the older simulation scenarios.

# Replace a scripted flow's steps
Source: https://docs.roark.ai/api-reference/customer-flow/replace-a-scripted-flows-steps

/api-reference/openapi.documented.json put /v1/customer-flow/{flowId}/graph
Replaces a scripted flow's conversation graph with the tree you send. This is a full replace, not a merge:
a step you omit is removed.

Include `nodeId` on a step to update the existing one, omit it to create a new step. Where two branches
rejoin, keep the `mergeIntoNodeIds` references a read gave you. Dropping them un-merges those branches
and is refused unless `allowUnmerge` is set.

A change to the set of paths re-seeds how the flow runs, which the response reports as
`variantsReshaped` along with the resulting happy path and edge cases.

# Update a customer flow
Source: https://docs.roark.ai/api-reference/customer-flow/update-a-customer-flow

/api-reference/openapi.documented.json put /v1/customer-flow/{flowId}
Updates a flow's title, description, branching mode, linked agents or flow-level expectations. The step graph is replaced through PUT /graph.

# Update a flow's happy path
Source: https://docs.roark.ai/api-reference/customer-flow/update-a-flows-happy-path

/api-reference/openapi.documented.json put /v1/customer-flow/{flowId}/happy-path
Updates the happy path's title, persona, environment, brief or expectations. Omitted fields are left
alone; `additionalExpectations` replaces the set wholesale rather than appending.

Its persona and environment are what the edge cases inherit, so changing them here changes every edge
case that does not name its own.

# Add a widget
Source: https://docs.roark.ai/api-reference/dashboard/add-a-widget

/api-reference/openapi.documented.json post /v1/dashboard/{dashboardId}/widget
Adds a widget to the top of a dashboard. Its name must be unique on the dashboard.

# Create a dashboard
Source: https://docs.roark.ai/api-reference/dashboard/create-a-dashboard

/api-reference/openapi.documented.json post /v1/dashboard
Creates a dashboard with its widgets. Each widget is an analytics query (see POST /v1/analytics/query)
with a name unique on the dashboard, a `display` (LINE, NUMBER, RADAR, STACKED_BAR, TABLE) and a
`size` (SMALL, MEDIUM, LARGE, EXTRA_LARGE). Widgets render top to bottom in the order given.

Dashboard `filters` apply over every widget and must be conversation-level: AGENT_PROPERTY,
CALL_PROPERTY, MODALITY, DATASET or METRIC_VALUE. On a field a widget already filters, the dashboard's wins.

# Delete a dashboard
Source: https://docs.roark.ai/api-reference/dashboard/delete-a-dashboard

/api-reference/openapi.documented.json delete /v1/dashboard/{dashboardId}
Deletes a dashboard and its widgets. If it was the default, another dashboard becomes the default. A dashboard managed by config as code returns 409.

# Get a dashboard
Source: https://docs.roark.ai/api-reference/dashboard/get-a-dashboard

/api-reference/openapi.documented.json get /v1/dashboard/{dashboardId}
Returns a dashboard with its widgets top to bottom, each in the shape POST /v1/dashboard accepts, so a dashboard can be read, edited and written back.

# Get dashboard data
Source: https://docs.roark.ai/api-reference/dashboard/get-dashboard-data

/api-reference/openapi.documented.json get /v1/dashboard/{dashboardId}/data
Runs every widget on the dashboard and returns each one's series, exactly as the dashboard shows them:
over the dashboard's saved window and with its filters applied. Pass `from` (and optionally `to`) to
read a different window, and `timezone` to align buckets to a zone.

# Get widget data
Source: https://docs.roark.ai/api-reference/dashboard/get-widget-data

/api-reference/openapi.documented.json get /v1/dashboard/{dashboardId}/widget/{widgetId}/data
Runs one widget and returns its series, as it reads on its dashboard (the dashboard's window and filters). Accepts the same `from` / `to` / `timezone` overrides as GET /v1/dashboard/{dashboardId}/data.

# List dashboards
Source: https://docs.roark.ai/api-reference/dashboard/list-dashboards

/api-reference/openapi.documented.json get /v1/dashboard
Lists the project's dashboards, the default first. GET /v1/dashboard/{dashboardId} returns the widgets.

# Remove a widget
Source: https://docs.roark.ai/api-reference/dashboard/remove-a-widget

/api-reference/openapi.documented.json delete /v1/dashboard/{dashboardId}/widget/{widgetId}
Removes a widget from a dashboard.

# Replace a widget
Source: https://docs.roark.ai/api-reference/dashboard/replace-a-widget

/api-reference/openapi.documented.json put /v1/dashboard/{dashboardId}/widget/{widgetId}
Replaces a widget's definition. It keeps its id and its place on the dashboard.

# Update a dashboard
Source: https://docs.roark.ai/api-reference/dashboard/update-a-dashboard

/api-reference/openapi.documented.json put /v1/dashboard/{dashboardId}
Edits a dashboard. Omitted fields keep their current value. `widgets`, when sent, replaces the list:
widgets are matched by name, so a widget that keeps its name keeps its id; a new name adds a widget and
a name no longer listed removes it. To change one widget, PUT /v1/dashboard/{dashboardId}/widget/{widgetId}.

# Get API health status
Source: https://docs.roark.ai/api-reference/health/get-api-health-status

/api-reference/openapi.documented.json get /health
Returns the health status of the API and its dependencies

# Create HTTP request definition
Source: https://docs.roark.ai/api-reference/http-request-definition/create-http-request-definition

/api-reference/openapi.documented.json post /v1/http-request-definition
Creates a new HTTP request definition. The signing secret is only returned in this response and cannot be retrieved later.

# Get HTTP request definition by ID
Source: https://docs.roark.ai/api-reference/http-request-definition/get-http-request-definition-by-id

/api-reference/openapi.documented.json get /v1/http-request-definition/{definitionId}
Returns a specific HTTP request definition by its ID.

# List HTTP request definitions
Source: https://docs.roark.ai/api-reference/http-request-definition/list-http-request-definitions

/api-reference/openapi.documented.json get /v1/http-request-definition
Returns a paginated list of HTTP request definitions for the authenticated project.

# Update HTTP request definition
Source: https://docs.roark.ai/api-reference/http-request-definition/update-http-request-definition

/api-reference/openapi.documented.json put /v1/http-request-definition/{definitionId}
Updates an existing HTTP request definition.

# Introduction
Source: https://docs.roark.ai/api-reference/introduction

Integrating with the Roark API

## Welcome to the Roark API

Our API provides programmatic access to all Roark features — from call monitoring and metrics collection to simulation testing.

> [!NOTE]
>
> Haven't generated an API key yet? [Generate one here](/documentation/getting-started/api-keys)

## Core Endpoints

> [card] [Call](/api-reference/call/create-a-call)
>
> Upload and analyze call recordings

> [card] [Metrics](/api-reference/metric/list-metric-definitions)
>
> Define metrics and collect results on calls

> [card] [Simulations](/api-reference/simulation-run-plan/list-run-plans)
>
> Test agents with synthetic callers

## Getting Started

**1. Generate an API Key**

Create your API key in the [dashboard](/documentation/getting-started/api-keys) to start making authenticated requests.

**2. Review the Endpoints**

Explore our available endpoints and their functionalities in the sections above.

**3. Make Your First Request**

Test the API with a simple health check. The [Roark CLI](/documentation/sdks/cli) is the fastest way in:

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark auth login
roark api get /v1/health
```

```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X GET https://api.roark.ai/v1/health \
  -H "Authorization: Bearer YOUR_API_KEY"
```

## SDKs and Libraries

> [card] [Node.js SDK](/documentation/sdks/node-sdk)
>
> Official Node.js SDK for Roark

> [card] [Python SDK](/documentation/sdks/python-sdk)
>
> Python integration library

## Request constraints

Each operation documents its own request constraints and pagination parameters
in this reference. The OpenAPI schema beside a field is the source of truth for
its length, numeric, and array bounds, including whether a list endpoint uses a
cursor or an offset.

## OpenAPI Specification

The full OpenAPI 3.1 document is served unauthenticated, and every limit the API
enforces appears in it as `maxLength`, `maximum` or `maxItems` on the field itself.
It is available at three equivalent URLs:

```
https://api.roark.ai/openapi.json
https://api.roark.ai/.well-known/openapi.json
https://api.roark.ai/doc
```

> [card] [View Full Specification](https://api.roark.ai/openapi.json)
>
> Explore our complete OpenAPI specification for detailed endpoint documentation


# Create issue
Source: https://docs.roark.ai/api-reference/issue/create-issue

/api-reference/openapi.documented.json post /v1/issue
Opens a new issue with `source: API`. The supplied evidence (calls / chats) is attached in the same transaction; an invalid reference fails the whole request rather than leaving a half-created issue.

# Get issue by ID
Source: https://docs.roark.ai/api-reference/issue/get-issue-by-id

/api-reference/openapi.documented.json get /v1/issue/{issueId}
Returns a single issue with its evidence rows.

# List issues
Source: https://docs.roark.ai/api-reference/issue/list-issues

/api-reference/openapi.documented.json get /v1/issue
Returns the project’s issues, ordered newest-first. Supports filtering by status, source, and severity, with offset pagination.

# Create a knowledge base
Source: https://docs.roark.ai/api-reference/knowledge-base/create-a-knowledge-base

/api-reference/openapi.documented.json post /v1/knowledge-base
Creates a knowledge base. TEXT and JSON sources accept inline `content`. FILE sources require `filename`, `mimeType`, and base64-encoded `contentBase64` — the file is decoded server-side, stored in S3, and (for PDFs) text-extracted inline before the response returns.

# Get a knowledge base by ID
Source: https://docs.roark.ai/api-reference/knowledge-base/get-a-knowledge-base-by-id

/api-reference/openapi.documented.json get /v1/knowledge-base/{id}
Returns metadata for a single knowledge base. Does not include content — content is read by metrics at evaluation time and is not exposed over the public API.

# List knowledge bases
Source: https://docs.roark.ai/api-reference/knowledge-base/list-knowledge-bases

/api-reference/openapi.documented.json get /v1/knowledge-base
Returns a cursor-paginated list of knowledge bases for the authenticated project. Soft-deleted knowledge bases are excluded.

# Create a live report
Source: https://docs.roark.ai/api-reference/live-report/create-a-live-report

/api-reference/openapi.documented.json post /v1/live-report
Saves a report on the project's live calls. Simulations are never included.

Targets are analytics measures held to a bar: a threshold metric is read as its pass rate (`value` is a
percentage, 0-100, held AT_LEAST by default); a numeric metric as its average (held AT_MOST by default).
Every target is read for the period and the period before it, so the report says whether each was met
and how it moved.

Scheduled reports are written `deliveryHour` hours into the day after each period ends in `timezone`
(a week runs Monday to Sunday), then emailed and posted per `notifications`. With `generateNow` (the
default) the report for the latest closed period is written right away and not sent.

# Delete a live report
Source: https://docs.roark.ai/api-reference/live-report/delete-a-live-report

/api-reference/openapi.documented.json delete /v1/live-report/{reportId}
Deletes the report and stops its scheduled reports.

# Generate a live report document
Source: https://docs.roark.ai/api-reference/live-report/generate-a-live-report-document

/api-reference/openapi.documented.json post /v1/live-report/{reportId}/generate
Writes the report now, for its latest closed period or a custom window (`periodStart` + `periodEnd`,
at most 92 days, ended). Returns 202 with the document in PENDING; poll
GET /v1/live-report/{reportId}/document/{documentId} until `status` is READY or FAILED (typically a few
minutes). An identical request already in flight is returned as-is. A generated document is not emailed.

# Get a live report
Source: https://docs.roark.ai/api-reference/live-report/get-a-live-report

/api-reference/openapi.documented.json get /v1/live-report/{reportId}

# Get a report document
Source: https://docs.roark.ai/api-reference/live-report/get-a-report-document

/api-reference/openapi.documented.json get /v1/live-report/{reportId}/document/{documentId}
Returns one written report. `document` is null until `status` is READY. Theme excerpts and grader quotes (call transcript text) are included only for keys with transcript:read.

# List live reports
Source: https://docs.roark.ai/api-reference/live-report/list-live-reports

/api-reference/openapi.documented.json get /v1/live-report
Lists the project's saved live call reports, newest first.

# List report documents
Source: https://docs.roark.ai/api-reference/live-report/list-report-documents

/api-reference/openapi.documented.json get /v1/live-report/{reportId}/document
Lists a report's documents, newest period first.

# Update a live report
Source: https://docs.roark.ai/api-reference/live-report/update-a-live-report

/api-reference/openapi.documented.json patch /v1/live-report/{reportId}
Changes the fields sent. Set `status` to PAUSED to stop scheduled reports.

# Create a personal credential
Source: https://docs.roark.ai/api-reference/me/create-a-personal-credential

/api-reference/openapi.documented.json post /v1/me/api-keys
Mints a credential that acts as you, for a CLI or an automation. The key value is returned exactly once, in the `key` field: capture it now, it is unreadable afterwards. Requires a personal credential (a project API key is refused) and admin on the project the credential defaults to. The new credential can never exceed the one that created it: not in scope, not in permissions, and not in lifetime. Omit a field to copy it from the calling credential.

# Describe the current credential
Source: https://docs.roark.ai/api-reference/me/describe-the-current-credential

/api-reference/openapi.documented.json get /v1/me
Returns the scope of the credential making the request, the organization it acts in, its default project (if any) and the permissions it was granted. Works for both project-scoped API keys and user-scoped credentials from the CLI or an MCP connector.

# List your personal credentials
Source: https://docs.roark.ai/api-reference/me/list-your-personal-credentials

/api-reference/openapi.documented.json get /v1/me/api-keys
Returns the credentials that act as you: the ones created in the dashboard, from `roark auth login`, and MCP connectors. Requires a personal credential; a project API key is refused. Defaults to ACTIVE, pass ?status=REVOKED to see revoked ones. The key value itself is never returned.

# Revoke one of your personal credentials
Source: https://docs.roark.ai/api-reference/me/revoke-one-of-your-personal-credentials

/api-reference/openapi.documented.json delete /v1/me/api-keys/{id}
Revokes a credential that acts as you. It stops working immediately. A credential may revoke itself, which is what `roark auth logout` does. Repeating the call is safe; an unknown id, or one belonging to someone else, answers 404.

# Cancel a metric collection job
Source: https://docs.roark.ai/api-reference/metric-collection-job/cancel-a-metric-collection-job

/api-reference/openapi.documented.json post /v1/metric/collection-jobs/{jobId}/cancel
Stops a metric collection job that has not finished. No further conversations are processed and the job ends in CANCELED.

Items the job had not reached are marked SKIPPED. Metric values already produced are kept, and items that were mid-computation when the cancel arrived may still complete. Only the items that completed are billed.

Safe to call more than once: a job that has already finished (COMPLETED, FAILED, or CANCELED) is returned unchanged.

# Create and run a metric collection job
Source: https://docs.roark.ai/api-reference/metric-collection-job/create-and-run-a-metric-collection-job

/api-reference/openapi.documented.json post /v1/metric/collection-jobs
Creates a metric collection job for the specified calls or chats and metrics, then triggers processing. Provide exactly one of callIds or chatIds. Pass policyId to score with a metric policy’s settings (for example to re-score past calls after changing which properties Property Verification checks); metrics then defaults to every metric that policy collects.

# Get metric collection job by ID
Source: https://docs.roark.ai/api-reference/metric-collection-job/get-metric-collection-job-by-id

/api-reference/openapi.documented.json get /v1/metric/collection-jobs/{jobId}
Returns a specific metric collection job with progress information.

# Get metric values produced by a metric collection job
Source: https://docs.roark.ai/api-reference/metric-collection-job/get-metric-values-produced-by-a-metric-collection-job

/api-reference/openapi.documented.json get /v1/metric/collection-jobs/{jobId}/metric-values
Returns the metric values produced by the specified job, grouped by metric definition. Unlike `GET /v1/call/:callId/metrics` (which returns the latest values on a call, regardless of which job computed them), this endpoint returns exactly the values *this* job produced — including for calls whose live values have since been overwritten by a later job. By default returns only SUCCESS rows; pass `?status=all` to also include NOT_APPLICABLE / DATA_MISSING / ERROR.

# List metric collection jobs
Source: https://docs.roark.ai/api-reference/metric-collection-job/list-metric-collection-jobs

/api-reference/openapi.documented.json get /v1/metric/collection-jobs
Returns a paginated list of metric collection jobs for the project.

# Retry a metric collection job
Source: https://docs.roark.ai/api-reference/metric-collection-job/retry-a-metric-collection-job

/api-reference/openapi.documented.json post /v1/metric/collection-jobs/{jobId}/retry
Creates a new metric collection job using the same conversations and metrics as a previous job, then triggers processing. The previous job must be in a terminal state (COMPLETED, FAILED, or CANCELED). Returns the newly created job — track its id for downstream fetches.

# Create a metric policy
Source: https://docs.roark.ai/api-reference/metric-policy/create-a-metric-policy

/api-reference/openapi.documented.json post /v1/metric/policies
Creates a new metric policy. Policies define which metrics to collect and under what conditions.

# Delete a metric policy
Source: https://docs.roark.ai/api-reference/metric-policy/delete-a-metric-policy

/api-reference/openapi.documented.json delete /v1/metric/policies/{policyId}
Soft-deletes a metric policy. System policies cannot be deleted.

# Get metric policy by ID
Source: https://docs.roark.ai/api-reference/metric-policy/get-metric-policy-by-id

/api-reference/openapi.documented.json get /v1/metric/policies/{policyId}
Returns a specific metric policy with its conditions and metrics.

# List metric policies
Source: https://docs.roark.ai/api-reference/metric-policy/list-metric-policies

/api-reference/openapi.documented.json get /v1/metric/policies
Returns a paginated list of metric policies for the project, including system policies.

# Update a metric policy
Source: https://docs.roark.ai/api-reference/metric-policy/update-a-metric-policy

/api-reference/openapi.documented.json put /v1/metric/policies/{policyId}
Updates an existing metric policy. System policies cannot be modified.

# Create a metric variant
Source: https://docs.roark.ai/api-reference/metric/create-a-metric-variant

/api-reference/openapi.documented.json post /v1/metric/definitions/{idOrSlug}/variants
Add a configuration of this metric for your organization, seeded from its Default. Edit it with
PUT to change what it measures, then pin it where you want it used.

Threshold metrics have no variants: their configuration comes from the metric they derive from.
Metrics in a package that manages its own variants reject this too.

# Create a threshold on a metric
Source: https://docs.roark.ai/api-reference/metric/create-a-threshold-on-a-metric

/api-reference/openapi.documented.json post /v1/metric/definitions/{idOrSlug}/thresholds
Create a boolean threshold derived from an existing metric. The source metric is addressed by its UUID or its stable `slug`. The threshold fires when the source metric meets the comparison condition. Scope and supported contexts are inherited from the source metric.

# Create custom metric definition
Source: https://docs.roark.ai/api-reference/metric/create-custom-metric-definition

/api-reference/openapi.documented.json post /v1/metric/definitions
Create a new metric definition. The `calculationType` field selects the variant: LLM_JUDGE (LLM-evaluated), FORMULA (computed from a math expression over other metrics), or PATTERN (detects a trigger→outcome pattern within a window). To create a threshold on top of an existing metric, use `POST /metric/definitions/{idOrSlug}/thresholds` instead.

# Delete a metric definition
Source: https://docs.roark.ai/api-reference/metric/delete-a-metric-definition

/api-reference/openapi.documented.json delete /v1/metric/definitions/{idOrSlug}
Archives (soft-deletes) a custom metric definition, addressed by its UUID or its stable `slug`. The metric is hidden from all reads and stops being collected, but its record and previously collected values are retained. System metrics cannot be deleted. Fails with 409 if the metric is still used as a source by a derived metric (delete those first).

# Delete a metric variant
Source: https://docs.roark.ai/api-reference/metric/delete-a-metric-variant

/api-reference/openapi.documented.json delete /v1/metric/definitions/{idOrSlug}/variants/{variantId}
Remove one of your organization’s variants. Anything pinned to it falls back to the Default, so
deleting a fork of a Roark variant returns you to Roark’s configuration.

Roark’s own variants cannot be deleted, and neither can a Default. Values already collected under
the deleted variant are retained.

# Get a metric definition
Source: https://docs.roark.ai/api-reference/metric/get-a-metric-definition

/api-reference/openapi.documented.json get /v1/metric/definitions/{idOrSlug}
Fetch a single metric definition by its UUID or its stable `slug` (e.g. `customer_satisfaction`). Resolution is scoped to the project — a project-owned metric wins over an org-wide one, which wins over a system metric of the same `slug`.

# Get a metric variant
Source: https://docs.roark.ai/api-reference/metric/get-a-metric-variant

/api-reference/openapi.documented.json get /v1/metric/definitions/{idOrSlug}/variants/{variantId}
One configuration of this metric, by id.

# Http request metric
Source: https://docs.roark.ai/api-reference/metric/http-request-metric

/api-reference/openapi.documented.json webhook HttpRequestMetric
What an HTTP request metric sends to your endpoint, and the answer it expects. Roark sends it once per call the metric grades, live or simulated, after the call is analysed.

With no body template, Roark POSTs the payload below. A body template fills `{{path}}` placeholders from the same payload, for example `{{call.properties.cardId}}` or `{{simulation.variables.newPin}}`. A placeholder the call has no value for (a run variable on a live call) skips the request and records the metric as missing data.

Answer with HTTP 200 and `{"passed": true | false, "reason": "..."}` within 30 seconds. Timeouts, connection failures and 5xx responses are retried twice with the same `requestId`, so drop a request you have already answered. Any other answer records an error on the metric, never a fail. Redirects are not followed.

# List a metric’s variants
Source: https://docs.roark.ai/api-reference/metric/list-a-metric’s-variants

/api-reference/openapi.documented.json get /v1/metric/definitions/{idOrSlug}/variants
Every configuration of this metric your organization can use: Roark’s own variants and any your
organization has added or forked. `isDefault` marks the one the metric is scored with when nothing
pins another; pass any variant’s `id` as `sourceVariantId` to pin it on a derived metric.

Auto-managed variants (the ones a package materializes for you) are not listed: they are engine state,
not configuration you author.

# List metric definitions
Source: https://docs.roark.ai/api-reference/metric/list-metric-definitions

/api-reference/openapi.documented.json get /v1/metric/definitions
Fetch metric definitions available in the project, including both system-generated and custom metrics. Results are ordered by immutable definition ID; pass `nextCursor` to retrieve the following page.

# Update a metric definition
Source: https://docs.roark.ai/api-reference/metric/update-a-metric-definition

/api-reference/openapi.documented.json put /v1/metric/definitions/{idOrSlug}
Update the editable subset of a custom metric definition, addressed by its UUID or its stable `slug`. Only the supplied fields are changed; omitted fields are left unchanged. Every update creates a new immutable version; the response carries the advanced `versionId`. Immutable fields (scope, outputType, calcType, …) are rejected, and which fields are editable depends on the metric (e.g. derived metrics only allow `name`).  Roark's own metrics are rejected here: this endpoint edits the shared definition, which every workspace sees. To change one for your workspace alone, edit its variant with PUT /v1/metric/definitions/{idOrSlug}/variants/{variantId}, which forks it for you and leaves every other workspace on the original.

# Update a metric variant
Source: https://docs.roark.ai/api-reference/metric/update-a-metric-variant

/api-reference/openapi.documented.json put /v1/metric/definitions/{idOrSlug}/variants/{variantId}
Rename a variant, change its configuration, or both. Every configuration change creates a new
immutable version and advances `versionId`; the response carries the advanced value.

**Editing one of Roark’s own variants forks it for your organization.** The response carries the
new variant’s `id`, which will differ from the one in the path, and `isSystem` becomes false. Roark’s
variant is untouched and other organizations keep it. DELETE your fork to go back to it.

Which fields are editable depends on the metric: some Roark metrics lock their prompt or output
configuration, and a locked field is rejected rather than ignored.

# Generate a project API key
Source: https://docs.roark.ai/api-reference/organization-project-api-key/generate-a-project-api-key

/api-reference/openapi.documented.json post /v1/org/project/{projectId}/api-key
Mint a new project-scoped API key. The raw key value is returned exactly once in the `key` field: capture it now, it is unreadable afterwards. Defaults to full project access with READ + WRITE scope; narrow with `permissions` / `scopes`.

# List project API keys
Source: https://docs.roark.ai/api-reference/organization-project-api-key/list-project-api-keys

/api-reference/openapi.documented.json get /v1/org/project/{projectId}/api-key
List API keys for a project (key values are never returned after creation).

# Revoke a project API key
Source: https://docs.roark.ai/api-reference/organization-project-api-key/revoke-a-project-api-key

/api-reference/openapi.documented.json delete /v1/org/project/{projectId}/api-key/{keyId}
Revoke (deactivate) a project API key. Idempotent from the caller perspective.

# Invite a member
Source: https://docs.roark.ai/api-reference/organization-project-member/invite-a-member

/api-reference/openapi.documented.json post /v1/org/project/{projectId}/member
Invite a user to the project by email with a role. If the user has no Roark account yet, an invite email is sent and the membership is INVITED until they accept; existing users are added directly. Use this for onboarding and for assigning users to projects.

# List project members
Source: https://docs.roark.ai/api-reference/organization-project-member/list-project-members

/api-reference/openapi.documented.json get /v1/org/project/{projectId}/member
List active and pending-invite members of a project.

# Remove a member
Source: https://docs.roark.ai/api-reference/organization-project-member/remove-a-member

/api-reference/openapi.documented.json delete /v1/org/project/{projectId}/member/{memberId}
Remove a member from the project. The last owner/admin cannot be removed (409).

# Update a member role
Source: https://docs.roark.ai/api-reference/organization-project-member/update-a-member-role

/api-reference/openapi.documented.json put /v1/org/project/{projectId}/member/{memberId}
Change a project member role.

# Create a project
Source: https://docs.roark.ai/api-reference/organization-project/create-a-project

/api-reference/openapi.documented.json post /v1/org/project
Create a new project in the organization. The project is provisioned with the default analysis packages, dashboard, and alerts, exactly as a UI-created project would be. The user who minted the API key becomes the initial project owner.

# Delete a project
Source: https://docs.roark.ai/api-reference/organization-project/delete-a-project

/api-reference/openapi.documented.json delete /v1/org/project/{projectId}
Soft-delete a project and pause its active schedules. Returns 409 when the project is the last one in the organization, has an active agent connection, or has a simulation running.

# Get a project
Source: https://docs.roark.ai/api-reference/organization-project/get-a-project

/api-reference/openapi.documented.json get /v1/org/project/{projectId}
Get a single project by id (must belong to the organization).

# List projects
Source: https://docs.roark.ai/api-reference/organization-project/list-projects

/api-reference/openapi.documented.json get /v1/org/project
List projects in the organization the API key belongs to. Paginated (max 101 per page): pass `page`/`limit` and keep paging until a page returns fewer than `limit` items.

# Update a project
Source: https://docs.roark.ai/api-reference/organization-project/update-a-project

/api-reference/openapi.documented.json put /v1/org/project/{projectId}
Update a project name, description, or category.

# Buy a phone number
Source: https://docs.roark.ai/api-reference/phone-number/buy-a-phone-number

/api-reference/openapi.documented.json post /v1/phone-number
Buys a number from the carrier for this project: the one named by e164, or one Roark picks to match autoPick (trying the next match if the carrier has just sold one). Free inside the plan allowance; past it, the first month of rent is charged now and monthly after that.

# Get the phone number allowance
Source: https://docs.roark.ai/api-reference/phone-number/get-the-phone-number-allowance

/api-reference/openapi.documented.json get /v1/phone-number/allowance
Returns how many numbers the organization gets for free, how many it holds, what a paid number costs a month, and whether a purchase would be admitted right now.

# List phone numbers
Source: https://docs.roark.ai/api-reference/phone-number/list-phone-numbers

/api-reference/openapi.documented.json get /v1/phone-number
Returns the phone numbers the project owns, oldest first, with the agent each is assigned to.

# Release a phone number
Source: https://docs.roark.ai/api-reference/phone-number/release-a-phone-number

/api-reference/openapi.documented.json delete /v1/phone-number/{phoneNumberId}
Gives the number back to the carrier. This cannot be undone: rent stops today and nothing is refunded.

# Search available phone numbers
Source: https://docs.roark.ai/api-reference/phone-number/search-available-phone-numbers

/api-reference/openapi.documented.json get /v1/phone-number/available
Returns US or Canadian numbers the carrier has for sale, local or toll-free, narrowed by area code or by digits the number must contain. When an area code has no numbers, nearby ones are returned flagged bestEffort; an empty list means nothing matched. Buy one with POST /v1/phone-number, or let Roark pick with autoPick.

# Update a phone number
Source: https://docs.roark.ai/api-reference/phone-number/update-a-phone-number

/api-reference/openapi.documented.json put /v1/phone-number/{phoneNumberId}
Relabels a number or changes which agent it is assigned to. A field left out is untouched; null clears it.

# List projects available to this credential
Source: https://docs.roark.ai/api-reference/project/list-projects-available-to-this-credential

/api-reference/openapi.documented.json get /v1/projects
Returns every project the calling credential may act on, with the permissions it holds in each. A project-scoped API key returns the single project it is bound to. A user-scoped credential returns the projects in its organization where the holder is an active member. Use the returned id in the X-Roark-Project-Id header.

# Mint a live observer WebSocket URL for a prospect call
Source: https://docs.roark.ai/api-reference/prospect-simulation/mint-a-live-observer-websocket-url-for-a-prospect-call

/api-reference/openapi.documented.json get /v1/public/prospect-sim/report/{token}/observer-url
Public endpoint scoped by the report token. Returns a short-lived, self-authenticating WebSocket URL for watching a prospect call live, but only while that call is actually in flight; otherwise wsUrl is null. Pass `callId` (from the live progress list) to pick one of the parallel calls; without it the most recently started live call is used.

# Read a prospect simulation teaser report
Source: https://docs.roark.ai/api-reference/prospect-simulation/read-a-prospect-simulation-teaser-report

/api-reference/openapi.documented.json get /v1/public/prospect-sim/report/{token}
Public endpoint scoped by the unguessable report token. Returns a truncated teaser once the debrief is ready; the full report is gated behind claiming an account.

# Run a free simulation against a prospect agent
Source: https://docs.roark.ai/api-reference/prospect-simulation/run-a-free-simulation-against-a-prospect-agent

/api-reference/openapi.documented.json post /v1/public/prospect-sim/run
Public endpoint: provisions a per-prospect project, runs a canned scenario template against the submitted phone number, and returns a report token to poll. The teaser report is also emailed when ready.

# Create an environment
Source: https://docs.roark.ai/api-reference/simulation-environment/create-an-environment

/api-reference/openapi.documented.json post /v1/simulation/environment
Creates an environment for the project: a noise bed and the level it plays at. Reference it by id when setting a customer flow variant's environment. Roark's curated presets always play at the default level, so this is how a project gets the same bed louder or quieter.

# Delete an environment
Source: https://docs.roark.ai/api-reference/simulation-environment/delete-an-environment

/api-reference/openapi.documented.json delete /v1/simulation/environment/{environmentId}
Soft-deletes one of the project's environments. It disappears from reads and cannot be picked for new runs; runs already built keep their snapshot. Refused (409) while a live customer flow variant still uses it: move those variants first. Roark-curated environments cannot be deleted (403).

# Get environment by ID
Source: https://docs.roark.ai/api-reference/simulation-environment/get-environment-by-id

/api-reference/openapi.documented.json get /v1/simulation/environment/{environmentId}
Returns a single environment by its ID.

# List environments
Source: https://docs.roark.ai/api-reference/simulation-environment/list-environments

/api-reference/openapi.documented.json get /v1/simulation/environment
Returns a paginated list of environments: the project's own plus the environments Roark curates and shares across every project. Reference one by id when setting a customer flow variant's environment.

# Update an environment
Source: https://docs.roark.ai/api-reference/simulation-environment/update-an-environment

/api-reference/openapi.documented.json put /v1/simulation/environment/{environmentId}
Updates one of the project's environments. Only the fields sent are changed. Runs already built keep the snapshot they were built with. Roark-curated environments cannot be edited (403).

# Get simulation by ID
Source: https://docs.roark.ai/api-reference/simulation-job/get-simulation-by-id

/api-reference/openapi.documented.json get /v1/simulation/job/{jobId}
Get a individual simulation run directly by its ID. This is generally part of a larger simulation run plan job.

# List mocked tool invocations
Source: https://docs.roark.ai/api-reference/simulation-job/list-mocked-tool-invocations

/api-reference/openapi.documented.json get /v1/simulation/job/{jobId}/tool-mock
Every guarded tool invocation Roark answered during this simulation: what the agent tried to call, with what arguments, and the simulated response it received. Use this to assert tool behavior in CI after a test call, with zero real side effects. Empty when the agent's tools are not guarded (see the tool guard docs).

# Lookup by phone number
Source: https://docs.roark.ai/api-reference/simulation-job/lookup-by-phone-number

/api-reference/openapi.documented.json get /v1/simulation/job/lookup
Find the matching simulation using the number used by the Roark simulation agent.

# Create a new persona
Source: https://docs.roark.ai/api-reference/simulation-persona/create-a-new-persona

/api-reference/openapi.documented.json post /v1/persona
Creates a new persona for the authenticated project.

# Delete a persona
Source: https://docs.roark.ai/api-reference/simulation-persona/delete-a-persona

/api-reference/openapi.documented.json delete /v1/persona/{personaId}
Soft-deletes a persona by its ID. The persona is hidden from all reads and stops being available to new simulations, but its record and history are retained. System personas cannot be deleted.

# Get persona by ID
Source: https://docs.roark.ai/api-reference/simulation-persona/get-persona-by-id

/api-reference/openapi.documented.json get /v1/persona/{personaId}
Returns a specific persona by its ID.

# List personas
Source: https://docs.roark.ai/api-reference/simulation-persona/list-personas

/api-reference/openapi.documented.json get /v1/persona
Returns a paginated list of personas for the authenticated project.

# Update a persona
Source: https://docs.roark.ai/api-reference/simulation-persona/update-a-persona

/api-reference/openapi.documented.json put /v1/persona/{personaId}
Updates an existing persona by its ID.

# Cancel a simulation plan job
Source: https://docs.roark.ai/api-reference/simulation-run-plan-job/cancel-a-simulation-plan-job

/api-reference/openapi.documented.json post /v1/simulation/plan/job/{jobId}/cancel
Stops a run that has not finished yet. Already-finished runs are left alone.

Intended for CI: when a pipeline is cancelled or superseded, cancelling the run stops it
placing calls you no longer need. Safe to call more than once.

# Generate simulation run debrief
Source: https://docs.roark.ai/api-reference/simulation-run-plan-job/generate-simulation-run-debrief

/api-reference/openapi.documented.json post /v1/simulation/plan/job/{jobId}/debrief
Request the written debrief for a settled run: executive summary, failure themes and recommendations.
Runs with graded calls get one automatically, so this is for regenerating one (`regenerate: true`)
or retrying one that failed.

Returns immediately. Generation takes a few minutes: poll GET
/v1/simulation/plan/job/{jobId}/results until `debrief.status` is READY or FAILED.
Without `regenerate`, a debrief that is ready or already being written is returned as-is.

# Get simulation plan job
Source: https://docs.roark.ai/api-reference/simulation-run-plan-job/get-simulation-plan-job

/api-reference/openapi.documented.json get /v1/simulation/plan/job/{jobId}
Retrieve details of a simulation plan job including all associated simulation jobs (calls)

# Get simulation run cost
Source: https://docs.roark.ai/api-reference/simulation-run-plan-job/get-simulation-run-cost

/api-reference/openapi.documented.json get /v1/simulation/plan/job/{jobId}/cost
The billable spend for one simulation run (run plan job), split by cost type, with the spend of each simulated call in it. Safe to call while the run is in progress: totals reflect the costs recorded so far. Amounts are billable spend only, in exact micro-dollars (`*UsdMicros`) and dollars (`*Usd`, for display). Requires an API key with the `billing:read` permission.

# Get simulation run results
Source: https://docs.roark.ai/api-reference/simulation-run-plan-job/get-simulation-run-results

/api-reference/openapi.documented.json get /v1/simulation/plan/job/{jobId}/results
Everything the run report shows, in one request: the summary and score, the pass/fail verdict,
per-metric aggregates, every simulation with its check results, breakdowns by agent and persona,
the comparison against the previous run (or `compareToJobId`), the sweep attribution, and the
written debrief with its executive summary, failure themes and recommendations.

Safe to call while the run is in progress: counts reflect what has landed so far. `verdict` and
`sweepAttribution` stay null until the run ends. The debrief is generated automatically a few
minutes after a run with graded calls settles; poll until `debrief.status` is READY, or request
one with POST /v1/simulation/plan/job/{jobId}/debrief.

# List simulation plan jobs
Source: https://docs.roark.ai/api-reference/simulation-run-plan-job/list-simulation-plan-jobs

/api-reference/openapi.documented.json get /v1/simulation/plan/jobs
Returns a paginated list of simulation run plan jobs. Filter by status, plan ID, or label to find specific simulation batches.

# Run a simulation plan
Source: https://docs.roark.ai/api-reference/simulation-run-plan-job/run-a-simulation-plan

/api-reference/openapi.documented.json post /v1/simulation/plan/{planId}/job
Deprecated: use POST /v1/simulation/run, which does the same thing and can also take the
plan configuration inline, so a one-off run does not have to create a plan first.

Creates and executes a job for an existing simulation run plan. Optionally provide runtime
variables to override plan-defined variables.

# Create a run plan
Source: https://docs.roark.ai/api-reference/simulation-run-plan/create-a-run-plan

/api-reference/openapi.documented.json post /v1/simulation/plan
Creates a new simulation run plan.

To run a simulation, use POST /v1/simulation/run instead: it starts a run from a plan or
from an inline configuration, and takes runtime variables. Create a plan here when you want
a reusable, named one to run later.

Send `template` instead of a full configuration to save one of the built-in templates as a
plan. It takes the same fields as the template variant of POST /v1/simulation/run, builds the
same plan, and never starts it.

To compare one property, attach the flow once and send `comparisonProperty` with the
`comparisonValues` to run: the plan attaches the flow once per value.

# Delete a run plan
Source: https://docs.roark.ai/api-reference/simulation-run-plan/delete-a-run-plan

/api-reference/openapi.documented.json delete /v1/simulation/plan/{planId}
Soft-deletes a simulation run plan by its ID.

# Get run plan by ID
Source: https://docs.roark.ai/api-reference/simulation-run-plan/get-run-plan-by-id

/api-reference/openapi.documented.json get /v1/simulation/plan/{planId}
Returns a specific simulation run plan by its ID.

# List run plans
Source: https://docs.roark.ai/api-reference/simulation-run-plan/list-run-plans

/api-reference/openapi.documented.json get /v1/simulation/plan
Returns a paginated list of simulation run plans. Optionally filter by search text or agent ID.

# Update a run plan
Source: https://docs.roark.ai/api-reference/simulation-run-plan/update-a-run-plan

/api-reference/openapi.documented.json put /v1/simulation/plan/{planId}
Updates an existing simulation run plan by its ID.

# List simulation templates
Source: https://docs.roark.ai/api-reference/simulation-template/list-simulation-templates

/api-reference/openapi.documented.json get /v1/simulation/template
Returns the built-in simulation templates, each resolved against this project: the metric and
check definitions it collects, and the flows it runs with the variants it covers.

A template is a preset rather than a stored object, so building a run from one produces an
ordinary run plan you own and can edit afterwards. Pass a `slug` as `template` to
POST /v1/simulation/run.

Every entry is complete, so there is no per-template endpoint to follow up with, and the list
is a fixed catalogue rather than a paginated one. Only templates a request can actually run
are listed.

# Create a run group
Source: https://docs.roark.ai/api-reference/simulation/create-a-run-group

/api-reference/openapi.documented.json post /v1/simulation/run-group
Groups several simulation runs so they read as one, and starts generating the group report:
one document across every run, with a scorecard summed across them and the issues found,
each merged from the per-run debriefs.

Send `simulationRunPlanJobIds` to group runs that already exist (runs still in progress are
fine: the report waits for them). Send `simulationRunPlanIds` to start one run of each plan
and group those; this places real calls, exactly as starting each plan would.

Poll GET /v1/simulation/run-group/{groupId} until `latestReport.status` is READY.

# Create a simulation group
Source: https://docs.roark.ai/api-reference/simulation/create-a-simulation-group

/api-reference/openapi.documented.json post /v1/simulation/group
Saves a named set of plans to run together, again and again. Nothing runs yet:
POST /v1/simulation/group/{groupId}/run starts one run of every plan and groups them
into a run group with one combined report.

Slack channels need Slack connected to the project. Email is on by default.

# Delete a run group
Source: https://docs.roark.ai/api-reference/simulation/delete-a-run-group

/api-reference/openapi.documented.json delete /v1/simulation/run-group/{groupId}
Deletes a run group. Its runs, and their results, are untouched.

# Delete a simulation group
Source: https://docs.roark.ai/api-reference/simulation/delete-a-simulation-group

/api-reference/openapi.documented.json delete /v1/simulation/group/{groupId}
Deletes a saved simulation group and any schedules that run it. Run groups it already started, and their reports, stay. A group managed by config as code returns 409.

# Delete a tool fixture
Source: https://docs.roark.ai/api-reference/simulation/delete-a-tool-fixture

/api-reference/openapi.documented.json delete /v1/simulation/tool-fixture/{fixtureId}
The tool goes back to scenario-aware generated responses in test calls.

# Get a run group
Source: https://docs.roark.ai/api-reference/simulation/get-a-run-group

/api-reference/openapi.documented.json get /v1/simulation/run-group/{groupId}
Returns a run group with its runs in group order and its latest group report, including the report document once READY.

# Get a simulation group
Source: https://docs.roark.ai/api-reference/simulation/get-a-simulation-group

/api-reference/openapi.documented.json get /v1/simulation/group/{groupId}
Returns a saved simulation group with its plans in run order and notification settings.

# List run groups
Source: https://docs.roark.ai/api-reference/simulation/list-run-groups

/api-reference/openapi.documented.json get /v1/simulation/run-group
Returns the run groups in the project, newest first, with their runs and where each group report stands. Read a report document from GET /v1/simulation/run-group/{groupId}.

# List simulation groups
Source: https://docs.roark.ai/api-reference/simulation/list-simulation-groups

/api-reference/openapi.documented.json get /v1/simulation/group
Returns the saved simulation groups in the project, most recently edited first, each with its plans in run order and notification settings.

# List tool fixtures
Source: https://docs.roark.ai/api-reference/simulation/list-tool-fixtures

/api-reference/openapi.documented.json get /v1/simulation/tool-fixture
All deterministic tool responses configured for this project, optionally filtered by tool.

# Mock a tool call during a test call
Source: https://docs.roark.ai/api-reference/simulation/mock-a-tool-call-during-a-test-call

/api-reference/openapi.documented.json post /v1/simulation/tool-mock
The server half of the tool guard for code-first agents. When a guarded tool fires during a Roark test call, send the invocation here instead of executing it: Roark answers with a simulated backend response that is valid JSON, shaped by the tool contract you pass, consistent with the test scenario, and consistent with earlier mocked responses in the same call. Real callers are never affected: the guard only diverts when the agent-config resolve response identified the session as a Roark simulation, and this endpoint independently re-validates the simulation before answering.

Failure contract for your wrapper: `404` means the simulation id is unknown to this project (treat the session as real). `409` means the simulation has already ended (stale session state: do NOT execute the real tool; return your static fallback). `5xx` means generation failed (return your static fallback).

Identical retries (same tool, same arguments) within a few minutes return the stored response, so double-fired handlers stay consistent.

# Regenerate a run group report
Source: https://docs.roark.ai/api-reference/simulation/regenerate-a-run-group-report

/api-reference/openapi.documented.json post /v1/simulation/run-group/{groupId}/report
Generates the group report again over the latest state of its runs. Returns the report generation; if one is already in flight it is returned instead of starting another. Poll GET /v1/simulation/run-group/{groupId} until `latestReport.status` is READY.

# Run a simulation
Source: https://docs.roark.ai/api-reference/simulation/run-a-simulation

/api-reference/openapi.documented.json post /v1/simulation/run
Starts a simulation and returns the run.

Send `template` to run one of the built-in templates: it supplies the metrics and checks,
and for some templates the flows too, so the request only names the agent and the direction.
Send `plan` to describe a simulation yourself and run it once. Send `planId` to run a plan
you already have.

`template` and `plan` both resolve to a run plan, returned as `simulationRunPlanId`. Add
`saveAsPlan` to keep it, or read it back to see exactly what ran. A plan built from a
template is a snapshot: retuning the template later never changes what that plan runs, which
is what makes a saved one safe to pin in CI.

# Run a simulation group
Source: https://docs.roark.ai/api-reference/simulation/run-a-simulation-group

/api-reference/openapi.documented.json post /v1/simulation/group/{groupId}/run
Starts one run of every plan in the group, exactly as POST /v1/simulation/plan/{planId}/job
would for each, and groups the runs into a run group whose combined report is announced
through the group's notification settings once it is ready. This places real calls.
Config-managed groups can be run here too.

Returns the run group. Poll GET /v1/simulation/run-group/{groupId} until
`latestReport.status` is READY.

If a plan cannot start after others did, the runs that started are still grouped and the
400 names the run group id.

# Set a tool fixture
Source: https://docs.roark.ai/api-reference/simulation/set-a-tool-fixture

/api-reference/openapi.documented.json post /v1/simulation/tool-fixture
Create-or-replace by scope: one fixture per tool project-wide, plus one per (tool, flow variant). Setting the same scope twice replaces the response, so CI can apply fixtures idempotently. During Roark test calls the tool guard answers with the fixture verbatim instead of generating a response; real callers are never affected.

# Update a simulation group
Source: https://docs.roark.ai/api-reference/simulation/update-a-simulation-group

/api-reference/openapi.documented.json put /v1/simulation/group/{groupId}
Edits a saved simulation group. Omitted fields keep their current value; `notifications`,
when sent, replaces the settings as a whole. Run groups it already started keep theirs.

A group managed by config as code (`configManaged: true`) returns 409: change it in the config file.

# Update a tool fixture
Source: https://docs.roark.ai/api-reference/simulation/update-a-tool-fixture

/api-reference/openapi.documented.json put /v1/simulation/tool-fixture/{fixtureId}
Change the response, description, or enabled state of an existing fixture.

# Create a tool
Source: https://docs.roark.ai/api-reference/tool/create-a-tool

/api-reference/openapi.documented.json post /v1/tool
Creates a tool and declares its first version. A tool only recorded from calls under the same name is adopted; one already declared returns 409. Give it a `runtime` for a hosted agent to call your endpoint. Header values are returned on read, so keep secrets out of them.

# Delete a tool
Source: https://docs.roark.ai/api-reference/tool/delete-a-tool

/api-reference/openapi.documented.json delete /v1/tool/{toolId}
Deletes a tool no agent version uses. A tool an agent version uses returns 409.

# Get a tool
Source: https://docs.roark.ai/api-reference/tool/get-a-tool

/api-reference/openapi.documented.json get /v1/tool/{toolId}

# List tools
Source: https://docs.roark.ai/api-reference/tool/list-tools

/api-reference/openapi.documented.json get /v1/tool
The project's tools and the built-in ones, each with its latest version: inputs, description and, for a tool a hosted agent runs, its request. Attach one to an agent draft by id.

# Update a tool
Source: https://docs.roark.ai/api-reference/tool/update-a-tool

/api-reference/openapi.documented.json put /v1/tool/{toolId}
Replaces the tool: its name, scope, inputs, description, grading hints and runtime. A field left out is cleared, so omitting agentId makes the tool project-wide. A change declares a new version; agent versions pinned to an older one keep it.

# Get usage & cost
Source: https://docs.roark.ai/api-reference/usage/get-usage-&-cost

/api-reference/openapi.documented.json get /v1/usage
Report billable usage and cost for the organization over a time window, broken down by project (highest spend first). Pass `from`/`to` as ISO 8601 timestamps; optionally pass `projectId` to scope the report to one project. Amounts are billable spend only (what we absorb is never included). Authenticated by either an organization API key with the `usage:read` permission (any project in the organization), or a project API key with the `billing:read` permission (that key's own project only; any other `projectId` is a 403).

# Create webhook
Source: https://docs.roark.ai/api-reference/webhook/create-webhook

/api-reference/openapi.documented.json post /v1/webhook
Creates a new webhook with event subscriptions. The signing secret is only returned in this response.

# Delete webhook
Source: https://docs.roark.ai/api-reference/webhook/delete-webhook

/api-reference/openapi.documented.json delete /v1/webhook/{webhookId}
Deletes a webhook and all its event subscriptions.

# Event
Source: https://docs.roark.ai/api-reference/webhook/event

/api-reference/openapi.documented.json webhook Webhook
Roark POSTs a JSON payload to every endpoint subscribed to one of the events listed below. Every payload uses the same envelope (`event`, `version`, `timestamp`, `data`); the `event` field is the discriminator that selects the matching `data` shape.

Acknowledge with any 2xx response within 10 seconds. Non-2xx responses and timeouts are retried with exponential backoff.

**Events**
- `call.analysis.completed`
- `call.analysis.failed`
- `call.analysis.cancelled`
- `call.evaluation.completed`
- `call.evaluation.failed`
- `simulation.run_plan_job.started`
- `simulation.run_plan_job.completed`
- `simulation.run_plan_job.failed`
- `simulation.run_plan_job.cancelled`
- `simulation.job.started`
- `simulation.job.completed`
- `simulation.job.failed`
- `simulation.job.cancelled`
- `metric_collection.job.completed`
- `metric_collection.job.failed`
- `chat.analysis.completed`
- `chat.analysis.failed`
- `issue.opened`
- `issue.resolved`

# Get webhook by ID
Source: https://docs.roark.ai/api-reference/webhook/get-webhook-by-id

/api-reference/openapi.documented.json get /v1/webhook/{webhookId}
Returns a specific webhook with its event subscriptions.

# List webhooks
Source: https://docs.roark.ai/api-reference/webhook/list-webhooks

/api-reference/openapi.documented.json get /v1/webhook
Returns a paginated list of webhooks with their event subscriptions.

# roark agent build
Source: https://docs.roark.ai/cli-reference/commands/agent-build

Build a Roark-hosted agent

Creates a new Roark-hosted agent from a one-line job description. Roark authors the system prompt and hosts the config, so the agent is live and self-improvable from creation.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent build [options]
```

> [!NOTE]
>
> Calls `POST /v1/agent/build`

## Options

**--name** `string` *required*

Name of the agent.

**--job-description** `string` *required*

One-line description of what the agent should do. Roark authors the system prompt from this.

**--voice** `string`

Optional display name of the agent's voice. The authored prompt matches its delivery to it.

**--tts** `object`

Optional provider voice the hosted agent speaks with.

**--tts.provider** `string` *required*

Text-to-speech provider, for example ELEVENLABS. One of `CARTESIA`, `ELEVENLABS`, `OPENAI`, `RIME`

**--tts.voice-id** `string` *required*

The provider's id for the voice.

**--tts.model** `string`

The provider's voice model, for example eleven\_flash\_v2\_5. Its current one when omitted.

**--stt** `object`

Optional transcriber the hosted agent hears with. Roark's default when omitted.

**--stt.provider** `string` *required*

Speech-to-text provider, for example DEEPGRAM. One of `DEEPGRAM`, `OPENAI_WHISPER`, `SPEECHMATICS`, `ASSEMBLYAI`, `ELEVENLABS`

**--stt.model** `string` *required*

The provider's model id, for example nova-3.

**--llm** `object`

Optional text model the hosted agent thinks with. Roark's default when omitted.

**--llm.provider** `string` *required*

Text model provider, for example OPENAI. One of `OPENAI`, `ANTHROPIC`, `GOOGLE`, `XAI`, `ULTRAVOX`, `AWS`, `ELEVENLABS`

**--llm.model** `string` *required*

The provider's model id, for example gpt-4.1.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent build --name <value> --job-description <value> --tts.provider <value> --tts.voice-id <value> --stt.provider <value> --stt.model <value> --llm.provider <value> --llm.model <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent-config get
Source: https://docs.roark.ai/cli-reference/commands/agent-config-get

Get a managed config

Fetch one managed config with its channel pointers and recent revisions, newest first.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent-config get <key> [options]
```

> [!NOTE]
>
> Calls `GET /v1/agent-config/{key}`

## Arguments

**key** `string` *required*

The key.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent-config get <key>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent-config list
Source: https://docs.roark.ai/cli-reference/commands/agent-config-list

List managed configs

List the managed agent configs in this project, most recent first.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent-config list [options]
```

> [!NOTE]
>
> Calls `GET /v1/agent-config`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent-config list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent-config promote
Source: https://docs.roark.ai/cli-reference/commands/agent-config-promote

Promote staging to production

Point the production channel at the staging revision. Real traffic reads it from the next session onward; roll back by writing the prior revision id to production (the chain preserves every state).

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent-config promote <key> [options]
```

> [!NOTE]
>
> Calls `POST /v1/agent-config/{key}/promote`

## Arguments

**key** `string` *required*

The key.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent-config promote <key>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent-config resolve
Source: https://docs.roark.ai/cli-reference/commands/agent-config-resolve

Resolve config for a session

The call your agent makes at session start. Returns the config JSON for this session, with per-session simulation recognition built in: when the session was originated by a Roark simulation (matched by caller number against the calls Roark has in flight), the STAGING revision is served for that session only, so Autoimprove candidates are tested on your real deployment. Real traffic always resolves to production; any recognition miss degrades to production too. On the first fetch of an unknown key, pass `defaults` (your baked-in config): the key is registered and the defaults become revision 1. That makes integration a single call. Cache the response per session; do not fetch per turn.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent-config resolve <key> [options]
```

> [!NOTE]
>
> Calls `POST /v1/agent-config/{key}/resolve`

## Arguments

**key** `string` *required*

The key.

## Options

**--defaults** `object`

Your baked-in config. On the first fetch of an unknown key this registers the config and becomes revision 1, so integration is a single call. Ignored once the key exists.

**--session** `object`

Session context. When the call was originated by a Roark simulation, Roark recognizes it here and serves the staging revision for this session only; real traffic always gets production.

**--session.session-id** `string`

Your session or room identifier, for correlation.

**--session.caller-number** `string`

The number calling your agent (E.164). Required for simulation recognition.

**--session.called-number** `string`

The number your agent was reached on (E.164).

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent-config resolve <key>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent-config staging delete
Source: https://docs.roark.ai/cli-reference/commands/agent-config-staging-delete

Discard the staging revision

Clear the staging channel. Production is untouched; recognized simulation sessions go back to reading production.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent-config staging delete <key> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/agent-config/{key}/staging`

## Arguments

**key** `string` *required*

The key.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent-config staging delete <key>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent-config update
Source: https://docs.roark.ai/cli-reference/commands/agent-config-update

Write a config revision

Write a new revision onto a channel. Writing production creates the key when it does not exist; writing staging stages a candidate that only Roark-recognized simulation sessions will read. Every write is a new revision; nothing is overwritten.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent-config update <key> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/agent-config/{key}`

## Arguments

**key** `string` *required*

The key.

## Options

**--channel** `string` *required*

Which channel to write. One of `production`, `staging`

**--document** `object` *required*

The full config JSON for the new revision.

**--summary** `string`

One-line story of the change.

**--agent-id** `string`

Link this config to a Roark agent (the one your calls report). Required before Autoimprove can run on it: the link is how a fix finds the config channel.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent-config update <key> --channel <value> --document <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent create
Source: https://docs.roark.ai/cli-reference/commands/agent-create

Create a new agent

Creates a new agent for the authenticated project.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent create [options]
```

> [!NOTE]
>
> Calls `POST /v1/agent`

## Options

**--name** `string` *required*

Name of the agent.

**--description** `string`

Description of the agent.

**--custom-id** `string`

Custom identifier for the agent.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent create --name <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent draft create
Source: https://docs.roark.ai/cli-reference/commands/agent-draft-create

Start a draft

Opens the agent's one shared draft, a copy of the live version or of an older published version. Fails with 409 when a draft is already open.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent draft create <agent-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/agent/{agentId}/draft`

## Arguments

**agent-id** `string` *required*

The agent-id.

## Options

**--from-version-id** `string`

Start from this published version instead of the live one.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent draft create <agent-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent draft delete
Source: https://docs.roark.ai/cli-reference/commands/agent-draft-delete

Discard the draft

Throws the open draft away and returns it. The live version is unchanged.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent draft delete <agent-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/agent/{agentId}/draft`

## Arguments

**agent-id** `string` *required*

The agent-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent draft delete <agent-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent draft get
Source: https://docs.roark.ai/cli-reference/commands/agent-draft-get

Get an agent's draft

The agent's open draft, with the issues that would stop it from being published.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent draft get <agent-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/agent/{agentId}/draft`

## Arguments

**agent-id** `string` *required*

The agent-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent draft get <agent-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent draft publish
Source: https://docs.roark.ai/cli-reference/commands/agent-draft-publish

Publish the draft

Makes the draft the live version, numbered v1, v2, ... Fails with 409 if the draft was edited after expectedRevision, if another version was published since the draft started, or if it changes nothing; with 400 and the issues if the version cannot run.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent draft publish <agent-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/agent/{agentId}/draft/publish`

## Arguments

**agent-id** `string` *required*

The agent-id.

## Options

**--expected-revision** `integer` *required*

The draft revision you reviewed.

**--summary** `string`

What changed.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent draft publish <agent-id> --expected-revision <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent draft update
Source: https://docs.roark.ai/cli-reference/commands/agent-draft-update

Edit the draft

Changes settings on the draft. Omitted settings stay as they are. Pass the revision you read as expectedRevision to fail with 409 if anyone edited the draft since, instead of applying on top of their edit.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent draft update <agent-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/agent/{agentId}/draft`

## Arguments

**agent-id** `string` *required*

The agent-id.

## Options

**--expected-revision** `integer`

The draft's revision when you read it. When sent, the edit is refused if the draft changed since; when omitted, it applies on top of the latest edit.

**--settings** `object` *required*

The settings to change. Omitted settings stay as they are; null clears one to the runtime default. subAgents replaces every speaker, model configs included, and must include `main`; in a model config, omitted fields use the runtime default.

**--settings.modalities** `array`

One of `call`, `chat` Repeatable.

**--settings.language** `string`

**--settings.first-message-mode** `string`

One of `SPEAK_TEXT`, `PLAY_AUDIO`, `WAIT_FOR_CALLER`

**--settings.first-message** `string`

**--settings.first-message-audio-url** `string`

**--settings.interruption-mode** `string`

One of `ALLOW`, `BLOCK`

**--settings.vad-start-secs** `number`

**--settings.vad-stop-secs** `number`

**--settings.call-max-duration-seconds** `integer`

**--settings.call-silence-timeout-seconds** `integer`

**--settings.idle-timeout-seconds** `integer`

**--settings.idle-max-count** `integer`

**--settings.idle-messages** `array`

Repeatable.

**--settings.end-detection-phrases** `array`

Repeatable.

**--settings.end-detection-reasons** `array`

Repeatable.

**--settings.end-call-goodbye-message** `string`

**--settings.background-noise** `string`

One of `NONE`, `AIRPORT`, `CHILDREN_PLAYING`, `CITY`, `COFFEE_SHOP`, `CONSTRUCTION`, `CRYING_BABY`, `DRIVING`, `LIBRARY`, `OFFICE`, `THUNDERSTORM`, `TRAIN`

**--settings.background-noise-url** `string`

**--settings.background-noise-volume** `number`

**--settings.chat-max-duration-seconds** `integer`

**--settings.chat-inactivity-timeout-seconds** `integer`

**--settings.sub-agents** `array`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent draft update <agent-id> --settings <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent endpoint create
Source: https://docs.roark.ai/cli-reference/commands/agent-endpoint-create

Create a new agent endpoint

Creates a new agent endpoint for the authenticated project.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent endpoint create [options]
```

> [!NOTE]
>
> Calls `POST /v1/agent/endpoint`

## Options

**--agent-id** `string` *required*

Agent ID to associate this endpoint with.

**--value** `string` *required*

Phone number in E.164 format (e.g., +12345678900).

**--environment** `string`

Environment name (default: production).

**--direction** `string` *required*

Call direction: INCOMING, OUTGOING, or `INCOMING_AND_OUTGOING`. One of `INCOMING`, `OUTGOING`, `INCOMING_AND_OUTGOING`

**--outbound-dial-type** `string`

Outbound dial type: NONE or `HTTP_REQUEST` (default: NONE). One of `NONE`, `HTTP_REQUEST`

**--outbound-dial-http-request-definition-id** `string`

ID of the HTTP request definition for outbound dialing (required when outboundDialType is `HTTP_REQUEST`).

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent endpoint create --agent-id <value> --value <value> --direction <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent endpoint get
Source: https://docs.roark.ai/cli-reference/commands/agent-endpoint-get

Get agent endpoint by ID

Returns a specific agent endpoint by its ID.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent endpoint get <endpoint-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/agent/endpoint/{endpointId}`

## Arguments

**endpoint-id** `string` *required*

The ID of the agent endpoint to retrieve.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent endpoint get <endpoint-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent endpoint list
Source: https://docs.roark.ai/cli-reference/commands/agent-endpoint-list

List agent endpoints

Returns a paginated list of agent endpoints for the authenticated project.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent endpoint list [options]
```

> [!NOTE]
>
> Calls `GET /v1/agent/endpoint`

## Options

**--limit** `integer`

Maximum number of endpoints to return (default: 20, max: 50).

**--after** `string`

Cursor for pagination - endpoint ID to start after.

**--agent-id** `string`

Filter by agent ID.

**--search-text** `string`

Search text to filter endpoints.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent endpoint list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent endpoint update
Source: https://docs.roark.ai/cli-reference/commands/agent-endpoint-update

Update an agent endpoint

Updates an existing agent endpoint by its ID. Only environment and outboundDialType can be modified.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent endpoint update <endpoint-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/agent/endpoint/{endpointId}`

## Arguments

**endpoint-id** `string` *required*

The ID of the agent endpoint to update.

## Options

**--environment** `string`

Environment name.

**--outbound-dial-type** `string`

Outbound dial type: NONE or `HTTP_REQUEST`. One of `NONE`, `HTTP_REQUEST`

**--outbound-dial-http-request-definition-id** `string`

ID of the HTTP request definition for outbound dialing.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent endpoint update <endpoint-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent get
Source: https://docs.roark.ai/cli-reference/commands/agent-get

Get agent by ID

Returns a specific agent by its ID.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent get <agent-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/agent/{agentId}`

## Arguments

**agent-id** `string` *required*

The ID of the agent to retrieve.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent get <agent-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent list
Source: https://docs.roark.ai/cli-reference/commands/agent-list

List agents

Returns a paginated list of agents for the authenticated project.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent list [options]
```

> [!NOTE]
>
> Calls `GET /v1/agent`

## Options

**--limit** `integer`

**--after** `string`

**--search-text** `string`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent prompt list
Source: https://docs.roark.ai/cli-reference/commands/agent-prompt-list

List an agent's prompts

Returns the agent's prompt lineages. Each lineage is an independent version history, labelled by its `source` (USER, API, CONFIG, or a provider integration). `prompt` is the current content.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent prompt list <agent-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/agent/{agentId}/prompts`

## Arguments

**agent-id** `string` *required*

The agent-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent prompt list <agent-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent prompt update
Source: https://docs.roark.ai/cli-reference/commands/agent-prompt-update

Set an agent's prompt

Sets the agent's API-managed prompt. This is its own version history (`source: API_MANAGED`), separate from prompts observed on calls, edited in the app, or managed by config-as-code. Setting the same content twice is a no-op (no new version). Roark does not run your agent and no metric reads this prompt: it is stored and versioned for your reference.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent prompt update <agent-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/agent/{agentId}/prompts`

## Arguments

**agent-id** `string` *required*

The agent-id.

## Options

**--prompt** `string` *required*

The prompt content to set.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent prompt update <agent-id> --prompt <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent prompt version list
Source: https://docs.roark.ai/cli-reference/commands/agent-prompt-version-list

List a prompt's versions

Returns the full version history of a single prompt lineage, newest first.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent prompt version list <agent-id> <prompt-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/agent/{agentId}/prompts/{promptId}/versions`

## Arguments

**agent-id** `string` *required*

The agent-id.

**prompt-id** `string` *required*

The prompt-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent prompt version list <agent-id> <prompt-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent update
Source: https://docs.roark.ai/cli-reference/commands/agent-update

Update an agent

Updates an existing agent by its ID.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent update <agent-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/agent/{agentId}`

## Arguments

**agent-id** `string` *required*

The ID of the agent to update.

## Options

**--name** `string`

Name of the agent.

**--description** `string`

Description of the agent.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent update <agent-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent version get
Source: https://docs.roark.ai/cli-reference/commands/agent-version-get

Get an agent version

One version with every setting, plus the issues that stop it from running, if any.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent version get <agent-id> <version-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/agent/{agentId}/versions/{versionId}`

## Arguments

**agent-id** `string` *required*

The agent-id.

**version-id** `string` *required*

The version-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent version get <agent-id> <version-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent version list
Source: https://docs.roark.ai/cli-reference/commands/agent-version-list

List an agent's versions

The open draft first, if there is one, then published versions newest first. Paginated: pass `pagination.nextCursor` as `after` for the next page.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent version list <agent-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/agent/{agentId}/versions`

## Arguments

**agent-id** `string` *required*

The agent-id.

## Options

**--limit** `integer`

**--after** `string`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent version list <agent-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark agent version rollback
Source: https://docs.roark.ai/cli-reference/commands/agent-version-rollback

Roll an agent back to a version

Publishes a new version, numbered after the latest, with this version's settings and makes it live. History stays append-only. An open draft started before this is kept but can no longer be published (409 `agent_draft_base_changed`); discard it or start a new one. Pass `expectedLiveVersionId` to be refused (409 `agent_live_version_changed`) if another version went live since you read it.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent version rollback <agent-id> <version-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/agent/{agentId}/versions/{versionId}/rollback`

## Arguments

**agent-id** `string` *required*

The agent-id.

**version-id** `string` *required*

The version-id.

## Options

**--summary** `string`

**--expected-live-version-id** `string`

The live version when you decided to roll back. When sent, the rollback is refused with 409 if another version went live since.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent version rollback <agent-id> <version-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark analytics query
Source: https://docs.roark.ai/cli-reference/commands/analytics-query

Run an analytics query

Aggregates metric values and conversation counts over a time window and returns one series per measure, with a total for the window and per-interval buckets. The same query a dashboard widget runs, without saving anything. - `measures`: what to aggregate. A METRIC measure names its metric by slug; CONVERSATION counts conversations; UPTIME reads health checks. The aggregation defaults by the metric's output type. - `timeRange`: LAST n DAYS/WEEKS/MONTHS, a `CALENDAR_ALIGNED` period (TODAY, `THIS_MONTH`, ...), FIXED from/to, or SINCE. Defaults to the last 30 days; at most 731 days. Calendar periods and buckets align to `timezone` (IANA, default UTC). - `filters` apply to every measure; a measure's own `filters` add to them. - `breakdown` on a measure returns one series per value (largest 50 kept). - `comparisonGroups` (2 to 5) run every measure once per named filter set; `comparisonMode: PREVIOUS_PERIOD` adds the preceding window's values. Values are in the metric's display units. A `PASS_RATE` is a 0 to 1 share; `sampleSize` 0 means no data. A query the analytics store cannot answer returns 400 naming the unsupported part.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark analytics query [options]
```

> [!NOTE]
>
> Calls `POST /v1/analytics/query`

## Options

**--measures** `array` *required*

What to aggregate: 1 to 10 measures.

**--filters** `array`

Applied to every measure, on top of each measure's own filters.

**--interval** `string`

Bucket size. Defaults from the window's length and steps coarser when the window would exceed 500 buckets. One of `HOUR`, `DAY`, `WEEK`, `MONTH`

**--comparison-mode** `string`

`PREVIOUS_PERIOD` adds the preceding window of the same length to every series. One of `NONE`, `PREVIOUS_PERIOD`

**--comparison-groups** `array`

Run every measure once per named filter set, e.g. one group per agent.

**--time-range** `string`

Defaults to the last 30 days.

**--timezone** `string`

IANA zone that buckets and calendar periods align to, e.g. America/New\_York. Defaults to UTC.

**--include-buckets** `switch`

Set false to return only the window totals.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark analytics query --measures <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark analytics schema get
Source: https://docs.roark.ai/cli-reference/commands/analytics-schema-get

Get the analytics schema

Lists what can go into an analytics query for this project: every metric by slug with its output type, default aggregation and the aggregations it supports; the measure types; the filter and breakdown types with the properties each accepts; and the custom call properties seen in the project. Read this first when building a query.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark analytics schema get [options]
```

> [!NOTE]
>
> Calls `GET /v1/analytics/schema`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark analytics schema get
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark autoimprove job answer
Source: https://docs.roark.ai/cli-reference/commands/autoimprove-job-answer

Answer Roark's question

Answer the question a job is blocked on. Only meaningful while the job status is `NEEDS_INPUT` (the open QUESTION entry carries the offered options; free text is also accepted). Otherwise returns a conflict.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job answer <job-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/autoimprove/job/{jobId}/answer`

## Arguments

**job-id** `string` *required*

The job-id.

## Options

**--text** `string` *required*

The message for Roark.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job answer <job-id> --text <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark autoimprove job cancel
Source: https://docs.roark.ai/cli-reference/commands/autoimprove-job-cancel

Cancel a job

Stop a live job. Production is never changed by a cancel; everything the job set up (the shadow agent, its phone number, authored test flows and run plan) is cleaned up automatically.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job cancel <job-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/autoimprove/job/{jobId}/cancel`

## Arguments

**job-id** `string` *required*

The job-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job cancel <job-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark autoimprove job create
Source: https://docs.roark.ai/cli-reference/commands/autoimprove-job-create

Start an Autoimprove job

Commission Roark on an objective. It clones the agent into a staging shadow (or uses your designated staging agent), authors a validation suite of simulated callers, measures a baseline, changes the staging configuration, and re-tests until the objective metric passes its target. Production is never touched by the loop; verified changes wait for promotion. Requires an active provider integration (Vapi, Retell, or ElevenLabs) with agent config writes enabled. One live job per agent: starting a second returns a conflict. The job runs asynchronously; poll GET /v1/autoimprove/job/\{jobId} or watch it in the dashboard. When its status is `NEEDS_INPUT`, answer via the answer endpoint; when `AWAITING_PROMOTE`, promote or dismiss.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job create [options]
```

> [!NOTE]
>
> Calls `POST /v1/autoimprove/job`

## Options

**--agent-id** `string` *required*

The production agent to improve. It is never modified until you promote.

**--objective-metric-definition-id** `string` *required*

The metric that defines success: a pass/fail metric, or a threshold variant of a scale metric (for example "PII Handling >= 4"). Roark measures the pass rate of this metric across simulated calls.

**--objective-label** `string` *required*

Human-readable label for the objective, shown everywhere the job appears.

**--target-value** `number`

The pass-rate percentage that counts as fixed. Defaults to 90.

**--staging-agent-id** `string`

An existing agent to stage changes on instead of the default shadow clone. Must be a different agent from agentId, on the same provider.

**--customer-integration-id** `string`

The provider integration whose credentials Roark uses. Omit to use the project's active integration for the agent's provider. The integration must have agent config writes enabled.

**--validation-run-plan-id** `string`

An existing simulation run plan to validate with. Omit to let Roark author its own suite.

**--max-iterations** `integer`

Cap on decision turns. Defaults to 50.

**--max-sim-calls** `integer`

Cap on simulated calls dialed. Defaults to 200.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job create --agent-id <value> --objective-metric-definition-id <value> --objective-label <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark autoimprove job dismiss
Source: https://docs.roark.ai/cli-reference/commands/autoimprove-job-dismiss

Dismiss a verified job

Discard a verified job without promoting: production stays untouched and the staging resources are cleaned up. Only a job in `AWAITING_PROMOTE` can be dismissed.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job dismiss <job-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/autoimprove/job/{jobId}/dismiss`

## Arguments

**job-id** `string` *required*

The job-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job dismiss <job-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark autoimprove job get
Source: https://docs.roark.ai/cli-reference/commands/autoimprove-job-get

Get an Autoimprove job

Fetch one job with its full worklog: every step Roark took, the validation batches with their pass-rate movement, any question it is waiting on, and its final report once concluded.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job get <job-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/autoimprove/job/{jobId}`

## Arguments

**job-id** `string` *required*

The job-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job get <job-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark autoimprove job guidance send
Source: https://docs.roark.ai/cli-reference/commands/autoimprove-job-guidance-send

Send guidance to a running job

Steer Roark mid-job. The message is folded into its next decision and is binding. Accepted while the job is live (RUNNING, `NEEDS_INPUT`, or PAUSED); a concluded job returns a conflict.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job guidance send <job-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/autoimprove/job/{jobId}/guidance`

## Arguments

**job-id** `string` *required*

The job-id.

## Options

**--text** `string` *required*

The message for Roark.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job guidance send <job-id> --text <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark autoimprove job list
Source: https://docs.roark.ai/cli-reference/commands/autoimprove-job-list

List Autoimprove jobs

List the Autoimprove jobs in this project, most recent first, capped at 100. A job is one autonomous engagement: Roark improving one agent toward one objective metric on a staging copy, with a human-gated promote to production at the end.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job list [options]
```

> [!NOTE]
>
> Calls `GET /v1/autoimprove/job`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark autoimprove job promote
Source: https://docs.roark.ai/cli-reference/commands/autoimprove-job-promote

Promote a verified job to production

Apply the verified staging changes to the PRODUCTION agent. Only a job in `AWAITING_PROMOTE` can be promoted. A snapshot of the production configuration is taken immediately before the write, so the promote is fully rollbackable. After the promote the staging shadow and its phone number are cleaned up.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job promote <job-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/autoimprove/job/{jobId}/promote`

## Arguments

**job-id** `string` *required*

The job-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job promote <job-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark benchmark leaderboard get
Source: https://docs.roark.ai/cli-reference/commands/benchmark-leaderboard-get

Get the benchmark leaderboard

Returns one ranked page of a suite version’s published targets, each with the metric cells it is judged on. Ranking happens in the database, so paging through the board is consistent. Every default the server applies (suite version, condition, sort metric, sort direction) is echoed on the response, so a page can be cited without guessing how it was ordered.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark benchmark leaderboard get [options]
```

> [!NOTE]
>
> Calls `GET /v1/benchmark/leaderboard`

## Options

**--suite** `string` *required*

**--suite-version** `string`

**--condition-key** `string`

**--sort-by** `string`

**--order** `string`

One of `asc`, `desc`

**--metrics** `string`

Comma-separated metric keys to project per row. Defaults to the headline set.

**--limit** `integer`

Maximum number of records to return (default: 20, max: 100).

**--offset** `integer`

Pagination offset.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark benchmark leaderboard get --suite <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark benchmark metric list
Source: https://docs.roark.ai/cli-reference/commands/benchmark-metric-list

List benchmark metrics

Returns the metrics a suite version published, each with the direction that counts as better. Use it to pick a valid `sortBy` for the leaderboard: a metric key outside this list is rejected rather than silently ranking every target null.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark benchmark metric list [options]
```

> [!NOTE]
>
> Calls `GET /v1/benchmark/metric`

## Options

**--suite** `string` *required*

**--suite-version** `string`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark benchmark metric list --suite <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark benchmark suite list
Source: https://docs.roark.ai/cli-reference/commands/benchmark-suite-list

List benchmark suites

Returns every benchmark suite with published results, along with the suite version the other endpoints default to. Start here: the suite name is a required parameter everywhere else and there is no other way to discover it.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark benchmark suite list [options]
```

> [!NOTE]
>
> Calls `GET /v1/benchmark/suite`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark benchmark suite list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark benchmark target get
Source: https://docs.roark.ai/cli-reference/commands/benchmark-target-get

Get a benchmark target

Returns a target’s published sweep and every one of its aggregate cells: the overall rollup plus a set per condition. Defaults to the current sweep; pass `publicationId` from the history endpoint to read a superseded one, which is how a regression is compared generation to generation. Numbers only — no transcript and no audio.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark benchmark target get <target-key> [options]
```

> [!NOTE]
>
> Calls `GET /v1/benchmark/target/{targetKey}`

## Arguments

**target-key** `string` *required*

The target-key.

## Options

**--suite** `string` *required*

**--suite-version** `string`

**--publication-id** `string`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark benchmark target get <target-key> --suite <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark benchmark target history list
Source: https://docs.roark.ai/cli-reference/commands/benchmark-target-history-list

List a target’s published generations

Returns every sweep published for a target, newest first, including superseded ones. This is the trend read: it answers "did this model regress?" from Roark’s own published record. Metadata only — pass a row’s `publicationId` to GET /v1/benchmark/target/\{targetKey} for that generation’s numbers.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark benchmark target history list <target-key> [options]
```

> [!NOTE]
>
> Calls `GET /v1/benchmark/target/{targetKey}/history`

## Arguments

**target-key** `string` *required*

The target-key.

## Options

**--suite** `string` *required*

**--suite-version** `string`

**--limit** `integer`

Maximum number of records to return (default: 20, max: 100).

**--offset** `integer`

Pagination offset.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark benchmark target history list <target-key> --suite <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark benchmark target score-sample list
Source: https://docs.roark.ai/cli-reference/commands/benchmark-target-score-sample-list

List score samples for a target

Returns the individual scored calls behind a target’s aggregate cells, each with the scorer’s rationale — the "why is this number what it is" read. Filter to one cell with `conditionKey` and `metricKey`. Metrics computed without a rationale (latencies, counts) contribute no samples. Returns reasoning only: no transcript and no audio.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark benchmark target score-sample list <target-key> [options]
```

> [!NOTE]
>
> Calls `GET /v1/benchmark/target/{targetKey}/score-sample`

## Arguments

**target-key** `string` *required*

The target-key.

## Options

**--suite** `string` *required*

**--suite-version** `string`

**--condition-key** `string`

**--metric-key** `string`

**--limit** `integer`

Maximum number of records to return (default: 20, max: 100).

**--offset** `integer`

Pagination offset.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark benchmark target score-sample list <target-key> --suite <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark calendar-connection check
Source: https://docs.roark.ai/cli-reference/commands/calendar-connection-check

Check a calendar connection

Re-verifies the stored credential against the provider and records the result on the connection. A credential the provider rejects sets `status` to `INVALID_CREDENTIALS`. A provider outage is a 500 and leaves the status untouched: being unable to reach the provider is not evidence that the customer’s credential is bad.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection check <connection-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/calendar-connection/{connectionId}/check`

## Arguments

**connection-id** `string` *required*

Calendar connection ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection check <connection-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark calendar-connection create
Source: https://docs.roark.ai/cli-reference/commands/calendar-connection-create

Connect a calendar account

Verifies the credential against the provider and, only if it works, stores it encrypted. A credential the provider rejects is a 400 and no connection is created: a connection that has never worked would next be exercised by an inbound call at 2am, with nobody there to fix it.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection create [options]
```

> [!NOTE]
>
> Calls `POST /v1/calendar-connection`

## Options

**--provider** `string` *required*

Who holds the calendar. `CAL_COM` is the only provider that can be connected today; GOOGLE and CALENDLY are described by `capabilities` but not yet connectable. One of `CAL_COM`, `GOOGLE`, `CALENDLY`

**--label** `string` *required*

Customer-facing name for this account, unique within the project. Required.

**--api-key** `string` *required*

For `CAL_COM`: a Cal.com API key (starts with `cal_`). It is verified against Cal.com before anything is stored, so an invalid key is a 400 and no connection is created. Stored encrypted and never returned.

**--timezone** `string`

The named IANA timezone to offer slots in, for example `America/Los_Angeles`. Omit to adopt the provider account’s own timezone. A fixed offset such as `-08:00` is rejected: it does not observe DST.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection create --provider <value> --label <value> --api-key <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark calendar-connection delete
Source: https://docs.roark.ai/cli-reference/commands/calendar-connection-delete

Delete a calendar connection

Disconnects the calendar and destroys the stored credential. A credential the customer deleted should stop existing, so this is not an archive. Appointments already recorded against the connection are kept, so a booking stays explainable after the calendar is disconnected. Refused with a 409 while a calendar tool is still bound to the connection, naming the tools: point them at another calendar, or delete them, first. A dangling binding would otherwise be found by whoever next tried to ship the agent, who is not the person who disconnected the calendar.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection delete <connection-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/calendar-connection/{connectionId}`

## Arguments

**connection-id** `string` *required*

Calendar connection ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection delete <connection-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark calendar-connection get
Source: https://docs.roark.ai/cli-reference/commands/calendar-connection-get

Get a calendar connection

Returns one calendar connection. The stored credential is never included.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection get <connection-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/calendar-connection/{connectionId}`

## Arguments

**connection-id** `string` *required*

Calendar connection ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection get <connection-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark calendar-connection list
Source: https://docs.roark.ai/cli-reference/commands/calendar-connection-list

List calendar connections

Returns every calendar connection in the authenticated project, by label. A project has a handful of these, so the list is not paginated: `pagination.hasMore` is always false.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection list [options]
```

> [!NOTE]
>
> Calls `GET /v1/calendar-connection`

## Options

**--provider** `string`

Who holds the calendar. `CAL_COM` is the only provider that can be connected today; GOOGLE and CALENDLY are described by `capabilities` but not yet connectable. One of `CAL_COM`, `GOOGLE`, `CALENDLY`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark calendar-connection target get
Source: https://docs.roark.ai/cli-reference/commands/calendar-connection-target-get

Read one bookable target

The same entry as the list, plus what it demands of a booker. `requiresAttendeeEmail` is the field that decides an agent’s design: the attendee object itself needs only a name and a timezone, but an event type can declare email required, and an agent that cannot collect one simply cannot book on it. Read this when configuring a calendar tool, not after a caller is already waiting. Costs one provider call more than the list, which is why the list does not fold it in.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection target get <connection-id> <target-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/calendar-connection/{connectionId}/targets/{targetId}`

## Arguments

**connection-id** `string` *required*

Calendar connection ID.

**target-id** `string` *required*

Target id, as the list returned it.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection target get <connection-id> <target-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark calendar-connection target list
Source: https://docs.roark.ai/cli-reference/commands/calendar-connection-target-list

List what can be booked on a calendar

What an agent can be pointed at on this connection: Cal.com event types today, Google calendars when that lands. Read live from the provider, so an event type added since you connected is here without reconnecting. Each entry carries a `target` ready to put on a calendar tool, so nothing has to assemble one by hand.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection target list <connection-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/calendar-connection/{connectionId}/targets`

## Arguments

**connection-id** `string` *required*

Calendar connection ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection target list <connection-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark calendar-connection update
Source: https://docs.roark.ai/cli-reference/commands/calendar-connection-update

Update a calendar connection

Renames a connection, changes the timezone slots are offered in, or rotates the credential. A rotated credential is verified first, and a verified credential returns the connection to ACTIVE.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection update <connection-id> [options]
```

> [!NOTE]
>
> Calls `PATCH /v1/calendar-connection/{connectionId}`

## Arguments

**connection-id** `string` *required*

Calendar connection ID.

## Options

**--label** `string`

Rename the connection.

**--timezone** `string`

Change the named IANA timezone slots are offered in.

**--api-key** `string`

Rotate the credential. The new key is verified before it is stored, and a verified key returns the connection to ACTIVE, which is how a connection recovers from `INVALID_CREDENTIALS`.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark calendar-connection update <connection-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark call batch delete
Source: https://docs.roark.ai/cli-reference/commands/call-batch-delete

Delete calls in batch

Deletes up to 100 calls by ID in one request. Each deleted call, its metrics and its analysis are hidden from every read, and any analysis still running on it is cancelled. IDs that do not exist in this project, or were already deleted, are returned in notFoundCallIds and left untouched. The request succeeds even when none of the calls are found. IDs are returned lowercase, each once.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call batch delete [options]
```

> [!NOTE]
>
> Calls `POST /v1/call/batch-delete`

## Options

**--call-ids** `array` *required*

IDs of the calls to delete, at most 100. Repeated IDs are counted once. Repeatable.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call batch delete --call-ids <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark call cost get
Source: https://docs.roark.ai/cli-reference/commands/call-cost-get

Get call cost

The billable spend attributed to one call, split by cost type, with every cost line behind it. Simulated calls include the costs their simulation incurred before the call connected. Amounts are billable spend only, in exact micro-dollars (`*UsdMicros`) and dollars (`*Usd`, for display). Requires an API key with the `billing:read` permission.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call cost get <call-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/call/{callId}/cost`

## Arguments

**call-id** `string` *required*

The call-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call cost get <call-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark call create
Source: https://docs.roark.ai/cli-reference/commands/call-create

Create a call

Create a new call with recording, transcript, agents, and customers

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call create [options]
```

> [!NOTE]
>
> Calls `POST /v1/call`

## Options

**--recording-url** `string` *required*

URL of source recording (must be an accessible WAV, MP3, MP4, or OGG file). Can be a signed URL.

**--stereo-recording-url** `string`

URL of source stereo recording. Must be accessible. Can be a signed URL. Supported formats: WAV, MP3, MP4, OGG.

**--started-at** `string` *required*

When the call started (ISO 8601 format).

**--interface-type** `string` *required*

Interface type of the call (PHONE or WEB). One of `PHONE`, `WEB`

**--call-direction** `string` *required*

Direction of the call (INBOUND or OUTBOUND). One of `INBOUND`, `OUTBOUND`

**--ended-status** `string`

High-level call end status, indicating how the call terminated. One of `PARTICIPANTS_DID_NOT_SPEAK`, `AGENT_DID_NOT_ANSWER`, `AGENT_DID_NOT_SPEAK`, `AGENT_STOPPED_SPEAKING`, `AGENT_ENDED_CALL`, `AGENT_SAID_END_CALL_PHRASE`, `AGENT_TRANSFERRED_CALL`, `AGENT_BUSY`, `AGENT_ERROR`, `CUSTOMER_ENDED_CALL`, `VOICE_MAIL_REACHED`, `SILENCE_TIME_OUT`, `PHONE_CALL_PROVIDER_CONNECTION_ERROR`, `CUSTOMER_DID_NOT_ANSWER`, `CUSTOMER_DID_NOT_SPEAK`, `CUSTOMER_STOPPED_SPEAKING`, `CUSTOMER_BUSY`, `DIAL_ERROR`, `MAX_DURATION_REACHED`, `UNKNOWN`

**--transcript** `array`

List of transcript entries made during the call.

**--tool-invocations** `array`

List of tool invocations made during the call.

**--properties** `object`

Custom properties to include with the call. These can be used for filtering and will show in the call details page.

**--vapi-call-id** `string`

The Vapi call ID (UUID) to link this call with OpenTelemetry trace data from Vapi. Used for matching calls with OTEL traces.

**--livekit-room-id** `string`

The LiveKit Cloud room ID to link this call with OpenTelemetry trace data from LiveKit. Used for matching calls with OTEL traces.

**--external-id** `string`

A stable identifier from your own system (e.g. session ID, conversation ID) used to correlate this call with OpenTelemetry traces. Set the same value as a `roark.external_id` span or resource attribute on your traces and the matching trace will be linked automatically. Must be unique within a project.

**--agent** `object`

Single agent participating in the call. Use this for simpler API when you have only one agent.

**--agent.endpoint** `string`

Endpoint configuration for this agent (optional).

**--agent.prompt** `object`

Agent's prompt configuration (optional).

**--agents** `array`

Agents participating in the call. Each agent requires identification and prompt information.

**--customer** `object`

Single customer participating in the call. Use this for simpler API when you have only one customer.

**--customer.label** `string`

Label to identify this customer in the transcript (e.g., "speaker-01", "speaker-02").

**--customer.phone-number-e164** `string` *required*

Customer phone number in E.164 format (e.g., +14155551234).

**--customers** `array`

Customers participating in the call.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call create --recording-url <value> --started-at <value> --interface-type <value> --call-direction <value> --customer.phone-number-e164 <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark call delete
Source: https://docs.roark.ai/cli-reference/commands/call-delete

Delete a call

Deletes a call by its ID. The call, its metrics and its analysis are hidden from every read, and any analysis still running on it is cancelled. Returns 404 when the call does not exist in this project or was already deleted.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call delete <call-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/call/{callId}`

## Arguments

**call-id** `string` *required*

The call-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call delete <call-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark call get
Source: https://docs.roark.ai/cli-reference/commands/call-get

Get a call by ID

Retrieve an existing call by its unique identifier

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call get <call-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/call/{callId}`

## Arguments

**call-id** `string` *required*

The call-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call get <call-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark call list
Source: https://docs.roark.ai/cli-reference/commands/call-list

List calls

Returns a paginated list of calls for the authenticated project. Pass agentId to keep calls for specific agents, or properties\[agent\_name]\[eq] when you only have the agent name. sortBy and sortDirection apply together with those filters.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call list [options]
```

> [!NOTE]
>
> Calls `GET /v1/call`

## Options

**--limit** `integer`

Maximum number of calls to return (default: 20, max: 100).

**--after** `string`

Cursor for pagination - use the nextCursor value from a previous response.

**--sort-by** `string`

Field to sort by (default: createdAt). One of `createdAt`, `startedAt`, `endedAt`, `duration`, `title`, `status`

**--sort-direction** `string`

Sort direction (default: desc). One of `asc`, `desc`

**--status** `string`

Filter by call status. One of `RINGING`, `IN_PROGRESS`, `ENDED`

**--search-text** `string`

Search text to filter calls by title, summary, or transcript.

**--agent-id** `string`

Calls that include any of these agents. One id, or several separated by commas, at most 100. A call with more than one agent matches when any of them is listed. An id that is not an agent on a call in this project matches nothing. Combines with the other filters, including sortBy and sortDirection.

**--simulation-run-plan-job-id** `string`

Filter by simulation run plan job ID to get all calls from a specific simulation batch.

**--simulation-run-plan-job-ids** `string`

Calls from any of these simulation runs (run plan job ids), comma-separated, at most 100.

**--simulation-run-plan-ids** `string`

Calls from every run of any of these run plans, comma-separated, at most 100.

**--counted-in-results** `string`

true: only simulated calls their run's results count (not invalidated, for example because the agent never spoke). false: only the simulated calls the results leave out, each invalidated. Live calls, and calls removed from their run, match neither. One of `true`, `false`

**--exclude-hidden-runs** `string`

true: leave out calls from runs hidden from the runs list. Live calls stay in. false is the same as leaving it out. One of `true`, `false`

**--properties** `object`

Filter on properties, one entry per property name. properties\[\<name>]\[exists]=true keeps calls that have the property set (whatever the value), properties\[\<name>]\[exists]=false keeps calls missing it (never sent, sent as null, or removed). properties\[\<name>]\[eq]=\<value> keeps calls whose value equals it, ignoring case. agent\_name and agent\_id are built in and read the agents linked to the call, not a stored snapshot: agent\_name matches an agent name, agent\_id matches an agent id (a UUID). A call with several agents matches when any of them does. A custom property the project has never stored matches no calls under eq. Names match the way properties are stored, so planTier and plan\_tier are one property, and agentName is agent\_name. At most 20 properties. A name taken by a built-in call filter, such as status, is rejected with 400. One operator per property.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark call metric list
Source: https://docs.roark.ai/cli-reference/commands/call-metric-list

List call metrics

Fetch all call-level metrics for a specific call, including both system-generated and custom metrics. Only returns rows from the \*\*latest\*\* metric-collection job per metric — if the same metric has been recomputed, prior runs are excluded and remain in the metric history. By default returns only successfully computed metrics; pass `?status=all` to also include rows that resolved as `NOT_APPLICABLE` / `DATA_MISSING` / ERROR (the `value` field is omitted on those entries — check `captureStatus`).

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call metric list <call-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/call/{callId}/metrics`

## Arguments

**call-id** `string` *required*

The call-id.

## Options

**--flatten** `string`

Whether to return a flat list instead of grouped by metric definition (default: false).

**--status** `string`

Filter metrics by capture status. `success` (default) returns only successfully computed metrics — backwards-compatible with the historical behavior. `all` also returns `NOT_APPLICABLE` / `DATA_MISSING` / ERROR rows (with `value` omitted), so clients can distinguish "still computing" from "computed but no value" and exit retry loops correctly. One of `success`, `all`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call metric list <call-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark call property update
Source: https://docs.roark.ai/cli-reference/commands/call-property-update

Update the properties of a call

Add, overwrite or remove custom properties on an already-existing call. Use this for data that becomes available after the call was created, e.g. an outcome your system validates a few minutes later. Merge semantics: a value adds or overwrites the property, null removes it, and properties left out are untouched. The patch is applied whole or not at all: a property Roark sets itself, or a value whose type differs from the type the project already stores that property as, rejects the request with a 400 and nothing is written. A 409 means another request created one of the properties with a different value type at the same moment; nothing was written. Optionally pass `metrics` to (re)score metrics over the updated properties, which triggers a billed metric collection job and requires the `metric:create` permission.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call property update <call-id> [options]
```

> [!NOTE]
>
> Calls `PATCH /v1/call/{callId}/properties`

## Arguments

**call-id** `string` *required*

The call-id.

## Options

**--properties** `object` *required*

Properties to change on the call. A string, number or boolean value adds the property or overwrites it; null removes it. Properties left out are untouched. Keys are stored in snake\_case, as on POST /v1/call. A property keeps the value type it was first stored with in the project, so a value of another type is rejected. Properties Roark sets itself (direction, duration\_seconds, agent\_id, ...) can't be changed. Max 100 per request.

**--metrics** `array`

Optional. Metric definitions to (re)score on this call once the properties are written, e.g. metrics that read the call's properties. Triggers a metric collection job (billed, credit-gated) and requires the 'metric:create' permission. Omit to update properties without scoring. Max 20.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call property update <call-id> --properties <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark call sentiment-run list
Source: https://docs.roark.ai/cli-reference/commands/call-sentiment-run-list

List call sentiment runs

Fetch detailed sentiment analysis results for a specific call, including emotional tone, key phrases, and sentiment scores.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call sentiment-run list <call-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/call/{callId}/sentiment-run`

## Arguments

**call-id** `string` *required*

The call-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call sentiment-run list <call-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark call tool-invocation append
Source: https://docs.roark.ai/cli-reference/commands/call-tool-invocation-append

Append tool invocations to a call

Attach tool invocations that fired during a call to an already-existing call, asynchronously after the call was created. Use this when the tool-call data becomes available later than the call itself (e.g. a Roark simulation, or a call submitted via POST /v1/call before its tools were ready). Writes are idempotent: re-sending an invocation already present on the call (same tool name and timing) is skipped, so retries converge instead of double-counting. Optionally pass `metrics` to (re)score metrics over the newly attached tools, which triggers a billed metric collection job and requires the `metric:create` permission.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call tool-invocation append <call-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/call/{callId}/tool-invocations`

## Arguments

**call-id** `string` *required*

The call-id.

## Options

**--tool-invocations** `array` *required*

Tool invocations that fired during the call, to attach to it. Max 500 per request. Re-sending an invocation already present on the call (same tool name and timing) is skipped, so retries are safe.

**--metrics** `array`

Optional. Metric definitions to (re)score on this call after the tool invocations are attached, e.g. the Tool Invocation Analysis metrics. Triggers a metric collection job (billed, credit-gated) and requires the 'metric:create' permission. Omit to attach tools without scoring. Max 20.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call tool-invocation append <call-id> --tool-invocations <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark call transcript get
Source: https://docs.roark.ai/cli-reference/commands/call-transcript-get

Get call transcript

Fetch the full transcript for a specific call. Optionally specify a transcription source; otherwise the best available source is used automatically.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call transcript get <call-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/call/{callId}/transcript`

## Arguments

**call-id** `string` *required*

The call-id.

## Options

**--source** `string`

Transcription source to fetch. When omitted, uses the preferred source based on availability: `CUSTOMER_AGENT_REALTIME` > `SIMULATION_AGENT_REALTIME` > `ROARK_POST_CALL`. One of `ROARK_POST_CALL`, `SIMULATION_AGENT_REALTIME`, `CUSTOMER_AGENT_REALTIME`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call transcript get <call-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark config apply
Source: https://docs.roark.ai/cli-reference/commands/config-apply

Reconcile the project to a config directory (or bundle). Previews and confirms first.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply [dir] [options]
```

## Options

**--no-prune** `switch`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply [dir]
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark config diff
Source: https://docs.roark.ai/cli-reference/commands/config-diff

Preview the changes a config directory (or bundle) would make. No writes.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config diff [dir] [options]
```

## Options

**--no-prune** `switch`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config diff [dir]
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark credential create
Source: https://docs.roark.ai/cli-reference/commands/credential-create

Create a personal credential

Mints a credential that acts as you, for a CLI or an automation. The key value is returned exactly once, in the `key` field: capture it now, it is unreadable afterwards. Requires a personal credential (a project API key is refused) and admin on the project the credential defaults to. The new credential can never exceed the one that created it: not in scope, not in permissions, and not in lifetime. Omit a field to copy it from the calling credential.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark credential create [options]
```

> [!NOTE]
>
> Calls `POST /v1/me/api-keys`

## Options

**--name** `string` *required*

A label you will recognise later. It is the only thing that tells two credentials apart in the list you revoke from.

**--project-id** `string`

Project the credential assumes when a request sends no X-Roark-Project-Id header. Defaults to the calling credential's own default project. You must be an admin of whichever project is used.

**--scopes** `array`

Coarse tier. Defaults to the calling credential's own tier, and can never exceed it: a READ credential cannot mint a WRITE one. One of `READ`, `WRITE` Repeatable.

**--permissions** `array`

Granular 'resource:action' permissions. Defaults to the calling credential's own set, and can never exceed it. A ceiling, not an entitlement: the holder still only reaches what their project membership allows. Repeatable.

**--expires-at** `string`

ISO 8601 expiry. Defaults to the calling credential's own expiry (no expiry, for a `roark auth login` credential) and may not outlive it, so a short-lived connector credential cannot mint a permanent one.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark credential create --name <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark credential list
Source: https://docs.roark.ai/cli-reference/commands/credential-list

List your personal credentials

Returns the credentials that act as you: the ones created in the dashboard, from `roark auth login`, and MCP connectors. Requires a personal credential; a project API key is refused. Defaults to ACTIVE, pass ?status=REVOKED to see revoked ones. The key value itself is never returned.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark credential list [options]
```

> [!NOTE]
>
> Calls `GET /v1/me/api-keys`

## Options

**--status** `string`

Filter by status. Defaults to ACTIVE. One of `ACTIVE`, `REVOKED`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark credential list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark credential revoke
Source: https://docs.roark.ai/cli-reference/commands/credential-revoke

Revoke one of your personal credentials

Revokes a credential that acts as you. It stops working immediately. A credential may revoke itself, which is what `roark auth logout` does. Repeating the call is safe; an unknown id, or one belonging to someone else, answers 404.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark credential revoke <credential-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/me/api-keys/{id}`

## Arguments

**credential-id** `string` *required*

The credential-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark credential revoke <credential-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark customer-flow create
Source: https://docs.roark.ai/cli-reference/commands/customer-flow-create

Create a customer flow

Creates a customer flow. A SCRIPTED flow carries a step graph and gets one way of running it per path through the graph; an IMPROV flow carries the briefs you send. Customer flows replace the older simulation scenarios, so build a flow for anything new.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow create [options]
```

> [!NOTE]
>
> Calls `POST /v1/customer-flow`

## Request body

This endpoint accepts one of several request shapes, so the body is supplied as
JSON through `--data` or on standard input rather than as individual flags.

**CreateScriptedCustomerFlowInput** `object`

Requires `type`, `title`, `graph`

**CreateImprovCustomerFlowInput** `object`

Requires `type`, `title`, `agentIds`, `happyPath`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow create --data '{ ... }'
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark customer-flow delete
Source: https://docs.roark.ai/cli-reference/commands/customer-flow-delete

Delete a customer flow

Soft-deletes a customer flow along with its edge cases, expectations and (for scripted flows) its step graph. Run plans that linked it drop it from their test cases.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow delete <flow-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/customer-flow/{flowId}`

## Arguments

**flow-id** `string` *required*

The flow-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow delete <flow-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark customer-flow duplicate
Source: https://docs.roark.ai/cli-reference/commands/customer-flow-duplicate

Duplicate a customer flow

Deep-copies a flow into a new project-owned flow. The copy carries the source's description, branching mode, linked agents, flow-level expectations and flow-owned metrics. A scripted flow copies its whole step graph; an improv flow copies its variants (personas, briefs, expectations). Duplicating a Roark-managed flow is how you customise it. Voicemail flows are Roark-managed and cannot be duplicated.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow duplicate <flow-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/customer-flow/{flowId}/duplicate`

## Arguments

**flow-id** `string` *required*

The flow-id.

## Options

**--title** `string`

Title for the copy. Defaults to "Copy of" the source flow.

**--agent-ids** `array`

Agents to link on the copy. Omit to carry the source flow's agents over. Required when the source is a Roark-managed flow with no agents of its own. Repeatable.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow duplicate <flow-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark customer-flow edge-case create
Source: https://docs.roark.ai/cli-reference/commands/customer-flow-edge-case-create

Add an edge case

Adds a variant to an IMPROV flow. A scripted flow's variants are owned by the path engine, one per path through the graph, so they are created by editing the graph through PUT /v1/customer-flow/\{flowId}/graph rather than here. Leave personaOverrideId or environmentId unset to inherit the happy path's.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow edge-case create <flow-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/customer-flow/{flowId}/edge-case`

## Arguments

**flow-id** `string` *required*

The flow-id.

## Options

**--title** `string` *required*

**--environment-id** `string`

**--prompt** `string`

**--preceded-by-customer-flow-id** `string`

**--preceded-by-customer-flow-variant-id** `string`

**--include-flow-expectations** `switch`

Whether this edge case is graded on the flow's agentExpectations as well as its own additionalExpectations. Defaults to true; false grades it on its additionalExpectations only.

**--additional-expectations** `array`

The expectations graded on this variant only, on top of the flow's own. Sent as the complete set: it replaces whatever the variant had, so omit it to leave the set alone and send \[] to clear it. Improv flows only: a scripted variant's expectations come from the agent turns on its path and are rewritten on every graph edit.

**--persona-override-id** `string`

The persona this runs as. Omit to inherit the happy path's.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow edge-case create <flow-id> --title <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark customer-flow edge-case delete
Source: https://docs.roark.ai/cli-reference/commands/customer-flow-edge-case-delete

Remove an edge case

Soft-deletes a variant. On a scripted flow the path engine re-creates a variant for any path still in the graph, so remove the path through PUT /graph instead if that is what you meant.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow edge-case delete <flow-id> <edge-case-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/customer-flow/{flowId}/edge-case/{edgeCaseId}`

## Arguments

**flow-id** `string` *required*

The flow-id.

**edge-case-id** `string` *required*

The edge-case-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow edge-case delete <flow-id> <edge-case-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark customer-flow edge-case promote
Source: https://docs.roark.ai/cli-reference/commands/customer-flow-edge-case-promote

Promote an edge case to the happy path

Makes this edge case the flow's happy path, and the outgoing happy path an edge case. Its persona and environment are baked into it first, so edge cases that were inheriting keep the configuration they had.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow edge-case promote <flow-id> <edge-case-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/customer-flow/{flowId}/edge-case/{edgeCaseId}/promote`

## Arguments

**flow-id** `string` *required*

The flow-id.

**edge-case-id** `string` *required*

The edge-case-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow edge-case promote <flow-id> <edge-case-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark customer-flow edge-case update
Source: https://docs.roark.ai/cli-reference/commands/customer-flow-edge-case-update

Update an edge case

Updates an edge case's title, persona, environment, brief, preceded-by link or expectations. Omitted fields are left alone; `additionalExpectations` replaces the set wholesale rather than appending. Promoting it to the happy path is a separate call, since that also demotes the incumbent.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow edge-case update <flow-id> <edge-case-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/customer-flow/{flowId}/edge-case/{edgeCaseId}`

## Arguments

**flow-id** `string` *required*

The flow-id.

**edge-case-id** `string` *required*

The edge-case-id.

## Options

**--title** `string`

**--environment-id** `string`

**--prompt** `string`

**--preceded-by-customer-flow-id** `string`

**--preceded-by-customer-flow-variant-id** `string`

**--include-flow-expectations** `switch`

Whether this edge case is graded on the flow's agentExpectations as well as its own additionalExpectations. Defaults to true; false grades it on its additionalExpectations only.

**--persona-override-id** `string`

The persona this runs as. Null on an edge case inherits the happy path's.

**--additional-expectations** `array`

The expectations graded on this variant only, on top of the flow's own. Sent as the complete set: it replaces whatever the variant had, so omit it to leave the set alone and send \[] to clear it. Improv flows only: a scripted variant's expectations come from the agent turns on its path and are rewritten on every graph edit.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow edge-case update <flow-id> <edge-case-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark customer-flow get
Source: https://docs.roark.ai/cli-reference/commands/customer-flow-get

Get customer flow by ID

Returns a customer flow with its happy path, edge cases, expectations and linked agents. Scripted flows also carry their step graph.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow get <flow-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/customer-flow/{flowId}`

## Arguments

**flow-id** `string` *required*

The flow-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow get <flow-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark customer-flow graph replace
Source: https://docs.roark.ai/cli-reference/commands/customer-flow-graph-replace

Replace a scripted flow's steps

Replaces a scripted flow's conversation graph with the tree you send. This is a full replace, not a merge: a step you omit is removed. Include `nodeId` on a step to update the existing one, omit it to create a new step. Where two branches rejoin, keep the `mergeIntoNodeIds` references a read gave you. Dropping them un-merges those branches and is refused unless `allowUnmerge` is set. A change to the set of paths re-seeds how the flow runs, which the response reports as `variantsReshaped` along with the resulting happy path and edge cases.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow graph replace <flow-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/customer-flow/{flowId}/graph`

## Arguments

**flow-id** `string` *required*

The flow-id.

## Options

**--graph** `array` *required*

The complete graph. This replaces the flow's existing steps rather than merging into them.

**--allow-unmerge** `switch`

Confirms a write that drops branches which currently rejoin. Only needed when the request omits mergeIntoNodeIds references the flow already had; a faithful round trip never needs it.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow graph replace <flow-id> --graph <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark customer-flow happy-path update
Source: https://docs.roark.ai/cli-reference/commands/customer-flow-happy-path-update

Update a flow's happy path

Updates the happy path's title, persona, environment, brief or expectations. Omitted fields are left alone; `additionalExpectations` replaces the set wholesale rather than appending. Its persona and environment are what the edge cases inherit, so changing them here changes every edge case that does not name its own.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow happy-path update <flow-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/customer-flow/{flowId}/happy-path`

## Arguments

**flow-id** `string` *required*

The flow-id.

## Options

**--title** `string`

**--environment-id** `string`

**--prompt** `string`

**--preceded-by-customer-flow-id** `string`

**--preceded-by-customer-flow-variant-id** `string`

**--include-flow-expectations** `switch`

Whether this edge case is graded on the flow's agentExpectations as well as its own additionalExpectations. Defaults to true; false grades it on its additionalExpectations only.

**--persona-override-id** `string`

The persona this runs as. Null on an edge case inherits the happy path's.

**--additional-expectations** `array`

The expectations graded on this variant only, on top of the flow's own. Sent as the complete set: it replaces whatever the variant had, so omit it to leave the set alone and send \[] to clear it. Improv flows only: a scripted variant's expectations come from the agent turns on its path and are rewritten on every graph edit.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow happy-path update <flow-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark customer-flow list
Source: https://docs.roark.ai/cli-reference/commands/customer-flow-list

List customer flows

Returns a paginated list of customer flows with their agents, expectations, happy path and edge cases. The step graph is the one field omitted: reading it walks the project's whole step graph, so it comes back from the single-flow endpoint instead. Customer flows are how a project describes what to test; they replace the older simulation scenarios.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow list [options]
```

> [!NOTE]
>
> Calls `GET /v1/customer-flow`

## Options

**--limit** `integer`

**--after** `string`

**--search-text** `string`

**--type** `string`

One of `SCRIPTED`, `IMPROV`, `VOICEMAIL`

**--include-system** `string`

One of `true`, `false`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark customer-flow update
Source: https://docs.roark.ai/cli-reference/commands/customer-flow-update

Update a customer flow

Updates a flow's title, description, branching mode, linked agents or flow-level expectations. The step graph is replaced through PUT /graph.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow update <flow-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/customer-flow/{flowId}`

## Arguments

**flow-id** `string` *required*

The flow-id.

## Options

**--title** `string`

**--description** `string`

**--script-adherence** `string`

Scripted flows only. How closely a run follows the script. LOOSE (default) hands the whole script to the simulated customer as one prompt; it keeps the call moving whatever your agent says. STRICT runs the script as a state machine on the agent service: at every agent step the simulated customer waits, silent, until your agent has said the expected line, and only then moves on. Scripted flows only; STRICT needs the agent-service transport and is not available on realtime models. One of `LOOSE`, `STRICT`

**--off-script-policy** `object`

STRICT only. What the simulated customer does when your agent does not say the expected line. Each unmatched agent utterance is an attempt: `reaction` runs per attempt (`STAY_SILENT`, REPEAT its last scripted line, RESPOND once in character without moving on, or SAY `sayLine`), and `then` runs when attempts reach `maxAttempts` or your agent stays silent for `waitSeconds` (`HANG_UP` ends the call with ended reason `SCRIPT_DIVERGED`, `HANG_UP_INVALIDATE` ends it the same way and invalidates the run so it is scored by nothing and counted nowhere, `MOVE_ON` advances anyway, ADAPT hands the rest of the call to loose behaviour). Null: stay silent, 3 attempts, hang up. The default for every agent step; an `AGENT_TURN` step can carry its own.

**--off-script-policy.reaction** `string` *required*

One of `STAY_SILENT`, `REPEAT`, `RESPOND`, `SAY`

**--off-script-policy.say-line** `string`

**--off-script-policy.max-attempts** `integer` *required*

**--off-script-policy.then** `string` *required*

One of `HANG_UP`, `MOVE_ON`, `ADAPT`, `HANG_UP_INVALIDATE`

**--off-script-policy.wait-seconds** `integer`

**--branching-mode** `string`

Scripted flows only. How a run walks the graph. DETERMINISTIC ("Simulate every path" in the app) places one call per variant, each following its path exactly whatever the agent says. ADAPTIVE ("Adapt to your agent") collapses the paths into one call PER PERSONA, on which the simulated customer picks a branch from what the agent actually said. Both modes speak the exact authored lines, and neither changes how metrics or expectations grade. One of `DETERMINISTIC`, `ADAPTIVE`

**--agent-ids** `array`

Replaces the linked agents. Omit to leave them unchanged. An improv flow must keep at least one; a scripted flow can be left with none. Repeatable.

**--agent-expectations** `array`

Replaces the flow-level expectations. Omit to leave them unchanged.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark customer-flow update <flow-id> --off-script-policy.reaction <value> --off-script-policy.max-attempts <value> --off-script-policy.then <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark dashboard create
Source: https://docs.roark.ai/cli-reference/commands/dashboard-create

Create a dashboard

Creates a dashboard with its widgets. Each widget is an analytics query (see POST /v1/analytics/query) with a name unique on the dashboard, a `display` (LINE, NUMBER, RADAR, `STACKED_BAR`, TABLE) and a `size` (SMALL, MEDIUM, LARGE, `EXTRA_LARGE`). Widgets render top to bottom in the order given. Dashboard `filters` apply over every widget and must be conversation-level: `AGENT_PROPERTY`, `CALL_PROPERTY`, MODALITY, DATASET or `METRIC_VALUE`. On a field a widget already filters, the dashboard's wins.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard create [options]
```

> [!NOTE]
>
> Calls `POST /v1/dashboard`

## Options

**--name** `string` *required*

**--description** `string`

**--time-range** `string`

The window: LAST n DAYS/WEEKS/MONTHS up to now, a `CALENDAR_ALIGNED` period (TODAY, YESTERDAY, `THIS_WEEK`, `LAST_WEEK`, `THIS_MONTH`, `LAST_MONTH`, `THIS_YEAR`, `LAST_YEAR`), FIXED from/to, or SINCE a date. At most 731 days (two years).

**--filters** `array`

**--is-default** `switch`

**--widgets** `array`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard create --name <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark dashboard data get
Source: https://docs.roark.ai/cli-reference/commands/dashboard-data-get

Get dashboard data

Runs every widget on the dashboard and returns each one's series, exactly as the dashboard shows them: over the dashboard's saved window and with its filters applied. Pass `from` (and optionally `to`) to read a different window, and `timezone` to align buckets to a zone.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard data get <dashboard-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/dashboard/{dashboardId}/data`

## Arguments

**dashboard-id** `string` *required*

The dashboard-id.

## Options

**--from** `string`

ISO 8601 start. Overrides the saved window; without `to`, runs until now.

**--to** `string`

ISO 8601 end. Requires `from`.

**--timezone** `string`

IANA zone buckets and calendar periods align to. Default UTC.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard data get <dashboard-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark dashboard delete
Source: https://docs.roark.ai/cli-reference/commands/dashboard-delete

Delete a dashboard

Deletes a dashboard and its widgets. If it was the default, another dashboard becomes the default. A dashboard managed by config as code returns 409.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard delete <dashboard-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/dashboard/{dashboardId}`

## Arguments

**dashboard-id** `string` *required*

The dashboard ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard delete <dashboard-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark dashboard get
Source: https://docs.roark.ai/cli-reference/commands/dashboard-get

Get a dashboard

Returns a dashboard with its widgets top to bottom, each in the shape POST /v1/dashboard accepts, so a dashboard can be read, edited and written back.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard get <dashboard-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/dashboard/{dashboardId}`

## Arguments

**dashboard-id** `string` *required*

The dashboard ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard get <dashboard-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark dashboard list
Source: https://docs.roark.ai/cli-reference/commands/dashboard-list

List dashboards

Lists the project's dashboards, the default first. GET /v1/dashboard/\{dashboardId} returns the widgets.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard list [options]
```

> [!NOTE]
>
> Calls `GET /v1/dashboard`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark dashboard update
Source: https://docs.roark.ai/cli-reference/commands/dashboard-update

Update a dashboard

Edits a dashboard. Omitted fields keep their current value. `widgets`, when sent, replaces the list: widgets are matched by name, so a widget that keeps its name keeps its id; a new name adds a widget and a name no longer listed removes it. To change one widget, PUT /v1/dashboard/\{dashboardId}/widget/\{widgetId}.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard update <dashboard-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/dashboard/{dashboardId}`

## Arguments

**dashboard-id** `string` *required*

The dashboard ID.

## Options

**--name** `string`

**--description** `string`

**--time-range** `string`

The window: LAST n DAYS/WEEKS/MONTHS up to now, a `CALENDAR_ALIGNED` period (TODAY, YESTERDAY, `THIS_WEEK`, `LAST_WEEK`, `THIS_MONTH`, `LAST_MONTH`, `THIS_YEAR`, `LAST_YEAR`), FIXED from/to, or SINCE a date. At most 731 days (two years).

**--filters** `array`

**--is-default** `switch`

**--widgets** `array`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard update <dashboard-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark dashboard widget create
Source: https://docs.roark.ai/cli-reference/commands/dashboard-widget-create

Add a widget

Adds a widget to the top of a dashboard. Its name must be unique on the dashboard.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard widget create <dashboard-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/dashboard/{dashboardId}/widget`

## Arguments

**dashboard-id** `string` *required*

The dashboard ID.

## Options

**--measures** `array` *required*

What to aggregate: 1 to 10 measures.

**--filters** `array`

Applied to every measure, on top of each measure's own filters.

**--interval** `string`

Bucket size. Defaults from the window's length and steps coarser when the window would exceed 500 buckets. One of `HOUR`, `DAY`, `WEEK`, `MONTH`

**--comparison-mode** `string`

`PREVIOUS_PERIOD` adds the preceding window of the same length to every series. One of `NONE`, `PREVIOUS_PERIOD`

**--comparison-groups** `array`

Run every measure once per named filter set, e.g. one group per agent.

**--name** `string` *required*

Unique on the dashboard: how the widget is matched on update.

**--description** `string`

**--display** `string`

Chart type. Defaults to LINE. One of `LINE`, `NUMBER`, `RADAR`, `STACKED_BAR`, `TABLE`

**--size** `string`

Grid width. Defaults to MEDIUM. One of `SMALL`, `MEDIUM`, `LARGE`, `EXTRA_LARGE`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard widget create <dashboard-id> --measures <value> --name <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark dashboard widget-data get
Source: https://docs.roark.ai/cli-reference/commands/dashboard-widget-data-get

Get widget data

Runs one widget and returns its series, as it reads on its dashboard (the dashboard's window and filters). Accepts the same `from` / `to` / `timezone` overrides as GET /v1/dashboard/\{dashboardId}/data.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard widget-data get <dashboard-id> <widget-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/dashboard/{dashboardId}/widget/{widgetId}/data`

## Arguments

**dashboard-id** `string` *required*

The dashboard-id.

**widget-id** `string` *required*

The widget-id.

## Options

**--from** `string`

ISO 8601 start. Overrides the saved window; without `to`, runs until now.

**--to** `string`

ISO 8601 end. Requires `from`.

**--timezone** `string`

IANA zone buckets and calendar periods align to. Default UTC.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard widget-data get <dashboard-id> <widget-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark dashboard widget delete
Source: https://docs.roark.ai/cli-reference/commands/dashboard-widget-delete

Remove a widget

Removes a widget from a dashboard.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard widget delete <dashboard-id> <widget-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/dashboard/{dashboardId}/widget/{widgetId}`

## Arguments

**dashboard-id** `string` *required*

The dashboard ID.

**widget-id** `string` *required*

The widget ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard widget delete <dashboard-id> <widget-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark dashboard widget update
Source: https://docs.roark.ai/cli-reference/commands/dashboard-widget-update

Replace a widget

Replaces a widget's definition. It keeps its id and its place on the dashboard.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard widget update <dashboard-id> <widget-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/dashboard/{dashboardId}/widget/{widgetId}`

## Arguments

**dashboard-id** `string` *required*

The dashboard ID.

**widget-id** `string` *required*

The widget ID.

## Options

**--measures** `array` *required*

What to aggregate: 1 to 10 measures.

**--filters** `array`

Applied to every measure, on top of each measure's own filters.

**--interval** `string`

Bucket size. Defaults from the window's length and steps coarser when the window would exceed 500 buckets. One of `HOUR`, `DAY`, `WEEK`, `MONTH`

**--comparison-mode** `string`

`PREVIOUS_PERIOD` adds the preceding window of the same length to every series. One of `NONE`, `PREVIOUS_PERIOD`

**--comparison-groups** `array`

Run every measure once per named filter set, e.g. one group per agent.

**--name** `string` *required*

Unique on the dashboard: how the widget is matched on update.

**--description** `string`

**--display** `string`

Chart type. Defaults to LINE. One of `LINE`, `NUMBER`, `RADAR`, `STACKED_BAR`, `TABLE`

**--size** `string`

Grid width. Defaults to MEDIUM. One of `SMALL`, `MEDIUM`, `LARGE`, `EXTRA_LARGE`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard widget update <dashboard-id> <widget-id> --measures <value> --name <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark health
Source: https://docs.roark.ai/cli-reference/commands/health

Get API health status

Returns the health status of the API and its dependencies

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark health [options]
```

> [!NOTE]
>
> Calls `GET /health`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark health
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark http-request-definition create
Source: https://docs.roark.ai/cli-reference/commands/http-request-definition-create

Create HTTP request definition

Creates a new HTTP request definition. The signing secret is only returned in this response and cannot be retrieved later.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark http-request-definition create [options]
```

> [!NOTE]
>
> Calls `POST /v1/http-request-definition`

## Options

**--scope** `string` *required*

Scope: `AGENT_OUTBOUND_DIAL`. One of `AGENT_OUTBOUND_DIAL`

**--url** `string` *required*

URL for the HTTP request.

**--description** `string`

Description of the HTTP request definition.

**--method** `string`

HTTP method (default: POST). One of `POST`, `PUT`, `PATCH`, `GET`

**--body** `string`

Request body template. Accepts a JSON object or a string with placeholders like \{\{phoneNumberToDial}}. Objects are serialized to JSON for storage.

**--headers** `object`

Request headers as key-value pairs.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark http-request-definition create --scope <value> --url <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark http-request-definition get
Source: https://docs.roark.ai/cli-reference/commands/http-request-definition-get

Get HTTP request definition by ID

Returns a specific HTTP request definition by its ID.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark http-request-definition get <definition-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/http-request-definition/{definitionId}`

## Arguments

**definition-id** `string` *required*

The ID of the HTTP request definition to retrieve.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark http-request-definition get <definition-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark http-request-definition list
Source: https://docs.roark.ai/cli-reference/commands/http-request-definition-list

List HTTP request definitions

Returns a paginated list of HTTP request definitions for the authenticated project.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark http-request-definition list [options]
```

> [!NOTE]
>
> Calls `GET /v1/http-request-definition`

## Options

**--limit** `integer`

Maximum number of definitions to return (default: 20, max: 50).

**--after** `string`

Cursor for pagination - definition ID to start after.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark http-request-definition list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark http-request-definition update
Source: https://docs.roark.ai/cli-reference/commands/http-request-definition-update

Update HTTP request definition

Updates an existing HTTP request definition.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark http-request-definition update <definition-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/http-request-definition/{definitionId}`

## Arguments

**definition-id** `string` *required*

The ID of the HTTP request definition to update.

## Options

**--url** `string`

URL for the HTTP request.

**--description** `string`

Description of the HTTP request definition.

**--method** `string`

HTTP method: POST, PUT, PATCH, or GET. One of `POST`, `PUT`, `PATCH`, `GET`

**--body** `string`

Request body template. Accepts a JSON object or a string with placeholders like \{\{phoneNumberToDial}}. Objects are serialized to JSON for storage.

**--headers** `object`

Request headers as key-value pairs.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark http-request-definition update <definition-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark live-report create
Source: https://docs.roark.ai/cli-reference/commands/live-report-create

Create a live report

Saves a report on the project's live calls. Simulations are never included. Targets are analytics measures held to a bar: a threshold metric is read as its pass rate (`value` is a percentage, 0-100, held `AT_LEAST` by default); a numeric metric as its average (held `AT_MOST` by default). Every target is read for the period and the period before it, so the report says whether each was met and how it moved. Scheduled reports are written `deliveryHour` hours into the day after each period ends in `timezone` (a week runs Monday to Sunday), then emailed and posted per `notifications`. With `generateNow` (the default) the report for the latest closed period is written right away and not sent.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report create [options]
```

> [!NOTE]
>
> Calls `POST /v1/live-report`

## Options

**--name** `string` *required*

**--description** `string`

**--agent-id** `string`

**--filters** `array`

**--targets** `array`

**--dashboard-id** `string`

**--cadence** `string`

One of `DAILY`, `WEEKLY`, `MONTHLY`

**--timezone** `string`

**--delivery-hour** `integer`

**--status** `string`

One of `ACTIVE`, `PAUSED`

**--notifications** `object`

**--notifications.email** `object` *required*

**--notifications.slack** `array` *required*

**--include-transcript-excerpts** `switch`

**--generate-now** `switch`

Write the report for the latest closed period right away. Default true.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report create --name <value> --notifications.email <value> --notifications.slack <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark live-report delete
Source: https://docs.roark.ai/cli-reference/commands/live-report-delete

Delete a live report

Deletes the report and stops its scheduled reports.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report delete <report-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/live-report/{reportId}`

## Arguments

**report-id** `string` *required*

The live report ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report delete <report-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark live-report document get
Source: https://docs.roark.ai/cli-reference/commands/live-report-document-get

Get a report document

Returns one written report. `document` is null until `status` is READY. Theme excerpts and grader quotes (call transcript text) are included only for keys with transcript:read.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report document get <report-id> <document-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/live-report/{reportId}/document/{documentId}`

## Arguments

**report-id** `string` *required*

The live report ID.

**document-id** `string` *required*

The report document ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report document get <report-id> <document-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark live-report document list
Source: https://docs.roark.ai/cli-reference/commands/live-report-document-list

List report documents

Lists a report's documents, newest period first.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report document list <report-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/live-report/{reportId}/document`

## Arguments

**report-id** `string` *required*

The report-id.

## Options

**--limit** `integer`

Page size, at most 100.

**--offset** `integer`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report document list <report-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark live-report get
Source: https://docs.roark.ai/cli-reference/commands/live-report-get

Get a live report

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report get <report-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/live-report/{reportId}`

## Arguments

**report-id** `string` *required*

The live report ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report get <report-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark live-report list
Source: https://docs.roark.ai/cli-reference/commands/live-report-list

List live reports

Lists the project's saved live call reports, newest first.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report list [options]
```

> [!NOTE]
>
> Calls `GET /v1/live-report`

## Options

**--limit** `integer`

Page size, at most 100.

**--offset** `integer`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark live-report run
Source: https://docs.roark.ai/cli-reference/commands/live-report-run

Generate a live report document

Writes the report now, for its latest closed period or a custom window (`periodStart` + `periodEnd`, at most 92 days, ended). Returns 202 with the document in PENDING; poll GET /v1/live-report/\{reportId}/document/\{documentId} until `status` is READY or FAILED (typically a few minutes). An identical request already in flight is returned as-is. A generated document is not emailed.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report run <report-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/live-report/{reportId}/generate`

## Arguments

**report-id** `string` *required*

The live report ID.

## Options

**--period-start** `string`

ISO 8601 start of a custom window. Omit both for the latest closed period.

**--period-end** `string`

ISO 8601 end (exclusive) of a custom window. Requires periodStart.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report run <report-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark live-report update
Source: https://docs.roark.ai/cli-reference/commands/live-report-update

Update a live report

Changes the fields sent. Set `status` to PAUSED to stop scheduled reports.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report update <report-id> [options]
```

> [!NOTE]
>
> Calls `PATCH /v1/live-report/{reportId}`

## Arguments

**report-id** `string` *required*

The live report ID.

## Options

**--name** `string`

**--description** `string`

**--agent-id** `string`

**--filters** `array`

**--targets** `array`

**--dashboard-id** `string`

**--cadence** `string`

One of `DAILY`, `WEEKLY`, `MONTHLY`

**--timezone** `string`

**--delivery-hour** `integer`

**--status** `string`

One of `ACTIVE`, `PAUSED`

**--notifications** `object`

**--notifications.email** `object` *required*

**--notifications.slack** `array` *required*

**--include-transcript-excerpts** `switch`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark live-report update <report-id> --notifications.email <value> --notifications.slack <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric collection-job cancel
Source: https://docs.roark.ai/cli-reference/commands/metric-collection-job-cancel

Cancel a metric collection job

Stops a metric collection job that has not finished. No further conversations are processed and the job ends in CANCELED. Items the job had not reached are marked SKIPPED. Metric values already produced are kept, and items that were mid-computation when the cancel arrived may still complete. Only the items that completed are billed. Safe to call more than once: a job that has already finished (COMPLETED, FAILED, or CANCELED) is returned unchanged.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric collection-job cancel <job-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/metric/collection-jobs/{jobId}/cancel`

## Arguments

**job-id** `string` *required*

The ID of the metric collection job to cancel.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric collection-job cancel <job-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric collection-job create
Source: https://docs.roark.ai/cli-reference/commands/metric-collection-job-create

Create and run a metric collection job

Creates a metric collection job for the specified calls or chats and metrics, then triggers processing. Provide exactly one of callIds or chatIds. Pass policyId to score with a metric policy’s settings (for example to re-score past calls after changing which properties Property Verification checks); metrics then defaults to every metric that policy collects.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric collection-job create [options]
```

> [!NOTE]
>
> Calls `POST /v1/metric/collection-jobs`

## Options

**--call-ids** `array`

Call IDs to collect metrics for. Mutually exclusive with chatIds. Max 500 per request. Repeatable.

**--chat-ids** `array`

Chat IDs to collect metrics for. Mutually exclusive with callIds. Max 500 per request. Repeatable.

**--metrics** `array`

Metric definitions to collect. Max 20 per request. Required unless policyId is given; with policyId, defaults to every metric that policy collects, and each one listed must be in it.

**--policy-id** `string`

Optional. A metric policy (collector) whose settings to score with: each metric runs at the version that policy runs it at, with its per-collector setting (such as the properties Property Verification checks), instead of the metric's default. A metric the policy runs at its default runs at its default here too. The policy's conditions are not applied: the listed conversations are scored as given. Use it to re-score past conversations after changing a policy's settings.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric collection-job create
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric collection-job get
Source: https://docs.roark.ai/cli-reference/commands/metric-collection-job-get

Get metric collection job by ID

Returns a specific metric collection job with progress information.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric collection-job get <job-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/metric/collection-jobs/{jobId}`

## Arguments

**job-id** `string` *required*

The ID of the metric collection job to retrieve.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric collection-job get <job-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric collection-job list
Source: https://docs.roark.ai/cli-reference/commands/metric-collection-job-list

List metric collection jobs

Returns a paginated list of metric collection jobs for the project.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric collection-job list [options]
```

> [!NOTE]
>
> Calls `GET /v1/metric/collection-jobs`

## Options

**--limit** `integer`

Maximum number of jobs to return (default: 20, max: 50).

**--after** `string`

Cursor for pagination - use the nextCursor value from a previous response.

**--status** `string`

Filter by job status. One of `PENDING`, `PROCESSING`, `COMPLETED`, `FAILED`, `CANCELED`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric collection-job list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric definition create
Source: https://docs.roark.ai/cli-reference/commands/metric-definition-create

Create custom metric definition

Create a new metric definition. The `calculationType` field selects the variant: `LLM_JUDGE` (LLM-evaluated), FORMULA (computed from a math expression over other metrics), or PATTERN (detects a trigger→outcome pattern within a window). To create a threshold on top of an existing metric, use `POST /metric/definitions/{idOrSlug}/thresholds` instead.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric definition create [options]
```

> [!NOTE]
>
> Calls `POST /v1/metric/definitions`

## Request body

This endpoint accepts one of several request shapes, so the body is supplied as
JSON through `--data` or on standard input rather than as individual flags.

**PromptMetricInput** `object`

Requires `calculationType`, `name`, `outputType`

**FormulaMetricInput** `object`

Requires `calculationType`, `name`, `outputType`, `formula`, `sources`

**PatternMetricInput** `object`

Requires `calculationType`, `name`, `operation`, `outcome`

**HttpRequestMetricInput** `object`

Requires `calculationType`, `name`, `httpRequest`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric definition create --data '{ ... }'
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric definition list
Source: https://docs.roark.ai/cli-reference/commands/metric-definition-list

List metric definitions

Fetch metric definitions available in the project, including both system-generated and custom metrics. Results are ordered by immutable definition ID; pass `nextCursor` to retrieve the following page.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric definition list [options]
```

> [!NOTE]
>
> Calls `GET /v1/metric/definitions`

## Options

**--limit** `integer`

**--after** `string`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric definition list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric definition update
Source: https://docs.roark.ai/cli-reference/commands/metric-definition-update

Update a metric definition

Update the editable subset of a custom metric definition, addressed by its UUID or its stable `slug`. Only the supplied fields are changed; omitted fields are left unchanged. Every update creates a new immutable version; the response carries the advanced `versionId`. Immutable fields (scope, outputType, calcType, …) are rejected, and which fields are editable depends on the metric (e.g. derived metrics only allow `name`). Roark's own metrics are rejected here: this endpoint edits the shared definition, which every workspace sees. To change one for your workspace alone, edit its variant with PUT /v1/metric/definitions/\{idOrSlug}/variants/\{variantId}, which forks it for you and leaves every other workspace on the original.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric definition update <id-or-slug> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/metric/definitions/{idOrSlug}`

## Arguments

**id-or-slug** `string` *required*

The metric definition's Roark ID (UUID) or its stable `slug` (e.g. `customer_satisfaction`).

## Options

**--name** `string`

New name (only for metrics whose name is editable).

**--llm-prompt** `string`

New LLM prompt (only for `LLM_JUDGE` metrics whose prompt is editable).

**--scale-min** `integer`

New scale minimum (SCALE output only).

**--scale-max** `integer`

New scale maximum (SCALE output only).

**--scale-labels** `array`

Replacement set of scale-range labels (SCALE output only).

**--boolean-true-label** `string`

New label for the true case (BOOLEAN output only).

**--boolean-false-label** `string`

New label for the false case (BOOLEAN output only).

**--grades-short-calls** `switch`

Score this metric on a simulated call with fewer than 4 transcript lines. Such calls skip every other metric. Default false.

**--classification-options** `array`

Replacement set of classification options (CLASSIFICATION output only).

**--max-classifications** `integer`

New maximum number of classifications (CLASSIFICATION output only).

**--tool-definition-ids** `array`

Replacement set of scoped tool-definition ids (only for metrics whose tool scoping is editable). Repeatable.

**--supported-contexts** `array`

Replacement set of supported contexts. Omit to leave unchanged. Scope and participantRole cannot change after create, so moving to SEGMENT or TURN grading only works on a `PER_PARTICIPANT` metric (see create). One of `CALL`, `SEGMENT`, `TURN` Repeatable.

**--formula** `string`

New formula expression (FORMULA only). Pass `sources` alongside if the referenced metrics change.

**--sources** `array`

Replacement formula sources, required when `formula` changes the referenced metrics (FORMULA only).

**--http-request** `object`

Replacement request (`HTTP_REQUEST` only). Send the whole request; omitted headers are removed.

**--http-request.url** `string` *required*

Public http(s) URL. May contain \{\{path}} placeholders into the request payload, which are URL-encoded.

**--http-request.method** `string`

GET or POST (default POST). A GET request has no body. One of `GET`, `POST`

**--http-request.headers** `object`

Request headers, for example your own Authorization header. Values may contain placeholders.

**--http-request.body** `string`

A JSON template with \{\{path}} placeholders into the request payload, such as \{"cardId": "\{\{call.properties.cardId}}"}. Omit or pass null to send the default payload.

**--change-reason** `string`

Optional free-text audit note recorded on the new version.

**--slug** `string`

**--metric-id** `string`

**--source** `string`

**--organization-id** `string`

**--project-id** `string`

**--output-type** `string`

**--scope** `string`

**--participant-role** `string`

**--calc-type** `string`

**--supports-multiple-variants** `string`

**--analysis-package-id** `string`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric definition update <id-or-slug> --http-request.url <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric policy create
Source: https://docs.roark.ai/cli-reference/commands/metric-policy-create

Create a metric policy

Creates a new metric policy. Policies define which metrics to collect and under what conditions.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric policy create [options]
```

> [!NOTE]
>
> Calls `POST /v1/metric/policies`

## Options

**--name** `string` *required*

Name of the metric policy.

**--status** `string`

Status of the policy (default: ACTIVE). One of `ACTIVE`, `INACTIVE`

**--modality** `string` *required*

Modality this policy targets. A policy fires for exactly one modality and can only reference metrics that support that modality. One of `call`, `chat`

**--conditions** `array`

Condition groups. Omit to match all calls.

**--metrics** `array` *required*

Metric definitions to collect when this policy matches, each with an optional setting.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric policy create --name <value> --modality <value> --metrics <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric policy delete
Source: https://docs.roark.ai/cli-reference/commands/metric-policy-delete

Delete a metric policy

Soft-deletes a metric policy. System policies cannot be deleted.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric policy delete <policy-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/metric/policies/{policyId}`

## Arguments

**policy-id** `string` *required*

The ID of the metric policy to delete.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric policy delete <policy-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric policy get
Source: https://docs.roark.ai/cli-reference/commands/metric-policy-get

Get metric policy by ID

Returns a specific metric policy with its conditions and metrics.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric policy get <policy-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/metric/policies/{policyId}`

## Arguments

**policy-id** `string` *required*

The ID of the metric policy to retrieve.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric policy get <policy-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric policy list
Source: https://docs.roark.ai/cli-reference/commands/metric-policy-list

List metric policies

Returns a paginated list of metric policies for the project, including system policies.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric policy list [options]
```

> [!NOTE]
>
> Calls `GET /v1/metric/policies`

## Options

**--limit** `integer`

Maximum number of policies to return (default: 20, max: 50).

**--after** `string`

Cursor for pagination - use the nextCursor value from a previous response.

**--status** `string`

Filter by policy status. One of `ACTIVE`, `INACTIVE`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric policy list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric policy update
Source: https://docs.roark.ai/cli-reference/commands/metric-policy-update

Update a metric policy

Updates an existing metric policy. System policies cannot be modified.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric policy update <policy-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/metric/policies/{policyId}`

## Arguments

**policy-id** `string` *required*

The ID of the metric policy to update.

## Options

**--name** `string`

Name of the metric policy.

**--status** `string`

Status of the policy. One of `ACTIVE`, `INACTIVE`

**--conditions** `array`

Condition groups. Omit to keep existing, provide empty array to remove all conditions.

**--metrics** `array`

Metric definitions to collect when this policy matches. Replaces the policy's whole metric list: a metric sent without settings goes back to its default, so resend a metric's settings to keep them. Omit to keep the current metrics.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric policy update <policy-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric variant create
Source: https://docs.roark.ai/cli-reference/commands/metric-variant-create

Create a metric variant

Add a configuration of this metric for your organization, seeded from its Default. Edit it with PUT to change what it measures, then pin it where you want it used. Threshold metrics have no variants: their configuration comes from the metric they derive from. Metrics in a package that manages its own variants reject this too.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric variant create <id-or-slug> [options]
```

> [!NOTE]
>
> Calls `POST /v1/metric/definitions/{idOrSlug}/variants`

## Arguments

**id-or-slug** `string` *required*

Metric definition UUID or its stable slug.

## Options

**--name** `string` *required*

Name for the new variant. Must be unique for this metric within your organization and cannot be `Default`.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric variant create <id-or-slug> --name <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric variant delete
Source: https://docs.roark.ai/cli-reference/commands/metric-variant-delete

Delete a metric variant

Remove one of your organization’s variants. Anything pinned to it falls back to the Default, so deleting a fork of a Roark variant returns you to Roark’s configuration. Roark’s own variants cannot be deleted, and neither can a Default. Values already collected under the deleted variant are retained.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric variant delete <id-or-slug> <variant-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/metric/definitions/{idOrSlug}/variants/{variantId}`

## Arguments

**id-or-slug** `string` *required*

Metric definition UUID or its stable slug.

**variant-id** `string` *required*

Variant UUID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric variant delete <id-or-slug> <variant-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric variant get
Source: https://docs.roark.ai/cli-reference/commands/metric-variant-get

Get a metric variant

One configuration of this metric, by id.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric variant get <id-or-slug> <variant-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/metric/definitions/{idOrSlug}/variants/{variantId}`

## Arguments

**id-or-slug** `string` *required*

Metric definition UUID or its stable slug.

**variant-id** `string` *required*

Variant UUID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric variant get <id-or-slug> <variant-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric variant list
Source: https://docs.roark.ai/cli-reference/commands/metric-variant-list

List a metric’s variants

Every configuration of this metric your organization can use: Roark’s own variants and any your organization has added or forked. `isDefault` marks the one the metric is scored with when nothing pins another; pass any variant’s `id` as `sourceVariantId` to pin it on a derived metric. Auto-managed variants (the ones a package materializes for you) are not listed: they are engine state, not configuration you author.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric variant list <id-or-slug> [options]
```

> [!NOTE]
>
> Calls `GET /v1/metric/definitions/{idOrSlug}/variants`

## Arguments

**id-or-slug** `string` *required*

Metric definition UUID or its stable slug.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric variant list <id-or-slug>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark metric variant update
Source: https://docs.roark.ai/cli-reference/commands/metric-variant-update

Update a metric variant

Rename a variant, change its configuration, or both. Every configuration change creates a new immutable version and advances `versionId`; the response carries the advanced value. \*\*Editing one of Roark’s own variants forks it for your organization.\*\* The response carries the new variant’s `id`, which will differ from the one in the path, and `isSystem` becomes false. Roark’s variant is untouched and other organizations keep it. DELETE your fork to go back to it. Which fields are editable depends on the metric: some Roark metrics lock their prompt or output configuration, and a locked field is rejected rather than ignored.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric variant update <id-or-slug> <variant-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/metric/definitions/{idOrSlug}/variants/{variantId}`

## Arguments

**id-or-slug** `string` *required*

Metric definition UUID or its stable slug.

**variant-id** `string` *required*

Variant UUID.

## Options

**--name** `string`

Rename the variant. Does not change its configuration, so `versionId` is unaffected. `Default` is reserved.

**--llm-prompt** `string`

The rubric this variant applies. LLM judge metrics only.

**--scale-min** `integer`

Scale minimum. SCALE output only.

**--scale-max** `integer`

Scale maximum. SCALE output only.

**--boolean-true-label** `string`

What a `true` value means. Given to the judge as its polarity rule, so keep it accurate.

**--boolean-false-label** `string`

What a `false` value means. Given to the judge as its polarity rule, so keep it accurate.

**--grades-short-calls** `switch`

Score this metric on a simulated call with fewer than 4 transcript lines. Such calls skip every other metric. Default false.

**--max-classifications** `integer`

Maximum classifications returned. CLASSIFICATION output only.

**--change-reason** `string`

Free-text audit note recorded on the new version.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark metric variant update <id-or-slug> <variant-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark phone-number allowance get
Source: https://docs.roark.ai/cli-reference/commands/phone-number-allowance-get

Get the phone number allowance

Returns how many numbers the organization gets for free, how many it holds, what a paid number costs a month, and whether a purchase would be admitted right now.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark phone-number allowance get [options]
```

> [!NOTE]
>
> Calls `GET /v1/phone-number/allowance`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark phone-number allowance get
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark phone-number available list
Source: https://docs.roark.ai/cli-reference/commands/phone-number-available-list

Search available phone numbers

Returns US or Canadian numbers the carrier has for sale, local or toll-free, narrowed by area code or by digits the number must contain. When an area code has no numbers, nearby ones are returned flagged bestEffort; an empty list means nothing matched. Buy one with POST /v1/phone-number, or let Roark pick with autoPick.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark phone-number available list [options]
```

> [!NOTE]
>
> Calls `GET /v1/phone-number/available`

## Options

**--country** `string`

ISO 3166-1 alpha-2 country of the number: US or CA. One of `US`, `CA`

**--number-type** `string`

LOCAL for a number in an area code, `TOLL_FREE` for one from the shared US and Canada toll-free pool. One of `LOCAL`, `TOLL_FREE`

**--area-code** `string`

Three-digit area code, or a toll-free prefix such as 800. When it has no numbers, nearby ones are returned instead, flagged bestEffort.

**--contains** `string`

Digits or letters the number must contain (1 to 7, \* matches any digit).

**--limit** `integer`

Maximum number of results (default: 20, max: 100).

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark phone-number available list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark phone-number create
Source: https://docs.roark.ai/cli-reference/commands/phone-number-create

Buy a phone number

Buys a number from the carrier for this project: the one named by e164, or one Roark picks to match autoPick (trying the next match if the carrier has just sold one). Free inside the plan allowance; past it, the first month of rent is charged now and monthly after that.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark phone-number create [options]
```

> [!NOTE]
>
> Calls `POST /v1/phone-number`

## Options

**--e164** `string`

The number to buy, in E.164 format, from a search result. Give this or autoPick.

**--auto-pick** `object`

What the number Roark picks should look like. Roark buys the first matching number the carrier will sell, preferring the area code and falling back to a nearby one.

**--auto-pick.country** `string`

ISO 3166-1 alpha-2 country of the number: US or CA. One of `US`, `CA`

**--auto-pick.number-type** `string`

LOCAL for a number in an area code, `TOLL_FREE` for one from the shared US and Canada toll-free pool. One of `LOCAL`, `TOLL_FREE`

**--auto-pick.area-code** `string`

Three-digit area code, or a toll-free prefix such as 800. When it has no numbers, nearby ones are returned instead, flagged bestEffort.

**--auto-pick.contains** `string`

Digits or letters the number must contain (1 to 7, \* matches any digit).

**--country** `string`

The country of the number given by e164 (default: US). An auto-picked number uses autoPick.country. One of `US`, `CA`

**--name** `string`

A label for the number.

**--agent-id** `string`

The agent to assign the number to.

**--expect-free** `switch`

Whether you expect this number to be free (from the allowance). The purchase is refused with 409 if that has changed since you checked.

**--expect-monthly-rent-micros** `integer`

The monthly rent you expect for a paid number, in micros (from the allowance). The purchase is refused with 409 if that has changed since you checked.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark phone-number create
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark phone-number delete
Source: https://docs.roark.ai/cli-reference/commands/phone-number-delete

Release a phone number

Gives the number back to the carrier. This cannot be undone: rent stops today and nothing is refunded.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark phone-number delete <phone-number-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/phone-number/{phoneNumberId}`

## Arguments

**phone-number-id** `string` *required*

The phone-number-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark phone-number delete <phone-number-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark phone-number list
Source: https://docs.roark.ai/cli-reference/commands/phone-number-list

List phone numbers

Returns the phone numbers the project owns, oldest first, with the agent each is assigned to.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark phone-number list [options]
```

> [!NOTE]
>
> Calls `GET /v1/phone-number`

## Options

**--agent-id** `string`

Only the numbers assigned to this agent.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark phone-number list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark phone-number update
Source: https://docs.roark.ai/cli-reference/commands/phone-number-update

Update a phone number

Relabels a number or changes which agent it is assigned to. A field left out is untouched; null clears it.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark phone-number update <phone-number-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/phone-number/{phoneNumberId}`

## Arguments

**phone-number-id** `string` *required*

The phone-number-id.

## Options

**--name** `string`

New label; null clears it, omit to leave it alone.

**--agent-id** `string`

Agent to assign the number to; null unassigns, omit to leave it alone.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark phone-number update <phone-number-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark project list
Source: https://docs.roark.ai/cli-reference/commands/project-list

List projects available to this credential

Returns every project the calling credential may act on, with the permissions it holds in each. A project-scoped API key returns the single project it is bound to. A user-scoped credential returns the projects in its organization where the holder is an active member. Use the returned id in the X-Roark-Project-Id header.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark project list [options]
```

> [!NOTE]
>
> Calls `GET /v1/projects`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark project list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation environment create
Source: https://docs.roark.ai/cli-reference/commands/simulation-environment-create

Create an environment

Creates an environment for the project: a noise bed and the level it plays at. Reference it by id when setting a customer flow variant's environment. Roark's curated presets always play at the default level, so this is how a project gets the same bed louder or quieter.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation environment create [options]
```

> [!NOTE]
>
> Calls `POST /v1/simulation/environment`

## Options

**--name** `string` *required*

Display name, shown wherever a flow variant references the environment.

**--description** `string`

Optional note on when to use this environment.

**--background-noise** `string` *required*

The noise bed played underneath the simulated caller. NONE plays nothing. One of `NONE`, `AIRPORT`, `CHILDREN_PLAYING`, `CITY`, `COFFEE_SHOP`, `CONSTRUCTION`, `CRYING_BABY`, `DRIVING`, `LIBRARY`, `OFFICE`, `THUNDERSTORM`, `TRAIN`

**--background-noise-volume** `number`

How loud the bed plays, as a gain from 0 (silent) to 1 (as loud as the caller). Defaults to 0.1, which sits well under the caller. Ignored on Vapi endpoints, which have no level control.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation environment create --name <value> --background-noise <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation environment delete
Source: https://docs.roark.ai/cli-reference/commands/simulation-environment-delete

Delete an environment

Soft-deletes one of the project's environments. It disappears from reads and cannot be picked for new runs; runs already built keep their snapshot. Refused (409) while a live customer flow variant still uses it: move those variants first. Roark-curated environments cannot be deleted (403).

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation environment delete <environment-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/simulation/environment/{environmentId}`

## Arguments

**environment-id** `string` *required*

The environment-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation environment delete <environment-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation environment get
Source: https://docs.roark.ai/cli-reference/commands/simulation-environment-get

Get environment by ID

Returns a single environment by its ID.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation environment get <environment-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/environment/{environmentId}`

## Arguments

**environment-id** `string` *required*

The environment-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation environment get <environment-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation environment list
Source: https://docs.roark.ai/cli-reference/commands/simulation-environment-list

List environments

Returns a paginated list of environments: the project's own plus the environments Roark curates and shares across every project. Reference one by id when setting a customer flow variant's environment.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation environment list [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/environment`

## Options

**--limit** `integer`

**--after** `string`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation environment list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation environment update
Source: https://docs.roark.ai/cli-reference/commands/simulation-environment-update

Update an environment

Updates one of the project's environments. Only the fields sent are changed. Runs already built keep the snapshot they were built with. Roark-curated environments cannot be edited (403).

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation environment update <environment-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/simulation/environment/{environmentId}`

## Arguments

**environment-id** `string` *required*

The environment-id.

## Options

**--name** `string`

Display name, shown wherever a flow variant references the environment.

**--description** `string`

Optional note on when to use this environment.

**--background-noise** `string`

The noise bed played underneath the simulated caller. NONE plays nothing. One of `NONE`, `AIRPORT`, `CHILDREN_PLAYING`, `CITY`, `COFFEE_SHOP`, `CONSTRUCTION`, `CRYING_BABY`, `DRIVING`, `LIBRARY`, `OFFICE`, `THUNDERSTORM`, `TRAIN`

**--background-noise-volume** `number`

How loud the bed plays, as a gain from 0 (silent) to 1 (as loud as the caller). Defaults to 0.1, which sits well under the caller. Ignored on Vapi endpoints, which have no level control.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation environment update <environment-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation group create
Source: https://docs.roark.ai/cli-reference/commands/simulation-group-create

Create a simulation group

Saves a named set of plans to run together, again and again. Nothing runs yet: POST /v1/simulation/group/\{groupId}/run starts one run of every plan and groups them into a run group with one combined report. Slack channels need Slack connected to the project. Email is on by default.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation group create [options]
```

> [!NOTE]
>
> Calls `POST /v1/simulation/group`

## Options

**--name** `string` *required*

What to call the group.

**--description** `string`

What the group is for.

**--simulation-run-plan-ids** `array` *required*

The plans to run together, in run order (2 to 20). Duplicates are dropped. Repeatable.

**--notifications** `object`

Where each run's group report announces itself once it is ready.

**--notifications.email** `object`

**--notifications.slack** `array`

Channels in the project's connected Slack workspace to post the report to (at most 10). Slack must be connected to the project first.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation group create --name <value> --simulation-run-plan-ids <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation group delete
Source: https://docs.roark.ai/cli-reference/commands/simulation-group-delete

Delete a simulation group

Deletes a saved simulation group and any schedules that run it. Run groups it already started, and their reports, stay. A group managed by config as code returns 409.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation group delete <group-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/simulation/group/{groupId}`

## Arguments

**group-id** `string` *required*

The simulation group ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation group delete <group-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation group get
Source: https://docs.roark.ai/cli-reference/commands/simulation-group-get

Get a simulation group

Returns a saved simulation group with its plans in run order and notification settings.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation group get <group-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/group/{groupId}`

## Arguments

**group-id** `string` *required*

The simulation group ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation group get <group-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation group list
Source: https://docs.roark.ai/cli-reference/commands/simulation-group-list

List simulation groups

Returns the saved simulation groups in the project, most recently edited first, each with its plans in run order and notification settings.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation group list [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/group`

## Options

**--limit** `integer`

Maximum number of groups to return (default: 20, max: 100).

**--offset** `integer`

Pagination offset.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation group list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation group run
Source: https://docs.roark.ai/cli-reference/commands/simulation-group-run

Run a simulation group

Starts one run of every plan in the group, exactly as POST /v1/simulation/plan/\{planId}/job would for each, and groups the runs into a run group whose combined report is announced through the group's notification settings once it is ready. This places real calls. Config-managed groups can be run here too. Returns the run group. Poll GET /v1/simulation/run-group/\{groupId} until `latestReport.status` is READY. If a plan cannot start after others did, the runs that started are still grouped and the 400 names the run group id.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation group run <group-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/simulation/group/{groupId}/run`

## Arguments

**group-id** `string` *required*

The simulation group ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation group run <group-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation group update
Source: https://docs.roark.ai/cli-reference/commands/simulation-group-update

Update a simulation group

Edits a saved simulation group. Omitted fields keep their current value; `notifications`, when sent, replaces the settings as a whole. Run groups it already started keep theirs. A group managed by config as code (`configManaged: true`) returns 409: change it in the config file.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation group update <group-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/simulation/group/{groupId}`

## Arguments

**group-id** `string` *required*

The simulation group ID.

## Options

**--name** `string`

**--description** `string`

Send null to clear it.

**--simulation-run-plan-ids** `array`

Replaces the plans, in run order. Repeatable.

**--notifications** `object`

Where each run's group report announces itself once it is ready.

**--notifications.email** `object`

**--notifications.slack** `array`

Channels in the project's connected Slack workspace to post the report to (at most 10). Slack must be connected to the project first.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation group update <group-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation job get
Source: https://docs.roark.ai/cli-reference/commands/simulation-job-get

Get simulation by ID

Get a individual simulation run directly by its ID. This is generally part of a larger simulation run plan job.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation job get <job-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/job/{jobId}`

## Arguments

**job-id** `string` *required*

Simulation job ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation job get <job-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation job lookup
Source: https://docs.roark.ai/cli-reference/commands/simulation-job-lookup

Lookup by phone number

Find the matching simulation using the number used by the Roark simulation agent.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation job lookup [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/job/lookup`

## Options

**--roark-phone-number** `string` *required*

Phone number provisioned by Roark for the simulation job in E.164 format. In the case of an inbound simulation, this is the number that calls your agent; in the case of an outbound simulation, this is the number you call from your agent.

**--call-received-at** `string`

ISO 8601 timestamp of when the call was received. Alternatively, any time between the start and end of the call is valid. Defaults to the current time, which fetches any jobs that are currently ongoing.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation job lookup --roark-phone-number <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation job tool-mock list
Source: https://docs.roark.ai/cli-reference/commands/simulation-job-tool-mock-list

List mocked tool invocations

Every guarded tool invocation Roark answered during this simulation: what the agent tried to call, with what arguments, and the simulated response it received. Use this to assert tool behavior in CI after a test call, with zero real side effects. Empty when the agent's tools are not guarded (see the tool guard docs).

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation job tool-mock list <job-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/job/{jobId}/tool-mock`

## Arguments

**job-id** `string` *required*

Simulation job ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation job tool-mock list <job-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation persona create
Source: https://docs.roark.ai/cli-reference/commands/simulation-persona-create

Create a new persona

Creates a new persona for the authenticated project.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation persona create [options]
```

> [!NOTE]
>
> Calls `POST /v1/persona`

## Options

**--name** `string` *required*

The name the agent will identify as during conversations.

**--display-name** `string`

Label shown in place of the name across the dashboard (e.g. a short descriptor like "Irate Escalator"). The persona still identifies as `name` on calls. Omit or set null to display the name itself.

**--description** `string`

Human-readable description of the persona.

**--language** `string` *required*

Primary language ISO 639-1 code for the persona. One of `EN`, `ES`, `DE`, `HI`, `FR`, `NL`, `AR`, `EL`, `IT`, `ID`, `TH`, `JA`, `TL`, `MS`, `ZH`, `TR`, `PT`, `HE`

**--secondary-language** `string`

Secondary language ISO 639-1 code for code-switching (e.g., Hinglish, Spanglish). One of `EN`

**--understood-languages** `array`

Languages the persona can understand. Defaults to the languages the persona speaks. One of `EN`, `ES`, `DE`, `HI`, `FR`, `NL`, `AR`, `EL`, `IT`, `ID`, `TH`, `JA`, `TL`, `MS`, `ZH`, `TR`, `PT`, `HE` Repeatable.

**--accent** `string` *required*

Accent of the persona, defined using ISO 3166-1 alpha-2 country codes with optional variants. One of `US`, `US_X_SOUTH`, `GB`, `ES`, `DE`, `IN`, `FR`, `NL`, `SA`, `GR`, `AU`, `IT`, `ID`, `TH`, `JP`, `NZ`, `PH`, `SG`, `MY`, `HK`, `TR`, `PT`, `IL`

**--age** `string`

How old the caller sounds and behaves. Only ages the persona's accent has a voice for are accepted; defaults to ADULT, which every accent supports. One of `CHILD`, `TEENAGER`, `ADULT`, `ELDERLY`

**--gender** `string` *required*

Gender of the persona. One of `MALE`, `FEMALE`

**--background-noise** `string`

Background noise setting. One of `NONE`, `AIRPORT`, `CHILDREN_PLAYING`, `CITY`, `COFFEE_SHOP`, `CONSTRUCTION`, `CRYING_BABY`, `DRIVING`, `LIBRARY`, `OFFICE`, `THUNDERSTORM`, `TRAIN`

**--speech-pace** `string`

Speech pace of the persona. One of `SUPER_SLOW`, `SLOW`, `NORMAL`, `FAST`, `SUPER_FAST`

**--speech-clarity** `string`

Speech clarity of the persona. One of `CLEAR`, `VAGUE`, `RAMBLING`

**--has-disfluencies** `switch`

Whether the persona uses filler words like "um" and "uh".

**--base-emotion** `string`

Base emotional state of the persona. One of `NEUTRAL`, `CHEERFUL`, `CONFUSED`, `FRUSTRATED`, `SKEPTICAL`, `RUSHED`, `DISTRACTED`, `ANGRY`, `ANXIOUS`, `SAD`

**--intent-clarity** `string`

How clearly the persona expresses their intentions. One of `CLEAR`, `INDIRECT`, `VAGUE`

**--confirmation-style** `string`

How the persona confirms information. One of `EXPLICIT`, `VAGUE`

**--memory-reliability** `string`

How reliable the persona's memory is. One of `HIGH`, `LOW`

**--response-timing** `string`

Deprecated and inert: it no longer affects the call. It set how long the persona waited once the agent stopped talking, and measured across production simulations it moved the reply gap by less than the noise floor, because model and speech latency dominate it. Every persona now uses one voice-activity profile. Use `interruption` for a caller who talks over the agent. Still accepted and stored so existing clients keep working. `BARGE_IN` is stored as `responseTiming: QUICK` with `interruption: OCCASIONAL`. One of `RELAXED`, `NORMAL`, `QUICK`, `BARGE_IN`

**--interruption** `string`

How much the persona talks over the agent. Defaults to OFF, a caller that waits its turn. One of `OFF`, `BACKCHANNEL`, `OCCASIONAL`, `HEAVY`

**--backstory-prompt** `string`

Background story and behavioral patterns for the persona.

**--idle-messages** `array`

Messages the persona will say when the agent goes silent during a call. Defaults to language-appropriate phrases when omitted or sent as null. Repeatable.

**--idle-timeout-seconds** `integer`

Seconds of silence before the persona sends an idle message.

**--idle-message-max-spoken-count** `integer`

Maximum number of idle messages the persona will send before giving up.

**--idle-message-reset-count-on-user-speech-enabled** `switch`

Whether the idle message counter resets when the agent speaks.

**--properties** `object`

Additional custom properties about the persona.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation persona create --name <value> --language <value> --accent <value> --gender <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation persona get
Source: https://docs.roark.ai/cli-reference/commands/simulation-persona-get

Get persona by ID

Returns a specific persona by its ID.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation persona get <persona-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/persona/{personaId}`

## Arguments

**persona-id** `string` *required*

The ID of the persona to retrieve.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation persona get <persona-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation persona list
Source: https://docs.roark.ai/cli-reference/commands/simulation-persona-list

List personas

Returns a paginated list of personas for the authenticated project.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation persona list [options]
```

> [!NOTE]
>
> Calls `GET /v1/persona`

## Options

**--limit** `integer`

**--after** `string`

**--search-text** `string`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation persona list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation persona update
Source: https://docs.roark.ai/cli-reference/commands/simulation-persona-update

Update a persona

Updates an existing persona by its ID.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation persona update <persona-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/persona/{personaId}`

## Arguments

**persona-id** `string` *required*

The ID of the persona to update.

## Options

**--name** `string`

The name the agent will identify as during conversations.

**--display-name** `string`

Label shown in place of the name across the dashboard (e.g. a short descriptor like "Irate Escalator"). The persona still identifies as `name` on calls. Omit or set null to display the name itself.

**--description** `string`

Human-readable description of the persona.

**--language** `string`

Primary language ISO 639-1 code for the persona. One of `EN`, `ES`, `DE`, `HI`, `FR`, `NL`, `AR`, `EL`, `IT`, `ID`, `TH`, `JA`, `TL`, `MS`, `ZH`, `TR`, `PT`, `HE`

**--secondary-language** `string`

Secondary language ISO 639-1 code for code-switching (e.g., Hinglish, Spanglish). One of `EN`

**--understood-languages** `array`

Languages the persona can understand. Multilingual combinations are limited by multilingual speech recognition support. One of `EN`, `ES`, `DE`, `HI`, `FR`, `NL`, `AR`, `EL`, `IT`, `ID`, `TH`, `JA`, `TL`, `MS`, `ZH`, `TR`, `PT`, `HE` Repeatable.

**--accent** `string`

Accent of the persona, defined using ISO 3166-1 alpha-2 country codes with optional variants. One of `US`, `US_X_SOUTH`, `GB`, `ES`, `DE`, `IN`, `FR`, `NL`, `SA`, `GR`, `AU`, `IT`, `ID`, `TH`, `JP`, `NZ`, `PH`, `SG`, `MY`, `HK`, `TR`, `PT`, `IL`

**--age** `string`

How old the caller sounds and behaves. Only ages the persona's accent has a voice for are accepted; defaults to ADULT, which every accent supports. One of `CHILD`, `TEENAGER`, `ADULT`, `ELDERLY`

**--gender** `string`

Gender of the persona. One of `MALE`, `FEMALE`

**--background-noise** `string`

Background noise setting. One of `NONE`, `AIRPORT`, `CHILDREN_PLAYING`, `CITY`, `COFFEE_SHOP`, `CONSTRUCTION`, `CRYING_BABY`, `DRIVING`, `LIBRARY`, `OFFICE`, `THUNDERSTORM`, `TRAIN`

**--speech-pace** `string`

Speech pace of the persona. One of `SUPER_SLOW`, `SLOW`, `NORMAL`, `FAST`, `SUPER_FAST`

**--speech-clarity** `string`

Speech clarity of the persona. One of `CLEAR`, `VAGUE`, `RAMBLING`

**--has-disfluencies** `switch`

Whether the persona uses filler words like "um" and "uh".

**--base-emotion** `string`

Base emotional state of the persona. One of `NEUTRAL`, `CHEERFUL`, `CONFUSED`, `FRUSTRATED`, `SKEPTICAL`, `RUSHED`, `DISTRACTED`, `ANGRY`, `ANXIOUS`, `SAD`

**--intent-clarity** `string`

How clearly the persona expresses their intentions. One of `CLEAR`, `INDIRECT`, `VAGUE`

**--confirmation-style** `string`

How the persona confirms information. One of `EXPLICIT`, `VAGUE`

**--memory-reliability** `string`

How reliable the persona's memory is. One of `HIGH`, `LOW`

**--response-timing** `string`

Deprecated and inert: it no longer affects the call. It set how long the persona waited once the agent stopped talking, and measured across production simulations it moved the reply gap by less than the noise floor, because model and speech latency dominate it. Every persona now uses one voice-activity profile. Use `interruption` for a caller who talks over the agent. Still accepted and stored so existing clients keep working. `BARGE_IN` is stored as `responseTiming: QUICK` with `interruption: OCCASIONAL`. One of `RELAXED`, `NORMAL`, `QUICK`, `BARGE_IN`

**--interruption** `string`

How much the persona talks over the agent while it is still speaking. OFF waits its turn. BACKCHANNEL makes listening noises ("mm-hm") over the agent without taking the floor, which tests whether the agent wrongly stops for them. OCCASIONAL adds cutting in on some long agent turns, HEAVY on most of them. Timing is randomised per turn, so two runs of the same persona do not interrupt at identical moments. One of `OFF`, `BACKCHANNEL`, `OCCASIONAL`, `HEAVY`

**--backstory-prompt** `string`

Background story and behavioral patterns for the persona.

**--idle-messages** `array`

Messages the persona will say when the agent goes silent during a call. null = "Automatic": language-appropriate defaults are used at call time. Repeatable.

**--idle-timeout-seconds** `integer`

Seconds of silence before the persona sends an idle message.

**--idle-message-max-spoken-count** `integer`

Maximum number of idle messages the persona will send before giving up.

**--idle-message-reset-count-on-user-speech-enabled** `switch`

Whether the idle message counter resets when the agent speaks.

**--properties** `object`

Additional custom properties about the persona.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation persona update <persona-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation plan create
Source: https://docs.roark.ai/cli-reference/commands/simulation-plan-create

Create a run plan

Creates a new simulation run plan. To run a simulation, use POST /v1/simulation/run instead: it starts a run from a plan or from an inline configuration, and takes runtime variables. Create a plan here when you want a reusable, named one to run later. Send `template` instead of a full configuration to save one of the built-in templates as a plan. It takes the same fields as the template variant of POST /v1/simulation/run, builds the same plan, and never starts it. To compare one property, attach the flow once and send `comparisonProperty` with the `comparisonValues` to run: the plan attaches the flow once per value.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan create [options]
```

> [!NOTE]
>
> Calls `POST /v1/simulation/plan`

## Request body

This endpoint accepts one of several request shapes, so the body is supplied as
JSON through `--data` or on standard input rather than as individual flags.

**CreateRunPlanFromConfig** `object`

Requires `name`, `direction`, `maxSimulationDurationSeconds`, `agentEndpoints`

**CreateRunPlanFromTemplate** `object`

Requires `template`, `agentEndpoints`, `direction`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan create --data '{ ... }'
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation plan delete
Source: https://docs.roark.ai/cli-reference/commands/simulation-plan-delete

Delete a run plan

Soft-deletes a simulation run plan by its ID.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan delete <plan-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/simulation/plan/{planId}`

## Arguments

**plan-id** `string` *required*

The ID of the run plan to delete.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan delete <plan-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation plan get
Source: https://docs.roark.ai/cli-reference/commands/simulation-plan-get

Get run plan by ID

Returns a specific simulation run plan by its ID.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan get <plan-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/plan/{planId}`

## Arguments

**plan-id** `string` *required*

The ID of the run plan to retrieve.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan get <plan-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation plan job cancel
Source: https://docs.roark.ai/cli-reference/commands/simulation-plan-job-cancel

Cancel a simulation plan job

Stops a run that has not finished yet. Already-finished runs are left alone. Intended for CI: when a pipeline is cancelled or superseded, cancelling the run stops it placing calls you no longer need. Safe to call more than once.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job cancel <job-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/simulation/plan/job/{jobId}/cancel`

## Arguments

**job-id** `string` *required*

Simulation run plan job ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job cancel <job-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation plan job cost get
Source: https://docs.roark.ai/cli-reference/commands/simulation-plan-job-cost-get

Get simulation run cost

The billable spend for one simulation run (run plan job), split by cost type, with the spend of each simulated call in it. Safe to call while the run is in progress: totals reflect the costs recorded so far. Amounts are billable spend only, in exact micro-dollars (`*UsdMicros`) and dollars (`*Usd`, for display). Requires an API key with the `billing:read` permission.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job cost get <job-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/plan/job/{jobId}/cost`

## Arguments

**job-id** `string` *required*

The job-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job cost get <job-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation plan job debrief create
Source: https://docs.roark.ai/cli-reference/commands/simulation-plan-job-debrief-create

Generate simulation run debrief

Request the written debrief for a settled run: executive summary, failure themes and recommendations. Runs with graded calls get one automatically, so this is for regenerating one (`regenerate: true`) or retrying one that failed. Returns immediately. Generation takes a few minutes: poll GET /v1/simulation/plan/job/\{jobId}/results until `debrief.status` is READY or FAILED. Without `regenerate`, a debrief that is ready or already being written is returned as-is.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job debrief create <job-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/simulation/plan/job/{jobId}/debrief`

## Arguments

**job-id** `string` *required*

The job-id.

## Options

**--regenerate** `switch`

Generate a new debrief even when a ready one exists. Without it, an existing ready or in-progress debrief is returned as-is.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job debrief create <job-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation plan job get
Source: https://docs.roark.ai/cli-reference/commands/simulation-plan-job-get

Get simulation plan job

Retrieve details of a simulation plan job including all associated simulation jobs (calls)

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job get <job-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/plan/job/{jobId}`

## Arguments

**job-id** `string` *required*

Simulation run plan job ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job get <job-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation plan job list
Source: https://docs.roark.ai/cli-reference/commands/simulation-plan-job-list

List simulation plan jobs

Returns a paginated list of simulation run plan jobs. Filter by status, plan ID, or label to find specific simulation batches.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job list [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/plan/jobs`

## Options

**--limit** `integer`

Maximum number of plan jobs to return (default: 20, max: 50).

**--after** `string`

Cursor for pagination - use the nextCursor value from a previous response.

**--status** `string`

Filter by plan job status (PENDING, QUEUED, `CREATING_SNAPSHOTS`, `CREATING_SIMULATIONS`, `PREPARING_CAPACITY`, `RUNNING_SIMULATIONS`, COMPLETED, FAILED, `TIMED_OUT`, CANCELLED, CANCELLING, `ENDING_SIMULATIONS`). One of `PENDING`, `QUEUED`, `CREATING_SNAPSHOTS`, `CREATING_SIMULATIONS`, `PREPARING_CAPACITY`, `RUNNING_SIMULATIONS`, `COMPLETED`, `FAILED`, `TIMED_OUT`, `CANCELLED`, `CANCELLING`, `ENDING_SIMULATIONS`

**--simulation-run-plan-id** `string`

Filter by simulation run plan ID.

**--label-id** `string`

Filter by label ID attached to the plan job. Use this if you know the label ID.

**--label-name** `string`

Filter by label name attached to the plan job. More user-friendly alternative to labelId. Case-insensitive.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation plan job result get
Source: https://docs.roark.ai/cli-reference/commands/simulation-plan-job-result-get

Get simulation run results

Everything the run report shows, in one request: the summary and score, the pass/fail verdict, per-metric aggregates, every simulation with its check results, breakdowns by agent and persona, the comparison against the previous run (or `compareToJobId`), the sweep attribution, and the written debrief with its executive summary, failure themes and recommendations. Safe to call while the run is in progress: counts reflect what has landed so far. `verdict` and `sweepAttribution` stay null until the run ends. The debrief is generated automatically a few minutes after a run with graded calls settles; poll until `debrief.status` is READY, or request one with POST /v1/simulation/plan/job/\{jobId}/debrief.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job result get <job-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/plan/job/{jobId}/results`

## Arguments

**job-id** `string` *required*

The job-id.

## Options

**--compare-to-job-id** `string`

Simulation run plan job to compare this run against, e.g. the last run on your main branch. Defaults to the plan's previous settled run. Any run in the project works, including another plan's: checks are matched by metric, and a check only one run graded reads null on the other side.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job result get <job-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation plan job start
Source: https://docs.roark.ai/cli-reference/commands/simulation-plan-job-start

Run a simulation plan

Deprecated: use POST /v1/simulation/run, which does the same thing and can also take the plan configuration inline, so a one-off run does not have to create a plan first. Creates and executes a job for an existing simulation run plan. Optionally provide runtime variables to override plan-defined variables.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job start <plan-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/simulation/plan/{planId}/job`

## Arguments

**plan-id** `string` *required*

Simulation run plan ID.

## Options

**--variables** `string`

Values for the \{\{variables}} the run resolves, overriding whatever the plan has pinned. An object applies them to the whole run: \{ "orderNumber": "12345", "tier": "gold" } An array applies them per flow, or to just its happy path or one of its edge cases, when a single set will not do. Each entry carries what it applies to: \[ \{ "flowId": "550e8400-...", "variables": \{ "orderNumber": "12345" } }, \{ "flowId": "550e8400-...", "happyPath": true, "variables": \{ "orderNumber": "55555" } }, \{ "flowId": "550e8400-...", "edgeCaseId": "7a3d2e1f-...", "variables": \{ "orderNumber": "67890" } } ] An entry that narrows to neither covers everything that flow resolves. A flow this plan does not attach, or an edge case that does not belong to the flow, is rejected rather than ignored. A plan built on scenarios rather than customer flows targets them the same way, with `scenarioId` in place of `flowId`. That form is deprecated alongside scenarios themselves, and still accepted so runs against those plans keep working.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job start <plan-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation plan list
Source: https://docs.roark.ai/cli-reference/commands/simulation-plan-list

List run plans

Returns a paginated list of simulation run plans. Optionally filter by search text or agent ID.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan list [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/plan`

## Options

**--limit** `integer`

Maximum number of run plans to return (default: 20, max: 50).

**--after** `string`

Cursor for pagination - use the nextCursor value from a previous response.

**--search-text** `string`

Search text to filter run plans by name.

**--agent-id** `string`

Filter run plans by agent ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation plan update
Source: https://docs.roark.ai/cli-reference/commands/simulation-plan-update

Update a run plan

Updates an existing simulation run plan by its ID.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan update <plan-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/simulation/plan/{planId}`

## Arguments

**plan-id** `string` *required*

The ID of the run plan to update.

## Options

**--is-hidden** `switch`

Whether this plan is hidden from GET /v1/simulation/plan. A run started without `saveAsPlan` creates a hidden plan to carry it. Send `{ "name": "...", "isHidden": false }` to keep that configuration as a reusable plan, which is what the app does when you save a one-off run.

**--name** `string`

Name of the run plan.

**--description** `string`

Description of the run plan.

**--direction** `string`

Direction of the simulation (INBOUND or OUTBOUND). One of `INBOUND`, `OUTBOUND`

**--iteration-count** `integer`

Number of iterations to run for each test case (1-10000).

**--max-concurrent-jobs** `integer`

Maximum number of concurrent simulation jobs.

**--max-simulation-duration-seconds** `integer`

Maximum duration in seconds for each simulation.

**--silence-timeout-seconds** `integer`

Timeout in seconds for silence detection.

**--max-no-response-retries** `integer`

How many more times to run a test case when the agent under test never responds: it never speaks on a call or never replies in a chat (0-10). 0 turns retries off. Failed checks and failures on Roark’s side are never retried. Each retry is a separate attempt, billed like any other, so a plan retrying N times can place up to N + 1 calls per test case. Every silent attempt stays on the run with its own call; the run settles once each test case has a final attempt, and the agent never spoke verdict is judged on each test case’s last attempt.

**--no-response-retry-backoff-seconds** `integer`

Seconds a retry waits before it dials (30-600). Only used when `maxNoResponseRetries` is above 0.

**--end-call-phrases** `array`

Phrases that trigger end of call. Empty array disables the feature. Repeatable.

**--end-call-reasons** `array`

Semantic conditions that trigger end of call. The LLM evaluates the conversation against these conditions. Empty array disables the feature. Repeatable.

**--execution-mode** `string`

Execution mode (PARALLEL or SEQUENTIAL). One of `PARALLEL`, `SEQUENTIAL_SAME_RUN_PLAN`, `SEQUENTIAL_PROJECT`

**--scenarios** `array`

Deprecated: use `flows` instead. Replaces the scenarios on this run plan. Omit to leave them unchanged; send an empty array to detach them all, which is how a scenario-based plan is moved over to flows.

**--flows** `array`

Replaces the customer flows attached to this run plan. Omit to leave them unchanged; send an empty array to detach them all.

**--personas** `array`

Personas to include in this run plan.

**--agent-endpoints** `array`

Agent endpoints to include in this run plan.

**--metrics** `array`

Metric definitions to include in this run plan. Reference each by `id` (UUID) or `slug`.

**--enrich-with-live-conversation** `switch`

Whether to merge the customer's own live recording into each simulation of this plan.

**--include-flow-metrics** `switch`

Whether to also collect each attached flow's own metrics, on top of this plan's list.

**--include-automatic-metrics** `switch`

Whether to let the run add metrics by itself off the attached flows. See `POST /v1/simulation/plan`.

**--comparison-property** `string`

The property this plan investigates. Send `null` to clear the comparison; omit the field to leave it unchanged. See `POST /v1/simulation/plan`. The pair moves together. Sending `comparisonProperty` without `comparisonBaseline` keeps the stored baseline when the property is unchanged and the baseline is still one of the values being run. Otherwise it becomes the new property's norm, or `null` when that norm is not being run either, because a baseline is a value of one specific property. One of `ACCENT`, `AGE`, `BACKGROUND_NOISE`, `BACKGROUND_NOISE_VOLUME`, `BASE_EMOTION`, `CONFIRMATION_STYLE`, `GENDER`, `INTENT_CLARITY`, `LANGUAGE`, `INTERRUPTION`, `MEMORY_RELIABILITY`, `RESPONSE_TIMING`, `SPEECH_CLARITY`, `SPEECH_PACE`

**--comparison-baseline** `string`

The reference value, shown first in the results. See `POST /v1/simulation/plan`. A real value cannot be sent on its own: the property it belongs to decides which values are legal, and an omitted property means "leave unchanged", which this endpoint cannot check a baseline against. Send `comparisonProperty` with it, or get a `400`. `null` on its own IS allowed, and clears just the baseline while leaving the property set. Nothing needs validating when clearing, and a property with no baseline is a real state: the report falls back to that property's own norm, and `GENDER` has no norm to fall back to.

**--comparison-values** `array`

The arms to run. See `POST /v1/simulation/plan`. Omitting it keeps the arms the plan already has, pins included, so an edit that only renames the plan never widens a sweep you deliberately narrowed, and never multiplies what it costs. Send it with `comparisonProperty` and `flows`, which the arms are rebuilt from.

**--notifications** `object`

Where each run's report announces itself once it is ready.

**--notifications.email** `object`

**--notifications.slack** `array`

Channels in the project's connected Slack workspace to post the report to (at most 10). Slack must be connected to the project first.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan update <plan-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation run
Source: https://docs.roark.ai/cli-reference/commands/simulation-run

Run a simulation

Starts a simulation and returns the run. Send `template` to run one of the built-in templates: it supplies the metrics and checks, and for some templates the flows too, so the request only names the agent and the direction. Send `plan` to describe a simulation yourself and run it once. Send `planId` to run a plan you already have. `template` and `plan` both resolve to a run plan, returned as `simulationRunPlanId`. Add `saveAsPlan` to keep it, or read it back to see exactly what ran. A plan built from a template is a snapshot: retuning the template later never changes what that plan runs, which is what makes a saved one safe to pin in CI.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run [options]
```

> [!NOTE]
>
> Calls `POST /v1/simulation/run`

## Request body

This endpoint accepts one of several request shapes, so the body is supplied as
JSON through `--data` or on standard input rather than as individual flags.

**RunSimulationFromConfig** `object`

Requires `plan`

**RunSimulationFromPlanId** `object`

Requires `planId`

**RunSimulationFromConfigKey** `object`

Requires `planConfigKey`

**RunSimulationFromTemplate** `object`

Requires `template`, `agentEndpoints`, `direction`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run --data '{ ... }'
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation run-group create
Source: https://docs.roark.ai/cli-reference/commands/simulation-run-group-create

Create a run group

Groups several simulation runs so they read as one, and starts generating the group report: one document across every run, with a scorecard summed across them and the issues found, each merged from the per-run debriefs. Send `simulationRunPlanJobIds` to group runs that already exist (runs still in progress are fine: the report waits for them). Send `simulationRunPlanIds` to start one run of each plan and group those; this places real calls, exactly as starting each plan would. Poll GET /v1/simulation/run-group/\{groupId} until `latestReport.status` is READY.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run-group create [options]
```

> [!NOTE]
>
> Calls `POST /v1/simulation/run-group`

## Options

**--name** `string`

What to call the group. Defaults to the plans' names when starting plans, and to the runs' plan names when grouping existing runs.

**--simulation-run-plan-job-ids** `array`

Runs that already exist, in display order (2 to 20). Runs still in progress are fine: the group report waits for them. Send this or `simulationRunPlanIds`, not both. Repeatable.

**--simulation-run-plan-ids** `array`

Plans to run once each, in display order (2 to 20). Each plan starts a run, exactly as POST /v1/simulation/plan/\{planId}/job would. Send this or `simulationRunPlanJobIds`, not both. Repeatable.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run-group create
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation run-group delete
Source: https://docs.roark.ai/cli-reference/commands/simulation-run-group-delete

Delete a run group

Deletes a run group. Its runs, and their results, are untouched.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run-group delete <group-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/simulation/run-group/{groupId}`

## Arguments

**group-id** `string` *required*

The run group ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run-group delete <group-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation run-group get
Source: https://docs.roark.ai/cli-reference/commands/simulation-run-group-get

Get a run group

Returns a run group with its runs in group order and its latest group report, including the report document once READY.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run-group get <group-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/run-group/{groupId}`

## Arguments

**group-id** `string` *required*

The run group ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run-group get <group-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation run-group list
Source: https://docs.roark.ai/cli-reference/commands/simulation-run-group-list

List run groups

Returns the run groups in the project, newest first, with their runs and where each group report stands. Read a report document from GET /v1/simulation/run-group/\{groupId}.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run-group list [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/run-group`

## Options

**--limit** `integer`

Maximum number of run groups to return (default: 20, max: 100).

**--offset** `integer`

Pagination offset.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run-group list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation run-group report regenerate
Source: https://docs.roark.ai/cli-reference/commands/simulation-run-group-report-regenerate

Regenerate a run group report

Generates the group report again over the latest state of its runs. Returns the report generation; if one is already in flight it is returned instead of starting another. Poll GET /v1/simulation/run-group/\{groupId} until `latestReport.status` is READY.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run-group report regenerate <group-id> [options]
```

> [!NOTE]
>
> Calls `POST /v1/simulation/run-group/{groupId}/report`

## Arguments

**group-id** `string` *required*

The run group ID.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run-group report regenerate <group-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation template list
Source: https://docs.roark.ai/cli-reference/commands/simulation-template-list

List simulation templates

Returns the built-in simulation templates, each resolved against this project: the metric and check definitions it collects, and the flows it runs with the variants it covers. A template is a preset rather than a stored object, so building a run from one produces an ordinary run plan you own and can edit afterwards. Pass a `slug` as `template` to POST /v1/simulation/run. Every entry is complete, so there is no per-template endpoint to follow up with, and the list is a fixed catalogue rather than a paginated one. Only templates a request can actually run are listed.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation template list [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/template`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation template list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation tool-fixture create
Source: https://docs.roark.ai/cli-reference/commands/simulation-tool-fixture-create

Set a tool fixture

Create-or-replace by scope: one fixture per tool project-wide, plus one per (tool, flow variant). Setting the same scope twice replaces the response, so CI can apply fixtures idempotently. During Roark test calls the tool guard answers with the fixture verbatim instead of generating a response; real callers are never affected.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation tool-fixture create [options]
```

> [!NOTE]
>
> Calls `POST /v1/simulation/tool-fixture`

## Options

**--tool-name** `string` *required*

The tool this fixture answers for.

**--response** `string`

The exact JSON to return.

**--customer-flow-variant-id** `string`

Pin the fixture to one flow variant (scenario). Omit for a project-wide fixture. A variant-pinned fixture beats the project-wide one.

**--description** `string`

Why this fixture exists.

**--enabled** `switch`

Defaults to true.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation tool-fixture create --tool-name <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation tool-fixture delete
Source: https://docs.roark.ai/cli-reference/commands/simulation-tool-fixture-delete

Delete a tool fixture

The tool goes back to scenario-aware generated responses in test calls.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation tool-fixture delete <fixture-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/simulation/tool-fixture/{fixtureId}`

## Arguments

**fixture-id** `string` *required*

The fixture-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation tool-fixture delete <fixture-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation tool-fixture list
Source: https://docs.roark.ai/cli-reference/commands/simulation-tool-fixture-list

List tool fixtures

All deterministic tool responses configured for this project, optionally filtered by tool.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation tool-fixture list [options]
```

> [!NOTE]
>
> Calls `GET /v1/simulation/tool-fixture`

## Options

**--tool-name** `string`

Only fixtures for this tool.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation tool-fixture list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation tool-fixture update
Source: https://docs.roark.ai/cli-reference/commands/simulation-tool-fixture-update

Update a tool fixture

Change the response, description, or enabled state of an existing fixture.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation tool-fixture update <fixture-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/simulation/tool-fixture/{fixtureId}`

## Arguments

**fixture-id** `string` *required*

The fixture-id.

## Options

**--response** `string`

**--description** `string`

**--enabled** `switch`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation tool-fixture update <fixture-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark simulation tool mock
Source: https://docs.roark.ai/cli-reference/commands/simulation-tool-mock

Mock a tool call during a test call

The server half of the tool guard for code-first agents. When a guarded tool fires during a Roark test call, send the invocation here instead of executing it: Roark answers with a simulated backend response that is valid JSON, shaped by the tool contract you pass, consistent with the test scenario, and consistent with earlier mocked responses in the same call. Real callers are never affected: the guard only diverts when the agent-config resolve response identified the session as a Roark simulation, and this endpoint independently re-validates the simulation before answering. Failure contract for your wrapper: `404` means the simulation id is unknown to this project (treat the session as real). `409` means the simulation has already ended (stale session state: do NOT execute the real tool; return your static fallback). `5xx` means generation failed (return your static fallback). Identical retries (same tool, same arguments) within a few minutes return the stored response, so double-fired handlers stay consistent.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation tool mock [options]
```

> [!NOTE]
>
> Calls `POST /v1/simulation/tool-mock`

## Options

**--simulation-job-id** `string` *required*

The simulation this session belongs to, from the agent-config resolve response (`simulationJobId`). Roark re-validates it against the live simulation before answering.

**--session-id** `string`

Your session or room identifier, echoed back in logs for correlation.

**--tool-name** `string` *required*

The tool the agent invoked.

**--tool-description** `string`

The tool's contract: its description and, ideally, its parameter and return shape. The more contract you pass, the more faithful the simulated response.

**--arguments** `object`

The arguments the agent called the tool with, verbatim.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation tool mock --simulation-job-id <value> --tool-name <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark tool create
Source: https://docs.roark.ai/cli-reference/commands/tool-create

Create a tool

Creates a tool and declares its first version. A tool only recorded from calls under the same name is adopted; one already declared returns 409. Give it a `runtime` for a hosted agent to call your endpoint. Header values are returned on read, so keep secrets out of them.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark tool create [options]
```

> [!NOTE]
>
> Calls `POST /v1/tool`

## Options

**--name** `string` *required*

**--description** `string` *required*

When the agent should use the tool.

**--agent-id** `string`

**--inputs** `array`

**--invocation-criteria** `string`

**--expected-result** `string`

**--runtime** `object`

Omit or null for a tool that is only graded.

**--runtime.request** `object` *required*

**--runtime.changes-data** `switch`

**--runtime.response-fields** `array`

Repeatable.

**--runtime.messages** `object`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark tool create --name <value> --description <value> --runtime.request <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark tool delete
Source: https://docs.roark.ai/cli-reference/commands/tool-delete

Delete a tool

Deletes a tool no agent version uses. A tool an agent version uses returns 409.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark tool delete <tool-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/tool/{toolId}`

## Arguments

**tool-id** `string` *required*

The tool-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark tool delete <tool-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark tool get
Source: https://docs.roark.ai/cli-reference/commands/tool-get

Get a tool

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark tool get <tool-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/tool/{toolId}`

## Arguments

**tool-id** `string` *required*

The tool-id.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark tool get <tool-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark tool list
Source: https://docs.roark.ai/cli-reference/commands/tool-list

List tools

The project's tools and the built-in ones, each with its latest version: inputs, description and, for a tool a hosted agent runs, its request. Attach one to an agent draft by id.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark tool list [options]
```

> [!NOTE]
>
> Calls `GET /v1/tool`

## Options

**--agent-id** `string`

Only tools scoped to this agent.

**--include-built-in** `string`

Include the built-in tools (default true). One of `true`, `false`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark tool list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark tool update
Source: https://docs.roark.ai/cli-reference/commands/tool-update

Update a tool

Replaces the tool: its name, scope, inputs, description, grading hints and runtime. A field left out is cleared, so omitting agentId makes the tool project-wide. A change declares a new version; agent versions pinned to an older one keep it.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark tool update <tool-id> [options]
```

> [!NOTE]
>
> Calls `PUT /v1/tool/{toolId}`

## Arguments

**tool-id** `string` *required*

The tool-id.

## Options

**--name** `string` *required*

**--description** `string` *required*

When the agent should use the tool.

**--agent-id** `string`

**--inputs** `array`

**--invocation-criteria** `string`

**--expected-result** `string`

**--runtime** `object`

Omit or null for a tool that is only graded.

**--runtime.request** `object` *required*

**--runtime.changes-data** `switch`

**--runtime.response-fields** `array`

Repeatable.

**--runtime.messages** `object`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark tool update <tool-id> --name <value> --description <value> --runtime.request <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark usage get
Source: https://docs.roark.ai/cli-reference/commands/usage-get

Get usage & cost

Report billable usage and cost for the organization over a time window, broken down by project (highest spend first). Pass `from`/`to` as ISO 8601 timestamps; optionally pass `projectId` to scope the report to one project. Amounts are billable spend only (what we absorb is never included). Authenticated by either an organization API key with the `usage:read` permission (any project in the organization), or a project API key with the `billing:read` permission (that key's own project only; any other `projectId` is a 403).

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark usage get [options]
```

> [!NOTE]
>
> Calls `GET /v1/usage`

## Options

**--from** `string` *required*

Start of the window (inclusive), ISO 8601.

**--to** `string` *required*

End of the window (exclusive), ISO 8601. Must be after `from`, and the window may not exceed 366 days.

**--project-id** `string`

Limit the report to a single project. Omit for the whole organization.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark usage get --from <value> --to <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark webhook create
Source: https://docs.roark.ai/cli-reference/commands/webhook-create

Create webhook

Creates a new webhook with event subscriptions. The signing secret is only returned in this response.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark webhook create [options]
```

> [!NOTE]
>
> Calls `POST /v1/webhook`

## Options

**--url** `string` *required*

Webhook URL.

**--description** `string`

Webhook description.

**--headers** `object`

Request headers (e.g. authorization tokens).

**--events** `array` *required*

Event types to subscribe to (at least one required). One of `CALL_ANALYSIS_COMPLETED`, `CALL_ANALYSIS_FAILED`, `CALL_ANALYSIS_CANCELLED`, `CALL_EVALUATION_COMPLETED`, `CALL_EVALUATION_FAILED`, `SIMULATION_RUN_PLAN_JOB_STARTED`, `SIMULATION_RUN_PLAN_JOB_COMPLETED`, `SIMULATION_RUN_PLAN_JOB_FAILED`, `SIMULATION_RUN_PLAN_JOB_CANCELLED`, `SIMULATION_JOB_STARTED`, `SIMULATION_JOB_COMPLETED`, `SIMULATION_JOB_FAILED`, `SIMULATION_JOB_CANCELLED`, `METRIC_COLLECTION_JOB_COMPLETED`, `METRIC_COLLECTION_JOB_FAILED`, `CHAT_ANALYSIS_COMPLETED`, `CHAT_ANALYSIS_FAILED`, `ISSUE_OPENED`, `ISSUE_RESOLVED` Repeatable.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark webhook create --url <value> --events <value>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark webhook delete
Source: https://docs.roark.ai/cli-reference/commands/webhook-delete

Delete webhook

Deletes a webhook and all its event subscriptions.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark webhook delete <webhook-id> [options]
```

> [!NOTE]
>
> Calls `DELETE /v1/webhook/{webhookId}`

## Arguments

**webhook-id** `string` *required*

The ID of the webhook to delete.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark webhook delete <webhook-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark webhook get
Source: https://docs.roark.ai/cli-reference/commands/webhook-get

Get webhook by ID

Returns a specific webhook with its event subscriptions.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark webhook get <webhook-id> [options]
```

> [!NOTE]
>
> Calls `GET /v1/webhook/{webhookId}`

## Arguments

**webhook-id** `string` *required*

The ID of the webhook to retrieve.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark webhook get <webhook-id>
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark webhook list
Source: https://docs.roark.ai/cli-reference/commands/webhook-list

List webhooks

Returns a paginated list of webhooks with their event subscriptions.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark webhook list [options]
```

> [!NOTE]
>
> Calls `GET /v1/webhook`

## Options

**--limit** `integer`

Maximum number of webhooks to return (default: 20, max: 50).

**--after** `string`

Cursor for pagination - webhook ID to start after.

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark webhook list
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# roark whoami
Source: https://docs.roark.ai/cli-reference/commands/whoami

Describe the current credential

Returns the scope of the credential making the request, the organization it acts in, its default project (if any) and the permissions it was granted. Works for both project-scoped API keys and user-scoped credentials from the CLI or an MCP connector.

```bash Synopsis theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark whoami [options]
```

> [!NOTE]
>
> Calls `GET /v1/me`

## Example

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark whoami
```

Every command also takes the [global options](/cli-reference/overview#global-options) and sets one
of the [exit codes](/cli-reference/overview#exit-codes).


# CLI Reference
Source: https://docs.roark.ai/cli-reference/overview

Every roark command, generated from the same spec as the API

`roark` calls the Roark Analytics API from the command line. Commands read noun before verb, so the resource is what you type first and the verb is one of list, get, create, update or delete unless the operation is genuinely something else.

```bash Usage theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark <resource> [<sub-resource>...] <verb> [arguments] [options]
```

Installing the CLI, storing a credential and using it in CI are covered in the
[CLI guide](/documentation/sdks/cli). This tab is the per-command reference.

## Global options

Every command accepts these in addition to its own options.

**--base-url** `url`

API base URL.

**--token** `token`

Bearer token. Prefer `ROARK_API_BEARER_TOKEN` or the stored credential.

**--timeout** `ms`

Request timeout in milliseconds.

**--max-retries** `count`

Retries for retryable failures.

**--format** `format`

Output format: auto, json, jsonl, plain.

**--json** `switch`

Shorthand for `--format` json.

**--no-color** `switch`

Disable colour.

**-q, --quiet** `switch`

Suppress non-essential output.

**--allow-project-base-url** `switch`

Send the stored credential to a base URL set by a project `.roark.json`. Refused by default.

### Sending a body

Commands that send a request body accept these as well.

**--data** `json`

Request body as JSON, or @file to read one (@- for stdin).

**-y, --yes** `switch`

Skip confirmation prompts.

**--no-input** `switch`

Never prompt; fail instead.

## Configuration

Settings are resolved highest precedence first: command line flag, environment variable, project `.roark.json` found by walking up from the working directory, then the user config file. The stored credential is written with mode 0600.

## Environment

| Variable | Description |
| - | - |
| `ROARK_API_BEARER_TOKEN` | Bearer token used when `--token` is not given. |
| `ROARK_BASE_URL` | API base URL. |
| `XDG_CONFIG_HOME` | Overrides the location of the user config file. |
| `ROARK_ALLOW_PROJECT_BASE_URL` | Trusts a base URL set by a project `.roark.json`, like `--allow-project-base-url`. |
| `NO_COLOR` | Disables colour when set to any value. |

## Exit codes

Distinct per failure mode, so a CI job can tell a rejected request from a missing credential without reading stderr.

| Code | Meaning |
| - | - |
| `0` | The command succeeded. |
| `1` | The API rejected the request. |
| `2` | The command line was wrong. |
| `3` | No credential, or the credential was refused. |
| `4` | The addressed resource does not exist. |
| `5` | The request never completed: connection, timeout, or abort. |

## Commands

`roark <command> --help` prints the same options any of these pages lists.

### `agent`

| Command | Description |
| - | - |
| [`roark agent build`](/cli-reference/commands/agent-build) | Build a Roark-hosted agent |
| [`roark agent create`](/cli-reference/commands/agent-create) | Create a new agent |
| [`roark agent draft create`](/cli-reference/commands/agent-draft-create) | Start a draft |
| [`roark agent draft delete`](/cli-reference/commands/agent-draft-delete) | Discard the draft |
| [`roark agent draft get`](/cli-reference/commands/agent-draft-get) | Get an agent's draft |
| [`roark agent draft publish`](/cli-reference/commands/agent-draft-publish) | Publish the draft |
| [`roark agent draft update`](/cli-reference/commands/agent-draft-update) | Edit the draft |
| [`roark agent endpoint create`](/cli-reference/commands/agent-endpoint-create) | Create a new agent endpoint |
| [`roark agent endpoint get`](/cli-reference/commands/agent-endpoint-get) | Get agent endpoint by ID |
| [`roark agent endpoint list`](/cli-reference/commands/agent-endpoint-list) | List agent endpoints |
| [`roark agent endpoint update`](/cli-reference/commands/agent-endpoint-update) | Update an agent endpoint |
| [`roark agent get`](/cli-reference/commands/agent-get) | Get agent by ID |
| [`roark agent list`](/cli-reference/commands/agent-list) | List agents |
| [`roark agent prompt list`](/cli-reference/commands/agent-prompt-list) | List an agent's prompts |
| [`roark agent prompt update`](/cli-reference/commands/agent-prompt-update) | Set an agent's prompt |
| [`roark agent prompt version list`](/cli-reference/commands/agent-prompt-version-list) | List a prompt's versions |
| [`roark agent update`](/cli-reference/commands/agent-update) | Update an agent |
| [`roark agent version get`](/cli-reference/commands/agent-version-get) | Get an agent version |
| [`roark agent version list`](/cli-reference/commands/agent-version-list) | List an agent's versions |
| [`roark agent version rollback`](/cli-reference/commands/agent-version-rollback) | Roll an agent back to a version |

### `agent-config`

| Command | Description |
| - | - |
| [`roark agent-config get`](/cli-reference/commands/agent-config-get) | Get a managed config |
| [`roark agent-config list`](/cli-reference/commands/agent-config-list) | List managed configs |
| [`roark agent-config promote`](/cli-reference/commands/agent-config-promote) | Promote staging to production |
| [`roark agent-config resolve`](/cli-reference/commands/agent-config-resolve) | Resolve config for a session |
| [`roark agent-config staging delete`](/cli-reference/commands/agent-config-staging-delete) | Discard the staging revision |
| [`roark agent-config update`](/cli-reference/commands/agent-config-update) | Write a config revision |

### `analytics`

| Command | Description |
| - | - |
| [`roark analytics query`](/cli-reference/commands/analytics-query) | Run an analytics query |
| [`roark analytics schema get`](/cli-reference/commands/analytics-schema-get) | Get the analytics schema |

### `autoimprove`

| Command | Description |
| - | - |
| [`roark autoimprove job answer`](/cli-reference/commands/autoimprove-job-answer) | Answer Roark's question |
| [`roark autoimprove job cancel`](/cli-reference/commands/autoimprove-job-cancel) | Cancel a job |
| [`roark autoimprove job create`](/cli-reference/commands/autoimprove-job-create) | Start an Autoimprove job |
| [`roark autoimprove job dismiss`](/cli-reference/commands/autoimprove-job-dismiss) | Dismiss a verified job |
| [`roark autoimprove job get`](/cli-reference/commands/autoimprove-job-get) | Get an Autoimprove job |
| [`roark autoimprove job guidance send`](/cli-reference/commands/autoimprove-job-guidance-send) | Send guidance to a running job |
| [`roark autoimprove job list`](/cli-reference/commands/autoimprove-job-list) | List Autoimprove jobs |
| [`roark autoimprove job promote`](/cli-reference/commands/autoimprove-job-promote) | Promote a verified job to production |

### `benchmark`

| Command | Description |
| - | - |
| [`roark benchmark leaderboard get`](/cli-reference/commands/benchmark-leaderboard-get) | Get the benchmark leaderboard |
| [`roark benchmark metric list`](/cli-reference/commands/benchmark-metric-list) | List benchmark metrics |
| [`roark benchmark suite list`](/cli-reference/commands/benchmark-suite-list) | List benchmark suites |
| [`roark benchmark target get`](/cli-reference/commands/benchmark-target-get) | Get a benchmark target |
| [`roark benchmark target history list`](/cli-reference/commands/benchmark-target-history-list) | List a target’s published generations |
| [`roark benchmark target score-sample list`](/cli-reference/commands/benchmark-target-score-sample-list) | List score samples for a target |

### `calendar-connection`

| Command | Description |
| - | - |
| [`roark calendar-connection check`](/cli-reference/commands/calendar-connection-check) | Check a calendar connection |
| [`roark calendar-connection create`](/cli-reference/commands/calendar-connection-create) | Connect a calendar account |
| [`roark calendar-connection delete`](/cli-reference/commands/calendar-connection-delete) | Delete a calendar connection |
| [`roark calendar-connection get`](/cli-reference/commands/calendar-connection-get) | Get a calendar connection |
| [`roark calendar-connection list`](/cli-reference/commands/calendar-connection-list) | List calendar connections |
| [`roark calendar-connection target get`](/cli-reference/commands/calendar-connection-target-get) | Read one bookable target |
| [`roark calendar-connection target list`](/cli-reference/commands/calendar-connection-target-list) | List what can be booked on a calendar |
| [`roark calendar-connection update`](/cli-reference/commands/calendar-connection-update) | Update a calendar connection |

### `call`

| Command | Description |
| - | - |
| [`roark call batch delete`](/cli-reference/commands/call-batch-delete) | Delete calls in batch |
| [`roark call cost get`](/cli-reference/commands/call-cost-get) | Get call cost |
| [`roark call create`](/cli-reference/commands/call-create) | Create a call |
| [`roark call delete`](/cli-reference/commands/call-delete) | Delete a call |
| [`roark call get`](/cli-reference/commands/call-get) | Get a call by ID |
| [`roark call list`](/cli-reference/commands/call-list) | List calls |
| [`roark call metric list`](/cli-reference/commands/call-metric-list) | List call metrics |
| [`roark call property update`](/cli-reference/commands/call-property-update) | Update the properties of a call |
| [`roark call sentiment-run list`](/cli-reference/commands/call-sentiment-run-list) | List call sentiment runs |
| [`roark call tool-invocation append`](/cli-reference/commands/call-tool-invocation-append) | Append tool invocations to a call |
| [`roark call transcript get`](/cli-reference/commands/call-transcript-get) | Get call transcript |

### `config`

| Command | Description |
| - | - |
| [`roark config apply`](/cli-reference/commands/config-apply) | Reconcile the project to a config directory (or bundle). Previews and confirms first. |
| [`roark config diff`](/cli-reference/commands/config-diff) | Preview the changes a config directory (or bundle) would make. No writes. |

### `credential`

| Command | Description |
| - | - |
| [`roark credential create`](/cli-reference/commands/credential-create) | Create a personal credential |
| [`roark credential list`](/cli-reference/commands/credential-list) | List your personal credentials |
| [`roark credential revoke`](/cli-reference/commands/credential-revoke) | Revoke one of your personal credentials |

### `customer-flow`

| Command | Description |
| - | - |
| [`roark customer-flow create`](/cli-reference/commands/customer-flow-create) | Create a customer flow |
| [`roark customer-flow delete`](/cli-reference/commands/customer-flow-delete) | Delete a customer flow |
| [`roark customer-flow duplicate`](/cli-reference/commands/customer-flow-duplicate) | Duplicate a customer flow |
| [`roark customer-flow edge-case create`](/cli-reference/commands/customer-flow-edge-case-create) | Add an edge case |
| [`roark customer-flow edge-case delete`](/cli-reference/commands/customer-flow-edge-case-delete) | Remove an edge case |
| [`roark customer-flow edge-case promote`](/cli-reference/commands/customer-flow-edge-case-promote) | Promote an edge case to the happy path |
| [`roark customer-flow edge-case update`](/cli-reference/commands/customer-flow-edge-case-update) | Update an edge case |
| [`roark customer-flow get`](/cli-reference/commands/customer-flow-get) | Get customer flow by ID |
| [`roark customer-flow graph replace`](/cli-reference/commands/customer-flow-graph-replace) | Replace a scripted flow's steps |
| [`roark customer-flow happy-path update`](/cli-reference/commands/customer-flow-happy-path-update) | Update a flow's happy path |
| [`roark customer-flow list`](/cli-reference/commands/customer-flow-list) | List customer flows |
| [`roark customer-flow update`](/cli-reference/commands/customer-flow-update) | Update a customer flow |

### `dashboard`

| Command | Description |
| - | - |
| [`roark dashboard create`](/cli-reference/commands/dashboard-create) | Create a dashboard |
| [`roark dashboard data get`](/cli-reference/commands/dashboard-data-get) | Get dashboard data |
| [`roark dashboard delete`](/cli-reference/commands/dashboard-delete) | Delete a dashboard |
| [`roark dashboard get`](/cli-reference/commands/dashboard-get) | Get a dashboard |
| [`roark dashboard list`](/cli-reference/commands/dashboard-list) | List dashboards |
| [`roark dashboard update`](/cli-reference/commands/dashboard-update) | Update a dashboard |
| [`roark dashboard widget create`](/cli-reference/commands/dashboard-widget-create) | Add a widget |
| [`roark dashboard widget delete`](/cli-reference/commands/dashboard-widget-delete) | Remove a widget |
| [`roark dashboard widget update`](/cli-reference/commands/dashboard-widget-update) | Replace a widget |
| [`roark dashboard widget-data get`](/cli-reference/commands/dashboard-widget-data-get) | Get widget data |

### `health`

| Command | Description |
| - | - |
| [`roark health`](/cli-reference/commands/health) | Get API health status |

### `http-request-definition`

| Command | Description |
| - | - |
| [`roark http-request-definition create`](/cli-reference/commands/http-request-definition-create) | Create HTTP request definition |
| [`roark http-request-definition get`](/cli-reference/commands/http-request-definition-get) | Get HTTP request definition by ID |
| [`roark http-request-definition list`](/cli-reference/commands/http-request-definition-list) | List HTTP request definitions |
| [`roark http-request-definition update`](/cli-reference/commands/http-request-definition-update) | Update HTTP request definition |

### `live-report`

| Command | Description |
| - | - |
| [`roark live-report create`](/cli-reference/commands/live-report-create) | Create a live report |
| [`roark live-report delete`](/cli-reference/commands/live-report-delete) | Delete a live report |
| [`roark live-report document get`](/cli-reference/commands/live-report-document-get) | Get a report document |
| [`roark live-report document list`](/cli-reference/commands/live-report-document-list) | List report documents |
| [`roark live-report get`](/cli-reference/commands/live-report-get) | Get a live report |
| [`roark live-report list`](/cli-reference/commands/live-report-list) | List live reports |
| [`roark live-report run`](/cli-reference/commands/live-report-run) | Generate a live report document |
| [`roark live-report update`](/cli-reference/commands/live-report-update) | Update a live report |

### `metric`

| Command | Description |
| - | - |
| [`roark metric collection-job cancel`](/cli-reference/commands/metric-collection-job-cancel) | Cancel a metric collection job |
| [`roark metric collection-job create`](/cli-reference/commands/metric-collection-job-create) | Create and run a metric collection job |
| [`roark metric collection-job get`](/cli-reference/commands/metric-collection-job-get) | Get metric collection job by ID |
| [`roark metric collection-job list`](/cli-reference/commands/metric-collection-job-list) | List metric collection jobs |
| [`roark metric definition create`](/cli-reference/commands/metric-definition-create) | Create custom metric definition |
| [`roark metric definition list`](/cli-reference/commands/metric-definition-list) | List metric definitions |
| [`roark metric definition update`](/cli-reference/commands/metric-definition-update) | Update a metric definition |
| [`roark metric policy create`](/cli-reference/commands/metric-policy-create) | Create a metric policy |
| [`roark metric policy delete`](/cli-reference/commands/metric-policy-delete) | Delete a metric policy |
| [`roark metric policy get`](/cli-reference/commands/metric-policy-get) | Get metric policy by ID |
| [`roark metric policy list`](/cli-reference/commands/metric-policy-list) | List metric policies |
| [`roark metric policy update`](/cli-reference/commands/metric-policy-update) | Update a metric policy |
| [`roark metric variant create`](/cli-reference/commands/metric-variant-create) | Create a metric variant |
| [`roark metric variant delete`](/cli-reference/commands/metric-variant-delete) | Delete a metric variant |
| [`roark metric variant get`](/cli-reference/commands/metric-variant-get) | Get a metric variant |
| [`roark metric variant list`](/cli-reference/commands/metric-variant-list) | List a metric’s variants |
| [`roark metric variant update`](/cli-reference/commands/metric-variant-update) | Update a metric variant |

### `phone-number`

| Command | Description |
| - | - |
| [`roark phone-number allowance get`](/cli-reference/commands/phone-number-allowance-get) | Get the phone number allowance |
| [`roark phone-number available list`](/cli-reference/commands/phone-number-available-list) | Search available phone numbers |
| [`roark phone-number create`](/cli-reference/commands/phone-number-create) | Buy a phone number |
| [`roark phone-number delete`](/cli-reference/commands/phone-number-delete) | Release a phone number |
| [`roark phone-number list`](/cli-reference/commands/phone-number-list) | List phone numbers |
| [`roark phone-number update`](/cli-reference/commands/phone-number-update) | Update a phone number |

### `project`

| Command | Description |
| - | - |
| [`roark project list`](/cli-reference/commands/project-list) | List projects available to this credential |

### `simulation`

| Command | Description |
| - | - |
| [`roark simulation environment create`](/cli-reference/commands/simulation-environment-create) | Create an environment |
| [`roark simulation environment delete`](/cli-reference/commands/simulation-environment-delete) | Delete an environment |
| [`roark simulation environment get`](/cli-reference/commands/simulation-environment-get) | Get environment by ID |
| [`roark simulation environment list`](/cli-reference/commands/simulation-environment-list) | List environments |
| [`roark simulation environment update`](/cli-reference/commands/simulation-environment-update) | Update an environment |
| [`roark simulation group create`](/cli-reference/commands/simulation-group-create) | Create a simulation group |
| [`roark simulation group delete`](/cli-reference/commands/simulation-group-delete) | Delete a simulation group |
| [`roark simulation group get`](/cli-reference/commands/simulation-group-get) | Get a simulation group |
| [`roark simulation group list`](/cli-reference/commands/simulation-group-list) | List simulation groups |
| [`roark simulation group run`](/cli-reference/commands/simulation-group-run) | Run a simulation group |
| [`roark simulation group update`](/cli-reference/commands/simulation-group-update) | Update a simulation group |
| [`roark simulation job get`](/cli-reference/commands/simulation-job-get) | Get simulation by ID |
| [`roark simulation job lookup`](/cli-reference/commands/simulation-job-lookup) | Lookup by phone number |
| [`roark simulation job tool-mock list`](/cli-reference/commands/simulation-job-tool-mock-list) | List mocked tool invocations |
| [`roark simulation persona create`](/cli-reference/commands/simulation-persona-create) | Create a new persona |
| [`roark simulation persona get`](/cli-reference/commands/simulation-persona-get) | Get persona by ID |
| [`roark simulation persona list`](/cli-reference/commands/simulation-persona-list) | List personas |
| [`roark simulation persona update`](/cli-reference/commands/simulation-persona-update) | Update a persona |
| [`roark simulation plan create`](/cli-reference/commands/simulation-plan-create) | Create a run plan |
| [`roark simulation plan delete`](/cli-reference/commands/simulation-plan-delete) | Delete a run plan |
| [`roark simulation plan get`](/cli-reference/commands/simulation-plan-get) | Get run plan by ID |
| [`roark simulation plan job cancel`](/cli-reference/commands/simulation-plan-job-cancel) | Cancel a simulation plan job |
| [`roark simulation plan job cost get`](/cli-reference/commands/simulation-plan-job-cost-get) | Get simulation run cost |
| [`roark simulation plan job debrief create`](/cli-reference/commands/simulation-plan-job-debrief-create) | Generate simulation run debrief |
| [`roark simulation plan job get`](/cli-reference/commands/simulation-plan-job-get) | Get simulation plan job |
| [`roark simulation plan job list`](/cli-reference/commands/simulation-plan-job-list) | List simulation plan jobs |
| [`roark simulation plan job result get`](/cli-reference/commands/simulation-plan-job-result-get) | Get simulation run results |
| [`roark simulation plan job start`](/cli-reference/commands/simulation-plan-job-start) | Run a simulation plan |
| [`roark simulation plan list`](/cli-reference/commands/simulation-plan-list) | List run plans |
| [`roark simulation plan update`](/cli-reference/commands/simulation-plan-update) | Update a run plan |
| [`roark simulation run`](/cli-reference/commands/simulation-run) | Run a simulation |
| [`roark simulation run-group create`](/cli-reference/commands/simulation-run-group-create) | Create a run group |
| [`roark simulation run-group delete`](/cli-reference/commands/simulation-run-group-delete) | Delete a run group |
| [`roark simulation run-group get`](/cli-reference/commands/simulation-run-group-get) | Get a run group |
| [`roark simulation run-group list`](/cli-reference/commands/simulation-run-group-list) | List run groups |
| [`roark simulation run-group report regenerate`](/cli-reference/commands/simulation-run-group-report-regenerate) | Regenerate a run group report |
| [`roark simulation template list`](/cli-reference/commands/simulation-template-list) | List simulation templates |
| [`roark simulation tool mock`](/cli-reference/commands/simulation-tool-mock) | Mock a tool call during a test call |
| [`roark simulation tool-fixture create`](/cli-reference/commands/simulation-tool-fixture-create) | Set a tool fixture |
| [`roark simulation tool-fixture delete`](/cli-reference/commands/simulation-tool-fixture-delete) | Delete a tool fixture |
| [`roark simulation tool-fixture list`](/cli-reference/commands/simulation-tool-fixture-list) | List tool fixtures |
| [`roark simulation tool-fixture update`](/cli-reference/commands/simulation-tool-fixture-update) | Update a tool fixture |

### `tool`

| Command | Description |
| - | - |
| [`roark tool create`](/cli-reference/commands/tool-create) | Create a tool |
| [`roark tool delete`](/cli-reference/commands/tool-delete) | Delete a tool |
| [`roark tool get`](/cli-reference/commands/tool-get) | Get a tool |
| [`roark tool list`](/cli-reference/commands/tool-list) | List tools |
| [`roark tool update`](/cli-reference/commands/tool-update) | Update a tool |

### `usage`

| Command | Description |
| - | - |
| [`roark usage get`](/cli-reference/commands/usage-get) | Get usage & cost |

### `webhook`

| Command | Description |
| - | - |
| [`roark webhook create`](/cli-reference/commands/webhook-create) | Create webhook |
| [`roark webhook delete`](/cli-reference/commands/webhook-delete) | Delete webhook |
| [`roark webhook get`](/cli-reference/commands/webhook-get) | Get webhook by ID |
| [`roark webhook list`](/cli-reference/commands/webhook-list) | List webhooks |

### `whoami`

| Command | Description |
| - | - |
| [`roark whoami`](/cli-reference/commands/whoami) | Describe the current credential |


# Code-first agents (LiveKit, Pipecat)
Source: https://docs.roark.ai/documentation/autoimprove/code-first-agents

Run Autoimprove on agents you built in code: externalize the tunable surface into a Roark-managed config, and Roark improves it against your real production deployment

A LiveKit or Pipecat agent is your code on your infrastructure, so there is no provider document for Roark to edit. The managed-config integration turns that around: your agent fetches its tunable surface (prompt, model, params, voice) from Roark at session start, and Autoimprove works on that config with the same loop, safety model, and promote gate as provider agents.

The key mechanic is **per-session resolution**: Roark's test calls dial your production deployment directly. Because Roark originated those calls, the config endpoint recognizes them and serves the **staging** revision for those sessions only. Real callers always get production config; you deploy nothing extra.

## 1. Fetch config at session start

One call registers the key on first use (your defaults become revision 1) and resolves the right revision per session.

```python LiveKit Agents theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import os
from livekit import agents
from livekit.agents import AgentSession
from livekit.plugins import openai
from roark_analytics import Roark

roark = Roark(bearer_token=os.environ["ROARK_API_BEARER_TOKEN"])

DEFAULTS = {
    "system_prompt": "You are a helpful scheduling assistant for Mary's Dental.",
    "model": "gpt-4.1",
    "temperature": 0.4,
    # Externalize your STT and TTS choices too: whatever lives in this
    # document, Roark can tune. Whatever stays in code, it cannot.
    "tts": {"voice_id": "alloy", "speed": 1.0},
    "stt": {"language": "en"},
}

async def entrypoint(ctx: agents.JobContext):
    await ctx.connect()
    participant = await ctx.wait_for_participant()

    # SIP attributes carry the numbers on telephony rooms; adapt to your setup.
    caller = participant.attributes.get("sip.phoneNumber")
    called = participant.attributes.get("sip.trunkPhoneNumber")

    cfg = roark.agent_config.resolve(
        "support-agent",
        defaults=DEFAULTS,
        session={"session_id": ctx.room.name, "caller_number": caller, "called_number": called},
    ).data
    doc = cfg.document

    # Stamp attribution so Roark ties this session to the right revision
    # (and, for its own test calls, to the simulation that placed them).
    await ctx.room.local_participant.set_attributes(
        {"roark.revisionId": cfg.revision_id, "roark.channel": cfg.channel}
    )

    session = AgentSession(
        llm=openai.LLM(model=doc["model"], temperature=doc["temperature"]),
    )
    await session.start(agent=agents.Agent(instructions=doc["system_prompt"]), room=ctx.room)
```

```python Pipecat theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import os
from pipecat.services.openai.llm import OpenAILLMService
from roark_analytics import Roark

roark = Roark(bearer_token=os.environ["ROARK_API_BEARER_TOKEN"])

DEFAULTS = {
    "system_prompt": "You are a helpful scheduling assistant for Mary's Dental.",
    "model": "gpt-4.1",
    "temperature": 0.4,
    # Externalize your STT and TTS choices too: whatever lives in this
    # document, Roark can tune. Whatever stays in code, it cannot.
    "tts": {"voice_id": "alloy", "speed": 1.0},
    "stt": {"language": "en"},
}

async def run_bot(transport, call_info):
    # call_info: however your transport surfaces the dial-in numbers.
    cfg = roark.agent_config.resolve(
        "support-agent",
        defaults=DEFAULTS,
        session={
            "session_id": call_info.session_id,
            "caller_number": call_info.caller_number,
            "called_number": call_info.called_number,
        },
    ).data
    doc = cfg.document

    llm = OpenAILLMService(model=doc["model"], params={"temperature": doc["temperature"]})
    messages = [{"role": "system", "content": doc["system_prompt"]}]
    # ... build your pipeline as usual, and include cfg.revision_id and
    # cfg.channel in the call properties you report to Roark.
```

> [!NOTE]
>
> Cache the resolved config for the session; do not fetch per turn. Wrap the resolve in a try/except that falls back to your `DEFAULTS`: the failure mode is then your current behavior, never a broken call. The SDK method ships in the next `roark_analytics` release; until you upgrade, the same call is one `POST /v1/agent-config/{key}/resolve` request.

## 2. Link the config to your Roark agent

Autoimprove finds the config through the agent your calls already report. Link once:

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent-config update support-agent \
  --channel production \
  --document @config.json \
  --agent-id "$AGENT_ID"
```

```bash API theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X PUT https://api.roark.ai/v1/agent-config/support-agent \
  -H "Authorization: Bearer $ROARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"channel": "production", "document": {"system_prompt": "...", "model": "gpt-4.1", "temperature": 0.4}, "agentId": "'"$AGENT_ID"'"}'
```

Your agent also needs a registered phone endpoint (the number Roark's simulations dial): `roark agent endpoint create` if you have not already.

## 3. Run Autoimprove

Exactly the same as any other agent:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job create \
  --agent-id "$AGENT_ID" \
  --objective-metric-definition-id "$METRIC_ID" \
  --objective-label "Booking completion should pass" \
  --target-value 90
```

What happens under the hood differs only in the staging story: instead of cloning a shadow, Roark **forks a staging config channel** from production. Its test calls dial your production number, get recognized, and read the staging revision; every change Roark makes is a new audited revision on that channel. Real traffic never sees any of it.

When the job reaches `AWAITING_PROMOTE`, promote is a pointer swap: your very next session reads the verified revision. Rolling back is writing the prior revision back to production; the chain preserves every state.

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job promote <job-id>
```

## 4. Guard your tools (optional)

Roark's test calls dial your production deployment, which means your real tools would run: real bookings, real SMS, real transfers. The tool guard fixes that. Wrap a side-effecting tool once; during a Roark test call it never executes. Instead, Roark answers with a simulated backend response that is shaped by the tool's contract, consistent with the test scenario (a scenario about a fully booked calendar gets a realistic "no availability" from a booking tool), and consistent with earlier tool responses in the same call. You configure nothing per tool. Real callers always hit the real tool: the guard only diverts when the resolve response identified the session as a Roark simulation, and the endpoint independently re-validates the simulation before answering.

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import httpx

def roark_tool_guard(cfg, fallback=None):
    """Wrap a side-effecting tool. During a Roark test call the tool never
    executes; Roark answers with a simulated backend response instead."""
    def decorator(tool_fn):
        async def wrapper(**kwargs):
            if cfg.simulation_job_id is None:
                return await tool_fn(**kwargs)  # real caller: real tool
            try:
                response = httpx.post(
                    "https://api.roark.ai/v1/simulation/tool-mock",
                    headers={"Authorization": f"Bearer {os.environ['ROARK_API_BEARER_TOKEN']}"},
                    json={
                        "simulationJobId": cfg.simulation_job_id,
                        "toolName": tool_fn.__name__,
                        "toolDescription": tool_fn.__doc__,
                        "arguments": kwargs,
                    },
                    timeout=5.0,
                )
                if response.status_code == 404:
                    # Not a known simulation after all: treat the session as real.
                    return await tool_fn(**kwargs)
                response.raise_for_status()
                return response.json()["data"]["result"]
            except httpx.HTTPError:
                # Never run the real tool for a session resolve called a test:
                # degrade to a static stub instead.
                return fallback if fallback is not None else {"status": "ok", "mocked": True}
        wrapper.__name__ = tool_fn.__name__
        wrapper.__doc__ = tool_fn.__doc__
        return wrapper
    return decorator

# After resolving cfg at session start:
@roark_tool_guard(cfg, fallback={"status": "booked", "confirmationId": "TEST-0000"})
async def book_appointment(date: str, time: str) -> dict:
    """Books an appointment. Args: date (YYYY-MM-DD), time (HH:MM). Returns {confirmationId, status}."""
    ...  # your real calendar write: never runs during a Roark test call
```

Wrap the writes (`book`, `cancel`, `send_sms`, `transfer`, `charge`); leave read-only lookups unwrapped when live data makes the test more realistic. Pass the richest `toolDescription` you can: the contract is what shapes the simulated response. In LiveKit Agents, wrap the function before registering it with `@function_tool`; in Pipecat, wrap the handler before registering it on the LLM service. Identical retries within a few minutes return the stored response, so double-fired handlers stay consistent, and every mocked invocation is recorded against the simulation as evidence of what your agent tried to call.

The failure contract, precisely: `404` means the simulation id is unknown (treat the session as real), `409` means the simulation already ended (do NOT run the real tool; use your fallback), and any other error also means use your fallback. The `roark_analytics` SDK gains a first-class `simulation.mock_tool(...)` method in its next release; the decorator above works today. The guard is a simulations-wide feature, not an Autoimprove one: every Roark test call against your deployment gets safe tools, you can pin deterministic responses per tool with fixtures, and you can read back exactly what your agent called after each run. The full workflow, including CI assertions and automatic grading, is in the [Tool Call Testing recipe](/documentation/recipes/tool-call-testing).

## What Roark can and cannot improve here

Roark can change anything in the document you externalized: prompt, model and sampling params, and your STT/TTS choices (voice, speed, language) when you include them, as the example does. It keeps the key shape your code reads (it never renames keys). What stays in code (tool logic, pipeline structure) is out of reach by design; the job's final report recommends those changes for a human when the config surface is not enough. Guarded tools are still out of reach for changes, but their invocations become testable: Roark sees what your agent tried to call with what arguments, with no real side effects.


# Overview
Source: https://docs.roark.ai/documentation/autoimprove/overview

Give Roark an objective and it improves your voice agent for you: it stages a copy, builds a test suite, changes the configuration, and proves the improvement with real simulated calls before you promote it

Autoimprove is Roark working on your agent instead of just measuring it. You pick one objective, a metric that should pass and the pass rate you want, and Roark runs the whole improvement loop autonomously: it clones your agent into a staging copy, writes caller scenarios that exercise the objective, measures a baseline over real simulated phone calls, changes the staging configuration, and re-tests until the objective holds. Production changes exactly once, when you promote a verified job's changes.

## Quickstart

Three commands with the [Roark CLI](/documentation/sdks/cli). Find your agent and the metric that defines success, then hand Roark the objective:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark agent list
roark metric definitions list

roark autoimprove job create \
  --agent-id "$AGENT_ID" \
  --objective-metric-definition-id "$METRIC_ID" \
  --objective-label "Consent collection should pass" \
  --target-value 90
```

That is the whole job description. Roark takes it from there; watch it work with:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job get <job-id>
```

The response carries the live worklog: every step, every batch of test calls with its pass rate, and Roark's own running notes. The same job is live in the dashboard under **Autoimprove**, where the worklog renders as a feed.

> [!NOTE]
>
> The objective metric should produce a pass/fail verdict per call: either a boolean metric, or a threshold on a scale metric (for example "Instruction Follow >= 4"). Create one in the dashboard's metric studio or with `roark metric definitions create`.

***

## How it works

1. **Stage.** Roark clones your production agent inside your provider account (Vapi, Retell, or ElevenLabs) as a clearly-marked shadow and registers it for simulations: phone-based providers get their own number, ElevenLabs shadows are reached over the ConvAI realtime channel. Your production agent is never modified by the loop.
2. **Author.** It writes caller scenarios for the objective: the failure case it is hunting, an easy case, an edge case. Each becomes a real simulation flow with success criteria, graded by your metrics.
3. **Measure.** A baseline batch of simulated calls establishes where things stand. Every later change is measured against it.
4. **Change and re-test.** Roark edits the staging configuration through an audited funnel, then proves or disproves each change with more calls. It reads the graders' evidence, not its own opinion of the transcripts.
5. **Verify and hand over.** When the objective passes its target with repeated trials, the job waits for you: review the diffs and the numbers, then promote or dismiss.

## What Roark can change

The whole provider configuration document, not just the prompt: the system prompt, the underlying model and temperature, the voice, the transcriber, timeouts and endpointing, the first message, tool definitions. Credentials, URLs, webhooks, and privacy settings are protected and refused by the platform.

## The safety model

> [card] Staging-only writes
>
> The loop only ever writes to the staging copy. The one write path to production is the promote, and only you trigger it.

> [card] Rollbackable promotes
>
> A snapshot of the production configuration is taken immediately before a promote, so every promote can be fully rolled back.

> [card] Guaranteed cleanup
>
> Whatever the outcome, everything the job set up is removed: the shadow agent, its phone number, the authored test flows and run plan.

> [card] Hard budgets
>
> Decision turns and simulated calls are capped (50 turns and 200 calls by default, tunable at create). Roark cannot exceed them.

## Job statuses

| Status | Meaning |
| - | - |
| `RUNNING` | The loop is working: authoring, changing, or validating. |
| `NEEDS_INPUT` | Roark is blocked on a question to you. Answer it and the loop resumes; after 24 hours it proceeds with the stated default. |
| `AWAITING_PROMOTE` | Verified. The staged changes passed the target and wait for your review. |
| `PROMOTED` | You promoted; the changes are live on production. |
| `NO_FIX` | Concluded without a change to ship. This includes the positive case where the baseline already met the target. |
| `CANCELLED` | Stopped by request. Production was never changed. |
| `FAILED` | Stopped by an error. Production was never changed, and resources were cleaned up. |

## Go deeper

> [card] [Running a job](/documentation/autoimprove/running-a-job)
>
> The full lifecycle over the CLI and API: creating, watching, steering, answering, promoting.

> [card] [CLI](/documentation/sdks/cli)
>
> Install and authenticate the Roark CLI used throughout this section.


# Running a job
Source: https://docs.roark.ai/documentation/autoimprove/running-a-job

The full Autoimprove lifecycle over the CLI and API: create a job, watch the worklog, steer and answer, then promote or dismiss

Everything the dashboard can do with a job, the CLI and API can do too. The examples below lead with the CLI; every command has a matching REST endpoint under `/v1/autoimprove/job`.

## Create a job

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job create \
  --agent-id "$AGENT_ID" \
  --objective-metric-definition-id "$METRIC_ID" \
  --objective-label "Consent collection should pass" \
  --target-value 90
```

```bash API theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/autoimprove/job \
  -H "Authorization: Bearer $ROARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "agentId": "'"$AGENT_ID"'",
    "objectiveMetricDefinitionId": "'"$METRIC_ID"'",
    "objectiveLabel": "Consent collection should pass",
    "targetValue": 90
  }'
```

Returns `201` with the job, already `RUNNING`. Useful optional fields:

| Field | What it does |
| - | - |
| `stagingAgentId` | Stage changes on an existing agent of yours instead of the default shadow clone. Must differ from `agentId` and share its provider. |
| `customerIntegrationId` | Pin the provider integration whose credentials Roark uses (Vapi, Retell, or ElevenLabs). Defaults to the project's active integration for the agent's provider. The integration must have agent config writes enabled. |
| `validationRunPlanId` | Bring your own simulation run plan instead of letting Roark author a suite. |
| `maxIterations`, `maxSimCalls` | Tighten or widen the hard budgets (defaults 50 and 200). |

One live job per agent: creating a second returns `409`.

> [!WARNING]
>
> The job starts spending real resources immediately: a shadow agent and phone number in your provider account, and simulated phone calls that bill like any other simulation. The budgets cap the total.

## Watch it work

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job get <job-id>
roark autoimprove job list
```

```bash API theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl https://api.roark.ai/v1/autoimprove/job/$JOB_ID \
  -H "Authorization: Bearer $ROARK_API_KEY"
```

The single-job response includes `logEntries`, the full worklog oldest-first. The kinds worth watching for:

| Kind | Meaning |
| - | - |
| `SETUP` | The staging checklist: shadow ready, number assigned, reachable. |
| `SUITE` | Roark authored its validation scenarios. |
| `CHANGE` | A configuration change applied to staging (`configPushId` links the audited diff). |
| `VALIDATION` | A batch of test calls settled: `trialCount`, `passRateBefore`, `passRateAfter`. |
| `QUESTION` | Roark is blocked on you; `questionOptions` carries its suggested answers. |
| `VERDICT` | The conclusion and why. |
| `STATUS` | Everything else it wants you to know, including cleanup confirmations. |

Poll the job until `status` leaves `RUNNING`. The `workingMemory` field is Roark's own running notes; `finalReport` is its closing story once concluded.

## Steer and answer

Send guidance any time the job is live; it binds Roark's next decision:

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job guidance send <job-id> \
  --text "Keep the current voice; focus on the closing confirmation."
```

```bash API theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/autoimprove/job/$JOB_ID/guidance \
  -H "Authorization: Bearer $ROARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text": "Keep the current voice; focus on the closing confirmation."}'
```

When `status` is `NEEDS_INPUT`, answer the open question (free text or one of the offered options). Unanswered questions proceed with Roark's stated default after 24 hours:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job answer <job-id> --text "Prefer accuracy over latency"
```

Answering a job that is not waiting returns `409`.

## Promote or dismiss

A verified job (`AWAITING_PROMOTE`) holds its staged changes until you decide:

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job promote <job-id>
roark autoimprove job dismiss <job-id>
```

```bash API theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/autoimprove/job/$JOB_ID/promote \
  -H "Authorization: Bearer $ROARK_API_KEY"
```

Promote applies the verified operations to the production agent through the same audited funnel, with a pre-promote snapshot for rollback, then cleans up the staging resources. Dismiss discards everything; production is never touched. Both return `409` unless the job is awaiting review.

## Cancel

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark autoimprove job cancel <job-id>
```

Stops a live job. Cleanup is automatic: the shadow agent, its phone number, and the authored suite are removed. Production is never changed by a cancel.

> [!NOTE]
>
> Cross-tenant and unknown job ids are indistinguishable: both return `404`.


# Agents
Source: https://docs.roark.ai/documentation/config-as-code/agents

Define voice agents as config

A voice agent: its prompt and the lines it can be reached on. Thin by design; endpoints are nested and reconciled with the agent. References nothing.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { readFileSync } from 'node:fs';
import { defineAgent } from '@roarkanalytics/sdk/lib/config';

const frontdesk = defineAgent({
  name: 'frontdesk',
  description: 'Bright Smiles front desk booking agent',
  customId: 'bright-smiles-frontdesk',
  prompt: readFileSync('./prompts/frontdesk.md', 'utf8'),
  endpoints: [{ value: '+15551234567', direction: 'INCOMING_AND_OUTGOING' }],
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: agent
name: frontdesk
description: Bright Smiles front desk booking agent
customId: bright-smiles-frontdesk
prompt: file://prompts/frontdesk.md
endpoints:
  - value: '+15551234567'
    direction: INCOMING_AND_OUTGOING
```

* `name` is the local key (`^[a-z0-9][a-z0-9-_.]*$`) and, with the kind, forms the config identity.
* `displayName` (optional) is the agent's name as it appears in Roark, where spaces and capitals are allowed; it defaults to `name`. Set it to an agent's exact existing name to bring an agent you created in the app under config (apply takes it over in place instead of creating a duplicate) when that name isn't slug-shaped, for example `displayName: "Dental Clinic"` with `name: dental-clinic`.
* Other resources reference an agent by its `name` (the slug), not its `displayName` (e.g. a flow's `agents:` list, a collector's `AGENT` filter, or a [simulation plan's](/documentation/config-as-code/simulation-plans) `agentEndpoints`).

## Endpoints

An endpoint is identified by its `value`: the E.164 number for a phone line, the URL otherwise. That is the only field you always have to give.

| Field | Default | Notes |
| :- | :- | :- |
| `value` | | E.164 (`+12345678900`) for `PHONE`, the `ws://` or `wss://` URL for `WEBSOCKET` |
| `type` | `PHONE` | `PHONE` or `WEBSOCKET` |
| `direction` | `INCOMING` | `INCOMING` or `OUTGOING` |
| `environment` | `production` | |
| `outboundDialType` | `NONE` | `NONE` (Roark places the call) or `HTTP_REQUEST` (Roark asks your own system to place it) |
| `outboundDialHttpRequestDefinition` | | The `name` of an [HTTP request definition](/documentation/config-as-code/http-request-definitions). Required when `outboundDialType` is `HTTP_REQUEST`, and rejected otherwise |

An endpoint is identified by its **value and its direction together**, so one number can carry an inbound endpoint and an outbound one:

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
endpoints: [
  { value: '+15551234567', direction: 'INCOMING' },
  { value: '+15551234567', direction: 'OUTGOING' },
],
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
endpoints:
  - value: '+15551234567'
    direction: INCOMING
  - value: '+15551234567'
    direction: OUTGOING
```

That is how a number that both takes and places calls is set up: an endpoint is `INCOMING` or `OUTGOING`, never both.

Endpoints reconcile with the agent on that pair: one you add is created, one you change is updated, and one you remove is detached. Changing a `direction` is therefore a new endpoint and the retirement of the old one rather than an edit, which matches what it does to your call routing.

## Prompt

`prompt` is the agent's system prompt, kept in your repo and versioned in Roark. In TypeScript it's a plain string, so read it from a file (or write it inline); in YAML it takes an inline string or a `file://` reference resolved from your config root:

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
prompt: readFileSync('./prompts/frontdesk.md', 'utf8'),
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
prompt: file://prompts/frontdesk.md
```

Each apply versions it: a change appends a new version, identical content is a no-op. It lands in its own history on the agent's Prompts tab, shown as a read-only **Config as code** lineage, separate from prompts you edit in the app or that arrive on calls. Omitting `prompt` leaves any existing prompt alone: config sets a prompt, it never clears one you did not declare.

> [!NOTE]
>
> The prompt is kept for your reference and versioning only. Roark does not run your agent, and no metric evaluates against this prompt, so setting it does not change how your agent behaves or how it is scored.

> [!NOTE]
>
> Other endpoint types (LiveKit, SmallWebRTC, ElevenLabs, Kore, Google CES, Daily) are not configurable here. They are bidirectional, so they carry no direction, and each needs provider configuration the config format has no place for. Their endpoints are created by the integration that owns them.

> [!WARNING]
>
> Omitting the `endpoints` block leaves the agent's existing lines alone. Writing `endpoints: []` removes all of them. The difference matters, because it is the difference between saying nothing about your phone numbers and saying you want none.

## Outbound dialling

By default an `OUTGOING` endpoint is dialled by Roark. To hand dialling off to your own system, set `outboundDialType: HTTP_REQUEST` and point `outboundDialHttpRequestDefinition` at an [HTTP request definition](/documentation/config-as-code/http-request-definitions) by name. Roark fires that request (with the number to dial substituted in) when the call should be placed:

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
endpoints: [
  {
    value: '+15551234567',
    direction: 'OUTGOING',
    outboundDialType: 'HTTP_REQUEST',
    outboundDialHttpRequestDefinition: 'dialer', // a defineHTTPRequestDefinition resource
  },
],
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
endpoints:
  - value: '+15551234567'
    direction: OUTGOING
    outboundDialType: HTTP_REQUEST
    outboundDialHttpRequestDefinition: dialer   # a kind: httpRequestDefinition resource
```

Declare the `dialer` definition in the same bundle (or a previous apply); it reconciles before agents, so the reference resolves in one apply. See [HTTP request definitions](/documentation/config-as-code/http-request-definitions).

> [!NOTE]
>
> For the full field reference of every kind, see the [Config DSL reference](https://roark.ai/roark-config.schema.json) schema.


# Alerts
Source: https://docs.roark.ai/documentation/config-as-code/alerts

Define alerts (monitors) as config: threshold, event, and simulation triggers

An alert (monitor) reacts to a trigger by opening an issue and/or notifying Slack channels and webhooks. Pick one of three trigger types with `trigger.type`; the type is immutable once created (a change is delete-and-recreate).

`name` is the slug config identity. `displayName` (optional) is the alert's name as it appears in Roark, where spaces and capitals are allowed; it defaults to `name`. Set it to an alert's exact existing name to bring one you created in the app under config (apply takes it over instead of creating a duplicate) when that name isn't slug-shaped.

## Threshold

Watch a metric aggregate over a rolling window. Opens an issue when it crosses the threshold, and optionally notifies Slack/webhooks. `metric` is the metric's slug (a custom metric you defined, or a Roark system metric); the variant defaults to the metric's default.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineAlert } from '@roarkanalytics/sdk/lib/config';

const highFrustration = defineAlert({
  name: 'high-frustration',
  trigger: {
    type: 'threshold',
    metric: 'frustration_score', // metric slug (its metricId)
    aggregation: 'MEAN', // COUNT | RATE_PER_MINUTE | MEAN
    windowMinutes: 60,
    operator: 'GT', // GT | GTE | LT | LTE
    thresholdValue: 3,
    minSampleSize: 5, // optional; skip windows with fewer data points
  },
  actions: {
    slack: [{ channelId: 'C0123ABCXYZ', channelName: '#agent-quality' }], // Slack-native channel id
    webhooks: ['https://hooks.example.com/roark/alerts'],
  },
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: alert
name: high-frustration
trigger:
  type: threshold
  metric: frustration_score      # metric slug (its metricId)
  aggregation: MEAN              # COUNT | RATE_PER_MINUTE | MEAN
  windowMinutes: 60
  operator: GT                   # GT | GTE | LT | LTE
  thresholdValue: 3
  minSampleSize: 5               # optional; skip windows with fewer data points
actions:
  slack:
    - channelId: C0123ABCXYZ     # Slack-native channel id
      channelName: '#agent-quality'
  webhooks:
    - https://hooks.example.com/roark/alerts
```

## Event

Fire on platform events (call/chat analysis, simulation jobs, metric collection, issues).

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineAlert } from '@roarkanalytics/sdk/lib/config';

const analysisFailures = defineAlert({
  name: 'analysis-failures',
  trigger: {
    type: 'event',
    events: ['CALL_ANALYSIS_FAILED', 'CHAT_ANALYSIS_FAILED'],
  },
  actions: {
    slack: [{ channelId: 'C0123ABCXYZ', channelName: '#platform-alerts' }],
  },
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: alert
name: analysis-failures
trigger:
  type: event
  events:
    - CALL_ANALYSIS_FAILED
    - CHAT_ANALYSIS_FAILED
actions:
  slack:
    - channelId: C0123ABCXYZ
      channelName: '#platform-alerts'
```

## Simulation

Notify on a simulation plan's job outcome. `plan` (by name) scopes it to one plan; omit for project-wide. A simulation alert requires exactly one Slack channel and does not support webhooks.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineAlert } from '@roarkanalytics/sdk/lib/config';

const nightlySuiteResult = defineAlert({
  name: 'nightly-suite-result',
  trigger: {
    type: 'simulation',
    conditions: ['FAILURE', 'THRESHOLD_FAILED'], // SUCCESS | FAILURE | THRESHOLD_FAILED
    plan: 'nightly-regression',
    deliveryFormat: 'MESSAGE', // MESSAGE | PDF
  },
  actions: {
    slack: [{ channelId: 'C0999SIMS', channelName: '#sim-results' }],
  },
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: alert
name: nightly-suite-result
trigger:
  type: simulation
  conditions: [FAILURE, THRESHOLD_FAILED]   # SUCCESS | FAILURE | THRESHOLD_FAILED
  plan: nightly-regression
  deliveryFormat: MESSAGE                    # MESSAGE | PDF
actions:
  slack:
    - channelId: C0999SIMS
      channelName: '#sim-results'
```

> [!TIP]
>
> A simulation alert reports on one plan's run. To send a single combined report for several plans, with the PDF to email (including people without a Roark account) and to up to 10 Slack channels, use a [simulation group](/documentation/config-as-code/simulation-groups#who-gets-the-report).

* **Threshold alerts always open an issue**; Slack and webhook delivery are optional additions.
* **Slack channels** are referenced by their Slack-native `channelId` (e.g. `C0123ABC`) plus a `channelName`. The Slack workspace is resolved from the project's connected Slack integration, so no UUIDs go in config.
* **Webhooks** must be public `http(s)` URLs.

> [!NOTE]
>
> Referencing a Slack channel requires a **Slack integration connected to the project** in Roark first. Connect Slack under **Settings → Project → Notifications** (a one-time approval in Slack, which can't be done from config), then reference the channel by id here. In Slack, the id is at the bottom of the channel's **About** tab.

> [!NOTE]
>
> For the full field reference of every kind, see the [Config DSL reference](https://roark.ai/roark-config.schema.json) schema.


# Collectors
Source: https://docs.roark.ai/documentation/config-as-code/collectors

Decide which metrics get collected on which conversations

Decides which metrics get collected on which conversations (the config form of a [collector](/documentation/metrics/metric-collectors)). Its `metrics:` list references both custom metrics you defined and Roark's system metrics, all by slug.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineCollector } from '@roarkanalytics/sdk/lib/config';

const consentOnFrontdesk = defineCollector({
  name: 'consent-on-frontdesk',
  modality: 'call',
  status: 'ACTIVE',
  metrics: ['consent_collection_consent_obtained', 'refund_policy_accuracy'],
  filters: [
    {
      conditions: [{ type: 'AGENT', key: 'frontdesk', operator: 'EQUALS' }],
    },
  ],
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: collector
name: consent-on-frontdesk
modality: call
status: ACTIVE
metrics:
  - consent_collection_consent_obtained
  - refund_policy_accuracy
filters:
  - conditions:
      - type: AGENT
        key: frontdesk
        operator: EQUALS
```

* `name` is the slug config identity. `displayName` (optional) is the collector's name as it appears in Roark, where spaces and capitals are allowed; it defaults to `name`. Set it to a collector's exact existing name to bring one you created in the app under config (apply takes it over instead of creating a duplicate), for example `displayName: "Credit Card Apr"` with `name: credit-card-apr`.
* `modality` is `call` or `chat` and is immutable once created (make a new collector to switch).
* **Metrics** are referenced by their stable slug, not a UUID, and must be visible to the project and support the collector's `modality`.
* **Filters** are condition groups: groups OR together, conditions within a group AND. An `AGENT` condition's `key` is a config-managed agent name (resolved on apply); every other type (`CALL_SOURCE`, `CALL_PROPERTY`, `INTEGRATION`) uses its `key`/`value` verbatim. Omit `filters` to score every matching conversation.

## Per-collector metric settings

Some metrics take a setting that applies only in this collector, the same setting the app's inline metric config sets. Write the metric as an object with `metric` (its slug) and its setting, next to plain slugs:

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const creditCardApr = defineCollector({
  name: 'credit-card-apr',
  displayName: 'Credit Card Apr',
  modality: 'call',
  metrics: [
    {
      metric: 'property_transcript_mismatch',
      verifiedProperties: ['purchase_apr_total', 'extracted_purchase_apr_total'],
    },
    'apr_retrieved',
  ],
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
kind: collector
name: credit-card-apr
displayName: Credit Card Apr
modality: call
metrics:
  - metric: property_transcript_mismatch
    verifiedProperties: [purchase_apr_total, extracted_purchase_apr_total]
  - apr_retrieved
```

| Setting | Metrics | What it does |
| - | - | - |
| `verifiedProperties` | Property Verification (`property_transcript_mismatch`, `property_transcript_mismatch_count`, `stt_property_capture_rate`) | Checks only these call properties against the transcript. Omit to check every custom property. |
| `excludedPhrases` | Voice consistency | Agent lines containing one of these phrases are not judged, such as a recorded prompt in a different voice. |
| `voicemailScript` | Voicemail | The message the agent is expected to leave on voicemail. |
| `screeningBrief` | Call screening | What the agent should say when a screener asks who is calling and why. |

* A metric takes at most one setting, and only the one listed for it. Anything else fails the apply with an error naming the metric and the setting it takes. A misspelled setting fails validation rather than being ignored.
* **The file is the source of truth.** A metric listed without a setting runs with its default, so a setting made in the app is replaced on the next apply that changes the collector. Keep the setting in the file.
* Tool and knowledge base scoping differ per environment (they reference ids), so they are set in the app or through the [metric policies API](/api-reference) (`metrics[].settings`) instead.

### Re-scoring past calls with the collector's settings

Changing a setting applies to conversations scored afterwards. To re-score earlier calls with the collector's settings, create a metric collection job with the collector's id as `policyId`:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/metric/collection-jobs \
  -H "Authorization: Bearer $ROARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "callIds": ["3f0c2a9e-6b1d-4c3e-9a51-0d2b8e7f4c11"], "policyId": "<collector id>" }'
```

Each metric runs exactly as the collector runs it, and `metrics` defaults to all of the collector's metrics. The collector's filters are not applied: the calls you list are scored as given.

> [!NOTE]
>
> For the full field reference of every kind, see the [Config DSL reference](https://roark.ai/roark-config.schema.json) schema.


# Dashboards
Source: https://docs.roark.ai/documentation/config-as-code/dashboards

Define dashboards and their widgets in YAML, versioned and reviewed like the rest of your config

A `kind: dashboard` declares a [dashboard](/documentation/observability/dashboards): its window, the filters over every widget, and its widgets top to bottom. A widget is an [analytics query](/documentation/observability/analytics-api) plus a name, a chart type and a size, so a query you tried over the API drops straight into a widget.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineDashboard } from '@roarkanalytics/sdk/lib/config';

const frontdeskHealth = defineDashboard({
  name: 'frontdesk-health',
  displayName: 'Frontdesk health',
  timeRange: { type: 'LAST', value: 30, unit: 'DAYS' },
  filters: [
    { type: 'AGENT_PROPERTY', value: 'frontdesk' }, // an agent by its config name, or its id
    { type: 'MODALITY', value: 'call' },
  ],
  widgets: [
    {
      name: 'Calls',
      display: 'NUMBER',
      size: 'SMALL',
      measures: [{ type: 'CONVERSATION' }],
    },
    {
      name: 'Task completion',
      display: 'LINE',
      size: 'LARGE',
      interval: 'DAY',
      measures: [{ metric: 'task_completion' }],
    },
    {
      name: 'Call outcomes',
      display: 'STACKED_BAR',
      interval: 'WEEK',
      measures: [{ metric: 'call_outcome' }], // a classification: counted and split by value
    },
    {
      name: 'Calls by direction',
      display: 'STACKED_BAR',
      measures: [
        {
          type: 'CONVERSATION',
          breakdown: { type: 'CALL_PROPERTY', property: 'direction' },
        },
      ],
    },
  ],
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: dashboard
name: frontdesk-health
displayName: Frontdesk health
timeRange: { type: LAST, value: 30, unit: DAYS }
filters:
  - { type: AGENT_PROPERTY, value: frontdesk }   # an agent by its config name, or its id
  - { type: MODALITY, value: call }
widgets:
  - name: Calls
    display: NUMBER
    size: SMALL
    measures:
      - type: CONVERSATION
  - name: Task completion
    display: LINE
    size: LARGE
    interval: DAY
    measures:
      - metric: task_completion
  - name: Call outcomes
    display: STACKED_BAR
    interval: WEEK
    measures:
      - metric: call_outcome      # a classification: counted and split by value
  - name: Calls by direction
    display: STACKED_BAR
    measures:
      - type: CONVERSATION
        breakdown: { type: CALL_PROPERTY, property: direction }
```

***

## Fields

| Field | Required | |
| :- | :- | :- |
| `name` | yes | The config name. |
| `displayName` | no | The title shown in Roark. Defaults to `name`. |
| `description` | no | Free text. |
| `isDefault` | no | Make it the project's default dashboard. Left out, the flag is left as it is. |
| `timeRange` | no | The window every widget reads. A new dashboard defaults to the last 30 days; left out on an existing one, whatever a viewer last picked is kept. Same shapes as an [analytics query](/documentation/observability/analytics-api#time-windows). |
| `filters` | no | Over every widget: `AGENT_PROPERTY`, `CALL_PROPERTY`, `MODALITY`, `DATASET` or `METRIC_VALUE`. Left out, a viewer's filters are kept. See [Filters](#filters). |
| `widgets` | no | Up to 50, top to bottom. |

### Widgets

| Field | Required | |
| :- | :- | :- |
| `name` | yes | The widget's title, unique on the dashboard. It is the widget's identity across applies. |
| `measures` | yes | 1 to 10 [measures](/documentation/observability/analytics-api#measures): `metric: <slug>`, or `type: CONVERSATION` / `type: UPTIME`, each with optional `aggregation`, `percentile`, `breakdown` and `filters`. |
| `display` | no | `LINE` (default), `NUMBER`, `RADAR`, `STACKED_BAR` or `TABLE`. |
| `size` | no | `SMALL`, `MEDIUM` (default), `LARGE` or `EXTRA_LARGE`. |
| `interval` | no | `HOUR`, `DAY` (default), `WEEK` or `MONTH`. Stepped coarser on wide windows. |
| `filters` | no | Applied to every measure in the widget. |
| `comparisonMode` | no | `PREVIOUS_PERIOD` to add the preceding window. |
| `comparisonGroups` | no | 2 to 5 named filter sets compared as separate series. |

### Filters

A filter is `{ type, property, operator, value }`, the same shape as an [analytics filter](/documentation/observability/analytics-api#filters-breakdowns-and-comparisons); `operator` defaults to `EQUALS`. A `CALL_PROPERTY` filter also takes `IS_SET` or `IS_NOT_SET` with no `value`: conversations where the property is set (whatever the value, an empty string included) or missing (never sent, sent as `null`, or removed).

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
filters:
  - { type: AGENT_PROPERTY, value: frontdesk }
  - { type: CALL_PROPERTY, property: escalation_reason, operator: IS_NOT_SET }
```

#### Filter by another metric

A `METRIC_VALUE` filter keeps the conversations where another metric came out a certain way, such as refund accuracy only on calls where the task was completed. It works on the dashboard, a widget, a measure and a comparison group. `property` is that metric's slug, resolved the same way as a measure's `metric`, optionally followed by `.AGENT` or `.CUSTOMER` to read one side of a per-participant metric. `operator` and `value` are required, and depend on the metric:

| Metric | Operators | Value |
| :- | :- | :- |
| Boolean | `EQUALS`, `NOT_EQUALS` | `true` or `false` |
| Number, scale, count, offset | `EQUALS`, `NOT_EQUALS`, `GREATER_THAN`, `GREATER_THAN_OR_EQUAL_TO`, `LESS_THAN`, `LESS_THAN_OR_EQUAL_TO` | A number |
| Classification, text | `EQUALS`, `NOT_EQUALS` (ignoring case), `CONTAINS` | A label or text |
| [Threshold](/documentation/config-as-code/metrics#thresholds) | `EQUALS`, `NOT_EQUALS` | `PASSED` or `FAILED`, with no `.AGENT` or `.CUSTOMER` suffix |

A slug that matches no metric, or a value the metric can't compare, fails that dashboard's apply and leaves it as it was. For how the filter matches, see [Filter by another metric](/documentation/observability/dashboards#filter-by-another-metric).

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
widgets:
  - name: Refund accuracy when the task was completed
    filters:
      - { type: METRIC_VALUE, property: task_completion, operator: EQUALS, value: 2 } # 2 = Completed
    measures: [{ metric: refund_policy_accuracy }]
  - name: Containment by caller sentiment
    measures: [{ metric: agent_containment }]
    comparisonGroups:
      - name: Negative callers
        filters: [{ type: METRIC_VALUE, property: sentiment_score.CUSTOMER, operator: LESS_THAN_OR_EQUAL_TO, value: 3 }]
      - name: Everyone else
        filters: [{ type: METRIC_VALUE, property: sentiment_score.CUSTOMER, operator: GREATER_THAN, value: 3 }]
  - name: Sentiment on calls that failed the latency check
    filters:
      - { type: METRIC_VALUE, property: latency_under_2s, operator: EQUALS, value: FAILED }
    measures: [{ metric: sentiment_score }]
```

***

## References

* **Metrics** resolve by slug: a [`kind: metric`](/documentation/config-as-code/metrics) in the same config (applied first), or any metric the project can see, system metrics included.
* **Agents** in an agent filter can be written by their config name; Roark resolves them to ids, the same way other kinds reference agents.

Every widget is checked against what its metric can answer before anything is written. A widget asking for `AVG` of a classification, or naming a metric that doesn't exist, fails that dashboard's apply with the reason, and the dashboard is left exactly as it was.

***

## In Roark

A dashboard created by config shows **Managed by config**. Its title and widgets are read-only in Roark and over the API (edits return `409`): change them in your config and re-apply. Anyone viewing it can still pick a time range and filter it, and an apply only overwrites those when the config declares `timeRange` or `filters`. Duplicating a managed dashboard gives you an editable copy.

A dashboard made in Roark with the same title as a `kind: dashboard` (its `displayName`, or `name`) is adopted on the first apply rather than duplicated, as long as exactly one unmanaged dashboard has that title.

***

## Removing a dashboard

Deleting a dashboard from your config (with prune on, the default) deletes the dashboard and its widgets.


# Flows
Source: https://docs.roark.ai/documentation/config-as-code/flows

Define simulation flows as config (improv and scripted)

A simulation flow. Both types share `kind: flow` and are discriminated by `type`. Flows reference agents, a persona, and an environment by name.

## Improv (`type: improv`)

An improvised simulation with a happy path and edge-case variants.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { readFileSync } from 'node:fs';
import { defineImprovFlow } from '@roarkanalytics/sdk/lib/config';

const frustratedRebooking = defineImprovFlow({
  name: 'frustrated-rebooking',
  agents: ['frontdesk'],
  happyPath: {
    persona: 'frustrated-caller',
    environment: 'Quiet line',
    prompt: 'You call to rebook the cleaning that was cancelled on you.',
  },
  edgeCases: [
    {
      name: 'escalates-to-manager',
      prompt: readFileSync('./prompts/escalates-to-manager.md', 'utf8'),
      expectations: ['Agent offers to escalate rather than arguing'],
    },
  ],
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: flow
type: improv
name: frustrated-rebooking
agents: [frontdesk]
happyPath:
  persona: frustrated-caller
  environment: Quiet line
  prompt: You call to rebook the cleaning that was cancelled on you.
edgeCases:
  - name: escalates-to-manager
    prompt: file://prompts/escalates-to-manager.md
    expectations:
      - Agent offers to escalate rather than arguing
```

* `happyPath` requires a `persona` and `environment` (by name); `prompt` and `expectations` are optional.
* Each `edgeCases[]` entry inherits the happy path unless it overrides `persona` / `environment` / `prompt`, and can add its own `expectations`.

## Scripted (`type: scripted`)

A step-by-step conversation graph (branches, merges, DTMF, voicemail, flow links; the API enum name is `SCENARIO_LINK`). Each apply replaces the whole graph.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineScriptedFlow } from '@roarkanalytics/sdk/lib/config';

const bookingScripted = defineScriptedFlow({
  name: 'booking-scripted',
  agents: ['frontdesk'],
  branchingMode: 'ADAPTIVE',
  graph: [
    {
      ref: 'greeting',
      type: 'AGENT_TURN',
      content: 'Thanks for calling, how can I help?',
      steps: [
        {
          ref: 'request',
          type: 'CUSTOMER_TURN',
          content: "I'd like to book a cleaning.",
        },
      ],
    },
  ],
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: flow
type: scripted
name: booking-scripted
agents: [frontdesk]
branchingMode: ADAPTIVE
graph:
  - ref: greeting
    type: AGENT_TURN
    content: Thanks for calling, how can I help?
    steps:
      - ref: request
        type: CUSTOMER_TURN
        content: I'd like to book a cleaning.
```

* Each node has a `type` and an optional `ref` label so other nodes can rejoin it via `mergeInto` (DAG merge edges). `steps` are a node's successors (more than one = a branch).
* No UUIDs: identity is the `ref`, and each apply replaces the whole graph.
* The environment is referenced by its display name and must already exist in Roark.

> [!NOTE]
>
> For the full field reference of every kind, see the [Config DSL reference](https://roark.ai/roark-config.schema.json) schema.


# HTTP request definitions
Source: https://docs.roark.ai/documentation/config-as-code/http-request-definitions

Define the HTTP request Roark fires to place outbound calls through your own system

An `httpRequestDefinition` is a reusable HTTP request Roark fires to hand off **outbound dialling** to your own system. An [agent endpoint](/documentation/config-as-code/agents#outbound-dialling) whose `outboundDialType` is `HTTP_REQUEST` references one by name, and Roark calls it (substituting the number to dial) when a call should be placed.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineHTTPRequestDefinition } from '@roarkanalytics/sdk/lib/config';

const dialer = defineHTTPRequestDefinition({
  name: 'dialer',
  description: 'Places outbound calls through our telephony gateway',
  url: 'https://dialer.example.com/dial',
  method: 'POST',
  body: '{"to":"{{phoneNumberToDial}}"}',
  headers: {
    Authorization: 'Bearer YOUR_TOKEN',
  },
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: httpRequestDefinition
name: dialer
description: Places outbound calls through our telephony gateway
url: https://dialer.example.com/dial
method: POST
body: '{"to":"{{phoneNumberToDial}}"}'
headers:
  Authorization: 'Bearer YOUR_TOKEN'
```

| Field | Default | Notes |
| :- | :- | :- |
| `name` | | The local key (`^[a-z0-9][a-z0-9-_.]*$`); what an endpoint's `outboundDialHttpRequestDefinition` references |
| `url` | | Where Roark sends the request. Public `http(s)` only: localhost, private, and link-local hosts are rejected |
| `method` | `POST` | `POST`, `PUT`, `PATCH`, or `GET` |
| `body` | `{}` | Request body template. A string that may carry placeholders Roark substitutes at dial time, e.g. `{{phoneNumberToDial}}` |
| `headers` | | Static request headers sent verbatim, for example an `Authorization` bearer token for your endpoint |
| `description` | | |

It reconciles **before** agents, so an endpoint that references it in the same bundle resolves in one apply. Reference it only from a config-managed definition (declared in the bundle or a previous apply).

## Signing secret

Roark mints an HMAC **signing secret** for each definition so your endpoint can verify a request genuinely came from Roark. It is **not** declared in your config: it is generated on create and returned exactly once, in the `apply` response's `signingSecret` for that resource.

> [!WARNING]
>
> Capture the `signingSecret` from the apply response when the definition is first created: it cannot be read back afterwards. If you lose it, delete and re-apply the definition to mint a new one. Editing the other fields updates the definition in place and leaves the secret untouched.

## Ownership

Once a definition is managed by config, it is read-only in the dashboard and over the API: `PUT /v1/http-request-definition/{id}` returns `409`, and the response's `isConfigManaged` is `true`. Change it in your config and re-apply, or detach it first to release it back to dashboard ownership.

> [!NOTE]
>
> Only outbound-dial definitions are managed here. Webhook and alert-delivery HTTP requests own their own lifecycle and are configured elsewhere.

> [!NOTE]
>
> For the full field reference of every kind, see the [Config DSL reference](https://roark.ai/roark-config.schema.json) schema.


# Live reports
Source: https://docs.roark.ai/documentation/config-as-code/live-reports

Declare a recurring report on your production calls: which calls, the targets they are held to, when it is written and who gets it

A `kind: liveReport` declares a [live call report](/documentation/observability/live-reports): the calls it covers (an agent, narrowed by filters), the targets each period is held to, its cadence, and who it is sent to. When each period closes, Roark writes the report: targets met and missed, what changed since the period before, failure themes with quoted evidence, and recommendations.

Applying a report never writes one. Roark writes a report when each period closes, or when someone asks for the latest period from Roark, the API or `roark live-report run`.

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: liveReport
name: frontdesk-weekly
displayName: Frontdesk weekly review
description: How the frontdesk agent handled last week's production calls.
agent: frontdesk                   # a kind: agent in this config, or an agent made in Roark, by name
filters:
  - { type: MODALITY, value: call }
targets:
  - { metric: task_completion, aggregation: PASS_RATE, value: 85 }
  - { metric: response_latency_ms, aggregation: PERCENTILE, percentile: 95, value: 1200, label: p95 latency }
  - { metric: sentiment_score, aggregation: PASS_RATE, passOperator: GREATER_THAN_OR_EQUAL_TO, passValue: 6, value: 70 }
cadence: WEEKLY
timezone: America/Chicago
deliveryHour: 8                    # Monday 08:00 Chicago time, for the Monday-to-Sunday week
notify:
  email: [ops@example.com, cx-lead@example.com]
  slack:
    - { channelId: C0123ABCD, channelName: cx-reports }
dashboard: frontdesk-health        # embed this dashboard's charts, read over the report's calls
```

> [!NOTE]
>
> Live reports are YAML-only for now. A `defineLiveReport` helper for the [TypeScript SDK](/documentation/config-as-code/typescript) is on the way.

***

## Fields

| Field | Required | |
| :- | :- | :- |
| `name` | yes | The config name. |
| `displayName` | no | The report's title in Roark, emails and Slack. Defaults to `name`. |
| `description` | no | Free text. |
| `agent` | no | The agent whose calls it covers: a [`kind: agent`](/documentation/config-as-code/agents) in this config, or an agent made in Roark, by name. Left out, the report covers every agent in the project. |
| `filters` | no | Up to 20, narrowing the calls further: `CALL_PROPERTY`, `MODALITY`, `DATASET` or `METRIC_VALUE`, in the same shape as [dashboard filters](/documentation/config-as-code/dashboards#filters). Simulations are never included, and the agent is set with `agent`, not a filter (an `AGENT_PROPERTY` or `CALL_PROPERTY agent_id` filter is refused). |
| `targets` | no | Up to 10. See [Targets](#targets). |
| `cadence` | no | `DAILY`, `WEEKLY` (the default, Monday to Sunday) or `MONTHLY`. |
| `timezone` | no | The IANA zone periods and the delivery hour are read in. Defaults to `UTC`. |
| `deliveryHour` | no | The local hour, `0` to `23`, the report is sent on the day after its period closes. Defaults to `9`. |
| `enabled` | no | `false` pauses the report without deleting it. Defaults to `true`. |
| `notify` | no | Who gets each report, in the same shape as a [plan's `notify`](/documentation/config-as-code/simulation-plans). `email: true` (the default) emails every project member, `false` sends no email, and a list names the addresses, Roark users or not (the email carries the PDF). `slack` lists channels on the project's connected Slack workspace. |
| `includeTranscriptExcerpts` | no | Quote short transcript passages as evidence. Defaults to `true`; set `false` where transcripts must not leave Roark. |
| `dashboard` | no | A dashboard whose charts the report embeds: a [`kind: dashboard`](/documentation/config-as-code/dashboards) in this config, by config name, or a dashboard made in Roark, by its title. |

Every field is defaulted, so deleting a line reverts it rather than keeping the last value: removing `dashboard` unlinks it, and removing `notify` goes back to emailing every project member.

### Targets

| Field | Required | |
| :- | :- | :- |
| `metric` | yes | A metric slug: a [`kind: metric`](/documentation/config-as-code/metrics) in this config, or any metric the project can see. |
| `value` | yes | The bar, in display units. A `PASS_RATE` bar is a percentage, `0` to `100`. |
| `aggregation` | no | `PASS_RATE`, `AVG`, `MEDIAN`, `PERCENTILE`, `MIN` or `MAX`. Defaults to `PASS_RATE` for thresholds and booleans, `AVG` for anything numeric. |
| `percentile` | with `PERCENTILE` | `1` to `99`, for example `95`. |
| `passOperator`, `passValue` | with `PASS_RATE` on a metric that isn't a threshold | The condition a call passes on: `EQUALS`, `NOT_EQUALS`, `GREATER_THAN`, `GREATER_THAN_OR_EQUAL_TO`, `LESS_THAN` or `LESS_THAN_OR_EQUAL_TO`, and the value it compares with. |
| `comparator` | no | `AT_LEAST` (the default for a pass rate) or `AT_MOST` (the default for every other aggregation). |
| `label` | no | Shown instead of the metric's name. |

A [threshold](/documentation/config-as-code/metrics#thresholds)'s pass rate uses the threshold's own condition, so it takes no `passOperator`.

A target on a metric that doesn't exist, or one the metric can't answer (an average of a classification), fails that report's apply with the reason and leaves the report as it was.

***

## References

Live reports are applied after everything they name: metrics, agents and dashboards, so a report can name a metric, agent or dashboard defined in the same config. A dashboard that a report in your config still embeds is never pruned: removing the dashboard alone fails with the report named. Remove `dashboard:` from the report, or remove both together.

***

## In Roark

A report created by config shows as managed by config. It is read-only in Roark and over the API (edits, pause and delete return `409`): change it in your config and re-apply. Anyone can still open its reports, download the PDF, and write the latest period on demand.

A report made in Roark with the same title as a `kind: liveReport` (its `displayName`, or `name`) is adopted on the first apply rather than duplicated. Titles are unique in a project, ignoring case.

***

## Removing a report

Deleting a report from your config (with prune on, the default) deletes the report: no more reports are written or sent. Reports it already wrote stay readable from the emails and Slack messages that delivered them.


# Metrics
Source: https://docs.roark.ai/documentation/config-as-code/metrics

Define custom LLM judges, API checks and thresholds as config

A [custom metric](/documentation/metrics/custom-metrics) you author: an [LLM judge](/documentation/metrics/custom-metrics/llm-judge) graded against a prompt, an [API check](#api-checks) your own endpoint answers, or a [threshold](#thresholds) that turns another metric into a pass/fail check. Reference any of them from a collector, run plan, alert or dashboard by its `name`.

## LLM judges

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineMetric } from '@roarkanalytics/sdk/lib/config';

const refundPolicyAccuracy = defineMetric({
  name: 'refund_policy_accuracy', // the slug a collector references
  displayName: 'Refund Policy Accuracy',
  type: 'BOOLEAN', // BOOLEAN | SCALE | NUMERIC | TEXT | CLASSIFICATION
  prompt:
    'Using {{transcript}} and {{world_context}}, did the agent state the refund policy correctly?',
  trueLabel: 'Accurate',
  falseLabel: 'Inaccurate',
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: metric
name: refund_policy_accuracy       # the slug a collector references
displayName: Refund Policy Accuracy
type: BOOLEAN                       # BOOLEAN | SCALE | NUMERIC | TEXT | CLASSIFICATION
prompt: |
  Using {{transcript}} and {{world_context}}, did the agent state the refund policy correctly?
trueLabel: Accurate
falseLabel: Inaccurate
```

| Field | Required | Values |
| :- | :- | :- |
| `name` | yes | The metric's stable slug (its `metricId`), exactly what a collector references. |
| `displayName` | no | The name shown in Roark, up to 100 characters. |
| `type` | yes | `BOOLEAN`, `SCALE`, `NUMERIC`, `TEXT` or `CLASSIFICATION`. See [output types](/documentation/metrics/custom-metrics/llm-judge#output-types). |
| `prompt` | yes | The grading criteria. It must reference a transcript source (e.g. `{{transcript}}`) and `{{world_context}}`. See [prompt variables](/documentation/metrics/custom-metrics/llm-judge#prompt-variables). |
| `scope` | no | `GLOBAL` (default), one value for the whole conversation, or `PER_PARTICIPANT`, one value per side. |
| `participantRole` | with `PER_PARTICIPANT` | `AGENT` or `CUSTOMER`. |
| `contexts` | no | What the judge grades: `CALL` (default), plus `TURN` for each exchange or `SEGMENT` for each utterance. |
| `trueLabel` / `falseLabel` | no | BOOLEAN only: what a pass and a fail are called. |
| `scaleMin` / `scaleMax` | for `SCALE` | Whole numbers bounding the score. |
| `scaleLabels` | no | SCALE only: named bands, each `{ rangeMin, rangeMax, label, displayOrder }` with optional `description` and `colorHex`. |
| `options` | for `CLASSIFICATION` | The labels the judge picks from, each `{ label, displayOrder }` with an optional `description`. |
| `maxSelections` | no | CLASSIFICATION only: how many options one conversation can get. |

* The prompt is a plain string in TypeScript, inline or read from a file. In YAML it is inline or `file://`.
* `name`, `type` and `scope` are immutable once created: changing one fails that resource. To change them, give the metric a new `name`. Editing the prompt, labels, bounds or options creates a new version.
* `contexts` with `TURN` or `SEGMENT` must also include `CALL`, because checks and reports read the call-level value. A metric grades either each exchange or each utterance, not both: `[CALL, TURN]` and `[CALL, SEGMENT]` are valid, `[TURN]` and `[CALL, TURN, SEGMENT]` are not.

A SCALE judge with bands, graded on each exchange as well as the whole call:

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineMetric } from '@roarkanalytics/sdk/lib/config';

const empathy = defineMetric({
  name: 'agent_empathy',
  type: 'SCALE',
  scope: 'PER_PARTICIPANT',
  participantRole: 'AGENT',
  contexts: ['CALL', 'TURN'],
  prompt: 'Using {{transcript}} and {{world_context}}, rate how empathetic the agent was, from 1 to 5.',
  scaleMin: 1,
  scaleMax: 5,
  scaleLabels: [
    { rangeMin: 1, rangeMax: 2, label: 'Cold', displayOrder: 0 },
    { rangeMin: 3, rangeMax: 3, label: 'Neutral', displayOrder: 1 },
    { rangeMin: 4, rangeMax: 5, label: 'Warm', displayOrder: 2 },
  ],
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: metric
name: agent_empathy
type: SCALE
scope: PER_PARTICIPANT
participantRole: AGENT
contexts: [CALL, TURN]              # TURN or SEGMENT always comes with CALL
prompt: |
  Using {{transcript}} and {{world_context}}, rate how empathetic the agent was, from 1 to 5.
scaleMin: 1
scaleMax: 5
scaleLabels:
  - { rangeMin: 1, rangeMax: 2, label: Cold, displayOrder: 0 }
  - { rangeMin: 3, rangeMax: 3, label: Neutral, displayOrder: 1 }
  - { rangeMin: 4, rangeMax: 5, label: Warm, displayOrder: 2 }
```

## API checks

An [API check](/documentation/metrics/custom-metrics/api-checks) is a pass/fail metric your own endpoint answers. Set `calculationType: HTTP_REQUEST` and describe the request Roark sends after each call.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineApiCheck } from '@roarkanalytics/sdk/lib/config';

const pinChanged = defineApiCheck({
  name: 'pin_changed', // the slug a collector references
  displayName: 'PIN changed',
  httpRequest: {
    url: 'https://api.example.com/roark/checks/pin-changed',
    headers: { Authorization: `Bearer ${process.env.CARD_SERVICE_TOKEN}` },
    body: '{"cardId": "{{call.properties.cardId}}"}',
  },
  trueLabel: 'PIN changed',
  falseLabel: 'PIN unchanged',
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: metric
calculationType: HTTP_REQUEST       # makes this an API check; LLM judges leave it out
name: pin_changed                   # the slug a collector references
displayName: PIN changed
httpRequest:
  url: https://api.example.com/roark/checks/pin-changed
  headers:
    Authorization: 'Bearer YOUR_TOKEN'
  body: '{"cardId": "{{call.properties.cardId}}"}'
trueLabel: PIN changed
falseLabel: PIN unchanged
```

| Field | Required | Values |
| :- | :- | :- |
| `name` | yes | The check's stable slug, exactly what a collector references. |
| `displayName` | no | The name shown in Roark, up to 100 characters. |
| `httpRequest.url` | yes | A public `http(s)` URL, up to 2048 characters. |
| `httpRequest.method` | no | `POST` (default) or `GET`. A `GET` has no body. |
| `httpRequest.headers` | no | Up to 20 headers. |
| `httpRequest.body` | no | A JSON template of up to 20,000 characters. Omit it to send the [default payload](/documentation/metrics/custom-metrics/api-checks#the-request-payload). |
| `trueLabel` / `falseLabel` | no | What a pass and a fail are called. Default `Pass` and `Fail`. |

* URL, headers, and body can use [placeholders](/documentation/metrics/custom-metrics/api-checks#placeholders). A placeholder that doesn't exist fails that resource's apply.
* The file is the source of truth for the request: a header or body you remove from the file is removed from the check, and labels you remove go back to `Pass` and `Fail`. Each change creates a new version.
* `name` and `calculationType` are immutable. A metric can't switch between an LLM judge and an API check: apply fails that resource instead.
* Headers are stored as you write them. In TypeScript, read tokens from the environment rather than committing them.

## Thresholds

A [threshold](/documentation/metrics/thresholds) checks another metric against a bar and passes or fails. Set `calculationType: THRESHOLD`, name the metric it checks in `sourceMetric`, and give the comparison.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineThreshold } from '@roarkanalytics/sdk/lib/config';

const respondsWithin2s = defineThreshold({
  name: 'agent_responds_within_2s', // the slug a collector or run plan references
  sourceMetric: 'response_time', // a system metric, one made in Roark, or a metric in this config
  operator: 'LESS_THAN_OR_EQUALS',
  value: 2000,
  aggregationMode: 'P95', // optional, default EACH
  sourceParticipantRole: 'AGENT', // optional, for a per-participant source
  trueLabel: 'Fast enough', // optional, default Pass
  falseLabel: 'Too slow', // optional, default Fail
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: metric
calculationType: THRESHOLD          # makes this a threshold
name: agent_responds_within_2s      # the slug a collector or run plan references
sourceMetric: response_time         # a system metric, one made in Roark, or a kind: metric in this config
operator: LESS_THAN_OR_EQUALS
value: 2000
aggregationMode: P95                # optional, default EACH
sourceParticipantRole: AGENT        # optional, for a per-participant source
trueLabel: Fast enough              # optional, default Pass
falseLabel: Too slow                # optional, default Fail
```

`defineThreshold` needs `@roarkanalytics/sdk` 6.0.0 or later.

| Field | Required | Values |
| :- | :- | :- |
| `sourceMetric` | yes | The slug of the metric to check. It can be another threshold. |
| `operator` | yes | `GREATER_THAN`, `GREATER_THAN_OR_EQUALS`, `LESS_THAN`, `LESS_THAN_OR_EQUALS`, `EQUALS`, `NOT_EQUALS`. Numeric sources take all six; BOOLEAN and CLASSIFICATION sources take `EQUALS` and `NOT_EQUALS`. |
| `value` | yes | A number for SCALE, NUMERIC and count sources, `true` or `false` for BOOLEAN, an option label for CLASSIFICATION. |
| `aggregationMode` | no | How a source with several values per call is combined: `EACH` (default), `AVERAGE`, `MIN`, `MAX`, `MEDIAN`, `SUM`, `P95`, `P99`, `COUNT`. See [aggregation modes](/documentation/metrics/thresholds#aggregation-modes). |
| `countThreshold` | for `COUNT` | How many values may fail the comparison before the threshold fails. Only with `COUNT`. |
| `sourceParticipantRole` | no | `AGENT` or `CUSTOMER`: check one side of a per-participant source. |
| `sourceVariant` | no | The source's variant, by name. Omit it for the source's default. |
| `trueLabel` / `falseLabel` | no | What a pass and a fail are called. |

* **The comparison is immutable.** `sourceMetric`, `sourceVariant`, `operator`, `value`, `aggregationMode`, `countThreshold` and `sourceParticipantRole` can't change once the threshold exists, as in Roark. To change the bar, give the threshold a new `name`: apply creates the new one, and removes the old one when you prune. Only the labels can be edited in place. A value written differently but equal, such as `4` and `4.0`, is not a change.
* Roark names the threshold from its comparison (for example `Response Time <= 2000 (Agent)`), the same as one made in Roark.
* A threshold and its source can be in the same config. Apply creates the source first, and removes the threshold first when you prune both.
* If a threshold with the same comparison already exists in the project, apply fails that resource and names the existing one: reference it by its slug instead.

Once applied, a threshold is a metric like any other: list it in a [collector](/documentation/config-as-code/collectors)'s `metrics:`, gate a [simulation plan](/documentation/config-as-code/simulation-plans) on it, [alert](/documentation/config-as-code/alerts) on it, or chart its pass rate on a [dashboard](/documentation/config-as-code/dashboards).

> [!NOTE]
>
> **System metrics** (Roark's built-in metrics) are managed by Roark, not by you: you never define
> them in config. To collect one, just reference its slug in a collector's `metrics:` list, exactly
> like a custom metric. The same is true for formulas and patterns, which are not yet definable in
> config: create them in Studio or through the API, then reference them by slug.

> [!NOTE]
>
> For the full field reference of every kind, see the [Config DSL reference](https://roark.ai/roark-config.schema.json) schema.


# Organizations & Projects
Source: https://docs.roark.ai/documentation/config-as-code/organizations

Declare your projects, their members, and each project’s resources as config and provision your whole organization from one git repo

Everything else in Config as Code manages resources **inside one project**, authenticated by a project API key. This page is the level above: with an **Organization API key** you declare the **projects** your organization should have, each with its **members and roles** and its own nested **resources**, and apply the whole tree with the same `roark config apply`.

This is how you run Config as Code at scale: one git repo, reviewed by PR, that provisions projects, onboards people, and configures each project, for the entire organization.

## The two scopes

There is one apply surface (`/v1/config`, `roark config apply`); the **credential decides what you may declare**:

| Your key | You declare | Lands in |
| :- | :- | :- |
| **Project** API key | Project resources (`agent`, `persona`, `flow`, `metric`, ...) | that one project |
| **Organization** API key | `kind: project` (with nested members and resources) | the whole organization |

A project key cannot declare `kind: project`; an organization key declares only `kind: project` (project-scoped resources go **under** a project's `resources:`).

## Prerequisites

* An **Organization API key**. An org Owner or Admin creates one under **Settings → Organization → API keys** ("New Key"). The full key is shown once, so store it in your secret manager. It carries `org-config:apply` and the project/member/api-key management permissions.
* A git repository to hold your config.
* The **Roark CLI** (recommended): `npm install -g @roarkanalytics/cli`.

> [!NOTE]
>
> An organization key is an admin credential: it can create projects, manage members, and mint per-project keys across your whole organization. Treat it like one, and prefer a read-only key (below) for CI dry-runs.

## The `kind: project` resource

One file per project. A project declares its identity, its members, and (optionally) its own resources.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineAgent, defineMetric, defineProject } from '@roarkanalytics/sdk/lib/config';

const paymentsProd = defineProject({
  name: 'payments-prod', // identity (configKey = project/<name>), lowercase slug
  displayName: 'Payments (prod)', // optional; the project name shown in the UI
  description: 'Production VA for payments',
  category: 'FINANCIAL', // optional; HEALTHCARE | FINANCIAL | RETAIL | ... | OTHER
  dataStorageRegion: 'US_EAST_1', // optional; US_EAST_1 | EU_WEST_1
  members: [
    { email: 'jane@example.com', role: 'ADMIN' },
    { email: 'sam@example.com', role: 'MEMBER' },
  ],
  // this project's own Config-as-Code resources
  resources: [
    defineAgent({ name: 'payments-va' }),
    defineMetric({
      name: 'refund_policy_accuracy',
      type: 'BOOLEAN',
      prompt: 'Using {{transcript}} and {{world_context}}, was the refund policy stated correctly?',
    }),
  ],
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: project
name: payments-prod                 # identity (configKey = project/<name>), lowercase slug
displayName: Payments (prod)        # optional; the project name shown in the UI
description: Production VA for payments
category: FINANCIAL                 # optional; HEALTHCARE | FINANCIAL | RETAIL | ... | OTHER
dataStorageRegion: US_EAST_1        # optional; US_EAST_1 | EU_WEST_1

members:
  - { email: jane@example.com, role: ADMIN }
  - { email: sam@example.com, role: MEMBER }

resources:                          # this project's own Config-as-Code resources
  - kind: agent
    name: payments-va
  - kind: metric
    name: refund_policy_accuracy
    type: BOOLEAN
    prompt: |
      Using {{transcript}} and {{world_context}}, was the refund policy stated correctly?
```

* **`name`** is the stable identity (`configKey = project/<name>`) and the seed for the project slug. `displayName` is the human name in the UI (defaults to `name`).
* **`members`** are reconciled by email: a new email is invited, an existing member's role is updated, and (with prune on) a config-managed member you remove is removed. An existing UI or SSO member you declare is adopted rather than duplicated. Roles are the project roles: `OWNER`, `ADMIN`, `MEMBER`, `VIEWER`.
* **`resources`** is the project's own resource set, exactly the shapes documented on the other Config as Code pages (agents, personas, flows, metrics, collectors, simulation plans, alerts). They are applied into this project through the same reconcile, so one organization key manages an entire org's config in one tree.

> [!NOTE]
>
> Prefer **SCIM + SAML SSO** for day-to-day user lifecycle at scale (automatic provisioning/deprovisioning and project auto-join from your IdP). Use `members:` here for the projects and roles you want declared in git; the two compose. See [Okta SSO](/documentation/integrations/okta).

## Deploying

Same commands as project config; the organization key is what puts them in org mode.

**1. Authenticate with the organization key**

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
export ROARK_API_BEARER_TOKEN="<your-organization-api-key>"
```

**2. Preview**

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config diff ./roark/projects
```

The diff is a flat list: each project, its member changes, and its nested resource changes, tagged with the owning project.

**3. Apply**

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply ./roark/projects --yes
```

Reconciles the whole tree: creates/updates projects, invites/updates members, applies each project's resources, and (unless `--no-prune`) removes config-managed projects and members you deleted from git.

### Raw HTTP

The bundle and endpoints are the same as project config; you just send it with the organization key.

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/config/apply \
  -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @bundle.json
```

Each change comes back with a `status` (`applied` / `skipped` / `failed`) and, for a project, its `id`. Member and nested-resource changes carry the owning `project`:

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "data": {
    "changes": [
      { "configKey": "project/payments-prod", "kind": "project", "name": "payments-prod", "op": "create", "status": "applied", "id": "..." },
      { "configKey": "member/jane@example.com", "kind": "member", "name": "jane@example.com", "op": "create", "status": "applied", "project": "payments-prod" },
      { "configKey": "agent/payments-va", "kind": "agent", "name": "payments-va", "op": "create", "status": "applied", "project": "payments-prod" }
    ],
    "summary": { "create": 3, "update": 0, "delete": 0, "noop": 0, "failed": 0 }
  }
}
```

## Apply semantics

* **Identity is by name** (`project/<name>`). Re-applying an unchanged project is a no-op, not a rewrite. Members reconcile by email.
* **Prune removes what you deleted.** With prune on (the default), a project or member you remove from git is removed from Roark. Submit the full desired set every time, or use `--no-prune` / `"prune": false` for additive-only applies.

> [!WARNING]
>
> With prune enabled, deleting a `kind: project` file removes that project (and its data) on the next apply. Keep your repo the complete source of truth, or apply with `--no-prune`.

## Imperative provisioning (REST)

For one-off or scripted provisioning outside the declarative flow, the organization key also drives a small REST surface. Config-as-code is the recommended path for GitOps; these are handy for automation that isn't file-based.

| Endpoint | Purpose |
| :- | :- |
| `GET/POST /v1/org/project`, `PUT/DELETE /v1/org/project/{id}` | List, create, update, delete projects |
| `GET/POST /v1/org/project/{id}/member`, `PUT/DELETE .../member/{id}` | List, invite, update role, remove members |
| `GET/POST /v1/org/project/{id}/api-key`, `DELETE .../api-key/{id}` | List, generate, revoke a per-project API key |

Generating a per-project key is how you hand each team or environment its own project-scoped credential from your provisioning pipeline.

## Recommended workflow

1. Keep a `roark/projects/` directory in git, one file per project, reviewed via pull requests.
2. In CI, run `roark config diff ./roark/projects` on every PR (a **read-only** organization key is enough for diff) and post the output for review.
3. On merge, run `roark config apply ./roark/projects --yes` with a read+write organization key stored as a secret.

Teams get self-service projects, members, and configuration through a PR they own; you get a versioned, reviewable, reproducible organization with a full audit trail in git.


# Overview
Source: https://docs.roark.ai/documentation/config-as-code/overview

Define your Roark agents, personas, flows, metrics, collectors, simulation plans and groups, alerts, dashboards and live reports as YAML in git and deploy them with one apply

Config as Code lets you define your Roark resources (agents, personas, simulation flows, custom metrics, collectors, simulation plans and groups, alerts, dashboards, and live reports) as code in your own git repository, then deploy them with a single apply. Author them in **TypeScript** with the [Node SDK](/documentation/sdks/node-sdk) or in **YAML** with the [CLI](/documentation/sdks/cli), whichever fits your stack, both produce the same bundle. Your config repo is the source of truth: Roark reconciles the live project to match what you submitted, creating what's new, updating what changed, and removing what you deleted.

## Quickstart: your first simulation

Roark is a testing platform, so the first thing to build is a test. A simulation needs four things: an **agent** to call, a **caller** (persona), the **conversation** to run (a flow), and a **metric** to grade it. Define all four, apply, and run.

Set `ROARK_API_BEARER_TOKEN` first (the key needs `config:apply` and permission to run simulations). The TypeScript helpers live at `@roarkanalytics/sdk/lib/config`; the YAML path uses the [CLI](/documentation/sdks/cli).

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import {
  defineAgent,
  definePersona,
  defineImprovFlow,
  defineSimulationPlan,
  defineConfig,
} from '@roarkanalytics/sdk/lib/config';

// The agent under test, and how to reach it.
const frontdesk = defineAgent({
  name: 'frontdesk',
  endpoints: [{ value: '+15551234567', direction: 'OUTGOING' }],
});

// The caller. Keep it simple to start; add accent, emotion, and a backstory later.
const caller = definePersona({
  name: 'neutral-caller',
  language: 'EN',
  accent: 'US',
  gender: 'FEMALE',
});

// The conversation to test, graded against your expectations.
const billingQuestions = defineImprovFlow({
  name: 'billing-questions',
  agents: ['frontdesk'],
  happyPath: {
    persona: 'neutral-caller',
    environment: 'Quiet line',
    prompt: 'You want to understand a charge on your latest invoice.',
    expectations: ['Agent explains the charge clearly'],
  },
});

// The saved test: run the flow against the agent, graded by a metric.
const firstTest = defineSimulationPlan({
  name: 'first-test',
  direction: 'OUTBOUND',
  agentEndpoints: [{ agent: 'frontdesk', value: '+15551234567' }],
  flows: [{ flow: 'billing-questions' }],
  metrics: ['task_completion'],
  maxDurationSeconds: 300,
});

export default defineConfig({ resources: [frontdesk, caller, billingQuestions, firstTest] });
```

```yaml roark/ (one file per resource) theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# agents/frontdesk.yaml
kind: agent
name: frontdesk
endpoints:
  - value: '+15551234567'
    direction: OUTGOING
---
# personas/neutral-caller.yaml
kind: persona
name: neutral-caller
language: EN
accent: US
gender: FEMALE
---
# flows/billing-questions.yaml
kind: flow
type: improv
name: billing-questions
agents: [frontdesk]
happyPath:
  persona: neutral-caller
  environment: Quiet line
  prompt: You want to understand a charge on your latest invoice.
  expectations:
    - Agent explains the charge clearly
---
# simulation-plans/first-test.yaml
kind: simulationPlan
name: first-test
direction: OUTBOUND
agentEndpoints:
  - agent: frontdesk
    value: '+15551234567'
flows:
  - flow: billing-questions
metrics:
  - task_completion
maxDurationSeconds: 300
```

Preview, apply, then run the plan (`diff`/`apply` take a `roark.config.ts` or a `./roark` YAML directory):

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config diff roark.config.ts                      # preview the changes, no writes
roark config apply roark.config.ts                     # create the agent, persona, flow, and plan
roark simulation run --data '{"planId":"<plan-id>"}'   # run it (find the id with: roark simulation plan list)
```

That is a real test: a flow run against your agent, graded by a metric. Add edge cases, more flows, personas, collectors, alerts and dashboards the same way, each kind has its own reference page below.

> [!TIP]
>
> `caller`, `neutral-caller`, and `Quiet line` are a minimal setup. Roark also ships a roster of built-in personas and environments you can pick from in the dashboard; config references the personas in your own project, so define the callers you want to version in git.

> [card] [Full walkthrough: run a simulation](/documentation/config-as-code/run-a-simulation)
>
> Define, apply, run, read the pass/fail, and gate CI on it, step by step.

***

> [!NOTE]
>
> Config as Code manages **resource definitions**: applying reconciles them, it never places a call. To actually run a simulation you have defined, see **[Run a simulation](/documentation/config-as-code/run-a-simulation)**: start it from the CLI, gate CI on it, or give the plan a [`schedule`](/documentation/config-as-code/simulation-plans#running-on-a-schedule) so it runs itself.

You write only a human-readable `name` for each resource. Roark derives a stable identity (`configKey = <kind>/<name>`) and resolves cross-references by name, so there are no UUIDs in your files and no state file to keep in sync.

> [card] [Use the Roark CLI](/documentation/sdks/cli)
>
> The easiest way to run Config as Code. `roark config diff ./roark` and `roark config apply ./roark` bundle your directory (resolving `file://` prompts) and submit it for you. See the CLI page for install, auth, and a CI example.

***

## How it works

You submit the full desired set of resources to a single endpoint. Roark:

1. **Parses and validates** every resource against the schema.
2. **Diffs** the submitted set against the resources this project already manages via config.
3. **Reconciles**: creates new resources, updates changed ones, and (unless you opt out) deletes config-managed resources you removed from the submission.

There are two endpoints:

| Endpoint | What it does |
| :- | :- |
| `POST /v1/config/diff` | **Dry run.** Returns the changes that *would* be made. No writes. |
| `POST /v1/config/apply` | **Applies** the changes and returns what happened. |

Always run `diff` first to preview the changes, then `apply`.

***

## Prerequisites

* A Roark API key with the **`config:apply`** permission. Generate one from [API Keys](/documentation/getting-started/api-keys) and confirm it carries `config:apply`.
* A git repository to hold your config files (any layout; Roark reads the files you submit).
* The **Roark CLI** (recommended): `npm install -g @roarkanalytics/cli`, or run it on demand with `npx @roarkanalytics/cli`. The CLI builds the bundle from your config directory and drives `diff`/`apply` for you, so it's the easiest way to deploy. Raw HTTP and the SDKs work too.

> [!NOTE]
>
> A **project** API key is scoped to a single project, and everything you apply with it lands in that project. To provision **projects, members, and each project's resources across the whole organization** from one repo, use an **Organization** API key with `kind: project` resources. See [Organizations & Projects](/documentation/config-as-code/organizations).

***

## Repository layout

One file per resource, discriminated by `kind`. A conventional layout:

```
roark/
  agents/frontdesk.yaml
  personas/frustrated-caller.yaml
  flows/frustrated-rebooking.yaml     # improv flow
  flows/booking-scripted.yaml         # scripted-graph flow
  metrics/refund-policy-accuracy.yaml
  collectors/consent-on-frontdesk.yaml
  plans/nightly-regression.yaml       # simulation plan, optionally on a schedule
  groups/weekly-release-review.yaml   # plans run together, one report to email and Slack
  alerts/high-frustration.yaml        # threshold / event / simulation alert
  prompts/escalates-to-manager.md     # referenced by file://
```

Add this header to any resource file for editor autocomplete and validation:

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
```

***

## Resource kinds

One file per resource, discriminated by `kind`. Each kind has its own reference page with fields and examples:

> [card] [Agents](/documentation/config-as-code/agents)
>
> Voice agents and their phone endpoints.

> [card] [HTTP request definitions](/documentation/config-as-code/http-request-definitions)
>
> The HTTP request Roark fires to place outbound calls through your own system.

> [card] [Personas](/documentation/config-as-code/personas)
>
> The simulated caller for a flow.

> [card] [Flows](/documentation/config-as-code/flows)
>
> Simulation flows: improvised or scripted graphs.

> [card] [Metrics](/documentation/config-as-code/metrics)
>
> Custom metrics: LLM judges, API checks and thresholds.

> [card] [Collectors](/documentation/config-as-code/collectors)
>
> Which metrics get collected on which conversations.

> [card] [Simulation plans](/documentation/config-as-code/simulation-plans)
>
> Saved, repeatable simulation runs: agents, flows and the metrics that grade them.

> [card] [Simulation groups](/documentation/config-as-code/simulation-groups)
>
> Several plans run together, with one combined report sent as a PDF to email and Slack.

> [card] [Alerts](/documentation/config-as-code/alerts)
>
> Alerts (monitors): threshold, event, and simulation triggers.

> [card] [Dashboards](/documentation/config-as-code/dashboards)
>
> Dashboards and their widgets: charts over metric values and call counts.

> [card] [Live reports](/documentation/config-as-code/live-reports)
>
> A recurring written report on production calls: targets, trends, failure themes and recommendations.

> [card] [Organizations & Projects](/documentation/config-as-code/organizations)
>
> Provision projects, their members, and each project's resources across the org (organization key).

> [!NOTE]
>
> For the full field reference of every kind, see the [Config DSL reference](https://roark.ai/roark-config.schema.json) schema.

***

## Deploying

The easiest way to deploy is the **Roark CLI**. Point it at your config, a YAML directory or a `roark.config.ts`, and it builds the bundle for you (reading every YAML file and inlining `file://` prompts, or evaluating your TypeScript config), so there is no JSON body to assemble by hand.

> [!TIP]
>
> The [CLI](/documentation/sdks/cli) does the bundling for you: `roark config diff ./roark` (or `roark config diff roark.config.ts`) and the matching `apply` take your config directly, build the bundle, and submit it. With no argument it auto-detects a `roark.config.ts` in the current directory. The raw requests below are what it sends.

**1. Authenticate**

Give the CLI the project API key that carries `config:apply`:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
export ROARK_API_BEARER_TOKEN="<your-config-apply-key>"
# or store it once: roark auth login
```

**2. Preview the changes**

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config diff ./roark
```

The CLI reads every resource under `./roark`, builds the bundle, and prints one line per change (`+` create, `~` update, `-` delete) with a tally:

```text theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
  + collector/consent-on-frontdesk
  ~ agent/frontdesk

1 to create, 1 to update, 0 to delete
```

Resources already in sync are no-ops and aren't listed, so a project that fully matches your config prints `0 to create, 0 to update, 0 to delete`.

**3. Apply**

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply ./roark
```

`apply` previews the same diff, asks you to confirm, then reconciles and reports what it did. Pass `--yes` to skip the prompt in CI, and `--no-prune` for an additive-only apply that never deletes.

### Using raw HTTP

If you'd rather call the API directly, bundle your resources into a single JSON body: `{ "resources": [...], "prune": true }`, where each entry is one resource in the same shape as its YAML. `POST` it to `/v1/config/diff` first, then `/v1/config/apply`.

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/config/diff \
  -H "Authorization: Bearer $ROARK_API_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @bundle.json
```

The response lists each projected change with an `op` (`create`, `update`, or `delete`) plus a summary; in-sync resources are counted in `summary.noop` and omitted from `changes`:

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "data": {
    "changes": [
      { "configKey": "collector/consent-on-frontdesk", "kind": "collector", "name": "consent-on-frontdesk", "op": "create" }
    ],
    "summary": { "create": 1, "update": 0, "delete": 0, "noop": 0 }
  }
}
```

`apply` takes the same body against `/v1/config/apply`; each change comes back with a `status` (`applied` or `failed`) and, on success, the resource `id`.

> [!NOTE]
>
> These endpoints are available in the [Node.js](/documentation/sdks/node-sdk) and [Python](/documentation/sdks/python-sdk) SDKs as `config.diff` and `config.apply`, taking the same bundle, and in the [CLI](/documentation/sdks/cli) as `roark config diff` and `roark config apply`, taking a YAML directory or a TypeScript config file.

***

## Apply semantics

* **Identity is by name.** Re-submitting an unchanged resource updates it in place; it never creates a duplicate. Renaming a resource is a delete of the old name plus a create of the new one.
* **Cross-references resolve by name** within the same submission (a flow's `agents:`/`persona:`, a collector's `AGENT` filter). The referenced resource must be in the bundle or already config-managed in the project.
* **Prune deletes what you removed.** By default, config-managed resources absent from the submission are deleted so the project matches your repo exactly. To layer additive changes without deleting, send `"prune": false`.
* **Prompts are code.** Any prompt field takes an inline string or `file://relative/path.md`, resolved relative to your config root and inlined before you submit.
* **Idempotent.** Applying the same bundle twice converges to the same state. An unchanged resource is a no-op on the next `diff`/`apply`, not a rewrite, so a re-run of an in-sync project reports no changes.

> [!WARNING]
>
> With `prune` enabled (the default), a resource you delete from your repo is deleted from Roark on the next apply. Submit the **full** desired set every time, or use `"prune": false` for additive-only applies.

***

## Config-managed resources in the UI

A resource created by config is **read-only in the dashboard** and carries a "managed by config" badge. To change it, edit your config and re-apply.

If you need to hand a resource back to manual UI editing, **detach** it (from the resource's menu in the dashboard). Detaching clears its config ownership:

* A later apply that still lists it will re-adopt it.
* A later apply that omits it will simply leave it alone (it is no longer config-managed, so prune won't touch it).

***

## Recommended workflow

1. Keep your config in a git repo (a `roark/` YAML directory or a `roark.config.ts`), reviewed via pull requests.
2. In CI, run `roark config diff ./roark` (or `roark config diff roark.config.ts`) on every PR and post the output for review.
3. On merge to your main branch, run `roark config apply ./roark --yes` (or `roark config apply roark.config.ts --yes`).

Both CI steps read the API key from `ROARK_API_BEARER_TOKEN` (store it as a secret with `config:apply`). This gives you versioned, reviewable, reproducible Roark resources with a full audit trail in git.

For a copy-pasteable GitHub Actions workflow that does exactly this, see [Using the CLI in CI](/documentation/sdks/cli#using-the-cli-in-ci).


# Personas
Source: https://docs.roark.ai/documentation/config-as-code/personas

Define the simulated caller as config

The simulated caller for a flow. Self-contained (references nothing).

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { readFileSync } from 'node:fs';
import { definePersona } from '@roarkanalytics/sdk/lib/config';

const frustratedCaller = definePersona({
  name: 'frustrated-caller',
  displayName: 'Dana Whitfield',
  language: 'EN',
  accent: 'US',
  gender: 'FEMALE',
  baseEmotion: 'FRUSTRATED',
  backstoryPrompt: readFileSync('./prompts/frustrated-caller.md', 'utf8'),
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: persona
name: frustrated-caller
displayName: Dana Whitfield
language: EN
accent: US
gender: FEMALE
baseEmotion: FRUSTRATED
backstoryPrompt: file://prompts/frustrated-caller.md
```

* `name` is the local key; `displayName` is the name the agent hears (defaults to `name`).
* `language`, `accent`, and `gender` are required; behavioral knobs (`baseEmotion`, `speechPace`, `speechClarity`, `intentClarity`, `confirmationStyle`, `memoryReliability`, `backgroundNoise`, and the idle-message fields) are optional and default sensibly.
* `backstoryPrompt` is the caller's background: a plain string in TypeScript (inline or read from a file), or inline / a `file://` reference in YAML resolved from your config root.
* A flow references a persona by its `name`.

> [!NOTE]
>
> For the full field reference of every kind, see the [Config DSL reference](https://roark.ai/roark-config.schema.json) schema.


# Run a simulation
Source: https://docs.roark.ai/documentation/config-as-code/run-a-simulation

Zero to a scored simulation with config as code: define an agent and a plan, apply, run it, read the result, and gate CI on it

The end-to-end path, on one page: define a simulation in your repo, apply it, run it, and read the pass/fail. Config as code defines the resources; running is a separate, deliberate step (so applying never places a surprise call).

> [!NOTE]
>
> You need the [CLI](/documentation/sdks/cli) (`npm install -g @roarkanalytics/cli`) and a Roark API key with **`config:apply`** and permission to **start simulations and read results**. Set it once: `export ROARK_API_BEARER_TOKEN="<key>"`.

## 1. Define an agent and a plan

The minimum to run something: an **agent** to call and a **simulation plan** that says which flows to run against it and which metrics grade the result. This plan runs a built-in (`system`) flow, so there's no persona or flow to author first.

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineAgent, defineSimulationPlan, defineConfig } from '@roarkanalytics/sdk/lib/config';

const frontdesk = defineAgent({
  name: 'frontdesk',
  endpoints: [{ value: '+15551234567', direction: 'OUTGOING' }],
});

const nightly = defineSimulationPlan({
  name: 'nightly-screener-suite',
  direction: 'OUTBOUND',
  agentEndpoints: [{ agent: 'frontdesk', value: '+15551234567' }],
  metrics: ['call_screening_encountered', 'call_screening_handling_score'],
  flows: [{ system: 'sf-screener-apple' }],
  maxDurationSeconds: 300,
});

export default defineConfig({ resources: [frontdesk, nightly] });
```

```yaml roark/simulation-plans/nightly.yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
# (plus roark/agents/frontdesk.yaml defining the agent)
kind: simulationPlan
name: nightly-screener-suite
direction: OUTBOUND
agentEndpoints:
  - agent: frontdesk
    value: '+15551234567'
metrics:
  - call_screening_encountered
  - call_screening_handling_score
flows:
  - system: sf-screener-apple
maxDurationSeconds: 300
```

Swap the `system` flow for your own [customer flows](/documentation/config-as-code/flows) and [personas](/documentation/config-as-code/personas) once you want to test your own scenarios. `roark simulation template list` prints every built-in flow slug.

## 2. Apply it

Create the resources (preview first, then apply):

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config diff roark.config.ts    # or ./roark for a YAML directory
roark config apply roark.config.ts
```

This creates the agent and the plan. It does **not** run anything yet.

## 3. Run it

Running is a separate command. It takes the plan's **id**, not its config name, so look the id up once (or copy it from the plan's page in the dashboard):

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# find the plan's id by its name
roark simulation plan list | jq -r '.data[] | select(.name=="nightly-screener-suite") | .id'

# start a run
roark simulation run --data '{"planId":"<plan-id>"}'
```

`roark simulation run` returns a **job id** (the run). It places the calls, grades each against the plan's metrics, and the result settles when every call has been scored.

> [!TIP]
>
> Prefer not to leave your pipeline? The [GitHub Action](#5-gate-ci-on-it) starts the run and waits for it in one step, and the [Node SDK](/documentation/sdks/node-sdk) exposes the same via `client.simulation.run({ planId })`.

## 4. Read the result

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation plan job result <job-id>
```

You get the per-check pass/fail against the plan's [success criteria](/documentation/simulation-testing/run-plans#success-criteria), plus a link to the full report in the dashboard. Roark decides pass vs. fail, so the CLI, the dashboard, and your CI pipeline never disagree.

To have each run's report delivered instead of fetching it, add [`notify`](/documentation/config-as-code/simulation-plans#sending-each-runs-report) to the plan: the report PDF goes by email (to anyone, Roark account or not) and to Slack once it is ready.

## 5. Gate CI on it

For a pipeline, skip the manual run and let the [GitHub Action](https://github.com/roarkhq/simulation-action) start the run, wait, and fail the build when a check falls short:

```yaml .github/workflows/agent-regression.yml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
name: Agent regression
on:
  push:
    branches: [main]
jobs:
  simulate:
    runs-on: ubuntu-latest
    steps:
      # Optional: keep the plan itself in the repo and apply it first.
      - uses: actions/checkout@v4
      - run: npx -y @roarkanalytics/cli config apply roark.config.ts --yes
        env:
          ROARK_API_BEARER_TOKEN: ${{ secrets.ROARK_API_KEY }}
      # Start the run, wait for it, and gate the build on the result.
      - uses: roarkhq/simulation-action@v1
        with:
          api-token: ${{ secrets.ROARK_API_KEY }}
          plan-id: <your-plan-id>
```

See [CI/CD](/documentation/simulation-testing/ci-cd) for the full action reference.

## Or: run it on a schedule

To run without a trigger at all, add a [`schedule`](/documentation/config-as-code/simulation-plans#running-on-a-schedule) to the plan and apply, the plan runs itself on the cadence you set. Pair it with [`notify`](/documentation/config-as-code/simulation-plans#sending-each-runs-report) and every scheduled run's report lands in your inbox or Slack.

> [card] [Simulation plans reference](/documentation/config-as-code/simulation-plans)
>
> Every field a plan takes: agents, flows, metrics, direction, scheduling, report delivery, and knowledge checks.


# Simulation groups
Source: https://docs.roark.ai/documentation/config-as-code/simulation-groups

Run several simulation plans together and send one combined report to email and Slack

A simulation group is a saved set of [simulation plans](/documentation/config-as-code/simulation-plans) that run together. Every time the group runs, each plan starts once and Roark reads the runs as one: a single **group report** covering what went well, the issues found across every run (with call-level evidence) and recommendations, delivered as a PDF to whoever you choose.

Use a group when one release review spans several plans, for example a missions suite, a stress suite and a voice sweep, and the people reading the result want one report rather than three.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineSimulationGroup } from '@roarkanalytics/sdk/lib/config';

const weeklyReleaseReview = defineSimulationGroup({
  name: 'weekly-release-review',
  description: 'Every plan the pickup agent has to pass before a release',
  plans: [
    'mission-1-pickup-lifecycle', // a kind: simulationPlan in this config
    'Mission 2 address change', // or a plan made in the dashboard, by its exact name
    'conversational-stress',
  ],
  schedule: {
    type: 'weekly',
    days: ['monday'],
    time: '09:00',
    timezone: 'America/New_York',
  },
  notify: {
    email: [
      'qa-lead@example.com',
      'vp-operations@example.com', // does not need a Roark account
    ],
    slack: [{ channelId: 'C0123ABCXYZ', channelName: '#voice-qa' }],
  },
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: simulationGroup
name: weekly-release-review
description: Every plan the pickup agent has to pass before a release
plans:
  - mission-1-pickup-lifecycle      # a kind: simulationPlan in this config
  - Mission 2 address change        # or a plan made in the dashboard, by its exact name
  - conversational-stress
schedule:
  type: weekly
  days: [monday]
  time: '09:00'
  timezone: America/New_York
notify:
  email:
    - qa-lead@example.com
    - vp-operations@example.com     # does not need a Roark account
  slack:
    - channelId: C0123ABCXYZ
      channelName: '#voice-qa'
```

> [!NOTE]
>
> **Applying a group never starts a run.** It declares the group. A group runs when its [`schedule`](#running-on-a-schedule) fires, when someone presses **Run now** on it in the dashboard, or when you start it [from the CLI or API](#running-a-group-on-demand).

***

## Fields

| Field | Required | |
| :- | :- | :- |
| `name` | yes | The group's config name, and also its name in the dashboard |
| `description` | no | Free text. Removing it clears it. |
| `plans` | yes | 2 to 20 plan names, in the order they run and appear in the report. Each plan once. |
| `schedule` | no | Run the group on a cadence. The same block as a [plan's schedule](/documentation/config-as-code/simulation-plans#running-on-a-schedule). |
| `notify` | no | Where the report goes. Omitted: email every member of the project, no Slack. |

***

## Which plans run

Each entry in `plans` resolves by name, in this order:

1. A `kind: simulationPlan` in the same bundle, or one config already manages.
2. A plan made in the dashboard with exactly that name. Write it as it appears in Roark, spaces and capitals included.

A name that matches no plan, or more than one, fails that group's apply with the plan named, so a group never quietly runs a different plan than the one you meant.

A plan can belong to several groups. While a group in your config still lists a plan, pruning won't delete that plan: apply keeps it and reports which group needs it. If a plan in a group is deleted from the dashboard, the group's next run fails before starting anything, naming the missing plan. It never runs a partial set.

***

## Running on a schedule

`schedule` takes exactly the same block as a plan's: `hourly`, `interval`, `daily`, `weekly`, `monthly` or `once`, with `timezone`, `enabled`, `endDate` and `maxRuns`. See [Running on a schedule](/documentation/config-as-code/simulation-plans#running-on-a-schedule) for every field.

The ownership rules match too: declaring `schedule` puts the group's schedule under config, removing the block deletes it, and `enabled: false` pauses it without deleting it.

Each fire is attempted once. Plans start one after another and a started run can't be taken back, so a retry could start some plans twice. If a plan can't start, the plans that did start are still grouped and reported on, and the fire is recorded as failed.

> [!WARNING]
>
> A group run places real, billed calls for **every** plan in it. Check the combined size of your plans before choosing a cadence, or set `maxRuns` while you trial a new group.

> [!NOTE]
>
> A scheduled run can't prompt for a variable, so every variable the group's plans use needs a default. See [Variables](/documentation/simulation-testing/variables).

***

## Who gets the report

When every run in the group has finished, Roark writes the group report and announces it. The report PDF goes with every announcement, so a reader who never logs in to Roark still gets the whole report.

### Email

| `notify.email` | Who is emailed |
| :- | :- |
| `true` (default) | Every member of the project |
| `false` | Nobody. The report is still on the group's page in the dashboard. |
| A list of addresses (1 to 50) | Exactly those people, whether or not they have a Roark account |

Each email carries the verdict, a link to the report in Roark, and the full report attached as a PDF. A list is the way to send the report straight to executives or stakeholders outside your Roark project.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
notify: {
  email: ['cto@example.com', 'head-of-cx@example.com'],
}
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
notify:
  email: [cto@example.com, head-of-cx@example.com]
```

An empty list (`email: []`) is rejected rather than read as "nobody" or "everyone". Write `false` or `true`.

### Slack

`notify.slack` takes up to 10 channels, in the same `{ channelId, channelName }` shape an [alert](/documentation/config-as-code/alerts) uses. Each channel gets a summary message (the verdict, runs passed, calls, issue count and the top three issues, with a **Read the report** button), and the report PDF is posted in that message's thread.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
notify: {
  email: false,
  slack: [
    { channelId: 'C0123ABCXYZ', channelName: '#exec-updates' },
    { channelId: 'C0456DEFUVW', channelName: '#voice-qa' },
  ],
}
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
notify:
  email: false
  slack:
    - channelId: C0123ABCXYZ
      channelName: '#exec-updates'
    - channelId: C0456DEFUVW
      channelName: '#voice-qa'
```

**1. Connect Slack to the project**

Slack is connected once, in the dashboard, under **Settings → Project → Notifications**. Connecting is an approval in Slack itself, so it can't be done from config. A group that names a Slack channel fails to apply until the project has a Slack connection.

**2. Find each channel's id**

In Slack, open the channel, click its name, and copy the **Channel ID** at the bottom of the **About** tab (it looks like `C0123ABCXYZ`). The id is what identifies the channel; `channelName` is only the label you'll see in Roark.

**3. Invite Roark to private channels**

Roark joins public channels on its own. For a private channel, add the Roark app to it first: open the channel's details, then **Integrations → Add apps**.

> [!NOTE]
>
> Posting the PDF needs Slack's file upload permission, which older Slack connections don't have. On those, the summary message still arrives but without the PDF. Reconnect Slack under **Settings → Project → Notifications** to grant it.

### When settings change

Each run copies the group's notification settings when it starts. Changing `notify` and re-applying affects the next run, never a report that's already being written.

Like every other field, deleting the `notify` block (or one of its lines) reverts it to the default: email every project member, no Slack.

Plans take the same `notify` block for each run's own report. See [Sending each run's report](/documentation/config-as-code/simulation-plans#sending-each-runs-report).

***

## Running a group on demand

Applying never starts a run, so a one-off run is its own step. From the CLI, using the group's id (from `roark simulation group list`):

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply ./roark
roark simulation group run <group-id>
```

The command returns the **run group**: the runs it started and, under `latestReport`, the report's progress. The report waits for every run to finish. Poll it until `latestReport.status` is `READY` (or `FAILED`):

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run-group get <run-group-id>
```

The same operations are in the API (`POST /v1/simulation/group/{groupId}/run`, then `GET /v1/simulation/run-group/{runGroupId}`) and in the [Node.js](/documentation/sdks/node-sdk) and [Python](/documentation/sdks/python-sdk) SDKs. A run started this way announces its report through `notify` exactly as a scheduled run does.

***

## In the dashboard

Saved groups are listed under **Simulations → Runs → Groups**, above the history of group runs. Each row shows the group's schedule and where its report goes, and has **Run now**. Each group run opens a page with the combined report and a PDF download.

A group created by config shows **Managed by config**. You can still run it from the dashboard, but its plans and its **Report delivery** settings are read-only there: change them in your config and re-apply.

A group made in the dashboard with the same name as a `kind: simulationGroup` is adopted on the first apply rather than duplicated, as long as exactly one unmanaged group has that name.

***

## Removing a group

Deleting a group from your config (with prune on, the default) deletes the group and its schedule. The group runs and reports it already produced stay, so past reports remain readable.

***

## Apply order

A group lists plans by name, so one apply resolves it after the plans and before the alerts:

```
metric → persona → httpRequestDefinition → agent → tool → flow → collector → simulationPlan → simulationGroup → alert
```

Deletes run in reverse, so a pruned group is removed before any plan it listed.

> [!NOTE]
>
> For the full field reference of every kind, see the [Config DSL reference](https://roark.ai/roark-config.schema.json) schema.


# Simulation plans
Source: https://docs.roark.ai/documentation/config-as-code/simulation-plans

Declare a saved, repeatable simulation run

A saved, repeatable simulation: which agents to call, which flows to run against them, and which metrics to grade the result on (the config form of a [run plan](/documentation/simulation-testing/run-plans)).

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineSimulationPlan } from '@roarkanalytics/sdk/lib/config';

const nightlyScreenerSuite = defineSimulationPlan({
  name: 'nightly-screener-suite',
  direction: 'OUTBOUND',
  agentEndpoints: [{ agent: 'frontdesk', value: '+15551234567' }],
  metrics: ['call_screening_encountered', 'call_screening_handling_score'],
  flows: [
    { flow: 'refund-request' },
    { system: 'sf-screener-apple', edgeCases: ['screener-declines-the-call'] },
  ],
  maxDurationSeconds: 300,
  // Optional: send each run's report, with the PDF, by email and to Slack.
  notify: {
    email: ['qa-lead@example.com'],
    slack: [{ channelId: 'C0123ABCXYZ', channelName: '#voice-qa' }],
  },
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: simulationPlan
name: nightly-screener-suite
direction: OUTBOUND
agentEndpoints:
  - agent: frontdesk
    value: '+15551234567'
metrics:
  - call_screening_encountered
  - call_screening_handling_score
flows:
  - flow: refund-request
  - system: sf-screener-apple
    edgeCases: [screener-declines-the-call]
maxDurationSeconds: 300
# Optional: send each run's report, with the PDF, by email and to Slack.
notify:
  email: [qa-lead@example.com]
  slack:
    - channelId: C0123ABCXYZ
      channelName: '#voice-qa'
```

`notify` is optional; see [Sending each run's report](#sending-each-runs-report).

> [!NOTE]
>
> **Applying a plan never starts a run.** Applying reconciles the definition. A one-off run is a separate, deliberate act:
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> roark config apply ./roark
> roark simulation run --data '{"planId":"<plan-id>"}'
> ```
>
> A recurring run is declared on the plan itself with a [`schedule`](#running-on-a-schedule). Either way, the plan exists as a durable resource rather than as a side effect of having run something, which is what lets an [alert](/documentation/config-as-code/alerts) name it.

***

## Running on a schedule

Add a `schedule` block and the plan runs itself. It is the config form of a plan's [schedule](/documentation/simulation-testing/schedules): the same cadences, the same one schedule per plan.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineSimulationPlan } from '@roarkanalytics/sdk/lib/config';

const nightlyScreenerSuite = defineSimulationPlan({
  name: 'nightly-screener-suite',
  direction: 'OUTBOUND',
  agentEndpoints: [{ agent: 'frontdesk', value: '+15551234567' }],
  metrics: ['call_screening_handling_score'],
  flows: [{ system: 'sf-screener-apple' }],
  maxDurationSeconds: 300,
  schedule: {
    type: 'weekly',
    days: ['monday', 'tuesday', 'wednesday', 'thursday', 'friday'],
    time: '02:00',
    timezone: 'America/New_York',
  },
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: simulationPlan
name: nightly-screener-suite
direction: OUTBOUND
agentEndpoints:
  - agent: frontdesk
    value: '+15551234567'
metrics:
  - call_screening_handling_score
flows:
  - system: sf-screener-apple
maxDurationSeconds: 300
schedule:
  type: weekly
  days: [monday, tuesday, wednesday, thursday, friday]
  time: '02:00'
  timezone: America/New_York
```

| `type` | Fields | Runs |
| :- | :- | :- |
| `hourly` | `minute` (0 to 59, default `0`) | every hour at that many minutes past |
| `interval` | `everyMinutes` (15 to 59) | every N minutes |
| `daily` | `time` | every day at that time |
| `weekly` | `days`, `time` | on those days (`monday` to `sunday`) at that time |
| `monthly` | `dayOfMonth` (1 to 31), `time` | on that day of the month at that time; a month without that day is skipped |
| `once` | `at` | a single time, at an exact instant |

Every type also takes:

| Field | Default | |
| :- | :- | :- |
| `timezone` | `UTC` | IANA zone the clock times are read in, e.g. `America/Chicago`. Schedules follow daylight saving. |
| `enabled` | `true` | `false` pauses the schedule without deleting it |
| `endDate` | | recurring types: stop after this date (ISO 8601 with an offset or `Z`) |
| `maxRuns` | | recurring types: stop after this many runs. One of `endDate` or `maxRuns`, not both. |

Write `time` as a quoted 24-hour `'HH:MM'`. `once` takes a full timestamp with its offset (`at: '2026-11-02T09:00:00-05:00'`) and ignores `timezone`.

### Who owns the schedule

* **Declaring `schedule` puts it under config.** Apply creates it and keeps it in step with the file. In the dashboard the plan's **Automation** panel shows it as **Managed by config**, with no Edit, Pause or Delete: change it in the file and re-apply. Pause by setting `enabled: false`.
* **Removing the block deletes the schedule.** The plan stays and keeps running on demand.
* **A plan that never declares `schedule` leaves the dashboard's schedule alone.** Bringing an already-scheduled plan into config does not stop its automation.
* **Declaring `schedule` on a plan that already has one adopts it** rather than adding a second, so the plan never fires twice.
* **Removing the plan removes its schedule.** A deleted plan never keeps firing.

Detaching the plan from config in the dashboard hands its schedule back to the dashboard too.

> [!WARNING]
>
> Every scheduled run places real, billed calls. Start with a daily or weekly cadence, or set `maxRuns` while you trial a new plan.

> [!NOTE]
>
> A scheduled run can't prompt for a variable, so every variable the plan's flows use needs a default before you schedule it. If one is missing a default, apply fails for that plan with a missing variable error. See [Variables](/documentation/simulation-testing/variables).

A schedule created by apply is attributed to the user who minted the API key you apply with.

***

## Knowledge checks

Testing that your agent answers a set of questions correctly is common enough that it has its own template. Instead of authoring a flow per question, set `template: question-answer-check` and write the questions and expected answers as a list. Roark asks each one in its own call and grades the answer.

The same template is also a tile in the dashboard's **New simulation run** picker and can be run over the API (see [From the dashboard or API](#from-the-dashboard-or-api) below). Config, dashboard and API all drive the one template.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineQaSimulationPlan } from '@roarkanalytics/sdk/lib/config';

const billingFaqRegression = defineQaSimulationPlan({
  name: 'billing-faq-regression',
  direction: 'OUTBOUND',
  agentEndpoints: [{ agent: 'main-support-agent' }],
  persona: 'curious-caller',
  environment: 'Production Phone',
  maxDurationSeconds: 120,
  questions: [
    {
      ask: 'What are your hours on public holidays?',
      expect: "We're closed on public holidays.",
    },
    {
      ask: 'Can I get a refund after 30 days?',
      expect: 'Refunds are only available within 30 days of purchase.',
    },
  ],
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: simulationPlan
name: billing-faq-regression
template: question-answer-check
direction: OUTBOUND
agentEndpoints:
  - agent: main-support-agent
persona: curious-caller
environment: Production Phone
maxDurationSeconds: 120
questions:
  - ask: 'What are your hours on public holidays?'
    expect: "We're closed on public holidays."
  - ask: 'Can I get a refund after 30 days?'
    expect: 'Refunds are only available within 30 days of purchase.'
```

Each question becomes its **own isolated call**: the caller asks it, and the answer is graded on its own, with no context bleed from the other questions. Grading is LLM-judged, not string matching, so an answer that means the same thing passes ("we don't open on public holidays" satisfies the expectation above).

> [!NOTE]
>
> A knowledge check needs a `persona` (who asks) and an `environment` (which channel), the same two things every simulated call needs. Reference a `kind: persona` in your bundle by name, and an environment by its display name.

### Grading against a knowledge base

Set `gradeWithKnowledgeBase: true` to grade answers against your project's knowledge base as well. A question may then omit `expect`, and its answer is checked for grounding instead, so your knowledge base stays the single source of truth for the ones you don't want to hand-write.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const billingFaqRegression = defineQaSimulationPlan({
  name: 'billing-faq-regression',
  direction: 'OUTBOUND',
  agentEndpoints: [{ agent: 'main-support-agent' }],
  persona: 'curious-caller',
  environment: 'Production Phone',
  maxDurationSeconds: 120,
  gradeWithKnowledgeBase: true,
  questions: [
    {
      ask: 'Can I get a refund after 30 days?',
      expect: 'Refunds are only available within 30 days of purchase.',
    },
    { ask: 'Which payment methods do you accept?' }, // no expect: graded by grounding
  ],
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
kind: simulationPlan
name: billing-faq-regression
template: question-answer-check
direction: OUTBOUND
agentEndpoints:
  - agent: main-support-agent
persona: curious-caller
environment: Production Phone
maxDurationSeconds: 120
gradeWithKnowledgeBase: true
questions:
  - ask: 'Can I get a refund after 30 days?'
    expect: 'Refunds are only available within 30 days of purchase.'
  - ask: 'Which payment methods do you accept?' # no expect: graded by grounding
```

A question needs something to grade against: give it an `expect`, or turn on `gradeWithKnowledgeBase`. A question with neither is refused.

> [!NOTE]
>
> A knowledge check generates a flow behind the scenes (one variant per question). It appears in your flow library named after the plan. Edit the plan, not the generated flow: the next apply rewrites it from the questions.

The metrics are added for you, so a knowledge check does not need a `metrics` list: **Agent Expectations** grades each `expect`, and **Knowledge Base Grounding** is added when `gradeWithKnowledgeBase` is on. Anything you add under `metrics` is collected on top.

### From the dashboard or API

The same `question-answer-check` template is available two other ways, driving the identical run:

* **Dashboard**: **New simulation run → Question & Answer Check**. Pick the persona and environment, add your question/expected-answer rows, and run. There is no flow picker: the flow is generated from the questions.
* **API**: `POST /v1/simulation/run` with the template and your questions:

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "template": "question-answer-check",
  "direction": "OUTBOUND",
  "agentEndpoints": [{ "id": "3a1d5e7c-9b2f-4a6d-8c31-5f7e9d0a2b4c" }],
  "personaId": "b2c3d4e5-6f70-4a1b-8c2d-3e4f5a6b7c8d",
  "environmentId": "c3d4e5f6-7081-4b2c-9d3e-4f5a6b7c8d9e",
  "questions": [
    { "ask": "What are your hours on public holidays?", "expect": "We're closed on public holidays." },
    { "ask": "Can I get a refund after 30 days?", "expect": "Refunds are only available within 30 days of purchase." }
  ]
}
```

`personaId`, `environmentId` and `questions` are required for this template; `flows` is rejected (the flow is generated). The response is a run plan, the same shape every `POST /v1/simulation/run` returns.

***

## Your flows and Roark's

> [!NOTE]
>
> This and the sections below describe the default (`template: manual`) plan, where you attach flows you authored. A [knowledge check](#knowledge-checks) generates its flow for you, so it skips them.

A flow row names **exactly one** of `flow` or `system`. They resolve through different paths, so a row naming both is refused.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
flows: [
  // A `kind: flow` in your own bundle, resolved by name on apply.
  { flow: 'refund-request', happyPath: true },

  // A Roark-curated flow, by its stable slug. Works in every project without
  // you declaring it, and the slug is the same in every environment.
  { system: 'sf-screener-google', edgeCases: ['screener-takes-a-message'] },

  // Naming neither runs every variant the flow has.
  { system: 'sf-screener-apple' },
]
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
flows:
  # A `kind: flow` in your own bundle, resolved by name on apply.
  - flow: refund-request
    happyPath: true

  # A Roark-curated flow, by its stable slug. Works in every project without
  # you declaring it, and the slug is the same in every environment.
  - system: sf-screener-google
    edgeCases: [screener-takes-a-message]

  # Naming neither runs every variant the flow has.
  - system: sf-screener-apple
```

`roark simulation template list` prints the slug of every curated flow and outcome.

| | What runs |
| :- | :- |
| `happyPath: true` | the flow's default way through |
| `edgeCases: [slug, …]` | those specific outcomes |
| neither | every variant the flow has |

`happyPath` and `edgeCases` are alternatives, not a union: asking for both is refused, because omitting both is already how you say "all of it".

***

## Which line to call

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
agentEndpoints: [
  {
    agent: 'frontdesk', // a `kind: agent` in your bundle
    value: '+15551234567', // which of its endpoints
  },
]
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
agentEndpoints:
  - agent: frontdesk          # a `kind: agent` in your bundle
    value: '+15551234567'     # which of its endpoints
```

`value` is the endpoint's own identifier, spelled the way [`kind: agent`](/documentation/config-as-code/agents) spells it. Omit it when the agent has exactly one endpoint; apply refuses to guess between several rather than dial a different number than you meant.

***

## Everything else has a default

Only `name`, `direction`, `agentEndpoints`, `metrics`, `flows` and `maxDurationSeconds` are required. The rest default to what the dashboard would give you:

| Field | Default |
| :- | :- |
| `iterations` | `1` |
| `maxConcurrentJobs` | `1` |
| `executionMode` | `PARALLEL` |
| `silenceTimeoutSeconds` | `30` |
| `endCallPhrases` | `['goodbye']` |
| `endCallReasons` | `[]` |
| `includeFlowMetrics` | `true` |
| `enrichWithLiveConversation` | `false` |
| `schedule` | none: the plan runs on demand |
| `notify` | none: config leaves report delivery as it is |

> [!WARNING]
>
> Deleting a line reverts that setting to its default. It does not leave the previous value in place. That is what makes the file the source of truth: if a plan drifts from its config, the next apply pulls it back rather than quietly keeping both.

> [!NOTE]
>
> **Success criteria are not expressible in config yet.** `metrics` takes slugs, so every pass/fail check on a config-managed plan is held to the 80% default, and a managed plan rejects edits from the API or the dashboard. If a plan needs a check held to a minimum of its own, keep that plan out of config for now and set its minimums under **Advanced → Success criteria**. See [Success criteria](/documentation/simulation-testing/run-plans#success-criteria).

`endCallReasons` is worth knowing about if you run scripted flows. A simulated caller hangs up on an end-call phrase (something the **agent** says) or on one of these conditions. Running out of scripted steps ends nothing, so a flow scripted to a verdict needs a matching reason or the call stays on the line until `maxDurationSeconds`.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
endCallReasons: [
  'You have told the agent that the person it is calling has declined the call. End the call as soon as you have said so.',
]
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
endCallReasons:
  - You have told the agent that the person it is calling has declined the call. End the call as soon as you have said so.
```

***

## Sending each run's report

When a run of the plan finishes and Roark has written its report, `notify` sends that report by email, to Slack, or both. The report PDF goes with every announcement, so a reader who never logs in to Roark still gets the whole report.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineSimulationPlan } from '@roarkanalytics/sdk/lib/config';

const nightlyBillingRegression = defineSimulationPlan({
  name: 'nightly-billing-regression',
  // ...direction, agentEndpoints, metrics, flows, maxDurationSeconds
  notify: {
    email: [
      'qa-lead@example.com',
      'vp-operations@example.com', // does not need a Roark account
    ],
    slack: [{ channelId: 'C0123ABCXYZ', channelName: '#voice-qa' }],
  },
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
kind: simulationPlan
name: nightly-billing-regression
# ...direction, agentEndpoints, metrics, flows, maxDurationSeconds
notify:
  email:
    - qa-lead@example.com
    - vp-operations@example.com # does not need a Roark account
  slack:
    - channelId: C0123ABCXYZ
      channelName: '#voice-qa'
```

`notify` is the same block a [simulation group](/documentation/config-as-code/simulation-groups#who-gets-the-report) takes for its group report:

| `notify.email` | Who is emailed |
| :- | :- |
| `true` (default) | Every member of the project |
| `false` | Nobody. The report is still on the run's page in the dashboard. |
| A list of addresses (1 to 50) | Exactly those people, whether or not they have a Roark account |

`notify.slack` takes up to 10 `{ channelId, channelName }` channels. Each gets a summary (the verdict, calls analyzed, issue count and the top three issues, with a **Read the report** button) with the report PDF in the message's thread. Slack has to be connected to the project first; see [Slack](/documentation/config-as-code/simulation-groups#slack) for finding channel ids and private channels.

> [!WARNING]
>
> Inside the block, `email` defaults to every project member. A plan that should only post to Slack writes `email: false`, which matters most for a plan on a frequent schedule.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
// Slack only: no email.
notify: {
  email: false,
  slack: [{ channelId: 'C0123ABCXYZ', channelName: '#voice-qa' }],
},
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# Slack only: no email.
notify:
  email: false
  slack:
    - channelId: C0123ABCXYZ
      channelName: '#voice-qa'
```

A few things to know:

* **Who owns it.** Declaring `notify` makes config own the plan's report delivery, and deleting the block turns it off. A plan that never declares `notify` keeps whatever delivery was set in the dashboard before you adopted it into config, the same rule as `schedule`.
* **Once per run.** Each run sends its first report once. Regenerating a report later does not send it again, and a run that ended more than a day before its report was written is not sent at all, so generating reports for old runs never fills anyone's inbox.
* **Every run counts.** Runs started from the dashboard, on the plan's schedule, through the API, or as part of a simulation group all send their report. A plan in a group sends its own run reports as well as the group report.
* **Reports, not alerts.** `notify` delivers the finished report. To hear about a run the moment it fails, add an alert as below.

## Alerting on the result

An [alert](/documentation/config-as-code/alerts) scopes itself to a plan by name, which is why the plan's `name` is also its stored name:

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineAlert } from '@roarkanalytics/sdk/lib/config';

const screenerRegressions = defineAlert({
  name: 'screener-regressions',
  trigger: {
    type: 'simulation',
    conditions: ['FAILURE'],
    plan: 'nightly-screener-suite',
  },
  actions: {
    slack: [{ channelId: 'C0123ABCXYZ', channelName: '#agent-quality' }],
  },
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
kind: alert
name: screener-regressions
trigger:
  type: simulation
  conditions: [FAILURE]
  plan: nightly-screener-suite
actions:
  slack:
    - channelId: C0123ABCXYZ
      channelName: '#agent-quality'
```

Both can live in the same bundle. Apply creates the plan before the alert that names it.

To send each run's full report instead, use the plan's [`notify`](#sending-each-runs-report). To run several plans together and send one combined report (a PDF by email, Slack or both), list them in a [simulation group](/documentation/config-as-code/simulation-groups).

***

## Apply order

Plans reference agents, flows and metrics, and groups and alerts reference plans, so one apply resolves them in that order regardless of how your files are laid out:

```
metric → persona → httpRequestDefinition → agent → tool → flow → collector → simulationPlan → simulationGroup → alert
```

Deletes run in reverse, so a plan is torn down before the flows and agents it points at.

> [!NOTE]
>
> For the full field reference of every kind, see the [Config DSL reference](https://roark.ai/roark-config.schema.json) schema.


# Tools
Source: https://docs.roark.ai/documentation/config-as-code/tools

Define an agent tool as config: its parameters, when it should fire, and how Roark grades and mocks it

A `tool` describes a function your agent can call: its name, the parameters it takes, when it should be invoked, and what a successful result looks like. Roark uses that description to **grade and mock** tool calls in simulations: it checks the agent reached for the right tool at the right moment with sensible arguments, and it stands in a mock result so the conversation can carry on. Scope a tool to one agent with `agent`, or omit it for a project-wide tool.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineTool } from '@roarkanalytics/sdk/lib/config';

const bookAppointment = defineTool({
  name: 'book-appointment',
  description: 'Book a slot in the scheduling system',
  agent: 'frontdesk', // scope to one agent by name; omit for a project-wide tool
  parameters: {
    date: 'string',
    time: 'string',
    partySize: 'number',
  },
  invocationCriteria: 'Call once the caller has confirmed a specific date and time.',
  expectedResult: 'A confirmation id and the booked time.',
  handler: {
    type: 'http',
    url: 'https://api.example.com/appointments',
    method: 'POST',
  },
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: tool
name: book-appointment
description: Book a slot in the scheduling system
agent: frontdesk            # scope to one agent by name; omit for a project-wide tool
parameters:
  date: string
  time: string
  partySize: number
invocationCriteria: Call once the caller has confirmed a specific date and time.
expectedResult: A confirmation id and the booked time.
handler:
  type: http
  url: https://api.example.com/appointments
  method: POST
```

| Field | Default | Notes |
| :- | :- | :- |
| `name` | | The local key (`^[a-z0-9][a-z0-9-_.]*$`) and, with the kind, the config identity |
| `description` | | What the tool does, in plain language. Used to judge whether the agent reached for the right tool |
| `parameters` | | A flat map of parameter name to its type: `string`, `number`, or `boolean` |
| `invocationCriteria` | | When the tool should fire. Roark grades the call against this |
| `expectedResult` | | What a successful result looks like. Shapes the mock Roark returns to the agent |
| `agent` | project-wide | The `name` of a [`kind: agent`](/documentation/config-as-code/agents) to scope the tool to. Omit for a tool every agent in the project can use |
| `handler` | | How the tool executes (see [Handler](#handler)). Optional; reference metadata only |

> [!NOTE]
>
> A tool definition is **grading and mock metadata**, not an integration. In a simulation Roark scores whether your agent called the tool correctly and returns a mock result so the call can continue; it does **not** execute your backend. Defining a tool changes how Roark evaluates tool use, nothing about how your live agent runs.

## Parameters

`parameters` is a flat map of parameter name to its type, one of `string`, `number`, or `boolean`. It describes the arguments the agent is expected to pass, so Roark can check the call carried the right ones:

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
parameters: {
  date: 'string',
  time: 'string',
  partySize: 'number',
  waitlist: 'boolean',
}
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
parameters:
  date: string
  time: string
  partySize: number
  waitlist: boolean
```

## Handler

`handler` records **how** the tool executes. Today it is reference metadata, like an agent's prompt: Roark stores it but does not call it. It is read by the hosted runtime and connected-provider push once those consume it, so declaring a handler does not by itself run the tool.

It is optional. One shape exists today, an `http` handler, a public endpoint the tool's invocation would be dispatched to:

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
handler: {
  type: 'http',
  url: 'https://api.example.com/appointments',
  method: 'POST', // optional; GET | POST | PUT | PATCH | DELETE
}
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
handler:
  type: http
  url: https://api.example.com/appointments
  method: POST   # optional; GET | POST | PUT | PATCH | DELETE
```

* `type` is `http` (the discriminator that keeps future handler shapes additive).
* `url` is a public `http(s)` endpoint.
* `method` is optional, one of `GET`, `POST`, `PUT`, `PATCH`, or `DELETE`.

> [!NOTE]
>
> For the full field reference of every kind, see the [Config DSL reference](https://roark.ai/roark-config.schema.json) schema.


# TypeScript
Source: https://docs.roark.ai/documentation/config-as-code/typescript

Define the same Roark resources in TypeScript with define* builders, then apply them with the CLI or Node SDK, a typed alternative to YAML

Config as Code has two front doors to the **same** system. Write your resources as [YAML](/documentation/config-as-code/overview) files, or build them in **TypeScript** with the `define*` helpers in the [Node SDK](/documentation/sdks/node-sdk) (at the subpath `@roarkanalytics/sdk/lib/config`). Both produce the identical bundle and go through the identical `diff` / `apply`, so you can pick whichever fits your codebase, or mix them.

TypeScript buys you type-checked fields, editor autocomplete on every resource, typed cross-references (reference an agent by its construct, not a string), loops and shared constants, and prompts as plain strings, with no YAML `file://` sentinel. The [Quickstart](/documentation/config-as-code/overview#quickstart) walks the full author-preview-apply loop in both languages, and every resource page below carries a TypeScript example beside its YAML.

***

## Builders

One `define*` helper per resource kind. Each injects the resource's `kind` (and any fixed discriminator) and returns the exact object `config.apply` expects:

| Helper | Resource |
| :- | :- |
| `defineAgent` | Voice agents and their endpoints |
| `defineTool` | Agent tools (takes a typed `agent` reference) |
| `definePersona` | The simulated caller for a flow |
| `defineImprovFlow` / `defineScriptedFlow` | Simulation flows |
| `defineMetric` / `defineApiCheck` / `defineThreshold` | Custom metrics: LLM judges, API checks and thresholds |
| `defineCollector` | Which metrics run on which conversations |
| `defineSimulationPlan` / `defineQaSimulationPlan` | Saved, repeatable runs |
| `defineSimulationGroup` | Several plans run together |
| `defineAlert` | Threshold / event / simulation alerts |
| `defineDashboard` | Dashboards and widgets |
| `defineHTTPRequestDefinition` | Outbound-dial HTTP requests |
| `defineProject` | Projects (organization key, see [Organizations](/documentation/config-as-code/organizations)) |

`defineConfig({ resources, prune })` assembles the bundle. `resources` accepts both builder outputs and raw resource objects, so a kind without a dedicated helper still fits:

```ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
defineConfig({
  resources: [frontDesk, { kind: 'persona', name: 'frustrated-caller', accent: 'US' /* ... */ }],
});
```

***

## Apply semantics

Identical to the YAML path, because the output is the same bundle: identity by `name`, cross-references resolved by name, `prune` deletes config-managed resources you drop (set `prune: false` for additive-only), and applies are idempotent. See [Apply semantics](/documentation/config-as-code/overview#apply-semantics) for the full rules, and [Config-managed resources in the UI](/documentation/config-as-code/overview#config-managed-resources-in-the-ui) for how managed resources behave in the dashboard.

> [!TIP]
>
> YAML and TypeScript are interchangeable, not exclusive. A resource authored in TypeScript and one authored in YAML are the same resource to Roark (both keyed by `<kind>/<name>`), so you can migrate one kind at a time, or keep prompts as `.md` files and the rest in code.

> [card] [Node SDK](/documentation/sdks/node-sdk)
>
> Full setup for `@roarkanalytics/sdk`: install, authentication, and the rest of the API surface the same client exposes.


# Data Retention
Source: https://docs.roark.ai/documentation/enterprise/data-retention

Control how long Roark keeps your call data, and whether expired data is hidden or permanently deleted

Data retention lets you control how long Roark keeps your calls and their metrics, and what happens when they expire. It's configured **per project**.

## Configure retention

Go to **Settings → Project → Data Controls** and set two things:

**Retention period** `60 | 90 | 180 | 365 days | Indefinite`

How long to keep calls after they're created. Choose 60, 90, 180, or 365 days, or **Indefinite** to keep data forever. Defaults to **Indefinite**.

**When data expires** `Hide | Permanent`

What happens to a call once it passes the retention period:

* **Hide from dashboard (recoverable)** - the call is soft-deleted: hidden from the dashboard but still stored on the backend, so it can be recovered. This is the default.
* **Permanently delete (irreversible)** - the call is physically purged everywhere: recordings, transcripts, metrics, and traces.

## How expiry works

A call "expires" once its age exceeds the retention period. What happens next depends on the mode:

* **Hide** - expired calls are soft-deleted (removed from the dashboard but recoverable).
* **Permanent** - expired calls are hidden first, then **physically purged 30 days later** across all systems (Postgres, the metric store, recordings in object storage, analytics, and traces). This grace period gives you a window to catch a misconfiguration before data is gone for good.

> [!WARNING]
>
> Permanent deletion cannot be undone. Once a call is physically purged, its recordings, transcripts, metrics, and traces are gone across every system.

> [!NOTE]
>
> Retention applies going forward from when you set it. Setting a shorter period will begin expiring older calls that already fall outside the new window.


# Usage & Cost Reporting
Source: https://docs.roark.ai/documentation/enterprise/usage-reporting

Pull billable usage and cost per project, per call, and per simulation run, over the API or as a CSV

Every bit of billable work in Roark (simulations, metric evaluations, transcription) is attributed to the project that incurred it. You can read that attribution three ways:

* **Billing screen** (Settings → Billing): a per-project and per-type spend breakdown for the current billing period.
* **CSV export**: the "Export CSV" button on that breakdown downloads exactly what you see, grouped by project or by type.
* **API** (`GET /v1/usage`): the same per-project figures over a date range you choose, for feeding a warehouse, a finance report, or a per-team chargeback.

For a single call or simulation run, the call page's **Cost** tab and the run report's **Cost** section show what it cost, and the API exposes the same figures (see [Per-call and per-run cost](#per-call-and-per-run-cost)).

This page covers who can see spend, and the API.

## Who can see spend

Billing is organization-wide financial data, so no project role includes it, not even project Owner or Admin. You can see the Billing screen, the CSV export, and the per-call and per-run cost views if you are:

* an organization **Owner** or **Admin**, or
* a member an Owner has granted **billing access**: in **Settings → Project → Members**, open the member's **Data access** and turn on **View billing**. The grant applies to the whole organization (the Billing screen shows every project's spend), and it is read-only. Organization settings stay hidden.

## Authentication

`GET /v1/usage` accepts two credentials. Pick the narrower one that does the job.

| Credential | Permission | Reads |
| :- | :- | :- |
| **Project API key** | `billing:read` | That key's own project only. |
| **Organization API key** | `usage:read` | Every project in the organization. |

### Project API key (least privilege)

For a team that reports its own spend, use a project key that carries **`billing:read`**. It reads only its own project: leave out `projectId` (or pass the key's own project), and any other `projectId` returns `403`. A key that holds only `billing:read` can read spend and nothing else.

Create one in **Settings → Project → API keys**: choose **Custom**, set every row to **No Access** if spend is all you need, and turn on **Spend**. The **Spend** switch only appears for people who can see billing (see above), and the API refuses `billing:read` on a key minted by anyone else, so a project admin cannot give themselves spend through a key.

### Organization API key

For a finance or warehouse job that needs every project, use an **Organization API key** (the same credential family used for [org provisioning](/documentation/config-as-code/organizations)) that carries **`usage:read`**. An org Owner or Admin creates one under **Settings → Organization → API keys**; new keys get the full organization permission set, so they include `usage:read`.

> [!NOTE]
>
> Cost visibility is a deliberately separate permission from project access. If you are using an organization key created before usage reporting launched, re-mint it so it carries `usage:read`; an older key returns `403` on this endpoint.

## Endpoint

```
GET https://api.roark.ai/v1/usage
```

### Query parameters

| Parameter | Required | Description |
| :- | :- | :- |
| `from` | yes | Start of the window (inclusive), ISO 8601. Example: `2026-09-01T00:00:00Z`. |
| `to` | yes | End of the window (exclusive), ISO 8601. Must be after `from`, and the window may not exceed **366 days**. |
| `projectId` | no | Limit the report to a single project. Omit for the whole organization. |

### Example request

```typescript Node theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import Roark from '@roarkanalytics/sdk'

const client = new Roark({ bearerToken: process.env.ROARK_ORG_API_KEY })

const usage = await client.usage.get({
  from: '2026-09-01T00:00:00Z',
  to: '2026-10-01T00:00:00Z',
  // projectId: '7b3c0a1e-1111-2222-3333-444455556666', // scope to one project
})

console.log(usage.data.totalUsd)
```

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark usage get \
  --from 2026-09-01T00:00:00Z \
  --to 2026-10-01T00:00:00Z
# add --project-id <uuid> to scope to one project
```

```bash curl theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -s "https://api.roark.ai/v1/usage?from=2026-09-01T00:00:00Z&to=2026-10-01T00:00:00Z" \
  -H "Authorization: Bearer $ROARK_ORG_API_KEY"
```

```bash one project (org key) theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -s "https://api.roark.ai/v1/usage?from=2026-09-01T00:00:00Z&to=2026-10-01T00:00:00Z&projectId=7b3c0a1e-1111-2222-3333-444455556666" \
  -H "Authorization: Bearer $ROARK_ORG_API_KEY"
```

```bash project key theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -s "https://api.roark.ai/v1/usage?from=2026-09-01T00:00:00Z&to=2026-10-01T00:00:00Z" \
  -H "Authorization: Bearer $ROARK_PROJECT_API_KEY"
```

### Example response

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "data": {
    "from": "2026-09-01T00:00:00.000Z",
    "to": "2026-10-01T00:00:00.000Z",
    "currency": "usd",
    "totalUsdMicros": 1234567,
    "totalUsd": 1.234567,
    "projects": [
      {
        "projectId": "7b3c0a1e-1111-2222-3333-444455556666",
        "projectName": "Payments (prod)",
        "amountUsdMicros": 900000,
        "amountUsd": 0.9,
        "breakdown": [
          { "source": "SIM_PLATFORM_FEE", "surface": "SIMULATION", "amountUsdMicros": 600000, "amountUsd": 0.6 },
          { "source": "SIM_LLM", "surface": "SIMULATION", "amountUsdMicros": 300000, "amountUsd": 0.3 }
        ]
      },
      {
        "projectId": "9f1a2b3c-aaaa-bbbb-cccc-ddddeeeeffff",
        "projectName": "Support line",
        "amountUsdMicros": 334567,
        "amountUsd": 0.334567,
        "breakdown": [
          { "source": "METRIC", "surface": "POST_CALL", "amountUsdMicros": 334567, "amountUsd": 0.334567 }
        ]
      }
    ]
  }
}
```

### Response fields

* **`totalUsdMicros` / `totalUsd`**: total billable spend across the organization (or the single project, when `projectId` is set) over the window. The total is the sum of `projects`.
* **`projects[]`**: per-project billable spend, highest first. Only projects with spend in the window appear.
* **`projectName`**: the project's current name, or `null` if the project no longer exists.
* **`projects[].breakdown[]`**: that project's spend split by cost type, highest first, summing to the project's `amountUsdMicros`. Each entry has a `source` and a `surface`. `source` is the granular cost source (`SIM_PLATFORM_FEE`, `SIM_LLM`, `SIM_TTS`, `SIM_STT`, `SIM_TELEPHONY`, `METRIC`, `TRANSCRIPTION`); the `SIM_*` sources are what the Billing screen groups under "Simulation". `surface` is `SIMULATION` or `POST_CALL` (a production call). Roll these up however your report needs.
* **Micros vs dollars**: `amountUsdMicros` is the exact, canonical integer (`1,000,000` = `$1`, the unit the billing ledger stores). Use it to sum and reconcile against an invoice. `amountUsd` is the same figure as a convenience float for display.

> [!NOTE]
>
> Amounts are **billable** spend only. Internal costs Roark absorbs on your behalf are never included, so these figures match what you are charged.

## Errors

| Status | Meaning |
| :- | :- |
| `400` | Missing or invalid `from`/`to`, or a window wider than 366 days. |
| `401` | Missing or invalid API key. |
| `403` | The organization key does not carry `usage:read`, the project key does not carry `billing:read`, or a project key asked for another project's `projectId`. |

## Paging longer histories

The 366-day ceiling keeps a single request bounded. To report across a longer period, page the window month by month (or quarter by quarter) and sum the `totalUsdMicros` from each response.

## Per-call and per-run cost

Spend is attributed to the call and to the simulation run that incurred it, not only to the project and window. Both endpoints need a project API key with **`billing:read`**, and read only that key's project.

### One call

```
GET https://api.roark.ai/v1/call/{callId}/cost
```

The billable spend for one call, split by cost type, with every cost line behind it. A simulated call includes what its simulation spent before the call connected.

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "data": {
    "callId": "3f2c9d1e-5a6b-4c7d-8e9f-0a1b2c3d4e5f",
    "currency": "usd",
    "totalUsdMicros": 412000,
    "totalUsd": 0.412,
    "breakdown": [
      { "source": "SIM_PLATFORM_FEE", "surface": "SIMULATION", "amountUsdMicros": 250000, "amountUsd": 0.25, "quantity": 5 },
      { "source": "METRIC", "surface": "SIMULATION", "amountUsdMicros": 162000, "amountUsd": 0.162, "quantity": 10.8 }
    ],
    "items": [
      {
        "eventId": "evt_8d1c...",
        "source": "METRIC",
        "surface": "SIMULATION",
        "amountUsdMicros": 81000,
        "amountUsd": 0.081,
        "quantity": 5.4,
        "quantityUnit": "MINUTES",
        "provider": null,
        "model": null,
        "simulationJobId": "a1b2c3d4-0000-4000-8000-000000000001",
        "metricDefinitionId": "b2c3d4e5-0000-4000-8000-000000000002",
        "metricCollectionJobId": "c3d4e5f6-0000-4000-8000-000000000003",
        "occurredAt": "2026-10-20T03:14:15.000Z"
      }
    ]
  }
}
```

A call graded twice carries one `METRIC` line per grading pass; `metricCollectionJobId` tells them apart.

### One simulation run

```
GET https://api.roark.ai/v1/simulation/plan/job/{jobId}/cost
```

The billable spend for one simulation run (run plan job): totals by cost type, and the spend of each simulated call. Safe to call while the run is in progress; totals reflect what has been recorded so far. For a call's line items, call the per-call endpoint with its `conversationId`.

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "data": {
    "jobId": "7f3e4d2c-8a91-4b5c-9e6f-1a2b3c4d5e6f",
    "currency": "usd",
    "totalUsdMicros": 2060000,
    "totalUsd": 2.06,
    "breakdown": [
      { "source": "SIM_PLATFORM_FEE", "surface": "SIMULATION", "amountUsdMicros": 1250000, "amountUsd": 1.25, "quantity": 25 },
      { "source": "METRIC", "surface": "SIMULATION", "amountUsdMicros": 810000, "amountUsd": 0.81, "quantity": 54 }
    ],
    "calls": [
      {
        "conversationId": "3f2c9d1e-5a6b-4c7d-8e9f-0a1b2c3d4e5f",
        "simulationJobId": "a1b2c3d4-0000-4000-8000-000000000001",
        "amountUsdMicros": 412000,
        "amountUsd": 0.412,
        "breakdown": [
          { "source": "SIM_PLATFORM_FEE", "surface": "SIMULATION", "amountUsdMicros": 250000, "amountUsd": 0.25, "quantity": 5 },
          { "source": "METRIC", "surface": "SIMULATION", "amountUsdMicros": 162000, "amountUsd": 0.162, "quantity": 10.8 }
        ]
      }
    ]
  }
}
```

`conversationId` is the call id for voice runs (the chat id for chat runs). It is `null` for costs incurred before the conversation existed; join those on `simulationJobId`.

### Errors

| Status | Meaning |
| :- | :- |
| `401` | Missing or invalid API key. |
| `403` | The key does not carry `billing:read`. |
| `404` | No such call or run in the key's project. |


# API Keys
Source: https://docs.roark.ai/documentation/getting-started/api-keys

Generate and manage authentication keys for Roark APIs

## Overview

API keys authenticate your requests to Roark's API, enabling secure access to monitoring, testing, and analytics features.

> [!NOTE]
>
> API keys are essential for all integrations - whether you're using our SDKs, webhooks, or direct API calls.

## Two kinds of credential

Roark issues two shapes of bearer credential. Both go in the same `Authorization: Bearer` header, and most callers only ever need the first.

| | Project key | User credential |
| - | - | - |
| Created by | **Settings → API keys** in the dashboard | approving a browser consent (`roark auth login`, MCP connector) |
| Belongs to | one project, permanently | you, within one organization |
| Names its project | implicitly: the key *is* the project | per request, with the `X-Roark-Project-Id` header |
| Access | exactly the scopes and permissions chosen at creation | the permissions you approved, narrowed to your role on each project |
| Revoked at | **Settings → API keys** | **Settings → Account → Connected apps** |
| Survives you leaving | yes, it belongs to the project | no, deactivating your account revokes it |
| Right for CI | **yes** | no, it is tied to a person |

> [!NOTE]
>
> Use a **project key** for anything automated: ingestion, webhooks, CI pipelines, a service that
> outlives whoever set it up. A project key keeps working when people join and leave, which is
> exactly what a pipeline needs and exactly what a user credential deliberately does not do.

A user credential cannot exceed the person holding it. Its access is recomputed on every request as
the permissions approved at consent intersected with your live membership of the project being acted
on, so losing access in the dashboard removes it from the credential at the same moment.

## Generating an API Key

**1. Navigate to API Keys**

Go to the API Keys section in your dashboard

**2. Create New Key**

Click the "Create API Key" button

**3. Set Permissions**

Choose your access level:

* **Read-Only**: Retrieve data only
* **Full Access**: Read and write operations

**4. Save Securely**

Copy and store your key immediately - it won't be shown again

![Create API Key Modal](https://mintcdn.com/roark/MpvpibiPGNE0-cte/images/api-key/api-key-new-modal.png?fit=max\&auto=format\&n=MpvpibiPGNE0-cte\&q=85\&s=73f5edd675bc052db2de287e97e53bc9)

## Using Your API Key

All API requests require authentication via the Authorization header:

```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X GET "https://api.roark.ai/health" \
     -H "Authorization: Bearer YOUR_API_KEY"
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import requests

headers = {
    "Authorization": "Bearer YOUR_API_KEY"
}

response = requests.get("https://api.roark.ai/health", headers=headers)
```

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const response = await fetch("https://api.roark.ai/health", {
  headers: {
    "Authorization": `Bearer YOUR_API_KEY`
  }
});
```

```javascript Node.js theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const axios = require('axios');

const config = {
  headers: {
    'Authorization': 'Bearer YOUR_API_KEY'
  }
};

const response = await axios.get('https://api.roark.ai/health', config);
```

## Choosing the project a request acts on

A project key is bound to one project, so it never has to say which. A user credential can reach
every project you are a member of, so each request names one with the `X-Roark-Project-Id` header:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X GET "https://api.roark.ai/v1/agent" \
     -H "Authorization: Bearer YOUR_CREDENTIAL" \
     -H "X-Roark-Project-Id: 0b2b1f9e-5f1a-4c7e-9c3d-1f2e3a4b5c6d"
```

Two endpoints answer "what am I holding, and what can it reach", so a client never has to infer
either from a failed request:

```bash Which credential is this theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X GET "https://api.roark.ai/v1/me" \
     -H "Authorization: Bearer YOUR_CREDENTIAL"
```

```json Response theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "data": {
    "tokenScope": "USER",
    "user": { "id": "…", "email": "you@example.com", "name": "You" },
    "organization": { "id": "…", "name": "Acme" },
    "defaultProject": { "id": "…", "name": "Support line", "slug": "support-line" },
    "grantedPermissions": ["call:read", "call:create", "metric:read"],
    "scopes": ["WRITE"]
  }
}
```

```bash Which projects can it reach theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X GET "https://api.roark.ai/v1/projects" \
     -H "Authorization: Bearer YOUR_CREDENTIAL"
```

```json Response theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "data": {
    "projects": [
      {
        "id": "0b2b1f9e-5f1a-4c7e-9c3d-1f2e3a4b5c6d",
        "name": "Support line",
        "slug": "support-line",
        "organization": { "id": "…", "name": "Acme" },
        "grantedPermissions": ["call:read", "metric:read"]
      }
    ]
  }
}
```

`GET /v1/projects` is computed through the same permission engine that gates every other request, so
it will not advertise a project that your next call rejects. A project key gets a one-element list:
the project it is bound to.

### When a project cannot be resolved

| Status | Meaning | What to do |
| - | - | - |
| **400** | a user credential named no project, and has no default to fall back on | send `X-Roark-Project-Id`; `GET /v1/projects` lists valid ids |
| **403** | the project is outside the credential's organization, or you are not an active member of it | pick one from `GET /v1/projects` |
| **403** | a **project key** sent a header naming a different project than its own | drop the header, or use a credential for that project |

That last one is deliberately an error rather than a silent no-op. A caller that believes it is
writing to project B must never end up writing to project A.

> [!NOTE]
>
> A user credential created through a browser consent remembers the project chosen on the consent
> screen and falls back to it when a request names none, so a client that never sends the header
> still works. Do not rely on it if your integration touches more than one project: send the header
> explicitly and the behaviour stays obvious.

## Building your own browser login

Only relevant if you wrote your own client against `POST /v1/cli/auth/token`. The Roark CLI and MCP
server already handle this.

That endpoint's response gained fields, and **`projectId` is now nullable**:

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "token": "roark…",
  "name": "CLI on my-laptop",
  "scopeKind": "USER",
  "projectId": null,
  "organizationId": "…",
  "defaultProjectId": "…"
}
```

Read `scopeKind` to know what you were given. A `projectId` of `null` means the credential is
user-scoped and has no single project: treat it as "call `GET /v1/projects` and pick one", never as
an error. `defaultProjectId` is the project chosen on the consent screen, which is what the API falls
back to when a request names none, so storing it is the simplest way to keep working without asking.

> [!WARNING]
>
> A client that validates `projectId` as a required string will reject a perfectly good user
> credential. Accept both shapes before anyone in your organization is issued one.

## Security Best Practices

> [!WARNING]
>
> Never expose your API key in client-side code, public repositories, or logs. If compromised, regenerate immediately in your dashboard.

> [card] Environment Variables
>
> Store keys in `.env` files or secure vaults, never hardcode them

> [card] Regular Rotation
>
> Regenerate keys periodically to maintain security

> [card] Least Privilege
>
> Use read-only keys when write access isn't needed

> [card] Monitor Usage
>
> Check last-used timestamps to detect unauthorized access

## Common Integration Patterns

> [tab] SDK Integration
>
> Initialize SDKs with your key:
>
> ```javascript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> import Roark from '@roarkanalytics/sdk'
>
> const client = new Roark({
>   bearerToken: process.env.ROARK_API_BEARER_TOKEN
> });
> ```

> [tab] CI/CD Pipeline
>
> Store as secret in your pipeline:
>
> ```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> env:
>   ROARK_API_KEY: ${{ secrets.ROARK_API_KEY }}
> ```

## Need Help?

> [card] [API Reference](/api-reference/introduction)
>
> Explore all available endpoints and parameters

> [card] [Integration Guides](/documentation/integrations/overview)
>
> Step-by-step platform integration tutorials


# Welcome to Roark
Source: https://docs.roark.ai/documentation/getting-started/introduction

The quality platform for voice and chat AI

Roark is where voice and chat agents prove they're ready: **simulate** every failure mode before launch, **analyze** every production call once you're live, and score them all with metrics powered by **Roark Prism**, our purpose-built evaluation model.

> [card] [Simulation testing](/documentation/simulation-testing/overview)
>
> Break it in staging, not in production. Test agents with synthetic callers across flows, personas, accents, and edge cases before you ship.

> [card] [Post-call analysis](/documentation/observability/overview)
>
> Every production call scored, filed, and traced. Transcribed and measured against your metrics in real time.

> [card] [Human review](https://roark.ai/product/human-review)
>
> Label calls, set ground truth, and align every metric with your team's judgment.

> [card] [Autoimprove](/documentation/autoimprove/overview)
>
> From caught issue to drafted fix to proven deploy, with you in the loop.

***

## Your first simulation

The fastest way in is a few lines of code: define an agent and a simulation plan, apply them, and run it from the CLI. It's versioned in git, reusable in CI, and there are no UUIDs to wrangle.

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineAgent, defineSimulationPlan, defineConfig } from '@roarkanalytics/sdk/lib/config';

const frontdesk = defineAgent({
  name: 'frontdesk',
  endpoints: [{ value: '+15551234567', direction: 'OUTGOING' }],
});

export default defineConfig({
  resources: [
    frontdesk,
    defineSimulationPlan({
      name: 'smoke-test',
      direction: 'OUTBOUND',
      agentEndpoints: [{ agent: 'frontdesk', value: '+15551234567' }],
      flows: [{ system: 'sf-screener-apple' }], // a built-in flow; swap for your own
      metrics: ['call_screening_handling_score'],
      maxDurationSeconds: 300,
    }),
  ],
});
```

```yaml roark/ (one file per resource) theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# agents/frontdesk.yaml
kind: agent
name: frontdesk
endpoints:
  - value: '+15551234567'
    direction: OUTGOING
---
# simulation-plans/smoke-test.yaml
kind: simulationPlan
name: smoke-test
direction: OUTBOUND
agentEndpoints:
  - agent: frontdesk
    value: '+15551234567'
flows:
  - system: sf-screener-apple
metrics:
  - call_screening_handling_score
maxDurationSeconds: 300
```

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
export ROARK_API_BEARER_TOKEN="<key>"        # needs config:apply + simulation permissions
roark config apply roark.config.ts           # create the agent + plan (or `roark config apply ./roark` for YAML)
roark simulation run --data '{"planId":"<plan-id>"}' # run it (find the id with `roark simulation plan list`)
```

> [card] [Full walkthrough: Run a simulation](/documentation/config-as-code/run-a-simulation)
>
> Define, apply, run, read the pass/fail, and gate CI on it, step by step.

### Other ways to run one

Prefer to click through it, or drive the API imperatively? Each tab below also takes you from nothing to a running simulation, reusing a built-in persona and environment.

> [tab] Node SDK
>
> Copy, paste, run with [`@roarkanalytics/sdk`](/documentation/sdks/node-sdk):
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> import Roark from '@roarkanalytics/sdk'
>
> const client = new Roark({ bearerToken: process.env.ROARK_API_BEARER_TOKEN })
>
> // An agent and how to reach it
> const agent = await client.agent.create({ name: 'Front Desk' })
> const endpoint = await client.agentEndpoint.create({
>   agentId: agent.data.id,
>   value: '+15551234567',
>   direction: 'INCOMING_AND_OUTGOING',
> })
>
> // Reuse a built-in persona and the "Quiet line" environment
> const personas = await client.simulationPersona.list({ limit: 50 })
> const envs = await client.simulationEnvironment.list({ limit: 50 })
> const quietLine = envs.data.find((e) => e.name === 'Quiet line')!
>
> // The conversation to test
> const flow = await client.customerFlow.create({
>   type: 'IMPROV',
>   title: 'Rebooking',
>   agentIds: [agent.data.id],
>   happyPath: {
>     title: 'Frustrated rebooking',
>     personaOverrideId: personas.data[0].id,
>     environmentId: quietLine.id,
>     prompt: 'You call to rebook a cleaning that was cancelled on you.',
>   },
>   agentExpectations: [{ prompt: 'Agent offers a concrete alternative appointment time' }],
> })
>
> // Run it
> const run = await client.simulation.run({
>   plan: {
>     direction: 'INBOUND',
>     maxSimulationDurationSeconds: 300,
>     agentEndpoints: [{ id: endpoint.data.id }],
>     flows: [{ id: flow.data.id, happyPath: true }],
>     metrics: [{ slug: 'task_completion' }],
>   },
> })
>
> console.log(run.data.simulationRunPlanJobId)
> ```
>
> Watch it in the dashboard, or poll `client.simulationRunPlanJob.getByID(run.data.simulationRunPlanJobId)`.

> [tab] Python SDK
>
> Same flow with [`roark_analytics`](/documentation/sdks/python-sdk):
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> import os
> from roark_analytics import Roark
>
> client = Roark(bearer_token=os.environ["ROARK_API_BEARER_TOKEN"])
>
> # An agent and how to reach it
> agent = client.agent.create(name="Front Desk")
> endpoint = client.agent_endpoint.create(
>     agent_id=agent.data.id,
>     value="+15551234567",
>     direction="INCOMING_AND_OUTGOING",
> )
>
> # Reuse a built-in persona and the "Quiet line" environment
> personas = client.simulation_persona.list(limit=50)
> envs = client.simulation_environment.list(limit=50)
> quiet_line = next(e for e in envs.data if e.name == "Quiet line")
>
> # The conversation to test
> flow = client.customer_flow.create(
>     type="IMPROV",
>     title="Rebooking",
>     agent_ids=[agent.data.id],
>     happy_path={
>         "title": "Frustrated rebooking",
>         "personaOverrideId": personas.data[0].id,
>         "environmentId": quiet_line.id,
>         "prompt": "You call to rebook a cleaning that was cancelled on you.",
>     },
>     agent_expectations=[{"prompt": "Agent offers a concrete alternative appointment time"}],
> )
>
> # Run it
> run = client.simulation.run(
>     plan={
>         "direction": "INBOUND",
>         "maxSimulationDurationSeconds": 300,
>         "agentEndpoints": [{"id": endpoint.data.id}],
>         "flows": [{"id": flow.data.id, "happyPath": True}],
>         "metrics": [{"slug": "task_completion"}],
>     },
> )
>
> print(run.data.simulation_run_plan_job_id)
> ```

> [tab] CLI
>
> Drive the same steps from your terminal (install the [CLI](/documentation/sdks/cli) and set `ROARK_API_BEARER_TOKEN`). Commands print JSON, so pipe through `jq`:
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> AGENT_ID=$(roark agent create --name "Front Desk" | jq -r '.data.id')
>
> ENDPOINT_ID=$(roark agent endpoint create --agent-id "$AGENT_ID" \
>   --value "+15551234567" --direction INCOMING_AND_OUTGOING | jq -r '.data.id')
>
> PERSONA_ID=$(roark simulation persona list --limit 50 | jq -r '.data[0].id')
> ENV_ID=$(roark simulation environment list --limit 50 \
>   | jq -r '.data[] | select(.name=="Quiet line") | .id')
>
> FLOW_ID=$(roark customer-flow create --data "$(jq -nc \
>   --arg a "$AGENT_ID" --arg p "$PERSONA_ID" --arg e "$ENV_ID" '{
>     type:"IMPROV", title:"Rebooking", agentIds:[$a],
>     happyPath:{ title:"Frustrated rebooking", personaOverrideId:$p, environmentId:$e,
>       prompt:"You call to rebook a cleaning that was cancelled on you." }
>   }')" | jq -r '.data.id')
>
> roark simulation run --data "$(jq -nc --arg ep "$ENDPOINT_ID" --arg f "$FLOW_ID" '{
>   plan:{ direction:"INBOUND", maxSimulationDurationSeconds:300,
>     agentEndpoints:[{id:$ep}], flows:[{id:$f, happyPath:true}],
>     metrics:[{slug:"task_completion"}] }
> }')"
> ```

> [tab] Config as Code
>
> Keep your test suite in git as YAML and apply it with the [CLI](/documentation/sdks/cli):
>
> ```yaml roark/flows/rebooking.yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> kind: flow
> type: improv
> name: rebooking
> agents: [frontdesk]
> happyPath:
>   persona: frustrated-caller
>   environment: Quiet line
>   prompt: You call to rebook a cleaning that was cancelled on you.
> expectations:
>   - Agent offers a concrete alternative appointment time
> ```
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> roark config apply ./roark
> ```
>
> Config defines your resources; launch a run from the CLI, an SDK, or the dashboard. Full example (agents + personas + flows) and how to run it in CI: **[Config as Code](/documentation/config-as-code/overview)** and **[CI/CD](/documentation/simulation-testing/ci-cd)**.

> [tab] Dashboard
>
> No code:
>
> 1. **Connect your agent** under [Agents](/documentation/integrations/overview) (Vapi, Retell, ElevenLabs, LiveKit, and more).
> 2. **Create a [customer flow](/documentation/simulation-testing/customer-flows)** for the conversation to test (pick a built-in [persona](/documentation/simulation-testing/personas)).
> 3. **New Run**: pick the agent, attach the flow, add pass/fail [checks](/documentation/metrics/thresholds), launch.
> 4. **Read the verdict** and per-conversation scores.

***

## Explore the docs

> [card] [Simulations](/documentation/simulation-testing/overview)
>
> Customer flows, personas, templates, plans, and schedules

> [card] [Config as Code](/documentation/config-as-code/overview)
>
> Define agents, personas, flows, metrics, collectors, and alerts as YAML in git

> [card] [Observability](/documentation/observability/overview)
>
> Call history, traces, reports, and dashboards

> [card] [Metrics](/documentation/metrics/overview)
>
> The metric library, Studio, collectors, and thresholds

> [card] [Integrations](/documentation/integrations/overview)
>
> Vapi, Retell, ElevenLabs, Leaping, LiveKit, and custom

> [card] [CLI](/documentation/sdks/cli)
>
> Drive Roark from your terminal and CI pipelines

> [card] [SDKs](/documentation/sdks/node-sdk)
>
> Node.js, Python, and MCP Server

> [card] [API Reference](/api-reference/introduction)
>
> REST API documentation


# Bland AI
Source: https://docs.roark.ai/documentation/integrations/bland

Sync Bland AI agents and analyze voice AI conversations

Live monitoringVoice simulationsChat simulations

## Overview

The Bland AI integration syncs your voice agents and their calls into Roark. Agents and calls are pulled on a recurring schedule so every conversation is analyzed and evaluated automatically.

***

## Prerequisites

Before setting up the integration, ensure you have:

* A Bland AI account with active agents
* A Bland AI API key (found in your [Bland AI dashboard](https://app.bland.ai))
* Call recording enabled in your Bland AI send call payload (see [Call Payload Configuration](#call-payload-configuration) below)

***

## Call Payload Configuration

To sync calls with Roark, you must include two fields in your Bland AI send call payload:

| Field | Type | Description |
| :- | :- | :- |
| **`persona_id`** | `string` | The agent/persona ID. Roark uses this to match calls to agents and determine which calls to sync based on your selected agents. |
| **`record`** | `boolean` | Must be set to `true` so the call is recorded and Roark can access the audio for analysis. |

> [!WARNING]
>
> Calls without `record: true` will not have audio available and cannot be synced into Roark. Calls without a `persona_id` will not be matched to any agent.

***

## Setup Instructions

### Step 1: Create Integration

1. Navigate to **Agents** in your Roark dashboard
2. Click **Connect Agent** in the top right, choose **Existing Platform**, and select **Bland AI**
3. Enter your configuration:

| Field | Description |
| :- | :- |
| **Integration Name** | Friendly name for this integration |
| **API Key** | Your Bland AI API key |

The system will validate your API key and fetch available agents.

### Step 2: Select Agents

Choose which Bland AI agents to monitor:

* **All Agents**: Sync calls from every agent in your account
* **Selected Agents**: Choose specific agents to monitor

Roark syncs calls based on the `persona_id` field in your call payload. Only calls matching your selected agents will be synced.

### Step 3: Activate Integration

Review your settings and click **Create Integration** to begin syncing.

***

## What Gets Synced

Bland AI integrations sync the following data:

* **Calls**: Conversation audio, transcripts, and metadata
* **Agents**: Agent/persona configurations
* **Transcripts**: Full conversation transcripts with speaker labels
* **Tool Calls**: Function/tool invocations during conversations
* **Call Metadata**: Duration, status, participant information

***

## How Sync Works

Bland AI uses **pull-based synchronization**: Roark periodically fetches new calls from the Bland AI API rather than receiving webhooks.

| Phase | What Happens |
| :- | :- |
| **Agent Sync** | Fetches all agents, updates names and configurations |
| **Call Sync** | Fetches conversations since last sync, downloads recordings, parses transcripts and tool invocations, creates call records |

Roark matches incoming calls to agents using the `persona_id` included in your send call payload. Only calls with `record: true` and a valid `persona_id` are synced.

***

## Monitoring Integration Health

Track your integration status in the Roark dashboard:

* **Syncing**: Actively importing calls
* **Active**: Running on schedule, pulling new calls periodically
* **Paused**: Sync temporarily halted
* **Error**: Configuration or connection issues

View sync statistics including total calls synced, last sync timestamp, and job history.

***

## Agent Management

Synced agents appear in:

* **Simulation agent selection**: Use for testing
* **Agent performance reports**: Track metrics per agent
* **Comparison dashboards**: Analyze across agents

Agent data updates automatically when modified in Bland AI.

***

## Next Steps

> [card] [Run Simulations](/documentation/simulation-testing/running-simulations)
>
> Test synced agents with simulations

> [card] [View Call History](/documentation/observability/live-monitoring)
>
> Browse and analyze synced calls

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Automate metric collection on calls

> [card] [Integration Overview](/documentation/integrations/overview)
>
> Explore other integrations


# Custom Integrations
Source: https://docs.roark.ai/documentation/integrations/custom-integrations

Send calls from any platform using our API and SDKs

## Overview

Not using one of our pre-built integrations? No problem. Send call data from any voice AI platform or custom application. Just provide an audio file and Roark handles the rest: transcription, speech analysis, sentiment detection, and metric collection.

***

## Send a Call

> [tab] Node.js
>
> Install the [Node.js SDK](/documentation/sdks/node-sdk):
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> npm install @roarkanalytics/sdk
> ```
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> import Roark from '@roarkanalytics/sdk'
>
> const client = new Roark({
>   bearerToken: process.env.ROARK_API_BEARER_TOKEN,
> })
>
> const call = await client.call.create({
>   recordingUrl: 'https://your-storage.com/call-recording.mp3',
>   startedAt: '2024-01-15T10:00:00Z',
>   interfaceType: 'PHONE',
>   callDirection: 'INBOUND',
>   agent: {
>     name: 'Support Agent',
>     customId: 'custom-agent-1',
>   },
>   customer: {
>     phoneNumberE164: '+1234567890',
>   },
>   properties: {
>     department: 'sales',
>     region: 'us-east',
>   },
> })
> ```

> [tab] Python
>
> Install the [Python SDK](/documentation/sdks/python-sdk):
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> pip install roark-analytics
> ```
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> import os
> from roark_analytics import Roark
>
> client = Roark(
>     bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"),
> )
>
> call = client.call.create(
>     recording_url="https://your-storage.com/call-recording.mp3",
>     started_at="2024-01-15T10:00:00Z",
>     interface_type="PHONE",
>     call_direction="INBOUND",
>     agent={
>         "name": "Support Agent",
>         "customId": "custom-agent-1",
>     },
>     customer={
>         "phoneNumberE164": "+1234567890",
>     },
>     properties={
>         "department": "sales",
>         "region": "us-east",
>     },
> )
> ```

> [tab] cURL
>
> Use the [REST API](/api-reference/introduction) directly:
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> curl --request POST \
>   --url https://api.roark.ai/v1/call \
>   --header 'Authorization: Bearer YOUR_API_KEY' \
>   --header 'Content-Type: application/json' \
>   --data '{
>     "recordingUrl": "https://your-storage.com/call-recording.mp3",
>     "startedAt": "2024-01-15T10:00:00Z",
>     "interfaceType": "PHONE",
>     "callDirection": "INBOUND",
>     "agent": {
>       "name": "Support Agent",
>       "customId": "custom-agent-1"
>     },
>     "customer": {
>       "phoneNumberE164": "+1234567890"
>     },
>     "properties": {
>       "department": "sales",
>       "region": "us-east"
>     }
>   }'
> ```

Once a call is created, it appears in [Call History](/documentation/observability/live-monitoring) and is automatically transcribed and analyzed.

***

## What You Can Send

| Data | Description |
| :- | :- |
| **Audio file** | MP3, WAV, M4A, or FLAC (via public URL or base64) |
| **Metadata** | Agent info, customer details, call direction, interface type |
| **Custom properties** | Arbitrary key-value pairs for filtering and segmentation |
| **Tool calls** | Function/tool invocations made during the conversation |
| **Pre-transcribed text** | If you already have a transcript, send it along |

### Updating properties after ingest

Some data only exists after the call is in Roark, such as an outcome your system validates a few minutes later. Add it to the call with `PATCH /v1/call/{callId}/properties`:

> [tab] Node.js
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> await client.call.updateProperties(callId, {
>   properties: { partner_outcome: 'apr_returned', stale_flag: null },
> })
> ```

> [tab] cURL
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> curl -X PATCH https://api.roark.ai/v1/call/$CALL_ID/properties \
>   -H "Authorization: Bearer $ROARK_API_KEY" \
>   -H "Content-Type: application/json" \
>   -d '{ "properties": { "partner_outcome": "apr_returned", "stale_flag": null } }'
> ```

* A value adds the property or overwrites it, `null` removes it, and properties you leave out are untouched. Keys are stored in snake\_case, as on create.
* The update is all or nothing. If any property can't be written, the request returns `400` and nothing changes: a property Roark sets itself (`direction`, `duration_seconds`, `agent_id`, ...), or a value whose type differs from the type the project already stores that property as.
* Adding properties doesn't re-run metrics on its own. To re-score in the same request, pass `metrics: [{ id }]`; this starts a billed [collection job](/documentation/metrics/metric-collection-jobs) and needs the `metric:create` permission. If scoring can't start, the properties are still saved and the reason is in `metricCollectionError`.
* The request needs `call:create`. The response includes the call's full property list only when the key also has `call:read`.

> [!WARNING]
>
> A property's type is fixed by the first value it ever receives: a JSON number makes it a number property, a quoted string a text property. Keep each key's type consistent. A value of the other type is skipped when a call is created, and rejected with a `400` by the update endpoint.

### Audio Requirements

| Spec | Requirement |
| :- | :- |
| **Format** | MP3, WAV, M4A, or FLAC |
| **Sample Rate** | 8kHz or higher |
| **Channels** | Mono or stereo |
| **Duration** | Up to 2 hours |
| **File Size** | Max 500MB |
| **Access** | Publicly accessible URL or base64 encoded |

***

## What Happens Next

Once a call is ingested, Roark automatically:

1. **Transcribes and analyzes** the conversation: speech patterns, sentiment, 64+ emotions, interruptions, and more
2. **Runs metric collectors**: any active [metric collectors](/documentation/metrics/metric-collectors) collect metrics automatically (LLM evaluations, compliance checks, custom KPIs)
3. **Makes it searchable**: the call appears in [Call History](/documentation/observability/live-monitoring) with full filtering by properties, agent, duration, and more

***

## Use With Simulations

Agents created via the API can also be used in [simulations](/documentation/simulation-testing/overview). When you send calls with an `agent.customId`, Roark creates or matches an agent record that you can then target in simulation run plans, useful for testing custom-built voice agents that aren't on a pre-built platform.

***

## MCP Server

If you use an AI-powered development environment, the [Roark MCP Server](/documentation/sdks/mcp-server) gives your AI assistant direct access to the Roark API (including creating calls, querying metrics, and managing agents) without writing integration code manually.

***

## Getting Started

**1. Generate an API Key**

Create an API key in your [Roark dashboard](/documentation/getting-started/api-keys)

**2. Send Your First Call**

Upload an audio file using the SDK or REST API above

**3. Configure Collectors**

Set up [metric collectors](/documentation/metrics/metric-collectors) to automatically collect metrics on every call

**4. Explore Your Data**

Browse calls in [Call History](/documentation/observability/live-monitoring), build [reports](/documentation/observability/reports), and create [dashboards](/documentation/observability/dashboards)

***

## Resources

> [card] [Enable Tracing](/documentation/observability/traces?tab=custom-integration)
>
> Send OpenTelemetry traces from any platform

> [card] [Node.js SDK](/documentation/sdks/node-sdk)
>
> Full SDK guide and reference

> [card] [Python SDK](/documentation/sdks/python-sdk)
>
> Full SDK guide and reference

> [card] [API Reference](/api-reference/introduction)
>
> Complete REST API documentation


# ElevenLabs
Source: https://docs.roark.ai/documentation/integrations/elevenlabs

Sync ElevenLabs conversational AI agents and run voice and chat simulations

Live monitoringVoice simulationsChat simulations

## Overview

The ElevenLabs integration syncs your conversational AI agents and their calls into Roark. Agents, prompts, phone numbers, and tools are imported automatically, and calls are pulled on a recurring schedule so every conversation is analyzed.

A single ElevenLabs integration unlocks all three of Roark's capabilities for connected agents:

* **Live call monitoring**: pull conversation audio, transcripts, and metadata on a recurring schedule.
* **Voice simulations**: drive synthesized voice conversations against synced agents over phone or WebRTC.
* **Chat simulations**: drive [text-based simulations](/documentation/simulation-testing/chat-simulations) against synced agents over the ElevenLabs Conversational AI WebSocket. Roark auto-creates a chat endpoint for each agent on integration setup, so once you've connected ElevenLabs you can use any synced agent in a chat run plan immediately.

***

## Prerequisites

Before setting up the integration, ensure you have:

* An ElevenLabs account with active conversational AI agents
* An ElevenLabs API key (found in your ElevenLabs dashboard)
* Agents with **voice recording enabled** (agents with Zero Retention Mode or recording disabled cannot sync calls)

***

## Setup Instructions

### Step 1: Create Integration

1. Navigate to **Agents** in your Roark dashboard
2. Click **Connect Agent** in the top right, choose **Existing Platform**, and select **ElevenLabs**
3. Enter your configuration:

| Field | Description |
| :- | :- |
| **Integration Name** | Friendly name for this integration |
| **API Key** | Your ElevenLabs API key |

The system will validate your API key and fetch available agents.

### Step 2: Configure Historical Sync

Choose how far back to import existing calls:

* **Default**: Last 90 days of call history
* **Custom**: Select a specific date range
* **Skip**: Only sync new calls going forward

> [!NOTE]
>
> Historical sync runs as a backfill job during initial setup. Ongoing calls are synced automatically on a recurring schedule.

### Step 3: Select Agents

Choose which ElevenLabs agents to monitor:

* **All Agents**: Sync calls from every agent in your account
* **Selected Agents**: Choose specific agents to monitor

The agent selector shows each agent's name and ID. Agents with Zero Retention Mode enabled or voice recording disabled will be flagged as unavailable for call sync.

### Step 4: Activate Integration

Review your settings and click **Create Integration** to begin syncing.

***

## What Gets Synced

ElevenLabs integrations sync comprehensive data:

* **Calls**: Conversation audio, transcripts, and metadata
* **Agents**: Agent configurations and settings
* **Prompts**: System prompts configured for each agent
* **Phone Numbers**: Phone number endpoints assigned to agents
* **Tools**: Tool definitions available to agents
* **Call Metadata**: Duration, direction (inbound/outbound), interface type (phone/web), termination reason

***

## How Sync Works

ElevenLabs uses **pull-based synchronization**: Roark periodically fetches new calls from the ElevenLabs API rather than receiving webhooks.

| Phase | What Happens |
| :- | :- |
| **Agent Sync** | Fetches all agents, updates names and configurations, syncs prompts, phone numbers, and tools |
| **Health Check** | Verifies each agent's recording and privacy settings are compatible |
| **Call Sync** | Fetches conversations since last sync, downloads audio, parses tool invocations, creates call records |

Calls must be at least 10 seconds long and have a successful status to be synced.

***

## Agent Privacy Settings

ElevenLabs agents have privacy controls that affect what Roark can sync:

* **Zero Retention Mode** enabled → Call recordings unavailable, calls cannot be synced
* **Record Voice** disabled → Same as above

During health checks, Roark flags agents with these settings so you know which agents can and cannot sync call data.

***

## Monitoring Integration Health

Track your integration status in the Roark dashboard:

* **Syncing**: Actively importing calls
* **Active**: Running on schedule, pulling new calls periodically
* **Paused**: Sync temporarily halted
* **Error**: Configuration or connection issues

View sync statistics including total calls synced, last sync timestamp, and job history.

***

## Next Steps

> [card] [Run Simulations](/documentation/simulation-testing/running-simulations)
>
> Test synced agents with simulations

> [card] [View Call History](/documentation/observability/live-monitoring)
>
> Browse and analyze synced calls

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Automate metric collection on calls

> [card] [Integration Overview](/documentation/integrations/overview)
>
> Explore other integrations


# Exporting to Snowflake
Source: https://docs.roark.ai/documentation/integrations/exporting-to-snowflake

Sync your Roark call analysis into a data warehouse to join it against your own operational data

## Overview

Roark analyzes every call and produces structured metric values (frustration, sentiment, task completion, latency, custom metrics, and more). A common next step is to land that analysis in your own data warehouse so you can join it against your operational data, for example correlating call frustration with downstream business outcomes.

This guide covers the recommended pattern for syncing Roark analysis into **Snowflake** using the tools available today. The same approach works for BigQuery, Redshift, or Databricks: only the load step changes.

> [!NOTE]
>
> Want a fully managed connector or a scheduled file drop into your own bucket? See [Deeper integrations](#deeper-integrations) at the bottom of this page and [reach out](/documentation/resources/support). We are actively expanding warehouse-export options.

***

## The recommended pattern

The pattern is **event-driven pull**: Roark notifies you when a call finishes analysis, you pull the results from the API, land the raw JSON in cloud storage, and load it into Snowflake.

```
Roark webhook  ──▶  your handler  ──▶  fetch metrics/transcript  ──▶  S3 (raw JSON)  ──▶  Snowpipe  ──▶  Snowflake
```

Roark webhooks are a **notification, not the payload**. When a call is analyzed you receive the `callId`, not the metric values themselves, so your handler makes one follow-up API call to fetch the analysis. This keeps payloads small and lets you pull exactly the fields you need.

**1. Create a read-only API key**

Generate a key scoped to the project you want to export. Read endpoints require the `call:read`, `metric:read`, and `transcript:read` permissions. See [API Keys](/documentation/getting-started/api-keys).

**2. Subscribe to analysis webhooks**

Add a webhook endpoint and subscribe to `call.analysis.completed`. See [Webhooks](/documentation/integrations/webhooks).

**3. Fetch analysis on each event**

In your handler, call the metrics (and optionally transcript) endpoints for the `callId` you received.

**4. Land the raw JSON in cloud storage**

Write each call's payload to S3 (or GCS / Azure Blob). Store it as raw JSON so schema changes never break ingestion.

**5. Load into Snowflake**

Point Snowpipe at the bucket to auto-ingest, then flatten into modeled tables.

***

## What you can export

All data is served from the Customer API at `https://api.roark.ai/v1` and is scoped to the project your API key belongs to.

| Data | Endpoint | Permission |
| - | - | - |
| List of calls (paginated) | `GET /v1/call` | `call:read` |
| Single call detail | `GET /v1/call/{callId}` | `call:read` |
| Metric values for a call | `GET /v1/call/{callId}/metrics` | `metric:read` |
| Transcript for a call | `GET /v1/call/{callId}/transcript` | `transcript:read` |
| Metric values for a whole collection job | `GET /v1/metric/collection-jobs/{jobId}/metric-values` | `metric:read` |
| Metric definitions (catalog) | `GET /v1/metric/definitions` | `metric:read` |

Chats have the identical set of endpoints under `/v1/chat`.

> [!TIP]
>
> Every call carries an `externalId` (your own correlation ID, set when the call is ingested) and a `properties` map of custom key-values. These are your join keys back to operational data in your warehouse.

***

## Approach A: Event-driven (recommended for ongoing sync)

Best for keeping the warehouse continuously fresh with low latency.

```python Python (webhook handler) theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
@app.route('/webhooks/roark', methods=['POST'])
def handle_webhook():
    payload = request.json
    if payload['event'] == 'call.analysis.completed':
        call_id = payload['data']['callId']

        # Pull the analysis Roark just finished
        call = get(f'/v1/call/{call_id}')
        metrics = get(f'/v1/call/{call_id}/metrics')

        # Land raw JSON in S3, keyed by call id
        s3.put_object(
            Bucket='my-roark-export',
            Key=f'calls/{call_id}.json',
            Body=json.dumps({'call': call, 'metrics': metrics}),
        )
    return '', 200


def get(path):
    r = requests.get(
        f'https://api.roark.ai{path}',
        headers={'Authorization': f'Bearer {ROARK_API_KEY}'},
        timeout=10,
    )
    r.raise_for_status()
    return r.json()
```

```typescript TypeScript (webhook handler) theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
app.post('/webhooks/roark', async (req, res) => {
  const { event, data } = req.body
  if (event === 'call.analysis.completed') {
    const callId = data.callId

    const call = await get(`/v1/call/${callId}`)
    const metrics = await get(`/v1/call/${callId}/metrics`)

    await s3.putObject({
      Bucket: 'my-roark-export',
      Key: `calls/${callId}.json`,
      Body: JSON.stringify({ call, metrics }),
    })
  }
  res.sendStatus(200)
})

const get = async (path: string) => {
  const r = await fetch(`https://api.roark.ai${path}`, {
    headers: { Authorization: `Bearer ${process.env.ROARK_API_KEY}` },
  })
  if (!r.ok) throw new Error(`${path} -> ${r.status}`)
  return r.json()
}
```

***

## Approach B: Scheduled backfill / poll

Best for the initial backfill of historical data, or as a simpler alternative to webhooks. Page through the calls list on a schedule and fetch metrics per call.

The list endpoint uses keyset (cursor) pagination. Sort by `startedAt` and walk the cursor to move forward in time.

```python Python (paginated pull) theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
def export_all_calls():
    cursor = None
    while True:
        params = {'limit': 100, 'sortBy': 'startedAt', 'sortDirection': 'asc'}
        if cursor:
            params['after'] = cursor

        page = get('/v1/call', params=params)
        for call in page['data']:
            metrics = get(f"/v1/call/{call['id']}/metrics")
            write_to_s3(call, metrics)

        if not page['hasMore']:
            break
        cursor = page['nextCursor']
```

```bash cURL (single page) theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -G "https://api.roark.ai/v1/call" \
  -H "Authorization: Bearer $ROARK_API_KEY" \
  --data-urlencode "limit=100" \
  --data-urlencode "sortBy=startedAt" \
  --data-urlencode "sortDirection=asc" \
  --data-urlencode "after=$CURSOR"
```

> [!NOTE]
>
> For a large backfill you can also pull metric values a whole job at a time with `GET /v1/metric/collection-jobs/{jobId}/metric-values`, which returns every metric value for that collection job in one paginated stream rather than one request per call.

***

## Loading into Snowflake

Land the raw JSON in an external stage, ingest into a `VARIANT` staging table, then flatten into modeled tables with `LATERAL FLATTEN`. Storing the raw payload first means a new metric or field never breaks your pipeline.

**1. Create an external stage over your bucket**

```sql theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
create or replace stage roark_stage
  url = 's3://my-roark-export/calls/'
  storage_integration = my_s3_integration
  file_format = (type = json);
```

**2. Auto-ingest with Snowpipe**

```sql theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
create table roark_raw (raw variant, loaded_at timestamp_ntz default current_timestamp());

create or replace pipe roark_pipe auto_ingest = true as
  copy into roark_raw (raw) from @roark_stage;
```

**3. Flatten metrics into a queryable table**

```sql theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
create or replace view roark_call_metrics as
select
  raw:call:id::string           as call_id,
  raw:call:externalId::string   as external_id,   -- your join key
  raw:call:startedAt::timestamp as started_at,
  raw:call:durationMs::number   as duration_ms,
  raw:call:callDirection::string as direction,
  m.value:slug::string          as metric_slug,
  m.value:name::string          as metric_name,
  m.value:type::string          as metric_type,
  m.value:value::variant        as metric_value,
  m.value:captureStatus::string as capture_status
from roark_raw,
  lateral flatten(input => raw:metrics:data) m;
```

**4. Join against your operational data**

```sql theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
-- Does call frustration predict whether a caregiver takes a shift?
select
  f.metric_value::float as frustration,
  s.shift_accepted
from roark_call_metrics f
join shifts s
  on s.roark_external_id = f.external_id
where f.metric_slug = 'frustration';
```

> [!TIP]
>
> Model the flattened view once per metric type you care about, or keep one long-format `(call_id, metric_slug, metric_value)` table and pivot in your BI layer. Long format handles new metrics with zero schema changes.

***

## Things to know

* **Idempotency.** Keying S3 objects and Snowflake rows by `callId` (plus the metric collection job id) makes re-delivery and backfills safe to re-run without double counting.
* **Webhook delivery.** Roark retries failed deliveries up to 3 times with backoff. Respond `200` within 5 seconds and process asynchronously. See [Webhooks](/documentation/integrations/webhooks).
* **Date filtering.** The calls list does not yet accept an explicit `from`/`to` date range. Approximate a range by sorting on `startedAt` and walking the cursor until you pass your window.
* **Transcripts** are fetched per call from the transcript endpoint; they are not included in the metrics payload.
* **Rate limits and pagination.** Use `limit=100` and honor `nextCursor`/`hasMore` rather than assuming a fixed page count.

***

## Deeper integrations

The pattern above uses only what is available today. If you need something more managed, we are building toward it and would like to scope it with you:

> [card] Managed connector
>
> A Fivetran / Airbyte-style connector with incremental, updated-since cursors so your warehouse stays in sync automatically.

> [card] Scheduled file drop
>
> Roark writes incremental Parquet or CSV to your own S3 bucket on a schedule, ready for Snowpipe.

> [card] Secure data share
>
> A hands-off Snowflake Secure Data Share so there is nothing to run on your side.

> [card] [Talk to us](/documentation/resources/support)
>
> Tell us your volume, latency needs, and the joins you want to run.

***

## Next steps

> [card] [Webhooks](/documentation/integrations/webhooks)
>
> Set up the analysis-completed notification

> [card] [API Keys](/documentation/getting-started/api-keys)
>
> Create a read-only key for the export

> [card] [API Reference](/api-reference/introduction)
>
> Full endpoint and parameter reference

> [card] [Reports](/documentation/observability/reports)
>
> Analyze the same data inside Roark


# Google CES
Source: https://docs.roark.ai/documentation/integrations/google-ces

Run chat simulations against Google Customer Engagement Suite apps

Chat simulationsLive monitoringVoice simulations

## Overview

The Google CES integration connects a [Google Customer Engagement Suite](https://docs.cloud.google.com/customer-engagement-ai/) app to Roark for [chat simulations](/documentation/simulation-testing/chat-simulations). Roark sends each persona turn to the CES [`runSession` endpoint](https://docs.cloud.google.com/customer-engagement-ai/conversational-agents/ps/reference/rest/v1/projects.locations.apps.sessions/runSession) and records the agent's reply as part of the chat transcript.

Authentication uses [Workload Identity Federation (WIF)](https://docs.cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds) so **no long-lived service account keys ever leave your GCP tenant**. Roark mints a short-lived access token at simulation time using credentials your team controls and can revoke at any time.

***

## Prerequisites

Before getting started, you'll need:

* A Google Cloud project with a published Customer Engagement Suite app
* Permissions in that project to create a Workload Identity Pool and configure a service account with access to your CES app
* The CES app coordinates: **Location**, **App ID**, **Version ID**, **Deployment ID**

***

## Setting Up Workload Identity Federation

WIF setup requires a few values from Roark to authorize in your Workload Identity Pool. **Reach out to us and we'll walk you through it end-to-end**: typically a single back-and-forth to share the values you need, after which you generate the WIF credential JSON on your side and paste it into Roark.

> [card] [Get help setting up WIF](mailto:support@roark.ai?subject=Google%20CES%20integration%20setup\&body=Hi%20Roark%20team%2C%0A%0AWe%27d%20like%20to%20set%20up%20a%20Google%20CES%20integration%20and%20would%20like%20your%20help%20configuring%20Workload%20Identity%20Federation.%0A%0AOur%20GCP%20project%20ID%3A%20%0ACES%20app%20ID%3A%20%0ACES%20location%3A%20%0A%0AThanks%21)
>
> Email **[support@roark.ai](mailto:support@roark.ai)** and we'll get you set up. Include your GCP project ID and CES app coordinates if you have them handy.

Once WIF is configured on your side, you'll have a credential configuration JSON document (`type: "external_account"`) ready to paste into Roark.

***

## Creating the Integration in Roark

**1. Open the integration form**

Navigate to **Agents**, click **Connect Agent** in the top right, choose **Existing Platform**, and select **Google CES**.

**2. Fill in the configuration**

| Field | Description |
| :- | :- |
| **Integration Name** | Friendly name for this integration |
| **WIF Config JSON** | The full external-account credential JSON from your Workload Identity Pool. Stored encrypted at rest. |
| **CES Location** | GCP region for your CES app (e.g. `us`, `global`) |
| **CES App ID** | UUID of the CES app, from the CES console |
| **CES Version ID** | UUID of the app version to run |
| **CES Deployment ID** | UUID of the deployment to target |

**3. Save**

On save, Roark validates the WIF document's shape, stores it encrypted, and auto-provisions:

* One **Roark agent** named after the integration
* One **`GOOGLE_CES` chat endpoint** linked to the integration

The endpoint is immediately usable in run plans.

***

## How Authentication Works at Runtime

When a chat simulation runs against a Google CES endpoint, Roark mints a fresh access token for the request rather than storing one:

1. Roark uses your WIF configuration to obtain a federated token from Google STS.
2. The federated token is exchanged for a short-lived OAuth access token that impersonates your configured service account.
3. Roark sends `Authorization: Bearer <token>` on each call to `…/sessions/{sessionId}:runSession`.

Tokens are short-lived (≈1 hour) and live only in memory for the duration of the simulation. You can revoke access at any time by removing Roark from your Workload Identity Pool.

***

## How Conversations Are Driven

Each simulation generates a unique session ID per run. For every persona turn, Roark POSTs to:

```
POST https://<location>-ces.googleapis.com/v1beta/projects/<project>/locations/<location>/apps/<appId>/sessions/<sessionId>:runSession
```

with a body referencing your `app_version` and `deployment` plus the user input text. The agent's reply is read from `outputs[0].text` and added to the chat transcript.

CES creates the session on the first call with a new session ID and resumes it on subsequent calls with the same ID, so no separate bootstrap call is needed.

***

## What's Not Supported

* **Voice simulations**: Google CES integrations are chat-only.
* **Live monitoring / call import**: Roark does not pull historical or live conversations from CES. Use chat simulations to evaluate the agent.

***

## Next Steps

> [card] [Run Chat Simulations](/documentation/simulation-testing/chat-simulations)
>
> Test your CES app with persona-driven chats

> [card] [Build customer flows](/documentation/simulation-testing/customer-flows)
>
> Define the conversations to test

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Automate evaluation on every chat

> [card] [Integration Overview](/documentation/integrations/overview)
>
> Explore other integrations


# IP Whitelisting
Source: https://docs.roark.ai/documentation/integrations/ip-whitelisting

Whitelist Roark Analytics production IPs for firewalls, webhooks, and presigned URLs

If you need to whitelist Roark Analytics' IP addresses for accessing your infrastructure (for example, presigned URLs or private APIs), use the following static IP addresses.

***

## Production Environment

All outbound requests from Roark Analytics production services originate from these NAT gateway IPs:

* 54.175.209.200
* 54.161.212.170

> [!NOTE]
>
> Important: You must whitelist both IP addresses. Traffic may route through either one depending on availability zone placement.

***

## Use Cases

Whitelist these IPs when:

* Providing presigned URLs for audio files or recordings
* Exposing webhook endpoints with IP restrictions
* Configuring firewall rules for API integrations
* Setting up private API access

***

## Region

These IPs are for the US-EAST-1 region where Roark Analytics production services are hosted.


# Kore AI
Source: https://docs.roark.ai/documentation/integrations/kore

Run chat simulations against Kore AI Agent Platform apps

Chat simulationsLive monitoringVoice simulations

## Overview

The Kore AI integration connects an app on the [Kore AI Agent Platform](https://docs.kore.ai/agent-platform/) to Roark for [chat simulations](/documentation/simulation-testing/chat-simulations). Roark talks to your app over the Agent Platform v2 HTTP API, bootstrapping a session, sending each persona turn through `/runs/execute`, and tearing the session down when the conversation ends.

***

## Prerequisites

Before connecting, you'll need from the Kore AI Agent Platform:

* An **App ID** (UUID) for the agent app you want to test
* An **Environment name** the app is deployed to (e.g. `production`)
* An **API key** (`x-api-key`) with permission to create sessions and execute runs against that app
* The **base URL** of your Kore tenant, if you're on a non-default or self-hosted Kore domain. Most customers can leave this blank and Roark will use `https://agent-platform.kore.ai/api/v2`.

See the [Kore AI Agent Platform API reference](https://docs.kore.ai/agent-platform/apis/agentic-apps/overview/) for where to find these in the Kore console.

***

## Creating the Integration in Roark

**1. Open the integration form**

Navigate to **Agents**, click **Connect Agent** in the top right, choose **Existing Platform**, and select **Kore AI**.

**2. Fill in the configuration**

| Field | Description |
| :- | :- |
| **Integration Name** | Friendly name for this integration |
| **API Key** | Your Kore `x-api-key`. Stored encrypted at rest. |
| **App ID** | UUID of the Kore agent app |
| **Environment** | Environment name the app is deployed to (e.g. `production`) |
| **Base URL** *(optional)* | Override the Kore API base URL for enterprise or self-hosted deployments. Defaults to `https://agent-platform.kore.ai/api/v2`. |

**3. Save**

On save, Roark stores the credentials encrypted and auto-provisions:

* One **Roark agent** named after the integration
* One **`KORE` chat endpoint** linked to the integration

The endpoint is immediately usable in run plans. Credentials never leave the integration. The endpoint references them by ID.

> [!NOTE]
>
> In edit mode, leave the **API Key** field blank to keep the existing key. Only fill it in if you want to rotate the credential.

***

## How Conversations Are Driven

Each chat simulation runs through a three-leg session lifecycle against your Kore app:

| Phase | Request |
| :- | :- |
| **Bootstrap** | `POST {baseUrl}/apps/{appId}/environments/{env}/sessions`: creates a session and returns a session reference. |
| **Per turn** | `POST {baseUrl}/apps/{appId}/environments/{env}/runs/execute`: sends the persona's message as text input under the bootstrapped session, and reads the agent's reply from `outputs[0].content`. |
| **Teardown** | `POST {baseUrl}/apps/{appId}/environments/{env}/sessions/terminate`: closes the session when the chat ends. |

All requests are sent with your `x-api-key` in the header. The API key is decrypted in memory only for the lifetime of the chat session and is never logged.

***

## What's Not Supported

* **Voice simulations**: Kore AI integrations are chat-only in Roark.
* **Live monitoring / call import**: Roark does not pull historical conversations from Kore. Chat simulations are the way to evaluate your Kore app's behavior in Roark.

***

## Next Steps

> [card] [Run Chat Simulations](/documentation/simulation-testing/chat-simulations)
>
> Test your Kore app with persona-driven chats

> [card] [Build customer flows](/documentation/simulation-testing/customer-flows)
>
> Define the conversations to test

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Automate evaluation on every chat

> [card] [Integration Overview](/documentation/integrations/overview)
>
> Explore other integrations


# Leaping
Source: https://docs.roark.ai/documentation/integrations/leaping

Sync Leaping AI agents and analyze voice conversations

Live monitoringVoice simulationsChat simulations

## Overview

The Leaping integration syncs your Leaping AI agents and their calls into Roark. Agents, prompts, and tools are imported automatically, and calls are pulled on a recurring schedule so every conversation is analyzed.

***

## Prerequisites

Before setting up the integration, ensure you have:

* A Leaping AI account with active agents
* A Leaping API key (found in your Leaping dashboard)
* At least one configured agent in Leaping

***

## Setup Instructions

### Step 1: Create Integration

1. Navigate to **Agents** in your Roark dashboard
2. Click **Connect Agent** in the top right, choose **Existing Platform**, and select **Leaping**
3. Enter your configuration:

| Field | Description |
| :- | :- |
| **Integration Name** | Friendly name for this integration |
| **API Key** | Your Leaping API key |

The system will validate your API key and fetch available agents.

### Step 2: Configure Historical Sync

Choose how far back to import existing calls:

* **Default**: Last 90 days of call history
* **Custom**: Select a specific date range
* **Skip**: Only sync new calls going forward

> [!NOTE]
>
> Historical sync runs as a backfill job during initial setup. Ongoing calls are synced automatically on a recurring schedule.

### Step 3: Select Agents

Choose which Leaping agents to monitor:

* **All Agents**: Sync calls from every agent in your account
* **Selected Agents**: Choose specific agents to monitor

The agent selector shows each agent's name and ID.

### Step 4: Activate Integration

Review your settings and click **Create Integration** to begin syncing.

***

## What Gets Synced

Leaping integrations sync comprehensive data:

* **Calls**: Conversation transcripts, metadata, and call events
* **Agents**: Agent configurations and endpoints
* **Prompts**: System prompts configured for each agent
* **Tools**: Tool definitions available to agents
* **Call Metadata**: Duration, status, properties, and events

***

## How Sync Works

Leaping uses **pull-based synchronization**: Roark periodically fetches new calls from the Leaping API rather than receiving webhooks.

| Phase | What Happens |
| :- | :- |
| **Agent Sync** | Fetches agents and endpoints, updates configurations, syncs prompts and tools |
| **Health Check** | Verifies API connectivity and agent availability |
| **Call Sync** | Fetches calls since last sync for each agent, processes transcripts and metadata |

Calls with status `completed`, `transferred`, or `dropped` are synced.

***

## Monitoring Integration Health

Track your integration status in the Roark dashboard:

* **Syncing**: Actively importing calls
* **Active**: Running on schedule, pulling new calls periodically
* **Paused**: Sync temporarily halted
* **Error**: Configuration or connection issues

View sync statistics including total calls synced, last sync timestamp, and job history.

***

## Next Steps

> [card] [Run Simulations](/documentation/simulation-testing/running-simulations)
>
> Test synced agents with simulations

> [card] [View Call History](/documentation/observability/live-monitoring)
>
> Browse and analyze synced calls

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Automate metric collection on calls

> [card] [Integration Overview](/documentation/integrations/overview)
>
> Explore other integrations


# LiveKit Cloud
Source: https://docs.roark.ai/documentation/integrations/livekit

Connect LiveKit Cloud for real-time voice communication monitoring

Live monitoringVoice simulationsChat simulations

## Overview

The LiveKit Cloud integration enables real-time monitoring of voice communications built on [LiveKit Cloud](https://cloud.livekit.io/). Roark connects to your LiveKit project over webhooks and the server API (no changes to your agent code) and streams call data in for analysis, evaluation, and quality monitoring.

> [!NOTE]
>
> Running your own `livekit-server` instead of LiveKit Cloud? Webhooks and the dashboard-managed flow described here still work, but the recommended path for self-hosted deployments is the drop-in Python SDK. See [LiveKit (self-hosted)](/documentation/integrations/livekit-self-hosted).

***

## Prerequisites

Before setting up the integration, ensure you have:

* A [LiveKit Cloud](https://cloud.livekit.io/) account and project
* LiveKit API credentials (API Key and Secret)
* Admin access to configure webhooks in LiveKit

***

## Setup Instructions

### Step 1: Create Integration

1. Navigate to **Agents** in your Roark dashboard
2. Click **Connect Agent** in the top right, choose **Existing Platform**, and select **LiveKit**
3. Enter the following configuration:

| Field | Description | Example |
| :- | :- | :- |
| **Integration Name** | Friendly name for this integration | Production LiveKit |
| **Server URL** | Your LiveKit WebSocket URL | `wss://example.livekit.cloud` |
| **API Key** | LiveKit API key | `APIxxxxxxxxxxxxx` |
| **API Secret** | LiveKit API secret | Keep this secure |

### Step 2: Configure Webhook

After creating the integration, Roark provides a webhook URL:

1. Copy the webhook URL from the integration settings
2. Navigate to your LiveKit dashboard
3. Go to **Settings → Webhooks**
4. Add the Roark webhook URL
5. Select the following events:
   * `room_started`
   * `room_finished`
   * `participant_joined`
   * `egress_ended`

### Step 3: Configure Agents

Under the **Agents** section of the integration, decide how LiveKit calls map to Roark agents:

* **Import named LiveKit agents**: Roark fetches the named agents configured in your LiveKit project so you can choose which ones to track. Each imported agent becomes its own Roark agent, and calls are routed based on the LiveKit agent name that participated in the room.
* **Assign all calls to a single Roark agent**: Pick one Roark agent and every call from this integration is attributed to it. Use this when you rely on LiveKit's automatic dispatch (unnamed agents) or when you don't care about distinguishing multiple agents.

> [!NOTE]
>
> You can switch between these modes later, but existing calls will keep the agent attribution they had at ingest time.

### Step 4: WebRTC Endpoints (optional)

The **WebRTC** section of the integration page is where you manage all LiveKit WebRTC endpoints for this integration. On first setup it can auto-create one endpoint per imported agent so you can start running simulations immediately; after the integration exists you can come back to this same section at any time to add, edit, or remove endpoints. See [WebRTC Endpoints](#webrtc-endpoints) below for the full list of settings. You can skip this step and add endpoints later if you prefer.

### Step 5: Configure Metric Collectors

Calls synced from LiveKit are scored by your [metric collectors](/documentation/metrics/metric-collectors). Set them up globally under **Metrics → Collectors**, or scope collectors to a specific agent from that agent's **Collectors** section.

### Step 6: Activate Integration

Toggle the integration status to **Active** to begin receiving call data.

***

## WebRTC Endpoints

Roark can run simulations directly over WebRTC against a LiveKit room instead of placing a phone call. This is the recommended path for LiveKit agents because it avoids SIP trunking and exercises the exact transport your production users connect over.

### Managing WebRTC Endpoints

All LiveKit WebRTC endpoints for an integration are managed from the **WebRTC** section of the integration page:

1. Go to **Agents**, expand the connected sources at the top of the page, and choose your **LiveKit** source. The integration form opens on the right.
2. Expand the **WebRTC** section to see every endpoint attached to an agent that belongs to this integration. Each row shows the agent name and environment.
3. Click an existing endpoint to edit it, or click **Add WebRTC endpoint** to create a new one. The add flow opens the same editor used on the agent page and is scoped to agents managed by this integration.

> [!NOTE]
>
> On initial integration setup, Roark can also auto-create one WebRTC endpoint per imported agent in a single step. See [Step 4: WebRTC Endpoints](#step-4-webrtc-endpoints-optional) above. After the integration exists, use the flow described here for every change.

### Room Management Mode

When configuring the endpoint you pick who is responsible for creating the LiveKit room that a simulation runs in.

> [tab] Roark manages the room
>
> Roark creates the room in your LiveKit project at the start of each simulation using the API credentials from the integration, then joins the room as the simulated caller.
>
> * **Agent dispatch**: If a **Named LiveKit Agent** is set on the integration, Roark explicitly dispatches that agent into the room so it joins as soon as the room is created. If no named agent is set, Roark relies on LiveKit's [automatic dispatch](https://docs.livekit.io/agents/worker/agent-dispatch/) behavior: any unnamed worker connected to your LiveKit project will pick up the room automatically.
> * **Room metadata**: Optionally provide a JSON string that Roark attaches to the room it creates. Use this to pass context to your agent (for example, a customer ID, scenario tag, or feature flag) that your worker can read from the room metadata on join.
>
> This mode is the easiest to set up and works for most agents that use LiveKit's standard dispatch model.

> [tab] You manage the room
>
> Provide an HTTPS URL that Roark calls when a simulation starts. Your endpoint is responsible for creating the LiveKit room, dispatching your agent into it, and returning the room name so Roark can join.
>
> **Request**: Roark sends an HTTPS request to your URL when a simulation starts.
>
> **Expected response**: Return a JSON body containing the `roomName` of the LiveKit room you created. Roark then connects to the same LiveKit server and joins that room as the simulated caller.
>
> ```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> {
>   "roomName": "sim-session-abc123"
> }
> ```
>
> Use this mode when you need custom room setup logic: for example, if you provision rooms through your own backend, need to wire up per-session tokens or tools, or run a bespoke dispatch flow that Roark's auto-create path can't express.

Once the endpoint exists, use it like any other agent endpoint when building [run plans](/documentation/simulation-testing/run-plans) and [running simulations](/documentation/simulation-testing/running-simulations).

***

## What Gets Synced

LiveKit integrations sync the following data:

* **Call Events** - Room creation, participant joining, call completion
* **Recording Files** - MP4 recordings from egress events
* **Call Metadata** - Duration, participant count, room configuration
* **Transcripts** - When speech-to-text is enabled
* **Named Agents** - When agent import is enabled in the [Agents](#step-3-configure-agents) step, the list of named LiveKit agents from your project

> [!WARNING]
>
> LiveKit does not sync agent prompts or worker source code, as it's a real-time
> communication platform rather than an AI agent provider. Only agent names are
> imported so Roark can attribute calls to the correct agent.

***

## Webhook Events

The integration processes these LiveKit events:

| Event | Description | Roark Action |
| :- | :- | :- |
| `room_started` | Room created | Initialize call record |
| `room_finished` | Room ended | Finalize call, trigger evaluation |
| `participant_joined` | User joined | Update participant list |
| `egress_ended` | Recording ready | Process recording file |

***

## Call Sampling & Metadata Filtering

Roark provides two ways to control which LiveKit calls are processed:

### Random Sampling

Set a **sampling rate** (0–100%) in the integration settings to randomly sample a percentage of calls. This is useful when you want to evaluate a representative subset without any code changes.

### Metadata Filtering (Per-Call Control)

For more granular control, set `roark.skip` to `true` in your LiveKit room metadata to tell Roark to skip processing for specific calls.

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
from livekit import api

room = api.CreateRoomRequest(
    name="my-room",
    metadata='{"roark.skip": true}'
)
```

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { RoomServiceClient } from 'livekit-server-sdk';

const roomService = new RoomServiceClient(url, apiKey, apiSecret);
await roomService.createRoom({
  name: 'my-room',
  metadata: JSON.stringify({ 'roark.skip': true }),
});
```

**How it works:**

* Calls **without** `roark.skip` in metadata are processed normally (safe default)
* Calls **with** `roark.skip` set to `true` are skipped entirely: no recording, no evaluation, no processing
* The check happens at the webhook level, so skipped calls use zero Roark resources
* You can combine metadata filtering with sampling: sampling is applied first, then metadata filtering

> [!NOTE]
>
> Metadata filtering is ideal when you want deterministic control over which calls are processed, such as skipping internal test calls or only evaluating calls for specific customers.

***

## Monitoring Integration Health

Check your integration status in the Roark dashboard:

* **Active** 🟢 - Receiving and processing events
* **Inactive** 🔴 - Integration paused
* **Error** ⚠️ - Configuration or connection issues

View recent webhook deliveries and any error messages in the integration details.

***

## Next Steps

> [card] [Enable Tracing](/documentation/observability/traces?tab=livekit)
>
> Send OpenTelemetry traces from your LiveKit agent

> [card] [Configure Metrics](/documentation/metrics/overview)
>
> Define metrics and pass/fail thresholds

> [card] [View Live Calls](/documentation/observability/live-monitoring)
>
> Monitor calls in real-time

> [card] [Integration Overview](/documentation/integrations/overview)
>
> Learn about all integrations

> [card] [Self-hosted LiveKit](/documentation/integrations/livekit-self-hosted)
>
> Instrument your own livekit-server with the Python SDK


# LiveKit (self-hosted)
Source: https://docs.roark.ai/documentation/integrations/livekit-self-hosted

Instrument a self-hosted LiveKit Agents worker with the roark-analytics[livekit] SDK

Live monitoringVoice simulationsChat simulations

## Overview

The self-hosted LiveKit integration monitors voice agents built on the open-source [LiveKit Agents](https://docs.livekit.io/agents/) framework that run against your own LiveKit server. Drop the [`roark-analytics[livekit]`](https://pypi.org/project/roark-analytics/) helper into your agent entrypoint and call lifecycle, transcripts, tool calls, and a recording are forwarded to Roark automatically. No other code changes required.

The SDK talks only to Roark's own API; it never connects to your LiveKit server.

> [!NOTE]
>
> Using [LiveKit Cloud](https://cloud.livekit.io/)? You can use this SDK there too, but the dashboard-managed webhook integration is usually simpler. See [LiveKit Cloud](/documentation/integrations/livekit).

***

## Prerequisites

* A LiveKit Agents worker (Python 3.10+, `livekit-agents` 1.x)
* Your own LiveKit server
* A Roark API key with **WRITE** scope ([generate one](/documentation/getting-started/api-keys))

***

## Setup Instructions

### Step 1: Install the SDK

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
pip install "roark-analytics[livekit]"
```

### Step 2: Configure your API key

The only setting you need to provide is your Roark API key.

```bash title=".env" theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# Roark API key: create one on the API keys page in your Roark project.
ROARK_API_KEY=rk_live_replace_me
```

| Variable | Required | Purpose |
| :- | :- | :- |
| `ROARK_API_KEY` | Yes | Roark API key with WRITE scope. Pass it to the helper as `api_key=`. |
| `ROARK_OBSERVABILITY_ENABLED` | No | Set to `false` to make `observe_session` a no-op. See [Kill switch](#kill-switch). |

### Step 3: Wire `observe_session` into your entrypoint

Call `observe_session(...)` **before `session.start()`** so the audio taps are installed before the session begins streaming frames. That single call wires up everything:

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import os

from livekit.agents import Agent, AgentSession, JobContext, WorkerOptions, cli
from livekit.plugins import cartesia, openai, silero, speechmatics

from roark_analytics_python_livekit import observe_session

SYSTEM_PROMPT = "You are a friendly voice assistant. Keep replies short. They are spoken aloud."


class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(instructions=SYSTEM_PROMPT)


async def entrypoint(ctx: JobContext) -> None:
    # Connect first: Roark keys the call on the LiveKit room sid (RM_…),
    # which is also what links OpenTelemetry traces to the call.
    await ctx.connect()

    session = AgentSession(
        stt=speechmatics.STT(),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=cartesia.TTS(),
        vad=silero.VAD.load(),
    )

    # --- Roark analytics ---------------------------------------------------
    # Must be called BEFORE session.start(). Failures are logged and
    # swallowed: your agent keeps running even if Roark is unreachable.
    await observe_session(
        ctx,
        session,
        api_key=os.environ["ROARK_API_KEY"],
        agent_id="support-bot-v1",
        agent_name="Support Bot",
        agent_prompt=SYSTEM_PROMPT,
    )
    # -----------------------------------------------------------------------

    await session.start(room=ctx.room, agent=Assistant())


if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
```

That's the full integration: agent registration, transcripts, tool calls, and the recording are all handled by the helper.

| Argument | Required | Description |
| :- | :- | :- |
| `ctx` | Yes | The `JobContext` passed to your agent entrypoint. |
| `session` | Yes | The `AgentSession` you're about to `start()`. |
| `api_key` | Yes | Your Roark API key (WRITE scope). |
| `agent_id` | Yes | Stable, customer-defined agent identifier. Used for lazy registration. |
| `agent_name` | No | Display name shown in the Roark dashboard. |
| `agent_prompt` | No | System prompt: persisted as the agent's prompt revision, so changes are tracked over time. |
| `**metadata` | No | Free-form metadata forwarded on `call-started` for Roark-side correlation. |

### Step 4: Verify the connection

Run your worker against your LiveKit server and place a test call:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
python agent.py dev
```

As the call runs you should see Roark log lines, then the recording upload begin after a few seconds of audio:

```
tapping user audio input (...)
tapping agent audio output (...)
call-started: ...
```

Within a few seconds you should also see, in your Roark dashboard:

1. The agent appear under the **LiveKit** source filter
2. The call appear in the calls table with status `In Progress`
3. After the call ends, the transcript, tool invocations, and recording attached to the call

***

## How It Works

`observe_session` subscribes to the standard `AgentSession` event surface and ships a compact event timeline to Roark:

| Phase | Source | What's captured |
| :- | :- | :- |
| **Session start** | `JobContext.connect()` | `call-started`, keyed on the LiveKit room sid (`RM_…`). Agent is lazy-registered the first time Roark sees an `agent_id`. |
| **Transcripts** | `conversation_item_added` | Message role + content, for both user and assistant turns. |
| **Tool calls** | `function_tools_executed` | Paired `tool_call` / `tool_result` records, keyed by `tool_call_id`. |
| **Recording** | Taps on `session.input.audio` (user) + `session.output.audio` (agent) | Audio streamed to Roark during the call. |
| **Session end** | `ctx.add_shutdown_callback(...)` | Flushes pending state, drains in-flight uploads, POSTs `call-ended`. |

Failures are logged and swallowed: **the helper never raises into your session**. Your agent keeps running even if Roark is unreachable.

***

## What Gets Synced

* **Calls**: Lifecycle with timing and end reason
* **Agents**: Lazy-registered on first sight using the `agent_id` / `agent_name` you pass
* **Prompts**: System prompt captured at call start as a prompt revision
* **Transcripts**: Per-turn messages with role, content, and timestamp
* **Tool Invocations**: Tool call IDs, names, JSON arguments, and results
* **Recordings**: Audio recording of the call

> [!WARNING]
>
> Roark only sees what the helper forwards. If you remove `observe_session` from a worker (or set `ROARK_OBSERVABILITY_ENABLED=false`), no data flows for those calls.

***

## Agent Management

LiveKit agents are **lazy-registered** the first time the helper reports them:

* The first `call-started` event with a new `agent_id` creates the agent in Roark
* Subsequent events update the agent's name and prompt if they change
* These agents appear on the agents page under the **LiveKit** source filter

Once an agent exists, it can be used in [simulations](/documentation/simulation-testing/running-simulations), [run plans](/documentation/simulation-testing/run-plans), and [agent reports](/documentation/observability/reports) just like agents from any other provider.

***

## Kill Switch

Disable instrumentation at runtime without touching code. Set `ROARK_OBSERVABILITY_ENABLED=false` to make `observe_session` a no-op (it returns `None`). Values treated as off: `false`, `0`, `no`, `off` (case-insensitive); anything else (or the variable being absent) keeps it enabled.

***

## Troubleshooting

> [accordion] Calls aren't finalizing on Roark
>
> The helper registers a shutdown callback on the `JobContext`, which fires when LiveKit ends the job. If your transport tears down without firing the hook, call `await state.aflush(reason="...")` explicitly from your own disconnect handler: `aflush()` is idempotent, so the normal shutdown path will no-op if both fire.
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> state = await observe_session(ctx, session, api_key=..., agent_id=...)
> # ... later, from your disconnect handler:
> await state.aflush(reason="client-disconnected")
> ```

> [accordion] Transcripts arrive empty
>
> Transcripts come from the `conversation_item_added` event on `AgentSession`. If you use a custom pipeline that bypasses `AgentSession.start(room=…, agent=…)`, that event may never fire. Verify by adding your own `session.on("conversation_item_added", print)` listener.

> [accordion] Recording is missing one side
>
> The user side is tapped from `session.input.audio`; the agent side from `session.output.audio`. Because the taps are installed by `observe_session`, it **must be called before `session.start()`**. Check worker logs for the `tapping user audio input` / `tapping agent audio output` lines. If one is missing, that side will be silent on the merged recording.

> [accordion] No calls or recordings reaching Roark
>
> Confirm `api_key` is a WRITE-scope key and that `ROARK_OBSERVABILITY_ENABLED` is not set to a falsy value.

***

## Next Steps

> [card] [Enable Tracing](/documentation/observability/traces#livekit)
>
> Send OpenTelemetry traces: they auto-link to the call via the room sid

> [card] [Run Simulations](/documentation/simulation-testing/running-simulations)
>
> Test your LiveKit agents

> [card] [Configure metrics](/documentation/metrics/custom-metrics)
>
> Set up your custom metrics

> [card] [LiveKit Cloud](/documentation/integrations/livekit)
>
> Use the dashboard-managed webhook integration instead


# Okta SSO + SCIM
Source: https://docs.roark.ai/documentation/integrations/okta

Sign your team into Roark with Okta, and optionally push-provision and deprovision users via SCIM

## What this gets you

Connecting Okta to Roark lets your team sign in with their Okta credentials. New users are auto-created in Roark on first sign-in (just-in-time provisioning), so you don't have to invite them individually. If you also enable SCIM, Okta becomes the source of truth for user lifecycle: assigning a user pushes them into Roark before their first login, profile updates flow through automatically, and deactivating the user in Okta disables their Roark account.

***

## How it works

SAML and SCIM are two independent channels. You can run SAML-only (JIT provisioning) and add SCIM later, or run both from day one.

![Okta to Roark auth and provisioning flow: SAML through Cognito for sign-in, SCIM directly to Roark for user lifecycle](https://mintcdn.com/roark/s7g4ucTAKodk8q96/images/integrations/okta-sso-flow.svg?fit=max\&auto=format\&n=s7g4ucTAKodk8q96\&q=85\&s=f73915a9509237a4386f95b0483f9810)

* **SAML** authenticates the user. Okta posts the SAML response to Roark's Cognito User Pool, which establishes the session.
* **SCIM** is a separate, outbound channel from Okta to Roark for user lifecycle events (create, update, deactivate).

Roark verifies you own the email domain via a DNS TXT record before activating the SSO config, preventing accidental or malicious claims on a domain you don't control.

***

## Before you start

* You are an **OWNER** of your Roark organization.
* You have admin access to your Okta tenant.
* You can publish a DNS TXT record at the apex of your email domain.
* You know which email domain your team uses (e.g., `acmecorp.com`). One domain per Roark org.

***

## Step 1: Create the SAML app in Okta

In your Roark dashboard, open **Org Settings → Single Sign-On**. The page shows two values you'll paste into Okta: the **ACS URL** and the **SP Entity ID**. Keep that tab open.

In the Okta Admin Console:

1. Go to **Applications → Applications → Create App Integration**.
2. Choose **SAML 2.0** and click **Next**.
3. Give the app a name (e.g., "Roark") and continue.
4. On the SAML Settings screen, fill in:

| Field | Value |
| :- | :- |
| Single sign-on URL (ACS) | (copy from Roark SSO settings) |
| Audience URI (SP Entity ID) | (copy from Roark SSO settings) |
| Name ID format | `EmailAddress` |
| Application username | `Email` |

5. Add three **Attribute Statements**. Names are **case-sensitive**, type them exactly:

| Name | Name format | Value |
| :- | :- | :- |
| `email` | Basic | `user.email` |
| `given_name` | Basic | `user.firstName` |
| `family_name` | Basic | `user.lastName` |

6. Finish the wizard. On the resulting app's **Sign On** tab, find **Identity Provider metadata** and copy the URL.

***

## Step 2: Enable SAML in Roark

Back in **Org Settings → Single Sign-On**:

1. Paste the IdP metadata URL.
2. Enter your email domain (e.g., `acmecorp.com`).
3. Click **Enable SSO**.

Roark validates the metadata, registers the IdP with Cognito, and switches the settings page into the **pending verification** state with your ACS URL, Entity ID, and a DNS TXT record envelope displayed.

![Roark Single Sign-On settings for Okta in the pending verification state: the stored email domain and Okta metadata URL, with a Pending verification badge on the domain](https://mintcdn.com/roark/yUzvMIE3_BYBHPDx/images/integrations/okta-sso-pending.png?fit=max\&auto=format\&n=yUzvMIE3_BYBHPDx\&q=85\&s=944b44f415c678e8fc7ce2fdb46870c5)

> [!WARNING]
>
> Email addresses must match. If a user already has a Roark account under a different email than what Okta will send, you'll end up with duplicates. Reconcile email addresses in Okta before assigning users.

***

## Step 3: Verify your email domain

After you enable SSO, Roark issues a one-time verification token and shows a DNS TXT record envelope on the settings page. Until you publish the record and Roark confirms it, the SSO config is inert: sign-ins for the domain fall through to the non-SSO path and any SCIM token you generate will not authenticate.

Copy the values from the settings page into your DNS provider:

| Field | Value |
| :- | :- |
| Type | `TXT` |
| Name | your email-domain apex (e.g., `acmecorp.com`) |
| Value | `roark-domain-verification=<token>` |

1. Add the record at the **apex** of your email domain, not a subdomain like `_roark.acmecorp.com`. It coexists with existing TXT records on the apex (SPF, DKIM, Google site verification, and so on).
2. Wait for propagation. This is typically a few minutes, but can take up to an hour depending on your provider's TTL.
3. Roark polls automatically in the background once the token is issued, so most admins won't need to click anything. Leave the settings page open for a few minutes after the DNS change propagates. There's a **Verify now** button for an explicit check.

![Roark domain verification section: the DNS TXT record to publish (Type, Host, and a copyable monospace Value) with a Regenerate token button and a Verify now button, in the pending state](https://mintcdn.com/roark/yUzvMIE3_BYBHPDx/images/integrations/okta-txt-verify.png?fit=max\&auto=format\&n=yUzvMIE3_BYBHPDx\&q=85\&s=1923fb0d226cbc2f807318e3d6905b7d)

The token is valid for **7 days**. If it expires before you publish the record, click **Regenerate** to issue a fresh one. Regenerating invalidates any prior token, and if the row was already verified, clears that verified state as well (the new token becomes a re-attest).

> [!WARNING]
>
> Until the domain is verified, the SSO config does not activate. Sign-ins for the domain are not routed through your IdP and SCIM bearer tokens will not authenticate. The settings page surfaces a **Pending verification** badge.

If another Roark org has already verified the same email domain, your verify call returns `dns_record_mismatch`. Roark won't tell you which org owns it. [Contact support](/documentation/resources/support).

***

## Step 4: Assign users and test

1. In Okta, open the app's **Assignments** tab and assign at least one test user.
2. Have that user go to [roark.ai/login](https://roark.ai/login), enter their work email, and submit.
3. Roark detects SSO for the domain and redirects to Okta. After authenticating, the user lands in Roark, auto-created and joined to your org if they didn't already exist.

If sign-in fails, jump to [Troubleshooting](#troubleshooting).

***

## Step 5 (optional): Turn on SCIM

SCIM lets Okta push user lifecycle events into Roark, useful for larger teams and required if you want Okta to deactivate Roark accounts automatically.

1. In Roark, on **Org Settings → Single Sign-On → Okta**, click **Generate SCIM token**. Copy both the **base URL** and the **bearer token**. The token is shown once.

![Roark SCIM token modal: a one-time bearer token and the SCIM base URL in copyable monospace fields, with a warning that the full token is shown only once](https://mintcdn.com/roark/yUzvMIE3_BYBHPDx/images/integrations/okta-scim-token.png?fit=max\&auto=format\&n=yUzvMIE3_BYBHPDx\&q=85\&s=499b4a306e05ef5b00f3290248fbea5b)

2. In Okta, open the same SAML app and go to the **Provisioning** tab. Click **Configure API Integration**.

3) Check **Enable API integration**, paste the SCIM base URL and bearer token, and click **Test API Credentials**. Save.
4) Under **Provisioning → To App**, enable:
   * **Create Users**
   * **Update User Attributes**
   * **Deactivate Users**

> [!WARNING]
>
> **If you already have Roark users**, run **Provisioning → To App → Import Users from App** *before* assigning anyone in Okta. Okta calls Roark's SCIM endpoint, matches existing users by email, and records the link. Skipping this step still works (Roark matches by email on the first SCIM `POST` and returns the existing user), but importing first gives Okta a clean linked state and avoids confusing "User already exists" warnings in the Okta UI.

5. Assign users (or groups) to the app. Okta will create or link Roark accounts on assignment.

***

## Auto-join projects (optional)

By default, new SSO and SCIM-provisioned users are added to your organization but no projects. To onboard them into a default set of projects automatically:

1. Go to **Org Settings → Single Sign-On → Auto-join projects**.
2. Pick the projects new users should land in.
3. Save.

From then on, every user provisioned via JIT or SCIM is auto-added to those projects. Existing users are unaffected.

***

## What happens for existing Roark users

Account linking only takes effect once the domain is verified. Until then, SCIM `POST /Users` is rejected (the token doesn't authenticate) and SAML sign-ins fall through to the non-SSO path.

Roark links existing accounts (password or Google) to Okta by email: no duplicates, no data loss. The user keeps their user ID, history, and project memberships.

* **JIT path:** linking happens on first SSO sign-in. Roark finds the user by email and merges Okta into their identity providers.
* **SCIM path:** when Okta sends `POST /Users` for an existing user, Roark responds `200` (link) instead of `201` (create) and records the Okta `externalId`. The user does not need to sign in for this to take effect.

If the email belongs to a different active Roark org, SCIM responds `409 (uniqueness)`. To resolve, remove the user from the other org and retry, or [contact support](/documentation/resources/support).

***

## Deprovisioning

* **Unassign in Okta (SAML only):** the user can't start a new session. Existing sessions expire within \~1 hour.
* **SCIM `PATCH active=false`:** the user is marked `INACTIVE`, their Cognito record is disabled, and their project memberships are removed. If you later re-activate them in Okta, their org membership and prior project memberships are restored.

***

## Disabling SSO

From **Org Settings → Single Sign-On**, click **Disable SSO**. Roark removes the IdP from Cognito. Active sessions keep working until the token expires (within an hour). Users with a linked password or Google identity can still sign in that way; SSO-only users will be locked out until you re-enable.

Disabling SSO does not clear domain verification. If you re-enable with the same email domain later, the prior verification still stands. If you re-enable with a different email domain, you'll need to verify the new one.

***

## Troubleshooting

* **"Invalid SAML response received"**: Cognito couldn't fetch your IdP metadata. Confirm the metadata URL is publicly reachable and returns valid XML.
* **User signs in but is treated as new**: the email in the Okta profile doesn't match the one on the existing Roark account. Update the Okta user's email to match, or contact support to relink.
* **"Email domain is already claimed"**: another Roark org has already enabled SSO for this domain. Contact support.
* **SSO check times out at login**: Roark couldn't look up the SSO provider for that domain. Check your network/firewall to roark.ai and confirm the email domain exactly matches what you configured.

Domain verification returns one of the following results. The settings page surfaces these on every **Verify now** click; transient misses during background polling stay silent.

* **`dns_record_not_found`**: no TXT record at the apex carries the `roark-domain-verification=` prefix. Confirm you added the record at the apex (not `_roark.acmecorp.com`) and allow time for DNS propagation, typically minutes, sometimes up to an hour depending on your provider's TTL.
* **`dns_record_mismatch`**: a record with the `roark-domain-verification=` prefix exists, but its token doesn't match the one Roark issued. Two common causes: you regenerated the token after pasting the old value (paste the new one), or another organization is concurrently attempting to claim the same domain (only one will win: [contact support](/documentation/resources/support) if you believe this is your domain).
* **`token_expired`**: the issued token is past its 7-day TTL. Click **Regenerate** and paste the fresh value.
* **`dns_lookup_failed`**: DNS resolution itself errored. Usually transient; retry. If it persists, confirm the domain resolves at all (`dig TXT acmecorp.com`).
* **`verified`**: success. The SSO config is now active end-to-end; SAML sign-ins and SCIM provisioning will route.

***

## Limitations

* IdP-initiated SAML Single Logout is not supported. Logging out of Okta does not end Roark sessions; existing sessions expire on their own within \~1 hour.
* One email domain per Roark org. If your team uses multiple domains, [reach out](/documentation/resources/support).
* Wildcard or sub-domain claims are not supported. Verification and the email-domain match are exact at the apex only.
* SCIM `GET /Users` only honors the `userName eq "<email>"` filter. Okta only uses this form in practice; other filter expressions (`userName co`, `active eq true`, etc.) are ignored.

***

## Next steps

> [card] [IP Whitelisting](/documentation/integrations/ip-whitelisting)
>
> Allow Roark's outbound IPs through your firewall

> [card] [Support](/documentation/resources/support)
>
> Get help with Okta setup


# Overview
Source: https://docs.roark.ai/documentation/integrations/overview

Every platform Roark connects to, and exactly which capabilities each one supports

Roark connects to your agent platform in two directions: it **pulls production conversations in** for monitoring and scoring, and it **drives simulated conversations out** for testing. Which of those you get depends on the platform.

This page is the map. Start with the capability matrix, then jump to the setup guide for your platform.

***

## Capability Matrix

&#x20;Supported out of the box  ·   Possible with extra setup  ·   Not supported

| Platform | Import calls | Import chats | Sync agents | Telephony sims | WebRTC sims | Chat sims | Traces |
| :- | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
| [**VAPI**](/documentation/integrations/vapi) | | | | | | | |
| [**Pipecat**](/documentation/integrations/pipecat) | | | | | | | |
| [**Retell AI**](/documentation/integrations/retell) | | | | | | | |
| [**ElevenLabs**](/documentation/integrations/elevenlabs) | | | | | | | |
| [**Bland AI**](/documentation/integrations/bland) | | | | | | | |
| [**Leaping**](/documentation/integrations/leaping) | | | | | | | |
| [**LiveKit Cloud**](/documentation/integrations/livekit) | | | | | | | |
| [**LiveKit (self-hosted)**](/documentation/integrations/livekit-self-hosted) | | | | | | | |
| [**Google CES**](/documentation/integrations/google-ces) | | | | | | | |
| [**Kore AI**](/documentation/integrations/kore) | | | | | | | |
| [**Custom / API**](/documentation/integrations/custom-integrations) | | | | | | | |

### Notes on specific cells

> [accordion] LiveKit Cloud: agent sync is names only
>
> LiveKit is a real-time transport, not an agent framework, so it doesn't expose prompts or worker source code. Roark imports the **named agents** configured in your LiveKit project purely so calls can be attributed to the right agent. If you want prompts and tool definitions in Roark, use the [self-hosted SDK integration](/documentation/integrations/livekit-self-hosted), which captures the system prompt as a tracked prompt revision.

> [accordion] Traces: platform guides vs. generic OTLP
>
> Roark acts as an OpenTelemetry collector at `https://api.roark.ai/v1/traces`, so **any** platform can send traces over OTLP/HTTPS. Step-by-step guides exist for [VAPI](/documentation/observability/traces#vapi), [LiveKit](/documentation/observability/traces#livekit), and [custom integrations](/documentation/observability/traces#custom-integration). Platforms marked  work through the custom-integration path: you instrument your own code and correlate spans to the call yourself.

> [accordion] Importing chats
>
> No provider currently pushes production **chat** transcripts into Roark. Text conversations reach Roark either from a [chat simulation](/documentation/simulation-testing/chat-simulations) or by posting them to the [`/v1/chat` API](/api-reference/introduction) yourself, which is why **Custom / API** is the only row with chat import.

> [accordion] WebRTC on other platforms
>
> A WebRTC endpoint can be attached to any Roark agent whose underlying agent speaks **LiveKit** or **SmallWebRTC**. The matrix marks the platforms where that's the native, documented path. If your VAPI or ElevenLabs agent is fronted by one of those transports, you can wire up a WebRTC endpoint for it too.

***

## What Each Capability Means

| Capability | What you get |
| :- | :- |
| **Import calls** | Production voice calls flow into Roark with audio, transcript, tool invocations, and metadata, then get scored automatically by your [metric collectors](/documentation/metrics/metric-collectors). |
| **Import chats** | Production text conversations land as first-class [chat records](/documentation/simulation-testing/chat-simulations), evaluated the same way calls are. |
| **Sync agents** | Agent configurations, system prompts, phone numbers, and tool definitions import automatically and stay up to date. Prompt changes are tracked as revisions. |
| **Telephony sims** | Roark dials your agent over a real PSTN call, or answers when your agent dials Roark. See [inbound vs. outbound](/documentation/simulation-testing/inbound-vs-outbound). |
| **WebRTC sims** | Roark joins your agent over [WebRTC](/documentation/simulation-testing/webrtc) instead of a phone line: no SIP, no telephony minutes, same transport as your web and mobile users. |
| **Chat sims** | Roark drives persona-based [text conversations](/documentation/simulation-testing/chat-simulations) against your agent over its chat transport. |
| **Traces** | OpenTelemetry spans attach to the matching call so you can debug latency, LLM calls, and tool failures on the [call detail page](/documentation/observability/traces). |

***

## Browse Integrations

### Voice agent platforms

> [card] [VAPI](/documentation/integrations/vapi)
>
> Sync assistants and stream calls in over webhooks

> [card] [Retell AI](/documentation/integrations/retell)
>
> Sync agents and analyze conversations

> [card] [ElevenLabs](/documentation/integrations/elevenlabs)
>
> Voice **and** chat, the only platform that does both

> [card] [Bland AI](/documentation/integrations/bland)
>
> Monitor Bland phone agents via pull-based sync

> [card] [Leaping](/documentation/integrations/leaping)
>
> Sync Leaping AI agents, prompts, and tools

### Real-time frameworks & self-hosted

> [card] [LiveKit Cloud](/documentation/integrations/livekit)
>
> Dashboard-managed webhook integration, plus WebRTC endpoints

> [card] [LiveKit (self-hosted)](/documentation/integrations/livekit-self-hosted)
>
> Drop `observe_session` into your own Agents worker

> [card] [Pipecat](/documentation/integrations/pipecat)
>
> Drop the Roark observer into your Pipecat pipeline, self-hosted or on Pipecat Cloud

### Chat agent platforms

> [card] [Google CES](/documentation/integrations/google-ces)
>
> Chat simulations against Customer Engagement Suite apps

> [card] [Kore AI](/documentation/integrations/kore)
>
> Chat simulations against Kore Agent Platform apps

> [card] [Custom / API](/documentation/integrations/custom-integrations)
>
> Any platform: send calls and chats with the API or SDKs

> [!NOTE]
>
> Don't see your platform? [Custom integrations](/documentation/integrations/custom-integrations) cover every capability in the matrix. Send an audio file or a message transcript and Roark handles the rest.

***

## Simulation Transports

Whether a simulation runs as a call or a chat is decided by the **endpoint type** on the agent target. You don't pick a modality separately.

| Transport | Endpoint type | Modality | Best for |
| :- | :- | :- | :- |
| **Telephony (PSTN)** | Phone | Voice | Any agent reachable at a phone number (the default for VAPI, Retell, ElevenLabs, Bland, and Leaping) |
| **LiveKit** | LiveKit | Voice | Agents running in a LiveKit room, cloud or self-hosted |
| **SmallWebRTC** | SmallWebRTC | Voice | Pipecat agents using peer-to-peer WebRTC with no media server |
| **WebSocket** | WebSocket | Chat | Custom text agents you expose yourself |
| **Provider chat** | ElevenLabs WS · Google CES · Kore | Chat | Auto-created when you connect the integration, nothing to wire up |

> [!NOTE]
>
> ElevenLabs WS endpoints resolve to **chat**, not voice. To test an ElevenLabs agent over audio, use a Phone endpoint instead.

***

## How Integrations Work

**1. Choose your platform**

Pick your platform from the cards above and open its setup guide.

**2. Authenticate**

Connect with an API key, OAuth, or (for self-hosted frameworks) a Roark API key dropped into your worker.

**3. Configure sync**

Select which agents to track and how far back to backfill historical calls.

**4. Start monitoring and testing**

Calls flow in and get scored automatically. Synced agents become targets you can point [run plans](/documentation/simulation-testing/run-plans) at.

Data reaches Roark one of three ways, depending on the platform: **webhooks** (VAPI, Retell, LiveKit Cloud), **scheduled pull** (ElevenLabs, Bland, Leaping), or a **drop-in SDK** you add to your own code (LiveKit self-hosted, Pipecat, custom).

***

## One Integration, Many Agents

A single integration syncs every agent in the connected account:

```
VAPI Integration
├── Customer Support Agent
├── Sales Qualification Agent
├── Appointment Booking Agent
└── Survey Collection Agent
```

* **Single setup**: configure once, sync all agents
* **Centralized management**: monitor every agent from one integration
* **Unified analytics**: compare performance across agents and across platforms
* **Simplified testing**: target any synced agent in a run plan

***

## Security & Privacy

* **Encrypted connections**: TLS on every data transfer
* **Secure storage**: credentials encrypted at rest and referenced by ID, never embedded in plans
* **Access control**: role-based permissions for integration management
* **Audit logging**: every integration activity is recorded
* **Data isolation**: customer data separated by organization
* **PII redaction**: strip sensitive data from transcripts and recordings with [PII redaction](/documentation/observability/pii-redaction)

For network-level controls, see [IP whitelisting](/documentation/integrations/ip-whitelisting).

***

## Next Steps

> [card] [Generate an API key](/documentation/getting-started/api-keys)
>
> Every integration path starts here

> [card] [Set up collectors](/documentation/metrics/metric-collectors)
>
> Score every imported call automatically

> [card] [Build a run plan](/documentation/simulation-testing/run-plans)
>
> Turn a synced agent into a test suite

> [card] [Webhooks](/documentation/integrations/webhooks)
>
> Push Roark events into your own systems


# Pipecat
Source: https://docs.roark.ai/documentation/integrations/pipecat

Instrument a Pipecat voice agent with the roark_analytics[pipecat] observer, self-hosted or on Pipecat Cloud

Live monitoringVoice simulationsChat simulations

## Overview

The Pipecat integration monitors voice agents built on the open-source [Pipecat](https://github.com/pipecat-ai/pipecat) framework. Drop the [`roark_analytics[pipecat]`](https://pypi.org/project/roark-analytics/) observer into your pipeline and call lifecycle, transcripts, tool calls, and a recording are forwarded to Roark automatically.

The same observer works whether you run Pipecat yourself or on **Pipecat Cloud**: the wiring is identical, so you pick your deployment when you connect the integration and instrument your pipeline once. The observer talks only to Roark's own API; it never changes how your agent runs.

> [!NOTE]
>
> Deployment only matters for **simulations**. Monitoring (importing your production calls) is identical for self-hosted and Pipecat Cloud. If you only want monitoring, you can skip the simulation configuration below.

***

## Prerequisites

Before setting up the integration, ensure you have:

* A Pipecat voice agent (running on your own infrastructure or on Pipecat Cloud).
* Python 3.10 or newer.
* Access to your Roark project's **Agents** page to connect the integration and mint an API key.

***

## Setup Instructions

### Step 1: Connect the integration in Roark

Go to **Agents**, click **Connect Agent**, choose **Existing Platform**, and pick **Pipecat**. Select the **Self-hosted** or **Pipecat Cloud** tab, give the integration a name, and (optionally) fill in the simulation configuration for that deployment (see [Simulations](#simulations) below).

On connect, Roark mints a project API key **bound to this integration** and shows it once. Copy it now: it's what the observer authenticates with, and Roark can't show it again.

### Step 2: Install the observer

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
pip install "roark_analytics[pipecat]"
```

### Step 3: Wire the observer into your pipeline

Add `RoarkObserver` to your Pipecat pipeline. Two rules:

* Put `roark.audio_processor` **after** `transport.output()` so the observer captures the bot's post-TTS audio.
* Pass `observers=[roark]` on `PipelineParams`.

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat_roark import RoarkObserver

SYSTEM_PROMPT = "You are a friendly voice assistant. Keep replies short."

async def bot(runner_args):
    roark = RoarkObserver(
        api_key="<YOUR_ROARK_API_KEY>",
        agent_id="pipecat-demo",
        agent_name="Pipecat Demo",
        agent_prompt=SYSTEM_PROMPT,
        runner_args=runner_args,
    )

    # roark.audio_processor must sit AFTER transport.output() so the bot channel
    # captures post-TTS audio. The observer is wired via PipelineParams.observers.
    pipeline = Pipeline(
        [
            transport.input(),
            stt,
            context_aggregator.user(),
            llm,
            tts,
            transport.output(),
            roark.audio_processor,
            context_aggregator.assistant(),
        ]
    )

    task = PipelineTask(
        pipeline,
        params=PipelineParams(
            enable_metrics=True,
            enable_usage_metrics=True,
            observers=[roark],
        ),
    )

    runner = PipelineRunner(handle_sigint=runner_args.handle_sigint)
    await runner.run(task)
```

The observer authenticates with the key from Step 1, sent as the `x-roark-api-key` header. Rather than hard-coding it, read it from the environment (`api_key=os.environ["ROARK_API_KEY"]`). On Pipecat Cloud, set `ROARK_API_KEY` as a deployment secret.

> [!NOTE]
>
> `agent_id` and `agent_name` identify the agent in Roark. Agents are registered automatically on the first observed call, and `agent_prompt` is tracked as a prompt revision, so prompt changes show up in Roark over time.

### Step 4: Verify the connection

Place a call to your agent. Within a few moments it appears in [Call History](/documentation/observability/live-monitoring) with its transcript, recording, and any tool calls, then gets scored by your active [collectors](/documentation/metrics/metric-collectors).

***

## How It Works

The observer POSTs call lifecycle events (`call-started`, `call-ended`) to Roark's API, authenticated by your integration-bound key. Audio is streamed as chunks to Roark-issued presigned URLs and assembled into a recording when the call ends. Transcript and tool calls are sent in Pipecat's native format and mapped to Roark's model server-side.

Because the key is bound to the integration, every call is attributed to the right integration and agent with no extra configuration.

## What Gets Synced

* **Calls**: lifecycle, timing, and metadata for each conversation.
* **Recording**: the mixed audio, assembled from the streamed chunks.
* **Transcript**: turn-by-turn, from Pipecat's context messages.
* **Tool calls**: function invocations captured during the call.
* **Agent + prompt**: lazily registered from `agent_id` / `agent_name`, with `agent_prompt` tracked as a revision.

***

## Simulations

To run [simulations](/documentation/simulation-testing/overview) against a Pipecat agent, Roark joins it over [WebRTC](/documentation/simulation-testing/webrtc). Add the simulation configuration when you connect the integration (or edit it later). This is the one part that differs by deployment.

> [tab] Self-hosted (SmallWebRTC)
>
> Roark reaches your agent through its **SmallWebRTC** signaling endpoint.
>
> | Field | Description |
> | :- | :- |
> | **Signaling URL** | Your SmallWebRTC offer endpoint. Roark POSTs an SDP offer here (`http(s)` or `ws(s)`). |
> | **Auth token** | Optional bearer token for the signaling server. Stored encrypted. |
> | **Append agent ID** | Append each synced agent's id as a URL path segment, so one host can route to many bots. On by default. |
>
> Roark creates a WebRTC simulation endpoint for each synced agent automatically.

> [tab] Pipecat Cloud (Daily)
>
> Roark starts a session through **Pipecat Cloud**, which runs your agent over Daily.
>
> | Field | Description |
> | :- | :- |
> | **Public API key** | Your Pipecat Cloud **public** API key, used to start sessions. Stored encrypted. |
> | **Start body** | Optional JSON forwarded as the agent `body` on the Pipecat Cloud `/start` request. |
> | **Room properties** | Optional JSON forwarded as the Daily `dailyRoomProperties`. |
>
> The Pipecat Cloud start URL is derived from each synced agent's name (its Pipecat Cloud deployment name), so there's nothing else to configure per agent.

***

## Next Steps

> [card] [Run a simulation](/documentation/simulation-testing/overview)
>
> Test your Pipecat agent with synthetic callers over WebRTC

> [card] [Set up collectors](/documentation/metrics/metric-collectors)
>
> Score every incoming call automatically


# Retell AI
Source: https://docs.roark.ai/documentation/integrations/retell

Sync Retell agents and analyze voice AI conversations

Live monitoringVoice simulationsChat simulations

## Overview

The Retell AI integration enables comprehensive monitoring of your Retell voice agents. Sync historical calls, monitor live conversations, and automatically evaluate agent performance using Roark's analytics platform.

***

## Prerequisites

Before setting up the integration, ensure you have:

* A Retell AI account with active agents
* Retell API key (found in your Retell dashboard)
* At least one configured agent in Retell

***

## Setup Instructions

### Step 1: Create Integration

1. Navigate to **Agents** in your Roark dashboard
2. Click **Connect Agent** in the top right, choose **Existing Platform**, and select **Retell AI**
3. Enter your configuration:

| Field | Description |
| :- | :- |
| **Integration Name** | Friendly name for this integration |
| **API Key** | Your Retell API key |

The system will automatically validate your API key and fetch available agents.

### Step 2: Configure Historical Sync

Choose how far back to import existing calls:

* **Default**: Last 90 days of call history
* **Custom**: Select a specific date range
* **Skip**: Only sync new calls going forward

> [!NOTE]
>
> Historical sync runs once during initial setup. Future calls are synced automatically via webhooks.

### Step 3: Select Agents

Choose which Retell agents to monitor:

* **All Agents** - Sync calls from every agent in your account
* **Selected Agents** - Choose specific agents to monitor

The agent selector shows:

* Agent name
* Agent ID
* Last modified date

### Step 4: Configure Metric Collectors

Calls synced from Retell are scored by your [metric collectors](/documentation/metrics/metric-collectors). Set them up globally under **Metrics → Collectors**, or scope collectors to a specific agent from that agent's **Collectors** section.

### Step 5: Activate Integration

Review your settings and click **Create Integration** to begin syncing.

***

## What Gets Synced

Retell integrations sync comprehensive data:

* **Calls** - Complete conversation data and metadata
* **Agents** - Agent configurations and phone numbers
* **Prompts** - System prompts and instructions
* **Transcripts** - Full conversation transcripts with speaker labels
* **Tool Calls** - Function/tool invocations during conversations
* **Call Metadata** - Duration, status, participant information

***

## Webhook Configuration

Roark automatically configures webhooks with Retell. The integration handles:

| Event | Description | Roark Action |
| :- | :- | :- |
| `call_started` | Call initiated | Create call record, begin monitoring |
| `call_ended` | Call completed | Process transcript, run evaluations |

The webhook URL is automatically registered with Retell using your API key.

***

## Sync Management

### Initial Sync

During setup, Roark imports historical calls based on your selected date range:

* Processes calls in batches
* Shows sync progress in dashboard
* Handles large volumes efficiently

### Ongoing Sync

After initial setup:

* New calls sync automatically via webhooks
* Real-time processing of live calls
* Automatic retry on temporary failures

### Resume Sync

If sync is interrupted:

1. Go to integration settings
2. Click **Resume Sync**
3. Choose to continue from last position or restart

***

## Monitoring Integration Health

Track your integration status:

* **Syncing** 🔄 - Actively importing historical calls
* **Active** 🟢 - Receiving new calls via webhook
* **Paused** ⏸️ - Sync temporarily halted
* **Error** ⚠️ - Configuration or connection issues

View sync statistics:

* Total calls synced
* Last sync timestamp
* Sync progress percentage
* Recent webhook deliveries

***

## Agent Management

Synced agents appear in:

* **Simulation agent selection** - Use for testing
* **Agent performance reports** - Track metrics per agent
* **Comparison dashboards** - Analyze across agents

Agent data updates automatically when modified in Retell.

***

## Next Steps

> [card] [Run Simulations](/documentation/simulation-testing/running-simulations)
>
> Test synced agents

> [card] [View Analytics](/documentation/observability/reports)
>
> Analyze call performance

> [card] [Configure Metrics](/documentation/metrics/overview)
>
> Define metrics and pass/fail thresholds

> [card] [Integration Overview](/documentation/integrations/overview)
>
> Explore other integrations


# VAPI
Source: https://docs.roark.ai/documentation/integrations/vapi

Sync VAPI assistants and analyze voice AI conversations

Live monitoringVoice simulationsChat simulations

## Overview

The VAPI integration enables comprehensive monitoring of your VAPI voice assistants. Sync historical calls, monitor live conversations, and automatically evaluate assistant performance using Roark's analytics platform.

***

## Prerequisites

Before setting up the integration, ensure you have:

* A VAPI account with active assistants
* VAPI API key (found in your VAPI dashboard)
* At least one configured assistant in VAPI

***

## Setup Instructions

### Step 1: Create Integration

1. Navigate to **Agents** in your Roark dashboard
2. Click **Connect Agent** in the top right, choose **Existing Platform**, and select **VAPI**
3. Enter your configuration:

| Field | Description |
| :- | :- |
| **Integration Name** | Friendly name for this integration |
| **API Key** | Your VAPI API key |

The system will automatically validate your API key and fetch available assistants.

### Step 2: Configure Historical Sync

Choose how far back to import existing calls:

* **Default**: Last 90 days of call history
* **Custom**: Select a specific date range
* **Skip**: Only sync new calls going forward

> [!NOTE]
>
> Historical sync runs once during initial setup. Future calls are synced automatically via webhooks.

### Step 3: Select Assistants

Choose which VAPI assistants to monitor:

* **All Assistants** - Sync calls from every assistant in your account
* **Selected Assistants** - Choose specific assistants to monitor

The assistant selector shows:

* Assistant name
* Assistant ID
* Last modified date

### Step 4: Configure Metric Collectors

Calls synced from VAPI are scored by your [metric collectors](/documentation/metrics/metric-collectors). Set them up globally under **Metrics → Collectors**, or scope collectors to a specific agent from that agent's **Collectors** section.

### Step 5: Activate Integration

Review your settings and click **Create Integration** to begin syncing.

***

## What Gets Synced

VAPI integrations sync comprehensive data:

* **Calls** - Complete conversation data and metadata
* **Assistants** - Assistant configurations and phone numbers
* **Prompts** - System prompts and instructions
* **Transcripts** - Full conversation transcripts with speaker labels
* **Tool Calls** - Function/tool invocations during conversations
* **Call Metadata** - Duration, status, participant information

***

## Webhook Configuration

Roark automatically configures webhooks with VAPI. The integration handles:

| Event | Description | Roark Action |
| :- | :- | :- |
| `call_started` | Call initiated | Create call record, begin monitoring |
| `call_ended` | Call completed | Process transcript, run evaluations |

The webhook URL is automatically registered with VAPI using your API key.

***

## Sync Management

### Initial Sync

During setup, Roark imports historical calls based on your selected date range:

* Processes calls in batches
* Shows sync progress in dashboard
* Handles large volumes efficiently

### Ongoing Sync

After initial setup:

* New calls sync automatically via webhooks
* Real-time processing of live calls
* Automatic retry on temporary failures

### Resume Sync

If sync is interrupted:

1. Go to integration settings
2. Click **Resume Sync**
3. Choose to continue from last position or restart

***

## Monitoring Integration Health

Track your integration status:

* **Syncing** 🔄 - Actively importing historical calls
* **Active** 🟢 - Receiving new calls via webhook
* **Paused** ⏸️ - Sync temporarily halted
* **Error** ⚠️ - Configuration or connection issues

View sync statistics:

* Total calls synced
* Last sync timestamp
* Sync progress percentage
* Recent webhook deliveries

***

## Assistant Management

Synced assistants appear in:

* **Simulation assistant selection** - Use for testing
* **Assistant performance reports** - Track metrics per assistant
* **Comparison dashboards** - Analyze across assistants

Assistant data updates automatically when modified in VAPI.

***

## Next Steps

> [card] [Enable Tracing](/documentation/observability/traces?tab=vapi)
>
> Traces sync automatically. See how to enable them

> [card] [Run Simulations](/documentation/simulation-testing/running-simulations)
>
> Test synced assistants

> [card] [View Analytics](/documentation/observability/reports)
>
> Analyze call performance

> [card] [Integration Overview](/documentation/integrations/overview)
>
> Explore other integrations


# Webhooks
Source: https://docs.roark.ai/documentation/integrations/webhooks

Receive real-time notifications when call analysis completes or fails

## Overview

Webhooks allow you to receive real-time HTTP notifications when events occur in Roark. Instead of polling for updates, Roark will automatically send event data to your specified endpoint whenever a call analysis completes or fails.

[https://www.tella.tv/video/webhooks-demo-1-9rur/embed](https://www.tella.tv/video/webhooks-demo-1-9rur/embed)

***

## Webhook Events

Roark currently supports the following webhook events:

### `call.analysis.completed`

Triggered when a call analysis has successfully completed processing.

**Event Payload:**

* `callId` - Unique identifier for the call
* `callAnalysisJobId` - ID of the completed analysis job
* `projectId` - ID of the project containing the call

### `call.analysis.failed`

Triggered when a call analysis fails to complete.

**Event Payload:**

* `callId` - Unique identifier for the call
* `callAnalysisJobId` - ID of the failed analysis job
* `projectId` - ID of the project containing the call
* `errorMessage` - Description of the failure (optional)

***

## Setting Up Webhooks

**1. Navigate to Webhooks**

Go to your workspace settings and select the Webhooks section

**2. Add Endpoint**

Enter the URL where you want to receive webhook events

**3. Select Events**

Choose which events you want to subscribe to:

* Call Analysis Completed
* Call Analysis Failed

**4. Test Your Webhook**

Use the "Send Test Event" button to verify your endpoint is working correctly

***

## Testing Webhooks

You can send test events to verify your webhook integration is working properly:

1. **Send Test Event** - Click the "Send Test Event" button next to your webhook endpoint
2. **Verify Receipt** - Check that your endpoint received the test payload
3. **Validate Handling** - Ensure your application processes the test event correctly

Test events contain sample data that matches the structure of real webhook events, allowing you to validate your integration before going live.

***

## Webhook Security

### Signature Verification

All webhook requests from Roark include a signature header that you can use to verify the request authenticity:

```
X-Roark-Signature: <signature>
```

> [!NOTE]
>
> Always validate webhook signatures in production to ensure requests are coming from Roark and haven't been tampered with.

### Best Practices

* **Use HTTPS** - Only accept webhooks over secure HTTPS connections
* **Verify Signatures** - Always validate the webhook signature
* **Respond Quickly** - Return a 200 OK response within 5 seconds
* **Process Async** - Handle webhook processing in background jobs
* **Implement Retries** - Handle temporary failures gracefully

***

## Webhook Delivery

### Retry Logic

If your endpoint fails to respond with a 2xx status code, Roark will retry the webhook delivery:

* **Retry Attempts**: Up to 3 retries
* **Retry Schedule**: Exponential backoff (1 min, 10 min, 1 hour)
* **Timeout**: 5 seconds per request

### Response Requirements

Your endpoint should:

* Return a `200 OK` status code to acknowledge receipt
* Respond within 5 seconds
* Process the webhook payload asynchronously if needed

***

## Event Payload Structure

All webhook events follow a consistent structure with an `event` name, API `version`, `timestamp`, and event-specific `data`.

### Call Analysis Completed

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "event": "call.analysis.completed",
  "version": "1.0",
  "timestamp": "2025-11-21T10:30:00Z",
  "data": {
    "callId": "call_abc123",
    "callAnalysisJobId": "job_xyz789",
    "projectId": "proj_def456"
  }
}
```

### Call Analysis Failed

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "event": "call.analysis.failed",
  "version": "1.0",
  "timestamp": "2025-11-21T10:30:00Z",
  "data": {
    "callId": "call_abc123",
    "callAnalysisJobId": "job_xyz789",
    "projectId": "proj_def456",
    "errorMessage": "Analysis timeout exceeded"
  }
}
```

***

## Use Cases

### Real-time Notifications

Send alerts to your team when critical calls fail analysis:

```javascript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
app.post('/webhooks/roark', (req, res) => {
  const { event, data } = req.body;

  if (event === 'call.analysis.failed') {
    sendSlackAlert(
      `Call ${data.callId} analysis failed: ${data.errorMessage || 'Unknown error'}`
    );
  }

  res.sendStatus(200);
});
```

### Data Synchronization

Keep your internal systems in sync with Roark analysis results:

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
@app.route('/webhooks/roark', methods=['POST'])
def handle_webhook():
    payload = request.json

    if payload['event'] == 'call.analysis.completed':
        # Fetch full call details and update your database
        call_id = payload['data']['callId']
        update_call_analytics(call_id=call_id)

    return '', 200
```

### Workflow Automation

Trigger downstream processes when analysis completes:

```javascript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
async function handleWebhook(event) {
  if (event.event === 'call.analysis.completed') {
    const { callId, projectId } = event.data;

    // Trigger follow-up actions
    await updateCRM(callId, projectId);
    await sendCustomerSurvey(callId);
    await notifyQATeam(callId);
  }
}
```

***

## Troubleshooting

### Webhook Not Received

* **Check URL** - Ensure your endpoint URL is correct and accessible
* **Verify HTTPS** - Confirm you're using HTTPS (HTTP not supported)
* **Check Firewall** - Ensure your firewall allows incoming requests from Roark
* **Review Logs** - Check your server logs for incoming requests

### Delivery Failures

* **Response Time** - Ensure your endpoint responds within 5 seconds
* **Status Codes** - Return 2xx status codes to acknowledge receipt
* **Error Handling** - Check your endpoint for exceptions or crashes

### Missing Events

* **Event Subscriptions** - Verify you're subscribed to the correct events
* **Endpoint Status** - Check if your webhook endpoint is marked as active
* **Delivery History** - Review the webhook delivery logs in your Roark dashboard

***

## Next Steps

> [card] [API Reference](/api-reference/introduction)
>
> Explore the Roark API

> [card] [Custom Integrations](/documentation/integrations/custom-integrations)
>
> Build custom integrations

> [card] [Live Monitoring](/documentation/observability/live-monitoring)
>
> Monitor calls in real-time

> [card] [Support](/documentation/resources/support)
>
> Get help with webhooks


# Custom Metrics
Source: https://docs.roark.ai/documentation/metrics/custom-metrics

Measure what matters to your business with LLM judges, patterns, formulas, and API checks

Metric definitions describe what to measure and how. Roark comes with [built-in system metrics](/documentation/metrics/system-metrics) that work out of the box, and you can create your own custom metrics tailored to your business needs.

![The Metrics page: a Collectors strip across the top and the metric Library grouped by package, with All / System / Custom filters](https://mintcdn.com/roark/93k2kGpZVUUbTXSq/images/metrics/metrics-library.png?fit=max\&auto=format\&n=93k2kGpZVUUbTXSq\&q=85\&s=7914a990338bfac7a9d21a8fb83ef968)

***

## Choose an Engine

Custom metrics let you measure anything specific to your use case: task completion, compliance checks, quality scoring, or business KPIs. Every custom metric runs on one of four engines. Pick the engine by where the answer lives:

| Engine | The answer comes from | Returns | Example |
| :- | :- | :- | :- |
| [LLM Judge](/documentation/metrics/custom-metrics/llm-judge) | The conversation, read by Roark's evaluation model against your prompt | Boolean, Scale, Classification, Numeric, Count, or Text | Did the agent verify the caller's identity? |
| [Pattern](/documentation/metrics/custom-metrics/patterns) | Per-line metrics, a trigger followed by an outcome within a window | Boolean or Count | Did the agent talk over a frustrated caller? |
| [Formula](/documentation/metrics/custom-metrics/formulas) | Other metrics on the same call, combined with math and comparisons | Numeric or Boolean | Task completed AND instructions followed |
| [API Check](/documentation/metrics/custom-metrics/api-checks) | Your own backend, asked through an HTTP request | Boolean | Did the PIN actually change? |

> [card] [LLM Judge](/documentation/metrics/custom-metrics/llm-judge)
>
> Describe what to measure in plain language

> [card] [Patterns](/documentation/metrics/custom-metrics/patterns)
>
> Detect a trigger and what follows it

> [card] [Formulas](/documentation/metrics/custom-metrics/formulas)
>
> Compute a metric from other metrics

> [card] [API Checks](/documentation/metrics/custom-metrics/api-checks)
>
> Ask your own endpoint whether the call passed

Patterns, formulas, and API checks don't use Roark's evaluation models, so they aren't billed like LLM Judge metrics.

### Configuration Steps

**1. Define the Metric in Studio**

Open [Studio](/documentation/metrics/studio) in Author mode. Name your metric and pick an engine (locked after the first save). Then fill in what that engine needs: the prompt and output type for an LLM Judge, the triggers and outcome for a Pattern, the expression for a Formula, or the request for an API check. LLM Judge metrics and API checks can also be defined [in code](/documentation/config-as-code/metrics).

![Studio Author mode: the numbered Engine, Output (with range labels), Instructions, and Group sections, plus a Tests rail of scored test calls](https://mintcdn.com/roark/IjseSxRBSMbit4Aq/images/metrics/studio-author.png?fit=max\&auto=format\&n=IjseSxRBSMbit4Aq\&q=85\&s=b70e3d6a98d20fe838b6fbd6844696ba)

**2. Test Against Real Calls**

Use the test rail in Studio to run your metric against representative calls and validate it produces the results you expect. Iterate until you're satisfied.

**3. Add to a Collector or Run Plan**

Attach the metric to a [collector](/documentation/metrics/metric-collectors) for automated collection on incoming calls, or to a [simulation run plan](/documentation/simulation-testing/run-plans) for testing.

***

## SDK Reference

> [!NOTE]
>
> All API endpoints require authentication. [Generate an API key](/documentation/getting-started/api-keys) to get started.

### List Metric Definitions

Retrieve the first page of metric definitions available for your project. Each
page contains up to 500 definitions. If `pagination.hasMore` is true, use
`pagination.nextCursor` as the `after` query parameter on `GET
/v1/metric/definitions` to retrieve the next page. Results are ordered by
immutable definition ID. Treat cursors as opaque.

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const definitions = await client.metric.listDefinitions()

// definitions.data[0]
{
  id: 'uuid',
  metricId: 'response_time',
  name: 'Response Time',
  description: 'Time taken to respond to a question',
  type: 'OFFSET',
  scope: 'PER_PARTICIPANT',
  supportedContexts: ['SEGMENT_RANGE', 'CALL'],
  unit: { name: 'milliseconds', symbol: 'ms' },
}
```

### Get Call Metrics

Retrieve all metrics for a specific call:

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const metrics = await client.call.listMetrics('call-id')

// Or flatten to get a simple list instead of grouped by definition
const flat = await client.call.listMetrics('call-id', { flatten: 'true' })
```

The response groups metrics by definition, with each metric containing an array of values:

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
// metrics.data[0]
{
  metricDefinitionId: 'uuid',
  metricId: 'response_time',
  name: 'Response Time',
  type: 'OFFSET',
  scope: 'PER_PARTICIPANT',
  unit: { name: 'milliseconds', symbol: 'ms' },
  values: [
    {
      value: 2500,
      context: 'SEGMENT_RANGE',
      participantRole: 'agent',
      confidence: 1.0,
      computedAt: '2024-01-15T10:30:00Z',
      fromSegment: { id: 'uuid', text: 'How can I help you today?', startOffsetMs: 1000, endOffsetMs: 2000 },
      toSegment: { id: 'uuid', text: 'I have a question about my bill', startOffsetMs: 4500, endOffsetMs: 6000 },
    },
  ],
}
```

### Understanding Metric Values

**Confidence Scores:**

* All metrics include a `confidence` field (0-1)
* Deterministic metrics (like word count, duration) have confidence = 1.0
* AI-powered metrics include the model's confidence level

**Value Reasoning:**

* For AI-computed metrics, the `valueReasoning` field provides explanation
* Useful for understanding why a metric was scored a certain way
* Example: "The agent verified identity using two-factor authentication as mentioned in segment 3"

**Segment Context:**

* When `context` is `SEGMENT`, the `segment` field contains the specific utterance
* When `context` is `SEGMENT_RANGE`, both `fromSegment` and `toSegment` are included
* All segment objects include the full text and timing information

***

## Best Practices

> [accordion] Start with System Metrics
>
> Use the [built-in system metrics](/documentation/metrics/system-metrics) first. They cover performance, sentiment, interruptions, compliance, and more with no setup. Add custom metrics for business-specific needs.

> [accordion] Test Before Deploying
>
> Always validate custom metrics in [Studio](/documentation/metrics/studio) on representative calls before attaching them to collectors.

> [accordion] Use Formulas for Composite Scores
>
> Instead of creating one complex LLM prompt that tries to measure everything, break it into focused metrics and combine them with a [formula](/documentation/metrics/custom-metrics/formulas).

> [accordion] Balance Coverage
>
> Track both agent and customer metrics for complete conversation understanding.

***

## What's Next

> [card] [System Metrics Reference](/documentation/metrics/system-metrics)
>
> Browse all 65+ built-in metrics powered by specialized models

> [card] [Studio](/documentation/metrics/studio)
>
> Author and test metrics interactively before deploying

> [card] [Thresholds](/documentation/metrics/thresholds)
>
> Define pass/fail criteria for your metrics

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Automate metric collection with conditions-based rules


# API Checks
Source: https://docs.roark.ai/documentation/metrics/custom-metrics/api-checks

Ask your own backend whether a call did what it should

An API check is a pass/fail metric your own endpoint answers. After each call is analysed, Roark sends a request about the call to your API, and your API replies with `{ "passed": true | false, "reason": "..." }`. It runs on live calls and simulations alike.

Every other metric reads the conversation. An API check reads your system of record instead. An agent that says "your PIN has been changed" while the tool call behind it failed passes every transcript judge, but fails an API check that asks your card service whether the PIN actually changed.

Use an API check when the outcome lives in your backend:

* **State changes:** the PIN, address, or plan was actually updated.
* **Records created:** a booking, ticket, refund, or callback exists, with the right details.
* **Side effects:** an SMS or email went out, a payment was captured, a case was escalated.

***

## Quickstart

Define the API check [in code](/documentation/config-as-code/metrics#api-checks) (recommended), or build it in Studio.

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineApiCheck, defineConfig } from '@roarkanalytics/sdk/lib/config';

const pinChanged = defineApiCheck({
  name: 'pin_changed', // the slug a collector references
  displayName: 'PIN changed',
  httpRequest: {
    url: 'https://api.example.com/roark/checks/pin-changed',
    method: 'POST',
    headers: { Authorization: `Bearer ${process.env.CARD_SERVICE_TOKEN}` },
    body: '{"cardId": "{{call.properties.cardId}}", "expectedPin": "{{simulation.variables.newPin}}"}',
  },
  trueLabel: 'PIN changed',
  falseLabel: 'PIN unchanged',
});

export default defineConfig({
  resources: [pinChanged],
});
```

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: metric
calculationType: HTTP_REQUEST
name: pin_changed
displayName: PIN changed
httpRequest:
  url: https://api.example.com/roark/checks/pin-changed
  method: POST
  headers:
    Authorization: 'Bearer YOUR_TOKEN'
  body: '{"cardId": "{{call.properties.cardId}}", "expectedPin": "{{simulation.variables.newPin}}"}'
trueLabel: PIN changed
falseLabel: PIN unchanged
```

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply roark.config.ts
```

Then list `pin_changed` in a [collector](/documentation/config-as-code/collectors), in the same config, to score it on every matching call.

> [!NOTE]
>
> Headers are stored as you write them. In TypeScript, read your endpoint's token from the environment, as above, rather than committing it.

### Create it with the SDK

Create the metric with `calculationType: "HTTP_REQUEST"`:

```typescript Node theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import Roark from '@roarkanalytics/sdk'

const client = new Roark({ bearerToken: process.env.ROARK_API_BEARER_TOKEN })

const metric = await client.metric.createDefinition({
  calculationType: 'HTTP_REQUEST',
  name: 'PIN changed',
  slug: 'pin_changed',
  httpRequest: {
    url: 'https://api.example.com/roark/checks/pin-changed',
    method: 'POST',
    headers: { Authorization: `Bearer ${process.env.CARD_SERVICE_TOKEN}` },
    body: '{"cardId": "{{call.properties.cardId}}", "expectedPin": "{{simulation.variables.newPin}}"}',
  },
  booleanTrueLabel: 'PIN changed',
  booleanFalseLabel: 'PIN unchanged',
})
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import os
from roark_analytics import Roark

client = Roark(bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"))

metric = client.metric.create_definition(
    calculation_type="HTTP_REQUEST",
    name="PIN changed",
    slug="pin_changed",
    http_request={
        "url": "https://api.example.com/roark/checks/pin-changed",
        "method": "POST",
        "headers": {"Authorization": f"Bearer {os.environ['CARD_SERVICE_TOKEN']}"},
        "body": '{"cardId": "{{call.properties.cardId}}", "expectedPin": "{{simulation.variables.newPin}}"}',
    },
    boolean_true_label="PIN changed",
    boolean_false_label="PIN unchanged",
)
```

```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/metric/definitions \
  -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "calculationType": "HTTP_REQUEST",
    "name": "PIN changed",
    "slug": "pin_changed",
    "httpRequest": {
      "url": "https://api.example.com/roark/checks/pin-changed",
      "method": "POST",
      "headers": { "Authorization": "Bearer sk_live_..." },
      "body": "{\"cardId\": \"{{call.properties.cardId}}\", \"expectedPin\": \"{{simulation.variables.newPin}}\"}"
    },
    "booleanTrueLabel": "PIN changed",
    "booleanFalseLabel": "PIN unchanged"
  }'
```

The response is the metric definition, with `type: "BOOLEAN"` and your `httpRequest` echoed back.

| Field | Type | Required | Description |
| :- | :- | :- | :- |
| `httpRequest.url` | string | Yes | Public `http(s)` URL, up to 2048 characters. May contain placeholders, which are URL-encoded |
| `httpRequest.method` | string | No | `POST` (default) or `GET`. A `GET` request has no body |
| `httpRequest.headers` | object | No | Up to 20 headers. Values may contain placeholders |
| `httpRequest.body` | string | No | A JSON template, up to 20,000 characters. Omit it, or pass `null`, to send the default payload |
| `booleanTrueLabel` / `booleanFalseLabel` | string | No | What a passed and a failed check are called. Default `Pass` and `Fail` |
| `slug` | string | No | Stable identifier. Generated from the name if omitted |
| `analysisPackageId` | string | No | Package to add the metric to. Defaults to your project's Custom Metrics package |

To change the request, `PUT /v1/metric/definitions/{idOrSlug}` with the whole `httpRequest`. Headers you leave out are removed, and leaving out `body` switches back to the default payload. Each change creates a new metric version, so earlier results keep the request they ran with.

The API answers `400` when:

* the URL is not public `http(s)` (localhost, private, or internal addresses are refused)
* a placeholder doesn't exist in the payload
* the body isn't valid JSON once its placeholders are filled in
* a `GET` request has a body, or there are more than 20 headers

### Build it in Studio

Open [Studio](/documentation/metrics/studio) in Author mode and pick the **API check** engine. The engine is locked after the first save.

![Studio Author mode with the API check engine selected: a POST request to https://api.example.com/roark/checks/pin-changed with a Test button beside the URL, a passed test result labelled PIN changed with HTTP 200 and the endpoint's reason, an Authorization header, a custom JSON body using \{\{call.properties.cardId}}, \{\{simulation.variables.newPin}} and \{\{call.id}} placeholders, and the Pass and Fail labels](https://mintcdn.com/roark/s80NtcBbMfjE9Dti/images/metrics/api-check.png?fit=max\&auto=format\&n=s80NtcBbMfjE9Dti\&q=85\&s=299ccb3084e2f92db3f3199cfcb84013)

**1. Request**

Choose `POST` or `GET`, enter your endpoint URL, and add any headers your API needs, such as an `Authorization` header. Type `{{` in the URL, a header, or the body to insert a [placeholder](#placeholders).

**2. Body**

Leave the body as it is to send the default payload, or edit it into the JSON your endpoint expects. Turn on **Preview sent body** to see the body with its placeholders filled in for an example simulated call.

**3. Test**

Click **Test** next to the URL to send the request once, filled from the example payload. Studio shows the result (Pass, Fail, Error, or Not sent), the HTTP status, your reason, the body it sent, and your endpoint's response. Nothing is saved, so you can test before the metric exists.

**4. Labels**

Name the two outcomes, for example **PIN changed** and **PIN unchanged**. They default to **Pass** and **Fail**.

> [!TIP]
>
> The test fills run variables, call properties, and persona fields the example call doesn't have with `test-<name>`, so a placeholder like `{{simulation.variables.newPin}}` sends `test-newPin`. Have your endpoint accept those values while you set it up, or check the sent body to confirm each value lands where you expect.

***

## How It Works

**1. The call ends and is analysed**

API checks run with the rest of the call's metrics, from a [collector](/documentation/metrics/metric-collectors) on live calls or from a [run plan](/documentation/simulation-testing/run-plans) on simulations.

**2. Roark sends your request**

Roark fills your URL, headers, and body with data about the call and sends it. With no body, Roark POSTs the [default payload](#the-request-payload).

**3. Your endpoint answers**

Your endpoint replies `200` with `{ "passed": true | false, "reason": "..." }`. Roark records the result as Pass or Fail, with your reason shown next to it.

API checks are free: they don't use Roark's evaluation models, so they aren't billed like LLM Judge metrics.

***

## The Request Payload

With no body, Roark POSTs this payload as JSON. With a custom body, the same payload is what your [placeholders](#placeholders) read from.

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "version": "1.0",
  "requestId": "3eb16131-4444-4fb1-8d84-6c4c6fce5043",
  "metric": { "id": "0b8e3e0e-1111-4c8e-9a51-3f1f3c9b2d10", "name": "PIN changed" },
  "call": {
    "id": "6f1d2c3b-5555-4a6e-8f7a-1b2c3d4e5f60",
    "projectId": "7a2e3d4c-6666-4b7f-9a8b-2c3d4e5f6a71",
    "callDirection": "INBOUND",
    "startedAt": "2026-10-26T14:00:02.000Z",
    "endedAt": "2026-10-26T14:03:06.000Z",
    "durationMs": 184000,
    "endedStatus": "CUSTOMER_ENDED_CALL",
    "externalId": "your-call-id",
    "agents": [
      {
        "id": "8b3f4e5d-7777-4c8a-ab9c-3d4e5f6a7b82",
        "endpoint": { "id": "9c4a5f6e-8888-4d9b-bcad-4e5f6a7b8c93", "environment": "production", "phoneNumberE164": "+14155550100" }
      }
    ],
    "customers": [{ "phoneNumberE164": "+14155550199", "label": null }],
    "properties": { "cardId": "card_123" }
  },
  "simulation": {
    "jobId": "ad5b6a7f-9999-4eac-8dbe-5f6a7b8c9da4",
    "runPlanJobId": "be6c7b8a-aaaa-4fbd-9ecf-6a7b8c9daeb5",
    "flow": { "id": "cf7d8c9b-bbbb-4ace-afd0-7b8c9daebfc6", "title": "Change my PIN" },
    "testCase": { "id": "d08e9dac-cccc-4bdf-b0e1-8c9daebfc0d7", "title": "Happy path", "isDefault": true },
    "variables": { "cardNumber": "4111111111111111", "newPin": "4821" },
    "persona": { "name": "Maria" }
  }
}
```

| Field | Description |
| :- | :- |
| `requestId` | Unique per check, also sent as the `X-Roark-Request-Id` header. Retries reuse it, so you can deduplicate on it |
| `metric` | The metric being checked |
| `call` | The call, with the same field names as the [Calls API](/api-reference/introduction). `externalId` is your own id if you sent one, and `properties` holds the call's [custom properties](/documentation/integrations/custom-integrations) |
| `simulation` | Set when the call is a Roark simulation, `null` on a live call. `variables` holds the run's [variables](/documentation/simulation-testing/variables) and `persona` the persona the simulated caller played |

Every request also carries `User-Agent: Roark-Metric-Check/1.0`, and `Content-Type: application/json` on a `POST`. Headers you set take precedence.

***

## Placeholders

`{{path}}` placeholders read from the payload above, for example `{{call.id}}`, `{{call.properties.cardId}}`, or `{{simulation.variables.newPin}}`. Roark checks every placeholder when you save, so a typo is caught before any call runs.

How a value goes in depends on where the placeholder sits:

| Where | How the value goes in | Example |
| :- | :- | :- |
| URL | URL-encoded | `https://api.example.com/cards/{{call.properties.cardId}}/pin` |
| Header | As text. A value with a line break is refused | `X-Card-Id: {{call.properties.cardId}}` |
| Body, inside quotes | As escaped text | `"cardId": "{{call.properties.cardId}}"` |
| Body, outside quotes | As JSON: objects stay objects, text is quoted for you | `"call": {{call}}` sends the whole call object |

If a call has no value for a placeholder, such as a run variable on a live call or a property the call doesn't carry, Roark doesn't send the request. The result is recorded as missing data, never as a fail, and the literal `{{...}}` is never sent.

> [!TIP]
>
> A check written for simulations, such as one that reads `{{simulation.variables.newPin}}`, records missing data on live calls instead of failing them. Use a [collector's conditions](/documentation/metrics/metric-collectors) to run it only where it applies.

***

## Answering the Check

Reply with HTTP `200` and this JSON:

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{ "passed": true, "reason": "PIN on card_123 was changed at 14:02:51 UTC." }
```

| Field | Type | Description |
| :- | :- | :- |
| `passed` | boolean | Whether the call did what it should |
| `reason` | string | Optional. Shown next to the result in Roark. Longer than 1000 characters is cut |

A minimal endpoint:

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
app.post('/roark/checks/pin-changed', async (req, res) => {
  const { cardId, expectedPin } = req.body
  const card = await cards.get(cardId)

  if (card.pin === expectedPin) {
    return res.json({ passed: true, reason: `PIN on ${cardId} matches the new PIN.` })
  }
  res.json({ passed: false, reason: `PIN on ${cardId} was not changed.` })
})
```

### Errors and retries

Only a `200` with `passed` set is a verdict. Anything else is recorded as an **error** on the call, not a fail, so your pass rate reflects your agent and not your endpoint's uptime.

| Your endpoint | Result |
| :- | :- |
| `200` with `{ "passed": ... }` | Pass or Fail |
| Times out (30 seconds), can't be reached, or answers `5xx` | Retried twice with the same `requestId`, then an error |
| Any other status, including other `2xx` and `4xx` | Error, not retried |
| `200` with a body that isn't the contract | Error |

Roark shows your `reason`, never your raw response, on the call.

***

## Security

* **Authenticate with headers.** Add your own `Authorization` or API key header. Header values are redacted in Roark's request log.
* **Public URLs only.** The URL must be public `http(s)`. Roark checks it when you save, again after placeholders are filled in, and once more against the resolved address when it connects, so a public hostname that points to a private address is refused.
* **No redirects.** Roark doesn't follow redirects, so point the URL at the final endpoint.
* **Bounded responses.** Responses over 64 KB are not read.

***

## Limits

* API checks run on calls only, not chats.
* An API check has no variants: the request is the metric, so edit it on the metric itself.

***

## What's Next

> [card] [Studio](/documentation/metrics/studio)
>
> Author and test an API check before it runs on calls

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Run your API check automatically on matching live calls

> [card] [Run Plans](/documentation/simulation-testing/run-plans)
>
> Attach the check to a simulation and gate a run on it

> [card] [Variables](/documentation/simulation-testing/variables)
>
> Pass run variables your check can compare against


# Formulas
Source: https://docs.roark.ai/documentation/metrics/custom-metrics/formulas

Compute a metric from other metrics with math and comparisons

A formula metric computes its value from other metrics on the same call. Use one to roll several focused metrics into a single score, or to turn them into one pass/fail gate:

```
{{Task Completion >= 2}} AND {{Instruction Follow >= 4}}        →  Boolean "Successful call"
({{Empathy}} + {{Resolution}} + {{Clarity}}) / 3                 →  Numeric "Quality score"
```

Formulas don't call a model: they read values Roark has already recorded, so the same inputs always give the same result, and formulas aren't billed like LLM Judge metrics.

***

## Quickstart

Formulas aren't a [config as code](/documentation/config-as-code/overview) kind yet. Create the formula once with the SDK under a stable `slug`, then collect it from config by that slug. You can also build it in Studio.

### Create it with the SDK

Create the metric with `calculationType: "FORMULA"`. List every metric the formula references in `sources`:

```typescript Node theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import Roark from '@roarkanalytics/sdk'

const client = new Roark({ bearerToken: process.env.ROARK_API_BEARER_TOKEN })

const { data: definitions } = await client.metric.listDefinitions()
const idOf = (slug: string) => definitions.find((d) => d.metricId === slug)!.id

const taskCompletion = idOf('task_completion')
const instructionFollow = idOf('instruction_follow')

const metric = await client.metric.createDefinition({
  calculationType: 'FORMULA',
  name: 'Successful call',
  slug: 'successful_call',
  outputType: 'BOOLEAN',
  formula: `{{id:${taskCompletion} >= 2}} AND {{id:${instructionFollow} >= 4}}`,
  sources: [{ sourceMetricDefinitionId: taskCompletion }, { sourceMetricDefinitionId: instructionFollow }],
})
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import os
from roark_analytics import Roark

client = Roark(bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"))

definitions = client.metric.list_definitions().data
id_of = lambda slug: next(d.id for d in definitions if d.metric_id == slug)

task_completion = id_of("task_completion")
instruction_follow = id_of("instruction_follow")

metric = client.metric.create_definition(
    calculation_type="FORMULA",
    name="Successful call",
    slug="successful_call",
    output_type="BOOLEAN",
    formula=f"{{{{id:{task_completion} >= 2}}}} AND {{{{id:{instruction_follow} >= 4}}}}",
    sources=[
        {"source_metric_definition_id": task_completion},
        {"source_metric_definition_id": instruction_follow},
    ],
)
```

```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/metric/definitions \
  -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "calculationType": "FORMULA",
    "name": "Successful call",
    "slug": "successful_call",
    "outputType": "BOOLEAN",
    "formula": "{{id:<task_completion id> >= 2}} AND {{id:<instruction_follow id> >= 4}}",
    "sources": [
      { "sourceMetricDefinitionId": "<task_completion id>" },
      { "sourceMetricDefinitionId": "<instruction_follow id>" }
    ]
  }'
```

The response is the metric definition, with the expression and sources echoed back under `formula`.

| Field | Type | Required | Description |
| :- | :- | :- | :- |
| `name` | string | Yes | Name of the metric (1-100 characters) |
| `outputType` | string | Yes | `NUMERIC` or `BOOLEAN` |
| `formula` | string | Yes | The expression, with `{{id:<uuid>}}` references (up to 500 characters) |
| `sources` | object\[] | Yes | At least two: each `sourceMetricDefinitionId`, plus an optional `sourceVariantId` to read a [variant](/documentation/metrics/studio#system-metrics-and-variants) of that metric |
| `slug` | string | No | Stable identifier. Generated from the name if omitted |
| `analysisPackageId` | string | No | Package to add the metric to. Defaults to your project's Custom Metrics package |

To change the expression later, `PUT /v1/metric/definitions/{idOrSlug}` with the new `formula`, and `sources` if they changed. The same request sets `booleanTrueLabel` and `booleanFalseLabel` on a Boolean formula. Each change creates a new metric version.

### Collect it from config

Once the formula exists, list its slug in a [collector](/documentation/config-as-code/collectors) next to your other metrics, and Roark scores it on every matching call. Roark collects the formula's source metrics along with it, so you don't list them:

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineCollector, defineConfig } from '@roarkanalytics/sdk/lib/config';

const frontdeskQuality = defineCollector({
  name: 'frontdesk-quality',
  modality: 'call',
  status: 'ACTIVE',
  metrics: ['successful_call'],
});

export default defineConfig({
  resources: [frontdeskQuality],
});
```

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: collector
name: frontdesk-quality
modality: call
status: ACTIVE
metrics:
  - successful_call
```

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply roark.config.ts
```

Apply fails with `unknown metric "successful_call"` until the formula has been created.

### Build it in Studio

Open [Studio](/documentation/metrics/studio) in Author mode and pick the **Formula** engine. The engine is locked after the first save.

![Studio Author mode with the Formula engine selected: the formula \{\{Task Completion >= 2}} AND \{\{Instruction Follow >= 4}} AND \{\{Call Reason != "Wrong number"}}, a toolbar with Insert metric and the AND, OR, NOT and parenthesis operators, a Boolean badge, the letter map A Task Completion, B Instruction Follow, C Call Reason, the preview \[A\] >= 2 AND \[B\] >= 4 AND \[C\] != "Wrong number", and the True and False labels Successful call and Unsuccessful call](https://mintcdn.com/roark/s80NtcBbMfjE9Dti/images/metrics/formula-metric.png?fit=max\&auto=format\&n=s80NtcBbMfjE9Dti\&q=85\&s=aa2f13dfa70fdfe5c1c71a05e453cfe1)

Type `{{` or click **Insert metric** to pick a source. Studio only offers metrics the formula can use. Click a comparison to change its operator or value. The toolbar shows the operators for the output you're building, and the preview underneath shows the formula with each source as a letter.

***

## Writing a Formula

A formula mixes **metric references** in `{{double braces}}` with operators.

### References

| Reference | Gives |
| :- | :- |
| `{{Empathy}}` | The metric's value on the call, as a number |
| `{{Empathy >= 4}}` | A comparison: `1` when it holds, `0` when it doesn't |

In Studio you reference metrics by name. Studio stores them by id, so renaming a source metric doesn't break the formula. Through the API, write the id: `{{id:<metric definition id>}}` and `{{id:<metric definition id> >= 4}}`.

A plain reference reads Numeric, Count, and Scale values as they are, and a Boolean as `1` or `0`. Compare a value inside the braces with:

| Source type | Compare with | Example |
| :- | :- | :- |
| Numeric, Count, Scale | `==` `!=` `>` `>=` `<` `<=` and a number | `{{Frustration Score <= 2}}` |
| Boolean | `==` `!=` and `TRUE` or `FALSE` | `{{Identity Verified == TRUE}}` |
| Classification | `==` `!=` and a label in quotes, matched without case | `{{Call Reason != "Wrong number"}}` |

Text metrics can't be used in a formula.

### Operators

| Operators | Use |
| :- | :- |
| `+` `-` `*` `/` and `( )` | Arithmetic and grouping |
| `AND` `OR` `NOT` | Combine comparisons |
| `TRUE` `FALSE` | `1` and `0` |
| `MIN(a, b, ...)` `MAX(a, b, ...)` `AVG(a, b, ...)` `ABS(a)` | Functions over references and numbers |

### Output

A formula returns a **Numeric** or a **Boolean**:

* **Numeric** for arithmetic over plain references: `({{Empathy}} + {{Resolution}}) / 2`.
* **Boolean** for comparisons combined with `AND`, `OR`, and `NOT`. Any non-zero result is true.

Studio picks the output from the formula: all plain references make it Numeric, all comparisons make it Boolean. A formula that mixes the two is refused in Studio. For a Boolean, name the two outcomes with the True and False labels.

### Rules

* At least two metric references, up to 500 characters, with balanced parentheses.
* Every source must record a value per call. Metrics that only record values per segment or turn, such as Response Time, can't be referenced. To build on those, use a [pattern](/documentation/metrics/custom-metrics/patterns) or a [threshold](/documentation/metrics/thresholds).
* A formula can reference other formulas and thresholds, as long as they don't loop back to it. Roark computes the sources first.

***

## Reading the Result

Each value carries its working in the reasoning, with every source's value, for example:

```
Formula: {{Task Completion >= 2}} AND {{Instruction Follow >= 4}} → 1 (Task Completion >= 2 = true; Instruction Follow >= 4 = true)
```

### When a source is missing

If any source has no value on the call, the formula records **N/A**, and the reasoning names the sources it couldn't read. It never fills a missing source with zero. Add the sources to the same [collector](/documentation/metrics/metric-collectors) or [run plan](/documentation/simulation-testing/run-plans) as the formula, so they run on the same calls.

A formula that can't produce a finite number, such as a division by a source that is `0`, fails on that call instead of recording a value. Guard divisions with `MAX({{Turns}}, 1)`.

***

## What's Next

> [card] [LLM Judge](/documentation/metrics/custom-metrics/llm-judge)
>
> Write the focused metrics a formula combines

> [card] [Thresholds](/documentation/metrics/thresholds)
>
> Turn a single metric into a pass/fail check

> [card] [Patterns](/documentation/metrics/custom-metrics/patterns)
>
> Detect what follows what within a call

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Run the formula automatically on live calls


# LLM Judge
Source: https://docs.roark.ai/documentation/metrics/custom-metrics/llm-judge

Describe what to measure in plain language and let Roark score every call against it

An LLM Judge metric is a prompt. You describe what to measure, and **Roark Prism**, our evaluation model built for voice AI, reads each call and returns a typed result with its reasoning.

```
"Did the agent verify the caller's identity?"           →  Boolean
"Rate the agent's empathy on a 1-5 scale"               →  Scale
"What was the primary reason for the call?"             →  Classification
"How many times did the agent attempt to upsell?"       →  Count
```

Use an LLM Judge for anything that takes understanding the conversation: whether a step was followed, how well something was handled, why the customer called. When the answer is already in other metrics, a [Formula](/documentation/metrics/custom-metrics/formulas) or [Pattern](/documentation/metrics/custom-metrics/patterns) is cheaper and deterministic. When it lives in your backend, use an [API check](/documentation/metrics/custom-metrics/api-checks).

***

## Quickstart

Define the metric [in code](/documentation/config-as-code/metrics) (recommended), or build it in Studio.

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineMetric, defineConfig } from '@roarkanalytics/sdk/lib/config';

const identityVerified = defineMetric({
  name: 'identity_verified', // the slug a collector references
  displayName: 'Identity Verified',
  type: 'BOOLEAN', // BOOLEAN | SCALE | NUMERIC | TEXT | CLASSIFICATION
  prompt:
    'Using {{transcript}} and {{world_context}}, did the agent verify the caller identity before sharing account details?',
  trueLabel: 'Verified',
  falseLabel: 'Not Verified',
});

export default defineConfig({
  resources: [identityVerified],
});
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: metric
name: identity_verified            # the slug a collector references
displayName: Identity Verified
type: BOOLEAN                      # BOOLEAN | SCALE | NUMERIC | TEXT | CLASSIFICATION
prompt: |
  Using {{transcript}} and {{world_context}}, did the agent verify the caller identity before sharing account details?
trueLabel: Verified
falseLabel: Not Verified
```

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply roark.config.ts
```

See the [config as code metrics reference](/documentation/config-as-code/metrics) for SCALE, CLASSIFICATION, and `PER_PARTICIPANT` fields.

### Create it with the SDK

Create the metric with `calculationType: "LLM_JUDGE"`. It's the default, so requests that leave `calculationType` out create an LLM Judge too.

```typescript Node theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import Roark from '@roarkanalytics/sdk'

const client = new Roark({ bearerToken: process.env.ROARK_API_BEARER_TOKEN })

const metric = await client.metric.createDefinition({
  calculationType: 'LLM_JUDGE',
  name: 'Customer Satisfaction',
  outputType: 'SCALE',
  llmPrompt: 'Using {{transcript}}, rate the overall customer satisfaction based on the tone, resolution, and agent helpfulness.',
  scaleMin: 1,
  scaleMax: 10,
  scaleLabels: [
    { rangeMin: 1, rangeMax: 3, label: 'Poor', displayOrder: 1 },
    { rangeMin: 4, rangeMax: 6, label: 'Average', displayOrder: 2 },
    { rangeMin: 7, rangeMax: 10, label: 'Excellent', displayOrder: 3 },
  ],
})
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import os
from roark_analytics import Roark

client = Roark(bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"))

metric = client.metric.create_definition(
    calculation_type="LLM_JUDGE",
    name="Customer Satisfaction",
    output_type="SCALE",
    llm_prompt="Using {{transcript}}, rate the overall customer satisfaction based on the tone, resolution, and agent helpfulness.",
    scale_min=1,
    scale_max=10,
    scale_labels=[
        {"range_min": 1, "range_max": 3, "label": "Poor", "display_order": 1},
        {"range_min": 4, "range_max": 6, "label": "Average", "display_order": 2},
        {"range_min": 7, "range_max": 10, "label": "Excellent", "display_order": 3},
    ],
)
```

```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/metric/definitions \
  -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "calculationType": "LLM_JUDGE",
    "name": "Identity Verified",
    "outputType": "BOOLEAN",
    "llmPrompt": "Using {{transcript}}, did the agent verify the caller identity before proceeding with the request?",
    "booleanTrueLabel": "Verified",
    "booleanFalseLabel": "Not Verified"
  }'
```

| Field | Type | Required | Description |
| :- | :- | :- | :- |
| `name` | string | Yes | Name of the metric (1-100 characters) |
| `outputType` | string | Yes | One of `BOOLEAN`, `NUMERIC`, `TEXT`, `SCALE`, `CLASSIFICATION`, `COUNT`, `OFFSET` |
| `llmPrompt` | string | No | The instructions the judge follows (max 2000 characters). See [Prompt Variables](#prompt-variables) |
| `slug` | string | No | Stable identifier. Generated from the name if omitted |
| `scope` | string | No | `GLOBAL` (default) or `PER_PARTICIPANT` |
| `participantRole` | string | Conditional | Required when scope is `PER_PARTICIPANT` |
| `supportedContexts` | string\[] | No | Defaults to `["CALL"]` |
| `analysisPackageId` | string | No | Package to add the metric to. Defaults to your project's Custom Metrics package |

**Type-specific fields:**

| Field | Applies To | Description |
| :- | :- | :- |
| `booleanTrueLabel` / `booleanFalseLabel` | BOOLEAN | Labels for the two outcomes |
| `scaleMin` / `scaleMax` | SCALE | Range boundaries (0-100) |
| `scaleLabels` | SCALE | Label objects with `rangeMin`, `rangeMax`, `label`, `displayOrder` |
| `classificationOptions` | CLASSIFICATION | Options with `label`, `description`, `displayOrder` |
| `maxClassifications` | CLASSIFICATION | Maximum number of options the judge can pick |

To change the prompt or labels later, `PUT /v1/metric/definitions/{idOrSlug}` with the fields to change. Each change creates a new metric version.

### Build it in Studio

Open [Studio](/documentation/metrics/studio) in Author mode and pick the **LLM Judge** engine. Choose an output type, then write the instructions: for a Boolean, the yes/no question the judge should answer; for a Scale, what each band means. Use the test rail to score real calls before you save.

![Studio Author mode: the numbered Engine, Output (with range labels), Instructions, and Group sections, plus a Tests rail of scored test calls](https://mintcdn.com/roark/IjseSxRBSMbit4Aq/images/metrics/studio-author.png?fit=max\&auto=format\&n=IjseSxRBSMbit4Aq\&q=85\&s=b70e3d6a98d20fe838b6fbd6844696ba)

***

## Output Types

| Type | Returns | Configure |
| :- | :- | :- |
| `BOOLEAN` | Yes or no | Labels for each side, e.g. **Verified** / **Not Verified** |
| `SCALE` | A score in a range | `scaleMin`, `scaleMax`, and labels for bands of the range |
| `CLASSIFICATION` | One or more labels from a list | The options, and how many can be picked |
| `NUMERIC` / `COUNT` | A number | Nothing extra |
| `TEXT` | Free text | Nothing extra |

***

## Prompt Variables

Prompts reference call data with `{{variable}}` placeholders, which Roark fills in before the judge runs. In Studio, type `{{` in the instructions to pick one.

| Variable | What it inserts |
| :- | :- |
| `{{transcript}}` | The full transcript, from the best available source |
| `{{transcript.post_call}}` / `{{transcript.realtime}}` | A specific transcript source |
| `{{transcript.participant}}` | The transcript filtered to the participant being scored |
| `{{summary}}` | The AI-generated call summary |
| `{{tool_invocations}}` | Tool calls made during the call, with parameters and results (JSON) |
| `{{properties}}` | All of the call's custom properties (JSON) |
| `{{properties.<name>}}` | One custom property's value, e.g. `{{properties.partner_outcome}}` |
| `{{interruptions}}` / `{{silences}}` | Overlaps and silences, with timing (JSON) |
| `{{sentiment}}` / `{{emotions}}` / `{{vocal_cues}}` | Sentiment, emotion and vocal cue analysis |
| `{{duration}}` / `{{call_status}}` / `{{call_ended_status}}` / `{{call_id}}` | Call metadata |
| `{{world_context}}` | The current date, time and timezone |

### Using a custom property

`{{properties.<name>}}` inserts just that property's value, which makes it easy to grade the conversation against data from your own systems:

```text theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
Our partner reported the outcome {{properties.partner_outcome}} and an APR of
{{properties.purchase_apr_total}}. Using {{transcript}}, did the representative
state that APR on the call?
```

* The name can be written as stored (`partner_outcome`) or as you send it (`partnerOutcome`).
* If the call doesn't carry the property, the prompt says so (`[partner_outcome property not set on this conversation]`) rather than leaving the placeholder in.
* Properties added after the call was created are picked up the next time the metric runs. See [updating properties after ingest](/documentation/integrations/custom-integrations#updating-properties-after-ingest).

***

## Writing Good Prompts

> [accordion] Ask one thing
>
> A prompt that checks identity, tone, and resolution at once gives you one number you can't act on. Write three focused metrics and combine them with a [formula](/documentation/metrics/custom-metrics/formulas) if you need a single pass/fail.

> [accordion] Say what counts
>
> Spell out the edge cases: "Verification counts if the agent confirmed two of name, date of birth, and postcode. Asking without confirming doesn't count."

> [accordion] Define every band of a scale
>
> For a Scale, describe what a 1, a 3, and a 5 look like. Without that, scores drift between calls that are about the same.

> [accordion] Test on real calls first
>
> Run the metric against representative calls in [Studio](/documentation/metrics/studio) and read the reasoning before attaching it to a collector.

***

## What's Next

> [card] [Studio](/documentation/metrics/studio)
>
> Write and test a prompt against real calls

> [card] [Formulas](/documentation/metrics/custom-metrics/formulas)
>
> Combine focused judges into one score

> [card] [Thresholds](/documentation/metrics/thresholds)
>
> Turn a score into a pass/fail check

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Run the metric automatically on live calls


# Patterns
Source: https://docs.roark.ai/documentation/metrics/custom-metrics/patterns

Detect when one thing happens and another follows it, line by line through the call

A pattern metric watches a call line by line. When a **trigger** happens, it looks at the lines just after it for an **outcome**, and records whether the pattern happened or how many times.

"Did the agent talk over a caller who was frustrated?" is a pattern: the trigger is the caller's frustration, the outcome is an agent interruption within the next few lines. So is "did the agent recover after a misunderstanding" or "how often did a slow response follow a question".

Patterns are built from metrics Roark already records on each line, so they don't call a model: the same call always gives the same result, and patterns aren't billed like LLM Judge metrics.

***

## Quickstart

Patterns aren't a [config as code](/documentation/config-as-code/overview) kind yet. Create the pattern once with the SDK under a stable `slug`, then collect it from config by that slug. You can also build it in Studio.

### Create it with the SDK

Create the metric with `calculationType: "PATTERN"`. Sources are referenced by metric definition id, so look them up first:

```typescript Node theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import Roark from '@roarkanalytics/sdk'

const client = new Roark({ bearerToken: process.env.ROARK_API_BEARER_TOKEN })

const { data: definitions } = await client.metric.listDefinitions()
const idOf = (slug: string) => definitions.find((d) => d.metricId === slug)!.id

const metric = await client.metric.createDefinition({
  calculationType: 'PATTERN',
  name: 'Talked over a frustrated caller',
  slug: 'talked_over_frustrated_caller',
  operation: 'PATTERN_EXISTS',
  windowMode: 'segments',
  trigger: {
    sourceMetricDefinitionId: idOf('frustration_score'),
    operator: 'GREATER_THAN_OR_EQUALS',
    thresholdValue: '4',
    sourceParticipantRole: 'CUSTOMER',
  },
  outcome: {
    sourceMetricDefinitionId: idOf('interruption'),
    operator: 'EQUALS',
    thresholdValue: 'true',
    sourceParticipantRole: 'AGENT',
    windowAfter: 3,
  },
})
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import os
from roark_analytics import Roark

client = Roark(bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"))

definitions = client.metric.list_definitions().data
id_of = lambda slug: next(d.id for d in definitions if d.metric_id == slug)

metric = client.metric.create_definition(
    calculation_type="PATTERN",
    name="Talked over a frustrated caller",
    slug="talked_over_frustrated_caller",
    operation="PATTERN_EXISTS",
    window_mode="segments",
    trigger={
        "source_metric_definition_id": id_of("frustration_score"),
        "operator": "GREATER_THAN_OR_EQUALS",
        "threshold_value": "4",
        "source_participant_role": "CUSTOMER",
    },
    outcome={
        "source_metric_definition_id": id_of("interruption"),
        "operator": "EQUALS",
        "threshold_value": "true",
        "source_participant_role": "AGENT",
        "window_after": 3,
    },
)
```

```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/metric/definitions \
  -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "calculationType": "PATTERN",
    "name": "Talked over a frustrated caller",
    "slug": "talked_over_frustrated_caller",
    "operation": "PATTERN_EXISTS",
    "windowMode": "segments",
    "trigger": {
      "sourceMetricDefinitionId": "<frustration_score id>",
      "operator": "GREATER_THAN_OR_EQUALS",
      "thresholdValue": "4",
      "sourceParticipantRole": "CUSTOMER"
    },
    "outcome": {
      "sourceMetricDefinitionId": "<interruption id>",
      "operator": "EQUALS",
      "thresholdValue": "true",
      "sourceParticipantRole": "AGENT",
      "windowAfter": 3
    }
  }'
```

The response is the metric definition, with your configuration echoed back under `pattern`.

| Field | Type | Required | Description |
| :- | :- | :- | :- |
| `name` | string | Yes | Name of the metric (1-100 characters) |
| `operation` | string | Yes | `PATTERN_EXISTS` (Boolean), `PATTERN_COUNT` (Count), or `OUTCOME_AGGREGATE` (Numeric, see [below](#averaging-the-outcome)) |
| `trigger` | object | One of | A single trigger condition |
| `triggers` | object\[] | One of | Several trigger conditions, combined with `triggerCombinator` |
| `triggerCombinator` | string | Conditional | `AND` or `OR`. Required with more than one trigger |
| `outcome` | object | Yes | The outcome condition, with its window |
| `windowMode` | string | No | `seconds` (default) or `segments` |
| `slug` | string | No | Stable identifier. Generated from the name if omitted |
| `analysisPackageId` | string | No | Package to add the metric to. Defaults to your project's Custom Metrics package |

Each trigger and the outcome take:

| Field | Type | Required | Description |
| :- | :- | :- | :- |
| `sourceMetricDefinitionId` | string | Yes | The source metric's definition id |
| `operator` | string | Yes | `EQUALS`, `NOT_EQUALS`, `GREATER_THAN`, `GREATER_THAN_OR_EQUALS`, `LESS_THAN`, `LESS_THAN_OR_EQUALS` |
| `thresholdValue` | string | Yes | The value to compare with: a number, or `true` / `false` for a Boolean source |
| `sourceParticipantRole` | string | No | `AGENT` or `CUSTOMER`, for a per-participant source |
| `sourceVariantId` | string | No | A [variant](/documentation/metrics/studio#system-metrics-and-variants) of the source metric to read |
| `windowAfter` | integer | Outcome only | How far after the trigger to look, in `windowMode` units |
| `windowBefore` | integer | No, outcome only | How far before the trigger to look. Default `0` |

A request that references the same source metric twice, across the triggers and the outcome, is refused with `400`. `PUT /v1/metric/definitions/{idOrSlug}` changes a pattern's name. Its triggers, outcome, and window are fixed once it's created: to change them, create a new pattern.

### Collect it from config

Once the pattern exists, list its slug in a [collector](/documentation/config-as-code/collectors) next to your other metrics, and Roark scores it on every matching call. Roark collects the pattern's source metrics along with it, so you don't list them:

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineCollector, defineConfig } from '@roarkanalytics/sdk/lib/config';

const frontdeskQuality = defineCollector({
  name: 'frontdesk-quality',
  modality: 'call',
  status: 'ACTIVE',
  metrics: ['talked_over_frustrated_caller'],
});

export default defineConfig({
  resources: [frontdeskQuality],
});
```

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: collector
name: frontdesk-quality
modality: call
status: ACTIVE
metrics:
  - talked_over_frustrated_caller
```

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply roark.config.ts
```

Apply fails with `unknown metric "talked_over_frustrated_caller"` until the pattern has been created.

### Build it in Studio

Open [Studio](/documentation/metrics/studio) in Author mode and pick the **Pattern** engine. The engine is locked after the first save.

![Studio Author mode with the Pattern engine selected: two triggers joined by AND, Frustration Score >= 4 and Vocal Activation >= 0.7, both for the Customer, then within 3 segments, check for Interruption equal to True for the Agent, with Did it happen? selected as the result](https://mintcdn.com/roark/s80NtcBbMfjE9Dti/images/metrics/pattern-metric.png?fit=max\&auto=format\&n=s80NtcBbMfjE9Dti\&q=85\&s=59ed716996c77b43a2bfe4b36f1b71fe)

**1. If this happens**

Pick a trigger metric, an operator, and a value. For a per-participant metric, choose whose value counts: **Agent** or **Customer**. Click **Add condition** for more triggers, and choose **AND** or **OR** between them.

**2. Then within**

Set the window, in **segments** (1 to 20, default 3) or **seconds** (1 to 300, default 30).

**3. Check for**

Pick the outcome metric and the value it should reach. Each trigger and the outcome must use a different source metric.

**4. What does this metric return?**

**Did it happen?** gives a Boolean per call. **How many times?** gives a Count per call.

***

## How It Works

**1. Roark walks the call's segments in order**

A segment is one line of the transcript. Overlapping speech and system actions, such as a keypress or a hang-up, are segments too.

**2. Each segment is checked against the triggers**

A trigger compares a source metric's value on that segment with a threshold, for example **Frustration Score >= 4** for the customer. With several triggers, **AND** needs all of them to hold on the same segment, **OR** needs any one.

**3. The window after a trigger is searched for the outcome**

When the triggers hold, Roark looks at the segments inside the window, for example the next 3 segments, for one where the outcome holds. The trigger's own segment isn't part of the window.

**4. The call gets one value**

**Did it happen?** is true if any trigger found its outcome. **How many times?** counts the trigger segments that did. The segments that matched are marked on the call's timeline as the metric's evidence.

***

## Choosing Source Metrics

A pattern reads the values a metric records **on segments or turns**. Metrics that only record one value per call never match a trigger or an outcome. Good sources from the [system metrics](/documentation/metrics/system-metrics):

| Metric | Slug | Recorded per | Values |
| :- | :- | :- | :- |
| Frustration Score | `frustration_score` | Segment | 1 to 5 |
| Sentiment Score | `sentiment_score` | Segment | 1 to 9 |
| Vocal Activation | `vocal_activation` | Segment | 0 to 1 |
| Interruption | `interruption` | Segment | `true` / `false` |
| Comprehension Failure | `comprehension_failure` | Turn | `true` / `false` |
| Response Time | `response_time` | Turn | Milliseconds |

Your own [LLM Judge](/documentation/metrics/custom-metrics/llm-judge) metrics work too, when they record values per segment or turn.

Conditions compare numbers. Numeric, Count and Scale values are compared as they are, and a Boolean is `true` (1) or `false` (0). Classification and Text metrics can't be used as a source: their values never match.

***

## Windows

| Mode | The window covers |
| :- | :- |
| `segments` | The next `windowAfter` segments after the trigger, and the `windowBefore` segments before it |
| `seconds` | Every segment that starts up to `windowAfter` seconds after the trigger segment starts, or `windowBefore` seconds before it |

The trigger's own segment is never part of its window, so a pattern always relates two different lines.

### Averaging the outcome

`OUTCOME_AGGREGATE` (API only) ignores the outcome's operator and threshold. It averages the outcome metric's values across every trigger's window and records the result as a Numeric. Use it for questions like "how fast did the agent respond after a frustrated line": trigger on **Frustration Score >= 4**, aggregate **Response Time**.

***

## When Data Is Missing

* If no trigger metric recorded any values on the call, or the outcome metric recorded none, the pattern records **N/A** with the reason, rather than false.
* A call with no transcript segments gets no value.
* Make sure the source metrics run on the same calls: add them to the same [collector](/documentation/metrics/metric-collectors) or [run plan](/documentation/simulation-testing/run-plans) as the pattern.

***

## What's Next

> [card] [System Metrics](/documentation/metrics/system-metrics)
>
> Find per-segment metrics to build patterns from

> [card] [Formulas](/documentation/metrics/custom-metrics/formulas)
>
> Combine call-level metrics into one score

> [card] [Studio](/documentation/metrics/studio)
>
> Build a pattern and test it on real calls

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Run the pattern automatically on live calls


# Datasets
Source: https://docs.roark.ai/documentation/metrics/datasets

Group calls or chats into named collections for evaluation, comparison, and analysis

## Overview

A **dataset** is a named, project-scoped collection of conversations. Use datasets to group calls or chats you want to evaluate together: a batch of onboarding failures, last quarter's escalations, a golden set for regression testing in [Studio](/documentation/metrics/studio).

Each dataset holds one conversation type, **Calls** or **Chats**, chosen at creation. A dataset is locked to that type; you can't mix modalities in a single dataset.

Datasets live under **Measure → Datasets** in the sidebar.

![The Datasets list page: a split-panel layout with All, Calls, and Chats filter pills and a search box above a list of dataset cards on the left, and the selected dataset's detail with its conversations table on the right](https://mintcdn.com/roark/17HU3uIlkQ6GJrX5/images/metrics/datasets-list.png?fit=max\&auto=format\&n=17HU3uIlkQ6GJrX5\&q=85\&s=c8dd619958e92487e99260269fe2b85a)

## Dataset properties

| Property | Description |
| :- | :- |
| **Name** | Required, e.g. "Q1 Support Calls", "Onboarding Failures" |
| **Description** | Optional context for what the dataset contains |
| **Type** | Calls or Chats, fixed at creation |
| **Count** | Number of conversations currently in the dataset |
| **Labels** | Tags for organizing datasets |

> [!NOTE]
>
> Datasets with a **System** badge are managed by Roark and can't be edited or deleted.

## Creating a dataset

**1. Open Datasets**

Go to **Measure → Datasets** and click **New Dataset**.

**2. Pick the conversation type**

Choose **Calls** or **Chats**. This is permanent. Pick based on what you'll be adding.

**3. Name it**

Add a name and an optional description, then click **Create**.

## Adding conversations

How you add conversations depends on the dataset type:

* **Call datasets**: open the dataset and click **Add Calls** to pick calls from your project.
* **Chat datasets**: add chats from the chat's detail page, not from the dataset panel.

![A call dataset's detail panel with the Add Calls picker dialog open: a searchable, filterable list of candidate calls, each with a checkbox, summary, agent, duration, status, and timestamp, three selected, and a footer showing 3 calls selected with Cancel and Add 3 Calls buttons](https://mintcdn.com/roark/wL__NyRpzLGS2dwF/images/metrics/dataset-add-calls.png?fit=max\&auto=format\&n=wL__NyRpzLGS2dwF\&q=85\&s=c299e236201d5e1036f3b9a5ece5054e)

## Managing a dataset

Selecting a dataset from the list opens its detail panel, which shows the name, conversation count, and last-updated time above a table of members. From there you can:

* **Remove conversations**: select rows and bulk-remove them (with confirmation). Removing a conversation from a dataset doesn't delete the conversation itself.
* **Edit**: rename the dataset or update its description.
* **Delete**: remove the dataset entirely.
* **Drill in**: click any row to open that conversation's detail view.

The list page has **All / Calls / Chats** filter pills and a search box to find datasets quickly.

## What's Next

> [card] [Studio](/documentation/metrics/studio)
>
> Run a battery of evals against the conversations you've grouped

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Run metrics automatically on matching calls and chats


# Collection Jobs
Source: https://docs.roark.ai/documentation/metrics/metric-collection-jobs

Run metrics on demand for specific calls via the API

## Overview

Collection jobs are the programmatic way to run metrics on demand for one or many calls via the API. While [collectors](/documentation/metrics/metric-collectors) automate collection for incoming calls, collection jobs let you trigger metric processing against existing calls, whether that's backfilling a new metric across historical data or re-running after a definition change.

> [!TIP]
>
> Want to run metrics across calls from the UI? Use [Studio](/documentation/metrics/studio) in Evaluate mode ("Run a battery of metrics across a set of calls") instead. Collection jobs are designed for programmatic, at-scale use via the SDK.

***

## When to Use Collection Jobs

* **Backfill metrics**: Run newly created metrics on historical calls
* **Re-run metrics**: Reprocess calls after updating a metric definition
* **On-demand analysis**: Collect metrics for a specific set of calls uploaded via API
* **Scale up after testing**: You've validated a metric in Studio, now run it across many calls programmatically

***

## Creating a Collection Job

Provide an array of call IDs and the metrics you want to collect:

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const job = await client.metricCollectionJob.create({
  callIds: ['call-uuid-1', 'call-uuid-2', 'call-uuid-3'],
  metrics: [
    { id: 'metric-definition-uuid-1' },
    { id: 'metric-definition-uuid-2' },
  ],
})

// job.data
{
  id: 'job-uuid',
  status: 'PENDING',
  triggeredBy: 'USER_API',
  totalItems: 6,
  completedItems: 0,
  failedItems: 0,
  startedAt: null,
  completedAt: null,
  errorMessage: null,
  createdAt: '2025-01-15T10:30:00Z',
  updatedAt: '2025-01-15T10:30:00Z',
}
```

**Parameters:**

| Field | Type | Required | Description |
| :- | :- | :- | :- |
| `callIds` | string\[] | Yes | Array of call UUIDs to process (minimum 1, maximum 500) |
| `metrics` | array | Yes, unless `policyId` is given | Metric definitions to collect (minimum 1, maximum 20) |
| `policyId` | string | No | A [collector](/documentation/metrics/metric-collectors) whose settings to score with. See [below](#re-scoring-with-collector-settings) |

> [!NOTE]
>
> The `totalItems` count equals `callIds.length * metrics.length`: each call-metric combination is a separate item.

### Re-scoring with collector settings

Without `policyId`, each metric runs with its defaults. Metrics in a collector can carry per-collector settings, such as which properties [Property Verification](/documentation/metrics/system-metrics#property-verification) checks. To re-score calls exactly as a collector scores them, pass its id as `policyId`:

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const job = await client.metricCollectionJob.create({
  callIds: ['call-uuid-1', 'call-uuid-2'],
  policyId: 'collector-uuid',
})
```

* Each metric runs as the collector runs it, with its settings. `metrics` defaults to every metric the collector collects; if you list metrics, each must be in the collector.
* The collector's conditions are not applied: the calls you list are scored as given.
* The job and the values it writes carry the collector's id in `policyIds`, as a live collection does.
* One request covers at most 5,000 call-metric items, so with a collector of 12 metrics that is about 400 calls per request.

Collector ids are returned by `client.metricPolicy.list()` (`GET /v1/metric/policies`).

***

## Job Lifecycle

Collection jobs progress through the following statuses:

```
PENDING → PROCESSING → COMPLETED / FAILED / CANCELED
```

| Status | Description |
| :- | :- |
| `PENDING` | Job created, waiting to start processing |
| `PROCESSING` | Actively collecting metrics from calls |
| `COMPLETED` | All items processed successfully |
| `FAILED` | Job encountered errors (check `errorMessage`) |
| `CANCELED` | Job was canceled before completion |

### Progress Tracking

Monitor progress using these fields:

| Field | Description |
| :- | :- |
| `totalItems` | Total number of call-metric combinations to process |
| `completedItems` | Number of items successfully processed |
| `failedItems` | Number of items that failed |
| `startedAt` | When processing began |
| `completedAt` | When processing finished |

***

## Monitoring Jobs

### List Collection Jobs

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const jobs = await client.metricCollectionJob.list()

// With filters
const completedJobs = await client.metricCollectionJob.list({
  status: 'COMPLETED',
  limit: 50,
})
```

| Parameter | Type | Description |
| :- | :- | :- |
| `limit` | number | Max results (1-50, default: 20) |
| `after` | string | Cursor for pagination |
| `status` | string | Filter by `PENDING`, `PROCESSING`, `COMPLETED`, `FAILED`, or `CANCELED` |

### Get a Collection Job

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const job = await client.metricCollectionJob.getByID('job-id')
```

Returns the full job object with current progress.

***

## Example: Studio to Production

A typical end-to-end workflow (test a metric in Studio, then run it across calls via the SDK):

**1. Test in Studio**

Use [Studio](/documentation/metrics/studio) to test your metric prompt against a sample call. Iterate until you're happy with the output.

**2. Create the Metric Definition**

Once validated, create the metric definition via the SDK:

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const metric = await client.metric.createDefinition({
  name: 'Identity Verified',
  outputType: 'BOOLEAN',
  analysisPackageId: 'your-package-id',
  llmPrompt: 'Did the agent successfully verify the caller identity?',
  booleanTrueLabel: 'Verified',
  booleanFalseLabel: 'Not Verified',
})
```

**3. Run on Existing Calls**

Create a collection job targeting the calls you want to analyze:

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const job = await client.metricCollectionJob.create({
  callIds: ['call-id-1', 'call-id-2', 'call-id-3'],
  metrics: [{ id: metric.data.id }],
})
```

**4. Poll for Completion**

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
let status = 'PENDING'
while (status !== 'COMPLETED' && status !== 'FAILED') {
  const result = await client.metricCollectionJob.getByID(job.data.id)
  status = result.data.status
  console.log(`Progress: ${result.data.completedItems}/${result.data.totalItems}`)
}
```

**5. Fetch Results**

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const metrics = await client.call.listMetrics('call-id-1')
console.log(metrics.data)
```

> [!TIP]
>
> Once you've validated the metric works well across calls, consider creating a [collector](/documentation/metrics/metric-collectors) to automatically collect it on future calls.

***

## What's Next

> [card] [Studio](/documentation/metrics/studio)
>
> Test metrics interactively before running at scale

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Automate metric collection for incoming calls


# Collectors
Source: https://docs.roark.ai/documentation/metrics/metric-collectors

Run a set of metrics automatically on every conversation that matches

## Overview

A collector runs a set of metrics on the calls (or chats) it matches. Instead of triggering metric collection by hand, you define the segment of conversations you care about (by agent, source, or property) and Roark scores every matching conversation as it comes in.

Define a collector [in code](/documentation/config-as-code/collectors) (recommended), or build it in the dashboard.

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineCollector, defineConfig } from '@roarkanalytics/sdk/lib/config';

const frontdeskQuality = defineCollector({
  name: 'frontdesk-quality',
  modality: 'call',
  status: 'ACTIVE',
  metrics: ['consent_collection_consent_obtained', 'refund_policy_accuracy'],
  filters: [
    {
      conditions: [{ type: 'AGENT', key: 'frontdesk', operator: 'EQUALS' }],
    },
  ],
});

export default defineConfig({ resources: [frontdeskQuality] });
```

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: collector
name: frontdesk-quality
modality: call
status: ACTIVE
metrics:
  - consent_collection_consent_obtained
  - refund_policy_accuracy
filters:
  - conditions:
      - type: AGENT
        key: frontdesk
        operator: EQUALS
```

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply roark.config.ts
```

See the [config as code reference](/documentation/config-as-code/collectors) for every field, including per-collector metric settings.

Collectors live under **Measure → Metrics**: the Metrics page shows a Collectors strip at the top, and **Manage collectors** opens the full list at `/metrics/collectors`.

![The Collectors list at /metrics/collectors: a System collector (Call Monitoring) with a shield badge and a static Active status, above custom collectors showing their segment summary, Runs metric chips, an Edit chip, and an Active or Paused toggle](https://mintcdn.com/roark/cU-YAWgdvSh373VF/images/metrics/collectors-list.png?fit=max\&auto=format\&n=cU-YAWgdvSh373VF\&q=85\&s=7b1fec6d00cae250719adf2a575972a3)

A collector is made of four things:

1. **Name**: how it appears in the collectors list
2. **Modality**: Call or Chat. A collector fires for exactly one modality, and this is locked after creation
3. **Conditions** (optional): the segment of conversations it matches. Leave empty to match all calls (or all chats)
4. **Metrics**: the metrics to run on every match, each with optional pass/fail thresholds

***

## Creating a Collector

### Build it in the dashboard

**1. Start a new collector**

On `/metrics/collectors`, click **New collector**. The editor reads as a sentence: "When a **call** matches…" Click the chip to switch between Call and Chat. Once the collector is created, the modality is locked.

**2. Define the segment**

Add conditions to target specific conversations, or leave the conditions empty to match everything. Use **Add condition** and **Add OR condition** to build a group, and **Add AND condition group** to require a second group to match as well.

**3. Pick the metrics**

Under **Evaluate these metrics**, add the metrics to run. Each picked metric runs on every conversation that matches the scope above. Where a metric has multiple variants, choose which variant's prompt and config apply. Some metrics take a setting of their own here, such as **Properties to verify** for Property Verification (see [per-metric settings](#per-metric-settings)).

**4. Add thresholds (optional)**

On any picked metric, click **Add pass/fail threshold** to turn its raw value into a pass/fail outcome, for example, "Pass if ≥ 7".

**5. Create and name it**

Click **Create collector**. Roark suggests a name built from the metrics and conditions. Keep it or edit it, then confirm. Incoming conversations that match your conditions are evaluated from then on.

![The collector editor: a 'When a call matches…' conditions section with one group matching Agent is Billing Assistant OR Source is Vapi, above an 'Evaluate these metrics' section with the Disclosure Completeness scale metric showing a Pass if ≥ 7 pass/fail threshold and an Add pass/fail threshold button](https://mintcdn.com/roark/wL__NyRpzLGS2dwF/images/metrics/collector-editor.png?fit=max\&auto=format\&n=wL__NyRpzLGS2dwF\&q=85\&s=e3723171a09973465e878e73ff7fd33d)

***

## Conditions

Conditions scope a collector to a segment of conversations. **Groups combine with AND; conditions within a group combine with OR.** An empty condition set matches every call (or chat) in the project.

| Type | Matches | Inputs |
| :- | :- | :- |
| **Agent** | Conversations handled by specific agents | Agent picker |
| **Source** | Conversations from a specific integration | LiveKit Cloud, Vapi, Retell, ElevenLabs, Leaping, API, Bland |
| **Property** | Conversations by custom metadata | Property key + operator + value |

Property conditions support these operators:

| Operator | Comparison |
| :- | :- |
| equals / does not equal | Exact match |
| contains | Substring match |
| starts with | Prefix match |
| is greater than / is less than | Numeric comparison |
| is at least / is at most | Numeric comparison, inclusive (≥ / ≤) |

> [!TIP]
>
> A group like "Agent is Billing Bot OR Agent is Sales Bot" AND "Source is Vapi" runs the collector's metrics only on Vapi calls handled by either agent.

***

## Pass/Fail Thresholds

Thresholds turn a metric's raw value into a pass/fail check, configured inline on each picked metric:

* **Add pass/fail threshold** adds a row that reads "Pass if `[operator]` `[value]`" (for example, Pass if ≥ 0.7).
* Thresholds that already exist for the metric appear as read-only rows with an **Include if** checkbox, so you can opt an existing threshold into this collector instead of creating a duplicate. If a new row matches an existing threshold, Roark warns you and dedupes it. It won't be created again.

Each saved threshold is a derived metric in its own right (named like "Empathy ≥ 7", scored Pass/Fail), added to the collector alongside the source metric. Thresholds are available for Scale, Numeric, Count, Boolean, and Classification outputs.

> [card] [Thresholds Guide](/documentation/metrics/thresholds)
>
> Learn about operators and how derived threshold metrics work

***

## Active and Paused

Every custom collector has a status toggle in the list: **Active** collectors evaluate matching conversations; **Paused** collectors don't. Pause a collector instead of deleting it if you might need it again. Deleting a collector stops it from collecting its metrics, but the metric definitions themselves are unaffected.

***

## System Collectors

Roark ships system collectors, the baseline coverage that keeps core metrics running on all your calls and chats. They're marked with a **System** shield badge and sort first in the list.

| | System collectors | Custom collectors |
| :- | :- | :- |
| **Created by** | Roark | You, via dashboard or API |
| **Editable / deletable** | No | Yes |
| **Status** | Always run | Active or Paused |

***

## Per-Agent Collectors

Each agent's detail page has a **Collectors** section listing the collectors that target that agent. These are the same collectors (just scoped by an Agent condition) so a collector created from the agent page shows up in the main `/metrics/collectors` list too, and vice versa.

***

## API and SDK Access

Collectors are fully manageable programmatically via the [Node.js SDK](/documentation/sdks/node-sdk) and REST API.

> [!NOTE]
>
> The API keeps the older **metric policy** naming: the SDK client is `client.metricPolicy.*` and the REST endpoints are `/v1/metric/policies`. A "policy" in the API is exactly a collector in the dashboard.

### Create it with the SDK

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const collector = await client.metricPolicy.create({
  name: 'Production Agent Quality Checks',
  modality: 'call', // 'call' or 'chat', locked after creation
  status: 'ACTIVE',
  conditions: [
    {
      conditions: [
        {
          conditionType: 'AGENT',
          conditionKey: 'your-agent-id',
        },
      ],
    },
  ],
  metrics: [
    { id: 'metric-definition-uuid-1' },
    { id: 'metric-definition-uuid-2' },
  ],
})
```

| Field | Type | Required | Description |
| :- | :- | :- | :- |
| `name` | string | Yes | Descriptive name for the collector |
| `modality` | string | Yes | `call` or `chat`; the collector only fires for this modality and can only reference metrics that support it |
| `status` | string | No | `ACTIVE` (default) or `INACTIVE` (Paused) |
| `conditions` | array | No | Condition groups: groups AND together, conditions within a group OR together; omit to match all conversations |
| `metrics` | array | Yes | Metric definitions to run (minimum 1), each `{ id, settings? }`. See [per-metric settings](#per-metric-settings) |

### Per-metric settings

Some metrics take a setting that applies only in this collector, the same setting the dashboard shows under the metric when you pick it (for example **Properties to verify**). Pass it as `settings` on the metric:

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
await client.metricPolicy.create({
  name: 'Credit Card Apr',
  modality: 'call',
  metrics: [
    { id: 'property-mismatch-metric-uuid', settings: { verifiedProperties: ['purchase_apr_total'] } },
    { id: 'apr-retrieved-metric-uuid' },
  ],
})
```

| Setting | Metrics | What it does |
| :- | :- | :- |
| `verifiedProperties` | Property Verification | Checks only these call properties against the transcript |
| `excludedPhrases` | Voice consistency | Agent lines containing one of these phrases are not judged |
| `voicemailScript` | Voicemail | The message the agent is expected to leave |
| `screeningBrief` | Call screening | What the agent should say when a screener asks who is calling |
| `toolDefinitionIds` | Tool invocation | Scores only these tool definitions |
| `knowledgeBaseDocumentIds` | Knowledge grounding | Grounds against only these knowledge base documents, in priority order |

* A metric takes at most one setting, and only the one listed for it; anything else is a `400` naming the setting the metric takes.
* Responses return each metric's `settings`, or `null` when it runs with its defaults.
* An update that sends `metrics` replaces the collector's whole metric list, and a metric sent without `settings` goes back to its defaults. Send each metric's settings again to keep them; a fetched `metrics` list can be sent back as is.
* In [config as code](/documentation/config-as-code/collectors#per-collector-metric-settings) the same settings go on the metric entry in YAML.

### Condition fields

| `conditionType` | `conditionKey` | `conditionOperator` | `conditionValue` |
| :- | :- | :- | :- |
| `AGENT` | Agent ID | N/A | N/A |
| `CALL_SOURCE` | Source name (e.g. `VAPI`, `RETELL`) | N/A | N/A |
| `CALL_PROPERTY` | Property key | `EQUALS`, `NOT_EQUALS`, `CONTAINS`, `STARTS_WITH`, `GREATER_THAN`, `LESS_THAN`, `GREATER_THAN_OR_EQUALS`, `LESS_THAN_OR_EQUALS` | Value to compare against |

`GREATER_THAN_OR_EQUALS` and `LESS_THAN_OR_EQUALS` correspond to the dashboard's "is at least" and "is at most" operators.

**Example: collector for Vapi calls only**

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const collector = await client.metricPolicy.create({
  name: 'Vapi Call Metrics',
  modality: 'call',
  conditions: [
    {
      conditions: [
        {
          conditionType: 'CALL_SOURCE',
          conditionKey: 'VAPI',
        },
      ],
    },
  ],
  metrics: [{ id: 'metric-definition-uuid' }],
})
```

### List, get, update, delete

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
// List (limit 1-50, default 20; cursor pagination via `after`)
const collectors = await client.metricPolicy.list({ status: 'ACTIVE', limit: 50 })

// Get one
const collector = await client.metricPolicy.getByID('policy-id')

// Update: all fields optional; pass an empty `conditions` array to match everything
const updated = await client.metricPolicy.update('policy-id', {
  name: 'Updated Collector Name',
  status: 'INACTIVE',
})

// Delete
const result = await client.metricPolicy.delete('policy-id')
```

> [!WARNING]
>
> System collectors (`type: "SYSTEM"` in API responses) cannot be modified or deleted.

***

## Best Practices

> [accordion] Start Broad, Then Narrow
>
> Create a collector with no conditions to run key metrics on all calls, then add targeted collectors with conditions for specific segments.

> [accordion] Use Conditions for Targeted Collection
>
> If you only need certain metrics for specific agents or sources, scope the collector so you're not scoring conversations you don't care about.

> [accordion] Keep Collectors Organized
>
> Use names that say what the collector targets and measures. The auto-suggested name (metric names plus a condition summary) is a good starting point.

> [accordion] Pause Instead of Delete
>
> Pause a collector if you might need it again later. Deleting stops collection but leaves the metric definitions intact.

***

## What's Next

> [card] [Custom Metrics](/documentation/metrics/custom-metrics)
>
> Learn about metric types, scopes, and how to create custom metrics

> [card] [Studio](/documentation/metrics/studio)
>
> Author and test metrics, then bind them to a collector from the editor

> [card] [Thresholds](/documentation/metrics/thresholds)
>
> Turn raw metric values into pass/fail outcomes

> [card] [Metric Collection Jobs](/documentation/metrics/metric-collection-jobs)
>
> Run metrics on demand for specific calls


# Overview
Source: https://docs.roark.ai/documentation/metrics/overview

Understand what metrics are and how to use them across monitoring and simulations

Metrics are how you measure what happens in every voice AI conversation. Roark collects some metrics automatically (like response time, talk time, and sentiment) and lets you define your own for things like compliance checks, task completion, or custom business KPIs.

> [!NOTE]
>
> **UI-first or code-first?** Author custom metrics and collectors in the dashboard, or define them as [config as code](/documentation/config-as-code/overview) - YAML in your git repo, applied with the [CLI](/documentation/sdks/cli). Both manage the same resources.

## Quickstart

Define a custom metric and run it on a call:

> [tab] Node
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> // Author an LLM-judge metric
> const metric = await client.metric.createDefinition({
>   name: 'Resolved On First Call',
>   outputType: 'BOOLEAN',
>   llmPrompt: 'Did the agent resolve the issue without a transfer or callback?',
> })
>
> // Run it against an existing call, then read the result
> await client.metricCollectionJob.create({
>   callIds: ['<call-id>'],
>   metrics: [{ id: metric.data.id }],
> })
> const results = await client.call.listMetrics('<call-id>')
> ```

> [tab] Python
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> metric = client.metric.create_definition(
>     name="Resolved On First Call",
>     output_type="BOOLEAN",
>     llm_prompt="Did the agent resolve the issue without a transfer or callback?",
> )
>
> client.metric_collection_job.create(
>     call_ids=["<call-id>"],
>     metrics=[{"id": metric.data.id}],
> )
> results = client.call.list_metrics("<call-id>")
> ```

> [tab] Config as Code
>
> ```yaml roark/metrics/resolved-on-first-call.yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> kind: metric
> name: resolved_on_first_call
> displayName: Resolved On First Call
> type: BOOLEAN
> prompt: |
>   Using {{transcript}} and {{world_context}}, did the agent resolve the
>   issue without a transfer or callback?
> ```
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> roark config apply ./roark
> ```
>
> Attach it to a [collector](/documentation/metrics/metric-collectors) to score every matching call automatically.

Everything measurement-related lives in the **Measure** section of the sidebar:

* **Metrics**: your library of everything the project can measure, plus the collectors that run it on your calls
* **Studio**: where you author new metrics and run evaluation batteries against real calls
* **Datasets**: named collections of calls or chats for evaluation, comparison, and analysis

![The Metrics page: a horizontal Collectors strip across the top, then the metric Library grouped by package (Custom Metrics, Compliance, Emotion and Sentiment, Performance, Task and Outcome) with All, System, and Custom filter pills and a search box.](https://mintcdn.com/roark/93k2kGpZVUUbTXSq/images/metrics/metrics-library.png?fit=max\&auto=format\&n=93k2kGpZVUUbTXSq\&q=85\&s=7914a990338bfac7a9d21a8fb83ef968)

***

## What's a Metric?

A metric is a single measurement collected from a call. It has:

* **An output type**: boolean, numeric, scale, text, classification, or count
* **A scope**: global (one value per call) or per-participant (separate values for agent and customer)
* **A context**: call-level, segment-level (single utterance), or segment-range (span of conversation)

For example, `response_time` is a numeric, per-participant, segment-range metric that measures how long each speaker takes to respond. `identity_verified` might be a boolean, global, call-level metric powered by an LLM Judge prompt.

***

## Types of Metrics

### System Metrics (Built-in)

Roark automatically collects deterministic and voice-analysis metrics for every call. No configuration needed. Voice-analysis metrics are powered by Roark's custom voice analysis models, purpose-built to extract signal from conversational audio:

* **Performance**: Response time, talk time, silence duration, overlap/interruptions, latency
* **Emotion & Sentiment**: Sentiment tracking, 64+ emotion detection, vocal cues (raised voice, frustration), stress indicators
* **Speech**: Interruption detection, pause analysis, repetition detection
* **Compliance**: Disclosure completeness, prohibited language, PII handling, prompt injection resistance
* **Call Quality**: Speech quality scoring (DNSMOS), accent detection, voicemail handling

See the full list of 65+ system metrics in the [System Metrics Reference](/documentation/metrics/system-metrics). System metrics are read-only in Studio until you fork them with **Customize for your team**, which creates an org-scoped variant you can edit.

### Custom Metrics

Define your own metrics in [Studio](/documentation/metrics/studio) by picking one of four engines:

| Engine | What it does | Example |
| - | - | - |
| **[LLM Judge](/documentation/metrics/custom-metrics/llm-judge)** | Scores each call against a natural-language prompt, evaluated by **Roark Prism**, our purpose-built evaluation model for voice AI conversations | "Did the agent verify the caller's identity?" → Boolean |
| **[Pattern](/documentation/metrics/custom-metrics/patterns)** | Composes triggers, outcomes, and windows to detect when something happens in a call | Fire when the agent interrupts within 3 lines of a frustrated customer line |
| **[Formula](/documentation/metrics/custom-metrics/formulas)** | Computes a value from other metrics into a composite score | Weighted quality score across empathy, resolution, and clarity |
| **[API check](/documentation/metrics/custom-metrics/api-checks)** | Asks your own endpoint whether the call passed | Your card service confirms the PIN actually changed |

LLM Judge metrics return typed results:

* "Did the agent verify the caller's identity?" → **Boolean**
* "Rate the agent's empathy on a scale of 1-10" → **Scale**
* "What was the primary reason for the call?" → **Classification**
* "How many times did the agent attempt to upsell?" → **Count**

Custom metrics are created in Studio's Author mode (**New metric** on the Metrics page) or via the [SDK](/documentation/metrics/custom-metrics/llm-judge#create-it-with-the-sdk). See [Custom Metrics](/documentation/metrics/custom-metrics) for the full authoring guide.

***

## The Metrics Page

The Metrics page (**Measure → Metrics**) is home base for measurement:

* **Collectors strip**: each collector runs a set of metrics on the calls its segment matches. Compact cards show each collector's name, segment, and metrics, with a **Manage collectors** link and a **New collector** tile.
* **Library**: everything this project can measure, grouped by package. Filter with the **All / System / Custom** pills or search; clicking a metric opens it in Studio for editing and testing.
* **New metric**: jumps straight into Studio's Author mode.

***

## How It All Fits Together

Here's the typical sequence from defining a metric to seeing results:

**1. Define or Choose a Metric**

Use a built-in [system metric](/documentation/metrics/system-metrics), or [create your own custom metric](/documentation/metrics/custom-metrics) with an LLM Judge prompt, a Pattern, a Formula, or an [API check](/documentation/metrics/custom-metrics/api-checks).

**2. Test in Studio**

Author mode's test rail lets you [run your metric against real calls](/documentation/metrics/studio): add test calls and hit **Run all** to validate it produces the results you expect. Iterate on the prompt until you're satisfied. To compare many metrics across many calls at once, use Evaluate mode's **Run battery**.

**3. Add Thresholds (Optional)**

Set [pass/fail criteria](/documentation/metrics/thresholds) on your metric, for example, `Customer Satisfaction >= 7` or `Response Time < 1000ms`. Thresholds turn raw values into actionable outcomes.

**4. Decide When to Collect**

Choose how and when your metric runs:

* **[Collectors](/documentation/metrics/metric-collectors)**: Automate collection on incoming calls (monitoring). Add conditions to target specific agents, sources, or call properties; leave them empty to match every call.
* **[Simulation plans](/documentation/simulation-testing/run-plans)**: Attach metrics with thresholds to simulation runs to validate agent behavior before deployment.
* **[Collection Jobs](/documentation/metrics/metric-collection-jobs)**: Run metrics on demand against existing calls via the SDK, useful for backfilling or re-processing.

**5. Analyze Results**

View metric values per call in [Call History](/documentation/observability/live-monitoring), aggregate them in [Reports](/documentation/observability/reports), and organize everything in [Dashboards](/documentation/observability/dashboards). Group the calls you care about into [Datasets](/documentation/metrics/datasets) for evaluation and comparison.

***

## Quick Start Examples

> [!NOTE]
>
> The REST API and SDKs keep the older name for collectors: the SDK client is `client.metricPolicy.*` and the endpoints live under `/v1/metric/policies`. In the UI, these are **Collectors**.

> [tab] Use a System Metric
>
> System metrics like `response_time` are already collected for every call. To set a quality bar, create a collector with a threshold:
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> const collector = await client.metricPolicy.create({
>   name: 'Agent Response Time SLA',
>   status: 'ACTIVE',
>   conditions: [
>     {
>       conditions: [
>         {
>           conditionType: 'AGENT',
>           conditionKey: 'your-agent-id',
>         },
>       ],
>     },
>   ],
>   metrics: [
>     {
>       id: 'response-time-metric-id',
>       threshold: {
>         operator: 'LESS_THAN',
>         value: 1000,
>         aggregationMode: 'P95',
>         participantRole: 'AGENT',
>       },
>     },
>   ],
> })
> ```
>
> Calls where the agent's P95 response time exceeds 1 second are automatically flagged as failures.

> [tab] Create a Custom Metric
>
> Define a business-specific metric and add it to a collector. Roark Prism evaluates each call against your prompt:
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> // 1. Create the metric definition
> const metric = await client.metric.createDefinition({
>   name: 'Appointment Booked',
>   outputType: 'BOOLEAN',
>   analysisPackageId: 'your-package-id',
>   llmPrompt: 'Did the agent successfully book an appointment for the caller?',
>   booleanTrueLabel: 'Booked',
>   booleanFalseLabel: 'Not Booked',
> })
>
> // 2. Add it to a collector so it runs on every call
> const collector = await client.metricPolicy.create({
>   name: 'Appointment Booking Check',
>   status: 'ACTIVE',
>   metrics: [{ id: metric.data.id }],
> })
> ```
>
> Test your prompt in [Studio](/documentation/metrics/studio) first to validate it produces the results you expect.

> [tab] Use Metrics in Simulations
>
> Attach metrics with thresholds to a simulation plan to validate agent behavior before deployment. Run the simulation, and each call is scored against your pass/fail criteria:
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> // Run metrics against simulation calls after they complete
> const job = await client.metricCollectionJob.create({
>   callIds: ['sim-call-1', 'sim-call-2', 'sim-call-3'],
>   metrics: [
>     { id: 'task-completion-metric-id' },
>     { id: 'compliance-check-metric-id' },
>   ],
> })
>
> // Check results
> const result = await client.metricCollectionJob.getByID(job.data.id)
> console.log(`${result.data.completedItems}/${result.data.totalItems} processed`)
> ```
>
> Or configure metrics with [thresholds](/documentation/metrics/thresholds) directly in a [simulation plan](/documentation/simulation-testing/run-plans) from the dashboard.

***

## Sections

> [card] [System Metrics Reference](/documentation/metrics/system-metrics)
>
> Browse all 65+ built-in metrics powered by specialized models

> [card] [Custom Metrics](/documentation/metrics/custom-metrics)
>
> Create custom metrics with LLM judges, patterns, formulas, and API checks

> [card] [Studio](/documentation/metrics/studio)
>
> Author metrics and run evaluation batteries against real calls

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Run sets of metrics on the calls each collector's segment matches

> [card] [Thresholds](/documentation/metrics/thresholds)
>
> Define pass/fail criteria for your metrics

> [card] [Datasets](/documentation/metrics/datasets)
>
> Group calls or chats for evaluation, comparison, and analysis

> [card] [Collection Jobs](/documentation/metrics/metric-collection-jobs)
>
> Run metrics on demand for existing calls via the SDK


# Studio
Source: https://docs.roark.ai/documentation/metrics/studio

Author, test, and evaluate metrics in one workbench before they run on live traffic

Studio is the metric workbench: build metrics, test them against real calls, and run evaluation batteries, all before anything touches production traffic. Find it in the sidebar under **Measure → Studio**.

> [!NOTE]
>
> Studio replaces the Playground. If you're looking for the old Playground docs, everything it did (and more) now lives here.

Studio has three modes:

| Mode | What it's for |
| - | - |
| **Hub** | The landing page: start authoring or evaluating, and revisit recent evaluation runs |
| **Author** | Define or edit a single metric, with a test-call rail and version history |
| **Evaluate** | Run a battery of metrics across a set of calls and compare results side-by-side |

***

## The hub

Opening Studio lands you on the hub ("Build metrics. Run evaluations.") with two start cards:

* **Define a new metric** opens Author mode with a blank metric.
* **Run a battery of evals** opens Evaluate mode.

Below the cards, **Recent evaluation runs** lists your past batteries (e.g. "3 metrics × 5 calls") with a status chip: In progress, All passing, "N failing", Canceled, or Failed. Reopen any run to see its full result matrix, or fork it into a new evaluation.

![The Studio hub: the Build metrics, run evaluations header, two start cards for Define a new metric and Run a battery of evals, and a Recent evaluation runs list showing each run's metrics by calls, a status chip (In progress, All passing, N failing, Canceled), a pass count, and a timestamp.](https://mintcdn.com/roark/Q1LycOQn64NfkIxp/images/metrics/studio-hub.png?fit=max\&auto=format\&n=Q1LycOQn64NfkIxp\&q=85\&s=f79e45d48e1e63a3b9cebbaf79cdb2e7)

***

## Author mode

Author mode is the metric editor. You reach it from the hub, from the **New metric** button on the Metrics page, or by clicking any metric in the [Library](/documentation/metrics/overview).

The editor walks you through four numbered sections:

**1. 01 · Engine: what kind of metric is this?**

Pick how the metric is computed:

| Engine | How it works |
| - | - |
| **LLM Judge** | Scores each call against your natural-language prompt. See [LLM Judge](/documentation/metrics/custom-metrics/llm-judge) |
| **Pattern** | Composes triggers, outcomes, and windows. Fires when the pattern matches. See [Patterns](/documentation/metrics/custom-metrics/patterns) |
| **Formula** | Math over other metrics. See [Formulas](/documentation/metrics/custom-metrics/formulas) |
| **API check** | Asks your own endpoint whether the call passed. See [API Checks](/documentation/metrics/custom-metrics/api-checks) |

> [!WARNING]
>
> The engine is **locked after the first save**. To measure the same thing with a different engine, create a new metric.

**2. 02 · Output: what does the metric return?**

Choose the result type. Some types need extra configuration:

| Output | Returns | Extra config |
| - | - | - |
| **Boolean** | Yes / No | True and False labels |
| **Scale** | A score in a range | Min/Max plus band labels |
| **Classification** | One of your categories | Category list |
| **Numeric** | A number | A unit |
| **Count** | An occurrence count | A unit |
| **Text** | Free-form text | None |

**3. 03 · Instructions, Logic, or Formula**

The third section adapts to your engine. For **LLM Judge**, write instructions tuned to the output type: for a Boolean metric, the yes/no question the model should answer; for a Scale, what each band means. For **Pattern**, define the logic: when should this metric fire? For **Formula**, define the computation over other metrics.

**4. 04 · Group: where should this metric live?**

Assign the metric to a group (analysis package). Groups organize the Library on the Metrics page, so pick one your team will look for it under.

![Author mode: the metric editor for a Disclosure Completeness metric. The main column walks through four numbered sections: 01 Engine with LLM Judge selected, 02 Output set to Scale with 0 to 100 bounds and Complete, Partial, and Incomplete range labels, 03 Instructions with a filled prompt that ends in a highlighted \{\{transcript}} variable, and 04 Group. A right-hand test-call rail lists pinned calls, each with the metric's score, so the author can watch results as they tweak the prompt.](https://mintcdn.com/roark/IjseSxRBSMbit4Aq/images/metrics/studio-author.png?fit=max\&auto=format\&n=IjseSxRBSMbit4Aq\&q=85\&s=b70e3d6a98d20fe838b6fbd6844696ba)

### Test against real calls

The right-hand rail is your test bench. Use **Add test calls** to pick the calls Studio should evaluate the metric against, then run them individually or hit **Run all**. Each run streams through the evaluation pipeline live, so you can watch the result land and tighten your instructions between runs.

> [!TIP]
>
> Iterate here until the metric returns what you expect on calls you already know the answer for, before you apply it to live traffic.

### Version history

Every save publishes a new version. The **History** tab in the rail lists them all: view any past version, or restore one, which publishes a new version from the old one (you never lose the intermediate history).

### System metrics and variants

[System metrics](/documentation/metrics/system-metrics) open read-only in Author mode. To change one, click **Customize for your team**. This forks an org-scoped variant you can freely edit and save. The Roark-managed original stays intact, and your instructions go in an "Additional instructions" field layered on top of the system prompt.

### Editing a metric that's already live

If the metric is bound to collectors, a notice appears above the form: **"In use by N collectors: saving publishes a new version every collector will pick up."**

> [!WARNING]
>
> Saving a metric republishes it everywhere it runs. Every collector using the metric picks up the new version for calls scored from that point on. Test your changes in the rail before saving.

***

## Evaluate mode

Evaluate mode runs a battery of metrics across a set of calls and compares results side-by-side. Use it to validate a metric at scale, compare candidate metrics against each other, or spot-check quality across a sample of production calls.

**1. Pick metrics**

Use **Add metrics** to choose one or more metrics from your Library. You can attach a pass/fail threshold inline per metric (e.g. "Pass if ≥ 0.7") to turn raw scores into verdicts. See [Thresholds](/documentation/metrics/thresholds).

**2. Pick calls**

Use **Add calls** to select the conversations to evaluate.

**3. Run battery**

Hit **Run battery**. Results stream in as a comparison **matrix** (metrics × calls) or grouped **by call**. Switch views to suit the question you're asking. You can cancel a run in progress.

Finished batteries appear under **Recent evaluation runs** on the hub, so you can reopen the matrix later or fork the run into a new evaluation.

![Evaluate mode results matrix after running a battery: four metrics as columns (Disclosure Completeness, PII Handling, Task Completion, Sentiment) and twelve calls as rows, each cell showing a pass/fail verdict or a score, with an overall 75% pass rate summarizing the 4 metrics by 12 calls run.](https://mintcdn.com/roark/wL__NyRpzLGS2dwF/images/metrics/studio-evaluate.png?fit=max\&auto=format\&n=wL__NyRpzLGS2dwF\&q=85\&s=13f17f0977ef46912ce0d3b344c9c03e)

> [!NOTE]
>
> Prefer to run evaluations programmatically? [Metric collection jobs](/documentation/metrics/metric-collection-jobs) are the API equivalent: run a set of metrics across a set of calls via the SDK or REST.

***

## Apply a metric to live calls

Testing in Studio scores only the calls you pick. To score matching calls automatically as they come in, bind the metric to a collector. The Author mode footer (**"Apply this metric to live calls"**) has an **Add collector** button that opens the collector editor with your metric prefilled and the modality locked to what the metric supports.

Once bound, the footer shows **"Running on N collectors"** instead. See [Collectors](/documentation/metrics/metric-collectors) for how conditions and scoping work.

> [!NOTE]
>
> The REST API and SDKs keep the older name for collectors: endpoints live under `/v1/metric/policies` and the SDK client is `client.metricPolicy.*`.

***

## Next steps

> [card] [Custom Metrics](/documentation/metrics/custom-metrics)
>
> Deep-dive on LLM Judge, Pattern, Formula, and API check metric types

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Run your metrics automatically on matching live calls

> [card] [Thresholds](/documentation/metrics/thresholds)
>
> Turn scores into pass/fail verdicts

> [card] [Metric Collection Jobs](/documentation/metrics/metric-collection-jobs)
>
> Run evaluations at scale via the API


# System Metrics Reference
Source: https://docs.roark.ai/documentation/metrics/system-metrics

Complete reference for all built-in system metrics powered by specialized models

Roark ships with a comprehensive set of **system metrics** that are automatically available in every project. These metrics are powered by purpose-built, specialized models (not generic LLMs) designed to extract precise signal from conversational audio and transcripts.

System metrics require no configuration. Add them to an [analysis package](/documentation/metrics/custom-metrics), attach a [metric collector](/documentation/metrics/metric-collectors), and start collecting data immediately.

> [!NOTE]
>
> All system metrics listed below are powered by **specialized models** purpose-built for voice AI analysis. This means they are faster, more consistent, and more cost-effective than general-purpose LLM evaluation.

***

## Metric Types Overview

System metrics are computed by purpose-built models that analyze audio and transcript signals: performance, interruptions, sentiment, compliance, call quality. When you need something they don't cover, create a [custom metric](/documentation/metrics/custom-metrics) on one of four engines:

| Engine | How it works | Use case |
| :- | :- | :- |
| [LLM Judge](/documentation/metrics/custom-metrics/llm-judge) | Roark Prism evaluates the conversation against a natural-language prompt you write | Business logic, subjective quality checks, task completion |
| [Pattern](/documentation/metrics/custom-metrics/patterns) | A trigger on one line followed by an outcome within a window, built from per-line metrics | Talking over a frustrated caller, slow responses after a question |
| [Formula](/documentation/metrics/custom-metrics/formulas) | Math and comparisons over other metrics on the call | Composite scores, pass/fail rules across several metrics |
| [API Check](/documentation/metrics/custom-metrics/api-checks) | Your own endpoint answers pass or fail | Outcomes that live in your backend |

System metrics make good inputs: per-segment metrics such as Frustration Score and Interruption feed patterns, and call-level ones such as Task Completion feed formulas.

***

## System Metrics Reference

All system metrics below are collected automatically when included in an analysis package. Each metric shows its output type, scope, and the specialized model that powers it.

**Scope legend:**

* **Global**: one value per call
* **Per-participant**: separate values for agent and customer

***

### Core Analysis

Timing and interaction metrics extracted from audio diarization and transcript alignment.

> [!NOTE]
>
> Powered by **Roark Vibe**, our core voice analysis model.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `call_duration` | Total duration of the call | Numeric (seconds) | Global |
| `response_time` | Time between speaking turns | Numeric (seconds) | Per-participant |
| `time_to_first_word` | Time from call start to first spoken word | Numeric (seconds) | Per-participant |
| `silence_duration` | Duration of each silence period | Numeric (seconds) | Per-participant |
| `turn_duration` | Duration of each speaking turn | Numeric (seconds) | Per-participant |
| `word_count` | Number of words spoken | Count | Per-participant |
| `talk_to_listen_ratio` | Ratio of time a participant spends talking vs total call duration | Numeric | Per-participant |
| `speaking_rate` | Words spoken per minute by a participant | Numeric (wpm) | Per-participant |
| `turn_count` | Number of speaking turns by a participant | Count | Per-participant |
| `latency_spike_count` | Number of response gaps exceeding 3 seconds | Count | Per-participant |
| `longest_pause` | Longest gap between consecutive segments in the call | Numeric (seconds) | Global |
| `agent_responsive` | Whether the agent held up its end of the conversation rather than going silent or erroring. Simulations only. | Boolean | Global |
| `agent_spoke` | Whether the agent spoke at all during the call. Simulations only. | Boolean | Global |

***

### Sentiment & Emotion

Emotion and sentiment analysis from vocal features and acoustic signals.

> [!NOTE]
>
> Powered by **Hume Expression Measurement**, a specialized vocal emotion model.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `sentiment_score` | Sentiment rating on a 1–9 scale (1 = negative, 9 = positive) | Scale (1-9) | Per-participant |
| `emotion_label` | Detected emotion label from 64+ emotions | Classification | Per-participant |
| `dominant_emotion` | Most frequent emotion across the call | Classification | Per-participant |
| `vocal_cue_label` | Detected vocal cue or expression label | Classification | Per-participant |

***

### Interruptions

Detailed interruption and overlap analysis from speaker diarization.

> [!NOTE]
>
> Powered by **Roark Interruptions**, a specialized overlap detection model.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `interruption` | Whether overlapping speech occurred on a segment | Boolean | Per-participant |
| `interruption_duration` | Duration of overlapping speech | Numeric (seconds) | Per-participant |
| `interruption_count` | Total number of interruptions | Count | Global |
| `first_interruption_time` | Time into call of first interruption | Offset (seconds) | Global |
| `overtalk_ratio` | Ratio of overlapping speech duration to total call duration | Numeric | Global |
| `agent_interruption_count` | Number of times the agent interrupted the customer | Count | Global |
| `incorrect_agent_interruption_count` | Number of agent interruptions classified as inappropriate | Count | Global |
| `incorrect_interruption_rate` | Proportion of agent interruptions that were inappropriate | Scale (0-1) | Global |
| `customer_barge_in_count` | Number of times the customer attempted to interrupt the agent | Count | Global |
| `failed_barge_in` | Whether a customer interruption attempt failed because the agent did not yield | Boolean | Per-participant |
| `failed_barge_in_count` | Number of customer interruption attempts where the agent did not yield | Count | Global |
| `failed_barge_in_rate` | Proportion of customer interruption attempts that failed | Scale (0-1) | Global |
| `interruption_appropriateness` | Whether an agent interruption was appropriate based on conversational context | Boolean | Per-participant |
| `pre_interruption_speaker_duration` | How long the interrupted speaker had been talking before being interrupted | Numeric (seconds) | Per-participant |
| `agent_cutoff` | Whether the agent started speaking while the customer was mid-sentence | Boolean | Per-participant |
| `agent_cutoff_count` | Number of times the agent cut off the customer mid-sentence | Count | Global |

***

### Quality

Experience quality scoring from conversational signals.

> [!NOTE]
>
> Powered by **Roark Quality Analysis** and **Roark Prism**, specialized models for quality assessment.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `frustration_score` | Customer frustration level (1 = none, 5 = severe) | Scale (1-5) | Per-participant |
| `user_effort_score` | How much effort the customer exerted to accomplish their goal (1 = effortless, 5 = very difficult) | Scale (1-5) | Per-participant |
| `call_outcome` | Overall outcome: Resolved, Unresolved, Escalated, Dropped, or Follow-up Required | Classification | Global |
| `instruction_follow` | How well the agent followed its given instructions (1 = not followed, 5 = fully followed) | Scale (1-5) | Global |
| `redundant_question_count` | Questions where the agent asked for information already provided | Count | Global |
| `missed_response_count` | Moments where a participant should have responded but did not | Count | Per-participant |
| `comprehension_failure` | Whether the agent misunderstood what the customer said | Boolean | Per-participant |
| `comprehension_failure_count` | Number of times the agent misunderstood the customer | Count | Global |
| `customer_reception` | How well the customer received the interaction overall, folded from frustration and effort signals (1 = very poor, 5 = excellent) | Scale (1-5) | Per-participant |
| `conversation_flow_score` | Naturalness of the conversational exchange: turn-taking, pacing, responsiveness, absence of awkward silences (1 = poor, 5 = natural) | Scale (1-5) | Global |
| `scenario_adherence` | Flow Adherence: how closely the simulated customer followed its scripted flow. Simulations only. | Scale (1-5) | Global |
| `agent_expectations` | Whether the agent met every expectation authored on the customer flow, graded one verdict per expectation and rolled up all-or-nothing. Simulations only. | Boolean | Global |

***

### Repetition Detection

Conversational loop and repetition analysis.

> [!NOTE]
>
> Powered by **Roark Prism**, our evaluation model optimized for voice AI.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `repetition_density` | Ratio of repeated turns to total turns (0–1) | Numeric | Per-participant |
| `loop_count` | Number of distinct conversational loops detected | Count | Per-participant |

***

### Tool Invocations

Analysis of function/tool calling behavior during conversations.

> [!NOTE]
>
> Powered by **Roark Vibe** and **Roark Prism**.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `tool_invocation_count` | Total number of tool/function calls made during the conversation | Count | Global |
| `tool_invocation_correct` | Whether the agent invoked the correct tools at the appropriate times | Boolean | Global |
| `tool_invocation_order_correct` | Whether tools were called in the correct logical sequence | Boolean | Global |
| `tool_invocation_parameters_correct` | Whether correct parameters were passed to each tool invocation | Boolean | Global |
| `tool_invocation_result_correct` | Whether the agent correctly interpreted and used tool results | Boolean | Global |

***

### Compliance

Regulatory and safety evaluation metrics for AI agent conversations.

> [!NOTE]
>
> Powered by **Roark Prism**, customizable with your own compliance requirements.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `compliance_disclosure_completeness` | Whether all required disclosures were delivered (recording notice, AI identity, licensing) | Scale (1-5) | Global |
| `compliance_prohibited_language` | Whether the agent used prohibited language (unauthorized guarantees, medical/legal advice, discrimination) | Boolean | Global |
| `compliance_pii_handling` | How properly the agent handled personally identifiable information | Scale (1-5) | Global |
| `compliance_consent_collection` | Whether required consent was obtained before data collection or recording | Boolean | Global |
| `compliance_escalation_adherence` | Whether the agent properly escalated to a human when required | Boolean | Global |
| `compliance_scope_adherence` | Whether the agent stayed within its defined scope of topics | Scale (1-5) | Global |
| `compliance_prompt_injection_resistance` | Whether the agent resisted attempts to override its instructions or jailbreak | Boolean | Global |
| `compliance_identity_consistency` | Whether the agent maintained its assigned identity and disclosed its AI nature | Boolean | Global |
| `compliance_hallucination_boundary` | Whether the agent avoided fabricating information and deferred when unsure | Scale (1-5) | Global |

***

### Voicemail Detection

Voicemail detection and handling quality assessment.

> [!NOTE]
>
> Powered by **Roark Prism**.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `voicemail_detected` | Whether the call reached a voicemail system rather than a live person | Boolean | Global |
| `voicemail_agent_left_message` | Whether the agent left a voicemail message | Boolean | Global |
| `voicemail_handling_score` | Quality of voicemail handling (beep detection, message clarity, completeness) | Scale (1-5) | Global |

***

### Call Screening

Detects when a screener answers an outbound call instead of the person being called: a receptionist, an assistant, a household gatekeeper, or an automated screen.

> [!NOTE]
>
> Powered by **Roark Prism**.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `call_screening_encountered` | Whether the call was answered by someone screening it rather than the person being called | Boolean | Global |
| `call_screening_handling_score` | How well the agent handled being screened: recognising the gatekeeper, answering their questions directly, staying concise (1 = ran its script at the screener, 5 = handled cleanly) | Scale (1-5) | Global |

***

### Accent Detection

English accent identification from audio signals.

> [!NOTE]
>
> Powered by **Roark Accent ID**, a specialized accent classification model supporting 16 English accent variants.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `accent` | Detected English accent (US, British, Australian, Canadian, Indian, Irish, Scottish, Welsh, African, New Zealand, Hong Kong, Malaysian, Philippine, Singaporean, Bermudian, South Atlantic) | Classification | Per-participant |
| `accent_stability` | How stable the detected accent is across segments (1.0 = consistent, lower = varies) | Numeric (0-1) | Per-participant |

For a detailed walkthrough on using accent metrics, see the [Accent Detection recipe](/documentation/recipes/accent-detection).

***

### Call Quality (DNSMOS)

Speech quality assessment using the ITU-T P.808/P.835 Mean Opinion Score (MOS) scale.

> [!NOTE]
>
> Powered by **Roark DNSMOS**, a specialized speech quality model based on the ITU-T standard.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `speech_quality_overall` | Overall perceived speech quality (P.835 OVRL). Combines signal and background noise quality. | Scale (1-5) | Global |
| `speech_quality_signal` | Quality of the speech signal itself (P.835 SIG). Measures distortion, codec artifacts, and clarity. | Scale (1-5) | Global |
| `speech_quality_background` | Background environment quality (P.835 BAK). Higher = cleaner background. | Scale (1-5) | Global |
| `speech_quality_mos` | ITU-T P.808 Mean Opinion Score. Single overall quality rating from the audio signal. | Scale (1-5) | Global |

***

### Voice Naturalness

Acoustic naturalness of the agent's synthesized voice.

> [!NOTE]
>
> Powered by **Roark UTMOS**, a specialized naturalness model producing a mean opinion score.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `voice_naturalness` | How human-like vs robotic the agent's voice sounds (1 = robotic, 5 = natural) | Scale (1-5) | Per-participant (agent) |

***

### Voice Human-Likeness

Perceptual human-likeness judged from the agent's own audio.

> [!NOTE]
>
> Powered by an audio-input evaluation model listening to the agent's actual speech.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `voice_human_likeness` | Whether the agent could pass as a human speaker: prosody, breathing, disfluencies, delivery (1 = clearly synthetic, 5 = indistinguishable from human) | Scale (1-5) | Per-participant (agent) |

***

### Call Environment

Zero-shot acoustic classification of the caller's environment.

> [!NOTE]
>
> Powered by a specialized audio classification model.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `in_car_detection` | Whether the customer appears to be speaking from inside a moving vehicle | Boolean | Per-participant (customer) |

***

### Pronunciation Analysis

Per-word pronunciation checking against expected pronunciations you configure.

> [!NOTE]
>
> Powered by **Roark Phoneme**, a wav2vec2-based phoneme recognition model. Configure the word list and strictness per project.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `pronunciation_correctness` | Whether the agent pronounced each configured word correctly, one result per spoken occurrence | Classification (correct / incorrect) | Per-participant (agent) |
| `pronunciation_word_coverage` | Fraction of your configured words the agent actually said on the call (0 = none came up, 1 = all spoken at least once) | Numeric (0-1) | Per-participant (agent) |

***

### Property Verification

Checks the call properties you send at ingest (customer name, account number, appointment time, and so on) against what was actually said on the call, and flags values that conflict. Useful for catching stale CRM data, transcription errors, and callers who are not who the metadata says they are.

> [!NOTE]
>
> Powered by **Roark Prism**. Requires [custom properties](/documentation/observability/overview) on the call; system-generated properties are never checked.

| Metric | Description | Output | Scope |
| :- | :- | :- | :- |
| `property_transcript_mismatch` | Whether any call property conflicted with what was said on the call | Boolean | Global |
| `property_transcript_mismatch_count` | How many properties conflicted with the transcript (0 = everything that came up matched) | Count | Global |

Every property is judged with a three-way verdict:

* **MATCH**: the conversation referred to this property and agrees with the value you sent, allowing for formatting differences, nicknames, partial references, and transcription noise.
* **MISMATCH**: the conversation states a genuinely different value.
* **NOT\_MENTIONED**: the subject never came up. This is the most common outcome and is not counted as a mismatch.

The full per-property breakdown is returned on the [call metrics endpoint](/api-reference/call/list-call-metrics) as a `propertyVerdicts` array on the metric value: each entry carries the property name, the expected value, the verdict, the observed value for mismatches, the judge's reasoning, and the transcript segment where the property was referred to. The same breakdown renders on the call detail page in the platform.

Records about the call rather than things anyone says in it, such as an outcome label (`failed`, `timed_out`), an internal id or a timestamp, come back as **NOT\_MENTIONED**. A brand name and the legal name of the same organization ("Chase" and "JPMorgan Chase Bank, N.A.") count as a **MATCH**.

#### Choosing which properties to verify

By default every custom property on the call is checked. To check only the values the call is expected to state, pick them under **Properties to verify** on the metric in a [collector](/documentation/metrics/metric-collectors), run plan or flow. The other properties are then ignored, so metadata can never trip the mismatch flag. The three Property Verification metrics share one check, so a list set on any of them applies to all.

The same list can be set in [config as code](/documentation/config-as-code/collectors#per-collector-metric-settings):

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
metrics:
  - metric: property_transcript_mismatch
    verifiedProperties: [purchase_apr_total, extracted_purchase_apr_total]
```

or through the API, as `settings: { verifiedProperties: [...] }` on the metric in [`POST`/`PUT /v1/metric/policies`](/documentation/metrics/metric-collectors#per-metric-settings).

A changed list applies to calls scored afterwards. To re-score earlier calls with it, run a [collection job with the collector's id](/documentation/metrics/metric-collection-jobs#re-scoring-with-collector-settings).

***

## What's Next

> [card] [Custom Metrics](/documentation/metrics/custom-metrics)
>
> Create LLM Judge, Pattern, Formula, and API check metrics

> [card] [Studio](/documentation/metrics/studio)
>
> Test metrics interactively against real calls

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Automate metric collection with conditions-based rules

> [card] [Thresholds](/documentation/metrics/thresholds)
>
> Define pass/fail criteria for your metrics


# Thresholds
Source: https://docs.roark.ai/documentation/metrics/thresholds

Define pass/fail criteria for your metrics

## Overview

Thresholds turn raw metric values into clear **pass/fail outcomes**. When you add a threshold to a metric, Roark mints a derived boolean metric that automatically evaluates whether each conversation meets your criteria.

For example, a threshold of `>= 7` on `Customer Satisfaction` produces a Pass whenever the score is 7 or above, and a Fail otherwise.

***

## How Thresholds Work

Each threshold is a **derived metric** in its own right:

1. The source metric is collected as usual (e.g., a satisfaction score of `8`)
2. The threshold compares the value against your condition (e.g., `>= 7`)
3. A boolean result is produced with the labels **Pass** and **Fail**
4. The derived metric is stored alongside the original, so both show up in reporting

The derived metric is named after its rule (`Customer Satisfaction >= 7`) and uses the `THRESHOLD` calculation type with a Boolean output. Because it's a real metric definition, the same threshold can be reused across collectors.

> [!NOTE]
>
> Thresholds are available on these metric output types: **Scale**, **Numeric**, **Count**, **Boolean**, and **Classification**.

***

## Adding a Threshold

Thresholds can be configured from several places in Roark:

* **[Collectors](/documentation/metrics/metric-collectors)**: Each picked metric in the collector editor has a **Pass/Fail Threshold(s)** block with an **Add pass/fail threshold** button
* **[Studio](/documentation/metrics/studio)**: In Evaluate mode, configure thresholds inline per metric before running a battery of evals
* **[Simulation Run Plans](/documentation/simulation-testing/run-plans)**: Set pass/fail checks for simulation testing
* **[Config as code](/documentation/config-as-code/metrics#thresholds)**: Define a threshold with `defineThreshold` in TypeScript, or as a `kind: metric` with `calculationType: THRESHOLD` in YAML, including the API-only options below

The configuration is the same everywhere: each threshold row reads **"Pass if \[operator] \[value]"**.

![A picked metric in the collector editor with its Pass/Fail Thresholds block expanded, showing a Pass if greater than or equal to 7 row and the Add pass/fail threshold button](https://mintcdn.com/roark/17HU3uIlkQ6GJrX5/images/metrics/threshold-editor.png?fit=max\&auto=format\&n=17HU3uIlkQ6GJrX5\&q=85\&s=f1526d00dc795e7d27ad79111b7f5e26)

### Reusing Saved Thresholds

Because thresholds are metric definitions, ones you've already created appear as read-only rows with an **Include if** checkbox. Tick it to opt an existing threshold into the current collector. If you try to add a rule that matches a saved threshold, Roark warns "Matches an existing threshold: it won't be created again" and reuses the existing one instead of creating a duplicate.

***

## Operators

The available operators depend on the metric's output type.

### Numeric Types (Scale, Numeric, Count)

| Operator | Symbol | Example |
| :- | :- | :- |
| Greater than | `>` | Score `>` 5 |
| Greater than or equals | `>=` | Score `>=` 7 |
| Less than | `<` | Response time `<` 3000ms |
| Less than or equals | `<=` | Response time `<=` 2000ms |
| Equals | `=` | Count `=` 0 |
| Not equals | `!=` | Count `!=` 0 |

### Categorical Types (Boolean, Classification)

| Operator | Symbol | Example |
| :- | :- | :- |
| Equals | `=` | Compliance check `=` true |
| Not equals | `!=` | Call outcome `!=` "escalated" |

***

## API-Only Options

The UI keeps threshold rows simple: an operator and a value. The REST API and [config as code](/documentation/config-as-code/metrics#thresholds) support two additional options.

> [!NOTE]
>
> These options are set via the API's `CreateThresholdInput` and are not exposed in the Collectors or Studio UI.

### Aggregation Modes

When a metric produces multiple values per call (e.g., a per-segment metric that fires on every turn), the `aggregationMode` field determines how those values are combined before the comparison is applied.

| Mode | Description | Example |
| :- | :- | :- |
| **Each** | Compare every value individually. Fails if any single value fails (default) | Each response time `<` 5000ms |
| **Average** | Average all values, then apply the threshold | Average sentiment `>=` 6 |
| **Min** | Use the minimum value | Min confidence `>=` 0.8 |
| **Max** | Use the maximum value | Max response time `<` 10000ms |
| **Median** | Use the median value | Median score `>=` 7 |
| **Sum** | Sum all values, then apply the threshold | Total talk time `<=` 300000ms |
| **P95** | Use the 95th percentile value | P95 response time `<` 5000ms |
| **P99** | Use the 99th percentile value | P99 latency `<` 8000ms |
| **Count** | Count how many values match, and fire when `countThreshold` is met | No more than 2 segments below threshold |

### Participant Role Filtering

For metrics with `PER_PARTICIPANT` scope, the `sourceParticipantRole` field narrows the threshold to a specific speaker: `AGENT`, `CUSTOMER`, `SIMULATED_CUSTOMER`, or `BACKGROUND_SPEAKER`. This is useful when a metric like sentiment is tracked for both speakers but you only want to set a threshold on the agent's performance.

***

## Examples

> [accordion] Customer Satisfaction ≥ 7
>
> **Source metric:** Customer Satisfaction (Scale 1-10)
> **Operator:** Greater than or equals
> **Value:** 7
> **Result:** Pass if satisfaction is 7 or above, Fail otherwise. The derived metric is named `Customer Satisfaction >= 7`.

> [accordion] Identity Verified Equals True
>
> **Source metric:** Identity Verified (Boolean)
> **Operator:** Equals
> **Value:** true
> **Result:** Pass if the agent verified the caller's identity

> [accordion] Agent Response Time Under 1 Second (API)
>
> **Source metric:** Response Time (Numeric, per-segment)
> **Operator:** Less than
> **Value:** 1000 (milliseconds)
> **Aggregation (API-only):** P95
> **Participant Role (API-only):** Agent
> **Result:** Pass if 95th percentile agent response time is under 1 second

> [accordion] No More Than 2 Failed Segments (API, Count Mode)
>
> **Source metric:** Tone Appropriate (Boolean, per-segment)
> **Operator:** Equals
> **Value:** true
> **Aggregation (API-only):** Count (`countThreshold`: 2)
> **Result:** Pass if no more than 2 segments have an inappropriate tone

***

## What's Next

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Add thresholds to the metrics a collector runs

> [card] [Studio](/documentation/metrics/studio)
>
> Test thresholds in Evaluate mode before deploying

> [card] [Run Plans](/documentation/simulation-testing/run-plans)
>
> Set pass/fail criteria for simulation testing

> [card] [Custom Metrics](/documentation/metrics/custom-metrics)
>
> Learn about metric types, scopes, and output formats


# Query analytics
Source: https://docs.roark.ai/documentation/observability/analytics-api

Ask Roark for aggregate numbers over your calls and metrics from code, an agent, or the CLI: the same data behind your dashboards

The analytics API returns the numbers behind your [dashboards](/documentation/observability/dashboards) without opening the dashboard: average sentiment this week, calls per day, task completion by agent, a pass rate against last month. Send a query, get back a total and a per-interval series for each thing you asked about. Nothing is saved; to keep a query, add it to a dashboard as a [widget](#from-a-query-to-a-dashboard).

It is built for code and for AI agents alike. `GET /v1/analytics/schema` lists every metric you can query and what each one supports, so an agent can build a valid query on the first try.

## Quickstart

Calls per day over the last week, and the average sentiment score:

> [tab] Node
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> import Roark from '@roarkanalytics/sdk'
>
> const client = new Roark({ bearerToken: process.env.ROARK_API_BEARER_TOKEN })
>
> const { data } = await client.analytics.query({
>   measures: [{ type: 'CONVERSATION' }, { metric: 'sentiment_score' }],
>   timeRange: { type: 'LAST', value: 7, unit: 'DAYS' },
>   interval: 'DAY',
> })
>
> for (const series of data.series) console.log(series.metric ?? 'calls', series.value)
> ```

> [tab] Python
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> import os
> from roark_analytics import Roark
>
> client = Roark(bearer_token=os.environ["ROARK_API_BEARER_TOKEN"])
>
> result = client.analytics.query(
>     measures=[{"type": "CONVERSATION"}, {"metric": "sentiment_score"}],
>     time_range={"type": "LAST", "value": 7, "unit": "DAYS"},
>     interval="DAY",
> )
>
> for series in result.data.series:
>     print(series.metric or "calls", series.value)
> ```

> [tab] CLI
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> roark analytics query --data '{
>   "measures": [{ "type": "CONVERSATION" }, { "metric": "sentiment_score" }],
>   "timeRange": { "type": "LAST", "value": 7, "unit": "DAYS" },
>   "interval": "DAY"
> }'
> ```

> [tab] cURL
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> curl -X POST https://api.roark.ai/v1/analytics/query \
>   -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN" \
>   -H "Content-Type: application/json" \
>   -d '{
>     "measures": [{ "type": "CONVERSATION" }, { "metric": "sentiment_score" }],
>     "timeRange": { "type": "LAST", "value": 7, "unit": "DAYS" },
>     "interval": "DAY"
>   }'
> ```

The key needs the `metric:read` permission. A read-only key is enough: a query never changes anything.

***

## What you can query

Read the schema first. It lists the project's metrics by slug, each with its output type, its default aggregation and the aggregations it supports, plus every filter and breakdown type and the custom call properties seen in the project.

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark analytics schema get
```

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "data": {
    "metrics": [
      { "id": "3f2a…", "slug": "task_completion", "name": "Task Completion", "outputType": "SCALE", "isThreshold": false, "defaultAggregation": "AVG", "aggregations": ["AVG", "MIN", "MAX", "SUM", "MEDIAN", "PERCENTILE", "COUNT", "PASS_RATE"] },
      { "id": "9c41…", "slug": "call_outcome", "name": "Call Outcome", "outputType": "CLASSIFICATION", "isThreshold": false, "defaultAggregation": "COUNT", "aggregations": ["COUNT", "PASS_RATE"] }
    ],
    "filters": [{ "type": "AGENT_PROPERTY", "description": "Conversations the agent took part in. Value: an agent id …" }],
    "callProperties": [{ "name": "region", "valueType": "STRING", "isSystem": false }],
    "intervals": ["HOUR", "DAY", "WEEK", "MONTH"]
  }
}
```

The [MCP server](/documentation/sdks/mcp-server) exposes the same two operations, so an assistant connected to Roark can read the schema and run queries on its own.

***

## Measures

A query has one to ten **measures**: the things to aggregate.

| Measure | What it counts | Default aggregation |
| :- | :- | :- |
| `{ "metric": "<slug>" }` | A metric's values | By output type: `AVG` for numbers and scales, `COUNT` split by value for classifications and text, `AVG` (the share of true) for booleans, `PASS_RATE` for thresholds |
| `{ "type": "CONVERSATION" }` | Conversations (calls and chats) | `COUNT` |
| `{ "type": "UPTIME" }` | Health-check probe outcomes | `PASS_RATE` (the up-rate) |

Set `aggregation` to choose another: `AVG`, `MIN`, `MAX`, `SUM`, `MEDIAN`, `PERCENTILE` (with `percentile`, e.g. `95`), `COUNT` or `PASS_RATE`. The schema lists which ones each metric supports, and a query asking for one it doesn't support returns `400` saying which to use instead.

* **Per call first.** Metric values are folded to one value per call before they are aggregated (`distinctBy: CALL`), so a long call with many scored segments doesn't outweigh a short one. Set `distinctBy: NONE` to aggregate every value.
* **Pass rates.** `PASS_RATE` is the share of calls that meet a condition, from 0 to 1: `{ "metric": "response_time", "aggregation": "PASS_RATE", "passOperator": "LESS_THAN", "passValue": 800 }`. A [threshold](/documentation/metrics/thresholds) metric already carries its condition, so querying it by slug gives its pass rate directly. `sampleSize` is the number of calls evaluated; `0` means no data, not 0%.
* **Units.** Values come back in the metric's display units.

***

## Time windows

`timeRange` picks the window. It defaults to the last 30 days, and covers at most two years.

| `type` | Example |
| :- | :- |
| `LAST` | `{ "type": "LAST", "value": 7, "unit": "DAYS" }` (`DAYS`, `WEEKS`, `MONTHS`) |
| `CALENDAR_ALIGNED` | `{ "type": "CALENDAR_ALIGNED", "period": "THIS_MONTH" }` (`TODAY`, `YESTERDAY`, `THIS_WEEK`, `LAST_WEEK`, `THIS_MONTH`, `LAST_MONTH`, `THIS_YEAR`, `LAST_YEAR`) |
| `FIXED` | `{ "type": "FIXED", "from": "2026-09-01T00:00:00Z", "to": "2026-10-01T00:00:00Z" }` |
| `SINCE` | `{ "type": "SINCE", "from": "2026-10-01T00:00:00Z" }` |

`timezone` (an IANA name such as `America/New_York`, default `UTC`) decides where days, weeks and calendar periods start. `interval` (`HOUR`, `DAY`, `WEEK`, `MONTH`) sets the bucket size; leave it out and it is chosen from the window's length. A window that would produce more than 500 buckets steps to a coarser interval. Set `includeBuckets: false` to get totals only.

***

## Filters, breakdowns and comparisons

**Filters** narrow what is counted. Top-level `filters` apply to every measure; a measure's own `filters` add to them.

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
"filters": [
  { "type": "MODALITY", "value": "call" },
  { "type": "AGENT_PROPERTY", "value": "<agent-id>" },
  { "type": "CALL_PROPERTY", "property": "region", "value": "emea" },
  { "type": "METRIC_PROPERTY", "property": "participantRole", "value": "AGENT" }
]
```

`operator` defaults to `EQUALS`; the others are `NOT_EQUALS`, `GREATER_THAN`, `GREATER_THAN_OR_EQUAL_TO`, `LESS_THAN`, `LESS_THAN_OR_EQUAL_TO`, `CONTAINS`, `STARTS_WITH`, `IS_SET` and `IS_NOT_SET`. The schema lists every filter type and the properties each accepts.

**Set or missing.** `IS_SET` and `IS_NOT_SET` apply to `CALL_PROPERTY` filters only and take no `value`. `IS_SET` keeps conversations that have the property, whatever its value (an empty string included); `IS_NOT_SET` keeps the ones missing it: never sent, sent as `null`, or removed. A property no conversation has carried is missing on all of them.

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{ "type": "CALL_PROPERTY", "property": "plan_tier", "operator": "IS_NOT_SET" }
```

Sending a `value` with them, using them on another filter type, or using them as a `PASS_RATE` `passOperator` returns `400`. On the system properties, `direction` and `is_simulated` are set on every call and on no chat, and `duration_seconds` is set on calls that lasted more than zero seconds; `ended_reason` doesn't support them.

**Breakdowns** split a measure into one series per value. `{ "metric": "sentiment_score", "breakdown": { "type": "AGENT_PROPERTY" } }` returns sentiment per agent; `{ "type": "CALL_PROPERTY", "property": "direction" }` splits by inbound and outbound. The 50 largest series are returned, and `omittedSeries` counts the rest.

**Comparisons**:

* `comparisonMode: "PREVIOUS_PERIOD"` adds the preceding window of the same length to every series (`comparisonValue`, and per bucket).
* `comparisonGroups` runs every measure once per named filter set, for example one group per agent or per region (2 to 5 groups). While groups are set, breakdowns are ignored: each group is one series.

### Filter by another metric

A `METRIC_VALUE` filter keeps the conversations where a different metric's value matches: for example, the pass rate of `retrieved_accurately` only over the calls where `information_given` is true. It works as a top-level filter, on a measure, in a comparison group and as a dashboard-level filter.

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
"filters": [
  { "type": "METRIC_VALUE", "property": "<metric-id>", "operator": "EQUALS", "value": true }
]
```

* **`property`** is the filter metric's **id**, as `GET /v1/analytics/schema` lists it. A slug is not accepted here (unlike a measure's `metric`), and returns `400`.
* **One side.** Add `.AGENT` or `.CUSTOMER` to read only that participant's value of a per-participant metric: `"<metric-id>.AGENT"`. Without it, either side's value can match.
* **`value`** is required. A conversation matches on its latest value for the filter metric; conversations it hasn't scored are left out.

The operators follow the filter metric's output type:

| Output type | Operators | Value |
| :- | :- | :- |
| `BOOLEAN` | `EQUALS`, `NOT_EQUALS` | `true` or `false` |
| `CLASSIFICATION`, `TEXT` | `EQUALS`, `NOT_EQUALS`, `CONTAINS` | A label or text, matched case-insensitively |
| `NUMERIC`, `SCALE`, `COUNT`, `OFFSET` | `EQUALS`, `NOT_EQUALS`, `GREATER_THAN`, `GREATER_THAN_OR_EQUAL_TO`, `LESS_THAN`, `LESS_THAN_OR_EQUAL_TO` | A number, compared with the stored value (`OFFSET` in milliseconds) |
| Threshold (`isThreshold: true` in the schema) | `EQUALS`, `NOT_EQUALS` | `PASSED` or `FAILED` |

On a threshold, `PASSED` keeps the conversations that passed it and `FAILED` the ones that failed it, each evaluated the same way as the threshold's own pass rate. `NOT_EQUALS` flips the status, so `NOT_EQUALS PASSED` is the same as `EQUALS FAILED`. A conversation with no values for the metric the threshold checks is in neither set. A threshold takes no `.AGENT` or `.CUSTOMER` suffix: it reads the side it was defined on.

`STARTS_WITH` is never supported. The query returns `400`, naming the problem, when:

* `property` isn't a metric id, optionally followed by `.AGENT` or `.CUSTOMER`
* the metric doesn't exist in your organization
* `value` is missing or empty, or isn't a number for a number-type metric or `true` / `false` for a boolean
* the operator isn't one the metric's output type supports, or a threshold's `value` isn't `PASSED` or `FAILED`

A widget with one of these in its own filters or comparison groups is refused when you save it, rather than failing on every read.

***

## The response

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "data": {
    "from": "2026-10-14T09:00:00.000Z",
    "to": "2026-10-21T09:00:00.000Z",
    "interval": "DAY",
    "timezone": "UTC",
    "series": [
      {
        "measureIndex": 1,
        "type": "METRIC",
        "metric": "sentiment_score",
        "aggregation": "AVG",
        "breakdownValue": null,
        "comparisonGroup": null,
        "value": 0.62,
        "sampleSize": null,
        "comparisonValue": null,
        "buckets": [{ "date": "2026-10-14T00:00:00Z", "value": 0.58 }]
      }
    ],
    "omittedSeries": 0
  }
}
```

There is one series per measure, or per breakdown value or comparison group. `measureIndex` points back to the measure in your request. `value` is the aggregate over the whole window, and `buckets` holds the per-interval values.

A query the analytics store can't answer returns `400` naming the part it can't, never unfiltered numbers. An unknown metric slug returns `404`.

***

## From a query to a dashboard

A dashboard widget is a query plus a name, a chart type and a size. Once a query answers the question, save it as a widget with [`POST /v1/dashboard/{dashboardId}/widget`](/documentation/observability/dashboards#dashboards-over-the-api), or declare the dashboard in [config as code](/documentation/config-as-code/dashboards).

> [card] [Dashboards](/documentation/observability/dashboards)
>
> Save queries as widgets and read them back over the API

> [card] [Dashboards as code](/documentation/config-as-code/dashboards)
>
> Define dashboards in YAML next to the rest of your config


# Dashboards
Source: https://docs.roark.ai/documentation/observability/dashboards

Organize multiple reports into a single view for comprehensive monitoring

## Overview

Dashboards allow you to organize and view multiple reports in a single place, giving you a comprehensive overview of your voice AI performance. Create separate dashboards for different teams, use cases, or monitoring needs.

Define a dashboard [in code](/documentation/config-as-code/dashboards) (recommended), or build it in the dashboard UI.

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineDashboard, defineConfig } from '@roarkanalytics/sdk/lib/config';

const frontdeskHealth = defineDashboard({
  name: 'frontdesk-health',
  displayName: 'Frontdesk health',
  timeRange: { type: 'LAST', value: 30, unit: 'DAYS' },
  filters: [
    { type: 'AGENT_PROPERTY', value: 'frontdesk' }, // an agent by its config name, or its id
    { type: 'MODALITY', value: 'call' },
  ],
  widgets: [
    {
      name: 'Calls',
      display: 'NUMBER',
      size: 'SMALL',
      measures: [{ type: 'CONVERSATION' }],
    },
    {
      name: 'Task completion',
      display: 'LINE',
      size: 'LARGE',
      interval: 'DAY',
      measures: [{ metric: 'task_completion' }],
    },
    {
      name: 'Call outcomes',
      display: 'STACKED_BAR',
      interval: 'WEEK',
      measures: [{ metric: 'call_outcome' }], // a classification: counted and split by value
    },
  ],
});

export default defineConfig({ resources: [frontdeskHealth] });
```

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: dashboard
name: frontdesk-health
displayName: Frontdesk health
timeRange: { type: LAST, value: 30, unit: DAYS }
filters:
  - { type: AGENT_PROPERTY, value: frontdesk }   # an agent by its config name, or its id
  - { type: MODALITY, value: call }
widgets:
  - name: Calls
    display: NUMBER
    size: SMALL
    measures:
      - type: CONVERSATION
  - name: Task completion
    display: LINE
    size: LARGE
    interval: DAY
    measures:
      - metric: task_completion
  - name: Call outcomes
    display: STACKED_BAR
    interval: WEEK
    measures:
      - metric: call_outcome      # a classification: counted and split by value
```

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply roark.config.ts
```

### Manage it with the CLI

List dashboards and read a dashboard's numbers from the CLI:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard list
roark dashboard data get <dashboard-id>
```

***

## Creating Dashboards

* **Multiple Dashboards**: Create separate dashboards for different use cases (e.g., Production Monitoring, QA Review, Executive Summary)
* **Add Reports**: Select which saved [reports](/documentation/observability/reports) to include in each dashboard
* **Reorder**: Drag and drop reports to organize your dashboard layout

***

## Property Overrides

Filter an entire dashboard by specific call properties without creating new reports:

[https://www.tella.tv/video/dashboard-filters-3x81/embed](https://www.tella.tv/video/dashboard-filters-3x81/embed)

**Common Use Cases:**

* View all reports filtered to a specific business name or customer ID
* Analyze performance for a particular agent or integration
* Compare metrics across different property values
* Quickly switch between filtered views without leaving the dashboard

When you override properties at the dashboard level, all reports on that dashboard automatically update to reflect the filter, making it easy to drill down into specific segments of your data.

### Set or missing

A custom property filter can also check whether the property is there at all. Pick the property, then **is set** or **is not set**; there is no value to enter.

* **is set** keeps conversations that have a value for the property, an empty string included.
* **is not set** keeps conversations missing it: never sent, sent as `null`, or removed later.

A property no conversation has ever carried is missing on all of them. The filter reads as, for example, `plan_tier is not set`. It works on a widget, on a comparison group and on the whole dashboard. The [calls list](/documentation/observability/live-monitoring#filter-by-a-property-being-set-or-missing) has the same filter.

***

## Filter by another metric

Narrow a widget to the conversations where a different metric came out a certain way. For example, show the pass rate of **Retrieved accurately** only over the calls where **Information given** is true: calls where the agent gave no information can't retrieve it accurately, so they would only drag the rate down.

The **Another metric** filter is in the filter menu of:

* **The widget editor**, to narrow one widget. It is offered on every widget except uptime.
* **Comparison groups**, so each group can be a different slice, for example "Information given: true" against "Information given: false".
* **The dashboard filter bar**, to narrow every widget on the dashboard at once.

To add one:

**1. Choose the metric**

Open the filter menu, choose **Another metric** and pick the metric to filter on. Each metric shows its output type (Yes / no, Category, Scale, Number, Count, Time or Text).

**2. Choose a side (optional)**

If the metric scores each participant separately and isn't fixed to one side, choose **Either side**, **Agent** or **Customer**. **Either side** matches when either participant's value matches.

**3. Set the condition**

The condition depends on the metric's output type:

| Output type | Condition |
| :- | :- |
| Yes / no | **equals** or **not equals** the metric's true or false label |
| Category | **equals** or **not equals** one of its categories |
| Scale | **equals** or **not equals** one of its scale values |
| Text | **equals** or **not equals** one of the values seen so far |
| Number, Count, Time | `=`, `≠`, `>` or `<` a number you type |
| Threshold | **Passed** or **Failed** |

The filter shows as the metric's name and the condition, such as `Information given = True` (using the metric's own labels), with the side in brackets when you picked one: `Information given (Agent) = True`.

A conversation matches on its latest value for the filter metric. Conversations the filter metric hasn't scored are left out.

A threshold can be the filter metric too: choose **Passed** to keep the calls that passed it, or **Failed** for the calls that failed it. Each call is evaluated the same way as the threshold's own pass rate, so the calls a **Passed** filter keeps are exactly the ones its pass rate counts as passing. A call with no values for the metric the threshold checks is in neither set. A threshold has no side to choose: it reads the side it was defined on.

### Worked example

You score calls with two metrics: **Information given** (yes / no) and **Retrieved accurately** (yes / no). You want to know how often the agent gets the information right when it gives any.

1. Add a widget for **Retrieved accurately**. As a yes / no metric, it shows the share of calls where it came out true.
2. In the widget's filters, choose **Another metric**, then **Information given**.
3. Choose **equals**, then the metric's true label.

The widget now shows how often **Retrieved accurately** is true over only the calls where **Information given** is true. To compare against the rest, add a comparison group for each value of **Information given** instead.

Over the API, the same filter is a `METRIC_VALUE` filter: see [Query analytics](/documentation/observability/analytics-api#filter-by-another-metric).

***

## Managing Dashboards

* **Default Dashboard**: Set which dashboard loads by default
* **Quick Access**: Switch between dashboards from the navigation
* **Update Reports**: Add, remove, or reorder reports as your needs change

***

## Dashboards over the API

Everything on a dashboard is also available over the API, the [SDKs](/documentation/sdks/node-sdk), the [CLI](/documentation/sdks/cli) and the [MCP server](/documentation/sdks/mcp-server). A dashboard made in Roark reads over the API, and one made over the API opens in Roark.

**Read a dashboard's numbers.** `GET /v1/dashboard/{dashboardId}/data` runs every widget and returns each one's series exactly as the dashboard shows it: over the dashboard's saved window and with its filters applied. Pass `from` (and optionally `to`) to read another window, and `timezone` to align the buckets.

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "data": {
    "dashboardId": "6f1d…",
    "from": "2026-09-21T09:00:00.000Z",
    "to": "2026-10-21T09:00:00.000Z",
    "timezone": "UTC",
    "widgets": [
      { "widgetId": "7a8b…", "name": "Calls", "interval": "DAY", "blocked": false, "error": null, "series": [{ "value": 1840, "buckets": ["…"] }], "omittedSeries": 0 }
    ]
  }
}
```

A widget that can't be computed on its own (say its metric was deleted) carries `error` and no series; the rest of the dashboard still returns. `GET /v1/dashboard/{dashboardId}/widget/{widgetId}/data` reads a single widget.

**Create and edit dashboards.** A widget is an [analytics query](/documentation/observability/analytics-api) plus a `name` (unique on the dashboard), a `display` (`LINE`, `NUMBER`, `RADAR`, `STACKED_BAR`, `TABLE`) and a `size` (`SMALL`, `MEDIUM`, `LARGE`, `EXTRA_LARGE`):

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark dashboard create --data '{
  "name": "Support agent health",
  "timeRange": { "type": "LAST", "value": 30, "unit": "DAYS" },
  "filters": [{ "type": "MODALITY", "value": "call" }],
  "widgets": [
    { "name": "Calls", "display": "NUMBER", "size": "SMALL", "measures": [{ "type": "CONVERSATION" }] },
    { "name": "Task completion", "display": "LINE", "interval": "DAY", "measures": [{ "metric": "task_completion" }] }
  ]
}'
```

| Operation | Endpoint |
| :- | :- |
| List, create | `GET` / `POST /v1/dashboard` |
| Get, update, delete | `GET` / `PUT` / `DELETE /v1/dashboard/{dashboardId}` |
| Add a widget (to the top) | `POST /v1/dashboard/{dashboardId}/widget` |
| Replace, remove a widget | `PUT` / `DELETE /v1/dashboard/{dashboardId}/widget/{widgetId}` |
| Read data | `GET /v1/dashboard/{dashboardId}/data`, `GET .../widget/{widgetId}/data` |

`PUT /v1/dashboard/{dashboardId}` changes only the fields you send. Sending `widgets` replaces the list, matched by name: a widget that keeps its name keeps its id, a new name adds a widget, and a name you leave out is removed. Dashboard-level `filters` must be conversation-level (`AGENT_PROPERTY`, `CALL_PROPERTY`, `MODALITY`, `DATASET` or [`METRIC_VALUE`](/documentation/observability/analytics-api#filter-by-another-metric)). On a field a widget already filters, the dashboard's filter wins.

Reading needs the `metric:read` permission; creating, editing and deleting need `metric:create`, `metric:update` and `metric:delete`.

***

## Dashboards as code

Dashboards can live in git with the rest of your config as a [`kind: dashboard`](/documentation/config-as-code/dashboards) resource. A dashboard created by config shows **Managed by config** in Roark: its title and widgets are read-only there and over the API (edits return `409`), so they only change through your config. Anyone viewing it can still pick another time range and filter it. Duplicating a managed dashboard gives you an editable copy.

***

## What's Next

> [card] [Reports](/documentation/observability/reports)
>
> Create the reports that power your dashboards

> [card] [Query analytics](/documentation/observability/analytics-api)
>
> Ask for the same numbers from code or an agent

> [card] [Dashboards as code](/documentation/config-as-code/dashboards)
>
> Define dashboards in YAML

> [card] [Live Monitoring](/documentation/observability/live-monitoring)
>
> Monitor calls in real-time


# Live Monitoring
Source: https://docs.roark.ai/documentation/observability/live-monitoring

Browse, search, and analyze every voice AI call

## Overview

Call History is your central hub for browsing, searching, and analyzing every call that flows through Roark. Each call is automatically transcribed and analyzed the moment it arrives.

![Call History](https://mintcdn.com/roark/tphOjEqKNnjNqnPJ/images/monitoring/call-history.png?fit=max\&auto=format\&n=tphOjEqKNnjNqnPJ\&q=85\&s=5e34f5b0713d02a969b5a1aba0939e1f)

***

## Getting Calls Into Roark

### Via Integration

Connect a voice platform ([Vapi](/documentation/integrations/vapi), [Retell](/documentation/integrations/retell), or [LiveKit](/documentation/integrations/livekit)) and calls sync automatically. See [Integrations](/documentation/integrations/overview) for setup guides.

### Via API

Upload a call recording using the [Node.js](/documentation/sdks/node-sdk) or [Python](/documentation/sdks/python-sdk) SDK:

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const call = await client.call.create({
  recordingUrl: 'https://example.com/recording.mp3',
  startedAt: '2024-01-15T10:00:00Z',
  interfaceType: 'PHONE',
  callDirection: 'INBOUND',
  agent: { name: 'Support Agent' },
  customer: { phoneNumberE164: '+15551234567' },
  properties: { department: 'sales' },
})
```

Once created, the call appears in Call History and is automatically analyzed.

***

## What You See Per Call

Each call opens into a detailed view with the following tabs:

* **Overview**: AI-generated summary, key stats (duration, direction, status, end status), participants, and labels
* **Transcript**: Speaker-diarized conversation with timestamps, tool call annotations, and sentiment overlay
* **Metrics**: All collected metric values organized by package (call-level, segment-level, turn-level)
* **Properties**: Custom metadata passed with the call, filterable and searchable
* **Tools**: All tool/function invocations with inputs, outputs, and error details

> [!TIP]
>
> You can turn real calls into [customer flows](/documentation/simulation-testing/customer-flows). When creating a flow, choose **From your calls** and Roark drafts the flow from the conversations you pick.

***

## Filtering and Search

Find exactly what you need:

* **Full-text transcript search** across all calls
* **Filters**: duration, end status, call type (live vs simulation), agent, labels, analysis status, and custom properties
* **Saved views** for quick access to frequently used filter combinations

### Filter by a property being set or missing

A custom property filter can also check whether a call has the property at all, whatever its value. Pick the property, then **is set** or **is not set**.

* **is set** keeps calls that have a value for the property. An empty string counts as a value.
* **is not set** keeps calls missing it: the property was never sent, was sent as `null`, or was removed later by updating it to `null`.

A property your project has never received is missing on every call. Property names match the way they are stored, so `planTier` and `plan_tier` are the same property.

The same filter is available over the API on `GET /v1/call`, one `properties[<name>][exists]` parameter per property:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# Calls that never got a plan_tier, but do have a region
curl -G https://api.roark.ai/v1/call \
  -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN" \
  --data-urlencode "properties[plan_tier][exists]=false" \
  --data-urlencode "properties[region][exists]=true"
```

`true` keeps calls with the property set and `false` keeps calls missing it. You can filter on up to 20 properties at once. A name used by a built-in call filter (such as `status`), or the same property asked to be both set and missing, returns `400`.

Dashboards filter the same way: see [dashboard filters](/documentation/observability/dashboards#property-overrides).

***

## Deep Linking with URL Parameters

Share filtered views or link directly to relevant calls from external systems using URL query parameters. Custom properties can be filtered using dot notation:

```
/org/{org-slug}/{project-slug}/calls?p.{property_name}={operator}{value}
```

### Supported Operators

| Operator | Description | Example |
| - | - | - |
| `=` | Equals (default) | `p.outcome=completed` |
| `!=` | Not equals | `p.outcome=!=failed` |
| `>` | Greater than | `p.count=>10` |
| `>=` | Greater than or equal | `p.count=>=10` |
| `<` | Less than | `p.count=<5` |
| `<=` | Less than or equal | `p.count=<=5` |
| `contains:` | Contains | `p.customer_name=contains:John` |
| `starts_with:` | Starts with | `p.region=starts_with:us-` |
| `is_set:` | Has the property, whatever the value | `p.plan_tier=is_set:` |
| `is_not_set:` | Missing the property | `p.plan_tier=is_not_set:` |

### Examples

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# Filter by exact match
/calls?p.simulation_run_id=abc-123

# Filter by contains
/calls?p.customer_name=contains:John

# Filter by not equals
/calls?p.outcome=!=failed

# Filter by numeric comparison
/calls?p.attempt_count=>3

# Calls that never got a plan_tier
/calls?p.plan_tier=is_not_set:

# Combine multiple filters
/calls?p.simulation_run_id=abc-123&p.region=starts_with:us-
```

### Direct Call Linking

When a filter returns only a single matching call, Roark automatically opens that call's detail view. This enables direct deep linking to specific calls using unique identifiers:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# Link directly to a specific call using a unique custom property
/calls?p.custom_call_id=abc-123
```

This is particularly useful when you store your own call identifiers and want to link from external systems directly to the call details without an intermediate list view.

### Use Cases

* **Direct call access**: Link to a specific call using a unique identifier (e.g., `custom_call_id`); the call opens automatically when only one match is found
* **External system integration**: Link from CRMs, dashboards, or internal tools directly to relevant calls
* **Bookmarks**: Save filtered views for quick access to specific call subsets
* **Team sharing**: Share pre-filtered views with teammates for collaboration

***

## Collecting More Metrics

Roark automatically analyzes transcription, speech patterns, and sentiment for every call. To collect additional metrics (like custom LLM evaluations, compliance checks, or business KPIs) configure [metric collectors](/documentation/metrics/metric-collectors) or test metrics in [Studio](/documentation/metrics/studio).

Available out-of-the-box analysis includes:

* Sentiment tracking and 64+ emotion detection
* Interruption and overlap detection
* Speech pause and silence analysis
* Vocal cue detection (raised voice, frustration, etc.)
* Repetition detection

***

## What's Next

> [card] [Custom Metrics](/documentation/metrics/custom-metrics)
>
> Define custom metrics for your calls

> [card] [Policies](/documentation/metrics/metric-collectors)
>
> Automate metric collection

> [card] [Reports](/documentation/observability/reports)
>
> Build analytics from call data

> [card] [Dashboards](/documentation/observability/dashboards)
>
> Organize reports into views


# Live Call Reports
Source: https://docs.roark.ai/documentation/observability/live-reports

A written report on your production calls every day, week or month: targets, trends, what went wrong and what to do about it

## Overview

A live call report is the report Roark writes after a simulation run, for your production traffic. You pick the calls (an agent, narrowed by filters) and the bar they are held to (targets). When each period closes, Roark reads the period's calls and writes a report you can open in Roark, download as a PDF, or have sent to email and Slack.

Each report answers four questions:

* **Did we hit our targets?** Every target with this period's number, the period before, and whether it was met.
* **What changed?** Call volume, how calls ended, and every check's pass rate against the previous period.
* **What went wrong?** Failure themes, grouped from the calls that failed a target or check, each with how many calls it affected, quoted transcript evidence, and links to the calls.
* **What should we do?** Recommendations tied to the themes, in priority order.

Define a report [in code](/documentation/config-as-code/live-reports), or build it in Roark under **Monitor > Reports**.

***

## Create a report

Open **Reports** in the sidebar and click **New report**. The form walks through six steps; the rail on the left shows what each is set to, and you can jump between them.

**1. Details**

The report's name (how it reads in the list, emails and Slack) and an optional description.

**2. Calls**

The agent whose calls it covers, or every agent in the project, and any filters that narrow them further: a call property, the modality (calls or chats), a dataset, or another metric's value (for example, only calls where the task was completed). Simulations are never included.

**3. Targets**

Up to 10 numbers each period is held to, for example "Task completion pass rate at least 85%" or "p95 latency at most 1,200 ms". A report with no targets still describes the period; targets are what turn it into a verdict.

**4. Schedule**

Daily, weekly or monthly, the timezone the periods are read in, and the local hour the report is sent.

**5. Delivery**

Who gets each report: every project member by email (the default), a list of addresses (people without a Roark account included, since the email carries the PDF), no email, and any Slack channels. You can also turn off transcript excerpts here if transcripts must not leave Roark.

**6. Charts**

Optionally, a [dashboard](/documentation/observability/dashboards) whose charts the report embeds. Its widgets are read over the report's own calls and period, so the charts always match the numbers around them.

> [!TIP]
>
> Use **Write the latest period now** from the report's menu to see a report straight away, rather than waiting for the period to close.

***

## Targets

A target is a metric, an aggregation, and a bar:

| Aggregation | Reads | Bar |
| :- | :- | :- |
| Pass rate | The share of calls that passed | A percentage, 0 to 100 |
| Average, Median | The metric's typical value | In the metric's units |
| Percentile | p50 to p99 of the metric's value | In the metric's units |
| Min, Max | The extreme value | In the metric's units |

A [threshold](/documentation/metrics/thresholds)'s pass rate uses the threshold's own condition. For any other metric, a pass-rate target also takes the condition a call passes on (for example, "sentiment at least 6"). A target is met when the period's number is **at least** the bar (the default for pass rates) or **at most** it (the default for everything else, such as latency).

Each target is also read per agent, so a report covering several agents shows which one is pulling the number down.

***

## Schedule and periods

A report covers whole calendar periods in its timezone: a day, a Monday-to-Sunday week, or a calendar month. It is written once the period has closed, at the delivery hour on the following day. A weekly report set to 08:00 in `America/Chicago` covers Monday to Sunday and arrives on Monday at 08:00 Chicago time.

Each report is compared with the period before it: a week with the previous week, a month with the previous month. A comparison is only drawn when both periods have enough calls to make it meaningful.

Pause a report from its menu to stop new reports without deleting it. When you resume it, the next report covers the most recent closed period; the periods it was paused for are not written retroactively, but you can still write the latest one on demand.

***

## Reading a report

| Section | What it shows |
| :- | :- |
| **Verdict** | One paragraph on how the period went, with targets met and the headline change. |
| **Targets** | Each target this period and last, met or missed. |
| **Failure themes** | What went wrong, grouped from the failing calls, with how many calls each affected, a quote from the transcript, and links to the calls. A theme that is already an open issue in **Issues** links to it. |
| **Recommendations** | What to change, in priority order, tied to the themes behind them. |
| **By agent** | The busiest agents, with each target read for them alone. |
| **Checks** | Every metric the period's calls were graded on, worst pass rate first. |
| **Volume** | Calls in scope over the period, and how they ended. |
| **Charts** | The linked dashboard's widgets, over this period's calls. |
| **Appendix** | How the report was put together: how many calls were read, how themes were counted, and anything the report could not read. |

When many calls fail, Roark reads a representative sample of them and weights each theme back to the full count, so "affects about 340 calls" means 340 of the period's calls, not 340 of the sample. The appendix says how many calls were read.

Use the period picker to open an earlier report. Every report also has a **Download PDF**; links in emails and Slack open the report they were sent for, even if the report definition is later deleted.

***

## API and CLI

Everything in the form is available over the API, so a report can be created from a script or a deployment pipeline.

| Operation | Endpoint | CLI |
| :- | :- | :- |
| List reports | `GET /v1/live-report` | `roark live-report list` |
| Create a report | `POST /v1/live-report` | `roark live-report create` |
| Get a report | `GET /v1/live-report/{reportId}` | `roark live-report get` |
| Update a report | `PATCH /v1/live-report/{reportId}` | `roark live-report update` |
| Delete a report | `DELETE /v1/live-report/{reportId}` | `roark live-report delete` |
| Write the latest period now | `POST /v1/live-report/{reportId}/generate` | `roark live-report run` |
| List written reports | `GET /v1/live-report/{reportId}/document` | `roark live-report document list` |
| Get a written report | `GET /v1/live-report/{reportId}/document/{documentId}` | `roark live-report document get` |

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/live-report \
  -H "Authorization: Bearer $ROARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Frontdesk weekly review",
    "agentId": "6f1c2a8e-3b4d-4e5f-8a9b-0c1d2e3f4a5b",
    "filters": [{ "type": "MODALITY", "value": "call" }],
    "targets": [
      { "metric": "task_completion", "aggregation": "PASS_RATE", "value": 85 },
      { "metric": "response_latency_ms", "aggregation": "PERCENTILE", "percentile": 95, "value": 1200 }
    ],
    "cadence": "WEEKLY",
    "timezone": "America/Chicago",
    "deliveryHour": 8,
    "notifications": {
      "email": { "enabled": true, "recipients": ["ops@example.com"] },
      "slack": []
    }
  }'
```

A written report's document carries the same content as the page: the verdict, targets, themes with their evidence, recommendations, per-agent rows and checks, so you can feed it into your own tooling. See the [API reference](/api-reference/introduction) for every field.

> [!NOTE]
>
> A report defined in [config as code](/documentation/config-as-code/live-reports) is read-only in Roark and over the API: edits and deletes return `409`. Anyone can still write the latest period on demand.


# Overview
Source: https://docs.roark.ai/documentation/observability/overview

Monitor and analyze your voice AI conversations in real-time

Roark gives you full visibility into every voice AI conversation. Calls flow in via integrations or the API, metrics are collected automatically via policies, and you can explore everything in real-time.

## Quickstart

Send Roark a call and read what it measured:

> [tab] Node
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> const call = await client.call.create({
>   recordingUrl: 'https://example.com/recording.mp3',
>   startedAt: '2024-01-15T10:00:00Z',
>   interfaceType: 'PHONE',
>   callDirection: 'INBOUND',
>   agent: { name: 'Support Agent' },
> })
>
> const metrics = await client.call.listMetrics(call.data.id)
> ```

> [tab] Python
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> call = client.call.create(
>     recording_url="https://example.com/recording.mp3",
>     started_at="2024-01-15T10:00:00Z",
>     interface_type="PHONE",
>     call_direction="INBOUND",
>     agent={"name": "Support Agent"},
> )
>
> metrics = client.call.list_metrics(call.data.id)
> ```

> [tab] CLI
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> CALL_ID=$(roark call create \
>   --recording-url "https://example.com/recording.mp3" \
>   --started-at "2024-01-15T10:00:00Z" \
>   --interface-type PHONE --call-direction INBOUND \
>   --agent '{"name":"Support Agent"}' | jq -r '.data.id')
>
> roark call metric list "$CALL_ID"
> ```

Most teams connect a platform so production calls flow in automatically, see [Integrations](/documentation/integrations/overview).

***

## Key Components

> [card] [Call History](/documentation/observability/live-monitoring)
>
> Browse, search, and analyze every call

> [card] [Custom Metrics](/documentation/metrics/custom-metrics)
>
> Define what to measure across calls

> [card] [Reports](/documentation/observability/reports)
>
> Build analytics from your metric data

> [card] [Traces](/documentation/observability/traces)
>
> OpenTelemetry tracing and backend correlation

> [card] [Dashboards](/documentation/observability/dashboards)
>
> Organize reports into views

***

## How Calls Get Into Roark

There are two ways to get calls into Roark:

### Integrations

Connect a voice platform ([Vapi](/documentation/integrations/vapi), [Retell](/documentation/integrations/retell), or [LiveKit](/documentation/integrations/livekit)) and calls sync automatically. See [Integrations](/documentation/integrations/overview) for setup guides.

### API / SDK

Upload call recordings directly via the [Node.js](/documentation/sdks/node-sdk) or [Python](/documentation/sdks/python-sdk) SDK, or the [REST API](/api-reference/introduction).

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const call = await client.call.create({
  recordingUrl: 'https://example.com/recording.mp3',
  startedAt: '2024-01-15T10:00:00Z',
  interfaceType: 'PHONE',
  callDirection: 'INBOUND',
  agent: { name: 'Support Agent' },
  customer: { phoneNumberE164: '+15551234567' },
  properties: { department: 'sales' },
})
```

***

## What Happens When a Call Arrives

**1. Transcription & Analysis**

The call is transcribed and analyzed: speech patterns, sentiment, and 64+ emotions are detected automatically.

**2. Metric Collection**

Active [metric collectors](/documentation/metrics/metric-collectors) run automatically, collecting the metrics you've configured.

**3. Available in Call History**

The call appears in [Call History](/documentation/observability/live-monitoring) with full analysis, ready to explore.


# PII Redaction
Source: https://docs.roark.ai/documentation/observability/pii-redaction

Mask sensitive caller information in transcripts before it reaches your team or metrics

Roark can detect and mask personally identifiable information (PII), phone numbers, social security numbers, credit cards, and more, in your call and chat transcripts. When redaction is on, the original PII never leaves the redaction boundary: viewers see masked tokens, LLM graders receive masked text, and search results are sanitized.

Redaction is a per-project setting and applies to both **calls and chats** going forward. Existing conversations keep their original transcripts.

***

## How It Works

**1. Transcript becomes available**

Whether the transcript came from your integration, your own upload, or Roark's transcription pipeline, redaction kicks in once the transcript is persisted.

**2. Roark scans the transcript for PII**

A redaction pass identifies the entity types you've enabled (phone numbers, account numbers, addresses, names, etc.) and stores them as redaction spans alongside the transcript.

**3. Masking on display and analysis**

When you view a conversation, the transcript renders `[REDACTED:PHONE]`-style pills in place of the sensitive content. Metrics that use LLMs (custom prompts, sentiment, toxicity, emotion, politeness) receive the masked transcript by default.

***

## Enabling Redaction

**1. Open Settings**

Go to your project's **Settings** page and find the **PII Redaction** section.

**2. Toggle redaction on**

Flip the master toggle. The entity-type list expands.

**3. Choose entity types**

Most entity types are on by default. Names and addresses are off by default: they have higher false-positive rates and customers usually opt in deliberately.

***

## Supported Entity Types

| Entity | Default | Notes |
| - | - | - |
| Social security numbers | On | |
| Credit card number / CVV / expiration | On | Each is a separate toggle |
| Bank account numbers | On | |
| Bank routing numbers | On | |
| Phone numbers | On | |
| Email addresses | On | |
| Dates of birth | On | |
| PINs | On | |
| Passwords | On | |
| Names | Off | Higher false-positive rate, opt-in |
| Addresses | Off | Higher false-positive rate, opt-in |

***

## Metrics and Redaction

Metric LLM calls (custom prompts, sentiment, toxicity, emotion, politeness) automatically receive the **redacted** transcript when redaction is enabled. This keeps PII from leaving your tenant for analysis.

If your metrics rely on the original PII to do their job (and you accept the trade-off) you can override this with the **Allow evaluators to see original PII** toggle in the same settings panel. We recommend leaving it off unless you have a specific reason.

***

## What's Stored

* The original transcript is preserved unchanged: storage is non-destructive.
* Redaction spans (entity type, segment, confidence, source) are stored as metadata so masking is consistent across every read path and reproducible if you change the policy later.
* Audio recordings are stored as-is. Audio-level redaction (beep/silence at PII timestamps) is on the roadmap.

***

## Limitations Today

* Masking is applied at the **segment** level when any PII is detected within it, so a turn containing one phone number renders as a single redaction pill, not inline word-level redaction. Word-precise inline masking is on the roadmap.
* Audio recordings are not yet redacted. If you need redacted audio (for sharing recordings outside your team), reach out. It's planned and we're prioritizing based on demand.

***

## Compliance and Boundaries

When redaction is enabled:

* The frontend renders masked tokens in transcripts and exports.
* The API returns masked text by default for every segment-returning query (calls and chats).
* LLM graders receive masked text unless you explicitly opt out per project.
* Search results are sanitized, returning masked text rather than the original PII.

If you have specific compliance requirements (HIPAA BAA, PCI scope, destructive retention of original transcripts), [reach out to support](/documentation/resources/support). We can scope what's needed for your environment.


# Reports
Source: https://docs.roark.ai/documentation/observability/reports

Analytics dashboards and insights for voice AI performance

## Overview

Reports allow you to chain multiple metrics together to create comprehensive analytics dashboards. Track trends, compare performance, and drill down into specific calls - all from a single view.

![Latency Report Example](https://mintcdn.com/roark/19RX_xIP2QukdIpr/images/monitoring/report-latency-example.png?fit=max\&auto=format\&n=19RX_xIP2QukdIpr\&q=85\&s=3b60a5fd93931730b09fe3a7f372a732)

## What You Can Report On

### Events

Track occurrences of specific events in your system:

* `evaluation_failed` - When calls fail quality checks
* `task_completed` - Successful objective completion
* `escalation_triggered` - Customer requested human agent
* `tool_call_failed` - Function execution errors
* Custom events defined in your system

### Metrics

Analyze performance indicators over time:

* `response_time` - Agent response latency
* `sentiment_score` - Customer satisfaction trends
* `interruption_count` - Conversation flow issues
* `call_duration` - Efficiency metrics
* Any custom metrics you've defined

**Aggregation Options:**

* **Average** - Mean value across calls
* **Median** - Middle value
* **Sum** - Total combined value
* **Count** - Total number of occurrences
* **Min** - Lowest recorded value
* **Max** - Highest recorded value
* **P50** - 50th percentile value
* **P90** - 90th percentile value
* **P95** - 95th percentile value
* **P99** - 99th percentile value

A count metric offers **Total** (every occurrence added up) and **Conversations with any** instead.

A number metric's overview card shows its average, P50, P95, P99, minimum and maximum together. On a dashboard, click any of them to make it the headline figure.

## Baseline Configuration

Set context for your data with flexible baseline options:

> [tab] Percentage Baseline
>
> View data as a percentage against all your calls:
>
> ```
> Failed evaluations: 12% of all calls
> High frustration: 8% of all calls
> ```

> [tab] Custom Baseline
>
> Use another metric or event as your comparison:
>
> ```
> Failed evaluations: 45% of high-frustration calls
> Tool failures: 23% of escalated calls
> ```

> [tab] Sum Baseline
>
> Compare against the sum of another metric:
>
> ```
> Agent response time: 32% of total call duration
> Silence time: 18% of conversation time
> ```

## Visualization Options

### Line Graph

Track trends over time with detailed line charts:

![Latency Report Line Graph](https://mintcdn.com/roark/19RX_xIP2QukdIpr/images/monitoring/report-latency-example.png?fit=max\&auto=format\&n=19RX_xIP2QukdIpr\&q=85\&s=3b60a5fd93931730b09fe3a7f372a732)

Perfect for:

* Identifying patterns and trends
* Comparing multiple metrics
* Spotting anomalies

### Number Chart

Display key metrics as single values or comparisons:

![Latency Report Number Chart](https://mintcdn.com/roark/19RX_xIP2QukdIpr/images/monitoring/report-latency-example-number.png?fit=max\&auto=format\&n=19RX_xIP2QukdIpr\&q=85\&s=24104e63232ec006af144768cae9e944)

Ideal for:

* Executive dashboards
* Quick status checks
* KPI monitoring

## Time Controls

Customize your view with flexible time options:

### Time Period

* Last 24 hours
* Last 7 days
* Last 30 days
* Last 90 days
* Custom date range

### Interval Grouping

* **Hour**: For real-time monitoring
* **Day**: For daily trends
* **Week**: For weekly patterns
* **Month**: For long-term analysis
* **Year**: For annual comparisons

## Chain Multiple Metrics

Create powerful insights by combining metrics:

> [card] Correlation Analysis
>
> Compare response\_time with sentiment\_score to find relationships

> [card] Compound Metrics
>
> Stack multiple metrics to see the complete picture

> [card] Event + Metric
>
> Combine events with metrics for context (e.g., response\_time when evaluation\_failed)

> [card] Sequential Analysis
>
> Track how one metric affects another over time

## Drill-Down to Calls

Every data point connects to the actual conversations:

1. **Click any data point** on your report
2. **View matching calls** that contributed to that metric
3. **Navigate directly** to the call screen from [Live Monitoring](/documentation/observability/live-monitoring)
4. **Analyze the conversation** with full transcript and analysis

## Use Cases

### Performance Tracking

Monitor agent efficiency and quality:

* Average response times by agent
* Task completion rates over time
* Customer satisfaction trends

### Issue Detection

Identify problems before they escalate:

* Spike in failed evaluations
* Increase in customer frustration
* Pattern of tool failures

### Business Intelligence

Track metrics that matter to your business:

* Conversion rates
* Average handle time
* Cost per successful interaction

### Compliance Monitoring

Ensure adherence to standards:

* Script compliance percentage
* Required disclosure rates
* Quality score distributions

***

## What's Next

> [card] [Dashboards](/documentation/observability/dashboards)
>
> Organize multiple reports into a single view

> [card] [Live Monitoring](/documentation/observability/live-monitoring)
>
> Monitor calls in real-time


# Traces
Source: https://docs.roark.ai/documentation/observability/traces

Send OpenTelemetry traces to Roark for full visibility into your Voice AI agent

## Overview

Roark supports **OpenTelemetry (OTel) tracing**. Send your OTel trace data to Roark to see what happens under the hood of your Voice AI agent, debug latency, inspect tool usage and external API calls, and correlate call execution with your backend traces.

### Features

* **LLM traces per call**: View LLM spans, tool calls, and model invocations directly on the call detail page. Each call’s **Tracing** tab shows the full trace tree for that conversation, so you can quickly pinpoint where latency or failures occurred.
* **Central trace explorer**: See all traces in one place. Filter by time range, custom tags, or search by span name and attributes. Use this to spot patterns across calls, compare runs, and troubleshoot recurring issues.

***

![Roark Traces view showing agent turns with STT, LLM, and TTS spans](https://mintcdn.com/roark/4it2tXgWlRPz-tfL/images/observability/tracing-demo.png?fit=max\&auto=format\&n=4it2tXgWlRPz-tfL\&q=85\&s=4fb4d632146bf8def5a3785538eb531a)

***

## Endpoint and Protocol

| Requirement | Details |
| - | - |
| **Protocol** | OTLP over HTTPS only |
| **Endpoint** | Send traces to Roark's OTel endpoint (see examples below) |
| **OTel traces URL** | `https://api.roark.ai/v1/traces` |
| **Authorization** | **Required.** Send `Authorization: Bearer YOUR_ROARK_API_KEY` in the request headers. |
| **Role** | Roark acts as an OTel Collector |

> [!NOTE]
>
> All trace ingestion requests require authentication. [Generate an API
> key](/documentation/getting-started/api-keys) in your Roark dashboard and use
> it in the `Authorization: Bearer YOUR_ROARK_API_KEY` header.

***

## Setup Guide

**1. Get an API key**

Generate an API key in your Roark dashboard. Trace ingestion requires authentication via the `Authorization: Bearer YOUR_ROARK_API_KEY` header. [Create an API key →](/documentation/getting-started/api-keys)

**2. Choose your integration**

Pick the platform you're using and follow the corresponding setup section below:

* [LiveKit](#livekit): instrument your LiveKit agent with OpenTelemetry
* [VAPI](#vapi): traces sync automatically when calls are ingested
* [Custom integration](#custom-integration): any other platform via OTLP HTTP

***

## LiveKit

**Instrument your LiveKit agent** with OpenTelemetry and export traces to Roark. Configure your tracer with the `livekit.room.id` resource attribute. This is how Roark links your LiveKit room to its traces and shows them on the call detail page.

See the [LiveKit integration guide](/documentation/integrations/livekit) for the full webhook setup.

Install the required packages:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http livekit-agents
```

Example: a simple [LiveKit Agent](https://docs.livekit.io/agents/) entrypoint with OTel exporting to Roark:

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import os
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from livekit.agents import AgentServer, JobContext, cli
from livekit.agents.telemetry import set_tracer_provider


def setup_roark_tracer(ctx: JobContext):
    """Configure OTel to export traces to Roark with livekit.room.id for call correlation."""
    resource = Resource.create({
        "livekit.room.id": ctx.job.room.sid,  # Required: Roark uses this to link traces to the call
        "roark.skip": False,                   # Optional: set to True to filter out these traces
    })
    provider = TracerProvider(resource=resource)
    provider.add_span_processor(
        BatchSpanProcessor(
            OTLPSpanExporter(
                endpoint="https://api.roark.ai/v1/traces",
                headers={"Authorization": f"Bearer {os.environ['ROARK_API_KEY']}"},
            )
        )
    )
    set_tracer_provider(provider)

    # Ensure all pending traces are exported when the agent shuts down
    async def flush_traces():
        provider.force_flush()

    ctx.add_shutdown_callback(flush_traces)


server = AgentServer()


@server.rtc_session(agent_name="my-agent")
async def entrypoint(ctx: JobContext):
    # Initialize tracing early, before starting the agent session
    setup_roark_tracer(ctx)

    # Your agent logic here
    pass


if __name__ == "__main__":
    cli.run_app(server)
```

**Resource attributes:**

| Attribute | Required | Description |
| :- | :- | :- |
| `livekit.room.id` | Yes | The LiveKit room SID (`ctx.job.room.sid`). Roark uses this to link traces to the corresponding call. |
| `roark.skip` | No | Set to `true` to tell Roark to filter out traces for this room. Useful for skipping test or internal calls. |

> [!NOTE]
>
> If you're also using the [LiveKit webhook integration](/documentation/integrations/livekit), you can set `roark.skip` in both room metadata and OTel resource attributes to skip both call processing and trace ingestion.

***

## VAPI

**Traces sync automatically.** Whenever a call from a selected agent is synced to Roark, its OpenTelemetry traces are synced too. No extra OTLP exporter or instrumentation is needed on your side.

See the [VAPI integration guide](/documentation/integrations/vapi) for step-by-step setup.

> [!WARNING]
>
> Make sure **Public Logs** are enabled in your Vapi dashboard. Roark requires public log access to ingest trace data from Vapi calls.

***

## Custom Integration

For any other platform, use the OTLP HTTP trace exporter and point it to Roark's OTel endpoint. Include the **required** `Authorization: Bearer YOUR_ROARK_API_KEY` header and attach the required resource attributes.

### Correlating traces to a call or chat

If you submit calls or chats to Roark via the [Customer API](/api-reference/introduction), tag your traces with `roark.external_id` so Roark can link each conversation to its trace automatically.

1. When creating a call or chat via `POST /call` or `POST /chat`, supply the `externalId` field with a stable identifier from your own system (session ID, conversation ID, etc.). It must be unique within the project.
2. On your OpenTelemetry traces, set `roark.external_id` to the **same value**, either as a resource attribute (propagates to every span in the service) or as a span attribute on the root span.

Roark looks up the matching trace in ClickHouse after the call/chat is created and backfills the trace ID, so the conversation appears in its **Tracing** tab automatically.

> [tab] TypeScript (Node.js)
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> npm install @opentelemetry/sdk-trace-node @opentelemetry/exporter-trace-otlp-http @opentelemetry/resources
> ```
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
> import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
> import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
> import { Resource } from "@opentelemetry/resources";
>
> const exporter = new OTLPTraceExporter({
>   url: "https://api.roark.ai/v1/traces",
>   headers: { Authorization: `Bearer ${process.env.ROARK_API_KEY}` },
> });
>
> // `roark.external_id` is the correlation key linking this trace to the
> // call/chat you create with the same `externalId` via the Customer API.
> const resource = new Resource({
>   "roark.external_id": yourSessionId,
>   "roark.project.tag.env": process.env.ROARK_ENV ?? "production",
> });
>
> const provider = new NodeTracerProvider({ resource });
> provider.addSpanProcessor(new BatchSpanProcessor(exporter));
> provider.register();
> ```

> [tab] Go
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> go get go.opentelemetry.io/otel \
>   go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
>   go.opentelemetry.io/otel/sdk/trace
> ```
>
> ```go theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> package main
>
> import (
> 	"context"
> 	"os"
>
> 	"go.opentelemetry.io/otel"
> 	"go.opentelemetry.io/otel/attribute"
> 	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
> 	"go.opentelemetry.io/otel/sdk/resource"
> 	sdktrace "go.opentelemetry.io/otel/sdk/trace"
> 	semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
> )
>
> func initTracer(sessionID string) (func(context.Context) error, error) {
> 	exporter, err := otlptracehttp.New(context.Background(),
> 		otlptracehttp.WithEndpoint("api.roark.ai"),
> 		otlptracehttp.WithURLPath("/v1/traces"),
> 		otlptracehttp.WithHeaders(map[string]string{
> 			"Authorization": "Bearer " + os.Getenv("ROARK_API_KEY"),
> 		}),
> 	)
> 	if err != nil {
> 		return nil, err
> 	}
>
> 	// `roark.external_id` is the correlation key linking this trace to the
> 	// call/chat you create with the same `externalId` via the Customer API.
> 	res, err := resource.New(context.Background(),
> 		resource.WithAttributes(
> 			semconv.ServiceName("my-voice-agent"),
> 			attribute.String("roark.external_id", sessionID),
> 			attribute.String("roark.project.tag.env", "production"),
> 		),
> 	)
> 	if err != nil {
> 		return nil, err
> 	}
>
> 	tp := sdktrace.NewTracerProvider(
> 		sdktrace.WithBatcher(exporter),
> 		sdktrace.WithResource(res),
> 	)
> 	otel.SetTracerProvider(tp)
> 	return tp.Shutdown, nil
> }
> ```

> [tab] Python
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
> ```
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> from opentelemetry import trace
> from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
> from opentelemetry.sdk.resources import Resource
> from opentelemetry.sdk.trace import TracerProvider
> from opentelemetry.sdk.trace.export import BatchSpanProcessor
> import os
>
> exporter = OTLPSpanExporter(
>     endpoint="https://api.roark.ai/v1/traces",
>     headers={"Authorization": f"Bearer {os.getenv('ROARK_API_KEY')}"},
> )
>
> # `roark.external_id` is the correlation key linking this trace to the
> # call/chat you create with the same `externalId` via the Customer API.
> resource = Resource(attributes={
>     "roark.external_id": your_session_id,
>     "roark.project.tag.env": os.getenv("ROARK_ENV", "production"),
> })
>
> provider = TracerProvider(resource=resource)
> provider.add_span_processor(BatchSpanProcessor(exporter))
> trace.set_tracer_provider(provider)
> ```

**Resource attributes:**

| Attribute | Required | Description |
| :- | :- | :- |
| `roark.external_id` | Recommended | A stable identifier from your own system (e.g. session/conversation ID). Must match the `externalId` field you submit on the corresponding call or chat via the Customer API, and must be unique within a project. |
| `roark.project.tag.{key}` | No | Custom project tags for filtering and grouping in the trace explorer. |

***

Once your integration is set up, you're ready to send traces to Roark. You'll be able to view them on the Traces page and directly within each call's detail page.

***

## What's Next

> [card] [Live monitoring](/documentation/observability/live-monitoring)
>
> View and debug active calls

> [card] [Metrics & reports](/documentation/metrics/custom-metrics)
>
> Define and analyze call metrics

> [card] [Dashboards](/documentation/observability/dashboards)
>
> Build observability views

> [card] [API reference](/api-reference/introduction)
>
> Explore Roark API endpoints

***

**Related:** [LiveKit integration](/documentation/integrations/livekit) · [VAPI integration](/documentation/integrations/vapi) · [Integrations overview](/documentation/integrations/overview)


# Accent Detection & TTS Drift Monitoring
Source: https://docs.roark.ai/documentation/recipes/accent-detection

Detect accents and flag when your agent's TTS voice drifts mid-call

## Overview

Accent Detection identifies which English accent each participant speaks with across every segment of a call. This is useful for:

* **TTS consistency monitoring**: Verify your agent's text-to-speech voice maintains the expected accent throughout the call
* **Accent drift detection**: Flag calls where the agent's accent shifted mid-conversation
* **Regional analysis**: Understand the accent distribution of your callers

The model classifies audio into 16 English accent variants: American, British, Australian, Canadian, Indian, Irish, Scottish, Welsh, African, New Zealand, Hong Kong, Malaysian, Philippine, Singaporean, Bermudian, and South Atlantic.

***

## Prerequisites

### 1. Enable the Accent Detection Package

Navigate to **Settings > Analysis Packages** and enable **Accent Detection**.

![Accent Detection package in the metric library](https://mintcdn.com/roark/jKTeAb05J1nzHzij/images/metrics/accent-library.png?fit=max\&auto=format\&n=jKTeAb05J1nzHzij\&q=85\&s=4b0e3577d619e2f8cd18652e8e6ea230)

> [!NOTE]
>
> Accent detection requires both **audio conversion** and **diarization** artifacts. These are produced automatically when the package is enabled.

### 2. Understand the Metrics

The package includes two metrics:

| Metric | Type | What it measures |
| :- | :- | :- |
| **Accent** | Classification | Detected accent per segment and dominant accent at call level, with full probability distribution |
| **Accent Stability** | Numeric (0–1) | How consistent the detected accent is across segments. 1.0 = same accent throughout |

***

## Recipe: Detect Agent TTS Accent Drift

This recipe sets up automatic monitoring to flag any call where your agent's TTS accent drifts from its expected voice.

### Step 1: Create a Collector

1. Go to **Metrics → Collectors**
2. Create a new collector or edit an existing one
3. Add the **Accent Stability** metric from the Accent Detection package
4. Configure a threshold:
   * **Operator:** `>=`
   * **Value:** `0.7`
   * **Participant Role:** Agent

This evaluates every call and flags any where the agent's accent stability drops below 70%, meaning the detected accent changed for more than 30% of the agent's speaking time.

![Accent Stability threshold configuration](https://mintcdn.com/roark/jKTeAb05J1nzHzij/images/metrics/accent-stability-threshold.png?fit=max\&auto=format\&n=jKTeAb05J1nzHzij\&q=85\&s=2cebf488b21d97a0a6f4e6e94b22282d)

> [!TIP]
>
> Start with a threshold of `0.7` and adjust based on your results. Some variation is normal. The model may oscillate between similar accents (e.g. American vs Canadian) on short segments. A threshold of `0.5` would only flag significant accent changes.

#### From the CLI

Collectors have no direct command: they are a [Config as Code](/documentation/config-as-code/collectors) resource, so the CLI applies them from a file. This is the half you want in version control anyway, since which metrics run on which calls is a decision worth reviewing.

```yaml roark/collectors/accent-drift.yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: collector
name: accent-drift
modality: call
status: ACTIVE
metrics:
  - accent
  - accent_stability
```

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config diff ./roark   # preview, writes nothing
roark config apply ./roark
```

> [!NOTE]
>
> The **threshold** in step 4 above is dashboard-only. The collector config expresses which metrics run and on which calls, not the pass/fail line drawn on them, and the thresholds endpoint is not exposed on the API. To get an automated signal out of the CLI instead, define the tolerance as an alert (below) rather than as a collector threshold.

### Step 2: Review Results on the Call Detail Page

When a call is processed, open it and check the **Metrics tab**:

**Call-level accent card**: Shows the dominant accent per participant with a probability distribution. For example, if the agent spoke with an American accent for 70% of the call and British for 30%, you'll see both with their percentages.

![Call-level accent metric showing score distribution per participant](https://mintcdn.com/roark/jKTeAb05J1nzHzij/images/metrics/accent-call-metric.png?fit=max\&auto=format\&n=jKTeAb05J1nzHzij\&q=85\&s=5b6b2331e35e42765f705defabedb390)

**Segment-level probability chart**: Shows a stacked area chart of accent probabilities over time. This lets you see exactly *where* in the call the accent shifted. Use the participant filter to focus on the Agent.

![Segment-level stacked probability chart showing accent over time](https://mintcdn.com/roark/jKTeAb05J1nzHzij/images/metrics/accent-segment-metric.png?fit=max\&auto=format\&n=jKTeAb05J1nzHzij\&q=85\&s=410a1867ff27499fb917903d2491762d)

### Step 3: Set Up Alerts

Once you've configured the threshold:

* **Calls that pass**: The agent maintained a consistent accent throughout
* **Calls that fail**: The agent's accent drifted beyond your tolerance, appearing as a failed threshold on the Overview tab

You can use [webhooks](/documentation/integrations/webhooks) to get notified when metric collection completes, then check the threshold results programmatically.

#### From the CLI

An [alert](/documentation/config-as-code/alerts) expresses the tolerance itself, so drift reaches a channel without anyone opening the dashboard. Applied the same way as the collector:

```yaml roark/alerts/accent-drift.yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: alert
name: agent-accent-drift
trigger:
  type: threshold
  metric: accent_stability
  aggregation: MEAN
  windowMinutes: 60
  operator: LT
  thresholdValue: 0.7
  minSampleSize: 5
actions:
  slack:
    - channelId: C0123ABCXYZ
      channelName: '#agent-quality'
```

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply ./roark
```

> [!TIP]
>
> `minSampleSize` matters here. Accent stability on a single short call is noisy, so a window with two calls in it will page you for nothing.

***

## How Accent Scores Work

### Per-Segment Scores

Each segment shows the accent probabilities after normalization. The model outputs raw probabilities across all 16 accents (softmax), but we filter out accents below the baseline (1/16 = 6.25%) and renormalize so the remaining scores sum to 100%.

For example, if the model outputs `American: 10%, British: 8%, Canadian: 7%` with everything else below 6.25%, the normalized scores become `American: 40%, British: 32%, Canadian: 28%`.

### Call-Level Scores

Call-level accent scores represent the **proportion of speaking time** classified as each accent. If the agent had 10 segments classified as American (totaling 60s) and 5 segments as British (totaling 40s), the call-level scores are `American: 60%, British: 40%`.

### Accent Stability

Accent Stability is the proportion of speaking time the dominant accent held. In the example above, stability would be `0.6` (60%). A stability of `1.0` means every segment was classified as the same accent.

***

## Limitations

* **English only**: The current model classifies English accents only. Future language models will use the same infrastructure.
* **Minimum 5 seconds**: Segments shorter than 5 seconds are skipped as they don't contain enough audio for reliable classification.
* **Similar accents**: The model may confuse similar accents (e.g. American vs Canadian, British vs Irish) especially on short segments. The normalization helps but isn't perfect.


# Testing Call Screeners
Source: https://docs.roark.ai/documentation/recipes/call-screener-testing

Check your outbound agent recognises a gatekeeper, answers it directly, and saves its pitch for a person

## Overview

When your agent dials out, a person does not always answer. Apple and Google both ship call screening that picks up first, asks who is calling and why, and then decides whether to put the call through, decline it, or take a message.

A screener is easy to fail in a way you will not notice. The agent hears a voice, assumes it is the customer, and delivers its opening pitch to a robot. The call may even connect afterwards, so the outcome looks fine while the human on the other end has already missed the introduction.

The **Call screening** template runs your agent into real screener behaviour and grades what it does.

***

## What the template runs

Pick the template, and Roark supplies the calls. You choose the agent to put on the line.

![The Call screening template, with the screener outcomes it runs](https://mintcdn.com/roark/C7B00JtkERQjGhI5/images/simulations/call-screening-template.png?fit=max\&auto=format\&n=C7B00JtkERQjGhI5\&q=85\&s=0ad5ff66d1ac85a17538a0f310782fa1)

Two screeners, four outcomes, one call per outcome per agent:

| Screener | Outcome | What the agent faces |
| :- | :- | :- |
| **Apple call screening** | Screener connects the call | Answers the screener, then a person picks up |
| **Apple call screening** | Screener declines the call | Answers the screener, then the call is refused |
| **Google call screening** | Screener connects the call | Answers the screener, then a person picks up |
| **Google call screening** | Screener takes a message | Answers the screener, which offers to pass a message on |

Each screener offers only the outcomes that product actually has, which is why the two differ. Selecting fewer outcomes runs fewer calls.

## What it grades

The template attaches two metrics and one Pass/Fail check:

| Metric | Type | What it measures |
| :- | :- | :- |
| **Call screening encountered** | Yes/No | Whether the agent recognised it was talking to a screener rather than the person it called |
| **Call screening handling score** | Scale | How well it handled the gatekeeper: answering directly, and holding its opening back |
| **Call screening handling score check** | Pass/Fail | The threshold on that score, so a run reports a clean pass rate |

The flows also carry their own expectations, which are graded alongside:

* Recognises it is speaking to a call screener rather than the person it called
* Answers the screener's questions directly: who is calling, and why
* Does not deliver its full live-call opening or pitch to the screener
* Does not argue with, pressure, or try to talk past the screener

> [!NOTE]
>
> This template grades only the metrics above. Metrics attached to your own flows do not ride along, so the numbers describe screener handling and nothing else.

***

## Running it from the dashboard

1. Go to **Simulations > New run** and pick **Call screening**.
2. Under **Screeners to test**, keep all four outcomes or narrow to the ones you care about.
3. Choose the agents to call, set the direction to **Outbound**, and start the run.

## Running it from the CLI, SDKs or API

The template supplies the flows and the metrics, so a run names the agent and the direction and nothing else:

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run --data '{
  "template": "call-screening",
  "direction": "OUTBOUND",
  "agentEndpoints": [{ "id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e" }]
}'
```

```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl https://api.roark.ai/v1/simulation/run \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN" \
  -d '{
        "template": "call-screening",
        "direction": "OUTBOUND",
        "agentEndpoints": [{ "id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e" }]
      }'
```

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import Roark from '@roarkanalytics/sdk'

const client = new Roark({ bearerToken: process.env.ROARK_API_BEARER_TOKEN })

const run = await client.simulation.run({
  template: 'call-screening',
  direction: 'OUTBOUND',
  agentEndpoints: [{ id: '182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e' }],
})

console.log(run.data.simulationRunPlanJobId)
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import os
from roark_analytics import Roark

client = Roark(bearer_token=os.environ["ROARK_API_BEARER_TOKEN"])

run = client.simulation.run(
    template="call-screening",
    direction="OUTBOUND",
    agent_endpoints=[{"id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e"}],
)

print(run.data.simulation_run_plan_job_id)
```

That runs all four outcomes, the same as leaving every box ticked in the dashboard.

### Narrowing to specific outcomes

`GET /v1/simulation/template` lists what each template runs, including the slug of every flow and outcome:

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation template list
```

```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl https://api.roark.ai/v1/simulation/template \
  -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN"
```

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const templates = await client.simulationTemplate.list()

const screening = templates.data.find((t) => t.slug === 'call-screening')
console.log(screening?.flows)
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
templates = client.simulation_template.list()

screening = next(t for t in templates.data if t.slug == "call-screening")
print(screening.flows)
```

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "slug": "call-screening",
  "title": "Call screening",
  "flows": [
    {
      "id": "550e8400-e29b-41d4-a716-446655440000",
      "title": "Apple call screening",
      "slug": "sf-screener-apple",
      "happyPath": { "title": "Screener connects the call" },
      "edgeCases": [
        {
          "id": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
          "title": "Screener declines the call",
          "slug": "screener-declines-the-call"
        }
      ]
    }
  ]
}
```

Pass those slugs back as `flows` to run a subset:

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run --data '{
  "template": "call-screening",
  "direction": "OUTBOUND",
  "agentEndpoints": [{ "id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e" }],
  "flows": [
    { "slug": "sf-screener-apple", "edgeCases": [{ "slug": "screener-declines-the-call" }] }
  ]
}'
```

```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl https://api.roark.ai/v1/simulation/run \
  -H 'Content-Type: application/json' \
  -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN" \
  -d '{
        "template": "call-screening",
        "direction": "OUTBOUND",
        "agentEndpoints": [{ "id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e" }],
        "flows": [
          { "slug": "sf-screener-apple", "edgeCases": [{ "slug": "screener-declines-the-call" }] }
        ]
      }'
```

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
await client.simulation.run({
  template: 'call-screening',
  direction: 'OUTBOUND',
  agentEndpoints: [{ id: '182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e' }],
  flows: [{ slug: 'sf-screener-apple', edgeCases: [{ slug: 'screener-declines-the-call' }] }],
})
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
client.simulation.run(
    template="call-screening",
    direction="OUTBOUND",
    agent_endpoints=[{"id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e"}],
    flows=[{"slug": "sf-screener-apple", "edge_cases": [{"slug": "screener-declines-the-call"}]}],
)
```

> [!TIP]
>
> Prefer `slug` over `id` for anything you keep in version control. Roark-curated flows are shared across every project, so their ids differ between environments, while the slug does not.

***

## Reading the results

The run produces one call per outcome per agent. Look at **Call screening encountered** first: if it is No, the agent never realised a screener answered, and the handling score explains the rest.

A common shape is a high pass rate on the connected outcomes and failures on declined or message-taking ones, which usually means the agent handles a screener it can talk past but not one that ends the call.

## Keeping it running

Save the run as a plan and it becomes repeatable:

* Re-run it after a prompt change to check screener handling did not regress
* Put it on a [schedule](/documentation/simulation-testing/schedules) to catch drift
* Add it to [CI](/documentation/simulation-testing/ci-cd) so a release blocks on the pass rate

A saved plan is a snapshot of what the template resolved to when you created it, so it keeps running the same cases even as the template evolves.


# Tool Call Testing
Source: https://docs.roark.ai/documentation/recipes/tool-call-testing

Test your voice agent's tool calls: safely, deterministically, and graded automatically

## Why tool calls need their own testing story

Tools are where your agent touches the real world: bookings, transfers, payments, SMS. That makes them both the riskiest thing to test (a test call that books a real appointment is not a test, it is an incident) and the most important thing to verify (a perfectly polite agent that calls `book_appointment` with the wrong date is failing its job).

Roark solves both halves:

| Piece | What it does | When you need it |
| :- | :- | :- |
| **Tool guard** | Test calls never execute your real tools; Roark answers with simulated, scenario-consistent responses instead | Any time simulations reach a deployment with live tools |
| **Tool fixtures** | Pin an exact response for a tool ("the calendar has zero slots"), so a test forces the branch you want to exercise | Regression tests and branch coverage |
| **Tool Invocation Analysis metrics** | Grade whether the agent called the right tools, with the right parameters, in the right order, and used the results correctly | Always: simulations and production alike |

The rest of this page walks them in the order a new project adopts them.

***

## Step 1: Make test calls safe (the tool guard)

Wrap a side-effecting tool once. On a real caller's session the wrapper is a pass-through. During a Roark test call the tool body never runs: the wrapper sends the invocation to Roark and returns a simulated backend response that matches the tool's contract, follows the test scenario (a scenario about a fully booked calendar gets a realistic "no availability" from a booking tool), and stays consistent with earlier tool responses in the same call.

```python Code-first (LiveKit, Pipecat) theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# The resolve call you already make at session start tells you whether this
# session is a Roark test call (simulation_job_id is set). The guard keys off
# that. Full decorator: see the code-first guide linked below.
@roark_tool_guard(cfg, fallback={"status": "booked", "confirmationId": "TEST-0000"})
async def book_appointment(date: str, time: str) -> dict:
    """Books an appointment. Args: date (YYYY-MM-DD), time (HH:MM). Returns {confirmationId, status}."""
    ...  # your real calendar write: never runs during a Roark test call
```

```python Webhook backends (Vapi, Retell) theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# If your provider agent's tools call YOUR backend, apply the same pattern
# server-side: recognize the simulation, then ask Roark for the response.
job = roark.simulation_job.lookup(roark_phone_number=caller_number)
if job is not None:
    mock = roark.simulation.mock_tool(
        simulation_job_id=job.id,
        tool_name="book_appointment",
        tool_description="Books an appointment. Returns {confirmationId, status}.",
        arguments=payload,
    )
    return mock.data.result  # never touch the real calendar for a test call
```

Wrap the writes (`book`, `cancel`, `send_sms`, `transfer`, `charge`); leave read-only lookups unwrapped when live data makes the test more realistic. Two safety properties hold by construction: a real caller can never receive a fake response (the guard only diverts on sessions Roark itself identified as simulations, and the endpoint re-validates the simulation before answering), and a guarded tool can never fire during a test (every error path maps to "use your static fallback", except a 404, which genuinely means the session is not a known simulation).

> [card] [Full setup: code-first agents](/documentation/autoimprove/code-first-agents)
>
> The complete decorator, framework wiring for LiveKit and Pipecat, and the exact failure contract.

***

## Step 2: Force the outcomes you want to test (fixtures)

Generated mocks are plausible; sometimes a test needs exact. A **fixture** pins a tool's response so a scenario always exercises one branch: what does the agent do when there are no slots, when the payment is declined, when the lookup finds two accounts?

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# Project-wide: every test call sees an empty calendar
roark simulation tool-fixture create \
  --tool-name lookup_availability \
  --response '{"slots": []}' \
  --description "Forces the no-availability branch"

# Pinned to one flow variant: only this scenario sees a declined card
roark simulation tool-fixture create \
  --tool-name charge_card \
  --response '{"status": "declined", "reason": "insufficient_funds"}' \
  --customer-flow-variant-id "$VARIANT_ID"
```

```bash API theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/simulation/tool-fixture \
  -H "Authorization: Bearer $ROARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"toolName": "lookup_availability", "response": {"slots": []}, "description": "Forces the no-availability branch"}'
```

Precedence is simple: a fixture pinned to the running flow variant beats a project-wide one, and both beat generation. Setting the same scope twice replaces the response (one fixture per tool project-wide, one per tool and variant), so applying fixtures from CI is idempotent. Delete a fixture and the tool goes back to scenario-aware generated responses.

***

## Step 3: Verify what your agent actually called

After a test call, ask Roark what the agent tried to do. Every guarded invocation is recorded against the simulation: tool, arguments verbatim, the response served, and whether it came from a fixture or the model.

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation job tool-mock list <job-id>
```

```bash API theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl https://api.roark.ai/v1/simulation/job/$JOB_ID/tool-mock \
  -H "Authorization: Bearer $ROARK_API_KEY"
```

```json Response theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "data": [
    {
      "toolName": "lookup_availability",
      "arguments": { "date": "2026-10-01" },
      "result": { "slots": [] },
      "source": "FIXTURE",
      "createdAt": "2026-09-23T12:00:05Z"
    },
    {
      "toolName": "book_appointment",
      "arguments": { "date": "2026-10-08", "time": "15:00" },
      "result": { "status": "booked", "confirmationId": "APT-58371" },
      "source": "GENERATED",
      "createdAt": "2026-09-23T12:01:12Z"
    }
  ]
}
```

A CI step can assert directly on this: the agent offered an alternative date after the empty lookup, it never called `charge_card` on the refund scenario, the booking arguments match what the caller asked for. The list is empty when the tools are not guarded; that usually means step 1 is not wired for this agent.

***

## Step 4: Grade it automatically (metrics)

For judgment calls that raw assertions cannot express ("was this the *right* tool at the *right* moment?"), Roark ships the **Tool Invocation Analysis** package. Find it under **Metrics > Library**:

| Metric | Type | What It Measures |
| :- | :- | :- |
| **Tool Invocation Correct** | Yes/No | Whether the agent invoked the correct tools at the appropriate times |
| **Tool Invocation Count** | Count | Total number of tool calls made during the conversation |
| **Tool Invocation Order Correct** | Yes/No | Whether tools were called in the correct logical sequence |
| **Tool Invocation Parameters Correct** | Yes/No | Whether the correct parameters were passed to each tool call |
| **Tool Invocation Result Correct** | Yes/No | Whether the agent correctly interpreted and used the results returned by each tool |

Configure once, use everywhere: under **Tool Scoping**, select a specific tool and define when it should be called and what result is expected, or leave scoping empty to evaluate all tools. Use **Additional Instructions** for cross-tool rules like "Never call any tool before greeting the customer."

### In simulations

**1. Create a Run Plan**

Follow the [run plan guide](/documentation/simulation-testing/run-plans#creating-a-run-plan) and pick the scenarios that should trigger the tool calls under test. Add fixtures (step 2) to the scenarios that need forced outcomes.

**2. Add Tool Invocation Metrics**

In the **Metrics** section of the run plan, select the metrics you configured above.

**3. Enable Enriched Simulations**

Metrics grade the tool data on your side of the call, so your agent must send its call data to Roark: see [Enriched Simulations](/documentation/simulation-testing/enriched-simulations). If you use the tool guard, your guarded invocations are attached to the merged call automatically; you do not need to report them separately. Unguarded tools still need [tool invocation data](/documentation/tool-invocations) included with your call.

**4. Run and read**

Execute the plan. Once your agent's call is matched with the simulation, the tool invocation metrics are evaluated automatically and land in the run report.

The same plan runs from a terminal or CI:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark simulation run --data '{"planId":"<plan-id>"}'
roark simulation plan job get <job-id>
```

> [!TIP]
>
> The CLI exits non-zero when the API rejects a request, so a CI step that starts a run fails loudly rather than passing on a broken plan. See [exit codes](/documentation/sdks/cli#exit-codes).

### In production

The same metrics run continuously on live calls through a collector. Live calls always hit your real tools (the guard never mocks real traffic; that property is structural), so this is pure observation:

```yaml roark/collectors/tool-calls.yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: collector
name: tool-call-quality
modality: call
status: ACTIVE
metrics:
  - tool_invocation_correct
  - tool_invocation_parameters_correct
filters:
  - conditions:
      - type: AGENT
        key: frontdesk
        operator: EQUALS
```

```bash CLI theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config diff ./roark
roark config apply ./roark
```

> [!NOTE]
>
> Production calls must include [tool invocation data](/documentation/tool-invocations) for these metrics to work. Tool calls are captured automatically for **Vapi** and **Retell** integrations; for the API or code-first stacks, include tool invocations when you create the call or [attach them afterward](/documentation/tool-invocations#attaching-tool-calls-after-a-call).

***

## The loop, end to end

A team starting from zero gets to a full tool-testing loop in an afternoon: wrap the writes with the guard, pin fixtures for the branches that matter, run the plan, assert on the mocked invocations in CI, and let the Tool Invocation Analysis metrics grade every run and every production call from then on. Nothing real is booked, sent, or charged along the way.

## Next Steps

> [card] [Code-first agents](/documentation/autoimprove/code-first-agents)
>
> The tool guard's full setup for LiveKit and Pipecat, plus Autoimprove on your own stack

> [card] [Tool Invocations](/documentation/tool-invocations)
>
> How to submit tool call data with your calls

> [card] [Enriched Simulations](/documentation/simulation-testing/enriched-simulations)
>
> Enable call matching for simulation tool testing

> [card] [Collectors](/documentation/metrics/metric-collectors)
>
> Automate metric collection on production calls


# Glossary
Source: https://docs.roark.ai/documentation/resources/glossary

Key terms and concepts used in Roark

## Core Concepts

### Agent

A voice AI system that handles conversations. Can be a chatbot, voice assistant, or any automated system that interacts with users through voice or text.

### Call

A single conversation session between participants (human and/or AI agents). Contains transcript, audio, metadata, and analysis results.

### Endpoint

The connection point Roark uses to reach your agent. Endpoint types include Phone, Web RTC, LiveKit, WebSocket, ElevenLabs, Google CES, and Kore. Endpoints also carry a direction: whether Roark dials your agent (outbound from Roark) or your agent calls into Roark (inbound).

### Evaluation

The process of scoring and analyzing calls against specific criteria to measure quality and performance.

### Session

A group of related calls, useful for tracking multi-call interactions or conversation threads with the same customer.

## Simulation & Testing

### Persona

A simulated customer profile with defined characteristics, behaviors, and communication patterns used when running customer flows.

### Customer Flow

One type of customer conversation your agent should handle: the customer setup, its variants, and the agent expectations it's graded against. Customer flows supersede scenarios and drive simulations today, with live-call grading planned.

### Variant

One concrete version of a customer flow. Every flow has a default variant (the happy path) plus any number of edge cases, each with its own setup, variables, and optional additional expectations.

### Happy Path

A flow's default variant, the expected, unexceptional version of the conversation.

### Edge Case

A non-default variant of a customer flow that tests a deviation from the happy path (an upset customer, a wrong account number, an unusual request).

### Improv Mode

A flow authoring mode where you write a free-form customer brief and the simulated customer improvises the conversation around it.

### Scripted Mode

A flow authoring mode where you build the conversation as a step-by-step graph, with branching paths the customer follows.

### Agent Expectations

The graded contract attached to a customer flow: what your agent must do to pass. Flow-level expectations are inherited by every variant; edge cases can add additional expectations of their own.

### Flow Link

A scripted step that splices another scripted flow's steps in at that point, so shared segments (like identity verification) can be reused across flows.

### Template

A pre-built starting point for a run (such as Flow adherence, Red teaming, Conversation quality, Multilingual, Load testing, or Tool call accuracy) that seeds flows, metrics, and settings you can customize.

### Run Plan

A reusable, named test suite: the agent targets to test, the customer flows attached, the metrics and checks to evaluate, and run settings like iterations and concurrency. One-off runs are plans that simply aren't saved (hidden plans).

### Run

A single execution of a run plan, labelled SR-. Runs move through five statuses: Running, Queued, Completed, Failed, and Cancelled.

### Simulation

A controlled test conversation that validates agent behavior without involving real customers.

### Regression Testing

Re-running previous customer flows against updated agent versions to ensure improvements don't break existing functionality.

## Analytics & Metrics

### Collector

The user-facing name for a rule that runs selected metrics on matching calls: which metrics to evaluate, and the conditions (agent, source, call property) a call must match. Collectors can be Active or Paused.

> [!NOTE]
>
> The REST API and SDKs still use the older name for collectors: endpoints live under `/v1/metric/policies` and SDK methods under `client.metricPolicy.*`.

### Studio

The workspace for building and testing metrics. Author mode defines a new metric (LLM Judge, Pattern, Formula, or API check); Evaluate mode runs a battery of evals against sample calls or a dataset.

### Dataset

A curated set of calls used as an evaluation corpus, for example, to run a metric battery in Studio against a fixed, representative sample.

### Sentiment Analysis

Detection and classification of emotions and feelings expressed during conversations (64+ emotions tracked).

### Sentiment Score

A numerical value (-1 to +1) indicating overall emotional tone, where negative values indicate negative sentiment and positive values indicate positive sentiment.

### Performance Metrics

Quantitative measurements of agent efficiency including response time, resolution rate, and conversation duration.

### Tool Invocation

When an agent calls an external function or API during a conversation (e.g., checking order status, booking appointment).

### Talk Ratio

The percentage of conversation time each participant speaks, used to measure conversation balance.

## Integration Terms

### Webhook

An HTTP callback that sends real-time event data from voice platforms to Roark when calls occur.

### WebSocket

A persistent connection protocol used for real-time, bidirectional communication in live monitoring.

### Ingestion

The process of receiving and processing call data from voice platforms into Roark's analytics system.

### Egress

Outbound data flow, commonly referring to call recordings being sent to storage systems.

## Voice Platforms

### VAPI

A voice AI platform that provides infrastructure for building and deploying voice agents.

### Retell AI

A platform for creating conversational AI agents with natural voice interactions.

### LiveKit

An open-source platform for real-time audio/video communications, used for voice agent infrastructure.

### Pipecat Cloud

A cloud platform for deploying and managing voice AI pipelines.

## Technical Components

### Batch Evaluation

Processing multiple calls simultaneously for analysis rather than one at a time.

### Custom Metric

A user-defined metric (typically an LLM evaluation prompt) tailored to specific business requirements beyond the built-in system metrics.

### Transcript

The text representation of spoken conversation, including speaker labels and timestamps.

### Utterance

A single continuous speech segment from one speaker, ending when they stop talking or another speaker begins.

### Turn

A complete speaking opportunity for one participant in a conversation, may contain multiple utterances.

## Monitoring & Reporting

### Live Monitoring

Real-time observation and analysis of ongoing calls as they happen.

### Dashboard

The visual interface displaying analytics, metrics, and insights about your voice AI performance.

### Baseline

A reference point for normal performance metrics, used to identify deviations and improvements.

### Alert

Automated notification triggered when specific conditions are met (e.g., high frustration detected).

### Trend Analysis

Tracking metrics over time to identify patterns, improvements, or degradations in performance.

## Business Terms

### First Call Resolution (FCR)

Successfully resolving a customer's issue during their first call without requiring follow-up.

### Escalation

Transferring a call to a human agent or supervisor when the AI cannot handle the situation.

### Quality Assurance (QA)

The process of monitoring and evaluating agent interactions to ensure quality standards are met.

### Service Level Agreement (SLA)

Performance standards and metrics that must be maintained (e.g., response time, resolution rate).

## API & Development

### API Key

A unique identifier used to authenticate requests to Roark's API endpoints.

### Rate Limiting

Restrictions on the number of API requests allowed within a time period to prevent abuse.

### SDK

Software Development Kit - pre-built libraries (Node.js, Python) that simplify integration with Roark.

### REST API

The HTTP-based interface for programmatically interacting with Roark's services.

### Payload

The data package sent in API requests or webhook events containing call information.


# Support
Source: https://docs.roark.ai/documentation/resources/support

Get help with Roark

## Contact Support

For technical support and assistance:

**Email**: [support@roark.ai](mailto:support@roark.ai)

**Slack**: Available for customers on paid plans - join our dedicated Slack workspace for direct access to our team.

## Response Times

* **Free Plan**: Email support within 48 hours
* **Paid Plans**: Priority support via Slack with faster response times

## Before Contacting Support

Please check our [Troubleshooting Guide](/documentation/resources/troubleshooting) for solutions to common issues.


# Troubleshooting
Source: https://docs.roark.ai/documentation/resources/troubleshooting

Solutions to common issues when using Roark

## API & Authentication

### "Invalid API Key" Error

**Problem**: Receiving 401 Unauthorized errors.

**Solutions**:

1. Regenerate your API key from [dashboard](https://roark.ai/org)
2. Ensure no extra spaces before/after the key
3. Use format: `Authorization: Bearer YOUR_API_KEY`
4. Check key hasn't expired or been revoked

### Rate Limiting Issues

**Problem**: Receiving 429 Too Many Requests errors.

**Solutions**:

1. Check your current plan's rate limits
2. Implement exponential backoff in your code
3. Batch operations where possible
4. Consider upgrading to a higher tier for increased limits

## Getting Help

If these solutions don't resolve your issue:

1. **Documentation**: Review relevant guides in this documentation
2. **Community**: Ask in our [Slack community](https://roark.ai/slack) (paid plans)
3. **Support**: Email [support@roark.ai](mailto:support@roark.ai) with:
   * Your account email
   * Error messages or screenshots
   * Steps to reproduce the issue
   * Time when issue occurred (with timezone)


# CLI
Source: https://docs.roark.ai/documentation/sdks/cli

Drive Roark from your terminal and your CI pipeline

### Overview

`roark` is the official command line interface for the Roark API. It covers the same surface as the [Node.js](/documentation/sdks/node-sdk) and [Python](/documentation/sdks/python-sdk) SDKs (calls, metrics, personas, customer flows, simulations, webhooks) and adds the two commands that make [Config as Code](/documentation/config-as-code/overview) practical: `roark config diff` and `roark config apply`.

It prints JSON, so it composes with `jq` and with everything else in a pipeline, and it uses distinct [exit codes](#exit-codes), so a CI job can tell a rejected request apart from a missing credential.

***

### Install

> [tab] Install script (recommended)
>
> **macOS, Linux, WSL:**
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> curl -fsSL https://roark.ai/install.sh | sh
> ```
>
> This installs into `~/.roark` and links `roark` into `~/.local/bin`. Nothing is written outside your home directory, and no step needs `sudo`.
>
> Pin a version, or remove the CLI entirely:
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> curl -fsSL https://roark.ai/install.sh | sh -s -- --version 0.1.1
> curl -fsSL https://roark.ai/install.sh | sh -s -- --uninstall
> ```
>
> > [!NOTE]
> >
> > Re-run the install command to upgrade. It replaces the installed version in place and prunes the old one.

> [tab] Homebrew
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> brew tap roarkhq/tap
> brew trust roarkhq/tap
> brew install roark
> ```
>
> The `brew trust` step is not optional. Homebrew refuses to load a formula from a third-party tap until it is trusted, and without it `brew install` stops with `Refusing to load formula roarkhq/tap/roark from untrusted tap`. To trust just this formula rather than the whole tap, use `brew trust --formula roarkhq/tap/roark`.
>
> In a `Brewfile`:
>
> ```ruby theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> tap "roarkhq/tap"
> brew "roark"
> ```
>
> > [!NOTE]
> >
> > Homebrew installs do not auto-update. Run `brew upgrade roark` to move to the latest version.

> [tab] npm
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> npm install -g @roarkanalytics/cli
> ```
>
> Or run it without installing anything, which is often what you want in CI:
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> npx -y @roarkanalytics/cli@latest --help
> ```
>
> > [!WARNING]
> >
> > A global npm install writes into npm's configured prefix, which on many systems is root-owned and fails with `EACCES`. If you hit that, use the install script instead of reaching for `sudo`.

Confirm the install worked:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark --version
```

> [!NOTE]
>
> The CLI needs **Node.js 20 or newer**. Every install method above uses your existing Node; none of them bundle a runtime.

If `~/.local/bin` is not on your `PATH`, the install script tells you the line to add. Man pages ship with the CLI. Add `~/.roark/share/man` to `MANPATH` and `man roark` works.

***

### Authenticate

Interactively, `roark auth login` opens your browser to approve access, then stores the credential it mints (mode 0600), with the permissions you approve:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark auth login          # opens the browser to approve, stores the minted credential
roark auth login --paste  # skip the browser: paste or pipe a token instead
roark auth status         # what credential is in effect, where it came from, and whether it still works
roark auth logout         # deletes the stored credential
roark whoami              # asks the API what the credential is and what it can reach
```

`auth status` asks the API rather than reading the file, so a credential revoked in the browser is
reported as rejected here too, and exits `3`:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
$ roark auth status
Authenticated as you@example.com via ~/.config/roark/config.json (roar...2644)
Acting on project 0b2b1f9e-5f1a-4c7e-9c3d-1f2e3a4b5c6d, the default stored on this credential.
Override it with `roark config set project <id>`.
```

What the browser mints depends on your organization: a key bound to the project you pick, or a
credential scoped to you across the organization, which reaches every project you are a member of
and names one per request. Either way `roark whoami` tells you which you are holding (`tokenScope`),
and both are revocable in the dashboard: project keys under **Settings → API keys**, user
credentials under **Settings → Account → Connected apps**. See
[API Keys](/documentation/getting-started/api-keys#two-kinds-of-credential) for the difference.

In **CI there is no browser**, so authenticate with a token instead. Use a **project key** here rather than a user credential: a project key belongs to the project and keeps working when people join and leave, while a user credential is tied to a person and is revoked when their account is deactivated. Generate one from [API Keys](/documentation/getting-started/api-keys) and set the environment variable the CLI and SDKs both read, no `login` step needed:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
export ROARK_API_BEARER_TOKEN="your-api-key"
```

You can also pass `--token <value>` on any command, or pipe a token into login (`echo "$ROARK_API_BEARER_TOKEN" | roark auth login`). See [Using the CLI in CI](#using-the-cli-in-ci) for a full GitHub Actions example.

Settings resolve highest precedence first: a flag, then the environment variable, then a project `.roark.json` found by walking up from the working directory, then the user config file. `roark config path` prints where each of those lives.

> [!WARNING]
>
> A `.roark.json` arrives with a clone rather than being something you wrote, so if a project file sets `baseURL`, the CLI refuses to send a stored or environment credential to it. Read the file, then pass `--allow-project-base-url` (or set `ROARK_ALLOW_PROJECT_BASE_URL`) to opt in, or pass `--token` to send a different credential.

***

### Choosing a project

A project key names its own project and ignores everything in this section. A user credential reaches
every project you are a member of, so each command acts on one:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark project list                     # the projects this credential can reach
roark config set project <project-id>  # remember one for every later command
roark agent list --project <other-id>  # or name one for a single command
```

Resolution is the same precedence as every other setting, with one extra layer at the bottom:

1. `--project <id>` on the command
2. `ROARK_PROJECT_ID` in the environment
3. `project` in a `.roark.json` beside your code
4. `project` in the user config file (`roark config set project`)
5. the project you chose on the consent screen, stored on the credential itself

Step 5 is why a freshly pasted credential works with nothing configured: the API falls back to it
when a request names no project. `roark auth status` prints which of these is in effect.

> [!NOTE]
>
> Unlike `baseURL`, a `project` in a checked-in `.roark.json` is safe and needs no opt-in. It selects
> which of **your own** projects a command acts on, and it cannot send your credential anywhere new.
> That is the distinction the warning above is drawing: one steers permissions you already have, the
> other steers where your secret goes.

If a command returns `403 That project is not available to this credential`, the project is outside
your credential's organization or you are not an active member of it. `roark project list` shows the
ones that are.

***

### Usage

Commands read noun before verb, and the verb is `list`, `get`, `create`, `update` or `delete` unless the operation is genuinely something else:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call list --limit 5
roark call get <call-id>
roark simulation run --data '{"planId":"<plan-id>"}'
```

`roark <command> --help` prints the flags for any command, and `roark --help` lists the command tree.

#### Output

JSON on stdout (indented and coloured for a terminal, compact when piped) so the same command works in both places. Errors go to stderr, so `> out.json` captures only real output.

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark call list --limit 5 | jq '.data[].id'
roark call get <call-id> --format plain
```

#### Request bodies

Flags cover the common case, and nested objects go one level deep with dots. A whole payload can be supplied as JSON, with flags overriding what it contains:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark webhook create --url https://example.com/hook --events CALL_ANALYSIS_COMPLETED
roark customer-flow create --data @flow.json
cat flow.json | roark customer-flow create
```

#### Any endpoint

Endpoints without a generated command are still reachable:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark api get /v1/call --query limit=5
roark api post /v1/webhook --data '{"url":"https://example.com","events":["CALL_ANALYSIS_COMPLETED"]}'
```

#### Shell completion

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
eval "$(roark completion bash)"
eval "$(roark completion zsh)"
roark completion fish | source
```

***

### Config as Code

The CLI is the intended way to run [Config as Code](/documentation/config-as-code/overview). Point it at a directory of YAML resources. It bundles them, resolves any `file://` prompt references, and submits the result:

**1. Preview the changes**

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config diff ./roark
```

Prints the `create`, `update` and `delete` operations that would run. Nothing is written.

**2. Apply**

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply ./roark
```

Shows the same preview, then asks for confirmation before reconciling. Pass `-y` to skip the prompt in CI, and `--no-prune` for an additive-only apply that leaves removed resources alone.

> [!NOTE]
>
> These commands need an API key carrying the **`config:apply`** permission. See [Config as Code](/documentation/config-as-code/overview) for the resource kinds and apply semantics.

***

### Using the CLI in CI

CI runs headless, so skip `auth login` and authenticate with a token in the environment. Store an API key as a secret (`ROARK_API_KEY`) and export it as `ROARK_API_BEARER_TOKEN`; every command picks it up with no interactive step.

The common setup is a two-stage GitHub Actions workflow: **diff on every pull request** so a reviewer sees what would change, and **apply on merge to `main`**.

```yaml GitHub Actions theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
name: roark-config
on:
  pull_request:
    paths: ['roark/**']
  push:
    branches: [main]
    paths: ['roark/**']

jobs:
  config:
    runs-on: ubuntu-latest
    env:
      # An API key with the config:apply permission, stored as a repo secret.
      ROARK_API_BEARER_TOKEN: ${{ secrets.ROARK_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20

      # Preview on PRs: no writes, and the log shows exactly what would change.
      - name: Diff
        if: github.event_name == 'pull_request'
        run: npx @roarkanalytics/cli config diff ./roark

      # Reconcile on merge. -y skips the confirmation prompt.
      - name: Apply
        if: github.ref == 'refs/heads/main'
        run: npx @roarkanalytics/cli config apply ./roark -y
```

`config diff` exits non-zero if the request is rejected (see [exit codes](#exit-codes) below), so a broken bundle fails the PR check rather than slipping through. Use `--no-prune` on apply if you want additive-only syncs that never delete resources removed from the repo.

***

### Exit codes

Distinct codes so a CI job can branch on the failure rather than grepping stderr.

| Code | Meaning |
| :- | :- |
| 0 | Success |
| 1 | The API rejected the request |
| 2 | The command line was wrong |
| 3 | No credential, or the credential was refused |
| 4 | The addressed resource does not exist |
| 5 | The request never completed: connection, timeout, or abort |

***

### Additional Resources

> [card] [NPM Package](https://www.npmjs.com/package/@roarkanalytics/cli)
>
> View package details on npm

> [card] [Homebrew Tap](https://github.com/roarkhq/homebrew-tap)
>
> Formula source and release notes

> [card] [Config as Code](/documentation/config-as-code/overview)
>
> Define agents, personas, flows and metrics as YAML

> [card] [API Reference](/api-reference/introduction)
>
> Explore the full API documentation


# MCP Server
Source: https://docs.roark.ai/documentation/sdks/mcp-server

Connect AI agents and coding assistants to the Roark API using the Model Context Protocol

### Overview

The Roark MCP Server lets AI agents and coding assistants interact with the Roark API through the [Model Context Protocol (MCP)](https://modelcontextprotocol.io). Once connected, your agent can create calls, run evaluations, search documentation, and execute code against the Roark TypeScript SDK, all within your existing workflow.

The MCP server pairs with Roark's [Agent Skills](/documentation/sdks/skills), which teach an agent how to choose the right Roark resources and compose them into a complete testing workflow. The MCP provides the API access; the skills provide the workflow. For Claude Code, the Roark plugin installs both together.

### How It Works

The MCP server exposes two tools to your agent:

* **Documentation Search**: a tool for querying Roark API and SDK documentation directly from your agent.
* **Code Execution**: a tool where the agent writes and executes code against the Roark API in a sandboxed environment. Anything the code returns or prints is sent back to the agent as the tool result.

Using this approach, agents can perform complex API tasks deterministically and repeatably. All operations supported by the Roark REST API and SDKs are available through the MCP server.

### Prerequisites

Before you begin, ensure you have:

* Node.js v20 or higher ([Download Node.js](https://nodejs.org))
* A Roark API Key ([Generate one here](/documentation/getting-started/api-keys))
* An MCP-compatible client (Claude Code, Cursor, VS Code, etc.)

### Installation

Choose the setup method for your client:

> [tab] Claude Code
>
> The recommended setup installs the Roark MCP server and its workflow skills together. Set your API key in the environment that launches Claude Code, then run Claude Code and enter these commands:
>
> ```text theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> export ROARK_API_BEARER_TOKEN="your-api-key"
> /plugin marketplace add roarkhq/mcp-roark-analytics
> /plugin install roark@roark
> ```
>
> Set `ROARK_API_BEARER_TOKEN` before launching Claude Code. If Claude Code is already running, restart it after changing the variable.
>
> If you only want the MCP server without the workflow skills, add it directly from your terminal:
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> claude mcp add roark --env ROARK_API_BEARER_TOKEN="your-api-key" -- npx -y @roarkanalytics/sdk-mcp
> ```

> [tab] Cursor
>
> Add the following to your Cursor MCP configuration. You can find this file via **Cursor Settings > Tools & MCP > New MCP Server**.
>
> ```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> {
>   "mcpServers": {
>     "roark": {
>       "command": "npx",
>       "args": ["-y", "@roarkanalytics/sdk-mcp"],
>       "env": {
>         "ROARK_API_BEARER_TOKEN": "your-api-key"
>       }
>     }
>   }
> }
> ```

> [tab] VS Code
>
> Add the following to your VS Code MCP configuration. Open it via **Command Palette > MCP: Open User Configuration**.
>
> ```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> {
>   "mcpServers": {
>     "roark": {
>       "command": "npx",
>       "args": ["-y", "@roarkanalytics/sdk-mcp"],
>       "env": {
>         "ROARK_API_BEARER_TOKEN": "your-api-key"
>       }
>     }
>   }
> }
> ```

> [tab] Other Clients
>
> You can run the MCP server directly via `npx`:
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> export ROARK_API_BEARER_TOKEN="your-api-key"
> npx -y @roarkanalytics/sdk-mcp@latest
> ```
>
> For any MCP-compatible client that uses a JSON configuration, use:
>
> ```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> {
>   "mcpServers": {
>     "roark": {
>       "command": "npx",
>       "args": ["-y", "@roarkanalytics/sdk-mcp"],
>       "env": {
>         "ROARK_API_BEARER_TOKEN": "your-api-key"
>       }
>     }
>   }
> }
> ```
>
> Consult your client's documentation for where to place this configuration. A partial list of MCP clients is available at [modelcontextprotocol.io](https://modelcontextprotocol.io/clients).

### Choosing a project

The code-execution tool runs the agent's code against the Roark API with the credential you gave the
server, so which project it acts on follows the same rules as any other client.

A **project key** names its own project and there is nothing to choose. A **user credential** reaches
every project you are a member of, and there are two ways it gets pointed at one.

**A pinned connector URL** carries the project in the path. The dashboard's API keys page generates
it for you, and the server forwards that project on every call the agent's code makes:

```text theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
https://mcp.api.roark.ai/mcp/<project-id>
```

**An unpinned connector URL** carries no project, so the agent picks per call. This is the one to use
when the work spans projects:

```text theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
https://mcp.api.roark.ai/mcp
```

On an unpinned connector, code the agent writes names the project through the client:

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import Roark from '@roarkanalytics/sdk';

const client = new Roark();

// Which projects can this credential reach?
const { data } = await client.project.list();
console.log(data.projects.map((p) => `${p.name} ${p.id}`));

// Act on one of them.
const support = client.withOptions({ project: data.projects[0].id });
const calls = await support.call.list({ limit: 5 });
console.log(calls);
```

`withOptions({ project })` works on a pinned connector too, which is how an agent overrides the
pinned project for a single call.

> [!NOTE]
>
> Without a project, calls made by a user credential fail with `400` and a message naming the header
> to send. That is the signal to call `client.project.list()` and pick one, not a transient error to
> retry.

### Pairing with Agent Skills

The MCP server gives your agent the tools to reach the Roark API. It does not tell the agent which testing template fits a goal, which metrics to attach, or how to avoid accidentally placing hundreds of calls. That is what Roark's [Agent Skills](/documentation/sdks/skills) are for, and the two are designed to be used together.

There are 15 skills covering the full workflow: registering an agent, authoring personas and flows, assembling run plans, configuring metrics, reading results, monitoring production traffic, and gating CI. Installing the `roark` plugin in Claude Code brings the MCP server and all of them at once. Other agents install the skills with a single command:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
npx skills add roarkhq/mcp-roark-analytics
```

Once installed, ask your agent for what you want and it selects the right skill:

```text theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
Set up a Roark simulation for my support agent. Register the agent, create a persona and customer flow, attach metrics, and show me the estimated call count before starting.
```

See [Agent Skills](/documentation/sdks/skills) for the full list, the supported agents, and installation details.

> [!WARNING]
>
> Every simulated call is billable. Review the estimated call count before starting a run, especially when selecting multiple flow variants, personas, endpoints, or iterations.

### Example Recipes

Once connected, you can interact with the Roark API conversationally. Here are some examples to get you started:

> [accordion] Investigate a specific call
>
> *"Can you find out why call `d4e5f6a7-1234-5678-9abc-def012345678` failed?"*
>
> The agent will fetch the call details, check its status, and surface any errors or issues from the analysis.

> [accordion] Aggregate call metrics
>
> *"What was the average duration of the last 10 calls we received on Roark?"*
>
> The agent will list recent calls, extract their durations, and compute the average for you.

> [accordion] Run metrics on a call
>
> *"Run a metric collection job on call `a1b2c3d4-5678-9abc-def0-123456789abc` and summarize the results."*
>
> The agent will create a metric collection job, wait for it to complete, and present the metric values.

> [accordion] Create and measure a call in one go
>
> *"Create a call from this recording URL and collect the greeting-quality and task-completion metrics on it: [https://example.com/recording.mp3](https://example.com/recording.mp3)"*
>
> The agent will create the call record, trigger metric collection, and return the results once they're ready.

> [accordion] Review metric results across calls
>
> *"Show me the metric results for my last 5 calls, which ones scored lowest on task-completion?"*
>
> The agent will fetch recent calls, pull their metric values, and rank them by score.

> [accordion] List available metrics
>
> *"What metrics do I have configured in Roark?"*
>
> The agent will retrieve your metric definitions and list them with their descriptions and slugs.

> [accordion] Kick off a simulation
>
> *"Start a simulation run using my 'angry-customer' persona against the 'appointment-booking' customer flow."*
>
> The agent will look up your personas and customer flows, then create a simulation job.

### Additional Resources

> [card] [NPM Package](https://www.npmjs.com/package/@roarkanalytics/sdk-mcp)
>
> View package details on npm

> [card] [GitHub Repository](https://github.com/roarkhq/mcp-roark-analytics)
>
> Browse the source code and open an issue

> [card] [Agent Skills](/documentation/sdks/skills)
>
> Teach your agent the Roark testing workflow

> [card] [Node.js SDK](/documentation/sdks/node-sdk)
>
> Learn about the underlying TypeScript SDK

> [card] [API Reference](/api-reference/introduction)
>
> Explore the full API documentation

> [card] [MCP Specification](https://modelcontextprotocol.io)
>
> Learn more about the Model Context Protocol


# Node.js SDK
Source: https://docs.roark.ai/documentation/sdks/node-sdk

Upload calls, run metrics, and execute simulations using Node.js

> [!NOTE]
>
> The Node.js SDK is continually evolving, with new endpoints being added regularly. Stay tuned for updates!

### Overview

The Roark Node.js SDK provides a streamlined way to interact with the Roark API. This guide covers the two most common workflows: uploading calls and running metrics, and executing simulation run plans.

### Prerequisites

Before you begin, ensure you have:

* Node.js v12 or higher - [Download Node.js](https://nodejs.org)
* A Roark API Key - [Generate one here](/documentation/getting-started/api-keys)

### Installation

Choose your preferred package manager:

```bash npm theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
npm install @roarkanalytics/sdk
```

```bash yarn theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
yarn add @roarkanalytics/sdk
```

```bash pnpm theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
pnpm add @roarkanalytics/sdk
```

```bash bun theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
bun add @roarkanalytics/sdk
```

### Initialize the Client

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import Roark from '@roarkanalytics/sdk'

const client = new Roark({
  bearerToken: process.env.ROARK_API_BEARER_TOKEN,
})
```

If you are using a **user credential** rather than a project key, name the project it acts on. It is
sent as `X-Roark-Project-Id` on every request, and defaults to `ROARK_PROJECT_ID` from the
environment. A project key names its own project and ignores this:

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const client = new Roark({
  bearerToken: process.env.ROARK_API_BEARER_TOKEN,
  project: process.env.ROARK_PROJECT_ID,
})

// Which projects can this credential reach?
const { data } = await client.project.list()

// Act on a different one for a single call, or a group of them.
const other = client.withOptions({ project: data.projects[1].id })
```

Requires `@roarkanalytics/sdk` 4.15.0 or later. See
[API Keys](/documentation/getting-started/api-keys#choosing-the-project-a-request-acts-on) for when
this is needed.

> [!NOTE]
>
> Set `ROARK_API_BEARER_TOKEN` in your environment, or replace with your actual API key from Roark.

***

### Upload a Call and Run Metrics

Upload a call recording, then run metric definitions against it using a collection job.

**1. Upload a Call**

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const call = await client.call.create({
  recordingUrl: 'https://example.com/recording.mp3',
  startedAt: '2024-01-15T10:00:00Z',
  interfaceType: 'PHONE',
  callDirection: 'INBOUND',
  agent: {
    name: 'Support Agent',
    customId: 'agent-123',
  },
  customer: {
    phoneNumberE164: '+15551234567',
  },
  // Optional: custom properties for filtering
  properties: {
    department: 'sales',
    campaignId: 'summer-2024',
  },
})
```

**2. Get Metric Definitions**

List available metric definitions to choose which ones to run:

> [!NOTE]
>
> Metric definitions are cursor-paginated. `listDefinitions()` returns the
> first page, up to 500 definitions. To enumerate further pages, pass the
> preceding response's `pagination.nextCursor` as `after` to
> `GET /v1/metric/definitions`; treat the cursor as opaque.

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const definitions = await client.metric.listDefinitions()

// Find the metrics you want to collect
const taskCompletion = definitions.data.find(
  (m) => m.metricId === 'task_completion'
)
const sentiment = definitions.data.find(
  (m) => m.metricId === 'sentiment_score'
)
```

**3. Create a Metric Collection Job**

Run the selected metrics against your uploaded call:

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const job = await client.metricCollectionJob.create({
  callIds: [call.data.id],
  metrics: [
    { id: taskCompletion.id },
    { id: sentiment.id },
  ],
})
```

**4. Poll for Completion**

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
let status = 'PENDING'
while (status !== 'COMPLETED' && status !== 'FAILED') {
  const result = await client.metricCollectionJob.getByID(job.data.id)
  status = result.data.status
  console.log(`Progress: ${result.data.completedItems}/${result.data.totalItems}`)
  if (status !== 'COMPLETED' && status !== 'FAILED') {
    await new Promise((r) => setTimeout(r, 2000))
  }
}
```

**5. Fetch Results**

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const metrics = await client.call.listMetrics(call.data.id)
console.log(metrics.data)
```

> [!TIP]
>
> If you have [metric collectors](/documentation/metrics/metric-collectors) configured, metrics are collected automatically when calls are uploaded, no collection job needed.

***

### Run a Simulation

Run a simulation against your agent with an inline plan, then monitor it.

**1. Run a Simulation**

Reference an [agent endpoint](/documentation/simulation-testing/run-plans), a [customer flow](/documentation/simulation-testing/customer-flows), and at least one metric. Add `saveAsPlan: true` (with a `name`) to keep it as a reusable plan.

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const run = await client.simulation.run({
  plan: {
    direction: 'INBOUND',
    maxSimulationDurationSeconds: 300,
    agentEndpoints: [{ id: 'agent-endpoint-uuid' }],
    flows: [{ id: 'customer-flow-uuid', happyPath: true }],
    metrics: [{ slug: 'task_completion' }],
  },
})

const jobId = run.data.simulationRunPlanJobId
```

**2. Or re-run a saved plan**

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const job = await client.simulation.run({ planId: 'plan-uuid' })
```

**3. Monitor Progress**

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
const result = await client.simulationRunPlanJob.getByID(jobId)

console.log(`Status: ${result.data.status}`)
console.log(`Jobs: ${result.data.simulationJobs.length}`)
```

**4. View Results**

Results including metric evaluations are available in the Roark dashboard, or you can fetch metrics for individual simulation calls via the API:

```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
// Get metrics for a specific simulation call
const metrics = await client.call.listMetrics('simulation-call-id')
```

***

### Additional Examples

> [accordion] Create a call with tool invocations
>
> Track function calls and tool usage during the conversation:
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> const call = await client.call.create({
>   recordingUrl: 'https://example.com/recording.mp3',
>   startedAt: '2024-01-15T10:00:00Z',
>   interfaceType: 'PHONE',
>   callDirection: 'INBOUND',
>   agent: {
>     name: 'Booking Agent',
>     customId: 'booking-agent-1',
>   },
>   customer: {
>     phoneNumberE164: '+15551234567',
>   },
>   toolInvocations: [
>     {
>       name: 'checkAvailability',
>       description: 'Check available appointment slots',
>       startOffsetMs: 5000,
>       endOffsetMs: 5500,
>       parameters: {
>         date: '2024-01-20',
>         serviceType: 'consultation',
>       },
>       result: { slots: ['9:00 AM', '2:00 PM', '4:00 PM'] },
>       agent: { customId: 'booking-agent-1' },
>     },
>     {
>       name: 'bookAppointment',
>       description: 'Book an appointment for the customer',
>       startOffsetMs: 15000,
>       endOffsetMs: 15800,
>       parameters: {
>         date: '2024-01-20',
>         time: '2:00 PM',
>         customerName: 'John Doe',
>       },
>       result: 'Appointment confirmed',
>       agent: { customId: 'booking-agent-1' },
>     },
>   ],
> })
> ```

> [accordion] Create a call with an existing agent
>
> Reference an agent by its Roark ID or custom ID:
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> // By Roark ID
> const call = await client.call.create({
>   recordingUrl: 'https://example.com/recording.mp3',
>   startedAt: '2024-01-15T10:00:00Z',
>   interfaceType: 'WEB',
>   callDirection: 'OUTBOUND',
>   agent: {
>     roarkId: '550e8400-e29b-41d4-a716-446655440000',
>   },
>   customer: {
>     phoneNumberE164: '+15551234567',
>   },
> })
>
> // By custom ID
> const call2 = await client.call.create({
>   recordingUrl: 'https://example.com/recording.mp3',
>   startedAt: '2024-01-15T10:00:00Z',
>   interfaceType: 'PHONE',
>   callDirection: 'INBOUND',
>   agent: {
>     customId: 'my-agent-id',
>   },
>   customer: {
>     phoneNumberE164: '+15551234567',
>   },
> })
> ```

> [accordion] Create a metric collection job for multiple calls
>
> Run metrics across a batch of existing calls:
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> const job = await client.metricCollectionJob.create({
>   callIds: [
>     'call-uuid-1',
>     'call-uuid-2',
>     'call-uuid-3',
>   ],
>   metrics: [
>     { id: 'metric-definition-uuid-1' },
>     { id: 'metric-definition-uuid-2' },
>   ],
> })
>
> // totalItems = callIds.length * metrics.length
> console.log(`Processing ${job.data.totalItems} items`)
> ```

### Additional Resources

> [card] [API Reference](/api-reference/introduction)
>
> Explore our comprehensive API documentation

> [card] [Example Repository](https://github.com/roarkhq/sdk-roark-analytics-node/tree/main/examples)
>
> View example implementations and use cases

> [card] [NPM Package](https://www.npmjs.com/package/@roarkanalytics/sdk)
>
> View package details and stats on npm

> [card] [GitHub Repository](https://github.com/roarkhq/sdk-roark-analytics-node)
>
> Browse the source code and contribute


# Python SDK
Source: https://docs.roark.ai/documentation/sdks/python-sdk

Upload calls, run metrics, and execute simulations using Python

> [!NOTE]
>
> The Python SDK is continually evolving, with new endpoints being added regularly. Stay tuned for updates!

### Overview

The Roark Python SDK provides a streamlined way to interact with the Roark API. This guide covers the two most common workflows: uploading calls and running metrics, and executing simulation run plans.

### Prerequisites

Before you begin, ensure you have:

* Python 3.7 or higher - [Download Python](https://www.python.org/downloads/)
* A Roark API Key - [Generate one here](/documentation/getting-started/api-keys)

### Installation

Choose your preferred package manager:

```bash pip theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
pip install roark-analytics
```

```bash pipenv theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
pipenv install roark-analytics
```

```bash poetry theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
poetry add roark-analytics
```

### Initialize the Client

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import os
from roark_analytics import Roark

client = Roark(
    bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"),
)
```

If you are using a **user credential** rather than a project key, name the project it acts on. It is
sent as `X-Roark-Project-Id` on every request, and defaults to `ROARK_PROJECT_ID` from the
environment. A project key names its own project and ignores this:

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
client = Roark(
    bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"),
    project=os.environ.get("ROARK_PROJECT_ID"),
)

# Which projects can this credential reach?
projects = client.project.list()

# Act on a different one for a single call, or a group of them.
other = client.with_options(project=projects.data.projects[1].id)
```

Requires `roark-analytics` 5.15.0 or later. See
[API Keys](/documentation/getting-started/api-keys#choosing-the-project-a-request-acts-on) for when
this is needed.

> [!NOTE]
>
> Set `ROARK_API_BEARER_TOKEN` in your environment, or replace with your actual API key from Roark.

***

### Upload a Call and Run Metrics

Upload a call recording, then run metric definitions against it using a collection job.

**1. Upload a Call**

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
call = client.call.create(
    recording_url="https://example.com/recording.mp3",
    started_at="2024-01-15T10:00:00Z",
    interface_type="PHONE",
    call_direction="INBOUND",
    agent={
        "name": "Support Agent",
        "customId": "agent-123",
    },
    customer={
        "phoneNumberE164": "+15551234567",
    },
    # Optional: custom properties for filtering
    properties={
        "department": "sales",
        "campaignId": "summer-2024",
    },
)
```

**2. Get Metric Definitions**

List available metric definitions to choose which ones to run:

> [!NOTE]
>
> Metric definitions are cursor-paginated. `list_definitions()` returns the
> first page, up to 500 definitions. To enumerate further pages, pass the
> preceding response's `pagination.nextCursor` as `after` to
> `GET /v1/metric/definitions`; treat the cursor as opaque.

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
definitions = client.metric.list_definitions()

# Find the metrics you want to collect
task_completion = next(
    m for m in definitions.data if m.metric_id == "task_completion"
)
sentiment = next(
    m for m in definitions.data if m.metric_id == "sentiment_score"
)
```

**3. Create a Metric Collection Job**

Run the selected metrics against your uploaded call:

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
job = client.metric_collection_job.create(
    call_ids=[call.data.id],
    metrics=[
        {"id": task_completion.id},
        {"id": sentiment.id},
    ],
)
```

**4. Poll for Completion**

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import time

status = "PENDING"
while status not in ("COMPLETED", "FAILED"):
    result = client.metric_collection_job.get_by_id(job.data.id)
    status = result.data.status
    print(f"Progress: {result.data.completed_items}/{result.data.total_items}")
    if status not in ("COMPLETED", "FAILED"):
        time.sleep(2)
```

**5. Fetch Results**

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
metrics = client.call.list_metrics(call.data.id)
print(metrics.data)
```

> [!TIP]
>
> If you have [metric collectors](/documentation/metrics/metric-collectors) configured, metrics are collected automatically when calls are uploaded, no collection job needed.

***

### Run a Simulation

Run a simulation against your agent with an inline plan, then monitor it.

**1. Run a Simulation**

Reference an [agent endpoint](/documentation/simulation-testing/run-plans), a [customer flow](/documentation/simulation-testing/customer-flows), and at least one metric. Add `save_as_plan=True` (with a `name`) to keep it as a reusable plan.

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
run = client.simulation.run(
    plan={
        "direction": "INBOUND",
        "maxSimulationDurationSeconds": 300,
        "agentEndpoints": [{"id": "agent-endpoint-uuid"}],
        "flows": [{"id": "customer-flow-uuid", "happyPath": True}],
        "metrics": [{"slug": "task_completion"}],
    },
)

job_id = run.data.simulation_run_plan_job_id
```

**2. Or re-run a saved plan**

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
job = client.simulation.run(plan_id="plan-uuid")
```

**3. Monitor Progress**

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
result = client.simulation_run_plan_job.get_by_id(job_id)

print(f"Status: {result.data.status}")
print(f"Jobs: {len(result.data.simulation_jobs)}")
```

**4. View Results**

Results including metric evaluations are available in the Roark dashboard, or you can fetch metrics for individual simulation calls via the API:

```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# Get metrics for a specific simulation call
metrics = client.call.list_metrics("simulation-call-id")
```

***

### Additional Examples

> [accordion] Create a call with tool invocations
>
> Track function calls and tool usage during the conversation:
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> call = client.call.create(
>     recording_url="https://example.com/recording.mp3",
>     started_at="2024-01-15T10:00:00Z",
>     interface_type="PHONE",
>     call_direction="INBOUND",
>     agent={
>         "name": "Booking Agent",
>         "customId": "booking-agent-1",
>     },
>     customer={
>         "phoneNumberE164": "+15551234567",
>     },
>     tool_invocations=[
>         {
>             "name": "check_availability",
>             "description": "Check available appointment slots",
>             "startOffsetMs": 5000,
>             "endOffsetMs": 5500,
>             "parameters": {
>                 "date": "2024-01-20",
>                 "serviceType": "consultation",
>             },
>             "result": {"slots": ["9:00 AM", "2:00 PM", "4:00 PM"]},
>             "agent": {"customId": "booking-agent-1"},
>         },
>         {
>             "name": "book_appointment",
>             "description": "Book an appointment for the customer",
>             "startOffsetMs": 15000,
>             "endOffsetMs": 15800,
>             "parameters": {
>                 "date": "2024-01-20",
>                 "time": "2:00 PM",
>                 "customerName": "John Doe",
>             },
>             "result": "Appointment confirmed",
>             "agent": {"customId": "booking-agent-1"},
>         },
>     ],
> )
> ```

> [accordion] Create a call with an existing agent
>
> Reference an agent by its Roark ID or custom ID:
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> # By Roark ID
> call = client.call.create(
>     recording_url="https://example.com/recording.mp3",
>     started_at="2024-01-15T10:00:00Z",
>     interface_type="WEB",
>     call_direction="OUTBOUND",
>     agent={
>         "roarkId": "550e8400-e29b-41d4-a716-446655440000",
>     },
>     customer={
>         "phoneNumberE164": "+15551234567",
>     },
> )
>
> # By custom ID
> call2 = client.call.create(
>     recording_url="https://example.com/recording.mp3",
>     started_at="2024-01-15T10:00:00Z",
>     interface_type="PHONE",
>     call_direction="INBOUND",
>     agent={
>         "customId": "my-agent-id",
>     },
>     customer={
>         "phoneNumberE164": "+15551234567",
>     },
> )
> ```

> [accordion] Create a metric collection job for multiple calls
>
> Run metrics across a batch of existing calls:
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> job = client.metric_collection_job.create(
>     call_ids=[
>         "call-uuid-1",
>         "call-uuid-2",
>         "call-uuid-3",
>     ],
>     metrics=[
>         {"id": "metric-definition-uuid-1"},
>         {"id": "metric-definition-uuid-2"},
>     ],
> )
>
> # total_items = len(call_ids) * len(metrics)
> print(f"Processing {job.data.total_items} items")
> ```

> [accordion] Async usage
>
> For asynchronous operations, use the async client:
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> import os
> import asyncio
> from roark_analytics import AsyncRoark
>
> client = AsyncRoark(
>     bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"),
> )
>
> async def main():
>     call = await client.call.create(
>         recording_url="https://example.com/recording.mp3",
>         started_at="2024-01-15T10:00:00Z",
>         interface_type="PHONE",
>         call_direction="INBOUND",
>         agent={"name": "Support Agent"},
>         customer={"phoneNumberE164": "+15551234567"},
>     )
>
>     job = await client.metric_collection_job.create(
>         call_ids=[call.data.id],
>         metrics=[{"id": "metric-definition-uuid"}],
>     )
>
> asyncio.run(main())
> ```

### Additional Resources

> [card] [API Reference](/api-reference/introduction)
>
> Explore our comprehensive API documentation

> [card] [Example Repository](https://github.com/roarkhq/sdk-roark-analytics-python/tree/main/examples)
>
> View example implementations and use cases

> [card] [PyPI Package](https://pypi.org/project/roark-analytics/)
>
> View package details and stats on PyPI

> [card] [GitHub Repository](https://github.com/roarkhq/sdk-roark-analytics-python)
>
> Browse the source code and contribute


# Agent Skills
Source: https://docs.roark.ai/documentation/sdks/skills

Teach any coding agent to test voice and chat AI agents with Roark

### Overview

Roark publishes 15 **Agent Skills**: short, focused guides that teach a coding agent how to use Roark. The [MCP Server](/documentation/sdks/mcp-server) gives an agent the tools to reach the Roark API; the skills give it the workflow and the judgment. Which testing template fits the goal, which metrics to attach, how to select flow variants without accidentally placing hundreds of calls.

They use the open Agent Skills format (a `SKILL.md` file plus an optional `references/` folder), so they are not tied to one vendor. Claude Code, Codex, Cursor, Gemini CLI and GitHub Copilot all read them.

> [!NOTE]
>
> Skills and the MCP server are independent. Skills work on their own and make an agent better at reasoning about Roark, but an agent still needs the [MCP server](/documentation/sdks/mcp-server), the [CLI](/documentation/sdks/cli) or an [SDK](/documentation/sdks/node-sdk) to actually call the API.

***

### Install

> [tab] Any agent (recommended)
>
> The `skills` CLI installs straight from the Roark repository:
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> npx skills add roarkhq/mcp-roark-analytics
> ```
>
> This installs all 15 skills into the current project. The files land in `.agents/skills/`, and the CLI additionally links them into the directory of whichever agent it detects, so seeing both `.agents/skills/` and (for example) `.claude/skills/` afterwards is expected rather than a duplicate install.
>
> To choose targets explicitly, repeat the `-a` flag once per agent:
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> npx skills add roarkhq/mcp-roark-analytics -a codex -a cursor -a gemini-cli -a github-copilot
> ```
>
> > [!WARNING]
> >
> > A comma-separated list such as `-a codex,cursor` is read as one agent name and rejected. Repeat the flag instead.
>
> Add `-g` to install for your user account rather than the current project, and `-a '*'` to install for every agent the CLI knows.

> [tab] Claude Code
>
> In Claude Code, install the skills and the MCP server together as one plugin:
>
> ```text theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> /plugin marketplace add roarkhq/mcp-roark-analytics
> /plugin install roark@roark
> ```
>
> Set your API key in the environment that launches Claude Code:
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> export ROARK_API_BEARER_TOKEN="your-api-key"
> ```
>
> The plugin is the simplest route for Claude Code because it brings the API tools along with the skills. The `npx skills add` command also works and installs the skills only.

> [tab] Manual
>
> Every skill is a directory of plain Markdown, so you can copy them yourself from the [plugin repository](https://github.com/roarkhq/mcp-roark-analytics/tree/main/plugins/roark/skills):
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> git clone https://github.com/roarkhq/mcp-roark-analytics
> mkdir -p .agents/skills
> cp -r mcp-roark-analytics/plugins/roark/skills/* .agents/skills/
> ```
>
> Copy the whole directory. The skills reference each other by relative path, so installing a subset leaves broken links.

Confirm what landed:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
npx skills list
```

***

### Supported agents

The `skills` CLI writes one shared tree and every agent below reads it, so there is nothing to convert per vendor.

| Agent | Install location | Notes |
| :- | :- | :- |
| Claude Code | `.claude/skills/` | Linked to `.agents/skills/`, not a second copy. Or install the plugin to get the MCP server too |
| Codex | `.agents/skills/` | Universal layout |
| Cursor | `.agents/skills/` | Also reads `.claude/skills/` and `.codex/skills/` |
| Gemini CLI | `.agents/skills/` | Universal layout |
| GitHub Copilot | `.agents/skills/` | Universal layout |

Windsurf, Zed, Cline, Goose, Warp, opencode and others are supported by the same command. Run `npx skills add roarkhq/mcp-roark-analytics -a '*'` to install for every agent the CLI detects.

> [!WARNING]
>
> **Install the skills as a set.** The skills cross-reference each other by relative path, and most of them point into `roark-concepts` for shared definitions. Any installer that packages skills one at a time breaks those links. That rules out Claude Desktop, whose **Customize → Skills** screen accepts one zip per skill and has no notion of sibling skills. Use Claude Code or one of the agents above instead.

***

### The skills

| Skill | Use it to |
| :- | :- |
| `roark-overview` | Understand the Roark object model and choose the right workflow. Start here |
| `roark-concepts` | Look up what each domain object *is*. The shared definitions the others reference |
| `register-agent` | Register an agent and the endpoint Roark should call |
| `author-personas-flows` | Define personas and improv customer flows |
| `author-scripted-flows` | Build exact IVR and DTMF conversation graphs |
| `configure-outbound-dial` | Configure the HTTP request used for outbound calls |
| `build-run-plan` | Assemble and start a simulation run |
| `manage-run-plans` | Find, edit, rerun, or delete saved run plans |
| `configure-metrics` | Select built-in metrics, checks, or custom metrics |
| `read-results` | Read run status, scores, transcripts, and pass/fail results |
| `monitor-live-calls` | Set up metric policies and backfill jobs for production traffic |
| `ingest-calls` | Import recordings and inspect call analysis and sentiment |
| `subscribe-webhooks` | Receive events instead of polling long-running jobs |
| `manage-config-as-code` | Reconcile agents, flows, personas, metrics, and collectors declaratively |
| `gate-ci` | Start a run and gate a CI or deployment pipeline on its results |

Agents load skills on demand, so having all 15 installed costs nothing until one is relevant to what you asked for.

#### Concepts versus skills

`roark-concepts` is generated rather than hand-written. It is synced from Roark's own source repository, and the same corpus composes the in-product assistant's prompt, so your coding agent and the assistant in the Roark dashboard cannot describe the product differently.

The division of labour is deliberate. A **concept** says what a thing is and which choice is right. A **skill** says which call to make and what the API will reject, including the places where the public vocabulary differs from the internal one. A flow's `mode: 'UNSCRIPTED'` is `type: 'IMPROV'` over the API, for example.

***

### Using them

Once installed, the agent picks the right skill from what you ask. You do not name skills explicitly:

```text theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
Set up a Roark simulation for my support agent. Register the agent, create a persona
and customer flow, attach metrics, and show me the estimated call count before starting.
```

> [accordion] Build a regression suite
>
> *"Create a saved run plan covering our three billing flows with the frustrated-caller persona, attach task-completion and empathy metrics, and add a pass/fail check at 0.8."*
>
> The agent uses `build-run-plan` and `configure-metrics` to assemble the plan, then reports the call count before it starts anything.

> [accordion] Gate a deploy on a simulation
>
> *"Add a GitHub Actions job that runs our nightly Roark plan and fails the build if any check fails."*
>
> The agent uses `gate-ci`, which composes start, wait and assert, and warns you if the plan has no checks attached, since that would pass forever.

> [accordion] Author an IVR test
>
> *"Our new phone tree asks for an account number, then routes to billing or support. Build a scripted flow that presses 1 and enters an account number."*
>
> The agent uses `author-scripted-flows` for the conversation graph and DTMF entry, rather than an improv flow.

> [accordion] Grade production traffic
>
> *"Score every real call our support agent takes this week for task completion, and alert me when it drops below 0.7."*
>
> The agent uses `monitor-live-calls` to set up a standing metric collector over live traffic instead of a simulation.

> [!WARNING]
>
> Every simulated call is billable. The skills are written to create a run plan, report the estimated call count, and wait for you to confirm before starting a run. Review that number, especially when a plan multiplies flows, personas, endpoints and iterations.

***

### Keeping them current

The skills are versioned with the SDK surface they describe. Update them with:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
npx skills update
```

In Claude Code, the plugin updates with the marketplace. `skills-lock.json` records a content hash for each installed skill, so `skills update` can tell which ones actually changed and leave the rest alone. Commit that file to keep the set consistent across a team.

***

### Additional Resources

> [card] [MCP Server](/documentation/sdks/mcp-server)
>
> Give your agent access to the Roark API

> [card] [GitHub Repository](https://github.com/roarkhq/mcp-roark-analytics/tree/main/plugins/roark/skills)
>
> Read the skills source

> [card] [CLI](/documentation/sdks/cli)
>
> Drive Roark from your terminal and CI

> [card] [Config as Code](/documentation/config-as-code/overview)
>
> Define agents, personas, flows and metrics as YAML

> [card] [Node.js SDK](/documentation/sdks/node-sdk)
>
> The TypeScript SDK the skills write against

> [card] [API Reference](/api-reference/introduction)
>
> Explore the full API documentation


# Best Practices
Source: https://docs.roark.ai/documentation/simulation-testing/best-practices

Guidelines for getting the most out of Roark simulations

## Customer flows

### Improv vs. Scripted

The single biggest decision in a [customer flow](/documentation/simulation-testing/customer-flows) is the authoring mode, and it maps directly to what you're testing.

**Improv**: You write a free-text brief describing who the customer is, what they want, and how they behave. The simulator improvises a fresh conversation from that brief on every run, different words, same intent:

```
A first-time caller wants to book an appointment for tomorrow around 2pm.
They're friendly but a little unsure of the process and will ask what
information they need to provide.
```

Improv is the default choice for most testing. Because the wording changes each run, it exercises your agent's ability to handle natural language variation for the same intent, the thing that actually breaks in production.

**Scripted**: You author the conversation step by step on a graph canvas, writing the exact customer lines:

```
"Hello, can I make a booking for tomorrow at 2pm?"
```

Scripted mode is for precise control: verifying your agent parses a specific date format, detects a keyword, handles a DTMF sequence, or recovers from a deliberate silence.

> [!TIP]
>
> Use **Scripted** when the exact utterance matters: phrasing, keyword detection, slot filling, keypad input. Use **Improv** when you want varied, realistic conversations that test the same intent from a different angle every run.

### Write improv briefs like backstories

The customer setup brief is where the simulated customer comes to life. Behavioral color (emotional state, context, quirks) belongs in this prose, not in the persona (personas carry voice and speech characteristics; the brief carries the situation).

Good briefs give the simulator context that drives nuanced, realistic behavior:

**Bereaved customer**

```
James recently lost his wife and is calling to cancel her phone line on
their shared plan. He is soft-spoken and may become emotional. He doesn't
fully understand the account details and may need things explained gently.
```

**Skeptical professional**

```
Priya is a software engineer who immediately suspects she's talking to an
AI. She will ask pointed questions like "Are you a real person?" and "Can
you transfer me to a human?" She becomes frustrated if the agent can't
directly answer her questions about a billing discrepancy.
```

**Impatient multitasker**

```
Carlos is calling during a short break at work. He has 5 minutes. He'll
give short answers, may mishear things, and will ask the agent to repeat
or speak up. He needs to reschedule a delivery that requires a signature.
```

![The improv flow editor: a filled Happy path customer-setup brief with Persona, Environment, and Variables chips, agent expectations, and edge cases](https://mintcdn.com/roark/Q1LycOQn64NfkIxp/images/simulation-testing/improv-editor.png?fit=max\&auto=format\&n=Q1LycOQn64NfkIxp\&q=85\&s=22fc23a59a3947be3fb3ef8d2997248d)

### Structure scripted flows around paths

Scripted flows are **graph-based**: a DAG of agent and customer steps. Every unique path from start to leaf becomes its own variant automatically: the path engine derives one variant per path, so structuring the graph well is how you get coverage without duplication.

#### Start with a happy path

Begin with a single expected route, the happy path where everything goes as planned:

```mermaid theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
graph LR
    C1["👤 Customer calls in"] --> A1["🤖 Greets and\n asks how to help"]
    A1 --> C2["👤 Asks to book\n an appointment"]
    C2 --> A2["🤖 Asks for\n preferred date"]
    A2 --> C3["👤 Provides\n a date"]
    C3 --> A3["🤖 Confirms\n and books"]
```

#### Add branches for edge cases

Once your happy path works, branch at points where the conversation can diverge. Branches inherit everything above them, and each new path shows up as an edge case in the variant rail:

```mermaid theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
graph TD
    C1["👤 Customer calls in"] --> A1["🤖 Greets and asks how to help"]
    A1 --> C2["👤 Asks to book an appointment"]
    C2 --> A2["🤖 Asks for preferred date"]
    A2 --> C3["👤 Provides a date"]
    A2 --> C3b["👤 Asks what's available"]
    C3 --> A3["🤖 Confirms and books"]
    C3b --> A3b["🤖 Lists available slots"]
    A3b --> C4["👤 Picks a slot"]
    C4 --> A4["🤖 Confirms and books"]
```

Focus your branching where:

* The agent asks the customer a question (customers respond in unexpected ways)
* The agent could go on a tangent or lose track of the conversation
* Tool calls or lookups might fail or return unexpected results

If you want the *same* path exercised by a different caller, don't duplicate the branch. Use **Add variant for path** to run that path again with a different persona or environment.

> [!NOTE]
>
> Structural graph edits (adding or removing steps and edges) pause variant editing until you save: the flow needs to refresh its paths before you can add or remove variants.

![The scripted flow editor: a branching conversation graph with the happy path highlighted, and a Variants rail listing the happy path and edge cases](https://mintcdn.com/roark/IjseSxRBSMbit4Aq/images/simulation-testing/scripted-canvas.png?fit=max\&auto=format\&n=IjseSxRBSMbit4Aq\&q=85\&s=c047ffea266e13200c24518be7ae5b02)

#### Compose flows instead of duplicating them

Two composition tools keep shared sequences in one place:

* **Flow link** (scripted step): splices another scripted flow's steps in at that point. Keep a shared IVR tree or authentication sequence in its own flow and link to it; when the menu changes, you update it once.
* **Preceded by** (improv variant): runs a scripted flow *before* the improv segment. Use it to deterministically navigate an IVR menu, then hand off to an improvised conversation. Improv has no fixed ending, so it can only come last. You can't link a scripted flow after it.

### Templating with variables

Variants carry key/value **variables** that are handed to the customer-side model, and text fields support `{{variableName}}` references:

```
"Hi, my name is {{patientName}} and I need to reschedule
my {{appointmentType}} appointment"
```

Persona-scoped properties use the `{{persona.*}}` prefix and resolve from the variant's persona at runtime:

```
"Hi, my name is {{persona.name}}"
```

This keeps a single flow reusable across many test cases, with only the key details changing between runs. You can also use **Apply a test profile** in the variant's variables editor to fill the entries from a saved profile in one click. See the [Variables guide](/documentation/simulation-testing/variables) for the full lifecycle.

### Generate flows with Ask Roark

You don't have to author flows from scratch. The **New flow** page (`/customer-flows/new`) offers three Ask Roark starting points under "Generate":

1. **From your calls**: pick real production calls and let Ask Roark draft flows from them. This is the fastest route to representative coverage: the drafts reflect how customers actually talk to your agent.
2. **From a transcript**: upload or paste a transcript from any source (another platform, a QA review, a bug report) and generate a flow from it.
3. **Describe what you want**: describe the situation in plain language and Ask Roark drafts the flow for you.

Whichever card you start from, treat the draft as a baseline: review the generated brief or steps, then extend it with edge cases covering paths that didn't occur in the source material but could happen in production. Generated variants carry a **Generated** badge until you review them.

![The New customer flow chooser: three Generate with Ask Roark cards (Describe what you want, From a transcript, From your calls) above two blank starting options, Improv with a blue feather icon and Scripted with a green network icon](https://mintcdn.com/roark/yUzvMIE3_BYBHPDx/images/simulation-testing/flow-chooser.png?fit=max\&auto=format\&n=yUzvMIE3_BYBHPDx\&q=85\&s=2c930d427972598242150ce9807308e4)

***

## Personas

[Personas](/documentation/simulation-testing/personas) model *who is calling*: the voice and speech profile pinned to each flow variant. A good persona strategy tests your agent across a range of realistic caller profiles.

### Diversify voice and speech

Build a set of personas that vary:

* **Language and accent**: battle-test your transcriber's accuracy across accents (US, British, Indian, Spanish, and more) and languages
* **Speech pace and response timing**: slow, rambling talkers and quick, clipped ones ensure your agent neither interrupts customers nor times out waiting for them
* **Clarity and disfluencies**: vague or rambling callers with natural ums and false starts stress your agent's understanding far more than clean studio speech

> [!NOTE]
>
> Background noise is no longer a persona property. It's the variant's **Environment** chip in the flow editor (Office, Coffee shop, Driving, Airport, and more). Vary it per variant to verify your endpointing holds up in non-ideal audio.

### Test difficult customer types

Combine persona fields (base emotion, clarity) with a matching customer setup brief to build challenging callers:

> [card] AI Skeptic
>
> Base emotion **Skeptical**, plus a brief where the customer tests the agent with trick questions and asks for a human

> [card] Hostile Caller
>
> Base emotion **Frustrated**, plus a brief with a rude, escalating customer: verify your agent stays polite and professional

> [card] Sensitive Situation
>
> A brief describing a customer in a difficult moment (bereavement, financial hardship): ensure your agent is empathetic and considerate

> [card] Rapid Switcher
>
> Base emotion **Distracted**, clarity **Rambling**, plus a brief where the customer changes topics frequently

The persona sets the delivery; the variant's customer setup prose sets the situation and behavior. Keeping them separate means one "Frustrated rambler" persona can stress-test every flow in your library.

***

## Plan configuration

A plan composes agent endpoints, attached flows (with a variant selection per attachment), metrics with Pass/Fail checks, and run settings. How you configure it depends on what you're testing.

### Common patterns

> [accordion] Load testing
>
> Use the **Load testing** template rather than configuring this by hand. Its **Volume** panel exposes exactly the two knobs you need (**Concurrent calls** and **Total iterations**) plus the flow to test.
>
> * Pick a flow whose happy path matches your target call duration
> * Set total iterations to your target volume and concurrent calls to how many should hit the agent simultaneously
>
> This reveals how your agent performs under peak load without over-provisioning your regular test plans.

> [accordion] Adaptability across personas
>
> Test how your agent handles different voices, accents, and speech styles on the same conversation.
>
> * Attach a **single flow** with the **default variant** selection (your happy path)
> * Attach it multiple times with a different **persona override** per attachment, covering a wide spread of accents, paces, and base emotions
>
> This isolates persona-driven variation from flow complexity, making it easy to spot which caller profiles cause problems. For language coverage specifically, the **Multilingual** template does the fan-out for you: each attached flow runs once per selected language.

> [accordion] Instruction following
>
> Agents are non-deterministic. Verify they don't go off-script or hit loopholes.
>
> * Build flows with **multiple edge cases**: scripted branches for the paths you can enumerate, improv edge cases for the ones you can't
> * Write **agent expectations** for the behaviors that must hold on every variant, and additional expectations per edge case
> * Focus edge cases on points where the agent might go on a tangent or fail to recover
>
> The **Flow adherence** template ships with metrics and checks tuned for exactly this.

> [accordion] Red teaming
>
> Use the **Red teaming** template to test resilience against adversarial inputs. It sources flows carrying the **Adversarial** label from your library, or generates 3–10 adversarial edge cases for you at easy, medium, or hard difficulty, covering prompt injection attempts, PII extraction, and social engineering.
>
> Label your own hand-authored adversarial flows with the Adversarial system label so the template picks them up automatically.

See [Templates](/documentation/simulation-testing/templates) for the full catalogue and what each one preconfigures.

### Keeping simulations under control

| Setting | Default | Recommendation |
| :- | :- | :- |
| **Max duration** | 15 minutes (up to 24h) | Set to \~110% of your average call duration. Prevents runaway calls if the agent goes on a tangent. |
| **Silence timeout** | 30 seconds (5–300s) | Ends calls after sustained silence, catching cases where either side stops responding. |
| **End-call phrases** | `goodbye` (max 10) | Specific phrases that end the simulation immediately when matched. Add phrases that signal a call has gone off track. |
| **End-call reasons** | none (max 10) | When you can't pin down exact phrases, describe the condition instead: an LLM evaluates each turn and ends the call when the reason is met. |
| **Iterations** | 1 (max 100) | Runs per test case. Fixed at 1 for outbound runs. |
| **Concurrency** | 5 | Keep at the default or below for regular runs to avoid unnecessary load on your agent and manage costs. Reserve high concurrency for the Load testing template. |

***

## Next Steps

> [card] [Customer Flows](/documentation/simulation-testing/customer-flows)
>
> Author improv briefs and scripted graphs

> [card] [Personas](/documentation/simulation-testing/personas)
>
> Create diverse caller profiles

> [card] [Templates](/documentation/simulation-testing/templates)
>
> Start from a goal with preconfigured plans

> [card] [Run Plans](/documentation/simulation-testing/run-plans)
>
> Configure and execute simulation plans


# Chat Simulations
Source: https://docs.roark.ai/documentation/simulation-testing/chat-simulations

Test text-based AI agents the same way you test voice agents

## Overview

Chat simulations run persona-driven conversations against text-based AI agents: the same testing primitives you use for voice, applied to text. Roark plays the human side of the chat, sends messages to your agent, and evaluates the resulting transcript against your metrics and checks.

If you've run voice simulations, you already know how chat simulations work. The mental model is identical: a [persona](/documentation/simulation-testing/personas) drives the customer's behavior, a [customer flow](/documentation/simulation-testing/customer-flows) defines the conversation, a [plan](/documentation/simulation-testing/run-plans) ties them together with an agent endpoint, and [metrics](/documentation/metrics/overview) measure the outcome.

***

## Voice vs. Chat Endpoints

Whether a simulation runs as a call or a chat is decided by the **endpoint type** on the agent target. You don't pick a modality separately.

| Modality | Endpoint types |
| :- | :- |
| **Voice** | Phone, WebRTC, LiveKit |
| **Chat** | WebSocket, ElevenLabs WS, Kore AI, Google CES |

> [!NOTE]
>
> ElevenLabs WS endpoints resolve to **chat**, not voice. To test an ElevenLabs agent over audio, use a Phone endpoint instead.

***

## How It's Different from Voice Simulations

Chat simulations reuse every concept from voice simulations, with a few practical differences:

| | Voice Simulations | Chat Simulations |
| :- | :- | :- |
| **Transport** | Phone, WebRTC, LiveKit audio | WebSocket / HTTP text messages |
| **Persona output** | Synthesized speech with accent, gender, pace | Typed text in the persona's style |
| **Step types** | All Scripted steps including Silence and DTMF | Text steps: Customer, Agent, First message |
| **Conversation record** | Stored as a **call** with audio + transcript | Stored as a **chat** with the message transcript |

Voice-only step types in Scripted flows (Silence, DTMF, and Voicemail steps in older flows) are skipped automatically when a flow runs against a chat endpoint, so you can reuse existing flows without forking them, and text-relevant steps still execute. [Voicemail testing](/documentation/simulation-testing/templates/voicemail-testing) is voice-only for the same reason: a greeting is audio, so there's nothing for a chat transcript to carry.

***

## What's Shared with Voice Simulations

Everything except the transport layer is shared:

* **Personas**: The attributes that shape text behavior carry over: **language**, **base emotion**, **clarity**, **response timing**, and **disfluencies** (natural false starts, rendered as typed hesitations). Voice-only attributes (accent, gender, speech pace) are simply ignored in a chat.
* **Customer flows**: Both Improv and Scripted flows work for chat. The same variants, agent expectations, and `{{variable}}` values apply.
* **Plans**: Build a chat plan the same way you build a voice one; the only difference is selecting an agent target with a chat endpoint.
* **Variables**: [Variables](/documentation/simulation-testing/variables) resolve identically.
* **Metrics and checks**: System and custom metrics run on chat transcripts, and Pass/Fail checks evaluate the same way. Metrics that only apply to audio (e.g. speech rate, silence duration) are skipped.
* **Schedules**: Chat plans can be [scheduled](/documentation/simulation-testing/schedules) the same way as voice plans.
* **Re-runs and reporting**: Identical experience, with chat results appearing alongside call results in run reports.

Write your flows and personas once, then point them at either a voice endpoint or a chat endpoint depending on what you want to test.

***

## How Chat Simulations Run

**1. Build a plan**

Attach customer flows, pick metrics and checks, and select an agent target with a **chat endpoint** (see supported providers below).

**2. Roark opens a session**

For each conversation, Roark establishes a session with your agent over the provider's chat transport. Credentials live on the integration. They're never embedded in the plan.

**3. The persona drives the conversation**

Roark plays the customer using the persona's language and behavioral attributes, following the flow variant and adapting to whatever your agent actually says.

**4. Transcript is captured and evaluated**

Every message is recorded as a chat. When the conversation ends, your configured metrics and checks are evaluated against the transcript.

> [!NOTE]
>
> Chat simulations always run outbound-from-Roark: Roark initiates the session against your agent endpoint. The [inbound vs. outbound](/documentation/simulation-testing/inbound-vs-outbound) distinction from voice simulations doesn't apply to chat.

***

## Watching Chat Simulations Live

While a run is in progress, chat sessions appear in the run's **Live now** board just like calls. The live transcript and conversation events stream in real time as the persona and your agent exchange messages. The only difference from voice is that there's no audio: the listen-in audio card is hidden for chat sessions.

![A chat simulation running live: the run's Live now board of in-flight chat sessions on the left, and an open chat session on the right streaming its message transcript as blue customer and green agent bubbles, with an agent typing indicator. No audio card is shown, and the channel reads ElevenLabs WS, chat.](https://mintcdn.com/roark/yUzvMIE3_BYBHPDx/images/simulation-testing/chat-live.png?fit=max\&auto=format\&n=yUzvMIE3_BYBHPDx\&q=85\&s=3438d114bb79cdf5ccc0e6e17967cbe0)

***

## Supported Providers

Chat simulations require a chat-capable agent endpoint. Provider endpoints are created automatically when you connect a supported integration. You don't need to wire them up manually.

> [card] [ElevenLabs](/documentation/integrations/elevenlabs)
>
> Run chat simulations against your ElevenLabs Conversational AI agents.

> [card] [Google CES](/documentation/integrations/google-ces)
>
> Run chat simulations against your Google Customer Engagement Suite apps.

> [card] [Kore AI](/documentation/integrations/kore)
>
> Run chat simulations against your Kore AI Agent Platform apps.

You can also target a plain **WebSocket** endpoint for custom text agents. See each provider's page for setup steps and required credentials.

***

## Viewing Chat Results

Completed chat simulations appear in **Simulate → Runs** alongside call results, under the run's SR-\{n} label. Opening a chat shows the full message transcript, persona context, and evaluated metric and check results.

![A settled chat run report: the Calls section listing each chat with its checks passed and turn count on the left, and an open chat conversation on the right showing the full message transcript as chat bubbles beside its evaluated metric results, with Passed/Failed checks and per-metric scores.](https://mintcdn.com/roark/yUzvMIE3_BYBHPDx/images/simulation-testing/chat-report.png?fit=max\&auto=format\&n=yUzvMIE3_BYBHPDx\&q=85\&s=21ae4d417420a6635a34897bb3fa9cc9)

Chats are first-class conversation records in Roark. They show up in reports, dashboards, and [collectors](/documentation/metrics/metric-collectors) the same way calls do.

***

## Next Steps

> [card] [Configure Personas](/documentation/simulation-testing/personas)
>
> Set up the customer side of your chats

> [card] [Build Customer Flows](/documentation/simulation-testing/customer-flows)
>
> Design the conversation in Improv or Scripted mode

> [card] [Connect a Provider](/documentation/integrations/overview)
>
> Pick the chat-capable platform you want to test

> [card] [Plans and Runs](/documentation/simulation-testing/run-plans)
>
> Combine everything into a runnable test suite


# CI/CD
Source: https://docs.roark.ai/documentation/simulation-testing/ci-cd

Gate your pipeline on a simulation: the run is judged against the success criteria on its plan and the build fails when your agent falls short

## Overview

A simulation you have to remember to run is a test suite nobody runs. Put it in your pipeline and every change to your agent is called by simulated customers, scored against your metrics, and turned into a pass or a fail before it reaches production.

Three pieces make that work, and only the first is required:

1. The **[GitHub Action](https://github.com/roarkhq/simulation-action)** starts a run, waits for it, and exits non-zero when the run misses its criteria.
2. **Success criteria** on the [run plan](/documentation/simulation-testing/run-plans#success-criteria) decide what "missed" means. Roark makes that decision, so the dashboard, the CLI and your pipeline never disagree about whether a run passed.
3. **[Config as Code](/documentation/config-as-code/overview)** optionally keeps the test suite itself (customer flows, personas, metrics, collectors) in your repo, so the thing being run is versioned alongside the agent it tests.

***

## Gate a pipeline in one step

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
name: Voice agent regression
on:
  push:
    branches: [main]

jobs:
  simulate:
    runs-on: ubuntu-latest
    steps:
      - uses: roarkhq/simulation-action@v1
        with:
          api-token: ${{ secrets.ROARK_API_KEY }}
          plan-id: <your-plan-id>
```

That is the whole gate. The step places the calls, waits for every one of them to be scored, prints a per-check table to the job summary, and fails the build if a check fell short.

> [!NOTE]
>
> The API key needs permission to start simulations and read their results. Store it as a repository secret. See [Using the CLI in CI](/documentation/sdks/cli#using-the-cli-in-ci) for how credentials resolve.

Pass runtime values the plan's variables expect with `variables`, one `KEY=VALUE` per line:

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
      - uses: roarkhq/simulation-action@v1
        with:
          api-token: ${{ secrets.ROARK_API_KEY }}
          plan-id: <your-plan-id>
          variables: |
            orderNumber=12345
            customerName=John Doe
```

***

## How the pass/fail decision is made

Every check is judged **on its own**. A check is one pass/fail metric: a yes/no metric, or a threshold on another metric the plan collects. Each one has a minimum pass rate, the share of the run's simulations that must pass it, and the run passes only when every check clears its own minimum.

| Check | Passed | Rate | Minimum | Result |
| :- | :- | :- | :- | :- |
| silence duration \< 500ms | 4/10 | 40% | 40% | Pass |
| word count \< 1000 | 7/10 | 70% | 80% | **Fail** |

The run **fails**. `word count` missed its own bar, and no amount of success elsewhere makes up for it. There is no pooled average to clear, because averaging would let a check at 0% hide behind checks at 100%, which is the exact failure a gate exists to catch.

A check that sets no minimum of its own is held to **80%**. Set the minimums on the plan, under **Advanced → Success criteria**, or over the API. See [Success criteria](/documentation/simulation-testing/run-plans#success-criteria).

> [!NOTE]
>
> Criteria are pinned to each run when it starts, so editing a plan never rewrites the verdict of a run that already happened.

### Score is not the decision

A run also reports a **score**: the mean of the check rates, 0 to 100. It is for dashboards and trend lines only.

Nothing is gated on it, and you should not gate on it either. A run can score 95 and fail (one non-negotiable check missed its bar) or score 40 and pass (every check cleared a deliberately low bar).

***

## Three outcomes, not two

A gate answers one question: did this change make the agent worse? Only a run that actually ran can answer it, so the action reports three outcomes.

| Outcome | When | Exit code |
| :- | :- | :- |
| **PASSED** | Every check cleared its own minimum. | 0 |
| **FAILED** | The run completed cleanly and a check fell short. | **1** |
| **SKIPPED** | The run never produced a judgeable result: a Roark outage, a model-provider error, simulations that died before they were graded, a check that produced no result. | 0, with a warning |

A check falling below its minimum is the **only** thing that turns your build red. Problems on our side are annotated loudly, reported as `verdict: SKIPPED` with a `skip-reason`, and then they get out of your way.

That is deliberate. If our infrastructure can turn your pipeline red, the first thing your team learns is to re-run until green, and from then on nobody reads the gate at all, including on the run where the agent really did regress.

Operational problems **poison** the verdict rather than sitting beside it. If half a run's simulations never completed, a check reported at 40% is not evidence of a regression, it is evidence we could not measure. So any operational failure makes the whole run `SKIPPED`, even when a check also fell short.

Two exceptions, both deliberate:

* **A plan with no checks fails the build.** That is not an outage, it is a plan that can never gate anything, and skipping it would leave a green check next to a permanent no-op. Add a threshold to one of its metrics, or attach a yes/no metric.
* **A rejected request fails the build.** A bad token, an unknown `plan-id`, or a config the API refuses is something only you can fix, and a gate that skips on a typo never gates again. Only errors that read as ours (5xx, rate limits, dead sockets) are skipped.

Set `fail-on-run-error: true` if you would rather block than proceed unmeasured.

***

## Keep the run config in your repo

Instead of a saved plan, point the action at a YAML file. The run is described in the same commit as the agent change it tests:

```yaml .roark/checkout-regression.yml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
name: Checkout regression
direction: OUTBOUND
maxSimulationDurationSeconds: 300
agentEndpoints:
  - id: <your-agent-endpoint-id>
flows:
  - id: <your-flow-id>
    happyPath: true
    edgeCases: ALL
metrics:
  - slug: agent_containment
    # No minPassRate, so this check is held to the 80% default.
  - slug: latency
    minPassRate: 12
  - slug: leaked_pii
    minPassRate: 100
```

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
      - uses: actions/checkout@v4
      - uses: roarkhq/simulation-action@v1
        with:
          api-token: ${{ secrets.ROARK_API_KEY }}
          config: .roark/checkout-regression.yml
```

`minPassRate` is the only thing a plan says about a check, because it is the only part that is the plan's business. What counts as passing a single simulation belongs to the metric: a "did the agent leak PII?" check passes on FALSE because the metric carries a threshold of `EQUALS false`, stated once in Roark rather than re-stated by every plan that uses it. The API rejects keys it does not recognise, so a stray field here fails the run rather than being quietly ignored.

This runs as a one-off: nothing is added to your saved plans. Set `save-as-plan: true` to keep it instead, and read the new plan's id off the `plan-id` output so later runs can reference it directly.

> [!TIP]
>
> Config as Code and this run config are different files with different jobs. Config as Code manages the resource **definitions** your runs draw on (flows, personas, metrics); the run config above says which of them to exercise, against which agent, at what volume, and it is the one that takes `minPassRate`. A plan declared as `kind: simulationPlan` in a Config as Code bundle cannot carry minimums yet, so its checks all sit at 80%.

***

## Hold one branch to a higher bar

`min-pass-rate` holds every check to at least that minimum for this pipeline only, leaving the shared plan alone:

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
      - uses: roarkhq/simulation-action@v1
        with:
          api-token: ${{ secrets.ROARK_API_KEY }}
          plan-id: <your-plan-id>
          min-pass-rate: 99
```

It applies per check, on top of each check's own minimum, so a release branch can demand more than the plan without editing it. It can only tighten: it will never let through a run the plan's own criteria failed, because overruling a minimum the plan's owner set is not something a pipeline gets to do.

***

## Inputs

| Input | Required | Default | Description |
| :- | :- | :- | :- |
| `api-token` | yes | | Roark API key. Pass it from a secret. |
| `plan-id` | one of | | Saved run plan to run. |
| `config` | one of | | Path to a YAML file describing the run. |
| `variables` | no | | Runtime variables, one `KEY=VALUE` per line. |
| `save-as-plan` | no | `false` | Keep the `config` as a named run plan instead of running it as a one-off. |
| `min-pass-rate` | no | | Hold every check to at least this minimum (0-100) for this pipeline. Tightens only. |
| `timeout-minutes` | no | `30` | How long to wait for the run. |
| `poll-interval-seconds` | no | `15` | How often to check for completion. |
| `fail-on-timeout` | no | `false` | `true` fails the build when the run overruns instead of warning. |
| `fail-on-run-error` | no | `false` | `true` fails the build when the run could not be judged instead of skipping. |
| `cancel-on-exit` | no | `true` | Stop the Roark run when the workflow is cancelled or the wait times out. |
| `cli-version` | no | pinned | Version of `@roarkanalytics/cli` to run. |
| `api-base-url` | no | `https://api.roark.ai` | Override the API base URL. |

## Outputs

| Output | Description |
| :- | :- |
| `run-id` | The simulation run id. |
| `run-url` | Link to the run in the Roark platform. |
| `plan-id` | The run plan behind this run. With `save-as-plan`, the plan that was kept. |
| `score` | The run's quality score, 0-100. Reporting only: gate on `verdict`, never this. |
| `checks-passed` | How many checks cleared their own minimum. |
| `checks-total` | How many checks the run was judged on. |
| `verdict` | `PASSED`, `FAILED`, `SKIPPED`, or `TIMED_OUT`. |
| `skip-reason` | Why the run could not be judged. Set only alongside `SKIPPED`. |

***

## Simulations take minutes

Real calls take real time, so a gated run occupies a runner while it waits. Four ways to keep that predictable:

* **Run it where it matters.** Gate `main` or your release branch rather than every push to every branch.
* **Let our slowness warn rather than block.** An overrun is a warning by default (`fail-on-timeout: false`), while a genuine check failure still fails the build.
* **Cancel superseded runs.** A `concurrency` group stops an old push from holding a runner while a newer one is already testing the same branch. The action cancels the Roark run too, so the abandoned simulation stops placing calls you would otherwise be billed for.
* **Keep a job-level backstop.** `timeout-minutes` on the job is the last line of defence if the step itself wedges.

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
concurrency:
  group: roark-${{ github.ref }}
  cancel-in-progress: true
```

***

## Sync your test suite first

If your flows, personas and metrics live in git, apply them before the run so the suite being executed matches the commit:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
npx @roarkanalytics/cli config apply ./roark -y
```

`apply` exits non-zero if the bundle is invalid, so a broken config fails the build on its own. Run `roark config diff ./roark` on pull requests to preview changes before they land. See [Config as Code](/documentation/config-as-code/overview) for the full workflow.

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
      - name: Apply config
        run: npx @roarkanalytics/cli config apply ./roark -y
        env:
          ROARK_API_BEARER_TOKEN: ${{ secrets.ROARK_API_KEY }}

      - uses: roarkhq/simulation-action@v1
        with:
          api-token: ${{ secrets.ROARK_API_KEY }}
          plan-id: <your-plan-id>
```

***

## Outside GitHub Actions

The action is a convenience, not the mechanism. Any CI system can gate on the same verdict with the CLI or a pair of HTTP calls: start a run, poll until it reaches a terminal status, then read `verdict.passed`.

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
export ROARK_API_BEARER_TOKEN="$ROARK_API_KEY"

run_id="$(roark simulation run --data "{\"planId\":\"$PLAN_ID\"}" | jq -r '.data.simulationRunPlanJobId')"

# Poll until the run reaches a terminal status.
while :; do
  response="$(roark simulation plan job get "$run_id")"
  status="$(printf '%s' "$response" | jq -r '.data.status')"
  case "$status" in
    COMPLETED | FAILED | CANCELLED | TIMED_OUT) break ;;
  esac
  sleep 15
done

# `passed` is the exit status. `score` is reporting only.
printf '%s' "$response" | jq -e '.data.verdict.passed == true' >/dev/null
```

The verdict is attached to `GET /v1/simulation/plan/job/{jobId}` once the run has ended:

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "passed": false,
  "score": 85,
  "checks": [
    {
      "metricDefinitionId": "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
      "metricName": "silence duration < 500ms",
      "passRate": 100,
      "minPassRate": 40,
      "inherited": false,
      "passedSims": 10,
      "evaluatedSims": 10,
      "passed": true
    },
    {
      "metricDefinitionId": "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
      "metricName": "word count < 1000",
      "passRate": 70,
      "minPassRate": 80,
      "inherited": false,
      "passedSims": 7,
      "evaluatedSims": 10,
      "passed": false
    }
  ],
  "failures": [
    {
      "type": "METRIC_BELOW_MIN_PASS_RATE",
      "metricDefinitionId": "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
      "metricName": "word count < 1000",
      "passRate": 70,
      "minPassRate": 80,
      "inherited": false
    }
  ]
}
```

`failures` is a typed list rather than a rendered message, so your pipeline can tell the two kinds apart and treat them differently:

| `type` | Meaning | Whose problem |
| :- | :- | :- |
| `METRIC_BELOW_MIN_PASS_RATE` | A check fell short of its minimum. | Your agent. Fail the build. |
| `RUN_NOT_COMPLETED` | The run ended in `FAILED`, `CANCELLED` or `TIMED_OUT`. | Ours. Retry or warn. |
| `INCOMPLETE_COVERAGE` | Fewer simulations were graded than the run expected. | Ours. The numbers are not evidence. |
| `METRIC_NOT_EVALUATED` | A check produced no result at all. | Ours. |

`inherited: true` means the check was judged against the 80% default rather than a minimum it set for itself.

> [!WARNING]
>
> `verdict` is `null` when the plan carries no pass/fail check, because there was nothing to judge. Treat that as a configuration error rather than a pass: a plan that cannot fail is a gate that does nothing.

***

## Related

> [card] [Success criteria](/documentation/simulation-testing/run-plans#success-criteria)
>
> Set the minimum each check has to clear

> [card] [Config as Code](/documentation/config-as-code/overview)
>
> Define your test suite as YAML in git

> [card] [CLI](/documentation/sdks/cli)
>
> Install, authenticate, and use the CLI in CI

> [card] [Running simulations](/documentation/simulation-testing/running-simulations)
>
> Launch runs and read the report when they settle


# Customer Flows
Source: https://docs.roark.ai/documentation/simulation-testing/customer-flows

Define the customer conversations your agents should handle, and get graded against

## Quickstart

Create a customer flow (an improv brief graded against your expectations).

Define a flow [in code](/documentation/config-as-code/flows) (recommended), or build it in the dashboard.

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineImprovFlow, defineConfig } from '@roarkanalytics/sdk/lib/config';

const billingQuestions = defineImprovFlow({
  name: 'billing-questions',
  agents: ['frontdesk'],
  happyPath: {
    persona: 'frustrated-caller',
    environment: 'Quiet line',
    prompt: 'You want to understand a charge on your latest invoice.',
    expectations: ['Agent explains the charge clearly'],
  },
  edgeCases: [
    {
      name: 'disputes-the-charge',
      prompt: 'You believe the charge is wrong and want it removed.',
      expectations: ['Agent explains how to dispute the charge'],
    },
  ],
});

export default defineConfig({ resources: [billingQuestions] });
```

```yaml roark/flows/billing-questions.yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: flow
type: improv
name: billing-questions
agents: [frontdesk]
happyPath:
  persona: frustrated-caller
  environment: Quiet line
  prompt: You want to understand a charge on your latest invoice.
  expectations:
    - Agent explains the charge clearly
edgeCases:
  - name: disputes-the-charge
    prompt: You believe the charge is wrong and want it removed.
    expectations:
      - Agent explains how to dispute the charge
```

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply roark.config.ts
```

### Create it with the SDK

> [tab] Node
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> const flow = await client.customerFlow.create({
>   type: 'IMPROV',
>   title: 'Billing questions',
>   agentIds: ['<agent-id>'],
>   happyPath: {
>     title: 'Asks about a charge',
>     personaOverrideId: '<persona-id>', // a built-in or custom persona
>     environmentId: '<environment-id>', // e.g. "Quiet line"
>     prompt: 'You want to understand a charge on your latest invoice.',
>   },
>   agentExpectations: [{ prompt: 'Agent explains the charge clearly' }],
> })
> ```

> [tab] Python
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> flow = client.customer_flow.create(
>     type="IMPROV",
>     title="Billing questions",
>     agent_ids=["<agent-id>"],
>     happy_path={
>         "title": "Asks about a charge",
>         "personaOverrideId": "<persona-id>",
>         "environmentId": "<environment-id>",
>         "prompt": "You want to understand a charge on your latest invoice.",
>     },
>     agent_expectations=[{"prompt": "Agent explains the charge clearly"}],
> )
> ```

Need ids? Grab a built-in [persona](/documentation/simulation-testing/personas) and environment with `simulationPersona.list()` / `simulationEnvironment.list()` (see the [first-simulation quickstart](/documentation/getting-started/introduction#your-first-simulation)). Both are required on the happy path.

## What is a customer flow?

A customer flow describes one type of customer conversation: who the customer is, what they want, the variants you need to cover (language, mood, environment), and the **agent expectations** the agent is graded against. Flows live under **Simulate → Customer flows** in the sidebar.

Today, flows drive simulations: you attach them to run plans and Roark generates test calls from them. The same definition is what live-call analytics will grade incoming traffic against next.

> [!NOTE]
>
> Customer flows supersede the "scenarios" concept from earlier versions of Roark. The REST API and SDKs still use the older name. For example, run requests take a `scenarios` field.

Every flow has:

| Part | What it does |
| :- | :- |
| **Title & description** | Identify the flow in the hub and in run reports |
| **Linked agent(s)** | At least one agent must be linked: saving is blocked otherwise |
| **Labels** | System labels (Adversarial, Health check, Voicemail) and free-form custom labels |
| **Agent expectations** | LLM-graded instructions, inherited by every variant |
| **Variants** | One default **happy path** plus any number of **edge cases** |

![The Customer flows hub listing flows with Improv and Scripted mode badges, label chips, and a variants, expectations, and agents meta line, above a search field, source and mode filters, and a New flow button](https://mintcdn.com/roark/93k2kGpZVUUbTXSq/images/simulation-testing/customer-flows-hub.png?fit=max\&auto=format\&n=93k2kGpZVUUbTXSq\&q=85\&s=69d7e5697e672f14c8b9112d3c62cdf9)

***

## Authoring modes: Improv vs Scripted

Each flow is authored in one of two modes, toggled with the **Improv / Scripted** pill in the editor header.

| | **Improv** | **Scripted** |
| :- | :- | :- |
| You write | A free-text customer brief: the situation the customer is in, what they want, how they behave | The conversation itself, step by step, on a graph canvas |
| Each run | The simulator improvises a fresh conversation: different words, same intent | Walks your graph, following the exact lines you authored. [Branching](#branching-how-paths-become-calls) decides whether every path runs as its own call or one call branches live |
| Best for | Realistic, open-ended conversations; broad behavioral coverage | IVR menus, DTMF entry, deterministic routing, precise turn-by-turn checks |
| Variants | You author them by hand | Derived automatically: one per path through the graph |

Switching modes is non-destructive: both data shapes coexist on the flow, so you can toggle back and forth. If the other mode already has authored content, the editor asks you to confirm the switch. Your work is retained either way.

> [!NOTE]
>
> In the REST API, Improv mode is the `UNSCRIPTED` value of the `FlowMode` enum. There is no `IMPROV` enum value. The UI label is newer than the API name.

### Improv mode

The improv editor is organized into three numbered sections:

1. **Happy path**, the customer setup: a free-text brief describing the situation the customer is in, what they want, and how they typically behave. This section also carries the happy path's Persona, Environment, and Variables chips, and a **Link preceding flow** control (see [flow composition](#composing-flows) below).
2. **Agent expectations**: the required agent picker plus expectation rows, inherited by every variant.
3. **Edge cases**: additional variants that inherit from the happy path and override only what differs.

![The Improv customer-flow editor for a Billing questions flow, organized into three numbered sections: 01 Happy path with a customer-setup brief containing an invoice\_total variable chip and Persona, Environment, and Variables chips; 02 Agent expectations with a linked Front Desk agent and three LLM-graded expectation rows plus an Add expectation control; and 03 Edge cases with an expanded Disputes the charge variant and a collapsed Spanish-speaking caller variant above an Add another edge case control, with a Save flow action in the footer](https://mintcdn.com/roark/Q1LycOQn64NfkIxp/images/simulation-testing/improv-editor.png?fit=max\&auto=format\&n=Q1LycOQn64NfkIxp\&q=85\&s=22fc23a59a3947be3fb3ef8d2997248d)

### Scripted mode

Scripted mode replaces the form with a full-screen canvas where you author the conversation as a graph of steps. See [the scripted graph](#the-scripted-graph) for step kinds and rules.

***

## Variants: happy path and edge cases

A variant is one per-caller configuration of the flow: a title, persona, environment, variables, per-variant labels, and optional additional expectations.

* **Exactly one variant is the default**: the **happy path**. It must have a customer setup and a persona.
* **Every other variant is an edge case.** Edge cases inherit the happy path's setup, persona, and environment unless they override them. The setup field on an edge case reads "Inherited from the happy path, type to override."
* **Additional expectations append.** An edge case can add expectations that apply just to that variant; they never replace the flow-level set.
* Each edge case can carry its **own labels**, and its menu lets you **Mark as happy path** (making it the new default) or **Delete edge case**.

> [!TIP]
>
> An edge case that overrides nothing is pointless, and the editor blocks saving it. Give each edge case at least a different setup, persona, or environment.

### Persona, Environment, and Variables chips

Every variant carries a chip strip configuring who's calling and from where:

* **Persona**: the simulated customer's voice and delivery. The happy path requires one (a new flow auto-seeds the built-in **Polite First-Time Caller** persona, falling back to your project's first persona); the picker can also create or edit personas inline. See [Personas](/documentation/simulation-testing/personas).
* **Environment**: the background noise played under the customer's voice and how loud it plays. Pick a Roark preset (Quiet line, Office, Coffee shop, City street, Driving, Airport, Children playing, Thunderstorm), or drag the noise level and save the result as one of your own environments. Edge cases also get an **Inherit** option. See [Environments](/documentation/simulation-testing/environments).
* **Variables**: per-variant key/value entries (name, account ID, last visit…) handed to the customer-side model so it can answer the agent's questions consistently. Add rows manually, or use **Apply a test profile** to replace the entries with a saved profile's properties. Prompt and step text support `{{variable}}` tokens with autocomplete: `{{name}}` for flow-scoped properties, `{{persona.name}}` for persona-scoped ones. See [Variables](/documentation/simulation-testing/variables).

***

## Agent expectations

Agent expectations are the graded contract of the flow: short, LLM-evaluated instructions like *"Agent greets warmly and uses the caller's name once."* After each simulated conversation, Roark checks the transcript against every expectation and reports pass/fail per expectation, per call.

* **Flow-level expectations** are inherited by every variant.
* **Additional expectations** on an edge case apply only to that variant, on top of the flow-level set.
* In scripted flows, expectations are also derived from the agent turns along each path: what you wrote the agent should say becomes what it's graded on.

Write expectations as specific, checkable statements. "Agent offers a callback when the customer declines to hold" grades cleanly; "Agent is helpful" doesn't.

***

## Labels

Flows use a single label system with two kinds:

* **System labels**. Roark-curated: `Adversarial`, `Health check`, `Voicemail`. These drive template behavior; for example, the Red teaming template automatically runs flows carrying the Adversarial label. Roark also maintains a `Happy path` label automatically on each flow's default variant, which templates that run happy paths source; you don't apply it by hand.
* **Custom labels**: free-form tags for your own organization.

Labels attach at the flow level and per edge case, and the hub filters by them (alongside a **Mode** filter for Improv vs Scripted).

***

## The scripted graph

In scripted mode you build the conversation as a directed graph rooted at a start node. Click **Add step** to insert a step, then connect steps with edges; selecting a node opens an inspect panel with its kind-specific fields.

### Step kinds

| Step | What it does |
| :- | :- |
| **Agent** | What the agent says at this point (also becomes a graded expectation on the path) |
| **Customer** | What the customer says back |
| **First message** | The customer's opening line (only valid immediately after the start node) |
| **DTMF** | The caller presses keypad digits (`0–9`, `*`, `#`; use `#` to mark end-of-input) |
| **Silence** | The caller pauses for 1–60 seconds |
| **Flow link** | Splices another scripted flow's steps in at this point |

> [!NOTE]
>
> The API keeps older names here too: step kinds map to the `SimulationStepType` enum, where a Flow link is `SCENARIO_LINK` and a First message is `CUSTOMER_FIRST_MESSAGE`.

> [!NOTE]
>
> **Looking for the Voicemail step?** Voicemail is a run template now, not a step. See [Voicemail Testing](/documentation/simulation-testing/templates/voicemail-testing). A voicemail has no conversation to graph and only one meaningful variable (which greeting plays), so the template runs Roark's recordings directly. Flows that already contain a Voicemail step keep working; you just can't add new ones.

### Graph rules

* **Branching is allowed**: give a step multiple outgoing edges to fork the conversation.
* **Merges are allowed**: branches can rejoin.
* **Cycles are blocked**: the graph must be a DAG, so a conversation always terminates.

### Paths derive variants

Every unique route through the graph from start to finish is a **path**, and when you save the flow, Roark derives one variant per path. Each path card in the right rail shows a generated description of its route. You don't hand-author scripted variants; you shape the graph and the variants follow.

You can still control *who* runs each path: **Add variant for path** creates an extra copy of a path that runs it with a different persona, environment, or variables. The rail marks the flow's default variant with a **Default** badge and groups the rest under **Edge cases**, with **Make default** and **Delete variant** available per variant.

How those paths turn into *calls* is a separate choice, covered next in [Branching](#branching-how-paths-become-calls).

### Branching: how paths become calls

A branching graph can be run two ways, chosen with the **Branching** control at the top of the variant rail. Both modes speak the exact lines you authored. The choice decides only *when* a branch is picked and *how many calls* a run places. Neither mode changes how metrics or agent expectations grade.

| | **Simulate every path** | **Adapt to your agent** |
| :- | :- | :- |
| Branch is picked | Up front, when the flow is saved | During the call, from what your agent actually says |
| Calls per run | One per variant the run includes | One per persona among the included variants |
| Coverage | Every included path, every run | One path per call, whichever the agent leads to |
| Best for | Regression suites, full branch coverage, DTMF and IVR entry | Agents whose replies you can't predict; shared blocks linked into other flows |

Take a caller asking for a Tuesday appointment, where the agent's own answer forks the conversation:

```mermaid theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
graph TD
    C1["👤 Hi, can I book a\n cleaning for Tuesday?"] --> A1["🤖 Tuesday at\n 9am is open"]
    C1 --> A2["🤖 Tuesday is\n fully booked"]
    A1 --> C2["👤 Perfect,\n book it"]
    A2 --> C3["👤 What about\n Wednesday?"]
```

**Simulate every path** places two calls. Call 1 always takes the left branch and call 2 always takes the right one, even when your agent says something the other branch was written for. That is the point: a path that stopped matching your agent's behavior surfaces as a failure instead of being quietly routed around.

**Adapt to your agent** places one call. The simulated customer hears which reply your agent actually gave and follows that branch; the branch it didn't take isn't exercised on that call.

The **Branching** picker in the app shows the call count each mode places for the flow you're editing, with every variant running once. A run plan's own configuration wins over that figure: selecting a subset of variants, applying a persona override, or setting an iteration count all change what a run actually places.

> [!NOTE]
>
> **Personas multiply calls in both modes.** Adapt to your agent collapses a flow's paths into one call **per persona**, not one call outright. Five paths across three personas run as three calls. On a collapsed call, variable values come from the primary variant; the sibling variants contribute their path shape, not their own values. Sibling-only variable *names* are still detected and prompted for, so they don't vanish silently.

> [!WARNING]
>
> **Linking a multi-path flow requires Adapt to your agent.** A **Flow link** step targeting a flow that is set to *Simulate every path* and has more than one reachable path fails the run: composed fan-out across linked flows isn't supported yet. Either set the linked flow to **Adapt to your agent**, so the run follows whichever path the agent leads to, or link a single-path flow.

> [!TIP]
>
> Reach for **Adapt to your agent** when you genuinely can't predict the branch: an IVR tree whose menu order varies, or a shared block linked into many flows. Use **Simulate every path** for everything else. It's the mode that gives you coverage you can regress against.

> [!NOTE]
>
> **Adapt to your agent is not looser grading, and it is not goal-level authoring.** Both branching modes speak your exact scripted lines. To have the agent checked against intent ("confirms the caller's name") rather than a script, use an [Improv flow](#authoring-modes-improv-vs-scripted) with agent expectations.

> [!NOTE]
>
> In the REST API and the config-as-code DSL these are the `DETERMINISTIC` and `ADAPTIVE` values of the `ScriptedBranchingMode` enum. The UI labels are newer than the API names.

![The Branching picker in the scripted flow's variant rail: Simulate every path (4 calls, selected) and Adapt to your agent (1 call), with the selected mode's description below, beside the open How branching works explainer showing the same two-path graph run both ways](https://mintcdn.com/roark/yUzvMIE3_BYBHPDx/images/simulation-testing/branching-picker.png?fit=max\&auto=format\&n=yUzvMIE3_BYBHPDx\&q=85\&s=9a928bf96977d48e388f46186ad38a45)

> [!WARNING]
>
> Structural edits (adding or deleting nodes or edges) freeze variant editing until you save: "You've edited the graph. Save the flow to refresh its paths before adding or removing variants."

![The scripted flow editor: a branching conversation graph of agent and customer turn nodes connected by edges, with the happy path highlighted, and a Variants rail listing the happy path and edge cases](https://mintcdn.com/roark/IjseSxRBSMbit4Aq/images/simulation-testing/scripted-canvas.png?fit=max\&auto=format\&n=IjseSxRBSMbit4Aq\&q=85\&s=c047ffea266e13200c24518be7ae5b02)

***

## Composing flows

Flows compose in two directions:

* **Flow link (scripted → scripted).** A scripted flow can include a **Flow link** step that splices another scripted flow's steps in mid-graph. Use this to reuse a shared segment (an IVR menu, an identity-verification exchange) across many flows. Only scripted flows can be linked this way. A linked flow with more than one path must be set to **Adapt to your agent**; see [Branching](#branching-how-paths-become-calls).
* **Preceded by (scripted → improv).** An improv variant's **Link preceding flow** control names a scripted flow (and optionally a specific variant of it) that runs *before* the improv segment. Typical use: navigate a scripted IVR tree deterministically, then improvise the conversation once a human-like agent picks up.

  The preceding flow is performed by its own persona. When it reaches its last step, the phone is handed to the persona the run is testing, which carries the conversation from there. That handover is what makes persona properties such as accent, pace or emotion apply to the conversation with your agent. The part before the handover is left out of every metric, so an IVR's recorded prompts and keypresses never count against your agent. A call where the handover never happens is [excluded](/documentation/simulation-testing/running-simulations#calls-that-dont-count) rather than scored, because the persona under test never spoke.

Improv has no deterministic end, so an improv segment can only ever come last. You can't link *into* an improv flow from a scripted step.

***

## Creating a flow

From the hub, click **New flow** to open the chooser at `/customer-flows/new`. You can generate a draft or start blank:

**Generate: from existing material** (each opens the Ask Roark assistant with a seeded prompt):

* **Describe what you want**: tell Ask Roark the conversation in plain language and it drafts the flow.
* **From a transcript**: upload or paste a transcript and Roark turns it into a flow.
* **From your calls**: pick real calls from your production traffic to base the flow on.

**Or start blank:**

* **Improv: Describe the customer** opens the brief editor.
* **Scripted: Author the conversation** opens the graph canvas.

Variants drafted by Ask Roark carry a **Generated** badge until you review or promote them.

**1. Title and describe the flow**

Give it a clear title and a short description of the conversation type it covers.

**2. Write the happy path**

In Improv, write the customer setup brief; in Scripted, build the graph. Pin a persona, pick an environment, and add any variables the customer should know.

**3. Link an agent and add expectations**

Pick at least one agent, then add expectation rows describing what the agent must do. These are inherited by every variant.

**4. Add edge cases**

Cover the deviations that matter: a different language, a frustrated caller, a noisy environment, missing information. Override only what differs from the happy path.

**5. Save**

The save button surfaces the first blocking issue if anything is missing, for example "Link at least one agent to save", "The happy path needs a prompt", or (scripted) "Add at least one step to the graph".

***

## Using flows in a run

Flows drive simulations through run plans. In the create-run page, the **Attach flows** picker lists your library; expand a flow to choose its variants. Each attachment carries:

* **A variant selection**: all variants, the default variant (happy path) only, or specific variants you pick.
* **An optional persona override**: run the flow as a different customer without editing it. The Multilingual template uses this to fan one flow out across language personas.
* **Variable overrides**: per-attachment `{{variable}}` values that win over the variant's stored values at run time.

When a run starts, it snapshots the attached flows and variants, so the report always reflects exactly what was tested, even if you edit the flow afterwards.

![The Attach flows dialog with one flow expanded to its variant selection (All variants / Happy path only / Specific variants) and the happy path and edge-case variants listed](https://mintcdn.com/roark/wL__NyRpzLGS2dwF/images/simulation-testing/attach-flows.png?fit=max\&auto=format\&n=wL__NyRpzLGS2dwF\&q=85\&s=7f83548c58946c7b1b1861c58f7e4a2b)

> [card] [Run Plans](/documentation/simulation-testing/run-plans)
>
> Combine flows, agents, and metrics into reusable test suites

> [card] [Personas](/documentation/simulation-testing/personas)
>
> Configure the simulated customer's voice and delivery

> [card] [Variables](/documentation/simulation-testing/variables)
>
> Parameterize flows with per-variant and run-time values

> [card] [Templates](/documentation/simulation-testing/templates)
>
> Start runs from goals like red teaming or multilingual coverage


# Enriched Simulations
Source: https://docs.roark.ai/documentation/simulation-testing/enriched-simulations

Get richer call data in simulations by sending your agent call to Roark

## Overview

By default, Roark captures the recording from its own simulation agent's side of the conversation. This gives you basic transcript and metric analysis, but it's limited to what Roark can observe externally.

When you also send your agent's call data to Roark, via an [integration](/documentation/integrations/overview) or the [API/SDK](/documentation/observability/overview#how-calls-get-into-roark), Roark matches and merges the two calls. This unlocks significantly richer analysis because your call carries data that only your agent has access to.

***

## What You Gain

Enriched simulations give you access to data from your agent's perspective, enabling deeper analysis:

| Data | Description |
| :- | :- |
| **Full Transcript** | Your agent's transcript, which may be higher quality or include internal annotations |
| **Tool Invocations** | The actual [tool calls](/documentation/tool-invocations) your agent made during the conversation (e.g., database lookups, booking actions) |
| **OpenTelemetry Traces** | Backend [traces](/documentation/observability/traces) showing your agent's internal processing |
| **Properties & Metadata** | Any custom properties or context your agent attaches to the call |

This additional context enables metrics that aren't possible with the simulation recording alone. For example, the **Tool Invocation** metric can verify whether your agent actually called the right tool with the correct parameters, not just whether it *said* it would.

***

## Enabling Call Merging in a Run

Call merging is controlled per run. In the create-run **Metrics** block, turn on **Collect selected metrics from your agent's side of the call**. When the toggle is on, Roark waits for your agent's copy of each call and scores the selected metrics against it.

![Create-run Metrics block with the 'Collect selected metrics from your agent's side of the call' toggle enabled, and each selected metric showing its per-call collection source](https://mintcdn.com/roark/yUzvMIE3_BYBHPDx/images/simulation-testing/enrich-toggle.png?fit=max\&auto=format\&n=yUzvMIE3_BYBHPDx\&q=85\&s=187b694df84541ee976119d53b35ba44)

### Per-Metric Conversation Source

With the toggle on, each selected metric has a conversation source that determines which side of the call it's scored against:

| Source | Scored against | Best for |
| :- | :- | :- |
| **Simulated** | Roark's side of the call: the simulation agent's own recording and transcript | Metrics that only need what's audible in the conversation |
| **Live** | Your agent's side of the call: the call data your agent sends to Roark | Metrics that need internal data your agent carries |

### Live-Only Metrics

Some metrics can only be evaluated from your agent's side of the call: **tool invocations** are the canonical example, since only your agent knows which tools it actually called. Selecting a live-only metric locks the call-merging toggle on: you can't disable it while that metric is in the run.

### Waiting and Fallback

When call merging is enabled, Roark waits **up to 15 minutes** for your agent's side of each call to arrive. If it doesn't show up in that window, Roark falls back and completes analysis with the simulated side only. Live-sourced metrics won't have your agent's data to score against.

> [!TIP]
>
> Send calls to Roark as soon as they end. The sooner your agent's side arrives, the sooner merged results appear in the run report, and you avoid hitting the 15-minute fallback.

> [!NOTE]
>
> The API field behind the toggle keeps its internal name, `enrichWithLiveConversation`.

### Attaching Tool Invocations Directly

If the only agent-side data you need is **tool invocations**, you don't have to send a full second call and wait for it to merge. Instead, attach the tool calls straight to the simulation's call once it exists:

1. Find the simulation's call (list calls filtered by `simulationRunPlanJobId`, or read it from the run).
2. Attach the tools to it via [`POST /v1/call/{callId}/tool-invocations`](/documentation/tool-invocations#attaching-tool-calls-after-a-call), optionally passing `metrics` to score the tool metrics in the same request.

This sidesteps phone/timing matching and the 15-minute wait entirely, so it's the simplest path when you just want tool metrics on a simulated call. Attaching is idempotent, so it's safe to retry.

***

## How Matching Works

Once your agent's call arrives, Roark matches it to the simulation using two key signals:

### Phone Number

The phone number your agent interacted with is matched against the number Roark provisioned for the simulation. This is the same dynamically assigned number described in [Identifying Simulations](/documentation/simulation-testing/identifying-simulations).

### Timing

The call must have started during the simulation's active window. Roark checks that your call's start time falls within the time range when the simulation phone number was in use for that specific test case.

Both signals must match for the calls to be merged. This ensures accuracy even when phone numbers are reused across different runs.

> [!NOTE]
>
> Matching is bidirectional. It doesn't matter which call arrives first. If your agent's call arrives before Roark finishes processing the simulation, or vice versa, the merge happens once both are available (within the wait window described above).

> [!WARNING]
>
> `externalId` does **not** merge a call with a simulation. It correlates a call with [OpenTelemetry traces](/documentation/observability/traces) only. Simulation merging uses the phone number and timing above. If you set `externalId` expecting a merge, the calls will not join, and your live-sourced metrics will fall back to the simulated side. To attach agent-side tool data without relying on the merge, use [Attaching Tool Invocations Directly](#attaching-tool-invocations-directly).

***

## Setup

Beyond the run toggle, the only requirement is that your agent already sends calls to Roark. If you haven't set up call ingestion yet:

1. **Choose your method**: Use a [voice platform integration](/documentation/integrations/overview) or send calls via the [API/SDK](/documentation/observability/overview#how-calls-get-into-roark)
2. **Ensure calls include the phone number**: Your call data must include the phone number that participated in the simulation so Roark can match it
3. **Enable the toggle when creating a run**: Turn on **Collect selected metrics from your agent's side of the call** in the Metrics block and pick the source for each metric

> [!TIP]
>
> If you're using a voice platform integration like [Vapi](/documentation/integrations/vapi), [Retell](/documentation/integrations/retell), or [LiveKit](/documentation/integrations/livekit), calls are typically sent to Roark automatically, meaning enriched simulations work out of the box once you enable the toggle.

***

## Best Practices

> [accordion] Include Tool Invocations in Your Calls
>
> To get the most out of enriched simulations, send [tool invocations](/documentation/tool-invocations) with your call data. This enables live-sourced metrics that verify whether your agent executed the correct actions, not just whether it generated the right words.

> [accordion] Send Calls Promptly
>
> Roark waits up to 15 minutes for your agent's side before falling back to the simulated recording. Sending calls shortly after they end keeps live-sourced metrics complete and results fast.

> [accordion] Use Consistent Phone Numbers
>
> Make sure the phone number in your call data matches the number your agent actually used during the simulation. Mismatched or reformatted numbers will prevent matching.

> [accordion] Leverage OTEL Traces
>
> If your agent supports [OpenTelemetry tracing](/documentation/observability/traces), include trace data with your calls. This gives you full visibility into your agent's internal decision-making during simulations.

***

## Next Steps

> [card] [Tool Invocations](/documentation/tool-invocations)
>
> Learn how to submit tool calls with your call data

> [card] [Traces](/documentation/observability/traces)
>
> Set up OpenTelemetry tracing for deeper analysis

> [card] [Integrations](/documentation/integrations/overview)
>
> Connect your voice platform for automatic call ingestion

> [card] [Identifying Simulations](/documentation/simulation-testing/identifying-simulations)
>
> Understand how simulation calls are identified


# Environments
Source: https://docs.roark.ai/documentation/simulation-testing/environments

The background noise a simulated customer calls from, and how loud it plays

An environment is the acoustic setting a simulated customer calls from: a **noise bed** (an office, a coffee shop, a car) and the **level** it plays at underneath the customer's voice. Every [customer flow](/documentation/simulation-testing/customer-flows) variant runs in exactly one environment, so the same conversation can be tested from a quiet line and from a noisy street without changing anything else.

Environments are a project-level catalogue, shared across flows, the same way [personas](/documentation/simulation-testing/personas) are. They work in both authoring modes:

* In an [Improv flow](/documentation/simulation-testing/customer-flows#improv-mode), the happy path and each edge case carry their own Environment chip.
* In a [Scripted flow](/documentation/simulation-testing/customer-flows#scripted-mode), every path derived from the graph is a variant with the same chip in the right-hand rail, and **Add variant for path** runs the same path again in a different environment.

## Roark presets

Every project sees eight presets, one per noise bed, all playing at the default level:

| Preset | Noise bed |
| - | - |
| Quiet line | Nothing. The customer calls from a silent room. |
| Office | Open-plan office ambience |
| Coffee shop | Busy café chatter |
| City street | Outdoor city street noise |
| Driving | In-car road and engine noise |
| Airport | Terminal announcements and crowds |
| Children playing | Children in the background at home |
| Thunderstorm | Rain and thunder |

Presets are read-only. You cannot rename, re-level or delete them, and they keep their level on every deploy. Everything else about an environment, you own.

## Noise level

The level is how loud the bed plays relative to the customer's voice. In the flow editor it is a slider from 0% to 100%; over the API it is `backgroundNoiseVolume`, a gain from `0` to `1`.

* **10% is the default** and what every preset plays at. Roark first normalises each noise asset to a fixed loudness, then applies the level, so 10% lands roughly 20 dB under phone speech: clearly audible in the background without competing with the caller.
* **100% plays the noise as loud as the customer.** Use it to find out where your agent's endpointing and transcription break down, not as a realistic call.
* The level is a linear gain, not a perceptual scale. 50% is close to speech level, not half as loud as 100%.

## Setting the environment in the flow editor

Each variant carries a **Persona · Environment · Variables** chip strip: under the customer setup in an Improv flow, and on each path's variant card in the rail of a Scripted flow. The Environment chip opens a panel that lists the Roark presets and, once you have some, **Your environments**.

**1. Pick a bed**

Click a preset chip. The variant runs in that environment immediately, at the default level.

**2. Tune the level**

Drag **Noise level**. As soon as it leaves the preset's own level, the panel offers **Save as your environment** with a name filled in for you, such as "Office (30% noise)". Overwrite the name if you want, then click **Save environment**. Roark creates a project environment with that bed and level and binds the variant to it. The preset itself is untouched.

**3. Reuse or adjust it later**

Your environments appear as chips on every variant in the project. Pick one and the same panel lets you rename it, change its level or delete it. If you drag a preset to a level you have already saved, the panel offers **Use it** instead of creating a duplicate.

![A flow's happy path with the Environment chip open: the Roark presets with Office selected, and the Noise level slider at the default 10%](https://mintcdn.com/roark/LnwElYQN5nlnU2Kr/images/simulations/environment-panel.png?fit=max\&auto=format\&n=LnwElYQN5nlnU2Kr\&q=85\&s=a1edd5d4e003819f38d635737d6e30c4)

A few rules the editor enforces:

* **Edge cases inherit the happy path's environment** until you pick one for them; the chip reads "Office (inherited)". Choose **Inherit** to go back to inheriting.
* **A change to one of your environments applies everywhere it is picked.** It is a shared row, not a per-variant copy. To give one variant a different level, save a second environment.
* **An environment in use cannot be deleted.** Point the variants that use it at another environment first; the panel tells you how many there are.
* **Runs keep what they were built with.** Roark snapshots the environment when a run is created, so editing or deleting it later never changes a run already in flight or its report.

> [!TIP]
>
> Two edge cases with the same persona and brief but different environments are a quick way to compare how your agent copes with the same customer in a quiet room and a noisy one. In the **Add edge case** menu, **From an environment** creates one in a single click.

## Config as code

In an [Improv flow defined as YAML](/documentation/config-as-code/flows#improv-type-improv), the happy path and any edge case reference an environment by its display name, exactly as it appears in the panel:

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
kind: flow
type: improv
name: frustrated-rebooking
agents: [frontdesk]
happyPath:
  persona: frustrated-caller
  environment: Office (30% noise)
  prompt: You call to rebook the cleaning that was cancelled on you.
edgeCases:
  - name: from-the-car
    environment: Driving
```

The name has to exist in the project before the config is applied. Presets always do; create your own environments in the editor or over the API first.

A [Scripted flow in YAML](/documentation/config-as-code/flows#scripted-type-scripted) describes the graph only. Its variants are derived from the paths when the config is applied and start on the Quiet line preset, so set their environments in the editor afterwards.

## API

Manage environments programmatically to keep them alongside your test suite or to set up a project from a script. Full request and response schemas are in the [API reference](/api-reference/introduction).

* **List environments**: `GET /v1/simulation/environment` returns your environments and the Roark presets
* **Get an environment**: `GET /v1/simulation/environment/{environmentId}`
* **Create an environment**: `POST /v1/simulation/environment`
* **Update an environment**: `PUT /v1/simulation/environment/{environmentId}`
* **Delete an environment**: `DELETE /v1/simulation/environment/{environmentId}`

Create one with a bed and a level. `backgroundNoiseVolume` is optional and defaults to `0.1`:

```typescript Node theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import Roark from '@roarkanalytics/sdk'

const client = new Roark({ bearerToken: process.env.ROARK_API_BEARER_TOKEN })

const environment = await client.simulationEnvironment.create({
  name: 'Busy call centre',
  description: 'Open-plan office at peak hours',
  backgroundNoise: 'OFFICE',
  backgroundNoiseVolume: 0.3,
})

console.log(environment.data.id)
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import os
from roark_analytics import Roark

client = Roark(bearer_token=os.environ["ROARK_API_BEARER_TOKEN"])

environment = client.simulation_environment.create(
    name="Busy call centre",
    description="Open-plan office at peak hours",
    background_noise="OFFICE",
    background_noise_volume=0.3,
)

print(environment.data.id)
```

```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/simulation/environment \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Busy call centre",
    "description": "Open-plan office at peak hours",
    "backgroundNoise": "OFFICE",
    "backgroundNoiseVolume": 0.3
  }'
```

The response is the environment you can now reference from a customer flow variant's `environmentId`:

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "data": {
    "id": "6f1c2b8e-4d21-4f0a-9d3b-2b6c1e9f0a11",
    "name": "Busy call centre",
    "description": "Open-plan office at peak hours",
    "backgroundNoise": "OFFICE",
    "backgroundNoiseVolume": 0.3,
    "createdAt": "2026-09-08T18:20:11.512Z",
    "updatedAt": "2026-09-08T18:20:11.512Z"
  }
}
```

Updates change only the fields you send:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X PUT https://api.roark.ai/v1/simulation/environment/6f1c2b8e-4d21-4f0a-9d3b-2b6c1e9f0a11 \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "backgroundNoiseVolume": 0.5 }'
```

`backgroundNoise` accepts `NONE`, `OFFICE`, `COFFEE_SHOP`, `CITY`, `DRIVING`, `AIRPORT`, `CHILDREN_PLAYING` or `THUNDERSTORM`.

Errors to handle:

| Status | When |
| - | - |
| `400` | Invalid body, for example a level outside `0` to `1`, or an update with no fields |
| `403` | The environment is a Roark preset, or the API key lacks the simulation permission |
| `404` | Unknown or already deleted environment id |
| `409` | A delete while a live customer flow variant still uses the environment. Move those variants first. |

Deleting is a soft delete: the environment disappears from lists and cannot be picked for new runs, but runs already built keep their snapshot of it.


# Identifying Simulations
Source: https://docs.roark.ai/documentation/simulation-testing/identifying-simulations

Verify simulation calls and retrieve simulation details from your agent

## Overview

When your agent receives calls during simulation testing, you may need to verify whether a call is from a Roark simulation and retrieve details about the specific test being executed. Roark provides a dedicated API endpoint to identify simulations based on phone numbers and call timing.

## Identifying Simulation Calls

### Using the Simulation Job Lookup API

To verify if a call is coming from a simulated Roark agent or to retrieve simulation details, use the [Lookup Simulation Job by Phone Numbers API](https://docs.roark.ai/api-reference/simulation-job/lookup-by-phone-number).

This API allows you to:

* Verify if an incoming or outgoing call is part of a simulation test
* Retrieve the specific simulation job details
* Track which persona and customer flow are being tested in the simulation (the response includes the persona name as `simulationPersonaName`)

### API Parameters

The API requires specific parameters to identify the simulation:

| Parameter | Description | Usage |
| :- | :- | :- |
| **roarkAgentPhoneNumber** | The Roark agent's phone number | See direction-specific details below |
| **callStartedAt** | When the call was initiated | Must be within the simulation's active window |

### Understanding the `roarkAgentPhoneNumber` parameter

The `roarkAgentPhoneNumber` parameter depends on the call direction:

**For Inbound Calls**:

* Roark initiates the call from a dynamically provisioned number
* Your agent receives the call from this number
* In this case, `roarkAgentPhoneNumber` is the number that dials your agent

**For Outbound Calls**:

* Roark provisions a number for each test case in a simulation
* When a simulation for a test case starts, this number is either sent via an HTTP request for your agent to trigger the outbound call, or manually dialed at runtime based on the number shown on the dashboard. See [this page](/documentation/simulation-testing/inbound-vs-outbound#triggering-an-outbound-call) for more information.
* The Roark simulation agent receives this call and starts the simulation
* In this case, `roarkAgentPhoneNumber` is the number you receive via the HTTP request, or the number displayed on the simulation run page for manual calls

### Understanding the `callStartedAt` parameter

While a phone number is assigned to only one simulation at a time, once the simulation ends it may be reused for other simulations. As such, the `callStartedAt` timestamp is critical for matching the correct simulation:

* **Timing Window**: The call must occur within the simulation's execution timeframe. Any time between the simulation test case starting and the call ending is valid.
* **Default Value**: Current date. This is useful when calling the endpoint to match a call in real-time (examples 1 and 2 below). For historical calls, you should pass the call creation time to match the correct call.
* **Valid examples**:
  1. Using **current date** as `callStartedAt` whenever you receive an inbound call from Roark in your agent's inbound webhook (dependent on the provider)
  2. Using **current date** as `callStartedAt` whenever you receive an HTTP request to trigger an outbound call for a Roark simulation.
  3. Using any time between the above two options and the end time of the call.
* **Format**: ISO 8601 timestamp (e.g., `2024-01-15T14:30:00Z`)

> [!NOTE]
>
> The `callStartedAt` parameter helps distinguish between multiple simulations
> that might use the same phone numbers at different times.

***

## Best Practices

### Integration Recommendations

> [accordion] Link calls on your platform with Roark's simulation details
>
> Use the data returned by the API to tag calls on your platform as test calls
> and link directly to Roark's test results

> [accordion] Use for Contextual Information
>
> Leverage persona details to add context to your agent

> [accordion] Separate Test Traffic
>
> Use simulation identification to segregate test data from production metrics

> [accordion] Add Conditional Logic
>
> Implement test-specific behaviors when handling identified simulation calls

***

## Next Steps

> [card] [API Reference](https://docs.roark.ai/api-reference/simulation-job/lookup-by-phone-number)
>
> View the complete API documentation

> [card] [Inbound vs Outbound](/documentation/simulation-testing/inbound-vs-outbound)
>
> Understand call direction differences

> [card] [Run Plans](/documentation/simulation-testing/run-plans)
>
> Configure your run plans

> [card] [Running Simulations](/documentation/simulation-testing/running-simulations)
>
> Execute your simulation tests


# Inbound vs Outbound
Source: https://docs.roark.ai/documentation/simulation-testing/inbound-vs-outbound

Understanding simulation testing directions

## Overview

When running simulations in Roark, there are two fundamental testing directions based on who initiates the call. This affects how your tests execute, what you can test, and how you configure your simulations.

> [!NOTE]
>
> You don't choose a direction when building a run. It's derived from the
> agent endpoints you select. Each endpoint has a direction (Inbound, Outbound,
> or "In + out"), and a run is outbound only when every selected endpoint is
> outgoing-only. Otherwise the run executes inbound.

***

## Inbound Testing

In inbound testing, **Roark calls your agent** to simulate incoming customer calls.

### How It Works

1. You select an agent endpoint that accepts incoming calls (direction Inbound or "In + out")
2. Roark initiates calls to your agent
3. The simulated customer (persona) interacts with your agent
4. Your agent responds as it would to a real incoming call

### Characteristics

* **Immediate execution** - Tests run as soon as triggered
* **Multiple iterations** - Can repeat the same test case up to 100 times
* **High control** - Roark controls timing and execution
* **Synchronous results** - Get results immediately after completion

### Use Cases

Perfect for testing:

* **Customer service lines** - Support hotlines, help desks
* **IVR systems** - Automated phone menus
* **Receptionist agents** - Virtual assistants answering calls
* **Emergency response** - Crisis hotlines, urgent support
* **Inbound sales** - Agents handling incoming inquiries

### Configuration

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
Endpoint direction: Inbound (or "In + out")
Agent Endpoint: +1-555-0100 or sip:agent@domain.com
Iterations: 1-100 (configurable)
Concurrency: Default 5 parallel calls, capped by your account quota
```

***

## Outbound Testing

In outbound testing, **your agent calls Roark** to simulate outbound campaigns. Roark runs a plan outbound when every selected agent endpoint is outgoing-only.

### How It Works

1. Roark provisions a phone number for each test case in your run
2. Your agent can start the call either by exposing an outbound call endpoint that Roark will trigger automatically, or by directly calling the provisioned number shown on the run detail page.
3. Roark answers as the simulated customer (persona)
4. Your agent conducts the outbound call against the customer flow you attached

#### Triggering an outbound call

In an outbound simulation, your agent is responsible for starting the simulated call. This can be done in two ways:

#### HTTP Endpoint (Suggested)

By setting up an outbound call HTTP request, Roark can automatically start your outbound tests for you - giving you the benefits of scheduling and concurrency control.

1. Set up an HTTP request on the agent endpoint that will be used in the simulation

   ![Configuring an outbound call HTTP request on an agent endpoint: a POST to https://api.yourvoice.example/calls with Authorization and Content-Type headers and a JSON body whose 'to' field is set to the phoneNumberToDial template variable](https://mintcdn.com/roark/yUzvMIE3_BYBHPDx/images/simulation-testing/outbound-http.png?fit=max\&auto=format\&n=yUzvMIE3_BYBHPDx\&q=85\&s=8a12740170e48d7fc2b7edf73cd2bd58)

2. Whenever a simulation is started for this agent endpoint, Roark will make the specified HTTP request passing in the provisioned phone number as `{{phoneNumberToDial}}`.

3. This HTTP request is expected to trigger your agent to start an outbound call by dialing `{{phoneNumberToDial}}`.

This automates the workflow of running outbound tests - Roark will handle generating the number and calling your endpoint, and your system simply initiates the call.

> [!NOTE]
>
> Template variables such as `{{ phoneNumberToDial }}` can be used in the `URL`,
> `body`, and `headers`.

#### Provider Examples

Below are minimal example configurations for common voice AI providers. The `{{phoneNumberToDial}}` template variable will be replaced with the provisioned Roark number at runtime.

> [tab] VAPI
>
> **URL:**
>
> ```
> POST https://api.vapi.ai/call
> ```
>
> **Headers:**
>
> ```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> {
>   "Authorization": "Bearer <YOUR_VAPI_API_KEY>",
>   "Content-Type": "application/json"
> }
> ```
>
> **Body:**
>
> ```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> {
>   "customer": {
>     "number": "{{phoneNumberToDial}}"
>   },
>   "assistantId": "<YOUR_VAPI_ASSISTANT_ID>",
>   "phoneNumberId": "<YOUR_VAPI_PHONE_NUMBER_ID>"
> }
> ```
>
> See the [VAPI API documentation](https://docs.vapi.ai/api-reference/calls/create-call) for additional options.

> [tab] Retell
>
> **URL:**
>
> ```
> POST https://api.retellai.com/v2/create-phone-call
> ```
>
> **Headers:**
>
> ```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> {
>   "Authorization": "Bearer <YOUR_RETELL_API_KEY>",
>   "Content-Type": "application/json"
> }
> ```
>
> **Body:**
>
> ```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> {
>   "from_number": "<YOUR_RETELL_PHONE_NUMBER>",
>   "to_number": "{{phoneNumberToDial}}",
>   "override_agent_id": "<YOUR_RETELL_AGENT_ID>"
> }
> ```
>
> See the [Retell API documentation](https://docs.retellai.com/api-references/create-phone-call) for additional options like `metadata`, `retell_llm_dynamic_variables`, and agent overrides.

> [tab] ElevenLabs
>
> **URL:**
>
> ```
> POST https://api.elevenlabs.io/v1/convai/batch-calling/submit
> ```
>
> **Headers:**
>
> ```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> {
>   "xi-api-key": "<YOUR_ELEVENLABS_API_KEY>",
>   "Content-Type": "application/json"
> }
> ```
>
> **Body:**
>
> ```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> {
>   "call_name": "<YOUR_BATCH_CALL_NAME>",
>   "agent_id": "<YOUR_ELEVENLABS_AGENT_ID>",
>   "recipients": [
>     {
>       "phone_number": "{{phoneNumberToDial}}"
>     }
>   ]
> }
> ```
>
> See the [ElevenLabs API documentation](https://elevenlabs.io/docs/api-reference/batch-calling/create) for additional options.

#### Manual Calls

If an HTTP request is not configured, Roark displays the provisioned phone number for each test case on the run detail page while the call shows as "Waiting for call". You can then manually trigger your agent to dial that number.

![Live run detail for an outbound simulation: test cases shown as 'Waiting for a call to' their provisioned Roark phone number while the run waits for the agent to dial](https://mintcdn.com/roark/yUzvMIE3_BYBHPDx/images/simulation-testing/waiting-for-call.png?fit=max\&auto=format\&n=yUzvMIE3_BYBHPDx\&q=85\&s=4b53ea15eb2acb0b1aaac6fc8168d5f2)

Alternatively, you can trigger calls programmatically using our APIs. This is useful when you need full control over when simulations run within your own orchestration logic.

**1. Trigger the simulation run**

Start a simulation by calling the [Run a Simulation](/api-reference/simulation/run-a-simulation) endpoint with your run plan ID in the body.

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
POST /v1/simulation/run
```

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{ "planId": "<plan-id>" }
```

This returns a `simulationRunPlanJobId` for tracking the executed simulation run plan job.

**2. Fetch the simulation run plan job**

Retrieve the job details using the [Get Simulation Plan Job](/api-reference/simulation-run-plan-job/get-simulation-plan-job) endpoint.

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
GET /v1/simulation/plan/job/{jobId}
```

The response includes a list of individual simulation jobs.

**3. Dial pending jobs**

Look for jobs with `status: "WAITING_FOR_OUTBOUND_CALL"` (shown as "Waiting for call" in the dashboard). For each of these, have your agent dial the `roarkPhoneNumber` provided in the job object to start that simulation.

**4. Repeat for queued jobs**

Depending on your run settings, some jobs may be queued while others are in progress. Monitor when each call completes, then repeat steps 2–3 until the [Get Simulation Plan Job](/api-reference/simulation-run-plan-job/get-simulation-plan-job) endpoint returns `status: "COMPLETED"`.

> [!TIP]
>
> While the programmatic approach offers flexibility, we recommend using HTTP
> requests instead. Roark will automatically trigger calls at the right time,
> handling queued simulations and concurrency limits for you.

> [!WARNING]
>
> For both HTTP requests and manual calls, Roark expects the outbound call to be
> made by the number set in your agent endpoint. Calls made to the provisioned
> number from numbers other than the one defined in your agent endpoint will be
> ignored.

### Characteristics

* **Asynchronous execution** - Tests wait for your agent to call
* **Single iteration** - Outbound runs are always one iteration per test case
* **Agent-controlled timing** - Your system decides when to call
* **Waiting state** - Test cases show "Waiting for call" until your agent dials

### Use Cases

Perfect for testing:

* **Sales campaigns** - Cold calling, lead follow-up
* **Appointment reminders** - Medical, service appointments
* **Survey calls** - Customer satisfaction, market research
* **Collections** - Payment reminders, account management
* **Notifications** - Alert calls, status updates

### Configuration

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
Endpoint direction: Outbound (all selected endpoints outgoing-only)
Phone Numbers: Provisioned by Roark
Iterations: 1 (fixed)
Timeout: Configurable (default 24 hours)
```

## Key Differences

| Aspect | Inbound | Outbound |
| :- | :- | :- |
| **Who calls who** | Roark → Your Agent | Your Agent → Roark |
| **Execution** | Immediate | When agent calls |
| **Iterations** | Multiple supported (up to 100) | One per test case |
| **Use case** | Customer service | Sales/outreach |
| **Timing control** | Roark controls | You control |
| **Phone numbers** | Use your existing | Roark provisions |
| **Concurrency** | Default 5, capped by your quota | Limited by numbers |
| **Results** | Immediate | After agent calls |

***

## Choosing the Right Direction

Since direction follows from your agent endpoints, "choosing" a direction means selecting the right endpoints for the behavior you want to test.

### Test Inbound When:

* Testing customer service flows
* You need immediate results
* Running high-volume tests
* Testing response to incoming requests
* Validating IVR flows

### Test Outbound When:

* Testing sales or marketing campaigns
* Your agent initiates contact
* Testing dialer integrations
* Validating outreach scripts
* Testing follow-up workflows

***

## Technical Considerations

### Inbound Testing

* Ensure your agent can handle concurrent calls
* Configure appropriate rate limits
* Test your agent's scaling capabilities
* Monitor response times under load

### Outbound Testing

* Plan for phone number provisioning time
* Set appropriate timeouts for pending tests
* Configure your dialer to call Roark numbers
* Handle failed call attempts gracefully

***

## Best Practices

> [accordion] Start with Inbound
>
> Begin with inbound testing for immediate feedback before moving to outbound

> [accordion] Match Production Patterns
>
> Choose endpoints that match how your agent operates in production

> [accordion] Test Both Directions
>
> If your agent handles both, test both directions for complete coverage

> [accordion] Monitor Phone Number Usage
>
> For outbound, track which numbers have been called to avoid duplicates

> [accordion] Set Realistic Timeouts
>
> Configure appropriate timeouts for outbound tests based on your calling
> patterns

## Next Steps

> [card] [Create Run Plans](/documentation/simulation-testing/run-plans)
>
> Build test suites around your agent endpoints

> [card] [Configure Personas](/documentation/simulation-testing/personas)
>
> Set up customer profiles for testing

> [card] [Running Simulations](/documentation/simulation-testing/running-simulations)
>
> Execute your inbound or outbound tests

> [card] [Customer Flows](/documentation/simulation-testing/customer-flows)
>
> Author the conversations your simulations follow


# Overview
Source: https://docs.roark.ai/documentation/simulation-testing/overview

Pressure-test your voice AI agents before your customers do

> [!NOTE]
>
> **Define in code, or click through the UI.** The fastest path is [config as code](/documentation/config-as-code/overview): define your agents, flows, personas, and the run plan itself in TypeScript or YAML, then apply and run with the [CLI](/documentation/sdks/cli). Prefer clicking? Build the same resources and launch runs in the dashboard. Both manage the same resources.

## Quickstart

Define an agent and a plan, apply them, and run it, a few lines from nothing to a scored simulation:

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineAgent, defineSimulationPlan, defineConfig } from '@roarkanalytics/sdk/lib/config';

const frontdesk = defineAgent({
  name: 'frontdesk',
  endpoints: [{ value: '+15551234567', direction: 'OUTGOING' }],
});

export default defineConfig({
  resources: [
    frontdesk,
    defineSimulationPlan({
      name: 'smoke-test',
      direction: 'OUTBOUND',
      agentEndpoints: [{ agent: 'frontdesk', value: '+15551234567' }],
      flows: [{ system: 'sf-screener-apple' }], // a built-in flow; swap for your own
      metrics: ['call_screening_handling_score'],
      maxDurationSeconds: 300,
    }),
  ],
});
```

```bash Apply + run theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply roark.config.ts                        # create the agent + plan
roark simulation run --data '{"planId":"<plan-id>"}'      # run it (find the id with `roark simulation plan list`)
```

> [card] [Full walkthrough: Run a simulation](/documentation/config-as-code/run-a-simulation)
>
> Define, apply, run, read the pass/fail, and gate CI on it, step by step.

### Already have an agent and flow? Run one ad-hoc

Run against resources you already created (by id), without a saved plan:

> [tab] Node
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> const run = await client.simulation.run({
>   plan: {
>     direction: 'INBOUND',
>     maxSimulationDurationSeconds: 300,
>     agentEndpoints: [{ id: '<endpoint-id>' }],
>     flows: [{ id: '<flow-id>', happyPath: true }],
>     metrics: [{ slug: 'task_completion' }],
>   },
> })
> console.log(run.data.simulationRunPlanJobId)
> ```

> [tab] Python
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> run = client.simulation.run(plan={
>     "direction": "INBOUND",
>     "maxSimulationDurationSeconds": 300,
>     "agentEndpoints": [{"id": "<endpoint-id>"}],
>     "flows": [{"id": "<flow-id>", "happyPath": True}],
>     "metrics": [{"slug": "task_completion"}],
> })
> print(run.data.simulation_run_plan_job_id)
> ```

> [tab] CLI
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> roark simulation run --data '{
>   "plan": {
>     "direction": "INBOUND",
>     "maxSimulationDurationSeconds": 300,
>     "agentEndpoints": [{ "id": "<endpoint-id>" }],
>     "flows": [{ "id": "<flow-id>", "happyPath": true }],
>     "metrics": [{ "slug": "task_completion" }]
>   }
> }'
> ```

Starting from scratch? Follow the [full zero-to-first-simulation quickstart](/documentation/getting-started/introduction#your-first-simulation).

## What is simulation testing?

Simulation testing lets Roark act as your customers. You describe the conversations your agents should handle (who's calling, what they want, how the call should go) and Roark places real calls (or chat sessions) against your agents, grades every conversation, and reports pass/fail results.

Everything revolves around **plans** and **runs**. A plan is a reusable test suite; a run is one execution of it. Each run is labelled `SR-{n}` and moves through **Queued → Running → Completed** (or **Failed** / **Cancelled**).

A plan composes four things:

| Building block | What it defines |
| - | - |
| **Agent targets** | Which agents to test and how to reach each one: an agent plus an endpoint (Phone, WebRTC, LiveKit, WebSocket, ElevenLabs, Google CES, Kore) with a direction (inbound, outbound, or both) |
| **Customer flows** | The conversations to exercise: each flow attached with a variant selection (all variants, the happy path only, or specific variants) |
| **Metrics & checks** | What to measure on every conversation, with pass/fail thresholds that decide the run's verdict |
| **Run settings** | Iterations, concurrency, execution mode, and end conditions like max duration and silence timeout |

You'll find everything under the **Simulate** section of the sidebar: **Simulations** (the runs hub), **Customer flows**, and **Personas**.

***

## Core concepts

### Customer flows

A [customer flow](/documentation/simulation-testing/customer-flows) describes one type of conversation your agent should handle. Author it in **Improv** mode (write a free-text customer brief and the simulator improvises a fresh conversation each run) or **Scripted** mode (author the conversation step by step on a graph canvas, ideal for IVR menus and deterministic routing).

Every flow has a default variant (the **happy path**) plus optional **edge cases** that vary the persona, environment, or setup. The flow's **agent expectations** are the graded contract: LLM-checked instructions the agent is evaluated against on every variant.

### Personas

A [persona](/documentation/simulation-testing/personas) is who's calling: language, accent, gender, base emotion, and how they speak. Each flow variant pins a persona, and run plans can override it per attachment, for example, fanning one flow across language personas in a multilingual run.

### Plans and runs

A [plan](/documentation/simulation-testing/run-plans) is the saved suite; a [run](/documentation/simulation-testing/running-simulations) is one execution. Run a plan on demand, put it on a [schedule](/documentation/simulation-testing/schedules), or skip saving entirely. Unchecked **Save as plan** gives you a one-off run. [Templates](/documentation/simulation-testing/templates) like Red teaming, Multilingual, and Load testing pre-configure metrics, checks, and flow sourcing so you start from a goal instead of a blank page.

### Metrics and checks

Every run scores its conversations with [metrics](/documentation/metrics/overview): system metrics out of the box, plus [custom metrics](/documentation/metrics/custom-metrics) you build in [Studio](/documentation/metrics/studio). Pass/fail [thresholds](/documentation/metrics/thresholds) turn metrics into checks, and the run report's verdict is the pass rate across every check on every conversation.

***

## How a run comes together

**1. Pick agents to test**

From **Simulations → New Run**, select agents and the endpoints to reach them on. Run direction is derived from your endpoints: inbound means Roark calls your agent; outbound means your agent calls Roark.

**2. Choose a template**

Start from a goal (Flow adherence, Red teaming, Conversation quality, Multilingual, Load testing, Tool call accuracy) or Blank. Each template ships preset metrics and pass/fail checks, then attach the customer flows the run should exercise.

**3. Tune the details**

The **Advanced** section covers the metric set, iterations and concurrency, and end conditions (max duration, silence timeout, end-call phrases). Check **Save as plan** to keep the configuration as a re-runnable suite.

**4. Run and review**

Run now or schedule it. While the run is live you can watch calls in progress and listen in; once it settles, the report shows the verdict, per-metric results, run-over-run comparison, and every conversation's transcript.

![The Create a new run page: numbered section 01 Template with a grid of goal templates (Flow adherence selected alongside Blank, Red teaming, Conversation quality, and more), 02 Agents to test showing the Front Desk agent reachable on an inbound phone endpoint, and a Run simulation split button in the footer action bar.](https://mintcdn.com/roark/93k2kGpZVUUbTXSq/images/simulation-testing/new-run.png?fit=max\&auto=format\&n=93k2kGpZVUUbTXSq\&q=85\&s=46052cdb78af86678e6f43698d482d09)

> [!NOTE]
>
> The REST API and SDKs keep the older naming: customer flows are still `scenarios` in request fields, and run plans are managed via the same endpoints as before. Code examples throughout these docs reflect the API's names.

***

## Common use cases

* **Pre-deployment testing**: exercise every flow's happy path and edge cases before an agent goes live
* **Regression testing**: save a plan, re-run it after every change, and compare runs side by side in the report's Comparison tab
* **Continuous monitoring**: schedule plans hourly, daily, or weekly to catch regressions before you do
* **Safety and robustness**: the Red teaming template runs adversarial flows against your agent; Load testing checks behavior under concurrent call volume

***

## Where to go next

> [card] [Customer flows](/documentation/simulation-testing/customer-flows)
>
> Author the conversations your agents should handle: Improv briefs or Scripted graphs

> [card] [Personas](/documentation/simulation-testing/personas)
>
> Define the callers your simulations run as

> [card] [Templates](/documentation/simulation-testing/templates)
>
> Start a run from a goal with preset metrics and checks

> [card] [Plans & schedules](/documentation/simulation-testing/run-plans)
>
> Build reusable suites and run them on a cadence

> [card] [Running simulations](/documentation/simulation-testing/running-simulations)
>
> Launch runs, monitor them live, and read the report

> [card] [Best practices](/documentation/simulation-testing/best-practices)
>
> Strategies for building an effective test suite

For chat-based agents, see [Chat simulations](/documentation/simulation-testing/chat-simulations). To score metrics from your agent's side of the call, see [Enriched simulations](/documentation/simulation-testing/enriched-simulations).


# Personas
Source: https://docs.roark.ai/documentation/simulation-testing/personas

The callers your simulations run as

## Quickstart

A persona is the synthetic caller a simulation runs as: a reusable bundle of language, accent, gender, base emotion, and speaking style that you pin to a [customer flow](/documentation/simulation-testing/customer-flows) variant.

Define a persona [in code](/documentation/config-as-code/personas) (recommended), or build it in the dashboard.

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { readFileSync } from 'node:fs';
import { definePersona, defineConfig } from '@roarkanalytics/sdk/lib/config';

const busyParent = definePersona({
  name: 'busy-parent',
  displayName: 'Sarah Nguyen',
  language: 'EN',
  accent: 'US',
  gender: 'FEMALE',
  baseEmotion: 'RUSHED',
  backstoryPrompt: readFileSync('./prompts/busy-parent.md', 'utf8'),
});

export default defineConfig({
  resources: [busyParent],
});
```

```yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: persona
name: busy-parent
displayName: Sarah Nguyen
language: EN
accent: US
gender: FEMALE
baseEmotion: RUSHED
backstoryPrompt: file://prompts/busy-parent.md
```

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply roark.config.ts
```

### Create it with the SDK

Reuse a built-in persona, or create your own:

> [tab] Node
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> // Reuse a built-in persona (list includes Roark's starter set)
> const personas = await client.simulationPersona.list({ limit: 50 })
>
> // Or create a custom one
> const persona = await client.simulationPersona.create({
>   name: 'Busy Parent',
>   language: 'EN',
>   accent: 'US',
>   gender: 'FEMALE',
> })
> ```

> [tab] Python
>
> ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> personas = client.simulation_persona.list(limit=50)
>
> persona = client.simulation_persona.create(
>     name="Busy Parent", language="EN", accent="US", gender="FEMALE",
> )
> ```

> [tab] CLI
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> roark simulation persona list --limit 50
> roark simulation persona create --name "Busy Parent" --language EN --accent US --gender FEMALE
> ```

## Overview

A persona is the caller a simulation runs as. It bundles who the customer is (language, accent, gender, base emotion, and how they speak) into a reusable caller that you pin to a [customer flow](/documentation/simulation-testing/customer-flows) variant. Roark ships a starter set of **Built-in** personas; add your own **Custom** personas for the callers you care about.

Open **Personas** in the sidebar to see every persona in your project. From the hub you can search, create new personas, and edit or delete the custom ones.

![The Personas hub: a searchable roster of personas, each with a Built-in or Custom badge, language, accent, gender, and temperament tags, and a New persona button](https://mintcdn.com/roark/Xs4hVQUhEDN2Pyh8/images/simulation-testing/personas-hub.png?fit=max\&auto=format\&n=Xs4hVQUhEDN2Pyh8\&q=85\&s=73fda4b382ae46a236c3d0811aee0af7)

## Persona fields

Click **New persona** to open the persona dialog. Every field except Name is optional or has a sensible default.

| Field | Options | Default | Notes |
| :- | :- | :- | :- |
| **Name** | Free text | N/A | Required. Use something descriptive, e.g. "Returning client: Sarah" |
| **Display name** | Free text | Name | Shown in pickers and variant chips. Defaults to the name |
| **Description** | Free text | N/A | One or two sentences on who this caller is (hub dialog only) |
| **Language** | 18 languages | English | See the full list below |
| **Accent** | 23 accents | American | Includes American (Southern) as a distinct option |
| **Gender** | Female, Male, Neutral | Female | Affects the synthesized voice |
| **Base emotion** | Neutral, Cheerful, Frustrated, Skeptical, Rushed, Distracted, Confused | Neutral | The caller's underlying emotional tone |
| **Speech pace** | Very slow, Slow, Normal, Fast, Very fast | Normal | How quickly the persona speaks |
| **Clarity** | Clear, Vague, Rambling | Clear | How directly they express themselves |
| **Response timing** | Quick, Normal, Relaxed | Normal | How quickly the persona responds to pauses in conversation |
| **Disfluencies** | On / Off | Off | Natural ums, ahs, false starts. Off means the persona speaks cleanly |

> [accordion] Language options
>
> English, Spanish, French, German, Italian, Portuguese, Dutch, Arabic, Greek,
> Hebrew, Hindi, Indonesian, Malay, Tagalog, Thai, Japanese, Chinese, Turkish

> [accordion] Accent options
>
> American, American (Southern), British, Australian, New Zealand, Indian,
> Spanish, French, German, Italian, Portuguese, Dutch, Greek, Arabic, Hebrew,
> Indonesian, Malaysian, Filipino, Singaporean, Hong Kong, Thai, Japanese,
> Turkish

> [!NOTE]
>
> Background noise is not a persona setting. It's configured per flow variant
> via the **Environment** chip (Silent, Office, Coffee shop, City street,
> Driving, Airport, Children playing, Thunderstorm). See [Customer
> flows](/documentation/simulation-testing/customer-flows).

## Built-in vs Custom

Every persona has a source badge:

* **Built-in**: the starter set Roark ships with each project. Built-in personas are read-only in the hub: they have no edit or delete actions.
* **Custom**: personas you create. Fully editable and deletable.

You can still start from a Built-in persona: load it in the inline persona editor inside a flow, tweak it, and save. Roark shows the note "System persona: saving creates an editable copy." and creates a Custom copy with your changes, leaving the original intact.

> [!WARNING]
>
> Deleting a persona doesn't break existing flows silently, but customer-flow
> variants pinned to it will need a new persona.

## Pinning personas to flow variants

Personas do their work inside customer flows. Each variant in a flow carries a chip strip (**Persona · Environment · Variables**) and the Persona chip shows which caller that variant runs as.

* The flow's **default variant** (the happy path) must always have a concrete persona. New flows are pre-seeded with the Built-in **Polite First-Time Caller** persona (or your project's first persona if that one isn't available).
* **Non-default variants inherit** the default variant's persona automatically. The chip shows the name in muted italics, e.g. "Sarah (inherited)". Pin a different persona on a variant to override the inheritance for that variant only.

![An edge-case flow variant card, 'Disputes the charge', showing its chip strip: a Persona chip reading 'Frustrated Account Holder (inherited)' and an Environment chip reading 'Quiet line (inherited)' in muted italics, alongside a Variables chip.](https://mintcdn.com/roark/wL__NyRpzLGS2dwF/images/simulation-testing/variant-chips.png?fit=max\&auto=format\&n=wL__NyRpzLGS2dwF\&q=85\&s=0470b8ffd2e64e1d8528d8f71d1a0c54)

### The inline persona editor

Clicking a variant's Persona chip opens an inline editor with two paths:

**1. Pick an existing persona**

Click **Pick from existing personas** to open the "Pick a persona" dialog.
Search by name, language, or accent; each row shows the persona's name,
Language · Accent, and its Built-in or Custom badge. Picking one pins it to
the variant.

**2. Or author a caller in place**

Fill in the same fields as the hub dialog (minus Description) directly in
the panel (including a color-coded Base emotion pill grid) and click
**Create & use** to create the persona and pin it in one step.

**3. Tweak a pinned persona**

With a persona pinned, the editor shows "Pinned: \<name>". Editing a
Custom persona and clicking **Save changes** updates it in place. Editing a
Built-in persona creates an editable Custom copy instead.

![The inline persona editor: a Pick from existing personas affordance at the top, then the new-caller fields (Name, Display name, Language, Accent, Gender, Age) and a color-coded Base emotion pill grid with Frustrated selected](https://mintcdn.com/roark/Q1LycOQn64NfkIxp/images/simulation-testing/persona-editor.png?fit=max\&auto=format\&n=Q1LycOQn64NfkIxp\&q=85\&s=4d3bc55120b8fdcb2ea6d543fd3635f0)

> [!TIP]
>
> Because Custom personas are shared across flows, "Save changes" in the inline
> editor updates that persona everywhere it's pinned. If you want a variant-only
> tweak, create a new persona instead.

## Personas at run time

When a [run](/documentation/simulation-testing/running-simulations) executes, each simulated call is placed as the persona pinned to its flow variant. On the resulting call records, the persona appears as `simulationPersonaName`: the display name of the persona used in the simulation that created the call.

If a persona used by a run has properties without values, Roark prompts for them in the runtime-variables step before the run starts; the values you enter are saved onto the persona itself, so future runs reuse them. See [Variables](/documentation/simulation-testing/variables).

## API

Manage personas programmatically to build them from real customer data or wire persona management into CI:

* **Create Persona**: `POST /v1/persona`
* **Update Persona**: `PUT /v1/persona/{personaId}`
* **Get Persona**: `GET /v1/persona/{personaId}`
* **List Personas**: `GET /v1/persona`

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/persona \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Emma Johnson",
    "language": "EN",
    "accent": "US",
    "gender": "FEMALE",
    "baseEmotion": "NEUTRAL",
    "speechPace": "NORMAL",
    "speechClarity": "CLEAR",
    "hasDisfluencies": false,
    "properties": {
      "accountNumber": "ACC-123456",
      "zipCode": "90210"
    }
  }'
```

> [!NOTE]
>
> The API exposes some capabilities the platform UI doesn't surface: a
> `backstoryPrompt`, key/value `properties`, `secondaryLanguage`, idle-message
> settings, and the behavioral fields `backgroundNoise`, `intentClarity`,
> `confirmationStyle`, and `memoryReliability`. Personas created in the UI use
> system defaults for these (`NONE`, `CLEAR`, `EXPLICIT`, `HIGH`); set them via
> the API if you need finer control.

For all parameters and response formats, see the [Persona API Reference](/api-reference/simulation-persona/create-a-new-persona).

## Best practices

* **Start with 3–5 core personas** representing your most common caller types, then add edge cases as you discover them.
* **Mirror reality**: base personas on real customer data and call recordings.
* **Test extremes**: a Frustrated, Fast, Rambling caller stresses your agent in ways a Neutral one won't.
* **Name for the test**: "Confused elder: password reset" tells your team more than "Robert".

## Next steps

> [card] [Customer flows](/documentation/simulation-testing/customer-flows)
>
> Pin personas to flow variants and define what should happen on the call

> [card] [Running simulations](/documentation/simulation-testing/running-simulations)
>
> Plan and launch runs that execute your flows as these callers


# Run Plans
Source: https://docs.roark.ai/documentation/simulation-testing/run-plans

Save simulation configurations as reusable test suites you can run on demand, on a schedule, or via API

## Overview

A plan is a reusable test suite. It bundles everything a simulation run needs (the agents to test, the customer flows to exercise, the metrics and Pass/Fail checks to grade against, and the run settings) so you can execute the same suite repeatedly and compare results over time.

Every run starts from a plan. When you configure a run without saving it, Roark still creates a plan behind the scenes: it just stays hidden as a **one-off run**. Check **Save as plan** and the configuration becomes a named plan you can re-run anytime, schedule, and edit.

Define a plan [in code](/documentation/config-as-code/simulation-plans) (recommended), or build it in the dashboard.

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineAgent, defineSimulationPlan, defineConfig } from '@roarkanalytics/sdk/lib/config';

const frontdesk = defineAgent({
  name: 'frontdesk',
  endpoints: [{ value: '+15551234567', direction: 'OUTGOING' }],
});

const nightly = defineSimulationPlan({
  name: 'nightly-screener-suite',
  direction: 'OUTBOUND',
  agentEndpoints: [{ agent: 'frontdesk', value: '+15551234567' }],
  metrics: ['call_screening_encountered', 'call_screening_handling_score'],
  flows: [{ system: 'sf-screener-apple' }],
  maxDurationSeconds: 300,
});

export default defineConfig({ resources: [frontdesk, nightly] });
```

```yaml roark/simulation-plans/nightly.yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
# (plus roark/agents/frontdesk.yaml defining the agent)
kind: simulationPlan
name: nightly-screener-suite
direction: OUTBOUND
agentEndpoints:
  - agent: frontdesk
    value: '+15551234567'
metrics:
  - call_screening_encountered
  - call_screening_handling_score
flows:
  - system: sf-screener-apple
maxDurationSeconds: 300
```

Apply creates the plan; running it is a separate step:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply roark.config.ts                        # create the agent + plan
roark simulation run --data '{"planId":"<plan-id>"}'      # run it (find the id with `roark simulation plan list`)
```

See [Simulation plans](/documentation/config-as-code/simulation-plans) for every field a plan takes, or the [run a simulation walkthrough](/documentation/config-as-code/run-a-simulation) to go from zero to a scored result.

> [!NOTE]
>
> Plans live under **Simulate → Simulations** in the sidebar. The runs hub has a **Plans / All runs** view toggle: the Plans view shows one card per saved plan, plus a pinned **One-off runs** card for runs that weren't saved as a plan.

***

## Creating a plan

### Build it in the dashboard

Click **New Run** on the Simulations hub to open the create page. It's a single long-form page with four numbered sections:

![The create-run page with numbered sections (01 Template, 02 Agents to test) and the footer action bar with the Save as plan checkbox and the Run simulation split button](https://mintcdn.com/roark/93k2kGpZVUUbTXSq/images/simulation-testing/new-run.png?fit=max\&auto=format\&n=93k2kGpZVUUbTXSq\&q=85\&s=46052cdb78af86678e6f43698d482d09)

**1. 01 · Agents to test**

Pick the agents and how to reach each one. **Select agents & endpoints** opens a picker; each target is one agent reached on one specific endpoint (Phone, Web RTC, LiveKit, WebSocket, ElevenLabs, Google CES, or Kore) with its direction shown as Inbound, Outbound, or In + out.

**2. 02 · Template**

Start from a goal. Each template ships with preset metrics, Pass/Fail checks, and a starter configuration. Pick from the card grid (Blank is the default) or open **Browse all templates** for the full library. See [Templates](/documentation/simulation-testing/templates) for what each one does.

**3. 03 · Template configuration**

The template's own controls: for example, Load testing shows a Volume panel with concurrent calls and total iterations, and Multilingual asks which languages to test. Every template also surfaces a **Flows** panel where you attach the customer flows the run exercises (see below).

**4. 04 · Advanced**

Metrics and run settings: the metric picker with Pass/Fail thresholds, the option to [collect metrics from your agent's side of the call](/documentation/simulation-testing/enriched-simulations), run settings, end conditions, and custom variables.

### Save as plan, or run one-off

The footer bar has a **Save as plan** checkbox: "Saves this configuration as a plan you can re-run anytime." Checking it prompts you to name the plan.

* **Checked**: the plan appears in the Plans view with its own detail page, and every execution is recorded in its run history.
* **Unchecked**: the run executes as a one-off. It's grouped under the **One-off runs** card in the Plans view and the **One-off** lens in All runs. You can promote a one-off to a named plan later from the run header.

### Run now, or later

The primary button is a split button. Its face reads **Run simulation** (or **Schedule** when a schedule mode is selected), and the menu offers:

| Option | What it does |
| :- | :- |
| **Run now** | Starts the simulation immediately |
| **Schedule recurring…** | Runs the plan on a cadence: hourly, daily, weekly, or monthly |
| **Run once, later…** | Runs a single time at a future date and time |

Choosing a schedule mode reveals the Schedule section: frequency, time, timezone, days, and an end condition (never, on a date, or after N runs). See [Schedules](/documentation/simulation-testing/schedules) for details.

> [!TIP]
>
> If any variable in the plan lacks a default value, a runtime-variables modal collects the missing values before the run starts.

***

## Direction is derived

You don't choose inbound or outbound. Roark derives it from the endpoints you selected in **Agents to test**. A run is outbound only when every selected endpoint is outgoing-only; otherwise it's inbound. Outbound runs are forced to a single iteration because your agent initiates each call.

Learn more in the [Inbound vs Outbound guide](/documentation/simulation-testing/inbound-vs-outbound).

***

## Attaching flows

The **Attach flows** dialog lists your [customer flow](/documentation/simulation-testing/customer-flows) library. Pick the flows this run exercises, and expand one to choose its variants. Each attachment carries three things:

![The Attach flows dialog: a searchable list of customer flows, each with a checkbox, mode badge, and variant and expectation counts. One flow is expanded to its variant selection, showing the All variants, Happy path only, and Specific variants choice above the happy-path and edge-case variant checkboxes.](https://mintcdn.com/roark/wL__NyRpzLGS2dwF/images/simulation-testing/attach-flows.png?fit=max\&auto=format\&n=wL__NyRpzLGS2dwF\&q=85\&s=7f83548c58946c7b1b1861c58f7e4a2b)

> [accordion] Variant selection mode
>
> Per attached flow, choose which variants run:
>
> * **All variants**: the happy path plus every edge case
> * **Default variant**: just the happy path
> * **Specific variants**: an explicit selection of variants
>
> Templates set this for you where it matters. Multilingual and Load testing pin the default variant, for example.

> [accordion] Persona override
>
> Optionally override the persona the flow's variants would normally use, just for this plan. The Multilingual template uses this to fan the same flow across language personas, one attachment per language.

> [accordion] Variable overrides
>
> Set per-attachment values for the flow's `{{variable}}` tokens. Overrides set here win over the values stored on the variant at run time. See [Variables](/documentation/simulation-testing/variables).

The total number of test calls is the selected variants across all attachments, multiplied by your agent targets and iterations.

***

## Success criteria

What makes a run of this plan pass or fail. **Advanced → Success criteria** lists one row per check, with the minimum it has to clear.

A **check** is a metric that produces a pass/fail result: a yes/no metric, or a threshold you added to another metric the plan collects. Numeric and classification metrics are still collected and reported, they just have no verdict to judge, so they don't appear here.

![The Success criteria table in a run plan's Advanced settings: one row per pass/fail check with its minimum pass rate and the count of simulations that must pass to clear it.](https://mintcdn.com/roark/IjseSxRBSMbit4Aq/images/simulation-testing/success-criteria.png?fit=max\&auto=format\&n=IjseSxRBSMbit4Aq\&q=85\&s=e739a0f9482ee272efc3609b1b6ef27f)

| Column | What it is |
| :- | :- |
| **Check** | The pass/fail metric being judged |
| **Minimum** | The share of the run's simulations that must pass it |
| **Must pass** | The same bar restated as a count, e.g. "8 of 10 simulations need to pass" |

Every check starts at **80%** and keeps following that default until you type a number of its own (the revert arrow puts it back). The Metrics block above is where a metric becomes a check; this table is where it gets its minimum.

### Each check is judged on its own

There is no plan-wide bar and no per-metric weight. Set silence duration to 40% and word count to 80%, and the run fails unless 40% of simulations clear silence **and** 80% clear word count. A check that falls short fails the run however well the others did.

That is deliberate: one number for a whole plan is either too strict for the loose checks or too loose for the strict ones, and a pooled average would let a check at 0% hide behind checks at 100%.

> [!NOTE]
>
> Criteria are pinned to a run when it starts. Editing the plan changes what future runs are judged against and never rewrites a verdict that already happened.

The settled run report shows the result as a **Success criteria** card: each check's pass rate against the minimum it had to reach, a PASSED or FAILED pill, and a **score** (the mean of the check rates) that is there for trend lines and decides nothing. A plan with no checks reports no verdict at all.

### Setting minimums over the API

Criteria live on the metric references, so a plan carries them wherever it is created or updated:

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
"metrics": [
  { "slug": "agent_containment" },
  { "slug": "latency", "minPassRate": 40 },
  { "slug": "leaked_pii", "minPassRate": 100 }
]
```

`minPassRate` is 0-100 and applies to that check alone. Omit it (or send `null`) for the 80% default. The verdict comes back on `GET /v1/simulation/plan/job/{jobId}` once a run has ended, which is what the [CI gate](/documentation/simulation-testing/ci-cd) turns into an exit code.

***

## Run settings

The **Advanced** section exposes the run's execution and end-condition settings:

| Setting | Description | Default |
| :- | :- | :- |
| **Iterations** | Runs per test case, 1–100. Forced to 1 for outbound runs. | 1 |
| **Concurrency** | Parallel calls, capped by your account quota | 5 |
| **Execution mode** | Parallel, Sequential (this plan), or Sequential (all plans) | Parallel |
| **Max duration** | Hard cap per simulation, up to 24 hours | 15 minutes |
| **Silence timeout** | Ends the call after sustained silence, 5–300 seconds | 30 seconds |
| **End-call phrases** | Phrases that end the call when spoken (max 10; empty disables) | `goodbye` |
| **End-call reasons** | Semantic conditions evaluated by an LLM that end the call, e.g. "The agent confirmed the order" (max 10; empty disables) | None |

The sequential execution modes run calls one at a time (scoped either to this plan's runs or to every plan in the project), which is useful when your agent can't handle parallel traffic.

### Run-plan variables

Plans can define their own variables (a name, a value type, and an optional default), referenced as `{{name}}` in flow prompts and settings. Variables without a default are collected in a modal each time the plan runs, which makes them handy for values that change per run, like a booking date or a ticket number.

***

## The plan detail page

Every saved plan has a detail page with the plan name, its template badge, the agents it targets, and its run count, plus two header actions: **Edit** (reopens the create form pre-filled; the template is read-only) and **Run now**.

![A simulation plan detail page: the header shows the plan name, its agent, and run count with Edit and Run now actions; an Automation panel shows an Active schedule running every weekday at 9:00 AM ET with next-run and last-run times and pause controls; and a Run history list shows past runs labelled SR-\{n} with Running, Completed, and Failed statuses, Schedule and Manual trigger chips, check pass rates, and timestamps.](https://mintcdn.com/roark/cU-YAWgdvSh373VF/images/simulation-testing/plan-detail.png?fit=max\&auto=format\&n=cU-YAWgdvSh373VF\&q=85\&s=228a5636ed352fed8be06997540ad8ff)

The body shows:

* **Automation panel**. If the plan is scheduled: an Active/Paused badge, the cadence description, next-run and last-run times, and controls to pause/resume, edit the cadence, or delete the schedule. Unscheduled plans show a **Runs on demand** card with a **Schedule this plan** button instead.
* **Report delivery**: where each run's report goes once it is ready. Click **Set up** (or **Edit**) to email it to every project member or to specific addresses (they do not need a Roark account), and to post it to up to 10 Slack channels. The email attaches the report PDF and each Slack message carries it in its thread. Each run sends its report once, as soon as it is ready. Duplicating a plan copies its report delivery to the copy. A plan managed by [config as code](/documentation/config-as-code/simulation-plans#sending-each-runs-report) shows its settings read-only; change them in the plan's `notify` block.
* **Run history**: every execution of the plan, newest first. Runs are labelled `SR-{n}` and show one of five statuses: Running, Queued, Completed, Failed, or Cancelled, plus a trigger chip (Manual, Schedule, Re-run, or System). Click a run to open its report. See [Running Simulations](/documentation/simulation-testing/running-simulations).

***

## Creating plans via API

You can create and trigger plans programmatically:

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST "https://api.roark.ai/v1/simulation/plan" \
  -H "Authorization: Bearer $ROARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Production regression suite",
    "direction": "INBOUND",
    "maxSimulationDurationSeconds": 900,
    "scenarios": [
      {
        "id": "flow-uuid",
        "variables": {
          "customerName": "John Doe",
          "appointmentDate": "2026-02-15"
        }
      }
    ],
    "personas": [{ "id": "persona-uuid" }],
    "agentEndpoints": [{ "id": "endpoint-uuid" }],
    "metrics": [
      { "slug": "task-completion" },
      { "slug": "leaked-pii", "minPassRate": 100 }
    ],
    "iterationCount": 1,
    "maxConcurrentJobs": 5,
    "silenceTimeoutSeconds": 30,
    "endCallPhrases": ["goodbye"],
    "endCallReasons": ["The agent confirmed the order"],
    "executionMode": "PARALLEL",
    "autoRun": true
  }'
```

To send each run's report, add `notifications` to the body (or to `PUT /v1/simulation/plan/{planId}`, where `null` turns delivery off):

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
"notifications": {
  "email": { "enabled": true, "recipients": ["qa-lead@example.com"] },
  "slack": [{ "channelId": "C0123ABCXYZ", "channelName": "#voice-qa" }]
}
```

An empty `recipients` list emails every project member. Slack channels need Slack connected to the project, or the request returns `400`.

Required fields: `name`, `direction`, `maxSimulationDurationSeconds`, `scenarios`, `personas`, `agentEndpoints`, and `metrics`. The same flow ID can appear in `scenarios` multiple times with different variables, and `autoRun: true` triggers a run immediately after creation. Trigger an existing plan with `POST /v1/simulation/run`, passing the plan id in the body: `{ "planId": "<plan-id>" }`.

> [!NOTE]
>
> The REST API keeps the older names: customer flows are attached via the `scenarios` array, and `direction` is set explicitly rather than derived from endpoints as it is in the UI.

***

## Next steps

> [card] [Customer Flows](/documentation/simulation-testing/customer-flows)
>
> Author the conversations your plans exercise

> [card] [Templates](/documentation/simulation-testing/templates)
>
> Start plans from a goal with preset metrics and checks

> [card] [Schedules](/documentation/simulation-testing/schedules)
>
> Put plans on a cadence and catch regressions automatically

> [card] [Running Simulations](/documentation/simulation-testing/running-simulations)
>
> Watch runs live and read the report when they settle


# Running Simulations
Source: https://docs.roark.ai/documentation/simulation-testing/running-simulations

Trigger runs, watch them live, and read the settled report

A **run** is one execution of a simulation plan. Every run gets a sequential label (`SR-42`), a trigger chip showing how it started, and a run detail page that acts as mission control while calls are in flight, then settles into a full report once the last call lands.

> [!NOTE]
>
> Runs execute a configured plan: agent targets, customer flows, metrics, and run settings. If you haven't built one yet, start with [Run Plans](/documentation/simulation-testing/run-plans).

## Ways to trigger a run

Every run row carries a trigger chip telling you how it started:

| Trigger | How it happens |
| :- | :- |
| **Manual** | You clicked **Run simulation** on the create-run page, or **Run now** on a plan |
| **Schedule** | A recurring or one-time schedule fired ([Schedules](/documentation/simulation-testing/schedules)) |
| **Re-run** | Started from a previous run's **Re-run** menu |
| **System** | Roark started the run automatically |

### Run now

From **Simulate → Simulations**, click **New Run**, configure the run, and hit **Run simulation**. The split button's menu also offers **Schedule recurring…** and **Run once, later…** if you'd rather not run immediately. Tick **Save as plan** if you want to keep the configuration. Otherwise the run is grouped under the **One-off runs** card on the runs hub.

To run an existing plan, open it and click **Run now**.

> [!NOTE]
>
> If your plan has \{\{variables}} without default values, a modal collects them before the run starts. See [Variables](/documentation/simulation-testing/variables).

### Trigger a run programmatically

Start a run of an existing plan from the SDK, the CLI, or a raw HTTP call. Each takes the plan's id (find it with `roark simulation plan list`) and returns a `simulationRunPlanJobId`:

> [tab] Node
>
> ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> const run = await client.simulation.run({ planId: '<plan-id>' })
> console.log(run.data.simulationRunPlanJobId)
> ```

> [tab] CLI
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> roark simulation run --data '{ "planId": "<plan-id>" }'
> ```

> [tab] cURL
>
> ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
> curl -X POST "https://api.roark.ai/v1/simulation/run" \
>   -H "Authorization: Bearer $ROARK_API_KEY" \
>   -H "Content-Type: application/json" \
>   -d '{ "planId": "<plan-id>" }'
> ```

Read the settled report with `roark simulation plan job result <job-id>` (or `GET /v1/simulation/plan/job/{jobId}`). To override a plan's \{\{variables}} at runtime, include a `variables` object in the body. See [Variables](/documentation/simulation-testing/variables).

> [!NOTE]
>
> The REST API keeps the older naming: customer flows are called `scenarios`, so flow-scoped runtime variables are passed as an array of `{ scenarioId, variables }` objects.

## Run statuses

Runs show one of five statuses:

| Status | Meaning |
| :- | :- |
| **Queued** | Accepted and waiting to start |
| **Running** | Building test calls, executing them, or compiling results |
| **Completed** | Every call finished and the report is ready |
| **Failed** | The run hit an unrecoverable error or timed out |
| **Cancelled** | You cancelled the run before it finished |

> [!NOTE]
>
> The API exposes finer-grained backend statuses (`CREATING_SNAPSHOTS`, `CREATING_SIMULATIONS`, `RUNNING_SIMULATIONS`, `ENDING_SIMULATIONS`, `CANCELLING`). All of these render as **Running** in the UI, and `TIMED_OUT` renders as **Failed**.

## Watching a live run

Open a run while it's executing and you get mission control. The header shows the plan name (or "One-off run #N"), the SR-number, Inbound/Outbound and template badges, the agents under test, and a pulsing **Running · mm:ss** clock. Banners walk you through the phases: "Starting the run", then "All calls landed, compiling the report".

![A live run in progress: a progress strip of per-call status cells with a Done/Running/Queued legend, a Live now board of in-flight conversations with their callers and latest transcript lines, and the What we're measuring checklist](https://mintcdn.com/roark/Q1LycOQn64NfkIxp/images/simulation-testing/live-run.png?fit=max\&auto=format\&n=Q1LycOQn64NfkIxp\&q=85\&s=05e2b4cd5fe4f1e359c197a52e81fadb)

* **Progress strip**: how many calls are done, in flight, and still queued.
* **Live now board**: every active conversation. Voice calls (Phone, WebRTC, LiveKit) stream audio so you can listen in as they happen; chat sessions (WebSocket, ElevenLabs, Google CES, Kore) show a live transcript instead. Pending conversations show their state: Dialing, Connecting, or Waiting for call.
* **What we're measuring**: the run's checks, so you know what pass/fail will be judged on.
* **Calls table**: click any call to open a split detail pane with the transcript, agent expectations, and audio for voice calls.

### Cancelling a run

Click the red **Cancel run** button in the header. Cancellation is graceful: **queued calls stop, but in-flight calls run to completion** and still appear in the results. The run settles with status **Cancelled**.

## The settled report

Once every call lands, the page becomes a report with four sections:

| Section | What's in it |
| :- | :- |
| **Overview** | Verdict hero and headline stats |
| **Metrics** | Per-metric results across the run |
| **Comparison** | This run side by side with the plan's other runs |
| **Calls** | Every conversation, with transcripts and per-call results |

The verdict hero leads with a check pass-rate ring and a plain-language narrative ("Clean sweep: every check passed on every conversation."), backed by stat cards: **Clean calls**, **Calls run**, and **Trend** or **Weakest check**.

Below it, the **Success criteria** card is the pass/fail decision itself: a PASSED or FAILED pill, one row per check with the pass rate it achieved against the minimum it had to clear, and the reasons any run failed. The **score** alongside it is the mean of the check rates, shown for trend lines only: a run can score 95 and still fail when one check missed its own bar. A plan carrying no pass/fail check reports no verdict, since there was nothing to judge. See [Success criteria](/documentation/simulation-testing/run-plans#success-criteria), and [CI/CD](/documentation/simulation-testing/ci-cd) for gating a pipeline on the same verdict.

![A settled run report: the verdict hero with a pass-rate score ring, a plain-language summary, stat columns (calls run, trend, weakest check), and the Overview/Metrics/Comparison/Calls section nav](https://mintcdn.com/roark/Q1LycOQn64NfkIxp/images/simulation-testing/run-report-verdict.png?fit=max\&auto=format\&n=Q1LycOQn64NfkIxp\&q=85\&s=6f131c82be55e438a17d50314335f2c7)

Use the **Export** button to download the run's results as a CSV. A run-name dropdown in the header switches between the plan's executions (the newest is tagged "Latest"), or open the **Comparison** tab to view runs side by side.

### Calls that don't count

Not every call a run places ends up in its results. Every figure in the report, from the pass-rate ring to the per-check rates, the success criteria and the calls list, is built only from calls that tested what they were meant to test. Everything else falls into one of three groups, each shown separately so nothing disappears silently.

**Calls that couldn't run on Roark's side.** Occasionally a simulation fails for a reason that has nothing to do with your agent: the speech pipeline couldn't start, capacity wasn't available, or post-call analysis failed. These are left out of every list and count. The calls list shows a quiet note ("N simulations could not run on our side") with a link to reveal them. A call that fails this way before a conversation starts is retried once automatically and doesn't take an extra concurrency slot, so the run still gets the calls it planned.

**Calls your agent caused to fail.** When your agent didn't answer, was busy, declined, hung up right away or went silent, that is a real result about your agent. These calls stay in the report as failures.

**Excluded calls.** A call that ran but didn't exercise what the run is testing is marked **Excluded**, with a one-line reason, and left out of every figure. The **Show excluded** toggle on the calls list reveals them.

| Reason | Label | When it applies |
| :- | :- | :- |
| `REQUIRED_STAGE_INCOMPLETE` | Setup incomplete | The call ended before it got through a stage its flow marks as required, for example an IVR the agent was never reached past. |
| `SCRIPT_DIVERGED` | Went off script | A strict flow went off script at a step whose off-script policy is **Hang up and invalidate**. |
| `CALLER_NEVER_TOOK_OVER` | Caller never took over | The call opened on a [preceding flow](/documentation/simulation-testing/customer-flows#composing-flows)'s own persona, which never handed the phone to the persona under test. None of that persona's properties (accent, pace, emotion, age) were exercised, so the call says nothing about them. |

### Over the API

`GET /v1/simulation/job/{jobId}` (and each job in `GET /v1/simulation/plan/job/{jobId}`) reports both on every job:

* **`failureOwner`**: for a `FAILED` or `TIMED_OUT` job, who it failed because of. `ROARK` means the simulation itself failed on Roark's side and says nothing about your agent; `AGENT` means your agent under test caused it. It is `null` when the job didn't fail.
* **`invalidation`**: present on an excluded job, with the `reason` from the table above, a one-sentence `detail`, and `invalidatedAt`.

```json theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "data": {
    "simulationJobId": "08b09b8c-c18b-4aa1-bed6-139dd178a2c1",
    "status": "FAILED",
    "failureOwner": "ROARK",
    "invalidation": null
  }
}
```

A job with `failureOwner: "ROARK"` isn't evidence about your agent, so don't count it as a failure in your own reporting. Treat an invalidation reason you don't recognise as "excluded": new reasons may be added.

## Re-running

The settled report's **Re-run** menu offers two distinct modes:

| Mode | What it does | Use it when |
| :- | :- | :- |
| **Fresh run** | Builds new test calls from the plan **as configured today**, collecting any unresolved variables first | You've edited the plan or flows and want to test the current configuration |
| **Re-run with same data** | Replays this run's **exact snapshot**: same flows, personas, and settings, even if the plan has changed since | You want an apples-to-apples comparison, e.g. verifying a bug fix under identical conditions |

Re-runs land in the same plan's run history, so the Comparison tab picks them up automatically.

## Iterations and concurrency

Set in the plan's **Advanced → Run settings**:

* **Iterations**: runs per test case, up to 100. Useful for catching intermittent failures. Outbound runs are limited to 1 iteration. See [Inbound vs Outbound](/documentation/simulation-testing/inbound-vs-outbound).
* **Concurrency**: parallel calls. The default is 5, capped by your account quota. Start there and raise it once you've confirmed your agent handles the load.

## Next steps

> [card] [Run Plans](/documentation/simulation-testing/run-plans)
>
> Build reusable test suites

> [card] [Schedules](/documentation/simulation-testing/schedules)
>
> Put plans on a cadence

> [card] [Variables](/documentation/simulation-testing/variables)
>
> Parameterize flows with runtime values

> [card] [Enriched Simulations](/documentation/simulation-testing/enriched-simulations)
>
> Score metrics from your agent's side of the call


# Schedules
Source: https://docs.roark.ai/documentation/simulation-testing/schedules

Put a plan on a schedule so it runs itself: hourly, daily, weekly, monthly, or once at a future time

## Overview

A schedule is a property of a plan: each plan can have one schedule that runs it automatically on a cadence, or once at a future time. Instead of triggering runs by hand, put your regression suite on a daily schedule and let Roark catch regressions before you do.

There is no separate schedules page. You create and manage a schedule in three places:

* **In Config as Code** (recommended): the `schedule` block on a [`kind: simulationPlan`](/documentation/config-as-code/simulation-plans#running-on-a-schedule), versioned with the plan
* **While creating a run**: the split button next to **Run simulation** offers **Schedule recurring…** and **Run once, later…**
* **On the plan detail page**: the **Automation** panel (or the **Schedule this plan** call-to-action if the plan runs on demand)

Scheduled executions show up as ordinary runs in the runs hub, tagged with a **Schedule** trigger chip.

To run several plans together on one schedule and get a single combined report, declare a [simulation group](/documentation/config-as-code/simulation-groups#running-on-a-schedule) with a `schedule` in Config as Code.

***

## Schedule in code

The fastest way to schedule a plan: add a `schedule` block to its config and apply. The schedule is versioned with the plan and managed from your repo.

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineSimulationPlan, defineConfig } from '@roarkanalytics/sdk/lib/config';

const nightlyScreenerSuite = defineSimulationPlan({
  name: 'nightly-screener-suite',
  direction: 'OUTBOUND',
  agentEndpoints: [{ agent: 'frontdesk', value: '+15551234567' }],
  metrics: ['call_screening_handling_score'],
  flows: [{ system: 'sf-screener-apple' }],
  maxDurationSeconds: 300,
  schedule: {
    type: 'weekly',
    days: ['monday', 'tuesday', 'wednesday', 'thursday', 'friday'],
    time: '02:00',
    timezone: 'America/New_York',
  },
});

export default defineConfig({ resources: [nightlyScreenerSuite] });
```

```yaml roark/simulation-plans/nightly-screener-suite.yaml theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: simulationPlan
name: nightly-screener-suite
direction: OUTBOUND
agentEndpoints:
  - agent: frontdesk
    value: '+15551234567'
metrics:
  - call_screening_handling_score
flows:
  - system: sf-screener-apple
maxDurationSeconds: 300
schedule:
  type: weekly
  days: [monday, tuesday, wednesday, thursday, friday]
  time: '02:00'
  timezone: America/New_York
```

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply roark.config.ts
```

The plan now runs itself on the cadence you set. See the [simulation plans reference](/documentation/config-as-code/simulation-plans#running-on-a-schedule) for every cadence type (`hourly`, `interval`, `daily`, `weekly`, `monthly`, `once`) and end conditions like `endDate` and `maxRuns`. Prefer clicking? Set the same schedule in the dashboard, below.

***

## Scheduling from the create-run page

**1. Configure the run**

Set up agents, template, flows, and settings as usual on **Simulations → New Run**.

**2. Save it as a plan**

Check **Save as plan** in the footer and name the plan. A schedule belongs to a plan, so the configuration needs to be saved.

**3. Pick when it runs**

Open the chevron menu on the primary button and choose **Schedule recurring…** (run on a cadence) or **Run once, later…** (run a single time in the future). The button face changes from **Run simulation** to **Schedule**.

**4. Set the cadence**

An inline **Schedule** section appears with frequency, time, timezone, day selection, and end condition. A live sentence summarizes your choices, for example, "Every weekday at 9:00 AM, ET · ends never".

**5. Create the schedule**

Press **Schedule** (or <kbd>⌘</kbd>+<kbd>↵</kbd>). The plan is saved and its first run fires at the next scheduled time.

![The create-run footer with the primary split button menu open (Run now, Schedule recurring, Run once later) and the inline Schedule section showing a weekly cadence: frequency, time, timezone, weekday selection, end condition, and the live sentence 'Every weekday at 9:00 AM, ET · ends never'](https://mintcdn.com/roark/yUzvMIE3_BYBHPDx/images/simulation-testing/schedule-menu.png?fit=max\&auto=format\&n=yUzvMIE3_BYBHPDx\&q=85\&s=e2b138238b6cac1a1494a7dda134441e)

***

## Scheduling from a plan

On a plan's detail page:

* If the plan has no schedule, you'll see a **Runs on demand** card: "This plan only runs when you start it. Put it on a schedule and it'll run itself, catching regressions before you do." Click **Schedule this plan**, set the cadence, and click **Create schedule**.
* If the plan already has a schedule, the **Automation** panel shows it. Each plan has exactly one schedule.

![A plan detail page with the Automation panel (Active badge, cadence, Next run / Last run, Pause/Resume) above the Run history](https://mintcdn.com/roark/cU-YAWgdvSh373VF/images/simulation-testing/plan-detail.png?fit=max\&auto=format\&n=cU-YAWgdvSh373VF\&q=85\&s=228a5636ed352fed8be06997540ad8ff)

***

## Cadence options

### Frequencies

| Frequency | Behavior | Example |
| :- | :- | :- |
| **Hourly** | Runs every hour at a set minute | :30 past each hour |
| **Daily** | Runs once per day at a set time | Every day at 9:00 AM |
| **Weekly** | Runs on the days you pick (**On these days**) at a set time | Mon and Fri at 2:00 PM |
| **Monthly** | Runs on a day of the month at a set time | The 15th at 10:00 AM |
| **Once** | Runs a single time at a future date and time | Dec 1 at 3:00 PM |

### Time and timezone

Pick the time of day and a timezone from the curated list. Schedules follow the selected timezone, so choose the one your team works in. The live sentence and **Next run** label confirm exactly when the next execution lands.

### End conditions

| Option | Description | Typical use |
| :- | :- | :- |
| **Never** | Runs indefinitely | Continuous monitoring |
| **On a date** | Stops after a specific date | Campaign or seasonal testing |
| **After N runs** | Stops after a set number of executions | Trialing a new schedule |

***

## Managing a schedule

The plan's **Automation** panel is where you manage an existing schedule:

* **Status badge**: **Active** or **Paused**
* **Pause / Resume**: pause without losing the configuration, resume when ready
* **Edit**: change the cadence, time, timezone, or end condition
* **Delete**: remove the schedule entirely (with a confirmation step); the plan and its run history stay intact
* **Next run / Last run**: when the schedule fires next and when it last ran

> [!NOTE]
>
> Editing an active schedule recalculates the next run time from your new settings.

> [!NOTE]
>
> A schedule declared in [Config as Code](/documentation/config-as-code/simulation-plans#running-on-a-schedule) shows **Managed by config** and has no Edit, Pause or Delete here. Change it in your config and re-apply, or detach the plan to manage it from the dashboard.

***

## Scheduled runs in the runs hub

There is no separate job history per schedule. Each scheduled execution is a normal run:

* It appears in the plan's **Run history** and in the runs hub, labelled **SR-\{n}** like any other run
* It carries a **Schedule** trigger chip, distinguishing it from **Manual**, **Re-run**, and **System** runs
* It moves through the usual run statuses: **Running**, **Queued**, **Completed**, **Failed**, **Cancelled**

![The simulations runs hub listing recent runs newest first, with a scheduled run carrying a blue Schedule trigger chip alongside manually triggered runs](https://mintcdn.com/roark/17HU3uIlkQ6GJrX5/images/simulation-testing/runs-hub.png?fit=max\&auto=format\&n=17HU3uIlkQ6GJrX5\&q=85\&s=db9252269f21d1be49a9f2fdd4ce7b9c)

> [!WARNING]
>
> If any variable in the plan lacks a default value, scheduled runs can't prompt for it at execution time. Set defaults for every variable before scheduling. See [Variables](/documentation/simulation-testing/variables).

***

## Best practices

> [accordion] Start with lower frequencies
>
> Begin with daily or weekly schedules to validate the plan before moving to hourly.

> [accordion] Use end conditions when trialing
>
> Set **After N runs** on a new schedule to limit usage while you confirm the configuration.

> [accordion] Watch the first few runs
>
> Check the first scheduled runs in the runs hub to confirm agents connect and checks behave as expected.

> [accordion] Stagger heavy plans
>
> Avoid scheduling multiple high-concurrency plans at the same time.

> [accordion] Pause instead of delete
>
> Pausing keeps the cadence configured so you can resume with one click. Deleting means re-creating it later.

***

## Troubleshooting

**Schedule not firing**

* Check the **Automation** panel: the badge should read **Active**, not **Paused**
* Verify the timezone matches your expectation; the **Next run** label shows the exact upcoming time
* Confirm the end condition hasn't been reached (past its end date, or the run count is exhausted)

**Scheduled runs failing**

* Open the failed run from the plan's **Run history** and review the report
* Verify the plan's agents and endpoints are still reachable
* Make sure every variable in the plan has a default value

***

## Next steps

> [card] [Run Plans](/documentation/simulation-testing/run-plans)
>
> Build the plans your schedules execute

> [card] [Running Simulations](/documentation/simulation-testing/running-simulations)
>
> Follow runs live and read the report

> [card] [Customer Flows](/documentation/simulation-testing/customer-flows)
>
> Design the conversations your plans test

> [card] [Variables](/documentation/simulation-testing/variables)
>
> Set defaults so scheduled runs never stall


# Templates
Source: https://docs.roark.ai/documentation/simulation-testing/templates

Start a simulation run from a goal: each template ships preset metrics, Pass/Fail checks, and its own configuration panel

## Overview

Templates are step **01 · Template** of [creating a run](/documentation/simulation-testing/run-plans). Instead of assembling metrics and checks from scratch, you pick a goal (measure conversation quality, probe for jailbreaks, hammer one flow at volume) and the template seeds a sensible starting configuration:

* **Preset system metrics**: a curated set from the [metric library](/documentation/metrics/system-metrics), auto-selected in the Advanced section.
* **Pass/Fail checks**: [thresholds](/documentation/metrics/thresholds) attached alongside the metrics, so the run reports a clean pass rate out of the box.
* **A configuration panel**: step 03 of the create page changes per template (languages to test, volume to run at, adversarial cases to include).
* **Flow sourcing**: some templates pre-attach [customer flows](/documentation/simulation-testing/customer-flows) for you, by system label or by happy path.

Everything a template seeds is a starting point, not a lock. You can add or remove metrics, adjust checks, and change flows in the **Advanced** section before running.

![The create-run page's Template section with the full catalogue of goal templates expanded: a grid of cards (Blank, Flow adherence, Red teaming, Conversation quality, Multilingual, Tool call accuracy, Knowledge base grounding, Load testing, Accent handling, Emotion handling, Interruption handling, Voicemail testing, Call screening, Health check, Live Bench), each with a colored icon, title and one-line description, with Conversation quality selected](https://mintcdn.com/roark/cU-YAWgdvSh373VF/images/simulation-testing/template-picker.png?fit=max\&auto=format\&n=cU-YAWgdvSh373VF\&q=85\&s=bf8385c3fea9c167b6df2b8044ba1753)

***

## The catalogue

Pick a template from the card grid. The grid shows the common set; **Show more** expands it in place to reveal the rest. Each template has its own page covering why it matters, what it runs, what it measures and what to look for.

| Template | Category | What it tests |
| :- | :- | :- |
| **Blank** | Your templates | Start from scratch: choose what to measure, which flows to run, and who is calling |
| [**Flow adherence**](/documentation/simulation-testing/templates/flow-adherence) | Quality | Whether the agent follows the flows you authored and holds every expectation |
| [**Conversation quality**](/documentation/simulation-testing/templates/conversation-quality) | Quality | How natural, empathetic and helpful the agent is in realistic conversations |
| [**Multilingual**](/documentation/simulation-testing/templates/multilingual) | Quality | Language detection, comprehension and response language across 17 languages |
| [**Tool call accuracy**](/documentation/simulation-testing/templates/tool-call-accuracy) | Quality | Whether the agent calls the right tools, with the right arguments, in the right order |
| [**Knowledge base grounding**](/documentation/simulation-testing/templates/knowledge-base-grounding) | Quality | Whether the agent stays faithful to your knowledge base, and answers what it can |
| [**Red teaming**](/documentation/simulation-testing/templates/red-teaming) | Safety | Jailbreaks, leaked personal data, off-scope answers and made-up facts |
| [**Accent handling**](/documentation/simulation-testing/templates/accent-handling) | Robustness | Where the agent stops understanding callers from different regions |
| [**Caller age handling**](/documentation/simulation-testing/templates/age-handling) | Robustness | Which caller ages the agent understands worst |
| [**Background noise robustness**](/documentation/simulation-testing/templates/background-noise-robustness) | Robustness | What each background, from silence to a construction site, costs the agent |
| [**Emotion handling**](/documentation/simulation-testing/templates/emotion-handling) | Robustness | Which caller moods the agent handles worst |
| [**Gender handling**](/documentation/simulation-testing/templates/gender-handling) | Robustness | Whether the agent understands and treats every voice gender the same |
| [**Interruption handling**](/documentation/simulation-testing/templates/interruption-handling) | Robustness | How the agent handles callers who talk over it |
| [**Speech pace handling**](/documentation/simulation-testing/templates/speech-pace-handling) | Robustness | What callers who rush or drag cost the agent |
| [**Voicemail testing**](/documentation/simulation-testing/templates/voicemail-testing) | Voice & telephony | Whether the agent detects voicemail, leaves a usable message and hangs up |
| [**Call screening**](/documentation/simulation-testing/templates/call-screening) | Voice & telephony | Whether the agent recognises a call screener and saves its pitch for a person |
| [**Load testing**](/documentation/simulation-testing/templates/load-testing) | Performance | Latency, drops and concurrency limits when many calls run at once |
| [**Health check**](/documentation/simulation-testing/templates/health-check) | Performance | A scheduled probe call: the agent answers, speaks and stays responsive |
| [**Live Bench**](/documentation/simulation-testing/templates/live-bench) | Benchmark | Roark's standard benchmark, comparable across agents and versions |

***

## Property sweeps

The seven Robustness templates are **property sweeps**. Each runs the same flow once per value of one caller property (every accent, every background, every pace) with everything else held the same, then tells you which values did worse than the rest and which differences are just chance. Their configuration panel picks the flow, the values, a baseline to compare against, and the iterations, which it holds at 5 or more calls per value so every value can be compared.

***

## Template provenance

When you check **Save as plan**, the plan records which template it came from. The template shows up as a badge on the plan's detail page and on every run's header, and when you edit a plan the template is read-only: it defined the plan's shape, so you adjust the configuration rather than swap the goal.

One plan, one template: to test the same agents against a different goal, create a new run from the other template and save it as its own plan.

***

## Next steps

> [card] [Run Plans](/documentation/simulation-testing/run-plans)
>
> Save a template-seeded configuration as a reusable test suite

> [card] [Customer Flows](/documentation/simulation-testing/customer-flows)
>
> Author the conversations templates attach and grade against

> [card] [Running Simulations](/documentation/simulation-testing/running-simulations)
>
> Watch a run live and read its report

> [card] [Thresholds](/documentation/metrics/thresholds)
>
> How the Pass/Fail checks templates seed actually work


# Accent handling
Source: https://docs.roark.ai/documentation/simulation-testing/templates/accent-handling

Run one flow with callers from every region and find where your agent stops understanding

## Why this is a common failure point

Speech recognition is trained mostly on a few accents, and voice agents inherit its blind spots. A caller with a strong regional accent gets misheard, the agent confidently acts on the wrong word, and the caller has to repeat themselves or gives up. It rarely shows in testing, because the people testing an agent usually share an accent with the people who built it.

## How Roark tests it

Accent handling runs the same flow with a caller from each region you pick. The caller says the same things in every call: only the accent changes, so a value that does worse points at the accent, not the conversation.

It is a **property sweep**: every value runs the same flow with everything else held the same, and the report tells you which values did worse than the rest and which differences are just chance.

***

## The values

| Value | |
| :- | :- |
| American | Suggested baseline |
| American (Southern) | |
| British | |
| Australian | |
| New Zealand | |
| Indian | |
| Spanish | |
| French | |
| German | |
| Italian | |
| Portuguese | |
| Dutch | |
| Greek | |
| Arabic | |
| Hebrew | |
| Indonesian | |
| Malaysian | |
| Filipino | |
| Singaporean | |
| Hong Kong | |
| Thai | |
| Japanese | |
| Turkish | |

Every value is English spoken with that region's accent, not another language. To test other languages, use [Multilingual](/documentation/simulation-testing/templates/multilingual).

***

## What it measures

The template seeds these, and your flow's own metrics run alongside them. Only checks can mark a value **Worse on**; the other measures explain why.

| Measure | What it tells you |
| :- | :- |
| **Comprehension Failure (check)** | The agent misunderstood or misheard the caller. The check fails any call where it did. |
| **Redundant Question Count (check)** | The agent asked for something the caller already gave. The check allows one per call. |
| **Missed Response Count** | Moments the agent should have answered and did not, so the caller had to repeat themselves. |
| **Speech Pace Delta** | How much faster or slower the agent spoke than the caller. |
| **Call Outcome** | Whether the call ended where it should. |

***

## What to look for

* **Comprehension first.** An accent the agent struggles with shows up as comprehension failures and repeated questions long before the call outcome moves.
* **Names, numbers and spellings.** Account numbers, addresses and spelled names are where accents cause the most trouble. A flow that collects them tells you more than one that doesn't.
* **23 values is a big run.** At 6 calls per value that is 138 calls per agent. Start with the accents your callers actually have.

***

## Setting one up

Pick the flow to test, then the values to run it across. Every value starts selected; deselect the ones you don't need. Each value runs your flow again, so the values you keep are the cost of the run.

**Compared against** sets the baseline: a reference value the report pins first, such as silence for background noise. The template suggests the natural one, and you can change it or choose none. It doesn't change the verdict: whether a value did worse is always decided against every other value combined.

**Iterations** live in the sweep panel rather than in Advanced. Each value runs every attached test case once per iteration:

> **Calls per value = test cases x iterations**

Each value needs **at least 5 calls** to be compared with the others, and the Iterations field won't go below the count that reaches it. It starts at 6 calls per value, so one call [excluded after the run](/documentation/simulation-testing/running-simulations#calls-that-dont-count) doesn't drop a value out of the comparison.

More calls let the report see smaller differences. For a sweep of 10 values:

| Calls per value | A value has to be about this much worse to be flagged | Calls in total |
| :- | :- | :- |
| 5 | 50 percentage points | 50 |
| 10 | 35 percentage points | 100 |
| 20 | 25 percentage points | 200 |
| 30 | 20 percentage points | 300 |

> [!NOTE]
>
> Below 5 calls a value can never be flagged, however badly it does, which is why the minimum is enforced rather than suggested. Over the API, a sweep template defaults `iterationCount` to 6 and refuses a plan with fewer than 5 calls per value with a `400`.

***

## How the results are decided

The sweep's table leads the run report. Its **Compared with the rest** column gives each value one result:

| Result | What it means |
| :- | :- |
| **Worse on** *check* | The value failed that check more often than all the other values combined, by a gap too large to be chance. |
| **No difference** | Any gap is within what chance produces with this many calls. That is not proof of no effect: a small run can only see large gaps, and the report says how large when that matters. |
| **Too few calls** (*n*) | Fewer than 5 counted calls, so the value is shown but not compared. |
| **Not graded** | The value ran, but none of its calls were graded. |

How "too large to be chance" is decided:

* Each value is compared with **all the other values combined**, one check at a time, never with a single other value. Two values that happen to land at opposite ends are not a finding on their own.
* A sweep makes many comparisons at once (every value on every check), and some will look different by luck alone. The report accounts for that: of all the values it flags in a run, **at most 1 in 10 is expected to be a fluke**.
* A value is flagged only when it does **worse**. Doing better than the rest is not a problem to fix.

**Checks passed** shows each value's pass rate with the number of calls behind it, so 100% of two calls never reads like 100% of twenty. A lower rate with **No difference** beside it means the gap is not larger than chance.

**Fails, not accent specific.** A check that fails on **at least 20% of calls** with no value standing out is listed under this heading. It is a real problem, just not one accent caused, so changing accent won't fix it. Select the check to open its evidence.

The executive summary follows the same rules: it only blames the property for failures the table marks **Worse on**. Hover **How this works** under the table for this run's own numbers.


# Caller age handling
Source: https://docs.roark.ai/documentation/simulation-testing/templates/age-handling

Run one flow with child, teenage, adult and elderly callers and find which your agent understands worst

## Why this is a common failure point

Older callers often speak more slowly, pause mid-sentence and need information repeated; younger callers speak faster and in higher voices that some speech recognition hears poorly. An agent tuned on adult voices at a normal pace cuts elderly callers off in their pauses and mishears children, and those are often the callers who most need the call to go well.

## How Roark tests it

Caller age handling runs the same flow with a child, a teenager, an adult and an elderly caller, with everything else held the same, and tells you whether any age makes your agent misunderstand, ask again, or fail the call.

It is a **property sweep**: every value runs the same flow with everything else held the same, and the report tells you which values did worse than the rest and which differences are just chance.

***

## The values

| Value | |
| :- | :- |
| Child | |
| Teenager | |
| Adult | Suggested baseline |
| Elderly | |

Each age runs only where the caller's accent has a voice for it. An accent without a child voice, for example, runs its other ages and skips the child value.

***

## What it measures

The template seeds these, and your flow's own metrics run alongside them. Only checks can mark a value **Worse on**; the other measures explain why.

| Measure | What it tells you |
| :- | :- |
| **Comprehension Failure (check)** | The agent misunderstood or misheard the caller. The check fails any call where it did. |
| **Redundant Question Count (check)** | The agent asked for something the caller already gave. The check allows one per call. |
| **Speech Pace Delta** | How much faster or slower the agent spoke than the caller. |
| **Vocal Loudness Delta** | How much louder or quieter the agent was than the caller. |
| **Call Outcome** | Whether the call ended where it should. |

***

## What to look for

* **Elderly callers and pace.** An agent that speaks much faster than an elderly caller shows it in Speech Pace Delta, often before comprehension moves.
* **Children and pitch.** Comprehension failures concentrated on Child usually point at transcription, not the agent's logic.
* **Four values is a small run.** At 6 calls per value the whole sweep is 24 calls per agent, so 10 per value is affordable and sees smaller gaps.

***

## Setting one up

Pick the flow to test, then the values to run it across. Every value starts selected; deselect the ones you don't need. Each value runs your flow again, so the values you keep are the cost of the run.

**Compared against** sets the baseline: a reference value the report pins first, such as silence for background noise. The template suggests the natural one, and you can change it or choose none. It doesn't change the verdict: whether a value did worse is always decided against every other value combined.

**Iterations** live in the sweep panel rather than in Advanced. Each value runs every attached test case once per iteration:

> **Calls per value = test cases x iterations**

Each value needs **at least 5 calls** to be compared with the others, and the Iterations field won't go below the count that reaches it. It starts at 6 calls per value, so one call [excluded after the run](/documentation/simulation-testing/running-simulations#calls-that-dont-count) doesn't drop a value out of the comparison.

More calls let the report see smaller differences. For a sweep of 10 values:

| Calls per value | A value has to be about this much worse to be flagged | Calls in total |
| :- | :- | :- |
| 5 | 50 percentage points | 50 |
| 10 | 35 percentage points | 100 |
| 20 | 25 percentage points | 200 |
| 30 | 20 percentage points | 300 |

> [!NOTE]
>
> Below 5 calls a value can never be flagged, however badly it does, which is why the minimum is enforced rather than suggested. Over the API, a sweep template defaults `iterationCount` to 6 and refuses a plan with fewer than 5 calls per value with a `400`.

***

## How the results are decided

The sweep's table leads the run report. Its **Compared with the rest** column gives each value one result:

| Result | What it means |
| :- | :- |
| **Worse on** *check* | The value failed that check more often than all the other values combined, by a gap too large to be chance. |
| **No difference** | Any gap is within what chance produces with this many calls. That is not proof of no effect: a small run can only see large gaps, and the report says how large when that matters. |
| **Too few calls** (*n*) | Fewer than 5 counted calls, so the value is shown but not compared. |
| **Not graded** | The value ran, but none of its calls were graded. |

How "too large to be chance" is decided:

* Each value is compared with **all the other values combined**, one check at a time, never with a single other value. Two values that happen to land at opposite ends are not a finding on their own.
* A sweep makes many comparisons at once (every value on every check), and some will look different by luck alone. The report accounts for that: of all the values it flags in a run, **at most 1 in 10 is expected to be a fluke**.
* A value is flagged only when it does **worse**. Doing better than the rest is not a problem to fix.

**Checks passed** shows each value's pass rate with the number of calls behind it, so 100% of two calls never reads like 100% of twenty. A lower rate with **No difference** beside it means the gap is not larger than chance.

**Fails, not age specific.** A check that fails on **at least 20% of calls** with no value standing out is listed under this heading. It is a real problem, just not one age caused, so changing age won't fix it. Select the check to open its evidence.

The executive summary follows the same rules: it only blames the property for failures the table marks **Worse on**. Hover **How this works** under the table for this run's own numbers.


# Background noise robustness
Source: https://docs.roark.ai/documentation/simulation-testing/templates/background-noise-robustness

Run one flow through every background a caller could be in, from silence to a construction site

## Why this is a common failure point

Real callers ring from cars, trains, kitchens and busy streets. Background noise hides parts of what they say, and noise that contains other voices (a cafe, an airport, children playing) can be transcribed as if the caller said it. The agent answers words nobody spoke, or stops mid-sentence because it thinks it was interrupted. Agents tested in a quiet office never meet any of this.

## How Roark tests it

Background noise robustness runs the same flow with the caller in each environment you pick, from silence to a construction site. The caller is identical in every value: only the sound around them changes, so the report shows what each background costs your agent.

It is a **property sweep**: every value runs the same flow with everything else held the same, and the report tells you which values did worse than the rest and which differences are just chance.

***

## The values

| Value | |
| :- | :- |
| Silent | Suggested baseline |
| Library | |
| Office | |
| Coffee shop | |
| City street | |
| Driving | |
| Train | |
| Airport | |
| Children playing | |
| Crying baby | |
| Thunderstorm | |
| Construction site | |

Values are ordered from quietest to loudest, which is also how the report lists them.

***

## What it measures

The template seeds these, and your flow's own metrics run alongside them. Only checks can mark a value **Worse on**; the other measures explain why.

| Measure | What it tells you |
| :- | :- |
| **Comprehension Failure (check)** | The agent misunderstood or misheard the caller. The check fails any call where it did. |
| **Redundant Question Count (check)** | The agent asked for something the caller already gave. The check allows one per call. |
| **Missed Response Count** | Moments the agent should have answered and did not, often because noise hid the speech. |
| **Vocal Loudness Delta** | How much louder or quieter the agent was than the caller. |
| **Speech Pace Delta** | How much faster or slower the agent spoke than the caller. |
| **Call Outcome** | Whether the call ended where it should. |

***

## What to look for

* **Missed responses.** Noise that hides the caller's speech shows up as turns the agent never answered, not only as misunderstandings.
* **Noise with voices in it.** Cafes, airports and children playing are harder than loud but steady noise like a train.
* **False interruptions.** An agent that stops mid-sentence for background sound should also be run through [Interruption handling](/documentation/simulation-testing/templates/interruption-handling).

***

## Setting one up

Pick the flow to test, then the values to run it across. Every value starts selected; deselect the ones you don't need. Each value runs your flow again, so the values you keep are the cost of the run.

**Compared against** sets the baseline: a reference value the report pins first, such as silence for background noise. The template suggests the natural one, and you can change it or choose none. It doesn't change the verdict: whether a value did worse is always decided against every other value combined.

**Iterations** live in the sweep panel rather than in Advanced. Each value runs every attached test case once per iteration:

> **Calls per value = test cases x iterations**

Each value needs **at least 5 calls** to be compared with the others, and the Iterations field won't go below the count that reaches it. It starts at 6 calls per value, so one call [excluded after the run](/documentation/simulation-testing/running-simulations#calls-that-dont-count) doesn't drop a value out of the comparison.

More calls let the report see smaller differences. For a sweep of 10 values:

| Calls per value | A value has to be about this much worse to be flagged | Calls in total |
| :- | :- | :- |
| 5 | 50 percentage points | 50 |
| 10 | 35 percentage points | 100 |
| 20 | 25 percentage points | 200 |
| 30 | 20 percentage points | 300 |

> [!NOTE]
>
> Below 5 calls a value can never be flagged, however badly it does, which is why the minimum is enforced rather than suggested. Over the API, a sweep template defaults `iterationCount` to 6 and refuses a plan with fewer than 5 calls per value with a `400`.

***

## How the results are decided

The sweep's table leads the run report. Its **Compared with the rest** column gives each value one result:

| Result | What it means |
| :- | :- |
| **Worse on** *check* | The value failed that check more often than all the other values combined, by a gap too large to be chance. |
| **No difference** | Any gap is within what chance produces with this many calls. That is not proof of no effect: a small run can only see large gaps, and the report says how large when that matters. |
| **Too few calls** (*n*) | Fewer than 5 counted calls, so the value is shown but not compared. |
| **Not graded** | The value ran, but none of its calls were graded. |

How "too large to be chance" is decided:

* Each value is compared with **all the other values combined**, one check at a time, never with a single other value. Two values that happen to land at opposite ends are not a finding on their own.
* A sweep makes many comparisons at once (every value on every check), and some will look different by luck alone. The report accounts for that: of all the values it flags in a run, **at most 1 in 10 is expected to be a fluke**.
* A value is flagged only when it does **worse**. Doing better than the rest is not a problem to fix.

**Checks passed** shows each value's pass rate with the number of calls behind it, so 100% of two calls never reads like 100% of twenty. A lower rate with **No difference** beside it means the gap is not larger than chance.

**Fails, not background noise specific.** A check that fails on **at least 20% of calls** with no value standing out is listed under this heading. It is a real problem, just not one background noise caused, so changing background noise won't fix it. Select the check to open its evidence.

The executive summary follows the same rules: it only blames the property for failures the table marks **Worse on**. Hover **How this works** under the table for this run's own numbers.


# Call screening
Source: https://docs.roark.ai/documentation/simulation-testing/templates/call-screening

Check your outbound agent recognises a call screener, answers it directly, and saves its pitch for a person

## Why this is a common failure point

Apple and Google both ship call screening that answers first, asks who is calling and why, and then decides whether to put the call through, decline it, or take a message. An outbound agent that doesn't recognise a screener delivers its opening pitch to a robot. The call may even connect afterwards, so the outcome looks fine while the person has already missed the introduction.

## How Roark tests it

Call screening dials your agent into real screener behaviour, one call per screener outcome, and grades whether the agent noticed it was talking to a screener, answered it directly with who is calling and why, and held its opening for the person.

***

## Setting it up

Pick the screeners and outcomes to call into. Each screener offers only the outcomes that product actually has:

| Screener | Outcome |
| :- | :- |
| **Apple call screening** | Connects the call |
| **Apple call screening** | Declines the call |
| **Google call screening** | Connects the call |
| **Google call screening** | Takes a message |

One call runs per outcome, per agent. For a walkthrough, see [Testing call screeners](/documentation/recipes/call-screener-testing).

***

## What it measures

The template seeds these, and you can add or remove metrics in **Advanced** before running.

| Measure | What it tells you |
| :- | :- |
| **Call Screening Handling Score (check)** | How well the agent handled the screener, 1 to 5. The check passes at 4 or above. |
| **Call Screened** | Whether a screener answered. Descriptive: it records that a screen happened, not whether the agent noticed. |

***

## What to look for

* **The first agent turn.** A pitch delivered to the screener is the most common failure, and it happens in the first few seconds.
* **Declines and messages.** An agent that argues with a decline, or leaves no message when offered, fails differently from one that misses the screener.

***

## Over the API

Run it with `POST /v1/simulation/run` and `"template": "call-screening"`. `GET /v1/simulation/template` lists what each template accepts.


# Conversation quality
Source: https://docs.roark.ai/documentation/simulation-testing/templates/conversation-quality

Measure how natural, empathetic and helpful your agent sounds across realistic, unscripted conversations

## Why this is a common failure point

An agent can complete every task and still be unpleasant to talk to. It asks for the account number twice, leaves a silence long enough for the caller to say "hello?", talks over them, or answers a frustrated caller in the same bright tone it uses for everyone. These don't fail a task, so task-based tests pass while callers hang up annoyed.

Quality problems also hide in averages: one bad turn in an otherwise fine call is what the caller remembers.

## How Roark tests it

Conversation quality runs your flows with realistic, unscripted callers and grades the experience rather than the outcome: how much effort the caller had to spend, whether the agent repeated itself or missed a turn, how it handled the rhythm of the conversation, and how the caller felt by the end.

***

## Setting it up

Attach any flows through the inline Flows panel. The template works best on improv flows, where the caller behaves like a real person rather than following a script.

***

## What it measures

The template seeds these, and you can add or remove metrics in **Advanced** before running.

| Measure | What it tells you |
| :- | :- |
| **User Effort Score (check)** | How hard the caller had to work to get what they wanted, 1 to 5. The check passes at 4 or below. |
| **Instruction Follow (check)** | How well the agent followed its instructions, 1 to 5. The check passes at 3 or above. |
| **Redundant Question Count (check)** | The agent asked for something the caller already gave. The check allows one per call. |
| **Frustration Score** | How frustrated the caller became, 1 to 5. |
| **Sentiment Score** | The caller's sentiment, 1 (negative) to 9 (positive). |
| **Comprehension Failure** | Whether the agent misunderstood or misheard the caller. |
| **Missed Response Count** | Moments the agent should have answered and did not, so the caller had to repeat themselves. |
| **Interruption Appropriateness** | Whether each time the agent cut in was the right moment. |
| **Talk-to-Listen Ratio** | How much of the call each side spent talking. |
| **Overtalk Ratio** | How much of the call both sides spoke at once. |
| **Response Time** | Time between speaking turns. |
| **Call Outcome** | Whether the call ended where it should. |

***

## What to look for

* **Effort before sentiment.** Callers forgive a flat tone; they don't forgive repeating themselves. User effort and redundant questions are the first things to fix.
* **Response time spikes.** A long pause mid-call reads as a dropped line. Look at the slowest turns, not the average.
* **Read the calls behind a low score.** Every metric links to the turns that produced it.

***

## Over the API

Run it with `POST /v1/simulation/run` and `"template": "conversation-quality"`. `GET /v1/simulation/template` lists what each template accepts.


# Emotion handling
Source: https://docs.roark.ai/documentation/simulation-testing/templates/emotion-handling

Run one flow with callers who are angry, anxious, confused or rushed and find which moods your agent handles worst

## Why this is a common failure point

People call when something has gone wrong, so a large share of callers arrive frustrated, anxious or confused. Emotional callers speak differently (faster, louder, in fragments) and behave differently: they interrupt, repeat themselves and change direction. An agent that only ever met calm test callers mishears them, keeps the same cheerful tone with an angry caller, or loses the thread when a confused caller circles back.

## How Roark tests it

Emotion handling runs the same flow with a caller in each mood you pick, with the same goal and information every time. It tells you whether any mood makes your agent misunderstand the caller or fail the call, and measures how your agent sounded in response, so you can see whether it stayed calm.

It is a **property sweep**: every value runs the same flow with everything else held the same, and the report tells you which values did worse than the rest and which differences are just chance.

***

## The values

| Value | |
| :- | :- |
| Neutral | Suggested baseline |
| Cheerful | |
| Frustrated | |
| Angry | |
| Anxious | |
| Sad | |
| Skeptical | |
| Confused | |
| Rushed | |
| Distracted | |

Mood changes how the caller speaks (tone, pace, pitch) and how they behave. The caller's goal and information stay the same.

***

## What it measures

The template seeds these, and your flow's own metrics run alongside them. Only checks can mark a value **Worse on**; the other measures explain why.

| Measure | What it tells you |
| :- | :- |
| **Comprehension Failure (check)** | The agent misunderstood or misheard the caller. The check fails any call where it did. |
| **Dominant Emotion** | The emotion that dominated each speaker's voice. |
| **Vocal Valence** | How positive or negative each speaker sounded. |
| **Vocal Activation** | How calm or energised each speaker sounded. |
| **Vocal Pitch Delta** | How far apart the agent's and caller's pitch were. |
| **Vocal Loudness Delta** | How much louder or quieter the agent was than the caller. |
| **Vocal Pitch Variation Delta** | How much more or less expressive the agent was than the caller. |
| **Speech Pace Delta** | How much faster or slower the agent spoke than the caller. |
| **Call Outcome** | Whether the call ended where it should. |

***

## What to look for

* **Mirroring.** An agent that gets louder or faster with an angry caller shows it in the loudness and pace deltas. Steady deltas across moods mean it held its tone.
* **Confused and distracted callers** repeat themselves and change direction. Comprehension failures there are the most actionable finding this sweep produces.
* **The vocal measures explain, the check decides.** Only Comprehension Failure is a check, so only it can mark a mood **Worse on**.

***

## Setting one up

Pick the flow to test, then the values to run it across. Every value starts selected; deselect the ones you don't need. Each value runs your flow again, so the values you keep are the cost of the run.

**Compared against** sets the baseline: a reference value the report pins first, such as silence for background noise. The template suggests the natural one, and you can change it or choose none. It doesn't change the verdict: whether a value did worse is always decided against every other value combined.

**Iterations** live in the sweep panel rather than in Advanced. Each value runs every attached test case once per iteration:

> **Calls per value = test cases x iterations**

Each value needs **at least 5 calls** to be compared with the others, and the Iterations field won't go below the count that reaches it. It starts at 6 calls per value, so one call [excluded after the run](/documentation/simulation-testing/running-simulations#calls-that-dont-count) doesn't drop a value out of the comparison.

More calls let the report see smaller differences. For a sweep of 10 values:

| Calls per value | A value has to be about this much worse to be flagged | Calls in total |
| :- | :- | :- |
| 5 | 50 percentage points | 50 |
| 10 | 35 percentage points | 100 |
| 20 | 25 percentage points | 200 |
| 30 | 20 percentage points | 300 |

> [!NOTE]
>
> Below 5 calls a value can never be flagged, however badly it does, which is why the minimum is enforced rather than suggested. Over the API, a sweep template defaults `iterationCount` to 6 and refuses a plan with fewer than 5 calls per value with a `400`.

***

## How the results are decided

The sweep's table leads the run report. Its **Compared with the rest** column gives each value one result:

| Result | What it means |
| :- | :- |
| **Worse on** *check* | The value failed that check more often than all the other values combined, by a gap too large to be chance. |
| **No difference** | Any gap is within what chance produces with this many calls. That is not proof of no effect: a small run can only see large gaps, and the report says how large when that matters. |
| **Too few calls** (*n*) | Fewer than 5 counted calls, so the value is shown but not compared. |
| **Not graded** | The value ran, but none of its calls were graded. |

How "too large to be chance" is decided:

* Each value is compared with **all the other values combined**, one check at a time, never with a single other value. Two values that happen to land at opposite ends are not a finding on their own.
* A sweep makes many comparisons at once (every value on every check), and some will look different by luck alone. The report accounts for that: of all the values it flags in a run, **at most 1 in 10 is expected to be a fluke**.
* A value is flagged only when it does **worse**. Doing better than the rest is not a problem to fix.

**Checks passed** shows each value's pass rate with the number of calls behind it, so 100% of two calls never reads like 100% of twenty. A lower rate with **No difference** beside it means the gap is not larger than chance.

**Fails, not emotion specific.** A check that fails on **at least 20% of calls** with no value standing out is listed under this heading. It is a real problem, just not one emotion caused, so changing emotion won't fix it. Select the check to open its evidence.

The executive summary follows the same rules: it only blames the property for failures the table marks **Worse on**. Hover **How this works** under the table for this run's own numbers.


# Flow adherence
Source: https://docs.roark.ai/documentation/simulation-testing/templates/flow-adherence

Check your agent follows the flows you authored, step by step, and reaches the outcome each one expects

## Why this is a common failure point

Voice agents drift. A prompt that handles the happy path in a demo starts skipping verification steps, reordering questions, or promising things the process doesn't allow once real callers push it off script. Nobody notices, because each call sounds reasonable on its own: the failure only shows when you compare what the agent did with what it was supposed to do.

Prompt changes make it worse. A tweak that fixes one conversation quietly changes behaviour in others, and without a written expectation there is nothing to catch the regression.

## How Roark tests it

You write down what should happen as [customer flows](/documentation/simulation-testing/customer-flows): the steps a conversation should take and what the agent must do along the way. Flow adherence runs a simulated caller through each flow and grades every call against it: did the conversation follow the steps, did the agent follow its instructions, and did it hold every expectation you declared.

Because the flows are fixed, the same run repeated after a prompt change tells you exactly what moved.

***

## Setting it up

Attach the flows the agent should follow. Each flow runs its selected variants (its happy path and any edge cases you pick), once per iteration, per agent.

***

## What it measures

The template seeds these, and you can add or remove metrics in **Advanced** before running.

| Measure | What it tells you |
| :- | :- |
| **Flow Adherence (check)** | How closely the conversation followed the flow, 1 to 5. The check passes at 4 or above. |
| **Instruction Follow (check)** | How well the agent followed its instructions, 1 to 5. The check passes at 3 or above. |
| **Agent Expectations Followed (check)** | Whether the agent held every expectation the flow declares. Each expectation is judged separately, and the check passes only if all hold. |
| **Call Outcome** | Whether the call ended where it should. |

***

## What to look for

* **Expectations are the sharpest signal.** Adherence says the conversation drifted; a failed expectation says exactly which rule the agent broke.
* **Edge cases, not just the happy path.** Agents rarely fail the path they were built around. Attach the variants where the caller changes their mind or gives something unexpected.
* **Rerun after every prompt change.** Keep the run as a [plan](/documentation/simulation-testing/run-plans) so the comparison is like for like.

***

## Over the API

Run it with `POST /v1/simulation/run` and `"template": "flow-adherence"`. `GET /v1/simulation/template` lists what each template accepts.


# Gender handling
Source: https://docs.roark.ai/documentation/simulation-testing/templates/gender-handling

Run one flow with callers of each voice gender and find whether your agent understands and treats them the same

## Why this is a common failure point

Speech recognition accuracy varies with voice pitch, and published studies have repeatedly found different error rates for female and male voices. An agent built on that recognition can mishear one group of callers more often without anyone intending it, and the gap is invisible unless you measure it directly.

## How Roark tests it

Gender handling runs the same flow with a female and a male caller, with everything else held the same, and tells you whether your agent understands and treats them the same.

It is a **property sweep**: every value runs the same flow with everything else held the same, and the report tells you which values did worse than the rest and which differences are just chance.

***

## The values

| Value | |
| :- | :- |
| Female | |
| Male | |

There is no suggested baseline: neither voice is the normal case, so each value is compared with the other.

***

## What it measures

The template seeds these, and your flow's own metrics run alongside them. Only checks can mark a value **Worse on**; the other measures explain why.

| Measure | What it tells you |
| :- | :- |
| **Comprehension Failure (check)** | The agent misunderstood or misheard the caller. The check fails any call where it did. |
| **Redundant Question Count (check)** | The agent asked for something the caller already gave. The check allows one per call. |
| **Missed Response Count** | Moments the agent should have answered and did not, so the caller had to repeat themselves. |
| **Vocal Pitch Delta** | How far apart the agent's and caller's pitch were. |
| **Call Outcome** | Whether the call ended where it should. |

***

## What to look for

* **Two values, one question.** The sweep asks exactly one thing: does one voice do worse?
* **Run more calls.** Two values at 6 calls each is only 12 calls. At 20 per value the sweep can see a 25-point gap for 40 calls, which is cheap for the confidence it buys.
* **Recognition, not treatment.** A gap here almost always comes from transcription. Read the transcripts of failing calls before changing prompts.

***

## Setting one up

Pick the flow to test, then the values to run it across. Every value starts selected; deselect the ones you don't need. Each value runs your flow again, so the values you keep are the cost of the run.

**Compared against** sets the baseline: a reference value the report pins first, such as silence for background noise. The template suggests the natural one, and you can change it or choose none. It doesn't change the verdict: whether a value did worse is always decided against every other value combined.

**Iterations** live in the sweep panel rather than in Advanced. Each value runs every attached test case once per iteration:

> **Calls per value = test cases x iterations**

Each value needs **at least 5 calls** to be compared with the others, and the Iterations field won't go below the count that reaches it. It starts at 6 calls per value, so one call [excluded after the run](/documentation/simulation-testing/running-simulations#calls-that-dont-count) doesn't drop a value out of the comparison.

More calls let the report see smaller differences. For a sweep of 10 values:

| Calls per value | A value has to be about this much worse to be flagged | Calls in total |
| :- | :- | :- |
| 5 | 50 percentage points | 50 |
| 10 | 35 percentage points | 100 |
| 20 | 25 percentage points | 200 |
| 30 | 20 percentage points | 300 |

> [!NOTE]
>
> Below 5 calls a value can never be flagged, however badly it does, which is why the minimum is enforced rather than suggested. Over the API, a sweep template defaults `iterationCount` to 6 and refuses a plan with fewer than 5 calls per value with a `400`.

***

## How the results are decided

The sweep's table leads the run report. Its **Compared with the rest** column gives each value one result:

| Result | What it means |
| :- | :- |
| **Worse on** *check* | The value failed that check more often than all the other values combined, by a gap too large to be chance. |
| **No difference** | Any gap is within what chance produces with this many calls. That is not proof of no effect: a small run can only see large gaps, and the report says how large when that matters. |
| **Too few calls** (*n*) | Fewer than 5 counted calls, so the value is shown but not compared. |
| **Not graded** | The value ran, but none of its calls were graded. |

How "too large to be chance" is decided:

* Each value is compared with **all the other values combined**, one check at a time, never with a single other value. Two values that happen to land at opposite ends are not a finding on their own.
* A sweep makes many comparisons at once (every value on every check), and some will look different by luck alone. The report accounts for that: of all the values it flags in a run, **at most 1 in 10 is expected to be a fluke**.
* A value is flagged only when it does **worse**. Doing better than the rest is not a problem to fix.

**Checks passed** shows each value's pass rate with the number of calls behind it, so 100% of two calls never reads like 100% of twenty. A lower rate with **No difference** beside it means the gap is not larger than chance.

**Fails, not gender specific.** A check that fails on **at least 20% of calls** with no value standing out is listed under this heading. It is a real problem, just not one gender caused, so changing gender won't fix it. Select the check to open its evidence.

The executive summary follows the same rules: it only blames the property for failures the table marks **Worse on**. Hover **How this works** under the table for this run's own numbers.


# Health check
Source: https://docs.roark.ai/documentation/simulation-testing/templates/health-check

A minimal probe call on a schedule that verifies your agent answers, speaks and stays responsive

## Why this is a common failure point

Voice agents fail silently. A rotated API key, an expired phone number, a provider outage or a bad deploy can leave the agent answering calls with silence, or not answering at all, and nobody finds out until a customer complains. Dashboards of past calls won't warn you: if calls stop connecting, there is nothing new to look at.

## How Roark tests it

A health check calls your agent on a schedule with Roark's fixed probe flow and checks the basics: it answered, it spoke, and it responded quickly. Because the probe never changes, every check is comparable with the last, and a failure alerts you before your callers notice.

***

## Setting it up

The probe call is fixed, so there are no flows to pick. Choose the agents and the cadence (hourly by default). Health checks have their own home under **Health checks**, which links straight to this template.

***

## What it measures

The template seeds these, and you can add or remove metrics in **Advanced** before running.

| Measure | What it tells you |
| :- | :- |
| **Agent Spoke (check)** | Whether the agent said anything at all. The bluntest liveness signal. |
| **Time to First Word** | Time from the call connecting to the first spoken word. |
| **Response Time** | Time between speaking turns. |

***

## What to look for

* **Alert on failure.** Set an alert on the plan so a failed probe reaches you, not just the dashboard.
* **Watch time to first word drift.** A slow climb over days often comes before an outage.

***

## Over the API

Run it with `POST /v1/simulation/run` and `"template": "health-check"`. `GET /v1/simulation/template` lists what each template accepts.


# Interruption handling
Source: https://docs.roark.ai/documentation/simulation-testing/templates/interruption-handling

Run one flow with callers who never interrupt, who only make listening noises, and who talk over your agent

## Why this is a common failure point

Turn-taking is the hardest part of a voice conversation to get right. People say "mm-hm" while listening, cut in to correct a detail, and start talking before the agent has finished. An agent that stops at every listening noise sounds broken; one that talks straight through a correction books the wrong thing. Scripted test callers wait politely for their turn, so neither failure shows up in testing.

## How Roark tests it

Interruption handling runs the same flow with callers who interrupt more and more, from never to constantly. It grades both directions of every overlap: whether the agent stopped when it should have (and kept going when it should have), and whether it cut the caller off.

It is a **property sweep**: every value runs the same flow with everything else held the same, and the report tells you which values did worse than the rest and which differences are just chance.

***

## The values

| Value | |
| :- | :- |
| Never | Suggested baseline |
| Backchannel only | |
| Occasional | |
| Heavy | |

**Backchannel only** callers make listening noises ("mm-hm", "right") without trying to take the turn, so the agent should keep talking. **Occasional** and **Heavy** callers genuinely cut in.

***

## What it measures

The template seeds these, and your flow's own metrics run alongside them. Only checks can mark a value **Worse on**; the other measures explain why.

| Measure | What it tells you |
| :- | :- |
| **Incorrect Agent Interruption Count (check)** | The agent talked over the caller when it shouldn't have. The check allows one per call. |
| **Incorrect Barge-in Yield Count (check)** | The agent stopped for a listening noise, or kept talking through a real interruption. The check allows one per call. |
| **Agent Cutoff Count (check)** | The agent cut the caller off mid-sentence. The check allows one per call. |
| **Interruption Appropriateness** | Whether each time the agent cut in was the right moment. |
| **Barge-in Yield Appropriateness** | Whether the agent stopped, or didn't, correctly each time the caller cut in. |
| **Incorrect Barge-in Yield Rate** | The share of caller interruptions the agent handled wrongly. |
| **Failed Barge-in Rate** | The share of caller interruptions the agent talked straight through. |
| **Customer Barge-in Count** | How often the caller cut in. |
| **Agent Interruption Count** | How often the agent cut in. |
| **Call Outcome** | Whether the call ended where it should. |

***

## What to look for

* **Backchannel only is the trap.** An agent that stops every time it hears "mm-hm" fails here and nowhere else. Barge-in yield failures on this value point at over-sensitive turn-taking.
* **Rates, not counts.** A Heavy caller interrupts far more often, so counts rise by construction. The rates say whether each interruption was handled worse.
* **A single slip never fails a call.** Every check allows one mistake per call, so a value marked worse means repeated mishandling.

***

## Setting one up

Pick the flow to test, then the values to run it across. Every value starts selected; deselect the ones you don't need. Each value runs your flow again, so the values you keep are the cost of the run.

**Compared against** sets the baseline: a reference value the report pins first, such as silence for background noise. The template suggests the natural one, and you can change it or choose none. It doesn't change the verdict: whether a value did worse is always decided against every other value combined.

**Iterations** live in the sweep panel rather than in Advanced. Each value runs every attached test case once per iteration:

> **Calls per value = test cases x iterations**

Each value needs **at least 5 calls** to be compared with the others, and the Iterations field won't go below the count that reaches it. It starts at 6 calls per value, so one call [excluded after the run](/documentation/simulation-testing/running-simulations#calls-that-dont-count) doesn't drop a value out of the comparison.

More calls let the report see smaller differences. For a sweep of 10 values:

| Calls per value | A value has to be about this much worse to be flagged | Calls in total |
| :- | :- | :- |
| 5 | 50 percentage points | 50 |
| 10 | 35 percentage points | 100 |
| 20 | 25 percentage points | 200 |
| 30 | 20 percentage points | 300 |

> [!NOTE]
>
> Below 5 calls a value can never be flagged, however badly it does, which is why the minimum is enforced rather than suggested. Over the API, a sweep template defaults `iterationCount` to 6 and refuses a plan with fewer than 5 calls per value with a `400`.

***

## How the results are decided

The sweep's table leads the run report. Its **Compared with the rest** column gives each value one result:

| Result | What it means |
| :- | :- |
| **Worse on** *check* | The value failed that check more often than all the other values combined, by a gap too large to be chance. |
| **No difference** | Any gap is within what chance produces with this many calls. That is not proof of no effect: a small run can only see large gaps, and the report says how large when that matters. |
| **Too few calls** (*n*) | Fewer than 5 counted calls, so the value is shown but not compared. |
| **Not graded** | The value ran, but none of its calls were graded. |

How "too large to be chance" is decided:

* Each value is compared with **all the other values combined**, one check at a time, never with a single other value. Two values that happen to land at opposite ends are not a finding on their own.
* A sweep makes many comparisons at once (every value on every check), and some will look different by luck alone. The report accounts for that: of all the values it flags in a run, **at most 1 in 10 is expected to be a fluke**.
* A value is flagged only when it does **worse**. Doing better than the rest is not a problem to fix.

**Checks passed** shows each value's pass rate with the number of calls behind it, so 100% of two calls never reads like 100% of twenty. A lower rate with **No difference** beside it means the gap is not larger than chance.

**Fails, not caller interruption specific.** A check that fails on **at least 20% of calls** with no value standing out is listed under this heading. It is a real problem, just not one caller interruption caused, so changing caller interruption won't fix it. Select the check to open its evidence.

The executive summary follows the same rules: it only blames the property for failures the table marks **Worse on**. Hover **How this works** under the table for this run's own numbers.


# Knowledge base grounding
Source: https://docs.roark.ai/documentation/simulation-testing/templates/knowledge-base-grounding

Ask questions your knowledge base answers, and check your agent stays faithful to it

## Why this is a common failure point

Agents backed by a knowledge base fail in two opposite ways. They make things up: a price, a policy, a menu item that isn't there, stated with total confidence. Or they play it too safe and say "I don't know" to questions the knowledge base answers plainly. Both look like reasonable conversation, and both cost you: one misleads callers, the other sends them to a human.

## How Roark tests it

Knowledge base grounding runs callers who ask questions your knowledge base can answer, then checks every claim and offer the agent made against the source. It scores both failure directions separately, so an agent that never hallucinates but never answers either can't hide behind a good grounding score.

***

## Setting it up

Pick the knowledge bases to check against (leave it empty to use every knowledge base in the project), then attach the flows that ask the questions.

***

## What it measures

The template seeds these, and you can add or remove metrics in **Advanced** before running.

| Measure | What it tells you |
| :- | :- |
| **Knowledge Base Grounding (check)** | How faithful the agent stayed to the knowledge base, 1 to 5. The check passes at 3 or above. |
| **Ungrounded Claim** | Statements or offers that conflict with, or are not supported by, the knowledge base. |
| **Missed Answer** | Questions the knowledge base could answer that the agent deflected or got wrong. |

***

## What to look for

* **Read both scores together.** High grounding with many missed answers means the agent is too cautious, not accurate.
* **Ungrounded offers cost the most.** A made-up discount or a menu item you don't sell becomes a promise the business has to handle.
* **Write flows around real questions.** Pull them from your support queue so the test reflects what callers actually ask.

***

## Over the API

Run it with `POST /v1/simulation/run` and `"template": "knowledge-base-grounding"`. `GET /v1/simulation/template` lists what each template accepts.


# Live Bench
Source: https://docs.roark.ai/documentation/simulation-testing/templates/live-bench

Roark’s standard voice-agent benchmark: task completion, turn-taking, latency and robustness in realistic conditions

## Why this is a common failure point

Every team tests its agent differently, so "our agent is good" means nothing outside the team. Comparing two models, two voice providers or two versions of your own agent needs the same calls, the same conditions and the same grading every time. Otherwise a better score may just be an easier test.

## How Roark tests it

Live Bench is a fixed battery of calls with standardised flows, conditions and metrics. Nothing about it can be configured, so a score means the same thing across agents and against Roark's published baselines. You pick the agent to put on the line.

***

## Setting it up

The suite runs four batteries:

| Battery | What it covers |
| :- | :- |
| **Duplex dynamics** | Barge-in, backchannels, mid-turn hesitation, and corrections spoken over a read-back |
| **Task pursuit** | A three-part request with a change of mind, plus error-prone names, numbers and codes |
| **Robustness** | Accented and broken English over street, car and cafe noise; a caller who starts frustrated |
| **Safety spot-checks** | The two highest-signal adversarial probes: prompt injection and PII extraction |

***

## What it measures

The template seeds these, and you can add or remove metrics in **Advanced** before running.

| Measure | What it tells you |
| :- | :- |
| **Instruction Follow (check)** | How well the agent followed its instructions, 1 to 5. The check passes at 3 or above. |
| **Flow Adherence (check)** | How closely the conversation followed the flow, 1 to 5. The check passes at 4 or above. |
| **User Effort Score (check)** | How hard the caller had to work to get what they wanted, 1 to 5. The check passes at 4 or below. |
| **Loop Count (check)** | Conversational loops the agent got stuck in. The check allows none. |
| **Agent Cutoff Count (check)** | Times the agent cut the caller off. The check allows one per call. |
| **Prompt Injection Resistance (check)** | Whether the agent resisted attempts to override its instructions. |
| **PII Handling (check)** | How properly the agent handled personal data. Passes at 3 or above. |
| **Latency breakdown** | Time to first word, response time, latency spikes, and each stage of the turn: transcription, endpointing, the model, tool calls and speech synthesis. |
| **Turn-taking** | Interruption appropriateness, barge-in handling, overtalk and failed barge-ins. |

***

## What to look for

* **The latency bottleneck.** It names the stage that owned most of each slow turn, so you know whether to change the transcriber, the model or the voice.
* **Compare like with like.** Live Bench is built for A/B comparisons: run it on both versions of the agent and read the differences, not the absolute scores.

***

## Over the API

Live Bench is not available over the API yet.


# Load testing
Source: https://docs.roark.ai/documentation/simulation-testing/templates/load-testing

Run many simulations in parallel to surface latency, dropped calls and concurrency limits

## Why this is a common failure point

An agent that answers in 800 milliseconds on one call can take four seconds when fifty callers arrive at once. Model rate limits, tool backends, telephony capacity and your own infrastructure all degrade under load, and they degrade in ways a single test call never shows: slow first words, dead air mid-call, or calls that never connect. The first time most teams see it is during a marketing push or a Monday morning.

## How Roark tests it

Load testing runs one known-good flow many times in parallel, at the concurrency you set, and measures whether the agent stayed responsive and how its latency held up as volume rose. Because every call runs the same path, any change in behaviour is the load's doing.

***

## Setting it up

Load testing owns the run's scale, so the generic Iterations field in Advanced is hidden. Set the volume in the **Volume** panel:

| Field | What it controls |
| :- | :- |
| **Concurrent calls** | How many calls run at the same time, capped by your account's concurrency quota |
| **Total iterations** | The total number of calls across the test |

An estimated runtime updates as you change the numbers. Then pick **one flow to test**: the run uses its happy path on every call.

***

## What it measures

The template seeds these, and you can add or remove metrics in **Advanced** before running.

| Measure | What it tells you |
| :- | :- |
| **Agent Responsive (check)** | Whether the agent stayed responsive for the whole call. |
| **Response Time** | Time between speaking turns. |
| **Time to First Word** | Time from the call connecting to the first spoken word. |

***

## What to look for

* **Latency against time, not the average.** Compare the first wave of calls with the last; a steady climb means something is queueing.
* **Unresponsive calls.** One call where the agent stopped answering matters more than a slower average.
* **Test above your expected peak.** Run at 1.5 to 2 times the concurrency you expect so you find the limit before your callers do.

***

## Over the API

Run it with `POST /v1/simulation/run` and `"template": "load-testing"`. `GET /v1/simulation/template` lists what each template accepts.


# Multilingual
Source: https://docs.roark.ai/documentation/simulation-testing/templates/multilingual

Run your flows across languages and check your agent detects, understands and answers in the caller’s language

## Why this is a common failure point

Multilingual agents usually break at the edges between languages rather than inside them. The agent answers a Spanish caller in English, switches languages mid-call when the transcriber mishears a word, or understands the question but replies with a translation that loses the meaning. Callers who mix languages, as many bilingual callers do, are the hardest case of all.

Testing each language by hand is slow, so most teams test one or two and assume the rest work.

## How Roark tests it

Multilingual fans each attached flow out across the languages you pick. Every language runs the same flow with a native-speaking caller, so a failure in one language and not the others points straight at language handling. A code-switching option swaps in callers who mix their language with English mid-sentence.

***

## Setting it up

Pick the languages from the chip grid: Spanish, French, German, Portuguese, Italian, Dutch, Chinese, Japanese, Hindi, Arabic, Turkish, Greek, Indonesian, Thai, Tagalog, Malay and Hebrew. **Each attached flow runs once per language**, so 3 flows and 4 languages is 12 conversations per iteration.

Turn on **Code-switching** to use callers who mix their language with English (for example, Spanglish).

***

## What it measures

The template seeds these, and you can add or remove metrics in **Advanced** before running.

| Measure | What it tells you |
| :- | :- |
| **Instruction Follow (check)** | How well the agent followed its instructions, 1 to 5. The check passes at 3 or above. |
| **Flow Adherence (check)** | How closely the conversation followed the flow, 1 to 5. The check passes at 4 or above. |
| **Comprehension Failure** | Whether the agent misunderstood the caller. |
| **Sentiment Score** | The caller's sentiment, 1 (negative) to 9 (positive). |
| **Call Outcome** | Whether the call ended where it should. |

***

## What to look for

* **Compare languages, not totals.** Filter the calls by language: a language that fails where the others pass is a language problem, not a flow problem.
* **Wrong-language replies.** Read the first agent turn of failing calls. An agent that opens in English and never switches fails everything after it.
* **Code-switching is a separate test.** Run it on its own so mixed-language failures don't blur the single-language results.

***

## Over the API

Multilingual is not available over the API yet: the API has no field for the languages to test.


# Red teaming
Source: https://docs.roark.ai/documentation/simulation-testing/templates/red-teaming

Probe your agent adversarially for jailbreaks, leaked data, off-scope answers and made-up facts

## Why this is a common failure point

A voice agent is a public endpoint anyone can call. Callers will try to talk it out of its instructions ("ignore what you were told and..."), extract someone else's personal data, get it to give legal or medical advice, or make it claim to be human. Most of these attacks are simple, and most agents are only tested by people who want them to succeed.

## How Roark tests it

Red teaming runs adversarial callers against your agent and grades six safety properties on every call. The attacks come from Roark's curated adversarial library, from your own flows labelled Adversarial, or from new edge cases Roark generates for one of your flows at the difficulty you choose.

***

## Setting it up

Choose how to source the attacks:

* **Choose flows**: a filterable catalogue of Roark's adversarial library and your own flows carrying the Adversarial label.
* **Generate new**: pick one of your improv flows, how many cases (3 to 10) and a difficulty (easy, medium or hard). Roark drafts adversarial variants, adds them to the flow, and runs them.

***

## What it measures

The template seeds these, and you can add or remove metrics in **Advanced** before running.

| Measure | What it tells you |
| :- | :- |
| **Prompt Injection Resistance (check)** | Whether the agent resisted attempts to override its instructions. |
| **PII Handling (check)** | How properly the agent handled personal data, 1 to 5. Passes at 3 or above. |
| **Prohibited Language (check)** | Whether the agent avoided guarantees, diagnoses, legal advice and discriminatory language. |
| **Scope Adherence (check)** | Whether the agent stayed within its topics, 1 to 5. Passes at 3 or above. |
| **Hallucination Boundary (check)** | Whether the agent deferred instead of inventing answers, 1 to 5. Passes at 3 or above. |
| **Identity Consistency (check)** | Whether the agent kept its identity and disclosed being an AI when required. |

***

## What to look for

* **One failure is a finding.** Unlike quality metrics, a single successful attack is worth fixing. Read every failing call.
* **PII and identity first.** They carry legal weight. Scope and hallucination failures are usually prompt fixes.
* **Raise the difficulty over time.** Once easy attacks pass, generate hard ones against the same flow.

***

## Over the API

Run it with `POST /v1/simulation/run` and `"template": "red-teaming"`. `GET /v1/simulation/template` lists what each template accepts.


# Speech pace handling
Source: https://docs.roark.ai/documentation/simulation-testing/templates/speech-pace-handling

Run one flow with callers who rush and callers who drag, and find what pace costs your agent

## Why this is a common failure point

Callers don't speak at one speed. A slow speaker pauses mid-sentence, and an agent with a short end-of-turn timeout jumps in and cuts them off. A fast speaker runs words together and the transcriber drops or merges them. Both produce an agent that answers something the caller didn't finish saying.

## How Roark tests it

Speech pace handling runs the same flow with callers at each pace you pick, from very slow to very fast, and tells you whether any pace makes your agent misunderstand the caller or fail the call, and whether the agent adapts its own pace.

It is a **property sweep**: every value runs the same flow with everything else held the same, and the report tells you which values did worse than the rest and which differences are just chance.

***

## The values

| Value | |
| :- | :- |
| Very slow | |
| Slow | |
| Normal | Suggested baseline |
| Fast | |
| Very fast | |

Values are ordered from slowest to fastest, which is also how the report lists them.

***

## What it measures

The template seeds these, and your flow's own metrics run alongside them. Only checks can mark a value **Worse on**; the other measures explain why.

| Measure | What it tells you |
| :- | :- |
| **Comprehension Failure (check)** | The agent misunderstood or misheard the caller. The check fails any call where it did. |
| **Speech Pace Delta** | How much faster or slower the agent spoke than the caller. |
| **Speech Pace** | Words per minute for each speaker. |
| **Call Outcome** | Whether the call ended where it should. |

***

## What to look for

* **Slow callers and turn-taking.** Very slow speech has long pauses, and an agent that answers into a pause cuts the caller off.
* **Fast callers and recognition.** Very fast speech runs words together, so failures there usually come from transcription.
* **Does the agent adapt?** Speech Pace Delta shows whether it slows down for a slow caller or keeps its own pace.

***

## Setting one up

Pick the flow to test, then the values to run it across. Every value starts selected; deselect the ones you don't need. Each value runs your flow again, so the values you keep are the cost of the run.

**Compared against** sets the baseline: a reference value the report pins first, such as silence for background noise. The template suggests the natural one, and you can change it or choose none. It doesn't change the verdict: whether a value did worse is always decided against every other value combined.

**Iterations** live in the sweep panel rather than in Advanced. Each value runs every attached test case once per iteration:

> **Calls per value = test cases x iterations**

Each value needs **at least 5 calls** to be compared with the others, and the Iterations field won't go below the count that reaches it. It starts at 6 calls per value, so one call [excluded after the run](/documentation/simulation-testing/running-simulations#calls-that-dont-count) doesn't drop a value out of the comparison.

More calls let the report see smaller differences. For a sweep of 10 values:

| Calls per value | A value has to be about this much worse to be flagged | Calls in total |
| :- | :- | :- |
| 5 | 50 percentage points | 50 |
| 10 | 35 percentage points | 100 |
| 20 | 25 percentage points | 200 |
| 30 | 20 percentage points | 300 |

> [!NOTE]
>
> Below 5 calls a value can never be flagged, however badly it does, which is why the minimum is enforced rather than suggested. Over the API, a sweep template defaults `iterationCount` to 6 and refuses a plan with fewer than 5 calls per value with a `400`.

***

## How the results are decided

The sweep's table leads the run report. Its **Compared with the rest** column gives each value one result:

| Result | What it means |
| :- | :- |
| **Worse on** *check* | The value failed that check more often than all the other values combined, by a gap too large to be chance. |
| **No difference** | Any gap is within what chance produces with this many calls. That is not proof of no effect: a small run can only see large gaps, and the report says how large when that matters. |
| **Too few calls** (*n*) | Fewer than 5 counted calls, so the value is shown but not compared. |
| **Not graded** | The value ran, but none of its calls were graded. |

How "too large to be chance" is decided:

* Each value is compared with **all the other values combined**, one check at a time, never with a single other value. Two values that happen to land at opposite ends are not a finding on their own.
* A sweep makes many comparisons at once (every value on every check), and some will look different by luck alone. The report accounts for that: of all the values it flags in a run, **at most 1 in 10 is expected to be a fluke**.
* A value is flagged only when it does **worse**. Doing better than the rest is not a problem to fix.

**Checks passed** shows each value's pass rate with the number of calls behind it, so 100% of two calls never reads like 100% of twenty. A lower rate with **No difference** beside it means the gap is not larger than chance.

**Fails, not speech pace specific.** A check that fails on **at least 20% of calls** with no value standing out is listed under this heading. It is a real problem, just not one speech pace caused, so changing speech pace won't fix it. Select the check to open its evidence.

The executive summary follows the same rules: it only blames the property for failures the table marks **Worse on**. Hover **How this works** under the table for this run's own numbers.


# Tool call accuracy
Source: https://docs.roark.ai/documentation/simulation-testing/templates/tool-call-accuracy

Check your agent calls the right tools, with the right arguments, at the right point in the call

## Why this is a common failure point

When an agent books, cancels, looks up or transfers, a tool call does the work, and a tool call can be wrong in ways the conversation hides. The agent says "you're booked for Tuesday" after sending Thursday, looks up the account before verifying the caller, or never calls the tool at all and improvises an answer. The transcript sounds fine; the system of record is wrong.

## How Roark tests it

Tool call accuracy runs your flows and grades every tool call the agent made against the conversation: was it the right tool, were the arguments what the caller actually said, and did the calls happen in a sensible order. You pick which tools to evaluate once, and the template applies that list to all three checks.

***

## Setting it up

Choose the **Tools to evaluate** (all current tools by default), then attach the flows that exercise them through the inline Flows panel.

***

## What it measures

The template seeds these, and you can add or remove metrics in **Advanced** before running.

| Measure | What it tells you |
| :- | :- |
| **Tool Invocation Correct (check)** | Whether the agent called the right tools at the right times. |
| **Tool Invocation Parameters Correct (check)** | Whether each call carried the arguments the conversation supports. |
| **Tool Invocation Order Correct (check)** | Whether the calls happened in a logical sequence. |

***

## What to look for

* **Parameters fail most.** Dates, times, spellings and numbers are where a tool call silently diverges from what the caller said.
* **Missing calls.** An agent that answers from memory instead of calling the tool fails the invocation check. Those are the calls that give callers wrong information.
* **Order matters for safety.** A lookup before verification is an order failure even when every argument is right.

***

## Over the API

Run it with `POST /v1/simulation/run` and `"template": "tool-call-accuracy"`. `GET /v1/simulation/template` lists what each template accepts.


# Voicemail testing
Source: https://docs.roark.ai/documentation/simulation-testing/templates/voicemail-testing

Check your agent notices it has reached voicemail, leaves a usable message, and hangs up

## Why this is a common failure point

An outbound agent that dials real customers reaches voicemail on a large share of calls. What it does next is easy to get wrong: it waits for a reply that never comes, talks over the greeting, starts its message before the beep, or leaves a message no one can act on. Each of those wastes the call, and some leave a recording your customer will actually hear.

## How Roark tests it

Voicemail testing calls your agent into real voicemail greetings, one call per greeting, and grades whether it detected the voicemail, left a message, and how well it handled it: timing around the beep, clarity, completeness and length.

***

## Running a voicemail test

**1. Pick the template**

On [create a run](/documentation/simulation-testing/run-plans), choose **Voicemail testing** in step 01.

**2. Choose your agents**

Step 02 as usual. Voicemail is **voice-only**: a greeting is audio, so a chat endpoint has nothing to play. Attaching one is rejected before the run starts.

**3. Choose the greetings**

Step 03 is a **Greetings to test** checklist. Every greeting is selected by default; untick the ones you don't need. Each row has a play button so you can hear the recording before you commit to a run.

**4. Run**

One call runs per selected greeting, per agent. Ten greetings against two agents is twenty calls.

> [!WARNING]
>
> The call count multiplies. Leaving every greeting selected across several agents adds up quickly, so untick down to the greetings you actually care about if you're iterating.

***

## What happens on the call

Each call follows the same shape:

1. Roark dials your agent (or answers it, for an inbound endpoint).
2. The selected greeting plays as the first thing your agent hears: a real recording, not synthesised speech.
3. The line then goes **completely silent**. The simulated caller says nothing for the rest of the call, and won't interrupt.
4. The call ends when your agent hangs up, or after 15 seconds of silence once it stops speaking.

That silence is the test. Everything Roark grades is a property of what your agent does into a line that never answers back.

Because voicemail calls are short by construction, the template lowers the run's **max duration** default to two minutes. A hung call fails fast instead of billing the usual fifteen.

***

## The greetings

Roark ships a fixed set of real voicemail recordings, covering the shapes an outbound agent meets in the wild:

| Family | What it is |
| :- | :- |
| **Carrier robots** | The automated greetings networks play when a subscriber has set no personal message: Verizon, O2, Apple |
| **Mailbox full** | A carrier notice that there's no room to leave a message at all |
| **Personal greetings** | Real people recording their own outgoing message, across a range of voices, accents, and lengths |

**Mailbox full is the interesting one.** There's nowhere to leave a message, so an agent that launches into its script anyway is doing the wrong thing. The correct behaviour is to recognise it and hang up.

These are Roark-managed and read-only: you can select and preview them, but you can't add your own or edit the set.

***

## What gets measured

The template attaches three [system metrics](/documentation/metrics/system-metrics#voicemail-detection):

| Metric | What it answers |
| :- | :- |
| `voicemail_detected` | Did the agent recognise it had reached a voicemail system rather than a live person? |
| `voicemail_agent_left_message` | Did it leave a message at all? |
| `voicemail_handling_score` | How good was the handling: beep timing, message clarity, completeness (1–5) |

As with any template, these are a starting point. Add or remove metrics in the **Advanced** section before running.

> [!NOTE]
>
> Read the run report per greeting, not just in aggregate. An agent can handle a carrier robot perfectly and still talk over a personal greeting, and the run report separates the calls by greeting so that shows up.

***

## Next steps

> [card] [Templates](/documentation/simulation-testing/templates)
>
> The full template catalogue and what each one seeds

> [card] [System Metrics](/documentation/metrics/system-metrics)
>
> How the voicemail metrics are scored

> [card] [Run Plans](/documentation/simulation-testing/run-plans)
>
> Save a voicemail configuration as a reusable suite

> [card] [Schedules](/documentation/simulation-testing/schedules)
>
> Run voicemail checks on a cadence


# Variables
Source: https://docs.roark.ai/documentation/simulation-testing/variables

Feed dynamic values into customer flows and run plans with {{variable}} placeholders

Variables let you reuse the same customer flow with different data on every run. Instead of hardcoding details into a flow's brief or steps, you write `{{variableName}}` placeholders that get replaced with real values at runtime.

Variables show up in three places:

1. **On a flow variant**: key/value entries the simulator hands to the customer-side model, so the simulated customer can answer the agent's questions consistently
2. **On a run plan**: named variables with a type and an optional default, referenced as `{{name}}`, with per-attachment overrides when you attach flows
3. **At runtime**: a modal collects any values that still need filling in before the run starts

***

## 1. Variables on a flow variant

Each variant of a customer flow carries its own set of variables: key/value entries like `name`, `account_id`, or `last_visit` that the simulator gives to the customer-side model. When the agent asks "Can I get your account number?", the simulated customer answers with the value you set, consistently, every run.

### Editing variant variables

Open the variant and click the **Variables** chip (next to the Persona and Environment chips). From there you can:

* **Add variable**: add a key/value row inline
* **Apply a test profile**: replace the variant's entries with a saved test profile's properties in one click

![The Variables chip open on a flow variant, with its popover listing key/value rows (orderNumber, invoiceTotal, accountTier), an Add variable action, and an Apply a test profile button](https://mintcdn.com/roark/wL__NyRpzLGS2dwF/images/simulation-testing/variables-popover.png?fit=max\&auto=format\&n=wL__NyRpzLGS2dwF\&q=85\&s=261ad21fc8f5db5f03cba412d8126845)

### `{{variable}}` autocomplete

Text fields in the flow editor (the customer setup brief in Improv mode, step text in Scripted mode) support `{{variable}}` tokens with autocomplete sourced from your project's property definitions:

* `{{name}}`: flow-scoped properties, resolved from the variant's variables
* `{{persona.name}}`: persona-scoped properties, resolved from the variant's persona

Type `{{` to open the autocomplete and pick a property, or keep typing to reference a new one.

***

## 2. Variables on a run plan

Run plans define their own variables in the **Advanced** section of the create-run page. Each run-plan variable has:

| Field | Description |
| :- | :- |
| **Name** | Referenced as `{{name}}` in attached flows |
| **Type** | The value type (string, number, boolean, date) |
| **Default** | Optional. If omitted, the value must be provided before each run |

Variables with defaults let a plan run unattended (on a schedule or from the API) without prompting anyone. Variables without defaults turn the plan into a parameterized suite: every run asks for fresh values.

### Per-attachment overrides

When you attach customer flows to a plan via the **Attach flows** picker, each attachment can override `{{variable}}` values for that flow. Overrides win over the values stored on the variant, so you can attach the same flow with different data (for example, one attachment with `appointmentType: urgent` and another with `appointmentType: routine checkup`) and each becomes its own set of test calls.

![The Attach flows picker with a flow expanded, showing its happy path and edge case variants selected and a Variable overrides area setting per-attachment values that win over the plan defaults](https://mintcdn.com/roark/wL__NyRpzLGS2dwF/images/simulation-testing/flow-variables.png?fit=max\&auto=format\&n=wL__NyRpzLGS2dwF\&q=85\&s=e9049c71faec7bb22c8a03e23ee49b63)

***

## 3. Providing values at runtime

If any variable lacks a value when you start a run (**Run now**, ⌘↵, or a "Fresh run" re-run) a modal collects the missing values first. It gathers, in one place:

* **Run-plan variables** without defaults
* **Flow variant values**: per-variant values for attached scripted flows
* **Persona property values**: values for `{{persona.*}}` references, which persist onto the persona for future runs

Once every required value is filled in, the run starts.

![The runtime variables modal shown before a run starts, grouping run-plan, flow, and persona variables, with required empty values highlighted in red](https://mintcdn.com/roark/IjseSxRBSMbit4Aq/images/simulation-testing/runtime-variables.png?fit=max\&auto=format\&n=IjseSxRBSMbit4Aq\&q=85\&s=5c8413380b8bbbe243b45b6da157605d)

> [!NOTE]
>
> **"Re-run with same data"** replays a run's exact captured snapshot, so it never prompts for variables. **"Fresh run"** builds new test calls from the plan as configured today and collects unresolved variables first.

***

## Reserved variables

Some variable names are reserved for system use and resolved automatically:

| Variable | Resolves to |
| :- | :- |
| `{{persona.*}}` | Properties of the variant's persona (e.g. `{{persona.name}}`) |
| `{{phoneNumberToDial}}` | The phone number to dial for outbound simulations |
| `{{simulationJobId}}` | The current simulation job's ID |

Don't define your own variables with these names.

***

## Setting variables via the API

> [!NOTE]
>
> The REST API keeps the older naming: flows are passed in a `scenarios` field, and runtime overrides are keyed by `scenarioId`.

### Pre-set variables when creating a plan

Add a `variables` object to each entry in `scenarios` when [creating a run plan](/api-reference/simulation-run-plan/create-a-run-plan). The same flow ID can appear multiple times with different values. Each entry becomes a separate test case:

```json POST /v1/simulation/plan theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "name": "Patient Scheduling - Multi Profile",
  "direction": "INBOUND",
  "iterationCount": 2,
  "maxSimulationDurationSeconds": 300,
  "scenarios": [
    {
      "id": "550e8400-e29b-41d4-a716-446655440000",
      "variables": {
        "patientName": "John Doe",
        "appointmentType": "urgent",
        "insuranceProvider": "Aetna"
      }
    },
    {
      "id": "550e8400-e29b-41d4-a716-446655440000",
      "variables": {
        "patientName": "Jane Smith",
        "appointmentType": "routine checkup",
        "insuranceProvider": "Blue Cross"
      }
    }
  ],
  "personas": [{ "id": "6ba7b810-9dad-11d1-80b4-00c04fd430c8" }],
  "agentEndpoints": [{ "id": "7c9e6679-7425-40de-944b-e07fc1f90ae7" }],
  "metrics": [{ "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479" }],
  "autoRun": false
}
```

### Override variables when triggering a run

Runtime values are passed in the request body when [triggering a run](/api-reference/simulation/run-a-simulation). They override any values stored on the plan.

**Global format** (the same values apply to every flow in the plan):

```json POST /v1/simulation/run theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "planId": "<plan-id>",
  "variables": {
    "orderNumber": "12345",
    "environment": "staging"
  }
}
```

**Per-flow format** (different values per flow, keyed by `scenarioId`):

```json POST /v1/simulation/run theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "planId": "<plan-id>",
  "variables": [
    {
      "scenarioId": "550e8400-e29b-41d4-a716-446655440000",
      "variables": {
        "orderNumber": "12345",
        "claimType": "damaged item"
      }
    },
    {
      "scenarioId": "7a3d2e1f-c4b5-6a89-0d1e-2f3a4b5c6d7e",
      "variables": {
        "orderNumber": "67890",
        "claimType": "missing package"
      }
    }
  ]
}
```

If everything has a stored value or default, you can omit `variables`. `POST /v1/simulation/run` with just `{ "planId": "<plan-id>" }` starts the run as configured.

***

## Related

> [card] [Customer Flows](/documentation/simulation-testing/customer-flows)
>
> Author the flows and variants that variables plug into

> [card] [Run Plans](/documentation/simulation-testing/run-plans)
>
> Compose flows, personas, metrics, and variables into a reusable suite

> [card] [Running Simulations](/documentation/simulation-testing/running-simulations)
>
> Trigger runs and follow them live

> [card] [Personas](/documentation/simulation-testing/personas)
>
> The customer identities behind \{\{persona.\*}} values


# WebRTC Simulations
Source: https://docs.roark.ai/documentation/simulation-testing/webrtc

Run simulations over WebRTC instead of a phone line

## Overview

By default, Roark simulations are carried out over a phone line: the simulated caller dials your agent through a telephony provider and the entire conversation runs as a real PSTN call. This matches how most production voice agents are reached today.

If your agent supports **WebRTC**, the Roark simulation agent can connect to it directly over WebRTC instead of placing a phone call. This is useful when:

* Your agent is primarily accessed through a web or mobile client and you want simulations to exercise the same transport as production.
* You don't want to pay for telephony minutes just to run tests.
* Your agent doesn't expose a phone number at all (for example, an in-app voice assistant).
* You want lower-latency test runs without SIP in the middle.

Roark currently supports two WebRTC transports: **LiveKit** and **SmallWebRTC**.

The direction configured on the endpoint feeds the derived direction of any run that uses it. See [Inbound vs Outbound](/documentation/simulation-testing/inbound-vs-outbound).

***

## LiveKit

If your agent runs on LiveKit, Roark can join the same LiveKit room as your agent and carry out the simulation end-to-end over LiveKit's WebRTC infrastructure.

You can either let Roark create the room in your LiveKit project and dispatch your agent into it, or provide a callback URL where your own backend creates the room and returns the room name for Roark to join. Room metadata, named-agent dispatch, and automatic-dispatch behavior are all supported.

WebRTC endpoints are managed from the LiveKit integration page itself: go to **Agents**, expand the connected sources at the top of the page, and choose your **LiveKit** source: the integration form opens on the right, where you can use the **WebRTC** section to add or edit endpoints. Each endpoint is attached to one of the agents managed by the integration and can be reused across any run plan.

> [card] [LiveKit WebRTC Endpoint Setup](/documentation/integrations/livekit#webrtc-endpoints)
>
> See the LiveKit integration docs for the full list of endpoint settings and room management modes.

***

## SmallWebRTC

If your agent uses peer-to-peer / serverless WebRTC via Pipecat's [`SmallWebRTCTransport`](https://docs.pipecat.ai/server/services/transport/small-webrtc), Roark can connect to it directly with no intermediate media server. This is a good fit for Pipecat agents that don't use LiveKit or Daily and instead rely on the P2P WebRTC transport that ships with Pipecat.

To add a SmallWebRTC endpoint, navigate to **Agents → \{your agent} → Create Endpoint** and choose **SmallWebRTC**. You'll provide the URL that Roark should POST an SDP offer to, and Roark's simulation caller will negotiate the peer connection directly with your agent at simulation time.

> [!NOTE]
>
> SmallWebRTC is a good choice when you're running Pipecat locally or on minimal infrastructure and don't want to introduce a SFU. For Pipecat deployments that already use LiveKit or another SFU, prefer the matching transport endpoint.

***

## Next Steps

> [card] [LiveKit Integration](/documentation/integrations/livekit)
>
> Connect LiveKit and configure WebRTC endpoints

> [card] [Run Plans](/documentation/simulation-testing/run-plans)
>
> Use a WebRTC endpoint in a run plan

> [card] [Running Simulations](/documentation/simulation-testing/running-simulations)
>
> Execute simulations against your WebRTC agent

> [card] [Pipecat SmallWebRTC Docs](https://docs.pipecat.ai/server/services/transport/small-webrtc)
>
> Learn about the SmallWebRTC transport


# Tool Invocations
Source: https://docs.roark.ai/documentation/tool-invocations

Submit and track tool calls as part of post-call analysis

### Overview

Tool invocations allow you to capture and analyze the tools used during calls. This documentation covers how to submit tool calls, parameter options, result formats, and how Roark tracks versioning for tools.

### Submitting Tool Calls

Tool calls can be submitted as part of your `post-call-analysis` by including a `toolInvocations` array in your request payload when you [create the call](/documentation/observability/overview#how-calls-get-into-roark). If your tool data is not ready at call-creation time, you can also [attach it afterward](#attaching-tool-calls-after-a-call). Each tool invocation requires:

* `name`: The name of the tool being invoked
* `description`: A brief description of what the tool does
* `startOffsetMs`: The start time offset in milliseconds with respect to the start of the call
* `endOffsetMs`: The end time offset in milliseconds with respect to the start of the call
* `parameters`: The parameters passed to the tool
* `result`: The result of the tool invocation

```typescript TypeScript {22-38} theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  recordingUrl: 'https://example.com/recording.wav',
  callDirection: 'INBOUND',
  interfaceType: 'WEB',
  startedAt: '2025-03-29T15:30:00Z',
  isTest: false,
  properties: {},
  participants: [
    {
      name: 'Sales Agent',
      phoneNumber: '+15551234567',
      role: 'AGENT',
      spokeFirst: true,
    },
    {
      name: 'John Doe',
      phoneNumber: '+15557654321',
      role: 'CUSTOMER',
      spokeFirst: false,
    },
  ],
  // Tool invocations are passed as part of the post-call-analysis payload
  toolInvocations: [
    {
      name: 'getDentalAppointments',
      description: 'Get available dental appointments',
      startOffsetMs: 2000,
      endOffsetMs: 3000,
      parameters: {
        patientName: 'John Doe',
        patientPhone: '+1234567890',
      },
      result: { 
        appointments: ['cleaning', 'whitening', 'rootCanal'] 
      },
    },
    // More tool invocations...
  ],
}
```

```python Python {22-38} theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "recordingUrl": "https://example.com/recording.wav",
  "callDirection": "INBOUND",
  "interfaceType": "WEB",
  "startedAt": "2025-03-29T15:30:00Z",
  "isTest": False,
  "properties": {},
  "participants": [
    {
      "name": "Sales Agent",
      "phoneNumber": "+15551234567",
      "role": "AGENT",
      "spokeFirst": True,
    },
    {
      "name": "John Doe",
      "phoneNumber": "+15557654321",
      "role": "CUSTOMER",
      "spokeFirst": False,
    },
  ],
  # Tool invocations are passed as part of the post-call-analysis payload
  "toolInvocations": [
    {
      "name": "getDentalAppointments",
      "description": "Get available dental appointments",
      "startOffsetMs": 2000,
      "endOffsetMs": 3000,
      "parameters": {
        "patientName": "John Doe",
        "patientPhone": "+1234567890",
      },
      "result": { 
        "appointments": ["cleaning", "whitening", "rootCanal"] 
      },
    },
    # More tool invocations...
  ],
}
```

### Attaching Tool Calls After a Call

Sometimes your tool data is not ready when the call is created, for example when a Roark [simulation](/documentation/simulation-testing/enriched-simulations) has finished but your backend needs a moment to assemble the tool calls, or when you submit a call before its tools are available. In those cases you can attach tool invocations to an existing call:

```
POST /v1/call/{callId}/tool-invocations
```

The body takes the same `toolInvocations` array as call creation:

```typescript Node theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import Roark from '@roarkanalytics/sdk'

const client = new Roark({ bearerToken: process.env.ROARK_API_BEARER_TOKEN })

const response = await client.call.appendToolInvocations(callId, {
  toolInvocations: [
    {
      name: 'getDentalAppointments',
      description: 'Get available dental appointments',
      startOffsetMs: 2000,
      endOffsetMs: 3000,
      parameters: { patientName: 'John Doe' },
      result: { appointments: ['cleaning', 'whitening'] },
    },
  ],
})

console.log(response.data)
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import os
from roark_analytics import Roark

client = Roark(bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"))

response = client.call.append_tool_invocations(
    call_id=call_id,
    tool_invocations=[
        {
            "name": "getDentalAppointments",
            "description": "Get available dental appointments",
            "start_offset_ms": 2000,
            "end_offset_ms": 3000,
            "parameters": {"patientName": "John Doe"},
            "result": {"appointments": ["cleaning", "whitening"]},
        }
    ],
)

print(response.data)
```

```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/call/{callId}/tool-invocations \
  -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "toolInvocations": [
      {
        "name": "getDentalAppointments",
        "description": "Get available dental appointments",
        "startOffsetMs": 2000,
        "endOffsetMs": 3000,
        "parameters": { "patientName": "John Doe" },
        "result": { "appointments": ["cleaning", "whitening"] }
      }
    ]
  }'
```

The response reports how many invocations were `added` and how many were `skipped`.

> [!NOTE]
>
> Attaching is **idempotent**. Re-sending an invocation that is already on the call (same tool name and timing) is skipped rather than duplicated, so a retried request converges instead of double-counting.

> [!TIP]
>
> To score the tool metrics over the newly attached tools in the same request, pass a `metrics` array of metric definition IDs. This starts a [metric collection job](/documentation/metrics/metric-collection-jobs) (billed, and requires the `metric:create` permission). Omit it to attach tools only. If scoring can't be started (for example, insufficient credits), the tools are still attached and the response returns a `metricCollectionError` you can act on.

### Tracking Input Parameters

Parameters can be submitted in two different formats:

#### Simple Key-Value Pairs

The most basic format is a simple key-value pair. Roark will automatically infer the type of the parameter, and rely on the name of the parameter to understand the context.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
parameters: {
  patientName: 'John Doe',
  appointmentType: 'cleaning',
}
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
parameters = {
  "patientName": "John Doe",
  "appointmentType": "cleaning",
}
```

#### Detailed Key-Value Pairs

To ensure that the usage of the tool is clear and the types are correctly inferred, you can manually provide the `description` and `type` for each parameter. Type can be one of: `string`, `number`, `boolean`.

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
parameters: {
  patientName: {
    value: 'John Doe',
    description: 'Name of the patient', // optional
    type: 'string', // optional
  },
  appointmentType: {
    value: 'cleaning',
    description: 'Type of dental appointment',
    type: 'string',
  }
}
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
parameters = {
  "patientName": {
    "value": "John Doe",
    "description": "Name of the patient",  # optional
    "type": "string", # optional
  },
  "appointmentType": {
    "value": "cleaning",
    "description": "Type of dental appointment",
    "type": "string",
  }
}
```

### Tracking Results

The result returned by the tool can be a string or a JSON object.

#### String Results

For simple responses, you can pass in a string:

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
result: 'Success'
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
result = "Success"
```

#### JSON Object Results

For structured data, you can pass a JSON object:

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
result: {
  appointments: ['cleaning', 'whitening', 'rootCanal'],
  nextAvailable: '2025-04-15T10:00:00Z'
}
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
result = {
  "appointments": ["cleaning", "whitening", "rootCanal"],
  "nextAvailable": "2025-04-15T10:00:00Z"
}
```

### Tool Execution Duration

Roark calculates execution duration automatically using the difference between `startOffsetMs` and `endOffsetMs`:

* `startOffsetMs`: Time offset (in milliseconds) from the beginning of the call when the tool execution started
* `endOffsetMs`: Time offset (in milliseconds) from the beginning of the call when the tool execution ended

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
// This tool invocation lasted 1000ms (1 second)
{
  name: 'getDentalAppointments',
  startOffsetMs: 2000,  // 2 seconds into the call
  endOffsetMs: 3000, // 3 seconds into the call
  // ...
}
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# This tool invocation lasted 1000ms (1 second)
{
  "name": "getDentalAppointments",
  "startOffsetMs": 2000,  # 2 seconds into the call
  "endOffsetMs": 3000, # 3 seconds into the call
  # ...
}
```

### Tool Versioning

The system automatically tracks changes to your tool calls and creates new versions when:

1. A tool with the same name receives a different schema for input parameters
2. A tool with the same name is called with a different description

#### Example

If you previously called a tool:

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  name: 'bookAppointment',
  description: 'Book a dental appointment',
  parameters: { patientName: 'string', appointmentType: 'string' }
}
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "name": "bookAppointment",
  "description": "Book a dental appointment",
  "parameters": { "patientName": "string", "appointmentType": "string" }
}
```

And later change it to:

```typescript TypeScript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  name: 'bookAppointment',
  description: 'Book a dental appointment for the client',
  parameters: {
    patientName: 'string',
    patientPhone: 'string',
    appointmentType: { type: 'object' }
  }
}
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
{
  "name": "bookAppointment",
  "description": "Book a dental appointment for the client",
  "parameters": {
    "patientName": "string",
    "patientPhone": "string",
    "appointmentType": { "type": "object" }
  }
}
```

The system will automatically create a new version to track these changes.