API Checks

An API check is a pass/fail metric your own endpoint answers. After each call is analysed, Roark sends a request about the call to your API, and your API replies with { "passed": true | false, "reason": "..." }. It runs on live calls and simulations alike. Every other metric reads the conversation. An API check reads your system of record instead. An agent that says “your PIN has been changed” while the tool call behind it failed passes every transcript judge, but fails an API check that asks your card service whether the PIN actually changed. Use an API check when the outcome lives in your backend:

State changes: the PIN, address, or plan was actually updated.
Records created: a booking, ticket, refund, or callback exists, with the right details.
Side effects: an SMS or email went out, a payment was captured, a case was escalated.

Quickstart

API checks aren’t a config as code kind yet. Create the API check once with the SDK under a stable slug, then collect it from config by that slug. You can also build it in Studio.

Create it with the SDK

Create the metric with calculationType: "HTTP_REQUEST":

API checks need @roarkanalytics/sdk 5.17.0 or later. The Python SDK doesn’t support them yet: until it does, send the request with cURL or any HTTP client.

import Roark from '@roarkanalytics/sdk'

const client = new Roark({ bearerToken: process.env.ROARK_API_BEARER_TOKEN })

const metric = await client.metric.createDefinition({
  calculationType: 'HTTP_REQUEST',
  name: 'PIN changed',
  slug: 'pin_changed',
  httpRequest: {
    url: 'https://api.example.com/roark/checks/pin-changed',
    method: 'POST',
    headers: { Authorization: `Bearer ${process.env.CARD_SERVICE_TOKEN}` },
    body: '{"cardId": "{{call.properties.cardId}}", "expectedPin": "{{simulation.variables.newPin}}"}',
  },
  booleanTrueLabel: 'PIN changed',
  booleanFalseLabel: 'PIN unchanged',
})

import os
from roark_analytics import Roark

client = Roark(bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"))

metric = client.metric.create_definition(
    calculation_type="HTTP_REQUEST",
    name="PIN changed",
    slug="pin_changed",
    http_request={
        "url": "https://api.example.com/roark/checks/pin-changed",
        "method": "POST",
        "headers": {"Authorization": f"Bearer {os.environ['CARD_SERVICE_TOKEN']}"},
        "body": '{"cardId": "{{call.properties.cardId}}", "expectedPin": "{{simulation.variables.newPin}}"}',
    },
    boolean_true_label="PIN changed",
    boolean_false_label="PIN unchanged",
)

curl -X POST https://api.roark.ai/v1/metric/definitions \
  -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "calculationType": "HTTP_REQUEST",
    "name": "PIN changed",
    "slug": "pin_changed",
    "httpRequest": {
      "url": "https://api.example.com/roark/checks/pin-changed",
      "method": "POST",
      "headers": { "Authorization": "Bearer sk_live_..." },
      "body": "{\"cardId\": \"{{call.properties.cardId}}\", \"expectedPin\": \"{{simulation.variables.newPin}}\"}"
    },
    "booleanTrueLabel": "PIN changed",
    "booleanFalseLabel": "PIN unchanged"
  }'

The response is the metric definition, with type: "BOOLEAN" and your httpRequest echoed back.

Field	Type	Required	Description
`httpRequest.url`	string	Yes	Public `http(s)` URL, up to 2048 characters. May contain placeholders, which are URL-encoded
`httpRequest.method`	string	No	`POST` (default) or `GET`. A `GET` request has no body
`httpRequest.headers`	object	No	Up to 20 headers. Values may contain placeholders
`httpRequest.body`	string	No	A JSON template, up to 20,000 characters. Omit it, or pass `null`, to send the default payload
`booleanTrueLabel` / `booleanFalseLabel`	string	No	What a passed and a failed check are called. Default `Pass` and `Fail`
`slug`	string	No	Stable identifier. Generated from the name if omitted
`analysisPackageId`	string	No	Package to add the metric to. Defaults to your project’s Custom Metrics package

To change the request, PUT /v1/metric/definitions/{idOrSlug} with the whole httpRequest. Headers you leave out are removed, and leaving out body switches back to the default payload. Each change creates a new metric version, so earlier results keep the request they ran with. The API answers 400 when:

the URL is not public http(s) (localhost, private, or internal addresses are refused)
a placeholder doesn’t exist in the payload
the body isn’t valid JSON once its placeholders are filled in
a GET request has a body, or there are more than 20 headers

Collect it from config

Once the API check exists, list its slug in a collector next to your other metrics, and Roark scores it on every matching call:

import { defineCollector, defineConfig } from '@roarkanalytics/sdk/lib/config';

const frontdeskQuality = defineCollector({
  name: 'frontdesk-quality',
  modality: 'call',
  status: 'ACTIVE',
  metrics: ['pin_changed'],
});

export default defineConfig({
  resources: [frontdeskQuality],
});

# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: collector
name: frontdesk-quality
modality: call
status: ACTIVE
metrics:
  - pin_changed

roark config apply roark.config.ts

Apply fails with unknown metric "pin_changed" until the API check has been created.

Build it in Studio

Open Studio in Author mode and pick the API check engine. The engine is locked after the first save.

Request

Choose POST or GET, enter your endpoint URL, and add any headers your API needs, such as an Authorization header. Type {{ in the URL, a header, or the body to insert a placeholder.

Body

Leave the body as it is to send the default payload, or edit it into the JSON your endpoint expects. Turn on Preview sent body to see the body with its placeholders filled in for an example simulated call.

Test

Click Test next to the URL to send the request once, filled from the example payload. Studio shows the result (Pass, Fail, Error, or Not sent), the HTTP status, your reason, the body it sent, and your endpoint’s response. Nothing is saved, so you can test before the metric exists.

Labels

Name the two outcomes, for example PIN changed and PIN unchanged. They default to Pass and Fail.

The test fills run variables, call properties, and persona fields the example call doesn’t have with test-<name>, so a placeholder like {{simulation.variables.newPin}} sends test-newPin. Have your endpoint accept those values while you set it up, or check the sent body to confirm each value lands where you expect.

How It Works

The call ends and is analysed

API checks run with the rest of the call’s metrics, from a collector on live calls or from a run plan on simulations.

Roark sends your request

Roark fills your URL, headers, and body with data about the call and sends it. With no body, Roark POSTs the default payload.

Your endpoint answers

Your endpoint replies 200 with { "passed": true | false, "reason": "..." }. Roark records the result as Pass or Fail, with your reason shown next to it.

API checks are free: they don’t use Roark’s evaluation models, so they aren’t billed like LLM Judge metrics.

The Request Payload

With no body, Roark POSTs this payload as JSON. With a custom body, the same payload is what your placeholders read from.

{
  "version": "1.0",
  "requestId": "3eb16131-4444-4fb1-8d84-6c4c6fce5043",
  "metric": { "id": "0b8e3e0e-1111-4c8e-9a51-3f1f3c9b2d10", "name": "PIN changed" },
  "call": {
    "id": "6f1d2c3b-5555-4a6e-8f7a-1b2c3d4e5f60",
    "projectId": "7a2e3d4c-6666-4b7f-9a8b-2c3d4e5f6a71",
    "callDirection": "INBOUND",
    "startedAt": "2026-10-26T14:00:02.000Z",
    "endedAt": "2026-10-26T14:03:06.000Z",
    "durationMs": 184000,
    "endedStatus": "CUSTOMER_ENDED_CALL",
    "externalId": "your-call-id",
    "agents": [
      {
        "id": "8b3f4e5d-7777-4c8a-ab9c-3d4e5f6a7b82",
        "endpoint": { "id": "9c4a5f6e-8888-4d9b-bcad-4e5f6a7b8c93", "environment": "production", "phoneNumberE164": "+14155550100" }
      }
    ],
    "customers": [{ "phoneNumberE164": "+14155550199", "label": null }],
    "properties": { "cardId": "card_123" }
  },
  "simulation": {
    "jobId": "ad5b6a7f-9999-4eac-8dbe-5f6a7b8c9da4",
    "runPlanJobId": "be6c7b8a-aaaa-4fbd-9ecf-6a7b8c9daeb5",
    "flow": { "id": "cf7d8c9b-bbbb-4ace-afd0-7b8c9daebfc6", "title": "Change my PIN" },
    "testCase": { "id": "d08e9dac-cccc-4bdf-b0e1-8c9daebfc0d7", "title": "Happy path", "isDefault": true },
    "variables": { "cardNumber": "4111111111111111", "newPin": "4821" },
    "persona": { "name": "Maria" }
  }
}

Field	Description
`requestId`	Unique per check, also sent as the `X-Roark-Request-Id` header. Retries reuse it, so you can deduplicate on it
`metric`	The metric being checked
`call`	The call, with the same field names as the Calls API. `externalId` is your own id if you sent one, and `properties` holds the call’s custom properties
`simulation`	Set when the call is a Roark simulation, `null` on a live call. `variables` holds the run’s variables and `persona` the persona the simulated caller played

Every request also carries User-Agent: Roark-Metric-Check/1.0, and Content-Type: application/json on a POST. Headers you set take precedence.

Placeholders

{{path}} placeholders read from the payload above, for example {{call.id}}, {{call.properties.cardId}}, or {{simulation.variables.newPin}}. Roark checks every placeholder when you save, so a typo is caught before any call runs. How a value goes in depends on where the placeholder sits:

Where	How the value goes in	Example
URL	URL-encoded	`https://api.example.com/cards/{{call.properties.cardId}}/pin`
Header	As text. A value with a line break is refused	`X-Card-Id: {{call.properties.cardId}}`
Body, inside quotes	As escaped text	`"cardId": "{{call.properties.cardId}}"`
Body, outside quotes	As JSON: objects stay objects, text is quoted for you	`"call": {{call}}` sends the whole call object

If a call has no value for a placeholder, such as a run variable on a live call or a property the call doesn’t carry, Roark doesn’t send the request. The result is recorded as missing data, never as a fail, and the literal {{...}} is never sent.

A check written for simulations, such as one that reads {{simulation.variables.newPin}}, records missing data on live calls instead of failing them. Use a collector’s conditions to run it only where it applies.

Answering the Check

Reply with HTTP 200 and this JSON:

{ "passed": true, "reason": "PIN on card_123 was changed at 14:02:51 UTC." }

Field	Type	Description
`passed`	boolean	Whether the call did what it should
`reason`	string	Optional. Shown next to the result in Roark. Longer than 1000 characters is cut

A minimal endpoint:

app.post('/roark/checks/pin-changed', async (req, res) => {
  const { cardId, expectedPin } = req.body
  const card = await cards.get(cardId)

  if (card.pin === expectedPin) {
    return res.json({ passed: true, reason: `PIN on ${cardId} matches the new PIN.` })
  }
  res.json({ passed: false, reason: `PIN on ${cardId} was not changed.` })
})

Errors and retries

Only a 200 with passed set is a verdict. Anything else is recorded as an error on the call, not a fail, so your pass rate reflects your agent and not your endpoint’s uptime.

Your endpoint	Result
`200` with `{ "passed": ... }`	Pass or Fail
Times out (30 seconds), can’t be reached, or answers `5xx`	Retried twice with the same `requestId`, then an error
Any other status, including other `2xx` and `4xx`	Error, not retried
`200` with a body that isn’t the contract	Error

Roark shows your reason, never your raw response, on the call.

Security

Authenticate with headers. Add your own Authorization or API key header. Header values are redacted in Roark’s request log.
Public URLs only. The URL must be public http(s). Roark checks it when you save, again after placeholders are filled in, and once more against the resolved address when it connects, so a public hostname that points to a private address is refused.
No redirects. Roark doesn’t follow redirects, so point the URL at the final endpoint.
Bounded responses. Responses over 64 KB are not read.

Limits

API checks run on calls only, not chats.
An API check has no variants: the request is the metric, so edit it on the metric itself.

What’s Next

Studio

Author and test an API check before it runs on calls

Collectors

Run your API check automatically on matching live calls

Run Plans

Attach the check to a simulation and gate a run on it

Variables

Pass run variables your check can compare against

Getting Started

Simulations

Observability

Metrics

Autoimprove

Recipes

Integrations

Enterprise

Resources

Quickstart

Create it with the SDK

Collect it from config

Build it in Studio

How It Works

The Request Payload

Placeholders

Answering the Check

Errors and retries

Security

Limits

What’s Next

Studio

Collectors

Run Plans

Variables

​Quickstart

​Create it with the SDK

​Collect it from config

​Build it in Studio

​How It Works

​The Request Payload

​Placeholders

​Answering the Check

​Errors and retries

​Security

​Limits

​What’s Next

Studio

Collectors

Run Plans

Variables

Quickstart

Create it with the SDK

Collect it from config

Build it in Studio

How It Works

The Request Payload

Placeholders

Answering the Check

Errors and retries

Security

Limits

What’s Next