> ## Documentation Index > Fetch the complete documentation index at: https://docs.roark.ai/llms.txt > Use this file to discover all available pages before exploring further. # Run a simulation > Starts a simulation and returns the run. Send `template` to run one of the built-in templates: it supplies the metrics and checks, and for some templates the flows too, so the request only names the agent and the direction. Send `plan` to describe a simulation yourself and run it once. Send `planId` to run a plan you already have. `template` and `plan` both resolve to a run plan, returned as `simulationRunPlanId`. Add `saveAsPlan` to keep it, or read it back to see exactly what ran. A plan built from a template is a snapshot: retuning the template later never changes what that plan runs, which is what makes a saved one safe to pin in CI. ## OpenAPI ````yaml /api-reference/openapi.documented.json post /v1/simulation/run openapi: 3.1.0 info: title: Roark Analytics API description: >- The Roark Analytics API gives you access to the same API that powers the award winning Roark Analytics platform. version: 1.0.0 servers: - description: Production url: https://api.roark.ai security: - Bearer: [] tags: - name: Agent - name: Agent Endpoint - name: Call - name: Chat - name: Metric - name: Metric Policy - name: Metric Collection Job - name: Customer Flow - name: Customer Flow Edge Case - name: Simulation - name: Simulation Persona - name: Simulation Environment - name: Simulation Scenario - name: Simulation Run Plan - name: Simulation Template - name: Simulation Run Plan Job - name: Simulation Job - name: HTTP Request Definition - name: Webhook - name: Issue - name: Autoimprove - name: Agent Config - name: Knowledge Base - name: Config - name: Organization Project - name: Organization Project Member - name: Organization Project API Key - name: Organization Config - name: CLI Auth - name: Me - name: Project - name: Usage - name: Benchmark - name: Health - name: Prospect Simulation paths: /v1/simulation/run: post: tags: - Simulation summary: Run a simulation description: >- Starts a simulation and returns the run. Send `template` to run one of the built-in templates: it supplies the metrics and checks, and for some templates the flows too, so the request only names the agent and the direction. Send `plan` to describe a simulation yourself and run it once. Send `planId` to run a plan you already have. `template` and `plan` both resolve to a run plan, returned as `simulationRunPlanId`. Add `saveAsPlan` to keep it, or read it back to see exactly what ran. A plan built from a template is a snapshot: retuning the template later never changes what that plan runs, which is what makes a saved one safe to pin in CI. operationId: postV1SimulationRun requestBody: content: application/json: schema: $ref: '#/components/schemas/RunSimulationInput' responses: '200': description: The run that was started content: application/json: schema: type: object properties: data: $ref: '#/components/schemas/RunSimulationResponse' required: - data '400': content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' description: Validation error example: type: validation code: invalid_parameter message: The request was invalid param: email description: Bad Request '401': content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' description: Authentication error example: type: authentication code: unauthorized message: Authentication required description: Unauthorized '402': content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' description: Insufficient credit example: type: payment_required code: insufficient_credits message: >- Not enough credit to start this work. Add credit and try again. description: Payment Required '403': content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' description: Permission error example: type: forbidden code: permission_denied message: You do not have permission to access this resource description: Forbidden '404': content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' description: Not found error example: type: not_found code: resource_not_found message: The requested resource could not be found description: Not Found '429': content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' description: Rate limit error example: type: rate_limit code: too_many_requests message: Rate limit exceeded description: Too Many Requests '500': content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' description: Server error example: type: internal code: internal_error message: Internal server error description: Internal Server Error x-codeSamples: - lang: JavaScript source: |- import Roark from '@roarkanalytics/sdk'; const client = new Roark({ bearerToken: process.env['ROARK_API_BEARER_TOKEN'], // This is the default and can be omitted }); const response = await client.simulation.run({ plan: { agentEndpoints: [{ id: '182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e' }], direction: 'INBOUND', maxSimulationDurationSeconds: 300, }, }); console.log(response.data); - lang: Python source: |- import os from roark_analytics import Roark client = Roark( bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"), # This is the default and can be omitted ) response = client.simulation.run( plan={ "agent_endpoints": [{ "id": "182bd5e5-6e1a-4fe4-a799-aa6d9a6ab26e" }], "direction": "INBOUND", "max_simulation_duration_seconds": 300, }, ) print(response.data) components: schemas: RunSimulationInput: oneOf: - $ref: '#/components/schemas/RunSimulationFromConfig' - $ref: '#/components/schemas/RunSimulationFromPlanId' - $ref: '#/components/schemas/RunSimulationFromTemplate' description: >- A built-in template to run, a simulation to configure and run, or the id of a plan to run. RunSimulationResponse: type: object properties: simulationRunPlanJobId: type: string format: uuid description: The run. Poll it with GET /v1/simulation/plan/job/{jobId}. status: type: string enum: - PENDING - QUEUED - CREATING_SNAPSHOTS - CREATING_SIMULATIONS - PREPARING_CAPACITY - RUNNING_SIMULATIONS - COMPLETED - FAILED - TIMED_OUT - CANCELLED - CANCELLING - ENDING_SIMULATIONS description: >- Initial status. PENDING normally, or QUEUED when the plan runs sequentially and another job of its is still active. createdAt: type: string description: When the run was created, ISO 8601. simulationRunPlanId: type: string format: uuid description: >- The run plan behind this run, present whether or not it was saved. Pass it back as `planId` to run the same configuration again. savedAsPlan: type: boolean description: >- Whether that plan is listed by GET /v1/simulation/plan. False for an unsaved run, whose plan is hidden. simulationJobCount: type: integer description: How many simulated calls this run places. required: - simulationRunPlanJobId - status - createdAt - simulationRunPlanId - savedAsPlan - simulationJobCount description: A started simulation run. ErrorResponse: type: object properties: type: type: string enum: - validation - authentication - forbidden - not_found - conflict - payment_required - rate_limit - internal description: The error type category examples: - validation - authentication code: type: string description: Machine-readable error code identifier examples: - invalid_parameter - missing_required_field - unauthorized message: type: string description: Human-readable error message examples: - The request was invalid - Authentication required param: type: string description: The parameter that caused the error (if applicable) examples: - email - user_id details: description: Additional error context information required: - type - code - message RunSimulationFromConfig: type: object properties: plan: $ref: '#/components/schemas/InlineRunPlanConfig' saveAsPlan: type: boolean description: >- Keeps this configuration as a run plan, listed by GET /v1/simulation/plan and re-runnable with `planId`. Requires `plan.name`, since a plan you meant to keep should not be filed under a generated one. Omitted or false gives a one-off. The run still needs a plan to execute, so one is created either way, but it is hidden: it carries this run and nothing else. variables: anyOf: - type: object additionalProperties: type: string description: >- Global format: key-value pairs that apply to ALL scenarios in the plan example: orderNumber: '12345' environment: staging - type: array items: type: object properties: flowId: type: string format: uuid description: A customer flow this plan runs. happyPath: type: boolean const: true description: Narrow to the flow's happy path. edgeCaseId: type: string format: uuid description: Narrow to one edge case of that flow. variables: type: object additionalProperties: type: string description: The values to apply. required: - flowId - variables description: Values scoped to the flows this plan runs example: - flowId: 550e8400-e29b-41d4-a716-446655440000 variables: orderNumber: '12345' - flowId: 550e8400-e29b-41d4-a716-446655440000 happyPath: true variables: orderNumber: '55555' - flowId: 550e8400-e29b-41d4-a716-446655440000 edgeCaseId: 7a3d2e1f-c4b5-6a89-0d1e-2f3a4b5c6d7e variables: orderNumber: '67890' - type: array items: type: object properties: scenarioId: type: string format: uuid description: ID of the scenario to apply variables to variables: type: object additionalProperties: type: string description: Key-value pairs for this scenario required: - scenarioId - variables deprecated: true description: >- Deprecated, for plans built on scenarios. Plans built on customer flows target them with `flowId` instead. example: - scenarioId: 550e8400-e29b-41d4-a716-446655440000 variables: orderNumber: '12345' - scenarioId: 7a3d2e1f-c4b5-6a89-0d1e-2f3a4b5c6d7e variables: orderNumber: '67890' description: >- Values for the {{variables}} the run resolves, overriding whatever the plan has pinned. An object applies them to the whole run: { "orderNumber": "12345", "tier": "gold" } An array applies them per flow, or to just its happy path or one of its edge cases, when a single set will not do. Each entry carries what it applies to: [ { "flowId": "550e8400-...", "variables": { "orderNumber": "12345" } }, { "flowId": "550e8400-...", "happyPath": true, "variables": { "orderNumber": "55555" } }, { "flowId": "550e8400-...", "edgeCaseId": "7a3d2e1f-...", "variables": { "orderNumber": "67890" } } ] An entry that narrows to neither covers everything that flow resolves. A flow this plan does not attach, or an edge case that does not belong to the flow, is rejected rather than ignored. A plan built on scenarios rather than customer flows targets them the same way, with `scenarioId` in place of `flowId`. That form is deprecated alongside scenarios themselves, and still accepted so runs against those plans keep working. required: - plan additionalProperties: false title: New simulation description: >- Describe a simulation and run it, optionally keeping the configuration as a reusable plan. example: saveAsPlan: true plan: name: Billing regression direction: OUTBOUND maxSimulationDurationSeconds: 300 agentEndpoints: - id: 7c9e6679-7425-40de-944b-e07fc1f90ae7 metrics: - slug: customer_satisfaction flows: - id: 550e8400-e29b-41d4-a716-446655440000 happyPath: true edgeCases: ALL variables: orderNumber: '12345' RunSimulationFromPlanId: type: object properties: planId: type: string format: uuid description: >- The run plan to run, saved or hidden. Rename or unhide it with PUT /v1/simulation/plan/{planId}. variables: anyOf: - type: object additionalProperties: type: string description: >- Global format: key-value pairs that apply to ALL scenarios in the plan example: orderNumber: '12345' environment: staging - type: array items: type: object properties: flowId: type: string format: uuid description: A customer flow this plan runs. happyPath: type: boolean const: true description: Narrow to the flow's happy path. edgeCaseId: type: string format: uuid description: Narrow to one edge case of that flow. variables: type: object additionalProperties: type: string description: The values to apply. required: - flowId - variables description: Values scoped to the flows this plan runs example: - flowId: 550e8400-e29b-41d4-a716-446655440000 variables: orderNumber: '12345' - flowId: 550e8400-e29b-41d4-a716-446655440000 happyPath: true variables: orderNumber: '55555' - flowId: 550e8400-e29b-41d4-a716-446655440000 edgeCaseId: 7a3d2e1f-c4b5-6a89-0d1e-2f3a4b5c6d7e variables: orderNumber: '67890' - type: array items: type: object properties: scenarioId: type: string format: uuid description: ID of the scenario to apply variables to variables: type: object additionalProperties: type: string description: Key-value pairs for this scenario required: - scenarioId - variables deprecated: true description: >- Deprecated, for plans built on scenarios. Plans built on customer flows target them with `flowId` instead. example: - scenarioId: 550e8400-e29b-41d4-a716-446655440000 variables: orderNumber: '12345' - scenarioId: 7a3d2e1f-c4b5-6a89-0d1e-2f3a4b5c6d7e variables: orderNumber: '67890' description: >- Values for the {{variables}} the run resolves, overriding whatever the plan has pinned. An object applies them to the whole run: { "orderNumber": "12345", "tier": "gold" } An array applies them per flow, or to just its happy path or one of its edge cases, when a single set will not do. Each entry carries what it applies to: [ { "flowId": "550e8400-...", "variables": { "orderNumber": "12345" } }, { "flowId": "550e8400-...", "happyPath": true, "variables": { "orderNumber": "55555" } }, { "flowId": "550e8400-...", "edgeCaseId": "7a3d2e1f-...", "variables": { "orderNumber": "67890" } } ] An entry that narrows to neither covers everything that flow resolves. A flow this plan does not attach, or an edge case that does not belong to the flow, is rejected rather than ignored. A plan built on scenarios rather than customer flows targets them the same way, with `scenarioId` in place of `flowId`. That form is deprecated alongside scenarios themselves, and still accepted so runs against those plans keep working. required: - planId additionalProperties: false title: Existing plan description: Run a plan that already exists. example: planId: 3a1d5e7c-9b2f-4a6d-8c31-5f7e9d0a2b4c variables: orderNumber: '99887' RunSimulationFromTemplate: type: object properties: template: type: string minLength: 1 description: The template to run, as listed by GET /v1/simulation/template. example: health-check agentEndpoints: type: array items: type: object properties: id: type: string format: uuid required: - id minItems: 1 description: The agent endpoints to call. No template can know these. direction: type: string enum: - INBOUND - OUTBOUND description: Direction of the simulation (INBOUND or OUTBOUND) example: INBOUND flows: type: array items: $ref: '#/components/schemas/RunPlanFlowSelection' minItems: 1 description: >- The flows to run, in the same shape a run plan takes them. Required when the template lists no flows of its own: it presets what to measure, and this says what to measure it on. Optional when it does, where these REPLACE the ones it would have run, so you can narrow a suite to the cases you care about. Either way, GET /v1/simulation/template lists the flows and variant ids each template covers. On a template that sweeps a property, every value runs exactly what you select here: the happy path, the edge cases you name, or `edgeCases: "ALL"`. Each selected case is a call per value per iteration, so naming three edge cases triples the run. example: - slug: sf-prompt-injection edgeCases: - slug: data-embedded-injection - id: 3a1d5e7c-9b2f-4a6d-8c31-5f7e9d0a2b4c happyPath: true iterationCount: type: integer minimum: 1 maximum: 10000 description: >- Runs per test case (1-10000). Defaults to 1, or to 6 for a template that sweeps a property. A sweep needs at least 5 calls per value (test cases per value times iterations) to compare its values, and a lower count is refused with 400. example: 6 maxConcurrentJobs: type: integer minimum: 1 default: 5 description: Maximum number of concurrent simulation jobs example: 5 maxSimulationDurationSeconds: type: integer minimum: 1 description: >- Defaults to the template's `defaultMaxSimulationDurationSeconds`, as returned by GET /v1/simulation/template. silenceTimeoutSeconds: type: integer minimum: 1 default: 30 description: Timeout in seconds for silence detection example: 30 maxNoResponseRetries: type: integer minimum: 0 maximum: 10 default: 0 description: >- How many more times to run a test case when the agent under test never responds: it never speaks on a call or never replies in a chat (0-10). 0 turns retries off. Failed checks and failures on Roark’s side are never retried. Each retry is a separate attempt, billed like any other, so a plan retrying N times can place up to N + 1 calls per test case. Every silent attempt stays on the run with its own call; the run settles once each test case has a final attempt, and the agent never spoke verdict is judged on each test case’s last attempt. example: 2 noResponseRetryBackoffSeconds: type: integer minimum: 30 maximum: 600 default: 90 description: >- Seconds a retry waits before it dials (30-600). Only used when `maxNoResponseRetries` is above 0. example: 90 endCallPhrases: type: array items: type: string default: - goodbye description: Phrases that trigger end of call. Empty array disables the feature. example: - goodbye endCallReasons: type: array items: type: string description: >- Semantic conditions that trigger end of call. The LLM evaluates the conversation against these conditions. Defaults to the template's `defaultEndCallReasons`, as returned by GET /v1/simulation/template. Pass an empty array to run with none. example: - Order has been confirmed by the agent executionMode: type: string enum: - PARALLEL - SEQUENTIAL_SAME_RUN_PLAN - SEQUENTIAL_PROJECT default: PARALLEL description: Execution mode (PARALLEL or SEQUENTIAL) example: PARALLEL enrichWithLiveConversation: type: boolean default: false description: >- Merge the customer's own recording of the real call into each simulation, so metrics can be scored against the live leg as well as the simulated one. This is the API equivalent of the dashboard's live-enrichment toggle. With this on, the run provisions a phone number and holds each call open for up to 15 minutes waiting for a matching call to be posted to POST /v1/call. A call matches on the provisioned number (`roarkPhoneNumber` on the job) with a start time inside the simulation window. If nothing arrives, the simulation still completes and any `LIVE`-sourced metric produces no value. Required by any metric whose `requiresLiveConversation` is true: without it that metric is silently skipped. example: false additionalMetrics: type: array items: $ref: '#/components/schemas/RunPlanMetricRef' minItems: 1 description: >- Metrics to collect on top of the template's own, referenced by `id` or `slug` like a plan's `metrics`. The template's metrics and checks always run; naming one of them here again keeps it once, with the success criteria you set on it. example: - slug: call_ended_prematurely comparisonValues: type: array items: anyOf: - type: string minLength: 1 description: A value of `comparisonProperty`, run as its plain arm. - $ref: '#/components/schemas/ComparisonArm' minItems: 1 description: >- The arms of the sweep to run, for a template that sweeps one (GET /v1/simulation/template returns `sweep.property` for those that do). This is what the run costs: the flow is called once per arm, so ten arms is ten times the calls of one. Omit it to run every value the property has, plain, which for `accent-handling` is more than twenty. Send a subset to narrow it, for example the three accents you actually serve. An object entry pins something on that arm only, such as a noise level per bed on `background-noise-robustness`: `{ "value": "OFFICE", "backgroundNoiseVolume": 0.6 }` plays OFFICE at 60% while the other beds keep the default. See `POST /v1/simulation/plan`. example: - US - GB - IN comparisonBaseline: type: - string - 'null' description: >- The sweep's reference value, shown first in the results. Defaults to the template's own baseline, as returned by GET /v1/simulation/template. Whether a value did significantly worse does not depend on it: that is decided against every other value combined. Send it with `comparisonValues` and it must be one of them, or the request is rejected: anchoring every difference to an arm the run never made would measure it against nothing. Leave it out and the template's own baseline is used, and quietly dropped if your narrowing excluded it, since that one you did not choose. example: US questions: type: array items: type: object properties: ask: type: string minLength: 1 description: The question the caller asks the agent. example: What are your holiday hours? expect: type: string minLength: 1 description: The answer the agent must give, judged against the transcript. example: We're closed on public holidays. required: - ask - expect minItems: 1 description: >- For the `question-answer-check` template: the questions to ask and the answer expected for each. Every question runs as its own graded call. personaId: type: string format: uuid description: 'For `question-answer-check`: the persona that asks the questions.' environmentId: type: string format: uuid description: 'For `question-answer-check`: the environment the calls run in.' name: type: string minLength: 1 description: >- What to call this. Defaults to the template's name and the date, and required with `saveAsPlan`. saveAsPlan: type: boolean description: >- Keeps the resolved configuration as a run plan, listed by GET /v1/simulation/plan and re-runnable with `planId`. Requires `name`. variables: anyOf: - type: object additionalProperties: type: string description: >- Global format: key-value pairs that apply to ALL scenarios in the plan example: orderNumber: '12345' environment: staging - type: array items: type: object properties: flowId: type: string format: uuid description: A customer flow this plan runs. happyPath: type: boolean const: true description: Narrow to the flow's happy path. edgeCaseId: type: string format: uuid description: Narrow to one edge case of that flow. variables: type: object additionalProperties: type: string description: The values to apply. required: - flowId - variables description: Values scoped to the flows this plan runs example: - flowId: 550e8400-e29b-41d4-a716-446655440000 variables: orderNumber: '12345' - flowId: 550e8400-e29b-41d4-a716-446655440000 happyPath: true variables: orderNumber: '55555' - flowId: 550e8400-e29b-41d4-a716-446655440000 edgeCaseId: 7a3d2e1f-c4b5-6a89-0d1e-2f3a4b5c6d7e variables: orderNumber: '67890' description: >- Values for the {{variables}} the run resolves. An object applies them everywhere; an array targets a flow, its happy path, or one of its edge cases with `flowId`. The scenario-scoped form the other variants accept is not valid here: a template run is always flow-based, so there would be no scenario for it to reach. required: - template - agentEndpoints - direction additionalProperties: false title: From template description: Run one of the built-in templates against your agent. example: template: health-check direction: OUTBOUND agentEndpoints: - id: 7c9e6679-7425-40de-944b-e07fc1f90ae7 InlineRunPlanConfig: type: object properties: name: type: string minLength: 1 description: >- What to call this. Generated from the date when omitted, and required with `saveAsPlan`. example: Billing regression description: type: string description: Description of the run plan example: A run plan for testing inbound calls direction: type: string enum: - INBOUND - OUTBOUND description: Direction of the simulation (INBOUND or OUTBOUND) example: INBOUND iterationCount: type: integer minimum: 1 maximum: 10000 default: 1 description: Number of iterations to run for each test case (1-10000) example: 1 maxConcurrentJobs: type: integer minimum: 1 default: 5 description: Maximum number of concurrent simulation jobs example: 5 maxSimulationDurationSeconds: type: integer minimum: 1 description: Maximum duration in seconds for each simulation example: 300 silenceTimeoutSeconds: type: integer minimum: 1 default: 30 description: Timeout in seconds for silence detection example: 30 maxNoResponseRetries: type: integer minimum: 0 maximum: 10 default: 0 description: >- How many more times to run a test case when the agent under test never responds: it never speaks on a call or never replies in a chat (0-10). 0 turns retries off. Failed checks and failures on Roark’s side are never retried. Each retry is a separate attempt, billed like any other, so a plan retrying N times can place up to N + 1 calls per test case. Every silent attempt stays on the run with its own call; the run settles once each test case has a final attempt, and the agent never spoke verdict is judged on each test case’s last attempt. example: 2 noResponseRetryBackoffSeconds: type: integer minimum: 30 maximum: 600 default: 90 description: >- Seconds a retry waits before it dials (30-600). Only used when `maxNoResponseRetries` is above 0. example: 90 endCallPhrases: type: array items: type: string default: - goodbye description: Phrases that trigger end of call. Empty array disables the feature. example: - goodbye endCallReasons: type: array items: type: string default: [] description: >- Semantic conditions that trigger end of call. The LLM evaluates the conversation against these conditions. Empty array disables the feature. example: - Order has been confirmed by the agent executionMode: type: string enum: - PARALLEL - SEQUENTIAL_SAME_RUN_PLAN - SEQUENTIAL_PROJECT default: PARALLEL description: Execution mode (PARALLEL or SEQUENTIAL) example: PARALLEL scenarios: type: array items: type: object properties: id: type: string format: uuid description: Scenario ID variables: type: object additionalProperties: type: string description: >- Template variables for this scenario instance. The same scenario can appear multiple times with different variables. example: customerName: John Doe appointmentDate: '2024-02-15' required: - id minItems: 1 deprecated: true description: >- Deprecated: use `flows` instead. Scenarios to include in this run plan. The same scenario ID can appear multiple times with different variables. flows: type: array items: $ref: '#/components/schemas/RunPlanFlowSelection' minItems: 1 description: >- Customer flows to include in this run plan. The same flow can appear more than once with a different persona override, different variables, or different `overrides`: attaching it once per value of one property is how you compare that property without a template. personas: type: array items: type: object properties: id: type: string format: uuid required: - id minItems: 1 description: >- Personas to include in this run plan. Required with `scenarios`; ignored with `flows`, where each variant carries its own persona. agentEndpoints: type: array items: type: object properties: id: type: string format: uuid required: - id minItems: 1 description: Agent endpoints to include in this run plan metrics: type: array items: $ref: '#/components/schemas/RunPlanMetricRef' minItems: 1 description: >- Metric definitions to include in this run plan. Reference each by `id` (UUID) or `slug`. Optional when the attached `flows` carry the grading: metrics a flow declares itself (with `includeFlowMetrics`), or the Agent Expectations and Keypad Entry metrics a run adds for flows with expectations or expected keypad entries (with `includeAutomaticMetrics`). A plan with nothing to grade is rejected with a 400. includeFlowMetrics: type: boolean default: true description: >- Also collect each attached flow's own metrics, on top of the `metrics` named here. Default true, which is what you want when you brought your own flows and their graders. Set false for a run whose metric list is meant to be exhaustive: a template like Load Testing or Voicemail deliberately grades a narrow set, and inheriting every flow metric on top multiplies analysis cost across the volume without adding signal. GET /v1/simulation/template returns the value each template expects. example: true includeAutomaticMetrics: type: boolean default: true description: >- Let the run add metrics by itself off the attached flows, on top of the `metrics` named here. Two attach this way today: Agent Expectations wherever an attached flow has agent expectations written on it, and Keypad Entry wherever one has steps where the agent is expected to press keys. Both grade something authored on the flow that nothing else measures, which is why it is on by default. Set false when the `metrics` list is meant to be exhaustive: a plan testing only whether the caller can complete the flow may not want the agent graded on its expectations as well. False also pins the plan against any automatic metric Roark adds later. example: true comparisonProperty: type: - string - 'null' enum: - ACCENT - AGE - BACKGROUND_NOISE - BACKGROUND_NOISE_VOLUME - BASE_EMOTION - CONFIRMATION_STYLE - GENDER - INTENT_CLARITY - LANGUAGE - INTERRUPTION - MEMORY_RELIABILITY - RESPONSE_TIMING - SPEECH_CLARITY - SPEECH_PACE description: >- The property this run plan investigates: the one thing its arms differ by. Set it and the run report compares the arms on that property, so a run answers "what did background noise cost" rather than just "what did each arm score". Every value is a field already recorded on each call, so the report can label an arm `CRYING_BABY` rather than repeating a flow variant's title. Omit it and the report still compares when it can: it detects which property varies across the arms. Setting it is what tells the written summary what you were trying to find out, which detection cannot infer. example: BACKGROUND_NOISE comparisonBaseline: type: - string - 'null' description: >- The reference value of `comparisonProperty`, for example `NONE` for `BACKGROUND_NOISE` or `NORMAL` for `SPEECH_PACE`: shown first in the results. Must be a value that property can take. Whether a value did significantly worse does not depend on it: that is decided against every other value combined (see `sweepAttribution`). Stored rather than assumed, so the report can say "compared against US accent" instead of implying Roark decided which value is normal. Omit it and the property's own norm is used, as the dashboard prefills it, or none when your `comparisonValues` leave the norm out. `GENDER` has no norm, so choose the one you are testing against. example: NONE comparisonValues: type: array items: anyOf: - type: string minLength: 1 description: A value of `comparisonProperty`, run as its plain arm. - $ref: '#/components/schemas/ComparisonArm' minItems: 1 description: >- The arms to run, for a plan that sweeps `comparisonProperty`. This is what the plan costs: the flow is attached once per arm, so ten arms is ten times the calls of one. Attach each flow once, as you would without a comparison: the plan builds the arms, running the happy path or edge cases you selected under every arm. Built arms need at least 5 calls per arm (`iterationCount` times the test cases per arm), or the plan is refused with `400`. Flows that all carry `overrides` on `comparisonProperty` already are the arms and are kept as you wrote them; a mix of flows with and without one is refused. Each entry is one arm. A bare value runs it plain: `"CITY"`. An object runs the value with something pinned on that arm only, such as a noise level per bed: `{ "value": "OFFICE", "backgroundNoiseVolume": 0.6 }` plays OFFICE at 60% while the other beds keep the default. List a value more than once with different pins to run it as several arms: DRIVING at 0.7 and DRIVING at 1 are two arms, reported as `Driving (70% noise)` and `Driving (100% noise)`, and `"DRIVING"` beside them keeps the plain arm too. The sweep still varies one property; what an arm pins is part of "everything else" for that arm only, so the report still compares the arms on `comparisonProperty`. Omit it to run every value the property has, plain, which for `ACCENT` is more than twenty. A `comparisonBaseline` outside the values listed is rejected, because it would anchor every difference to an arm the run never made. A value the property cannot take, a pin the sweep cannot account for, or the same arm listed twice is rejected with `400`. Not stored as a field: the arms are the values. Reading the plan back returns them as its flow attachments, each with its pins as `overrides`. example: - NONE - value: OFFICE backgroundNoiseVolume: 0.6 - value: DRIVING backgroundNoiseVolume: 0.7 - value: DRIVING backgroundNoiseVolume: 1 enrichWithLiveConversation: type: boolean default: false description: >- Merge the customer's own recording of the real call into each simulation, so metrics can be scored against the live leg as well as the simulated one. This is the API equivalent of the dashboard's live-enrichment toggle. With this on, the run provisions a phone number and holds each call open for up to 15 minutes waiting for a matching call to be posted to POST /v1/call. A call matches on the provisioned number (`roarkPhoneNumber` on the job) with a start time inside the simulation window. If nothing arrives, the simulation still completes and any `LIVE`-sourced metric produces no value. Required by any metric whose `requiresLiveConversation` is true: without it that metric is silently skipped. example: false required: - direction - maxSimulationDurationSeconds - agentEndpoints additionalProperties: false description: 'The simulation to run: what to call, who calls it, and what to measure.' RunPlanFlowSelection: type: object properties: id: type: string format: uuid description: The customer flow to run. slug: type: string minLength: 1 description: >- The Roark-curated flow to run, by its stable slug. Use instead of `id` for a run you keep in version control: a curated flow’s id differs between deployments, its slug does not. Your own flows have no slug and are named by `id`. example: sf-prompt-injection happyPath: type: boolean description: >- Run the flow's happy path. Resolved when the run starts, so it follows the flow. edgeCases: anyOf: - type: string const: ALL - type: array items: $ref: '#/components/schemas/RunPlanEdgeCaseSelection' description: >- `"ALL"` runs every edge case the flow has when the run starts, so one added later is covered. An array runs only the ones you name, each able to carry its own persona override and values. personaOverrideId: type: - string - 'null' format: uuid description: >- Runs everything this attachment resolves as that persona instead of its own. variables: type: object additionalProperties: type: string description: Values for everything it resolves. example: customerName: John Doe appointmentDate: '2024-02-15' overrides: type: array items: $ref: '#/components/schemas/RunPlanFlowOverride' description: >- Persona and environment properties to change for this attachment only, without editing the persona or the environment themselves. Each entry patches the per-run snapshot this attachment records, so the flow runs as a caller with that accent, or over that background noise, and everything else stays as authored. This is how you attach the same flow twice and vary one thing between them, which is what a sweep template builds for you. One value per property; a property named twice is rejected. example: - property: ACCENT value: IN - property: BACKGROUND_NOISE value: TRAIN description: >- One customer flow attached to a run plan, and which of its ways of running you cover. Attaching the same flow more than once with different overrides is how you fan it out across personas or values. example: id: 550e8400-e29b-41d4-a716-446655440000 happyPath: true edgeCases: - id: 7c9e6679-7425-40de-944b-e07fc1f90ae7 variables: tier: premium RunPlanMetricRef: type: object properties: id: type: string format: uuid description: Metric definition UUID. Provide either this or `slug`, not both. slug: type: string minLength: 1 description: >- Stable metric slug (e.g. `customer_satisfaction`). Provide either this or `id`, not both. metricId: type: string minLength: 1 description: >- Alias of `slug` accepted for backwards compatibility. Use `slug` for new integrations. conversationSource: type: - string - 'null' enum: - SIMULATED - LIVE description: >- Which side of an enriched run this metric is scored on. Only meaningful with `enrichWithLiveConversation: true`, where a run has both a simulated conversation and the customer's own live recording of it. Defaults to `SIMULATED`. Use `LIVE` for a metric that must be measured against the real recording (audio quality, provider latency) rather than the simulated leg. `null` means the same as omitting it, so a plan read back from GET can be sent straight to PUT. minPassRate: type: - number - 'null' minimum: 0 maximum: 100 description: >- THE BAR, and the only thing that decides pass/fail. The share of the run's simulations that must pass this check, 0-100. Applied to this check alone and never pooled: silence duration at 40 and word count at 80 means the run fails unless 40% of sims clear silence AND 80% clear word count. Omit or `null` for the 80% default. example: 80 additionalProperties: false ComparisonArm: type: object properties: backgroundNoiseVolume: type: number minimum: 0 maximum: 1 description: >- The noise level this arm plays at, 0 to 1. The environment default is 0.1. Only a `BACKGROUND_NOISE` sweep may pin it. example: 0.6 value: type: string minLength: 1 description: The swept value this arm runs, a value of `comparisonProperty`. example: OFFICE required: - value additionalProperties: false description: >- One arm of a sweep: the swept value it runs, and what it pins besides that value. The OFFICE arm of a BACKGROUND_NOISE sweep playing at 60% while the other beds keep their level is `{ "value": "OFFICE", "backgroundNoiseVolume": 0.6 }`. The report compares the arms on the swept property only, so each sweep may pin just what its experiment calls for: a BACKGROUND_NOISE sweep may pin `backgroundNoiseVolume`, and no other sweep pins anything yet. A pin the sweep cannot account for is rejected with `400`. RunPlanEdgeCaseSelection: type: object properties: id: type: string format: uuid description: The edge case to run. slug: type: string minLength: 1 description: >- The edge case to run, by its stable slug, matched within this flow. Use instead of `id` for a run you keep in version control: a curated edge case’s id differs between deployments and changes outright if it is renamed. Your own edge cases have no slug and are named by `id`. example: data-embedded-injection personaOverrideId: type: - string - 'null' format: uuid description: Run this one as that persona instead of its own. variables: type: object additionalProperties: type: string description: Values for this one only. example: customerName: John Doe appointmentDate: '2024-02-15' RunPlanFlowOverride: type: object properties: property: type: string enum: - ACCENT - AGE - BACKGROUND_NOISE - BACKGROUND_NOISE_VOLUME - BASE_EMOTION - CONFIRMATION_STYLE - GENDER - INTENT_CLARITY - LANGUAGE - INTERRUPTION - MEMORY_RELIABILITY - RESPONSE_TIMING - SPEECH_CLARITY - SPEECH_PACE value: type: string minLength: 1 required: - property - value description: >- One persona or environment property, changed for this flow attachment only. securitySchemes: Bearer: type: http scheme: bearer bearerFormat: JWT ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.