> ## Documentation Index > Fetch the complete documentation index at: https://docs.roark.ai/llms.txt > Use this file to discover all available pages before exploring further. # Get simulation run results > Everything the run report shows, in one request: the summary and score, the pass/fail verdict, per-metric aggregates, every simulation with its check results, breakdowns by agent and persona, the comparison against the previous run (or `compareToJobId`), the sweep attribution, and the written debrief with its executive summary, failure themes and recommendations. Safe to call while the run is in progress: counts reflect what has landed so far. `verdict` and `sweepAttribution` stay null until the run ends. The debrief is generated automatically a few minutes after a run with graded calls settles; poll until `debrief.status` is READY, or request one with POST /v1/simulation/plan/job/{jobId}/debrief. ## OpenAPI ````yaml /api-reference/openapi.documented.json get /v1/simulation/plan/job/{jobId}/results openapi: 3.1.0 info: title: Roark Analytics API description: >- The Roark Analytics API gives you access to the same API that powers the award winning Roark Analytics platform. version: 1.0.0 servers: - description: Production url: https://api.roark.ai security: - Bearer: [] tags: - name: Agent - name: Agent Endpoint - name: Call - name: Chat - name: Metric - name: Metric Policy - name: Metric Collection Job - name: Customer Flow - name: Customer Flow Edge Case - name: Simulation - name: Simulation Persona - name: Simulation Environment - name: Simulation Scenario - name: Simulation Run Plan - name: Simulation Template - name: Simulation Run Plan Job - name: Simulation Job - name: HTTP Request Definition - name: Webhook - name: Issue - name: Autoimprove - name: Agent Config - name: Knowledge Base - name: Config - name: Organization Project - name: Organization Project Member - name: Organization Project API Key - name: Organization Config - name: CLI Auth - name: Me - name: Project - name: Usage - name: Benchmark - name: Health - name: Prospect Simulation paths: /v1/simulation/plan/job/{jobId}/results: get: tags: - Simulation Run Plan Job summary: Get simulation run results description: >- Everything the run report shows, in one request: the summary and score, the pass/fail verdict, per-metric aggregates, every simulation with its check results, breakdowns by agent and persona, the comparison against the previous run (or `compareToJobId`), the sweep attribution, and the written debrief with its executive summary, failure themes and recommendations. Safe to call while the run is in progress: counts reflect what has landed so far. `verdict` and `sweepAttribution` stay null until the run ends. The debrief is generated automatically a few minutes after a run with graded calls settles; poll until `debrief.status` is READY, or request one with POST /v1/simulation/plan/job/{jobId}/debrief. operationId: getV1SimulationPlanJobByJobIdResults parameters: - in: query name: compareToJobId schema: type: string format: uuid description: >- Simulation run plan job to compare this run against, e.g. the last run on your main branch. Defaults to the plan's previous settled run. Any run in the project works, including another plan's: checks are matched by metric, and a check only one run graded reads null on the other side. example: 9a8b7c6d-5e4f-4a3b-8c2d-1e0f9a8b7c6d - schema: type: string in: path name: jobId required: true responses: '200': description: The run’s results content: application/json: schema: type: object properties: data: $ref: '#/components/schemas/SimulationRunResults' required: - data '400': content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' description: Validation error example: type: validation code: invalid_parameter message: The request was invalid param: email description: Bad Request '401': content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' description: Authentication error example: type: authentication code: unauthorized message: Authentication required description: Unauthorized '403': content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' description: Permission error example: type: forbidden code: permission_denied message: You do not have permission to access this resource description: Forbidden '404': content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' description: Not found error example: type: not_found code: resource_not_found message: The requested resource could not be found description: Simulation run, or the `compareToJobId` run, not found '429': content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' description: Rate limit error example: type: rate_limit code: too_many_requests message: Rate limit exceeded description: Too Many Requests '500': content: application/json: schema: $ref: '#/components/schemas/ErrorResponse' description: Server error example: type: internal code: internal_error message: Internal server error description: Internal Server Error x-codeSamples: - lang: JavaScript source: >- import Roark from '@roarkanalytics/sdk'; const client = new Roark({ bearerToken: process.env['ROARK_API_BEARER_TOKEN'], // This is the default and can be omitted }); const response = await client.simulationRunPlanJob.getResults('jobId'); console.log(response.data); - lang: Python source: |- import os from roark_analytics import Roark client = Roark( bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"), # This is the default and can be omitted ) response = client.simulation_run_plan_job.get_results( job_id="jobId", ) print(response.data) components: schemas: SimulationRunResults: type: object properties: run: $ref: '#/components/schemas/SimulationRunResultsRun' summary: $ref: '#/components/schemas/SimulationRunResultsSummary' verdict: oneOf: - $ref: '#/components/schemas/SimulationRunPlanJobVerdict' - type: 'null' description: Null until the run ends, and for a run with no check to judge. attemptSummary: $ref: '#/components/schemas/SimulationRunPlanJobAttemptSummary' metrics: type: array items: $ref: '#/components/schemas/SimulationRunMetricResult' description: Checks first, then other metrics. calls: type: array items: $ref: '#/components/schemas/SimulationRunCallResult' breakdowns: type: object properties: byAgent: type: array items: $ref: '#/components/schemas/SimulationRunResultsGroup' byPersona: type: array items: $ref: '#/components/schemas/SimulationRunResultsGroup' required: - byAgent - byPersona comparison: oneOf: - $ref: '#/components/schemas/SimulationRunResultsComparison' - type: 'null' description: Null when there is no earlier run of the plan to compare against. sweepAttribution: oneOf: - $ref: '#/components/schemas/SimulationRunPlanJobSweepAttribution' - type: 'null' description: >- For a run that swept a property: which values the failures concentrate on. Null otherwise. debrief: oneOf: - $ref: '#/components/schemas/SimulationRunDebrief' - type: 'null' description: >- The latest debrief. Null when none was generated, e.g. a run with no graded calls. required: - run - summary - verdict - attemptSummary - metrics - calls - breakdowns - comparison - sweepAttribution - debrief description: Everything the run report shows, in one response. example: run: simulationRunPlanJobId: 7f3e4d2c-8a91-4b5c-9e6f-1a2b3c4d5e6f simulationRunPlanId: 550e8400-e29b-41d4-a716-446655440000 simulationRunPlanName: Appointment booking regression runNumber: 42 status: COMPLETED triggeredBy: TRIGGERED_FROM_API createdAt: '2026-10-11T14:00:00.000Z' startedAt: '2026-10-11T14:00:03.000Z' endedAt: '2026-10-11T14:04:41.000Z' durationMs: 278000 summary: totalCalls: 10 countedCalls: 10 completedCalls: 10 evaluatedCalls: 10 passedCalls: 7 failedCalls: 3 callPassRate: 70 score: 85 checkPassRate: 85 verdict: null attemptSummary: testCaseCount: 10 attemptCount: 10 silentAttemptCount: 0 retryCount: 0 stillSilentTestCaseCount: 0 metrics: - metricDefinitionId: bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb metricId: booked_appointment_check name: Booked appointment outputType: BOOLEAN isCheck: true unitSymbol: null evaluatedCalls: 10 sampleSize: 10 check: passedCalls: 7 failedCalls: 3 evaluatedCalls: 10 passRate: 70 numeric: null distribution: - value: 'true' count: 7 - value: 'false' count: 3 calls: - simulationJobId: 1a2b3c4d-5e6f-4a7b-8c9d-0e1f2a3b4c5d callId: 6e7f8a9b-0c1d-4e2f-9a3b-4c5d6e7f8a9b status: COMPLETED counted: true attemptNumber: 1 retryOfSimulationJobId: null personaName: Ana agentName: Front desk durationMs: 94000 passed: false checks: - metricDefinitionId: bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb metricName: Booked appointment passed: false values: [] breakdowns: byAgent: [] byPersona: [] comparison: null sweepAttribution: null debrief: null ErrorResponse: type: object properties: type: type: string enum: - validation - authentication - forbidden - not_found - conflict - payment_required - rate_limit - internal description: The error type category examples: - validation - authentication code: type: string description: Machine-readable error code identifier examples: - invalid_parameter - missing_required_field - unauthorized message: type: string description: Human-readable error message examples: - The request was invalid - Authentication required param: type: string description: The parameter that caused the error (if applicable) examples: - email - user_id details: description: Additional error context information required: - type - code - message SimulationRunResultsRun: type: object properties: simulationRunPlanJobId: type: string format: uuid simulationRunPlanId: type: string format: uuid simulationRunPlanName: type: - string - 'null' description: The run plan’s name. Null when the plan has since been deleted. runNumber: type: - integer - 'null' description: Per-project run number, shown as SR-42 in the app. status: type: string enum: - PENDING - QUEUED - CREATING_SNAPSHOTS - CREATING_SIMULATIONS - PREPARING_CAPACITY - RUNNING_SIMULATIONS - COMPLETED - FAILED - TIMED_OUT - CANCELLED - CANCELLING - ENDING_SIMULATIONS triggeredBy: type: string enum: - SCHEDULED - USER_TRIGGERED_FROM_UI - RE_RUN - TRIGGERED_FROM_API - SYSTEM createdAt: type: string startedAt: type: - string - 'null' endedAt: type: - string - 'null' durationMs: type: - integer - 'null' description: Wall-clock run time. Null until the run ends. required: - simulationRunPlanJobId - simulationRunPlanId - simulationRunPlanName - runNumber - status - triggeredBy - createdAt - startedAt - endedAt - durationMs SimulationRunResultsSummary: type: object properties: totalCalls: type: integer description: Every simulation of the run, retries included. countedCalls: type: integer description: 'Simulations whose results count: all but the invalidated ones.' completedCalls: type: integer description: Counted simulations that connected and produced a conversation. evaluatedCalls: type: integer description: Counted simulations at least one check graded. passedCalls: type: integer description: Evaluated simulations that passed every check. failedCalls: type: integer description: Evaluated simulations that failed at least one check. callPassRate: type: - number - 'null' minimum: 0 maximum: 100 description: >- passedCalls / evaluatedCalls, 0-100. Null when nothing was evaluated. score: type: - number - 'null' minimum: 0 maximum: 100 description: >- The run’s score: the mean of each check’s pass rate, the same number as `verdict.score`. 0-100. Null when nothing was evaluated. checkPassRate: type: - number - 'null' minimum: 0 maximum: 100 description: >- Every (check, simulation) verdict pooled. 0-100. Null when nothing was evaluated. required: - totalCalls - countedCalls - completedCalls - evaluatedCalls - passedCalls - failedCalls - callPassRate - score - checkPassRate SimulationRunPlanJobVerdict: type: object properties: passed: type: boolean description: >- Whether every check cleared its own `minPassRate` (and the run completed with full coverage). Use this as the CI exit status. Never derived from `score`: a run can score 95 and still fail, or score 40 and still pass. score: type: - number - 'null' minimum: 0 maximum: 100 description: >- The run's headline quality number, 0-100: the mean of each check's own pass rate. Every check weighs the same, however many simulations it evaluated, which is the same way `passed` treats them. It is the number the Roark dashboard shows for this run. REPORTING ONLY, for dashboards and trend lines. Nothing is judged against it. Null when nothing was evaluated. checks: type: array items: $ref: '#/components/schemas/SimulationRunPlanJobCheck' description: >- Every check the run was judged on, with its rate and the minimum it had to reach. Thresholds and authored yes/no metrics only. See `SimulationRunPlanJobCheck` for why a provider reading is not one. failures: type: array items: $ref: '#/components/schemas/SimulationRunPlanJobVerdictFailure' description: Every criterion the run missed. Empty when it passed. required: - passed - score - checks - failures description: >- Pass/fail verdict for the run, judged against the success criteria pinned on the run plan when the run started. example: passed: false score: 85 checks: - metricDefinitionId: aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa metricName: silence duration < 500ms passRate: 100 minPassRate: 40 inherited: false passedSims: 10 evaluatedSims: 10 passed: true - metricDefinitionId: bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb metricName: word count < 1000 passRate: 70 minPassRate: 80 inherited: false passedSims: 7 evaluatedSims: 10 passed: false failures: - type: METRIC_BELOW_MIN_PASS_RATE metricDefinitionId: bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb metricName: word count < 1000 passRate: 70 minPassRate: 80 inherited: false SimulationRunPlanJobAttemptSummary: type: object properties: testCaseCount: type: integer minimum: 0 description: >- Test cases in the run. Each one has a single final attempt that its result is read from. attemptCount: type: integer minimum: 0 description: >- Every simulation placed, retries included. Each is a separate, billed call. silentAttemptCount: type: integer minimum: 0 description: >- Simulations your agent never spoke on, including the ones a retry replaced. retryCount: type: integer minimum: 0 description: Simulations placed as a retry of one your agent never spoke on. stillSilentTestCaseCount: type: integer minimum: 0 description: >- Test cases whose final attempt your agent still never spoke on. This is what the `AGENT_NEVER_SPOKE` verdict failure counts. required: - testCaseCount - attemptCount - silentAttemptCount - retryCount - stillSilentTestCaseCount description: >- The run’s simulations against its test cases. Differs from a plain job count only when the plan retries simulations your agent never spoke on. SimulationRunMetricResult: type: object properties: metricDefinitionId: type: string format: uuid metricId: type: string description: The metric’s slug. name: type: string outputType: type: string enum: - COUNT - NUMERIC - BOOLEAN - SCALE - TEXT - CLASSIFICATION - OFFSET isCheck: type: boolean description: 'A threshold: one of the checks the run is judged on.' unitSymbol: type: - string - 'null' description: Unit of `numeric`, e.g. `ms`. evaluatedCalls: type: integer description: Counted simulations with at least one value. sampleSize: type: integer description: >- Values across those simulations. A per-turn metric has several per call. check: type: - object - 'null' properties: passedCalls: type: integer failedCalls: type: integer evaluatedCalls: type: integer passRate: type: - number - 'null' minimum: 0 maximum: 100 description: 0-100. Null when nothing was evaluated. required: - passedCalls - failedCalls - evaluatedCalls - passRate description: The check’s tally. Null for a metric that is not a check. numeric: type: - object - 'null' properties: average: type: number median: type: number p95: type: number min: type: number max: type: number required: - average - median - p95 - min - max description: >- For NUMERIC, SCALE, COUNT and OFFSET metrics. Null when nothing was captured, or without the `metric:read` permission. distribution: type: - array - 'null' items: type: object properties: value: type: string count: type: integer required: - value - count description: >- How many values took each reading, for BOOLEAN, SCALE and CLASSIFICATION metrics. Null without the `metric:read` permission. required: - metricDefinitionId - metricId - name - outputType - isCheck - unitSymbol - evaluatedCalls - sampleSize - check - numeric - distribution SimulationRunCallResult: type: object properties: simulationJobId: type: string format: uuid callId: type: - string - 'null' format: uuid description: >- Use with GET /v1/call/{callId} for the transcript, recording and every metric value. status: type: string enum: - QUEUED - CONNECTING - IN_CALL - PROCESSING - COMPLETED - ERROR - TIMED_OUT - CANCELLED - INVALIDATED - NOT_RUN description: >- QUEUED, CONNECTING, IN_CALL and PROCESSING while the simulation runs. Once settled: COMPLETED, TIMED_OUT (ran to the maximum duration, still graded), ERROR (failed on the agent’s side), NOT_RUN (failed on Roark’s side), CANCELLED, or INVALIDATED (ran, but does not count). counted: type: boolean description: 'False for an invalidated simulation: listed, never scored.' attemptNumber: type: integer retryOfSimulationJobId: type: - string - 'null' format: uuid personaName: type: - string - 'null' agentName: type: - string - 'null' durationMs: type: - integer - 'null' passed: type: - boolean - 'null' description: >- Passed every check that graded it. Null when no check graded it, or it does not count. checks: type: array items: type: object properties: metricDefinitionId: type: string format: uuid metricName: type: string passed: type: boolean required: - metricDefinitionId - metricName - passed values: type: array items: type: object properties: metricDefinitionId: type: string format: uuid metricName: type: string outputType: type: string enum: - COUNT - NUMERIC - BOOLEAN - SCALE - TEXT - CLASSIFICATION - OFFSET participantRole: type: - string - 'null' enum: - AGENT - CUSTOMER - SIMULATED_CUSTOMER - BACKGROUND_SPEAKER value: anyOf: - type: number - type: boolean - type: string - type: 'null' reasoning: type: - string - 'null' required: - metricDefinitionId - metricName - outputType - participantRole - value - reasoning description: >- Call-level values of the run’s other metrics. Empty without the `metric:read` permission. Per-turn values are on GET /v1/call/{callId}/metrics. required: - simulationJobId - callId - status - counted - attemptNumber - retryOfSimulationJobId - personaName - agentName - durationMs - passed - checks - values SimulationRunResultsGroup: type: object properties: key: type: string description: The agent or persona ID. label: type: string calls: type: integer description: Counted simulations in the group. evaluatedCalls: type: integer passedCalls: type: integer callPassRate: type: - number - 'null' minimum: 0 maximum: 100 description: 0-100. Null when nothing was evaluated. score: type: - number - 'null' minimum: 0 maximum: 100 description: The group’s own score. 0-100. Null when nothing was evaluated. required: - key - label - calls - evaluatedCalls - passedCalls - callPassRate - score SimulationRunResultsComparison: type: object properties: baselineRun: type: object properties: simulationRunPlanJobId: type: string format: uuid runNumber: type: - integer - 'null' status: type: string enum: - PENDING - QUEUED - CREATING_SNAPSHOTS - CREATING_SIMULATIONS - PREPARING_CAPACITY - RUNNING_SIMULATIONS - COMPLETED - FAILED - TIMED_OUT - CANCELLED - CANCELLING - ENDING_SIMULATIONS endedAt: type: - string - 'null' required: - simulationRunPlanJobId - runNumber - status - endedAt baselineSource: type: string enum: - PREVIOUS_RUN - REQUESTED description: >- REQUESTED when `compareToJobId` picked the baseline, PREVIOUS_RUN otherwise. currentScore: type: - number - 'null' minimum: 0 maximum: 100 baselineScore: type: - number - 'null' minimum: 0 maximum: 100 scoreDelta: type: - number - 'null' description: Current minus baseline, in points. currentCallPassRate: type: - number - 'null' minimum: 0 maximum: 100 baselineCallPassRate: type: - number - 'null' minimum: 0 maximum: 100 callPassRateDelta: type: - number - 'null' checks: type: array items: type: object properties: metricDefinitionId: type: string format: uuid metricName: type: string currentPassRate: type: - number - 'null' minimum: 0 maximum: 100 baselinePassRate: type: - number - 'null' minimum: 0 maximum: 100 delta: type: - number - 'null' required: - metricDefinitionId - metricName - currentPassRate - baselinePassRate - delta description: >- Every check either run was graded on. A side that did not grade it reads null. required: - baselineRun - baselineSource - currentScore - baselineScore - scoreDelta - currentCallPassRate - baselineCallPassRate - callPassRateDelta - checks SimulationRunPlanJobSweepAttribution: type: object properties: values: type: array items: $ref: '#/components/schemas/SimulationRunPlanJobSweepArm' description: Every value of the swept property, baseline first. checks: type: array items: $ref: '#/components/schemas/SimulationRunPlanJobSweepCheck' description: >- Every check the run was graded on: `PROPERTY_ATTRIBUTABLE` first, then `FOUND_UNATTRIBUTED`, then `WITHIN_NOISE`. testableValueCount: type: integer minimum: 0 description: How many values had enough counted simulations to be compared. testedComparisonCount: type: integer minimum: 0 description: >- How many value and check pairs were actually tested. A value can have enough simulations and still go untested (no other value graded that check, or too few verdicts on it). When this is 0 no comparison ran, so an empty `worseValues` everywhere means nothing was tested, not that no value did worse. minimumDetectableGap: type: - number - 'null' minimum: 0 maximum: 100 description: >- Roughly how many percentage points more often a value would need to fail than the rest of the run to be flagged at this sample size. Optimistic: it uses simulation counts rather than the verdicts on each check and ignores the correction, so checks graded on fewer simulations need larger gaps. Large when few simulations ran per value: finding no significant difference then means the run could not see one, not that none exists. caveats: type: array items: type: string description: >- Plain sentences on what this run could not see, ready to show to a reader. neverSpokeValues: type: array items: $ref: '#/components/schemas/SimulationRunPlanJobSweepNeverSpokeValue' description: >- The values your agent never spoke on significantly more often than the rest of the run, most significant first, tested the same way as `worseValues`. Empty on a run with no silent simulations. minArmCalls: type: integer exclusiveMinimum: 0 description: Counted simulations a value needs before it is compared. falseDiscoveryRate: type: number minimum: 0 maximum: 1 description: >- The Benjamini-Hochberg false discovery rate the comparisons are held to. foundUnattributedFailureRate: type: number minimum: 0 maximum: 100 description: >- The overall failure rate, 0-100, at which a check with no standout value is reported as `FOUND_UNATTRIBUTED`. required: - values - checks - testableValueCount - testedComparisonCount - minimumDetectableGap - caveats - neverSpokeValues - minArmCalls - falseDiscoveryRate - foundUnattributedFailureRate description: >- For a run that swept a property (accent, background noise, speech pace and so on): which check failures the property caused. Each value is compared with every other value combined using a one-sided Fisher exact test, and every comparison in the run is corrected together with Benjamini-Hochberg. A value is only called worse when the difference is statistically significant, so a value that happened to fail a few more simulations by chance is not reported as a problem. Invalidated simulations and simulations that failed on the Roark platform are left out. example: values: - key: ACCENT=US value: US label: American isBaseline: true attempted: 3 counted: 2 silentAttempts: 0 retries: 0 testable: false score: 50 - key: ACCENT=IN value: IN label: Indian isBaseline: false attempted: 8 counted: 8 silentAttempts: 0 retries: 0 testable: true score: 62.5 - key: ACCENT=GB value: GB label: British isBaseline: false attempted: 8 counted: 8 silentAttempts: 0 retries: 0 testable: true score: 68.8 checks: - metricDefinitionId: aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa metricName: Voice consistency attribution: FOUND_UNATTRIBUTED failed: 7 evaluated: 18 failureRate: 38.9 worseValues: [] testableValueCount: 2 testedComparisonCount: 8 minimumDetectableGap: 45 caveats: - >- American had fewer than 5 counted calls, so it is shown but not compared. - >- With about 8 calls per value, only large differences can be detected: a value would need to fail roughly 45 points more often than the rest to be reliably flagged. Smaller effects may exist and go unseen. neverSpokeValues: [] minArmCalls: 5 falseDiscoveryRate: 0.1 foundUnattributedFailureRate: 20 SimulationRunDebrief: type: object properties: debriefId: type: string format: uuid status: type: string enum: - PENDING - GENERATING - READY - FAILED description: >- PENDING and GENERATING while it is written (usually a few minutes), then READY or FAILED. errorMessage: type: - string - 'null' createdAt: type: string completedAt: type: - string - 'null' report: oneOf: - $ref: '#/components/schemas/SimulationRunDebriefReport' - type: 'null' description: Present once READY. required: - debriefId - status - errorMessage - createdAt - completedAt - report description: >- The run’s written debrief: executive summary, failure themes and recommendations. Generated automatically when a run with graded calls settles. example: debriefId: 3c2b1a09-8f7e-4d6c-9b5a-4e3d2c1b0a9f status: GENERATING errorMessage: null createdAt: '2026-10-11T14:05:12.000Z' completedAt: null report: null SimulationRunPlanJobCheck: type: object properties: metricDefinitionId: type: string format: uuid metricName: type: - string - 'null' passRate: type: - number - 'null' minimum: 0 maximum: 100 description: >- This check's own pass rate, 0-100: the share of sims it evaluated that passed it. Null when it evaluated nothing. minPassRate: type: number minimum: 0 maximum: 100 description: >- THE BAR it was judged against: the share of the run's simulations it had to pass. inherited: type: boolean description: >- Whether `minPassRate` is the 80% default (`true`) or a minimum this metric set for itself. passedSims: type: integer minimum: 0 evaluatedSims: type: integer minimum: 0 passed: type: boolean description: '`passRate >= minPassRate`.' required: - metricDefinitionId - passRate - minPassRate - inherited - passedSims - evaluatedSims - passed description: >- How one check did. Present for every check the run was judged on, passing or not. A check is a metric that yields a pass/fail: a threshold (`Silence Duration <= 2s`) or a yes/no metric you authored. Provider readings such as `Comprehension Failure` are observations, not checks: their `true` is whatever the underlying field happens to mean, so they carry no passing side and never appear here. Put a threshold on one to judge it. SimulationRunPlanJobVerdictFailure: oneOf: - type: object properties: type: type: string const: RUN_NOT_COMPLETED status: type: string enum: - PENDING - QUEUED - CREATING_SNAPSHOTS - CREATING_SIMULATIONS - PREPARING_CAPACITY - RUNNING_SIMULATIONS - COMPLETED - FAILED - TIMED_OUT - CANCELLED - CANCELLING - ENDING_SIMULATIONS description: The status the run actually ended in. required: - type - status - type: object properties: type: type: string const: INCOMPLETE_COVERAGE evaluatedCalls: type: integer minimum: 0 expectedCalls: type: integer minimum: 0 required: - type - evaluatedCalls - expectedCalls - type: object properties: type: type: string const: AGENT_NEVER_SPOKE neverSpokeCalls: type: integer minimum: 0 description: >- Test cases your agent never spoke on, judged on each test case’s last attempt: a silent simulation a retry later reached your agent on does not count here. totalCalls: type: integer minimum: 0 description: >- Every test case of the run whose last attempt reached your agent. maxShare: type: number minimum: 0 maximum: 100 description: >- The largest share of simulations, 0-100, your agent may never speak on while the run can pass. agentName: type: - string - 'null' description: >- The agent as the run tested it. Null when the simulations tested more than one agent. silentAttempts: type: integer minimum: 0 description: >- Every simulation your agent never spoke on, including the ones a retry replaced. Above `neverSpokeCalls` only when the plan retries silent simulations (`maxNoResponseRetries`). required: - type - neverSpokeCalls - totalCalls - maxShare description: >- Your agent never spoke on more than `maxShare` percent of the run. Those simulations are left out of every check, so the run fails on them whatever the checks say. - type: object properties: type: type: string const: METRIC_NOT_EVALUATED metricDefinitionId: type: string format: uuid metricName: type: - string - 'null' description: The check’s name, for rendering the failure. required: - type - metricDefinitionId - type: object properties: type: type: string const: METRIC_BELOW_MIN_PASS_RATE metricDefinitionId: type: string format: uuid metricName: type: - string - 'null' description: The check’s name, for rendering the failure. passRate: type: number minimum: 0 maximum: 100 minPassRate: type: number minimum: 0 maximum: 100 inherited: type: boolean description: >- Whether the missed minimum was the 80% default (`true`) or this metric's own. required: - type - metricDefinitionId - passRate - minPassRate - inherited description: >- One reason the run failed. Every reason is a hard fail: the run passes only when this list is empty. SimulationRunPlanJobSweepArm: type: object properties: key: type: string description: >- The arm's identity: the override signature its calls ran with, e.g. `BACKGROUND_NOISE=DRIVING;BACKGROUND_NOISE_VOLUME=0.7`. Join on this rather than on `value`: a sweep can run one value as several arms (Driving at 70% and at 100%), and those share a value. value: type: string description: The stored value of the swept property, e.g. `US`. label: type: string description: >- The value in words, with what else the arm pinned, e.g. `American` or `Driving (70% noise)`. isBaseline: type: boolean description: Whether this is the baseline the plan named. attempted: type: integer minimum: 0 description: >- Simulations run at this value. Simulations that failed on the Roark platform are left out, since they say nothing about your agent. counted: type: integer minimum: 0 description: >- Simulations that actually tested the property: not invalidated, and graded by at least one check. Only these are used in the comparison. silentAttempts: type: integer minimum: 0 description: >- Of `attempted`, the simulations your agent never spoke on, whether or not a retry followed. They are invalidated, so they are never in `counted`. retries: type: integer minimum: 0 description: >- Of `attempted`, the retries of simulations your agent never spoke on. Zero unless the plan retries silent simulations. testable: type: boolean description: >- Whether this value had at least `minArmCalls` counted simulations. A value below that is "insufficient data": it is reported with its numbers but never compared or ranked, because a rate from one or two calls cannot be told apart from luck. score: type: - number - 'null' minimum: 0 maximum: 100 description: >- The mean of each check's pass rate at this value, 0-100, the same rule as the run's `score`. Descriptive only: a lower score alone never makes a value worse. Null when nothing counted. required: - key - value - label - isBaseline - attempted - counted - silentAttempts - retries - testable - score description: One value of the swept property. SimulationRunPlanJobSweepCheck: type: object properties: metricDefinitionId: type: string format: uuid metricName: type: string attribution: type: string enum: - PROPERTY_ATTRIBUTABLE - FOUND_UNATTRIBUTED - WITHIN_NOISE description: >- How this check relates to the swept property: - `PROPERTY_ATTRIBUTABLE`: at least one value failed it significantly more often than every other value combined. `worseValues` names them. This is the only case in which a value is called worse. - `FOUND_UNATTRIBUTED`: no value stands out, but the check failed on at least `foundUnattributedFailureRate`% of counted simulations overall. A real issue with the agent on which no value stood out, so the run cannot tie it to the property. It does not show the property had no effect: a small run may be unable to see one. - `WITHIN_NOISE`: neither. Any differences between values are within what chance produces. failed: type: integer minimum: 0 evaluated: type: integer minimum: 0 failureRate: type: - number - 'null' minimum: 0 maximum: 100 description: >- Share of counted simulations across every value that failed the check, 0-100. worseValues: type: array items: $ref: '#/components/schemas/SimulationRunPlanJobSweepArmCheck' description: >- The values significantly worse than the rest, most significant first. Empty unless `attribution` is `PROPERTY_ATTRIBUTABLE`. required: - metricDefinitionId - metricName - attribution - failed - evaluated - failureRate - worseValues description: How one check's failures relate to the swept property. SimulationRunPlanJobSweepNeverSpokeValue: type: object properties: key: type: string description: >- The arm's identity: the override signature its calls ran with, e.g. `BACKGROUND_NOISE=DRIVING;BACKGROUND_NOISE_VOLUME=0.7`. Join on this rather than on `value`: a sweep can run one value as several arms (Driving at 70% and at 100%), and those share a value. value: type: string label: type: string silentAttempts: type: integer minimum: 0 description: Simulations at this value your agent never spoke on, retried or not. attempted: type: integer minimum: 0 description: Simulations run at this value. retries: type: integer minimum: 0 description: Retries placed at this value because your agent never spoke. restSilentAttempts: type: integer minimum: 0 description: >- Simulations your agent never spoke on across every other value combined. restAttempted: type: integer minimum: 0 description: Simulations run across every other value combined. adjustedPValue: type: number minimum: 0 maximum: 1 description: >- How likely a difference at least this large would be by chance alone, after correcting across the values (Benjamini-Hochberg). required: - key - value - label - silentAttempts - attempted - retries - restSilentAttempts - restAttempted - adjustedPValue description: >- A value your agent never spoke on significantly more often than the rest of the run. SimulationRunDebriefReport: type: object properties: generatedAt: type: string verdict: type: string enum: - PASSED - THRESHOLD_FAILED verdictSummary: type: string description: One-paragraph verdict, e.g. "Not ready to promote. ..." executiveSummary: type: object properties: paragraphs: type: array items: type: string sections: type: array items: type: object properties: heading: type: - string - 'null' body: type: string required: - heading - body required: - paragraphs - sections keyFindings: type: array items: type: object properties: title: type: string body: type: string callIds: type: array items: type: string description: The calls the finding is about. required: - title - body - callIds wentWell: type: array items: type: object properties: title: type: string body: type: string callIds: type: array items: type: string description: The calls the finding is about. required: - title - body - callIds themes: type: array items: type: object properties: slug: type: string title: type: string severity: type: string enum: - HIGH - MEDIUM - LOW impact: type: - string - 'null' enum: - CRITICAL - HIGH - MEDIUM - LOW prevalence: type: - string - 'null' enum: - WIDESPREAD - RECURRING - ISOLATED lifecycle: type: - string - 'null' enum: - NEW - PERSISTENT - IMPROVING - FIXED affectedCallCount: type: integer totalCallCount: type: integer checkNames: type: array items: type: string rootCauseChannel: type: - string - 'null' enum: - PROMPT - VOICE_CONFIG - STT - SPEECH_MODEL - LLM_MODEL - INFRA - EXTERNAL analysis: type: string callIds: type: array items: type: string required: - slug - title - severity - impact - prevalence - lifecycle - affectedCallCount - totalCallCount - checkNames - rootCauseChannel - analysis - callIds description: Failure themes, clustered across the run’s failing calls. recommendations: type: array items: type: object properties: order: type: integer title: type: string body: type: string severity: type: - string - 'null' enum: - HIGH - MEDIUM - LOW channel: type: - string - 'null' enum: - PROMPT - VOICE_CONFIG - STT - SPEECH_MODEL - LLM_MODEL - INFRA - EXTERNAL currentText: type: - string - 'null' description: 'For a prompt fix: the text to replace.' suggestedText: type: - string - 'null' description: 'For a prompt fix: the replacement.' themeSlugs: type: array items: type: string required: - order - title - body - severity - channel - currentText - suggestedText - themeSlugs trend: type: - object - 'null' properties: baselineRunPlanJobId: type: string baselineRunNumber: type: - integer - 'null' rows: type: array items: type: object properties: checkName: type: string previousPassRate: type: - number - 'null' minimum: 0 maximum: 100 currentPassRate: type: - number - 'null' minimum: 0 maximum: 100 lifecycle: type: - string - 'null' enum: - NEW - PERSISTENT - IMPROVING - FIXED required: - checkName - previousPassRate - currentPassRate - lifecycle required: - baselineRunPlanJobId - baselineRunNumber - rows description: The checks against the previous run, as of generation. required: - generatedAt - verdict - verdictSummary - executiveSummary - keyFindings - wentWell - themes - recommendations - trend SimulationRunPlanJobSweepArmCheck: type: object properties: key: type: string description: >- The arm's identity: the override signature its calls ran with, e.g. `BACKGROUND_NOISE=DRIVING;BACKGROUND_NOISE_VOLUME=0.7`. Join on this rather than on `value`: a sweep can run one value as several arms (Driving at 70% and at 100%), and those share a value. value: type: string label: type: string failed: type: integer minimum: 0 evaluated: type: integer minimum: 0 failureRate: type: - number - 'null' minimum: 0 maximum: 100 description: >- Share of the counted simulations at this value that failed the check, 0-100. restFailureRate: type: - number - 'null' minimum: 0 maximum: 100 description: >- The same share across every other value combined, 0-100. This is what the value is compared with. adjustedPValue: type: number minimum: 0 maximum: 1 description: >- How likely a difference at least this large would be by chance alone, after correcting for every comparison in the run (Benjamini-Hochberg). The value is called worse only when this is at most `falseDiscoveryRate`. required: - key - value - label - failed - evaluated - failureRate - restFailureRate - adjustedPValue description: >- A value that failed a check significantly more often than the rest of the run. securitySchemes: Bearer: type: http scheme: bearer bearerFormat: JWT ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.