LLM Judge

An LLM Judge metric is a prompt. You describe what to measure, and Roark Prism, our evaluation model built for voice AI, reads each call and returns a typed result with its reasoning.

"Did the agent verify the caller's identity?"           →  Boolean
"Rate the agent's empathy on a 1-5 scale"               →  Scale
"What was the primary reason for the call?"             →  Classification
"How many times did the agent attempt to upsell?"       →  Count

Use an LLM Judge for anything that takes understanding the conversation: whether a step was followed, how well something was handled, why the customer called. When the answer is already in other metrics, a Formula or Pattern is cheaper and deterministic. When it lives in your backend, use an API check.

Quickstart

Define the metric in code (recommended), or build it in Studio.

import { defineMetric, defineConfig } from '@roarkanalytics/sdk/lib/config';

const identityVerified = defineMetric({
  name: 'identity_verified', // the slug a collector references
  displayName: 'Identity Verified',
  type: 'BOOLEAN', // BOOLEAN | SCALE | NUMERIC | TEXT | CLASSIFICATION
  prompt:
    'Using {{transcript}} and {{world_context}}, did the agent verify the caller identity before sharing account details?',
  trueLabel: 'Verified',
  falseLabel: 'Not Verified',
});

export default defineConfig({
  resources: [identityVerified],
});

# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: metric
name: identity_verified            # the slug a collector references
displayName: Identity Verified
type: BOOLEAN                      # BOOLEAN | SCALE | NUMERIC | TEXT | CLASSIFICATION
prompt: |
  Using {{transcript}} and {{world_context}}, did the agent verify the caller identity before sharing account details?
trueLabel: Verified
falseLabel: Not Verified

roark config apply roark.config.ts

See the config as code metrics reference for SCALE, CLASSIFICATION, and PER_PARTICIPANT fields.

Create it with the SDK

Create the metric with calculationType: "LLM_JUDGE". It’s the default, so requests that leave calculationType out create an LLM Judge too.

import Roark from '@roarkanalytics/sdk'

const client = new Roark({ bearerToken: process.env.ROARK_API_BEARER_TOKEN })

const metric = await client.metric.createDefinition({
  calculationType: 'LLM_JUDGE',
  name: 'Customer Satisfaction',
  outputType: 'SCALE',
  llmPrompt: 'Using {{transcript}}, rate the overall customer satisfaction based on the tone, resolution, and agent helpfulness.',
  scaleMin: 1,
  scaleMax: 10,
  scaleLabels: [
    { rangeMin: 1, rangeMax: 3, label: 'Poor', displayOrder: 1 },
    { rangeMin: 4, rangeMax: 6, label: 'Average', displayOrder: 2 },
    { rangeMin: 7, rangeMax: 10, label: 'Excellent', displayOrder: 3 },
  ],
})

import os
from roark_analytics import Roark

client = Roark(bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"))

metric = client.metric.create_definition(
    calculation_type="LLM_JUDGE",
    name="Customer Satisfaction",
    output_type="SCALE",
    llm_prompt="Using {{transcript}}, rate the overall customer satisfaction based on the tone, resolution, and agent helpfulness.",
    scale_min=1,
    scale_max=10,
    scale_labels=[
        {"range_min": 1, "range_max": 3, "label": "Poor", "display_order": 1},
        {"range_min": 4, "range_max": 6, "label": "Average", "display_order": 2},
        {"range_min": 7, "range_max": 10, "label": "Excellent", "display_order": 3},
    ],
)

curl -X POST https://api.roark.ai/v1/metric/definitions \
  -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "calculationType": "LLM_JUDGE",
    "name": "Identity Verified",
    "outputType": "BOOLEAN",
    "llmPrompt": "Using {{transcript}}, did the agent verify the caller identity before proceeding with the request?",
    "booleanTrueLabel": "Verified",
    "booleanFalseLabel": "Not Verified"
  }'

Field	Type	Required	Description
`name`	string	Yes	Name of the metric (1-100 characters)
`outputType`	string	Yes	One of `BOOLEAN`, `NUMERIC`, `TEXT`, `SCALE`, `CLASSIFICATION`, `COUNT`, `OFFSET`
`llmPrompt`	string	No	The instructions the judge follows (max 2000 characters). See Prompt Variables
`slug`	string	No	Stable identifier. Generated from the name if omitted
`scope`	string	No	`GLOBAL` (default) or `PER_PARTICIPANT`
`participantRole`	string	Conditional	Required when scope is `PER_PARTICIPANT`
`supportedContexts`	string[]	No	Defaults to `["CALL"]`
`analysisPackageId`	string	No	Package to add the metric to. Defaults to your project’s Custom Metrics package

Type-specific fields:

Field	Applies To	Description
`booleanTrueLabel` / `booleanFalseLabel`	BOOLEAN	Labels for the two outcomes
`scaleMin` / `scaleMax`	SCALE	Range boundaries (0-100)
`scaleLabels`	SCALE	Label objects with `rangeMin`, `rangeMax`, `label`, `displayOrder`
`classificationOptions`	CLASSIFICATION	Options with `label`, `description`, `displayOrder`
`maxClassifications`	CLASSIFICATION	Maximum number of options the judge can pick

To change the prompt or labels later, PUT /v1/metric/definitions/{idOrSlug} with the fields to change. Each change creates a new metric version.

Build it in Studio

Open Studio in Author mode and pick the LLM Judge engine. Choose an output type, then write the instructions: for a Boolean, the yes/no question the judge should answer; for a Scale, what each band means. Use the test rail to score real calls before you save.

Studio Author mode: the numbered Engine, Output (with range labels), Instructions, and Group sections, plus a Tests rail of scored test calls

Output Types

Type	Returns	Configure
`BOOLEAN`	Yes or no	Labels for each side, e.g. Verified / Not Verified
`SCALE`	A score in a range	`scaleMin`, `scaleMax`, and labels for bands of the range
`CLASSIFICATION`	One or more labels from a list	The options, and how many can be picked
`NUMERIC` / `COUNT`	A number	Nothing extra
`TEXT`	Free text	Nothing extra

Prompt Variables

Prompts reference call data with {{variable}} placeholders, which Roark fills in before the judge runs. In Studio, type {{ in the instructions to pick one.

Variable	What it inserts
`{{transcript}}`	The full transcript, from the best available source
`{{transcript.post_call}}` / `{{transcript.realtime}}`	A specific transcript source
`{{transcript.participant}}`	The transcript filtered to the participant being scored
`{{summary}}`	The AI-generated call summary
`{{tool_invocations}}`	Tool calls made during the call, with parameters and results (JSON)
`{{properties}}`	All of the call’s custom properties (JSON)
`{{properties.<name>}}`	One custom property’s value, e.g. `{{properties.partner_outcome}}`
`{{interruptions}}` / `{{silences}}`	Overlaps and silences, with timing (JSON)
`{{sentiment}}` / `{{emotions}}` / `{{vocal_cues}}`	Sentiment, emotion and vocal cue analysis
`{{duration}}` / `{{call_status}}` / `{{call_ended_status}}` / `{{call_id}}`	Call metadata
`{{world_context}}`	The current date, time and timezone

Using a custom property

{{properties.<name>}} inserts just that property’s value, which makes it easy to grade the conversation against data from your own systems:

Our partner reported the outcome {{properties.partner_outcome}} and an APR of
{{properties.purchase_apr_total}}. Using {{transcript}}, did the representative
state that APR on the call?

The name can be written as stored (partner_outcome) or as you send it (partnerOutcome).
If the call doesn’t carry the property, the prompt says so ([partner_outcome property not set on this conversation]) rather than leaving the placeholder in.
Properties added after the call was created are picked up the next time the metric runs. See updating properties after ingest.

Writing Good Prompts

Ask one thing

A prompt that checks identity, tone, and resolution at once gives you one number you can’t act on. Write three focused metrics and combine them with a formula if you need a single pass/fail.

Say what counts

Spell out the edge cases: “Verification counts if the agent confirmed two of name, date of birth, and postcode. Asking without confirming doesn’t count.”

Define every band of a scale

For a Scale, describe what a 1, a 3, and a 5 look like. Without that, scores drift between calls that are about the same.

Test on real calls first

Run the metric against representative calls in Studio and read the reasoning before attaching it to a collector.

What’s Next

Studio

Write and test a prompt against real calls

Formulas

Combine focused judges into one score

Thresholds

Turn a score into a pass/fail check

Collectors

Run the metric automatically on live calls

Getting Started

Simulations

Observability

Metrics

Autoimprove

Recipes

Integrations

Enterprise

Resources

Quickstart

Create it with the SDK

Build it in Studio

Output Types

Prompt Variables

Using a custom property

Writing Good Prompts

What’s Next

Studio

Formulas

Thresholds

Collectors

​Quickstart

​Create it with the SDK

​Build it in Studio

​Output Types

​Prompt Variables

​Using a custom property

​Writing Good Prompts

​What’s Next

Studio

Formulas

Thresholds

Collectors

Quickstart

Create it with the SDK

Build it in Studio

Output Types

Prompt Variables

Using a custom property

Writing Good Prompts

What’s Next