> ## Documentation Index
> Fetch the complete documentation index at: https://docs.roark.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Judge

> Describe what to measure in plain language and let Roark score every call against it

An LLM Judge metric is a prompt. You describe what to measure, and **Roark Prism**, our evaluation model built for voice AI, reads each call and returns a typed result with its reasoning.

```
"Did the agent verify the caller's identity?"           →  Boolean
"Rate the agent's empathy on a 1-5 scale"               →  Scale
"What was the primary reason for the call?"             →  Classification
"How many times did the agent attempt to upsell?"       →  Count
```

Use an LLM Judge for anything that takes understanding the conversation: whether a step was followed, how well something was handled, why the customer called. When the answer is already in other metrics, a [Formula](https://docs.roark.ai/documentation/metrics/formulas.md) or [Pattern](https://docs.roark.ai/documentation/metrics/patterns.md) is cheaper and deterministic. When it lives in your backend, use an [API check](https://docs.roark.ai/documentation/metrics/api-checks.md).

***

## Output Types

| Type | Returns | Configure |
| :- | :- | :- |
| `BOOLEAN` | Yes or no | Labels for each side, e.g. **Verified** / **Not Verified** |
| `SCALE` | A score in a range | `scaleMin`, `scaleMax`, and labels for bands of the range |
| `CLASSIFICATION` | One or more labels from a list | The options, and how many can be picked |
| `NUMERIC` / `COUNT` | A number | Nothing extra |
| `TEXT` | Free text | Nothing extra |

***

## Creating an LLM Judge

### In Studio

Open [Studio](https://docs.roark.ai/documentation/metrics/studio.md) in Author mode and pick the **LLM Judge** engine. Choose an output type, then write the instructions: for a Boolean, the yes/no question the judge should answer; for a Scale, what each band means. Use the test rail to score real calls before you save.

![Studio Author mode: the numbered Engine, Output (with range labels), Instructions, and Group sections, plus a Tests rail of scored test calls](https://mintcdn.com/roark/IjseSxRBSMbit4Aq/images/metrics/studio-author.png?fit=max\&auto=format\&n=IjseSxRBSMbit4Aq\&q=85\&s=b70e3d6a98d20fe838b6fbd6844696ba)

### In code

Define the metric [in code](https://docs.roark.ai/documentation/config-as-code/metrics.md) to keep it in version control with the rest of your Roark config:

```typescript roark.config.ts theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import { defineMetric, defineConfig } from '@roarkanalytics/sdk/lib/config';

const identityVerified = defineMetric({
  name: 'identity_verified', // the slug a collector references
  displayName: 'Identity Verified',
  type: 'BOOLEAN', // BOOLEAN | SCALE | NUMERIC | TEXT | CLASSIFICATION
  prompt:
    'Using {{transcript}} and {{world_context}}, did the agent verify the caller identity before sharing account details?',
  trueLabel: 'Verified',
  falseLabel: 'Not Verified',
});

export default defineConfig({ resources: [identityVerified] });
```

```yaml YAML theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
# yaml-language-server: $schema=https://roark.ai/roark-config.schema.json
kind: metric
name: identity_verified            # the slug a collector references
displayName: Identity Verified
type: BOOLEAN                      # BOOLEAN | SCALE | NUMERIC | TEXT | CLASSIFICATION
prompt: |
  Using {{transcript}} and {{world_context}}, did the agent verify the caller identity before sharing account details?
trueLabel: Verified
falseLabel: Not Verified
```

```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
roark config apply roark.config.ts
```

See the [config as code metrics reference](https://docs.roark.ai/documentation/config-as-code/metrics.md) for SCALE, CLASSIFICATION, and `PER_PARTICIPANT` fields.

### With the SDK or API

Create the metric with `calculationType: "LLM_JUDGE"`. It's the default, so requests that leave `calculationType` out create an LLM Judge too.

```typescript Node theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import Roark from '@roarkanalytics/sdk'

const client = new Roark({ bearerToken: process.env.ROARK_API_BEARER_TOKEN })

const metric = await client.metric.createDefinition({
  calculationType: 'LLM_JUDGE',
  name: 'Customer Satisfaction',
  outputType: 'SCALE',
  llmPrompt: 'Using {{transcript}}, rate the overall customer satisfaction based on the tone, resolution, and agent helpfulness.',
  scaleMin: 1,
  scaleMax: 10,
  scaleLabels: [
    { rangeMin: 1, rangeMax: 3, label: 'Poor', displayOrder: 1 },
    { rangeMin: 4, rangeMax: 6, label: 'Average', displayOrder: 2 },
    { rangeMin: 7, rangeMax: 10, label: 'Excellent', displayOrder: 3 },
  ],
})
```

```python Python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
import os
from roark_analytics import Roark

client = Roark(bearer_token=os.environ.get("ROARK_API_BEARER_TOKEN"))

metric = client.metric.create_definition(
    calculation_type="LLM_JUDGE",
    name="Customer Satisfaction",
    output_type="SCALE",
    llm_prompt="Using {{transcript}}, rate the overall customer satisfaction based on the tone, resolution, and agent helpfulness.",
    scale_min=1,
    scale_max=10,
    scale_labels=[
        {"range_min": 1, "range_max": 3, "label": "Poor", "display_order": 1},
        {"range_min": 4, "range_max": 6, "label": "Average", "display_order": 2},
        {"range_min": 7, "range_max": 10, "label": "Excellent", "display_order": 3},
    ],
)
```

```bash cURL theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
curl -X POST https://api.roark.ai/v1/metric/definitions \
  -H "Authorization: Bearer $ROARK_API_BEARER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "calculationType": "LLM_JUDGE",
    "name": "Identity Verified",
    "outputType": "BOOLEAN",
    "llmPrompt": "Using {{transcript}}, did the agent verify the caller identity before proceeding with the request?",
    "booleanTrueLabel": "Verified",
    "booleanFalseLabel": "Not Verified"
  }'
```

| Field | Type | Required | Description |
| :- | :- | :- | :- |
| `name` | string | Yes | Name of the metric (1-100 characters) |
| `outputType` | string | Yes | One of `BOOLEAN`, `NUMERIC`, `TEXT`, `SCALE`, `CLASSIFICATION`, `COUNT`, `OFFSET` |
| `llmPrompt` | string | No | The instructions the judge follows (max 2000 characters). See [Prompt Variables](#prompt-variables) |
| `slug` | string | No | Stable identifier. Generated from the name if omitted |
| `scope` | string | No | `GLOBAL` (default) or `PER_PARTICIPANT` |
| `participantRole` | string | Conditional | Required when scope is `PER_PARTICIPANT` |
| `supportedContexts` | string\[] | No | Defaults to `["CALL"]` |
| `analysisPackageId` | string | No | Package to add the metric to. Defaults to your project's Custom Metrics package |

**Type-specific fields:**

| Field | Applies To | Description |
| :- | :- | :- |
| `booleanTrueLabel` / `booleanFalseLabel` | BOOLEAN | Labels for the two outcomes |
| `scaleMin` / `scaleMax` | SCALE | Range boundaries (0-100) |
| `scaleLabels` | SCALE | Label objects with `rangeMin`, `rangeMax`, `label`, `displayOrder` |
| `classificationOptions` | CLASSIFICATION | Options with `label`, `description`, `displayOrder` |
| `maxClassifications` | CLASSIFICATION | Maximum number of options the judge can pick |

To change the prompt or labels later, `PUT /v1/metric/definitions/{idOrSlug}` with the fields to change. Each change creates a new metric version.

***

## Prompt Variables

Prompts reference call data with `{{variable}}` placeholders, which Roark fills in before the judge runs. In Studio, type `{{` in the instructions to pick one.

| Variable | What it inserts |
| :- | :- |
| `{{transcript}}` | The full transcript, from the best available source |
| `{{transcript.post_call}}` / `{{transcript.realtime}}` | A specific transcript source |
| `{{transcript.participant}}` | The transcript filtered to the participant being scored |
| `{{summary}}` | The AI-generated call summary |
| `{{tool_invocations}}` | Tool calls made during the call, with parameters and results (JSON) |
| `{{properties}}` | All of the call's custom properties (JSON) |
| `{{properties.<name>}}` | One custom property's value, e.g. `{{properties.partner_outcome}}` |
| `{{interruptions}}` / `{{silences}}` | Overlaps and silences, with timing (JSON) |
| `{{sentiment}}` / `{{emotions}}` / `{{vocal_cues}}` | Sentiment, emotion and vocal cue analysis |
| `{{duration}}` / `{{call_status}}` / `{{call_ended_status}}` / `{{call_id}}` | Call metadata |
| `{{world_context}}` | The current date, time and timezone |

### Using a custom property

`{{properties.<name>}}` inserts just that property's value, which makes it easy to grade the conversation against data from your own systems:

```text theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}}
Our partner reported the outcome {{properties.partner_outcome}} and an APR of
{{properties.purchase_apr_total}}. Using {{transcript}}, did the representative
state that APR on the call?
```

* The name can be written as stored (`partner_outcome`) or as you send it (`partnerOutcome`).
* If the call doesn't carry the property, the prompt says so (`[partner_outcome property not set on this conversation]`) rather than leaving the placeholder in.
* Properties added after the call was created are picked up the next time the metric runs. See [updating properties after ingest](https://docs.roark.ai/documentation/integrations/custom-integrations.md#updating-properties-after-ingest).

***

## Writing Good Prompts

> **Ask one thing**
>
> A prompt that checks identity, tone, and resolution at once gives you one number you can't act on. Write three focused metrics and combine them with a [formula](https://docs.roark.ai/documentation/metrics/formulas.md) if you need a single pass/fail.

> **Say what counts**
>
> Spell out the edge cases: "Verification counts if the agent confirmed two of name, date of birth, and postcode. Asking without confirming doesn't count."

> **Define every band of a scale**
>
> For a Scale, describe what a 1, a 3, and a 5 look like. Without that, scores drift between calls that are about the same.

> **Test on real calls first**
>
> Run the metric against representative calls in [Studio](https://docs.roark.ai/documentation/metrics/studio.md) and read the reasoning before attaching it to a collector.

***

## What's Next

> **[Studio](https://docs.roark.ai/documentation/metrics/studio.md)**
>
> Write and test a prompt against real calls

> **[Formulas](https://docs.roark.ai/documentation/metrics/formulas.md)**
>
> Combine focused judges into one score

> **[Thresholds](https://docs.roark.ai/documentation/metrics/thresholds.md)**
>
> Turn a score into a pass/fail check

> **[Collectors](https://docs.roark.ai/documentation/metrics/metric-collectors.md)**
>
> Run the metric automatically on live calls


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.