> ## Documentation Index
> Fetch the complete documentation index at: https://docs.roark.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Studio

> Author, test, and evaluate metrics in one workbench before they run on live traffic

Studio is the metric workbench — build metrics, test them against real calls, and run evaluation batteries, all before anything touches production traffic. Find it in the sidebar under **Measure → Studio**.

<Note>
  Studio replaces the Playground. If you're looking for the old Playground docs, everything it did — and more — now lives here.
</Note>

Studio has three modes:

| Mode         | What it's for                                                                        |
| ------------ | ------------------------------------------------------------------------------------ |
| **Hub**      | The landing page — start authoring or evaluating, and revisit recent evaluation runs |
| **Author**   | Define or edit a single metric, with a test-call rail and version history            |
| **Evaluate** | Run a battery of metrics across a set of calls and compare results side-by-side      |

***

## The hub

Opening Studio lands you on the hub — "Build metrics. Run evaluations." — with two start cards:

* **Define a new metric** opens Author mode with a blank metric.
* **Run a battery of evals** opens Evaluate mode.

Below the cards, **Recent evaluation runs** lists your past batteries (e.g. "3 metrics × 5 calls") with a status chip — In progress, All passing, "N failing", Canceled, or Failed. Reopen any run to see its full result matrix, or fork it into a new evaluation.

***

## Author mode

Author mode is the metric editor. You reach it from the hub, from the **New metric** button on the Metrics page, or by clicking any metric in the [Library](/documentation/metrics/overview).

The editor walks you through four numbered sections:

<Steps>
  <Step title="01 · Engine — what kind of metric is this?">
    Pick how the metric is computed:

    | Engine        | How it works                                                                   |
    | ------------- | ------------------------------------------------------------------------------ |
    | **LLM Judge** | Scores each call against your natural-language prompt                          |
    | **Pattern**   | Composes triggers, outcomes, and time windows — fires when the pattern matches |
    | **Formula**   | Math over other metrics                                                        |

    <Warning>
      The engine is **locked after the first save**. To measure the same thing with a different engine, create a new metric.
    </Warning>
  </Step>

  <Step title="02 · Output — what does the metric return?">
    Choose the result type. Some types need extra configuration:

    | Output             | Returns                | Extra config             |
    | ------------------ | ---------------------- | ------------------------ |
    | **Boolean**        | Yes / No               | True and False labels    |
    | **Scale**          | A score in a range     | Min/Max plus band labels |
    | **Classification** | One of your categories | Category list            |
    | **Numeric**        | A number               | A unit                   |
    | **Count**          | An occurrence count    | A unit                   |
    | **Text**           | Free-form text         | —                        |
  </Step>

  <Step title="03 · Instructions, Logic, or Formula">
    The third section adapts to your engine. For **LLM Judge**, write instructions tuned to the output type — for a Boolean metric, the yes/no question the model should answer; for a Scale, what each band means. For **Pattern**, define the logic: when should this metric fire? For **Formula**, define the computation over other metrics.
  </Step>

  <Step title="04 · Group — where should this metric live?">
    Assign the metric to a group (analysis package). Groups organize the Library on the Metrics page, so pick one your team will look for it under.
  </Step>
</Steps>

### Test against real calls

The right-hand rail is your test bench. Use **Add test calls** to pick the calls Studio should evaluate the metric against, then run them individually or hit **Run all**. Each run streams through the evaluation pipeline live, so you can watch the result land and tighten your instructions between runs.

<Tip>
  Iterate here until the metric returns what you expect on calls you already know the answer for — before you apply it to live traffic.
</Tip>

### Version history

Every save publishes a new version. The **History** tab in the rail lists them all — view any past version, or restore one, which publishes a new version from the old one (you never lose the intermediate history).

### System metrics and variants

[System metrics](/documentation/metrics/system-metrics) open read-only in Author mode. To change one, click **Customize for your team** — this forks an org-scoped variant you can freely edit and save. The Roark-managed original stays intact, and your instructions go in an "Additional instructions" field layered on top of the system prompt.

### Editing a metric that's already live

If the metric is bound to collectors, a notice appears above the form: **"In use by N collectors — saving publishes a new version every collector will pick up."**

<Warning>
  Saving a metric republishes it everywhere it runs. Every collector using the metric picks up the new version for calls scored from that point on — test your changes in the rail before saving.
</Warning>

***

## Evaluate mode

Evaluate mode runs a battery of metrics across a set of calls and compares results side-by-side. Use it to validate a metric at scale, compare candidate metrics against each other, or spot-check quality across a sample of production calls.

<Steps>
  <Step title="Pick metrics">
    Use **Add metrics** to choose one or more metrics from your Library. You can attach a pass/fail threshold inline per metric (e.g. "Pass if ≥ 0.7") to turn raw scores into verdicts — see [Thresholds](/documentation/metrics/thresholds).
  </Step>

  <Step title="Pick calls">
    Use **Add calls** to select the conversations to evaluate.
  </Step>

  <Step title="Run battery">
    Hit **Run battery**. Results stream in as a comparison **matrix** (metrics × calls) or grouped **by call** — switch views to suit the question you're asking. You can cancel a run in progress.
  </Step>
</Steps>

Finished batteries appear under **Recent evaluation runs** on the hub, so you can reopen the matrix later or fork the run into a new evaluation.

<Note>
  Prefer to run evaluations programmatically? [Metric collection jobs](/documentation/metrics/metric-collection-jobs) are the API equivalent — run a set of metrics across a set of calls via the SDK or REST.
</Note>

***

## Apply a metric to live calls

Testing in Studio scores only the calls you pick. To score matching calls automatically as they come in, bind the metric to a collector. The Author mode footer — **"Apply this metric to live calls"** — has an **Add collector** button that opens the collector editor with your metric prefilled and the modality locked to what the metric supports.

Once bound, the footer shows **"Running on N collectors"** instead. See [Collectors](/documentation/metrics/metric-collectors) for how conditions and scoping work.

<Note>
  The REST API and SDKs keep the older name for collectors: endpoints live under `/v1/metric/policies` and the SDK client is `client.metricPolicy.*`.
</Note>

***

## Next steps

<CardGroup cols={2}>
  <Card title="Custom Metrics" icon="gauge" href="/documentation/metrics/custom-metrics">
    Deep-dive on LLM Judge, Pattern, and Formula metric types
  </Card>

  <Card title="Collectors" icon="funnel" href="/documentation/metrics/metric-collectors">
    Run your metrics automatically on matching live calls
  </Card>

  <Card title="Thresholds" icon="ruler" href="/documentation/metrics/thresholds">
    Turn scores into pass/fail verdicts
  </Card>

  <Card title="Metric Collection Jobs" icon="list-checks" href="/documentation/metrics/metric-collection-jobs">
    Run evaluations at scale via the API
  </Card>
</CardGroup>