Skip to main content
Studio is the metric workbench — build metrics, test them against real calls, and run evaluation batteries, all before anything touches production traffic. Find it in the sidebar under Measure → Studio.
Studio replaces the Playground. If you’re looking for the old Playground docs, everything it did — and more — now lives here.
Studio has three modes:
ModeWhat it’s for
HubThe landing page — start authoring or evaluating, and revisit recent evaluation runs
AuthorDefine or edit a single metric, with a test-call rail and version history
EvaluateRun a battery of metrics across a set of calls and compare results side-by-side

​
The hub

Opening Studio lands you on the hub — “Build metrics. Run evaluations.” — with two start cards:
  • Define a new metric opens Author mode with a blank metric.
  • Run a battery of evals opens Evaluate mode.
Below the cards, Recent evaluation runs lists your past batteries (e.g. “3 metrics × 5 calls”) with a status chip — In progress, All passing, “N failing”, Canceled, or Failed. Reopen any run to see its full result matrix, or fork it into a new evaluation.

​
Author mode

Author mode is the metric editor. You reach it from the hub, from the New metric button on the Metrics page, or by clicking any metric in the Library. The editor walks you through four numbered sections:
1

01 · Engine — what kind of metric is this?

Pick how the metric is computed:
EngineHow it works
LLM JudgeScores each call against your natural-language prompt
PatternComposes triggers, outcomes, and time windows — fires when the pattern matches
FormulaMath over other metrics
The engine is locked after the first save. To measure the same thing with a different engine, create a new metric.
2

02 · Output — what does the metric return?

Choose the result type. Some types need extra configuration:
OutputReturnsExtra config
BooleanYes / NoTrue and False labels
ScaleA score in a rangeMin/Max plus band labels
ClassificationOne of your categoriesCategory list
NumericA numberA unit
CountAn occurrence countA unit
TextFree-form text—
3

03 · Instructions, Logic, or Formula

The third section adapts to your engine. For LLM Judge, write instructions tuned to the output type — for a Boolean metric, the yes/no question the model should answer; for a Scale, what each band means. For Pattern, define the logic: when should this metric fire? For Formula, define the computation over other metrics.
4

04 · Group — where should this metric live?

Assign the metric to a group (analysis package). Groups organize the Library on the Metrics page, so pick one your team will look for it under.

​
Test against real calls

The right-hand rail is your test bench. Use Add test calls to pick the calls Studio should evaluate the metric against, then run them individually or hit Run all. Each run streams through the evaluation pipeline live, so you can watch the result land and tighten your instructions between runs.
Iterate here until the metric returns what you expect on calls you already know the answer for — before you apply it to live traffic.

​
Version history

Every save publishes a new version. The History tab in the rail lists them all — view any past version, or restore one, which publishes a new version from the old one (you never lose the intermediate history).

​
System metrics and variants

System metrics open read-only in Author mode. To change one, click Customize for your team — this forks an org-scoped variant you can freely edit and save. The Roark-managed original stays intact, and your instructions go in an “Additional instructions” field layered on top of the system prompt.

​
Editing a metric that’s already live

If the metric is bound to collectors, a notice appears above the form: “In use by N collectors — saving publishes a new version every collector will pick up.”
Saving a metric republishes it everywhere it runs. Every collector using the metric picks up the new version for calls scored from that point on — test your changes in the rail before saving.

​
Evaluate mode

Evaluate mode runs a battery of metrics across a set of calls and compares results side-by-side. Use it to validate a metric at scale, compare candidate metrics against each other, or spot-check quality across a sample of production calls.
1

Pick metrics

Use Add metrics to choose one or more metrics from your Library. You can attach a pass/fail threshold inline per metric (e.g. “Pass if ≥ 0.7”) to turn raw scores into verdicts — see Thresholds.
2

Pick calls

Use Add calls to select the conversations to evaluate.
3

Run battery

Hit Run battery. Results stream in as a comparison matrix (metrics × calls) or grouped by call — switch views to suit the question you’re asking. You can cancel a run in progress.
Finished batteries appear under Recent evaluation runs on the hub, so you can reopen the matrix later or fork the run into a new evaluation.
Prefer to run evaluations programmatically? Metric collection jobs are the API equivalent — run a set of metrics across a set of calls via the SDK or REST.

​
Apply a metric to live calls

Testing in Studio scores only the calls you pick. To score matching calls automatically as they come in, bind the metric to a collector. The Author mode footer — “Apply this metric to live calls” — has an Add collector button that opens the collector editor with your metric prefilled and the modality locked to what the metric supports. Once bound, the footer shows “Running on N collectors” instead. See Collectors for how conditions and scoping work.
The REST API and SDKs keep the older name for collectors: endpoints live under /v1/metric/policies and the SDK client is client.metricPolicy.*.

​
Next steps

Custom Metrics

Deep-dive on LLM Judge, Pattern, and Formula metric types

Collectors

Run your metrics automatically on matching live calls

Thresholds

Turn scores into pass/fail verdicts

Metric Collection Jobs

Run evaluations at scale via the API