Collectors

Overview

A collector runs a set of metrics on the calls (or chats) it matches. Instead of triggering metric collection by hand, you define the segment of conversations you care about — by agent, source, or property — and Roark scores every matching conversation as it comes in. Collectors live under Measure → Metrics: the Metrics page shows a Collectors strip at the top, and Manage collectors opens the full list at /metrics/collectors. A collector is made of four things:

Name — how it appears in the collectors list
Modality — Call or Chat. A collector fires for exactly one modality, and this is locked after creation
Conditions (optional) — the segment of conversations it matches. Leave empty to match all calls (or all chats)
Metrics — the metrics to run on every match, each with optional pass/fail thresholds

Creating a Collector

Start a new collector

On /metrics/collectors, click New collector. The editor reads as a sentence: “When a call matches…” — click the chip to switch between Call and Chat. Once the collector is created, the modality is locked.

Define the segment

Add conditions to target specific conversations, or leave the conditions empty to match everything. Use Add condition and Add OR condition to build a group, and Add AND condition group to require a second group to match as well.

Pick the metrics

Under Evaluate these metrics, add the metrics to run. Each picked metric runs on every conversation that matches the scope above. Where a metric has multiple variants, choose which variant’s prompt and config apply.

Add thresholds (optional)

On any picked metric, click Add pass/fail threshold to turn its raw value into a pass/fail outcome — for example, “Pass if ≥ 7”.

Create and name it

Click Create collector. Roark suggests a name built from the metrics and conditions — keep it or edit it, then confirm. Incoming conversations that match your conditions are evaluated from then on.

Conditions

Conditions scope a collector to a segment of conversations. Groups combine with AND; conditions within a group combine with OR. An empty condition set matches every call (or chat) in the project.

Type	Matches	Inputs
Agent	Conversations handled by specific agents	Agent picker
Source	Conversations from a specific integration	LiveKit Cloud, Vapi, Retell, ElevenLabs, Leaping, API, Bland
Property	Conversations by custom metadata	Property key + operator + value

Property conditions support these operators:

Operator	Comparison
equals / does not equal	Exact match
contains	Substring match
starts with	Prefix match
is greater than / is less than	Numeric comparison
is at least / is at most	Numeric comparison, inclusive (≥ / ≤)

A group like “Agent is Billing Bot OR Agent is Sales Bot” AND “Source is Vapi” runs the collector’s metrics only on Vapi calls handled by either agent.

Pass/Fail Thresholds

Thresholds turn a metric’s raw value into a pass/fail check, configured inline on each picked metric:

Add pass/fail threshold adds a row that reads “Pass if [operator] [value]” — for example, Pass if ≥ 0.7.
Thresholds that already exist for the metric appear as read-only rows with an Include if checkbox, so you can opt an existing threshold into this collector instead of creating a duplicate. If a new row matches an existing threshold, Roark warns you and dedupes it — it won’t be created again.

Each saved threshold is a derived metric in its own right (named like “Empathy ≥ 7”, scored Pass/Fail), added to the collector alongside the source metric. Thresholds are available for Scale, Numeric, Count, Boolean, and Classification outputs.

Thresholds Guide

Learn about operators and how derived threshold metrics work

Active and Paused

Every custom collector has a status toggle in the list: Active collectors evaluate matching conversations; Paused collectors don’t. Pause a collector instead of deleting it if you might need it again — deleting a collector stops it from collecting its metrics, but the metric definitions themselves are unaffected.

System Collectors

Roark ships system collectors — the baseline coverage that keeps core metrics running on all your calls and chats. They’re marked with a System shield badge and sort first in the list.

	System collectors	Custom collectors
Created by	Roark	You, via dashboard or API
Editable / deletable	No	Yes
Status	Always run	Active or Paused

Per-Agent Collectors

Each agent’s detail page has a Collectors section listing the collectors that target that agent. These are the same collectors — just scoped by an Agent condition — so a collector created from the agent page shows up in the main /metrics/collectors list too, and vice versa.

API and SDK Access

Collectors are fully manageable programmatically via the Node.js SDK and REST API.

The API keeps the older metric policy naming: the SDK client is client.metricPolicy.* and the REST endpoints are /v1/metric/policies. A “policy” in the API is exactly a collector in the dashboard.

Create

const collector = await client.metricPolicy.create({
  name: 'Production Agent Quality Checks',
  modality: 'call', // 'call' or 'chat' — locked after creation
  status: 'ACTIVE',
  conditions: [
    {
      conditions: [
        {
          conditionType: 'AGENT',
          conditionKey: 'your-agent-id',
        },
      ],
    },
  ],
  metrics: [
    { id: 'metric-definition-uuid-1' },
    { id: 'metric-definition-uuid-2' },
  ],
})

Field	Type	Required	Description
`name`	string	Yes	Descriptive name for the collector
`modality`	string	Yes	`call` or `chat`; the collector only fires for this modality and can only reference metrics that support it
`status`	string	No	`ACTIVE` (default) or `INACTIVE` (Paused)
`conditions`	array	No	Condition groups — groups AND together, conditions within a group OR together; omit to match all conversations
`metrics`	array	Yes	Metric definitions to run (minimum 1)

Condition fields

`conditionType`	`conditionKey`	`conditionOperator`	`conditionValue`
`AGENT`	Agent ID	—	—
`CALL_SOURCE`	Source name (e.g. `VAPI`, `RETELL`)	—	—
`CALL_PROPERTY`	Property key	`EQUALS`, `NOT_EQUALS`, `CONTAINS`, `STARTS_WITH`, `GREATER_THAN`, `LESS_THAN`, `GREATER_THAN_OR_EQUALS`, `LESS_THAN_OR_EQUALS`	Value to compare against

GREATER_THAN_OR_EQUALS and LESS_THAN_OR_EQUALS correspond to the dashboard’s “is at least” and “is at most” operators. Example: collector for Vapi calls only

const collector = await client.metricPolicy.create({
  name: 'Vapi Call Metrics',
  modality: 'call',
  conditions: [
    {
      conditions: [
        {
          conditionType: 'CALL_SOURCE',
          conditionKey: 'VAPI',
        },
      ],
    },
  ],
  metrics: [{ id: 'metric-definition-uuid' }],
})

List, get, update, delete

// List (limit 1-50, default 20; cursor pagination via `after`)
const collectors = await client.metricPolicy.list({ status: 'ACTIVE', limit: 50 })

// Get one
const collector = await client.metricPolicy.getByID('policy-id')

// Update — all fields optional; pass an empty `conditions` array to match everything
const updated = await client.metricPolicy.update('policy-id', {
  name: 'Updated Collector Name',
  status: 'INACTIVE',
})

// Delete
const result = await client.metricPolicy.delete('policy-id')

System collectors (type: "SYSTEM" in API responses) cannot be modified or deleted.

Best Practices

Start Broad, Then Narrow

Create a collector with no conditions to run key metrics on all calls, then add targeted collectors with conditions for specific segments.

Use Conditions for Targeted Collection

If you only need certain metrics for specific agents or sources, scope the collector so you’re not scoring conversations you don’t care about.

Keep Collectors Organized

Use names that say what the collector targets and measures — the auto-suggested name (metric names plus a condition summary) is a good starting point.

Pause Instead of Delete

Pause a collector if you might need it again later. Deleting stops collection but leaves the metric definitions intact.

What’s Next

Custom Metrics

Learn about metric types, scopes, and how to create custom metrics

Studio

Author and test metrics, then bind them to a collector from the editor

Thresholds

Turn raw metric values into pass/fail outcomes

Metric Collection Jobs

Run metrics on demand for specific calls

Getting Started

Observability

Metrics

Simulations

Recipes

Integrations

SDKs & Libraries

Resources

Overview

Creating a Collector

Conditions

Pass/Fail Thresholds

Thresholds Guide

Active and Paused

System Collectors

Per-Agent Collectors

API and SDK Access

Create

Condition fields

List, get, update, delete

Best Practices

What’s Next

Custom Metrics

Studio

Thresholds

Metric Collection Jobs

​Overview

​Creating a Collector

​Conditions

​Pass/Fail Thresholds

Thresholds Guide

​Active and Paused

​System Collectors

​Per-Agent Collectors

​API and SDK Access

​Create

​Condition fields

​List, get, update, delete

​Best Practices

​What’s Next

Custom Metrics

Studio

Thresholds

Metric Collection Jobs

Overview

Creating a Collector

Conditions

Pass/Fail Thresholds

Active and Paused

System Collectors

Per-Agent Collectors

API and SDK Access

Create

Condition fields

List, get, update, delete

Best Practices

What’s Next