Config declares the plan. It does not start it. Applying reconciles the definition; the run is a separate, deliberate act:That split is what lets an alert name a plan: the plan exists as a durable resource rather than as a side effect of having run something.
Knowledge checks
Testing that your agent answers a set of questions correctly is common enough that it has its own template. Instead of authoring a flow per question, settemplate: question-answer-check and write the questions and expected answers as a list. Roark asks each one in its own call and grades the answer.
The same template is also a tile in the dashboard’s New simulation run picker and can be run over the API (see From the dashboard or API below) — config, dashboard and API all drive the one template.
A knowledge check needs a
persona (who asks) and an environment (which channel), the same two things every simulated call needs. Reference a kind: persona in your bundle by name, and an environment by its display name.Grading against a knowledge base
SetgradeWithKnowledgeBase: true to grade answers against your project’s knowledge base as well. A question may then omit expect, and its answer is checked for grounding instead, so your knowledge base stays the single source of truth for the ones you don’t want to hand-write.
expect, or turn on gradeWithKnowledgeBase. A question with neither is refused.
A knowledge check generates a flow behind the scenes (one variant per question). It appears in your flow library named after the plan. Edit the plan, not the generated flow: the next apply rewrites it from the questions.
metrics list: Agent Expectations grades each expect, and Knowledge Base Grounding is added when gradeWithKnowledgeBase is on. Anything you add under metrics is collected on top.
From the dashboard or API
The samequestion-answer-check template is available two other ways, driving the identical run:
- Dashboard — New simulation run → Question & Answer Check. Pick the persona and environment, add your question/expected-answer rows, and run. There is no flow picker: the flow is generated from the questions.
- API —
POST /v1/simulation/runwith the template and your questions:
personaId, environmentId and questions are required for this template; flows is rejected (the flow is generated). The response is a run plan, the same shape every POST /v1/simulation/run returns.
Your flows and Roark’s
This and the sections below describe the default (
template: manual) plan, where you attach flows you authored. A knowledge check generates its flow for you, so it skips them.flow or system. They resolve through different paths, so a row naming both is refused.
roark simulation template list prints the slug of every curated flow and outcome.
happyPath and edgeCases are alternatives, not a union: asking for both is refused, because omitting both is already how you say “all of it”.
Which line to call
value is the endpoint’s own identifier, spelled the way kind: agent spells it. Omit it when the agent has exactly one endpoint; apply refuses to guess between several rather than dial a different number than you meant.
Everything else has a default
Onlyname, direction, agentEndpoints, metrics, flows and maxDurationSeconds are required. The rest default to what the dashboard would give you:
Success criteria are not expressible in config yet.
metrics takes slugs, so every pass/fail check on a config-managed plan is held to the 80% default, and a managed plan rejects edits from the API or the dashboard. If a plan needs a check held to a minimum of its own, keep that plan out of config for now and set its minimums under Advanced → Success criteria. See Success criteria.endCallReasons is worth knowing about if you run scripted flows. A simulated caller hangs up on an end-call phrase (something the agent says) or on one of these conditions. Running out of scripted steps ends nothing, so a flow scripted to a verdict needs a matching reason or the call stays on the line until maxDurationSeconds.
Alerting on the result
An alert scopes itself to a plan by name, which is why the plan’sname is also its stored name:
Apply order
Plans reference agents, flows and metrics, and alerts reference plans, so one apply resolves them in that order regardless of how your files are laid out:For the full field reference of every kind, see the Config DSL reference schema.