Skip to main content

​
Why this is a common failure point

Voice agents drift. A prompt that handles the happy path in a demo starts skipping verification steps, reordering questions, or promising things the process doesn’t allow once real callers push it off script. Nobody notices, because each call sounds reasonable on its own: the failure only shows when you compare what the agent did with what it was supposed to do. Prompt changes make it worse. A tweak that fixes one conversation quietly changes behaviour in others, and without a written expectation there is nothing to catch the regression.

​
How Roark tests it

You write down what should happen as customer flows: the steps a conversation should take and what the agent must do along the way. Flow adherence runs a simulated caller through each flow and grades every call against it: did the conversation follow the steps, did the agent follow its instructions, and did it hold every expectation you declared. Because the flows are fixed, the same run repeated after a prompt change tells you exactly what moved.

​
Setting it up

Attach the flows the agent should follow. Each flow runs its selected variants (its happy path and any edge cases you pick), once per iteration, per agent.

​
What it measures

The template seeds these, and you can add or remove metrics in Advanced before running.

​
What to look for

  • Expectations are the sharpest signal. Adherence says the conversation drifted; a failed expectation says exactly which rule the agent broke.
  • Edge cases, not just the happy path. Agents rarely fail the path they were built around. Attach the variants where the caller changes their mind or gives something unexpected.
  • Rerun after every prompt change. Keep the run as a plan so the comparison is like for like.

​
Over the API

Run it with POST /v1/simulation/run and "template": "flow-adherence". GET /v1/simulation/template lists what each template accepts.