Skip to main content

​
Why this is a common failure point

Speech recognition accuracy varies with voice pitch, and published studies have repeatedly found different error rates for female and male voices. An agent built on that recognition can mishear one group of callers more often without anyone intending it, and the gap is invisible unless you measure it directly.

​
How Roark tests it

Gender handling runs the same flow with a female and a male caller, with everything else held the same, and tells you whether your agent understands and treats them the same. It is a property sweep: every value runs the same flow with everything else held the same, and the report tells you which values did worse than the rest and which differences are just chance.

​
The values

There is no suggested baseline: neither voice is the normal case, so each value is compared with the other.

​
What it measures

The template seeds these, and your flow’s own metrics run alongside them. Only checks can mark a value Worse on; the other measures explain why.

​
What to look for

  • Two values, one question. The sweep asks exactly one thing: does one voice do worse?
  • Run more calls. Two values at 6 calls each is only 12 calls. At 20 per value the sweep can see a 25-point gap for 40 calls, which is cheap for the confidence it buys.
  • Recognition, not treatment. A gap here almost always comes from transcription. Read the transcripts of failing calls before changing prompts.

​
Setting one up

Pick the flow to test, then the values to run it across. Every value starts selected; deselect the ones you don’t need. Each value runs your flow again, so the values you keep are the cost of the run. Compared against sets the baseline: a reference value the report pins first, such as silence for background noise. The template suggests the natural one, and you can change it or choose none. It doesn’t change the verdict: whether a value did worse is always decided against every other value combined. Iterations live in the sweep panel rather than in Advanced. Each value runs every attached test case once per iteration:
Calls per value = test cases x iterations
Each value needs at least 5 calls to be compared with the others, and the Iterations field won’t go below the count that reaches it. It starts at 6 calls per value, so one call excluded after the run doesn’t drop a value out of the comparison. More calls let the report see smaller differences. For a sweep of 10 values:
Below 5 calls a value can never be flagged, however badly it does, which is why the minimum is enforced rather than suggested. Over the API, a sweep template defaults iterationCount to 6 and refuses a plan with fewer than 5 calls per value with a 400.

​
How the results are decided

The sweep’s table leads the run report. Its Compared with the rest column gives each value one result: How “too large to be chance” is decided:
  • Each value is compared with all the other values combined, one check at a time, never with a single other value. Two values that happen to land at opposite ends are not a finding on their own.
  • A sweep makes many comparisons at once (every value on every check), and some will look different by luck alone. The report accounts for that: of all the values it flags in a run, at most 1 in 10 is expected to be a fluke.
  • A value is flagged only when it does worse. Doing better than the rest is not a problem to fix.
Checks passed shows each value’s pass rate with the number of calls behind it, so 100% of two calls never reads like 100% of twenty. A lower rate with No difference beside it means the gap is not larger than chance. Fails, not gender specific. A check that fails on at least 20% of calls with no value standing out is listed under this heading. It is a real problem, just not one gender caused, so changing gender won’t fix it. Select the check to open its evidence. The executive summary follows the same rules: it only blames the property for failures the table marks Worse on. Hover How this works under the table for this run’s own numbers.