Code-first agents (LiveKit, Pipecat)

A LiveKit or Pipecat agent is your code on your infrastructure, so there is no provider document for Roark to edit. The managed-config integration turns that around: your agent fetches its tunable surface (prompt, model, params, voice) from Roark at session start, and Autoimprove works on that config with the same loop, safety model, and promote gate as provider agents. The key mechanic is per-session resolution: Roark’s test calls dial your production deployment directly. Because Roark originated those calls, the config endpoint recognizes them and serves the staging revision for those sessions only. Real callers always get production config; you deploy nothing extra.

1. Fetch config at session start

One call registers the key on first use (your defaults become revision 1) and resolves the right revision per session.

import os
from livekit import agents
from livekit.agents import AgentSession
from livekit.plugins import openai
from roark_analytics import Roark

roark = Roark(bearer_token=os.environ["ROARK_API_BEARER_TOKEN"])

DEFAULTS = {
    "system_prompt": "You are a helpful scheduling assistant for Mary's Dental.",
    "model": "gpt-4.1",
    "temperature": 0.4,
    # Externalize your STT and TTS choices too: whatever lives in this
    # document, Roark can tune. Whatever stays in code, it cannot.
    "tts": {"voice_id": "alloy", "speed": 1.0},
    "stt": {"language": "en"},
}

async def entrypoint(ctx: agents.JobContext):
    await ctx.connect()
    participant = await ctx.wait_for_participant()

    # SIP attributes carry the numbers on telephony rooms; adapt to your setup.
    caller = participant.attributes.get("sip.phoneNumber")
    called = participant.attributes.get("sip.trunkPhoneNumber")

    cfg = roark.agent_config.resolve(
        "support-agent",
        defaults=DEFAULTS,
        session={"session_id": ctx.room.name, "caller_number": caller, "called_number": called},
    ).data
    doc = cfg.document

    # Stamp attribution so Roark ties this session to the right revision
    # (and, for its own test calls, to the simulation that placed them).
    await ctx.room.local_participant.set_attributes(
        {"roark.revisionId": cfg.revision_id, "roark.channel": cfg.channel}
    )

    session = AgentSession(
        llm=openai.LLM(model=doc["model"], temperature=doc["temperature"]),
    )
    await session.start(agent=agents.Agent(instructions=doc["system_prompt"]), room=ctx.room)

import os
from pipecat.services.openai.llm import OpenAILLMService
from roark_analytics import Roark

roark = Roark(bearer_token=os.environ["ROARK_API_BEARER_TOKEN"])

DEFAULTS = {
    "system_prompt": "You are a helpful scheduling assistant for Mary's Dental.",
    "model": "gpt-4.1",
    "temperature": 0.4,
    # Externalize your STT and TTS choices too: whatever lives in this
    # document, Roark can tune. Whatever stays in code, it cannot.
    "tts": {"voice_id": "alloy", "speed": 1.0},
    "stt": {"language": "en"},
}

async def run_bot(transport, call_info):
    # call_info: however your transport surfaces the dial-in numbers.
    cfg = roark.agent_config.resolve(
        "support-agent",
        defaults=DEFAULTS,
        session={
            "session_id": call_info.session_id,
            "caller_number": call_info.caller_number,
            "called_number": call_info.called_number,
        },
    ).data
    doc = cfg.document

    llm = OpenAILLMService(model=doc["model"], params={"temperature": doc["temperature"]})
    messages = [{"role": "system", "content": doc["system_prompt"]}]
    # ... build your pipeline as usual, and include cfg.revision_id and
    # cfg.channel in the call properties you report to Roark.

Cache the resolved config for the session; do not fetch per turn. Wrap the resolve in a try/except that falls back to your DEFAULTS: the failure mode is then your current behavior, never a broken call. The SDK method ships in the next roark_analytics release; until you upgrade, the same call is one POST /v1/agent-config/{key}/resolve request.

2. Link the config to your Roark agent

Autoimprove finds the config through the agent your calls already report. Link once:

roark agent-config update support-agent \
  --channel production \
  --document @config.json \
  --agent-id "$AGENT_ID"

curl -X PUT https://api.roark.ai/v1/agent-config/support-agent \
  -H "Authorization: Bearer $ROARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"channel": "production", "document": {"system_prompt": "...", "model": "gpt-4.1", "temperature": 0.4}, "agentId": "'"$AGENT_ID"'"}'

Your agent also needs a registered phone endpoint (the number Roark’s simulations dial): roark agent endpoint create if you have not already.

3. Run Autoimprove

Exactly the same as any other agent:

roark autoimprove job create \
  --agent-id "$AGENT_ID" \
  --objective-metric-definition-id "$METRIC_ID" \
  --objective-label "Booking completion should pass" \
  --target-value 90

What happens under the hood differs only in the staging story: instead of cloning a shadow, Roark forks a staging config channel from production. Its test calls dial your production number, get recognized, and read the staging revision; every change Roark makes is a new audited revision on that channel. Real traffic never sees any of it. When the job reaches AWAITING_PROMOTE, promote is a pointer swap: your very next session reads the verified revision. Rolling back is writing the prior revision back to production; the chain preserves every state.

roark autoimprove job promote <job-id>

4. Guard your tools (optional)

Roark’s test calls dial your production deployment, which means your real tools would run: real bookings, real SMS, real transfers. The tool guard fixes that. Wrap a side-effecting tool once; during a Roark test call it never executes. Instead, Roark answers with a simulated backend response that is shaped by the tool’s contract, consistent with the test scenario (a scenario about a fully booked calendar gets a realistic “no availability” from a booking tool), and consistent with earlier tool responses in the same call. You configure nothing per tool. Real callers always hit the real tool: the guard only diverts when the resolve response identified the session as a Roark simulation, and the endpoint independently re-validates the simulation before answering.

import httpx

def roark_tool_guard(cfg, fallback=None):
    """Wrap a side-effecting tool. During a Roark test call the tool never
    executes; Roark answers with a simulated backend response instead."""
    def decorator(tool_fn):
        async def wrapper(**kwargs):
            if cfg.simulation_job_id is None:
                return await tool_fn(**kwargs)  # real caller: real tool
            try:
                response = httpx.post(
                    "https://api.roark.ai/v1/simulation/tool-mock",
                    headers={"Authorization": f"Bearer {os.environ['ROARK_API_BEARER_TOKEN']}"},
                    json={
                        "simulationJobId": cfg.simulation_job_id,
                        "toolName": tool_fn.__name__,
                        "toolDescription": tool_fn.__doc__,
                        "arguments": kwargs,
                    },
                    timeout=5.0,
                )
                if response.status_code == 404:
                    # Not a known simulation after all: treat the session as real.
                    return await tool_fn(**kwargs)
                response.raise_for_status()
                return response.json()["data"]["result"]
            except httpx.HTTPError:
                # Never run the real tool for a session resolve called a test:
                # degrade to a static stub instead.
                return fallback if fallback is not None else {"status": "ok", "mocked": True}
        wrapper.__name__ = tool_fn.__name__
        wrapper.__doc__ = tool_fn.__doc__
        return wrapper
    return decorator

# After resolving cfg at session start:
@roark_tool_guard(cfg, fallback={"status": "booked", "confirmationId": "TEST-0000"})
async def book_appointment(date: str, time: str) -> dict:
    """Books an appointment. Args: date (YYYY-MM-DD), time (HH:MM). Returns {confirmationId, status}."""
    ...  # your real calendar write: never runs during a Roark test call

Wrap the writes (book, cancel, send_sms, transfer, charge); leave read-only lookups unwrapped when live data makes the test more realistic. Pass the richest toolDescription you can: the contract is what shapes the simulated response. In LiveKit Agents, wrap the function before registering it with @function_tool; in Pipecat, wrap the handler before registering it on the LLM service. Identical retries within a few minutes return the stored response, so double-fired handlers stay consistent, and every mocked invocation is recorded against the simulation as evidence of what your agent tried to call. The failure contract, precisely: 404 means the simulation id is unknown (treat the session as real), 409 means the simulation already ended (do NOT run the real tool; use your fallback), and any other error also means use your fallback. The roark_analytics SDK gains a first-class simulation.mock_tool(...) method in its next release; the decorator above works today. The guard is a simulations-wide feature, not an Autoimprove one: every Roark test call against your deployment gets safe tools, you can pin deterministic responses per tool with fixtures, and you can read back exactly what your agent called after each run. The full workflow, including CI assertions and automatic grading, is in the Tool Call Testing recipe.

What Roark can and cannot improve here

Roark can change anything in the document you externalized: prompt, model and sampling params, and your STT/TTS choices (voice, speed, language) when you include them, as the example does. It keeps the key shape your code reads (it never renames keys). What stays in code (tool logic, pipeline structure) is out of reach by design; the job’s final report recommends those changes for a human when the config surface is not enough. Guarded tools are still out of reach for changes, but their invocations become testable: Roark sees what your agent tried to call with what arguments, with no real side effects.

Getting Started

Simulations

Observability

Metrics

Config as Code

Autoimprove

Recipes

Integrations

Enterprise

SDKs & Developer Tools

Resources

Code-first agents (LiveKit, Pipecat)

1. Fetch config at session start

2. Link the config to your Roark agent

3. Run Autoimprove

4. Guard your tools (optional)

What Roark can and cannot improve here

​1. Fetch config at session start

​2. Link the config to your Roark agent

​3. Run Autoimprove

​4. Guard your tools (optional)

​What Roark can and cannot improve here

1. Fetch config at session start

2. Link the config to your Roark agent

3. Run Autoimprove

4. Guard your tools (optional)

What Roark can and cannot improve here