> ## Documentation Index > Fetch the complete documentation index at: https://docs.roark.ai/llms.txt > Use this file to discover all available pages before exploring further. # Traces > Send OpenTelemetry traces to Roark for full visibility into your Voice AI agent ## Overview Roark supports **OpenTelemetry (OTel) tracing**. Send your OTel trace data to Roark to see what happens under the hood of your Voice AI agent, debug latency, inspect tool usage and external API calls, and correlate call execution with your backend traces. ### Features * **LLM traces per call** — View LLM spans, tool calls, and model invocations directly on the call detail page. Each call’s **Tracing** tab shows the full trace tree for that conversation, so you can quickly pinpoint where latency or failures occurred. * **Central trace explorer** — See all traces in one place. Filter by time range, custom tags, or search by span name and attributes. Use this to spot patterns across calls, compare runs, and troubleshoot recurring issues. *** Roark Traces view showing agent turns with STT, LLM, and TTS spans *** ## Endpoint and Protocol | Requirement | Details | | ------------------- | ------------------------------------------------------------------------------------- | | **Protocol** | OTLP over HTTPS only | | **Endpoint** | Send traces to Roark's OTel endpoint (see examples below) | | **OTel traces URL** | `https://api.roark.ai/v1/traces` | | **Authorization** | **Required.** Send `Authorization: Bearer YOUR_ROARK_API_KEY` in the request headers. | | **Role** | Roark acts as an OTel Collector | All trace ingestion requests require authentication. [Generate an API key](/documentation/getting-started/api-keys) in your Roark dashboard and use it in the `Authorization: Bearer YOUR_ROARK_API_KEY` header. *** ## Setup Guide Generate an API key in your Roark dashboard. Trace ingestion requires authentication via the `Authorization: Bearer YOUR_ROARK_API_KEY` header. [Create an API key →](/documentation/getting-started/api-keys) Pick the platform you're using and follow the corresponding setup section below: * [LiveKit](#livekit) — instrument your LiveKit agent with OpenTelemetry * [Pipecat](#pipecat) — enable Pipecat's built-in OTel and export to Roark * [VAPI](#vapi) — traces sync automatically when calls are ingested * [Custom integration](#custom-integration) — any other platform via OTLP HTTP *** ## LiveKit **Instrument your LiveKit agent** with OpenTelemetry and export traces to Roark. Configure your tracer with the `livekit.room.id` resource attribute — this is how Roark links your LiveKit room to its traces and shows them on the call detail page. See the [LiveKit integration guide](/documentation/integrations/livekit) for the full webhook setup. Install the required packages: ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}} pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http livekit-agents ``` Example: a simple [LiveKit Agent](https://docs.livekit.io/agents/) entrypoint with OTel exporting to Roark: ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}} import os from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from livekit.agents import AgentServer, JobContext, cli from livekit.agents.telemetry import set_tracer_provider def setup_roark_tracer(ctx: JobContext): """Configure OTel to export traces to Roark with livekit.room.id for call correlation.""" resource = Resource.create({ "livekit.room.id": ctx.job.room.sid, # Required: Roark uses this to link traces to the call "roark.skip": False, # Optional: set to True to filter out these traces }) provider = TracerProvider(resource=resource) provider.add_span_processor( BatchSpanProcessor( OTLPSpanExporter( endpoint="https://api.roark.ai/v1/traces", headers={"Authorization": f"Bearer {os.environ['ROARK_API_KEY']}"}, ) ) ) set_tracer_provider(provider) # Ensure all pending traces are exported when the agent shuts down async def flush_traces(): provider.force_flush() ctx.add_shutdown_callback(flush_traces) server = AgentServer() @server.rtc_session(agent_name="my-agent") async def entrypoint(ctx: JobContext): # Initialize tracing early, before starting the agent session setup_roark_tracer(ctx) # Your agent logic here pass if __name__ == "__main__": cli.run_app(server) ``` **Resource attributes:** | Attribute | Required | Description | | :---------------- | :------- | :---------------------------------------------------------------------------------------------------------- | | `livekit.room.id` | Yes | The LiveKit room SID (`ctx.job.room.sid`). Roark uses this to link traces to the corresponding call. | | `roark.skip` | No | Set to `true` to tell Roark to filter out traces for this room. Useful for skipping test or internal calls. | If you're also using the [LiveKit webhook integration](/documentation/integrations/livekit), you can set `roark.skip` in both room metadata and OTel resource attributes to skip both call processing and trace ingestion. *** ## Pipecat **Enable Pipecat's built-in OpenTelemetry** and export spans to Roark. Configure the tracer provider before constructing `PipelineTask`, pass `enable_tracing=True`, and use the same call ID for the observer's `pipecat_call_id` and the task's `conversation_id` so Roark can link each call to its trace. See the [Pipecat integration guide](/documentation/integrations/pipecat) for the observer setup. Install the required packages: ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}} pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http ``` Configure the tracer provider, then wire `enable_tracing` + matching IDs into the pipeline: ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}} import os import uuid from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat_roark import RoarkObserver def setup_roark_tracing() -> None: """Must run BEFORE constructing PipelineTask — Pipecat grabs the tracer provider at task-init time.""" resource = Resource.create({ "deployment.environment": os.getenv("ENVIRONMENT", "production"), # Optional: tag spans for the trace explorer. "roark.project.tag.env": os.getenv("ENVIRONMENT", "production"), }) provider = TracerProvider(resource=resource) provider.add_span_processor( BatchSpanProcessor( OTLPSpanExporter( endpoint="https://api.roark.ai/v1/traces", headers={"Authorization": f"Bearer {os.environ['ROARK_API_KEY']}"}, timeout=30, ) ) ) trace.set_tracer_provider(provider) setup_roark_tracing() # Generate ONE id and pass it to both sides so Roark can correlate. call_id = str(uuid.uuid4()) roark = RoarkObserver( api_key=os.environ["ROARK_API_KEY"], agent_id="support-bot-v1", pipecat_call_id=call_id, # appears on the Roark record as pipecatCallId ) task = PipelineTask( pipeline, params=PipelineParams(observers=[roark]), enable_tracing=True, conversation_id=call_id, # set as the conversation.id span attribute ) ``` **Resource and span attributes:** | Attribute | Required | Description | | :------------------------ | :------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `conversation.id` | Yes | Set automatically by Pipecat from `PipelineTask(conversation_id=...)`. Must match the observer's `pipecat_call_id` so Roark can link the trace to the call. | | `roark.project.tag.{key}` | No | Custom project tags for filtering and grouping in the trace explorer. | If you omit `conversation_id` / `pipecat_call_id`, Pipecat still emits traces and the observer still records the call — but Roark won't be able to associate the two, so the **Tracing** tab on the call detail page will be empty. *** ## VAPI **Traces sync automatically.** Whenever a call from a selected agent is synced to Roark, its OpenTelemetry traces are synced too. No extra OTLP exporter or instrumentation is needed on your side. See the [VAPI integration guide](/documentation/integrations/vapi) for step-by-step setup. Make sure **Public Logs** are enabled in your Vapi dashboard. Roark requires public log access to ingest trace data from Vapi calls. *** ## Custom Integration For any other platform, use the OTLP HTTP trace exporter and point it to Roark's OTel endpoint. Include the **required** `Authorization: Bearer YOUR_ROARK_API_KEY` header and attach the required resource attributes. ### Correlating traces to a call or chat If you submit calls or chats to Roark via the [Customer API](/api-reference/introduction), tag your traces with `roark.external_id` so Roark can link each conversation to its trace automatically. 1. When creating a call or chat via `POST /call` or `POST /chat`, supply the `externalId` field with a stable identifier from your own system (session ID, conversation ID, etc.). It must be unique within the project. 2. On your OpenTelemetry traces, set `roark.external_id` to the **same value** — either as a resource attribute (propagates to every span in the service) or as a span attribute on the root span. Roark looks up the matching trace in ClickHouse after the call/chat is created and backfills the trace ID, so the conversation appears in its **Tracing** tab automatically. ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}} npm install @opentelemetry/sdk-trace-node @opentelemetry/exporter-trace-otlp-http @opentelemetry/resources ``` ```typescript theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}} import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node"; import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base"; import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; import { Resource } from "@opentelemetry/resources"; const exporter = new OTLPTraceExporter({ url: "https://api.roark.ai/v1/traces", headers: { Authorization: `Bearer ${process.env.ROARK_API_KEY}` }, }); // `roark.external_id` is the correlation key linking this trace to the // call/chat you create with the same `externalId` via the Customer API. const resource = new Resource({ "roark.external_id": yourSessionId, "roark.project.tag.env": process.env.ROARK_ENV ?? "production", }); const provider = new NodeTracerProvider({ resource }); provider.addSpanProcessor(new BatchSpanProcessor(exporter)); provider.register(); ``` ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}} go get go.opentelemetry.io/otel \ go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \ go.opentelemetry.io/otel/sdk/trace ``` ```go theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}} package main import ( "context" "os" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" "go.opentelemetry.io/otel/sdk/resource" sdktrace "go.opentelemetry.io/otel/sdk/trace" semconv "go.opentelemetry.io/otel/semconv/v1.24.0" ) func initTracer(sessionID string) (func(context.Context) error, error) { exporter, err := otlptracehttp.New(context.Background(), otlptracehttp.WithEndpoint("api.roark.ai"), otlptracehttp.WithURLPath("/v1/traces"), otlptracehttp.WithHeaders(map[string]string{ "Authorization": "Bearer " + os.Getenv("ROARK_API_KEY"), }), ) if err != nil { return nil, err } // `roark.external_id` is the correlation key linking this trace to the // call/chat you create with the same `externalId` via the Customer API. res, err := resource.New(context.Background(), resource.WithAttributes( semconv.ServiceName("my-voice-agent"), attribute.String("roark.external_id", sessionID), attribute.String("roark.project.tag.env", "production"), ), ) if err != nil { return nil, err } tp := sdktrace.NewTracerProvider( sdktrace.WithBatcher(exporter), sdktrace.WithResource(res), ) otel.SetTracerProvider(tp) return tp.Shutdown, nil } ``` ```bash theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}} pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http ``` ```python theme={"theme":{"light":"everforest-light","dark":"everforest-dark"}} from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor import os exporter = OTLPSpanExporter( endpoint="https://api.roark.ai/v1/traces", headers={"Authorization": f"Bearer {os.getenv('ROARK_API_KEY')}"}, ) # `roark.external_id` is the correlation key linking this trace to the # call/chat you create with the same `externalId` via the Customer API. resource = Resource(attributes={ "roark.external_id": your_session_id, "roark.project.tag.env": os.getenv("ROARK_ENV", "production"), }) provider = TracerProvider(resource=resource) provider.add_span_processor(BatchSpanProcessor(exporter)) trace.set_tracer_provider(provider) ``` **Resource attributes:** | Attribute | Required | Description | | :------------------------ | :---------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `roark.external_id` | Recommended | A stable identifier from your own system (e.g. session/conversation ID). Must match the `externalId` field you submit on the corresponding call or chat via the Customer API, and must be unique within a project. | | `roark.project.tag.{key}` | No | Custom project tags for filtering and grouping in the trace explorer. | *** Once your integration is set up, you're ready to send traces to Roark. You'll be able to view them on the Traces page and directly within each call's detail page. *** ## What's Next View and debug active calls Define and analyze call metrics Build observability views Explore Roark API endpoints *** **Related:** [LiveKit integration](/documentation/integrations/livekit) · [VAPI integration](/documentation/integrations/vapi) · [Integrations overview](/documentation/integrations/overview)