Build a modular voice agent with Pipecat

build a modular voice agent with pipecat on exotel agentstream wire exotel phone audio to a pipecat pipeline using the official exotelframeserializer https //docs pipecat ai/api reference/server/services/serializers/exotel default stack deepgram stt → openai llm → cartesia tts at 8 khz pcm swap any stage in bot py without rewriting the exotel websocket glue sample code integrations/pipecat https //github com/exotel/agent stream/tree/main/integrations/pipecat in the agent stream https //github com/exotel/agent stream repo if you have not set up connect yet, start with connect voice ai with agentstream docid xb3tdmzuz6jbcqdsboqd when to use pipecat (vs speech to speech) pipecat cascade openai realtime / elevenlabs shape stt → llm → tts (you own each hop) native speech to speech best for swappable vendors, custom tools, cost control per stage lowest conversational latency and barge in feel expect turn detection + llm + tts before reply audio model speaks as it “thinks” use this guide when you want a composable agentstream bot for the most natural phone talk, prefer the openai realtime https //docs exotel com/exotel agentstream/openai realtime agentstream integration or elevenlabs https //docs exotel com/exotel agentstream/elevenlabss elevenagents agentstream integration guides what you get fastapi websocket host that speaks exotel agentstream ( /ws ) pipecat exotelframeserializer (16 bit linear pcm, typically 8 khz — not μ law) boot time cartesia greeting cache (same voice id as live tts) for near instant first media tuned turn taking silero vad + local smart turn (sub second silence stop) official pipecat exotel notes exotel websockets https //docs pipecat ai/pipecat/telephony/exotel websockets before you start item notes openai openai api key (llm replies) deepgram deepgram api key (stt) cartesia cartesia api key (+ optional cartesia voice id ) exotel account sid, api key, api token, exophone python 3 10+ virtualenv recommended public wss cloudflared or ngrok for the first test install git clone https //github com/exotel/agent stream git cd agent stream cp shared/env exotel example shared/ env exotel \# fill exotel values cd integrations/pipecat python3 m venv venv && source venv/bin/activate pip install r requirements txt cp env example env edit env (never commit this file) openai api key=sk openai model=gpt 4o mini deepgram api key= cartesia api key=sk car cartesia voice id=71a7ad14 091c 4e8e a314 022ece01c121 cartesia model=sonic 3 5 pipecat greeting=hi, how can i help? smart turn stop secs=0 8 server host=0 0 0 0 server port=8765 keep cartesia voice id / cartesia model / 8 khz pcm the same for the greeting cache and live tts so the agent voice stays consistent run the bot and open a tunnel terminal a cd integrations/pipecat source venv/bin/activate python server py on startup you should see cartesia tts websocket warmed greeting cache ready … pipecat exotel bridge ready — wss /ws uvicorn on 0 0 0 0 8765 websocket path /ws terminal b cloudflared tunnel url http //127 0 0 1 8765 place a connect call from the repo root set a && source shared/ env exotel && set +a python shared/place connect call py \\ \ to +91xxxxxxxxxx \\ \ stream url "wss\ //your cloudflare host/ws?sample rate=8000" streamurl wss\ //host/ws?sample rate=8000 port 8765 the ?sample rate=8000 query parameter is required for pstn answer the phone you should hear the cached greeting quickly, then speak a short turn and get an llm + cartesia reply architecture (what runs on each call) caller ←→ exotel agentstream (8 khz pcm) ↕ wss\ //…/ws fastapi + exotelframeserializer ↕ pipecat pipeline deepgram stt → user aggregator (vad + smart turn) → openai llm → cartesia tts → exotel media out piece role server py accept exotel wss, build transport + serializer, run one bot session bot py pipeline, vad / smart turn, llm, cartesia tts greeting cache py boot synthesize + silence trim; push pcm on connect voice config py shared voice / rate / encoding for cache and live tts pipecat does not auto hang up via exotel rest in this sample — ending the websocket ends the media bridge see the serializer docs https //docs pipecat ai/api reference/server/services/serializers/exotel verify boot greeting cache ready and voice=\<same cartesia voice id> connect cached greeting … queue ms≈0 after you speak smart turn completes in under 1s of silence (not 3s) reply audio sounds natural (sentence level cartesia), not stretched word by word pitch matches a normal phone call (8 khz end to end, no μ law mix up) latency expectations (honest) stage typical notes first greeting near 0 ms after pipeline ready pre cached cartesia pcm end of user turn 0 8s silence (configurable) smart turn stop secs (pipecat default is 3 0 — too slow for phones) llm ttfb 0 5–2s use gpt 4o mini (or similar) for voice cartesia ttfa 0 2s plus small leading silence from the model this is still a cascade speech to speech providers will feel snappier for free form chat go live checklist bot on a stable host with valid tls for wss\ // api keys only in secrets — not in git streamurl is wss\ //host/ws?sample rate=8000 greeting text / voice id match your brand smart turn stop secs tuned on real handsets (too low cuts users off mid thought) cartesia uses sentence aggregation for natural speech (do not force token flush with max buffer delay ms=0 unless you accept choppy audio) process supervision and clean disconnect handling capacity test one process ≠ unlimited concurrent calls troubleshoot symptom what to check no greeting boot cache failed — cartesia api key , network; look for greeting warm failed silence after connect tunnel url, path /ws , ?sample rate=8000 3s dead air before every reply smart turn still at default 3s — set smart turn stop secs=0 8 “hello……how……can……” stretched speech token level tts flush — use sentence aggregation (sample default) deep / slow voice sample rate mismatch — keep pipeline and cartesia at 8000 pcm hangup in a few seconds wrong streamurl or tunnel died slow llm switch openai model to a low latency chat model; shorten system prompt swap stt / llm / tts edit integrations/pipecat/bot py keep audio in sample rate / audio out sample rate = 8000 transport + exotelframeserializer unchanged in server py greeting cache voice settings in sync if you change cartesia (or re point cache at your new tts) upstream patterns pipecat examples/exotel chatbot https //github com/pipecat ai/pipecat examples/tree/main/exotel chatbot related repo readme integrations/pipecat/readme md pipecat exotelframeserializer https //docs pipecat ai/api reference/server/services/serializers/exotel connect voice ai api https //docs exotel com/exotel agentstream/connect voice ai api openai realtime https //docs exotel com/exotel agentstream/openai realtime agentstream integration · elevenlabs https //docs exotel com/exotel agentstream/elevenlabss elevenagents agentstream integration · sarvam https //docs exotel com/exotel agentstream/sarvam ai agentstream integration