Showcase: calling-ai-agent — An Open-Source AI Voice Calling Agent (Twilio, Pipecat, Deepgram, ElevenLabs)

Open-source showcase · github.com/Shivay00001/calling-ai-agent

TL;DR: calling-ai-agent is a full-stack AI voice calling agent that places real outbound phone calls through Twilio, holds a live spoken conversation using Deepgram speech-to-text, an LLM brain, and ElevenLabs text-to-speech wired together with Pipecat's voice pipeline, and is driven from a Next.js settings console. I read the backend line by line before writing this — it's a genuine working prototype, not a README-only repo. Status: working demo — it needs real API credentials (Twilio, Deepgram, ElevenLabs, an LLM provider) and a public URL for Twilio to reach it, and there are a few rough edges to fix before production.

1. What it is

The repository contains a complete AI voice-calling stack in two parts: a FastAPI backend (backend/bot.py, ~6.3 KB) that runs the real-time voice pipeline, and a Next.js frontend (frontend/src/app/page.tsx, ~7.7 KB) that acts as the operator console — you paste in your API keys, pick an LLM provider, type a phone number, and trigger a call. A Dockerfile ships the backend, and backend/.env.example documents the six credentials the system needs.

In plain terms: this is the skeleton of the kind of AI receptionist / AI sales-caller system that businesses paid consultants six figures to build in 2024–2025, as a self-hostable codebase you can read in an afternoon.

2. The problem it solves

Missed calls are the single most expensive leak in most local businesses — estimates put the small-business missed-call rate at 60–80%, and the majority of callers never try a second time. Human callers are great but don't scale: they cover business hours, cost real salaries, and churn. On the outbound side, sales development teams lose hours to dialing, voicemail, and gatekeepers.

A voice AI agent attacks both sides at once: it can answer inbound calls 24/7 (qualify, answer FAQs, book) and run outbound campaigns (reminders, re-engagement, appointment setting) without adding headcount. The hard part has always been the plumbing — streaming audio between a telephony provider, a transcription engine, an LLM, and a voice synthesizer with low enough latency that the conversation feels natural. This repo wires exactly that plumbing together using battle-tested components: Twilio for telephony, Pipecat for the pipeline, Deepgram for speech-to-text, ElevenLabs for the voice.

3. How it works — the call path, end to end

Reading bot.py from top to bottom, a single outbound call flows like this:

1. Trigger. POST /api/outbound-call accepts {"phone_number": "..."}. It builds a Twilio client from credentials (loaded by a get_api_key helper that checks a local settings.json first, then environment variables) and calls twilio_client.calls.create(to=..., from_=..., url=f"{base_url}/twiml"), returning the Twilio call SID.
2. Handoff to the voice stream. Twilio fetches the TwiML endpoint (POST /twiml), which returns a <Connect><Stream> document pointing at the backend's /ws websocket — the BASE_URL env var is translated from http(s):// to ws(s)://. This is why .env.example documents a public URL (e.g. an ngrok tunnel): Twilio's cloud must be able to reach your server.
3. The live pipeline. The /ws websocket handler builds a Pipecat pipeline:

   pipeline = Pipeline([
       transport.input(),
       stt,
       llm,
       tts,
       transport.output(),
   ])

   The transport is a FastAPIWebsocketTransport with VAD (voice activity detection) enabled and TwilioFrameSerializer handling Twilio's media frames. Audio flows: Twilio → Deepgram STT → LLM → ElevenLabs TTS → back to Twilio, with allow_interruptions=True so the caller can cut the agent off mid-sentence — the single most important detail for a call that feels human rather than robotic.
4. The brain. The LLM is swappable per call via a provider prefix: gpt-* routes to OpenAI, gemini/… routes through OpenAI-compatible Gemini endpoints, zhipu/… to Zhipu AI's GLM models, and claude/… to Anthropic's service (with a graceful fallback to GPT-4o if the Anthropic package isn't installed). The system prompt is deliberately short — "brief, polite, and conversational" — and on connection the agent queues a "Please greet the user." instruction so it opens the call.

4. The operator console

The Next.js frontend is a single-page console with two jobs: credential management and call triggering. It persists all keys in localStorage (keys named calling_openai_key, calling_twilio_sid, etc.), lets you switch the LLM provider from a dropdown, and on save POSTs everything to /api/settings/keys, which writes non-empty values into the backend's settings.json. The call panel takes a phone number and POSTs to /api/outbound-call, surfacing calling/success/error states. Nothing about the UI is fancy — but it's the right shape for an operator tool: keys, provider, number, call.

5b. A closer look at the settings layer

One design choice deserves more attention because it shapes how the system is operated: the get_api_key(key_name, env_fallback) helper. Every credential lookup first tries a local settings.json file, and only falls back to environment variables when the file is absent. The console's /api/settings/keys endpoint writes whatever non-empty values you paste into the UI into that file. This is a deliberate operator-experience tradeoff: instead of redeploying or editing env files every time you rotate a key or switch LLM providers, you paste the new key into the web UI and the next call uses it.

The tradeoff, stated plainly, is security: settings.json holds plaintext secrets on disk next to the application code. For a single-operator demo on a private machine that's acceptable; for anything shared, the file should be replaced with a secrets manager or at minimum restricted file permissions plus an authenticated settings endpoint. The shape of the solution is right — credentials as runtime configuration, switchable without redeploy — it just needs the storage backend upgraded before production.

Related: the frontend stores its copy of the keys in localStorage, which means anyone with access to the operator's browser profile can read them. Again, fine for a personal console, wrong for a multi-operator deployment. A small improvement with outsized impact: mark the settings endpoint to require auth and stop persisting raw keys in the browser beyond the session.

6. How to run it

The stack needs five real credentials: Twilio (account SID, auth token, a voice-enabled number), an LLM key, Deepgram, ElevenLabs, plus a public URL for Twilio webhooks. Roughly:

# Backend
cd backend
python -m venv venv && venv/Scripts/activate   # or source venv/bin/activate
pip install -r requirements.txt
# copy .env.example → .env, fill in keys, set BASE_URL to your ngrok URL
uvicorn bot:app --host 0.0.0.0 --port 8000

# Frontend
cd frontend
npm install
npm run dev

Then open the console, paste keys, enter a test number, and place the call. Dependencies are pinned sanely: fastapi, twilio, pipecat-ai[openai,elevenlabs,deepgram], python-dotenv.

6. Honest status: what works, what's rough

Working. The full outbound path — trigger → Twilio → TwiML stream → Pipecat pipeline with real STT/LLM/TTS — is genuinely implemented, not stubbed. Multi-provider LLM routing, the settings persistence layer, and the console UI are all real code. The Docker build is straightforward (python:3.11-slim, one service, uvicorn bot:app).

Rough edges (from reading the code, not guessing):

• Port mismatch: the README and Dockerfile use port 8000, bot.py's __main__ block uses 8006, and the frontend hardcodes localhost:8006 for its API calls. Align these before first run or the console can't reach the backend.
• Secrets hygiene: API keys are stored in plaintext in settings.json next to the code, and allow_origins=["*"] CORS is wide open. Fine for a demo on your machine; not for anything internet-facing.
• No call records: there's no transcript logging, no call-history endpoint, no recording metadata — the agent speaks and the evidence evaporates. Any business deployment needs a call log and transcript store.
• Twilio serializer uses a placeholder SID (TwilioFrameSerializer("dummy_sid")) — worth replacing with the real call SID for correct frame attribution at scale.
• No auth on any endpoint — anyone who can reach the server can place calls on your Twilio balance. Put it behind a reverse proxy with auth or a VPN.

This is a solid, readable prototype of the right architecture — the kind of repo that's genuinely useful as a starting point or a reference implementation, with the typical rough edges of a solo-built demo.

7. Roadmap — what would make it production-grade

1. Inbound handling — the TwiML endpoint already exists; wire incoming calls on the Twilio number into the same pipeline with a routing rule (inbound → receptionist prompt, outbound → sales prompt).
2. Transcripts + call log — persist every conversation (text frames are available in the Pipecat pipeline) with caller number, duration, and outcome; add a dashboard view in the console.
3. Booking integration — give the LLM function-calling tools (calendar check/booking, CRM lookup) so calls end in appointments, not just conversations.
4. Compliance — disclosure that the caller is an AI (required in the EU since the AI Act's Article 50 took effect in August 2026), consent logging for outbound calls, opt-out handling.
5. Hardening — API auth, secrets in a vault instead of settings.json, per-call cost tracking (Twilio + Deepgram + ElevenLabs + LLM all meter separately), and alerting on failures.

Frequently asked questions

The bottom line

calling-ai-agent is a rare thing in the AI-agent repo landscape: the code does what the README says. A Twilio → Pipecat → Deepgram → LLM → ElevenLabs voice loop, multi-provider brains, and a working operator console, in a codebase small enough to audit in an evening. It's a demo, not a product — bring your own credentials, fix the port mismatch, and don't expose it to the internet as-is — but as a starting point for anyone who wants to understand or build AI voice calling, it's the real thing.

Source: github.com/Shivay00001/calling-ai-agent — MIT licensed. Verified by reading backend/bot.py, frontend/src/app/page.tsx, the Dockerfile, and .env.example on the main branch.