Building Elle: The AI That Answers My Phone

If you call the number on my website right now, a voice picks up before the first ring finishes. Her name is Elle. She'll tell you she's my AI assistant, give you a quick rundown of what she can do, and then answer basically anything about my work, my projects, or my writing. She can book you a call with me. If you text the same number, she remembers the booking and will move it for you. If you open my website mid-call, she can pair your browser to the conversation and show you things while you talk.

I built her because I realized the best portfolio piece I could ship isn't a dashboard or a notebook. It's a phone number. A resume is a claim about what you can do. Elle is the claim, doing it, live, every time someone dials.

This post is the long version of how she works: the architecture, the latency war, the echo bug that made her interrupt herself at the same word every single call, and what the whole thing costs to run (spoiler: about three cents a minute, and my biggest expense is the server doing nothing).

The shape of the thing

Elle is one Node.js and TypeScript service running on Google Cloud Run. Around it sit four vendors doing exactly one job each:

• Twilio owns the phone number. When a call comes in, it opens a WebSocket to my server and streams the caller's raw audio: G.711 mu-law, 8 kHz, in 20 millisecond frames. Telephone audio, the same format the phone network has used for decades.
• Deepgram Nova listens to that stream and turns it into text, including deciding when the caller has finished talking, which turns out to be one of the most important knobs in the whole system.
• Claude Haiku is the brain. It gets my entire knowledge base in its prompt, a handful of tools (save a note, book a call, show an image on a paired browser), and a strict instruction to keep phone answers short.
• Deepgram Aura turns the reply back into speech, streamed straight back down the same WebSocket to Twilio.

That's the cascade: speech to text to reasoning to speech. The interesting part is that none of these vendors orchestrates the conversation. The turn-taking, the interruption handling, the audio pacing, the memory, the tools: all of that is my code, about 5,700 lines of TypeScript. You can buy this layer off the shelf now. Twilio sells it for $0.07 a minute. Deepgram sells it for $0.075. My version runs on about $0.03 a minute of raw parts, and I got to keep all the lessons.

The whole thing on one page. Blue is audio on the live call loop, green is text. Everything inside the dashed box is my code.

The one-second problem

Voice AI lives and dies on one number: the silence between when you stop talking and when the reply starts. Under a second feels like a conversation. Two seconds feels like a walkie-talkie. Three seconds and people start saying "hello? are you there?" into the phone, which I know because my logs are full of exactly that phrase from my early builds.

The naive pipeline is brutally slow, because everything waits for everything: wait for the full transcript, then wait for the full LLM reply, then wait for the full audio file, then play it. My first working version did something close to this and the pauses were painful.

Getting to roughly one second was not one fix. It was five:

• Stream the model, speak by sentence. The LLM reply streams token by token, and the moment a complete sentence exists (a regex watches for real terminal punctuation past a minimum length), that sentence goes to the voice while the rest is still generating. Elle starts saying the first sentence of her answer before she's finished thinking of the third.
• Keep the voice warm. Text-to-speech over a WebSocket has a handshake cost. So the connection opens once per call and stays open, and every utterance rides the same socket.
• No retrieval on the hot path. The obvious design is RAG: the model calls a search tool, waits for results, then answers. That's a whole extra model round-trip inside the latency budget. Instead, the phone prompt just contains the entire knowledge base. My resume, projects, and blog summaries are small enough that inline context is cheaper than the time a retrieval hop costs. The website chat, which nobody is holding a phone waiting for, uses proper retrieval with citations.
• Cap the answers. Phone replies are capped at 120 tokens. This is a personality decision as much as a latency one. Nobody wants a monologue from a phone call.
• Instrument everything. Every turn logs a timing line for each stage: when the endpoint fired, when the first LLM token landed, when the first sentence was ready, when the first audio frame left. When latency regressed, I didn't guess. I read which stage got slower.

Measured on real calls: first LLM token in 550 to 970 milliseconds, first audible word to the caller in 960 to 1,360 milliseconds. About a second of silence. A human pause.

Barge-in, or: she kept interrupting herself

The feature that separates a conversation from a phone tree is interruption. You should be able to talk over Elle, and she should stop. This sounds simple. It was the hardest thing in the entire build, and it produced my favorite bug.

The naive version uses voice activity detection: the transcription service fires an event when it hears speech, and you cancel whatever the bot is saying. The problem is that VAD fires on everything. A breath. A cough. A truck outside. And, fatally, the phone line echoing the bot's own voice back into the microphone. Elle would start talking, the line would echo her voice back, the VAD would decide the caller was speaking, and she would politely stop talking to let herself finish. Which she couldn't, because she'd stopped.

The fix came in layers, each one earned from a real failed call:

• Words, not noise. Interruption requires actual recognized text, at least six characters of it. Coughs and trucks don't transcribe. This killed most false positives.
• Grace windows. The first 1.2 seconds of any reply are protected, so the echo of her opening words can't cut her off instantly.
• Only interrupt a voice that's talking. Early on, "interruptions" that arrived while Elle was still thinking would cancel the reply being generated. The caller heard nothing, because the thing they supposedly interrupted hadn't started yet. Silence. "Are you there?" Now an interruption only counts while audio is actually playing.
• She knows what she sounds like. The last and best layer: everything Elle says goes into a rolling thirty-second window, and any incoming "caller speech" that is really just a chunk of her own recent words gets recognized as echo and ignored completely. It doesn't interrupt her, and it doesn't become a fake conversational turn for her to answer. She stopped replying to herself, which I realize is a sentence I never expected to write about a production system.

The greeting needed even more protection, because it's the longest thing she ever says and the echo of it is therefore the most likely thing to kill it. It cut off at the same spot every call, right after my name, like clockwork: the echo transcript crossed the interruption threshold at exactly that syllable. The greeting is now constitutionally uninterruptible. The system counts every millisecond of greeting audio queued versus sent, refuses all interruptions until the last frame is out the door, and refuses to start any conversational turn while the greeting is still being synthesized. If you talk over it, your words aren't lost; they queue, and she answers the moment the rundown ends.

The 20-millisecond metronome

Here's a bug I didn't expect: Elle slurred. Specifically, the ends of her sentences smeared, like a tape deck losing power.

Text-to-speech generates audio much faster than real time and delivers it in bursts. If you shove those bursts straight at the phone network, the playback buffer starves and stutters. So there's a pacer: audio queues up, and a timer releases exactly one 20 ms frame every 20 ms. Except JavaScript timers drift. A few milliseconds of lag per tick, compounding over a long sentence, and by the end the frames are arriving late. That's the slur.

The fix is to stop trusting the timer and trust the clock: every tick computes how many frames should have been sent by now based on wall-clock time, and sends however many are due. The timer ticks at 10 ms so it can catch up but never fall behind. First frame goes out instantly for latency; everything after marches to the metronome. She's been crisp since.

Memory, manners, and the shared phone problem

Elle remembers callers by phone number: names, contact info, things they told her. This produced a subtle and very human bug. A friend borrowed another friend's phone to call her, and Elle greeted him warmly by the wrong name. Later, a caller she'd never met got addressed like an old acquaintance, because someone else had once called from that number.

Phone numbers are not identities. The rule Elle lives by now: memory is context about the number, never about the person speaking. She will not call you by any name until you've told her one in the current conversation, no matter how confident her notes make her. If you give a different name than the notes expect, she goes with you and doesn't mention the discrepancy. It's the same courtesy a good human assistant would extend, which turned out to be the design principle behind most of her behavioral rules: when in doubt, ask what a great assistant would do, then write that down as a constraint.

She also texts, and that got legal fast

Booking a call by voice is nice. Changing it three days later shouldn't require calling back. So the same number answers SMS: text her and she can check, move, or cancel your booking, or make a new one, with the same brain and the same calendar.

What I didn't appreciate going in is that sending automated texts in the US is a regulated activity. Carriers require A2P 10DLC registration: a declared campaign, documented opt-in, mandatory STOP and HELP handling, disclosure messages, published terms. Building that properly meant a compliance gate that runs before any AI sees the message: STOP and its five synonyms kill everything and log the opt-out in a consent ledger; HELP returns the mandated identification and rates language; an opted-out number gets silence no matter what it sends until it opts back in. Every number's consent state, source, and timestamp are recorded, because "prove your opt-in" is a question carriers actually ask. Even the verbal path is covered: if you tell Elle on a call that texts are okay, she runs a proper consent script and a confirmation text with opt-out language goes out automatically, because verbal consent legally requires written confirmation.

Not the glamorous part of the project. Weirdly one of the parts I'm proudest of.

The browser joins the call

My favorite demo: while you're on the phone with her, ask to pair your browser. She reads you a six-digit code, you type it into my site, and your screen joins the call. The transcript streams live. She can put images on your screen while she talks about them. And as of this week, the channel runs both directions: there's a text box on the paired page, and anything you type lands in the live conversation as a real turn. You can ask a question by voice and the follow-up by keyboard, and she answers both out loud. One conversation, two input modes.

Under the hood this is small and satisfying: every call has an in-memory session with an event log and a set of listening browser sockets. Everything that happens fans out to every paired tab, and late joiners get the log replayed so a refreshed page shows the whole conversation. A typed message injects into the same queue that spoken utterances use, framed so the model knows it arrived by keyboard. Total new machinery: barely a hundred lines, because the pipeline was already shaped like a conversation rather than like a phone call.

What it costs

Per minute of conversation, at list prices: about nine tenths of a cent to Twilio for the call itself, eight tenths to Deepgram for listening, six tenths for speaking, and seven tenths to the LLM. Call it three cents a minute. A typical three-minute call is a dime. A full SMS exchange is a few cents. My friends stress-testing her for an evening cost me less than a coffee.

The real bill is stranger: the biggest line item is the server doing nothing. Cloud Run keeps one instance always warm so the first call of the day doesn't hit a cold start, and that costs $10 to $25 a month while the actual conversations cost pennies. At portfolio scale, availability costs more than usage. There's something clarifying about that: every additional person who calls her is nearly free. So call her.

What I actually learned

In my Moneyball series I wrote about the difference between finishing a dataset and building infrastructure. Elle is that lesson applied end to end. Nothing here is a notebook. Every subsystem has an offline verification suite that fakes its network dependencies and exercises the real logic: there's a fake Twilio socket and a fake LLM that prove barge-in flushes and cancels correctly, a fake calendar that proves bookings validate, a knowledge-base suite that proves queries land in the right sections. When I change the audio pacer at midnight, I know in eight seconds whether I broke the conversation, without placing a call.

The other lesson is that the hard problems in voice AI aren't the AI. The model was the easy part; I mostly just ask it to be brief and honest. The hard parts were telephone echo, timer drift, humans borrowing each other's phones, and the FCC. Real systems are mostly edges, and the edges are where I ended up learning the most.

What breaks the second you only have three layers

Draw a voice agent on a whiteboard and you get three boxes: speech to text, a language model, text to speech. Ears, brain, mouth. It's a clean picture, it's how I described Elle at the top of this post, and I now think it's wrong in a way that matters. Not incomplete. Wrong.

Here's the opinion, up front, so you can argue with it: a three-layer voice stack cannot hold a conversation, and no amount of tuning will get it there. It can hold an exchange. Question, answer, question, answer, politely taking turns like two people on a bad satellite link. That is not the same thing, and every hour I spent making Elle feel natural was an hour spent papering over the difference.

There's a fourth thing, and it isn't a box. It's whatever decides when to talk. Whether that pause meant "I'm finished" or "I'm thinking". Whether the caller talking over you is an interruption or just a cough. Whether to say "mhm" so the person knows you're still there. None of the three boxes owns that job, so you end up writing it yourself, and you find out fast that you're not writing a feature, you're writing a layer.

Here's what that actually looked like. My turn-taking runs on a silence timer, a thousand milliseconds of quiet and Elle decides you're done. That number is a confession. Set it lower and she cuts people off mid-thought. Set it higher and she feels slow on every single turn, including the ones where you clearly stopped talking. There is no value that is right for both, because the timer doesn't understand a word you said. A human waits differently after "so anyway" than after "what do you think?", and a timer cannot tell those apart.

Everything else in that layer is the same shape: a compensation for something that should have been modeled. Barge-in needs a minimum number of recognized characters, or a cough kills her sentence. It needs a grace window after she starts speaking, or the room's echo of her own voice interrupts her. It needs echo suppression on top of that, because she hears herself through the caller's speaker and starts answering her own question. Every one of those is a hand-tuned constant standing in for judgment.

And some things you cannot buy at any price. She can't murmur agreement while you're still talking, because the pipeline is strictly one turn at a time. Two people speaking at once has no representation in a system whose native unit is a finished transcript, so the most ordinary thing in human conversation is, architecturally, undefined. Worst of all, the text bottleneck that gives me verbatim compliance wording and swappable voices also throws away how something was said. Speech to text hands over the words and drops the hesitation, the sarcasm, the rising panic. Elle answers the sentence. She has no access to the delivery, and she never will, because that information died two layers upstream.

That's the part I want to be blunt about. These are not bugs in my implementation. Someone with more time would tune the constants better than I did and would hit exactly the same ceiling, because the ceiling is the shape of the pipeline. You cannot recover prosody after a transcript. You cannot represent overlap in a queue of turns. You cannot make a timer understand language. Three layers buys you a very good exchange, and then it stops.

Which is why the interaction layer has to be a model

I spent months assuming this was a tuning problem and that I just hadn't found the right constants. It isn't, and I want to be clear that this is not a private hunch I arrived at alone in a bedroom. Over the past year four independent sources, three of them labs with far better data than mine, published the same conclusion from four different directions.

Thinking Machines calls the turn-based stack a stopgap and argues interactivity belongs inside the architecture rather than in a harness bolted around it. Their line about a voice activity detector deciding you've finished speaking without understanding a word you said is, more or less exactly, the bug I've been describing. OpenAI took the turn detector out of GPT-Live's audio path entirely and pushed the heavier reasoning off to the side so it never blocks the conversation. ElevenLabs puts it in production terms: voice AI breaks in real conversation because it can't survive interruption, silence, or context that carries across turns. Sean Goedecke, writing as an engineer rather than a vendor, reads the same release and lands in the same place.

Read those four together and the useful thing is what they don't argue about. They disagree about where the interaction layer should live: fused into the weights, or written as orchestration around them. Not one of them treats its existence as optional. That is the claim I'd defend: the interaction layer is not a feature of a voice agent, it is a requirement, and if you didn't design one you have one anyway. Mine is the barge-in gate, the echo suppression, the metronome, and the cancellation token. It is the hardest code in the project, it is about a fifth of it, and I did not set out to write any of it. It accreted, one constant at a time, because the architecture left a hole and something had to fill it.

Which is the practical warning I'd give anyone starting one of these in 2027. You will estimate the three boxes, because the three boxes are the part you can see. The three boxes are the easy part. They are an afternoon of API calls. The schedule will be eaten by the layer nobody drew, and you will not recognise it as a layer while you're building it, because it arrives disguised as a list of small bugs.

So I built the exit before I needed it. The Twilio bridge doesn't know which architecture is behind it; it talks to an interface, and there are three implementations sitting behind that seam. The production cascade, and two fused speech-to-speech models that do their own turn-taking, one of which runs inside my own cloud project on credentials I already have. I can move a live call from one to the other and back without a deploy. Not because I need it today, but because the version of this that's actually good is going to be a model, and I'd rather the swap be one line than a rewrite.

What changes when you scale it

At my volume the economics are almost funny. Three cents a minute of conversation, and the largest line on the bill is the always-warm instance doing nothing, $10 to $25 a month so the first caller of the day doesn't wait for a cold start. Availability costs more than usage. Every extra caller is nearly free.

That inverts, and it inverts at a knowable point. Prompt caching is what buys the runway: the persona and the entire knowledge base are a stable prefix, so I pay full price for them once and a fraction thereafter, which is why per-call cost barely moves as conversations get longer. Push volume up and the hosted APIs stay linear while self-hosting is a step function that starts high and flattens. They cross around 80,000 talking minutes a month, where a single GPU starts to pencil out, and a fleet doesn't win outright until somewhere north of four million. Below the first crossing, renting is not the compromise, it's correct by an order of magnitude.

The parts that break first at scale aren't the model, though, they're the boring ones. SQLite rides along on the instance, which is fine for one warm container and wrong the moment there are two, so that becomes a managed database. Recording a call is cheap; storing and retaining and deleting them on a schedule is a compliance program. One phone number is a portfolio piece; a queue of them is a telephony product with a routing problem. And the abuse policy I wrote to keep a bored autodialer from running up a bill stops being a cost control and starts being a real trust and safety surface.

None of that is exotic. It's just the usual thing: the demo is the model, and the product is everything around it.

Here is the whole thing as one table, which is the version I wish someone had handed me before I started pricing this out.

The number I keep coming back to is the last row's neighbour. The same 6.25 million minutes staffed by people, at a loaded rate of ninety cents an agent-minute, is about $5.6 million a month. So the interesting comparison at the bottom of that table was never hosted against self-hosted. It's roughly $85,000 against $5.6 million, and at that ratio the unit price of a token is a rounding error. Which is a slightly uncomfortable thing to notice at the end of a post about shaving milliseconds off an audio pacer.

Elle's code, including everything this post describes, lives in my repositories, and the full engineering documentation is published: the system map, the schema, the voice pipeline, and file-by-file references. But honestly, the documentation isn't the deliverable.

The phone number is. Call her and ask her something hard. If she waits a beat too long before answering, now you know exactly which layer to blame.

Sources

The argument in the last three sections leans on four pieces published over the past year. If you only read one, read the first: it is the one that reframed this for me from a tuning problem into an architecture problem.

1. Thinking Machines Lab, Interaction Models: A Scalable Approach to Human-AI Collaboration. Argues interactivity belongs inside the model architecture rather than in a harness around it, and calls the turn-based stack a stopgap.
2. OpenAI, Continuous voice interaction with GPT-Live. The turn detector removed from the audio path, full duplex, with heavier reasoning escalated asynchronously so it never blocks the conversation.
3. ElevenLabs, Interaction models: Building natural human-AI dialogue. The same problem stated in production terms, and notable for describing an advanced cascade as the route toward it, which is the architecture Elle already is.
4. Sean Goedecke, Thinking Machines and interaction models. An engineer's read rather than a vendor's, reaching the same structural conclusion.

My own engineering documentation for Elle carries the same citations in its introduction, alongside the four-layer model and where this architecture sits on the fusion spectrum.