How to Build an AI Voice Intake Agent in 21 Days

The first AI agent can go live in 21 days when the scope is narrow. In 2026, a service company in Singapore or Vietnam can install a voice intake agent that answers calls, captures intent, and books the next action. The build uses an 11B speech model for the conversation, a low cost text model for routing, and a calendar system for handoff. AI enablement means the company owns the workflow, the prompts, and the records. This blueprint gives the stack, the sequence, and the failure points. It is written for a founder who needs coverage without hiring a full AI team. Start with one workflow. One input. One output. One measure. Use missed calls, booking requests, or order status calls. Do not start with a company wide assistant. A narrow agent ships. A broad assistant stalls. The goal is coverage. The caller gets an answer. The team gets a clean record. The business keeps control.

The problem

Service teams lose revenue on missed calls. A clinic misses a booking request during lunch. A logistics desk misses an order status call after hours. An agency misses a prospect while the team is on delivery. The caller needs one fact or one action. The business treats the call as low priority. Then the call becomes a lost appointment, a delayed order, or a cold prospect. The cost is quiet. It does not appear in a report. It appears as gaps in the schedule, idle staff, and discounting to win back late prospects.

The root cause is not effort. It is coverage. A human team cannot answer every call while doing the work. The phone system captures no intent. The inbox captures no urgency. The calendar shows no open slot until someone checks it. Every missed call creates a manual recovery task. Recovery is slower than intake. The longer the gap, the higher the chance the caller chooses another provider. The founder sees the symptom as a staffing issue. It is usually a routing issue. The call is not hard. It arrives at the wrong time and has no owner.

Who this is for

This is for a B2B service founder in Southeast Asia with revenue between $500k and $5M. The business gets calls or voice notes that require the same ten answers. A clinic, a freight desk, a maintenance firm, or a professional services office fits. The team already uses a calendar, a CRM, or a shared inbox. The founder wants coverage without adding headcount. The build also fits a business with one person answering too many calls. It gives that person a filtered queue, not more noise.

Is this the right fit

Fit: repeated call types, clear next actions, calendar or ticket system in place. Non fit: highly regulated medical advice, complex legal counseling, or calls that require senior judgment every time. The agent handles intake, not diagnosis. It captures facts, confirms intent, and books the next step. It does not replace a licensed professional. It removes the first delay.

System architecture

The system has five parts. The caller speaks to a telephony number. The voice model answers, listens, and replies. The orchestrator receives the transcript and routes the intent. The calendar or ticket system supplies availability. The text model writes the summary and the follow up message.

Inputs: an inbound call, a missed call callback, or a voice note from WhatsApp. Outputs: a classified intent, a booked appointment or ticket, and a written record in the CRM.

Flow:

1. The telephony provider answers the call and streams audio to the speech model.
2. The speech model returns a spoken reply and a transcript.
3. The orchestrator checks the transcript against a list of allowed intents.
4. The text model extracts name, phone, service, time, and urgency.
5. The calendar API returns open slots or the ticket system creates a record.
6. The agent confirms the slot with the caller.
7. The orchestrator writes the summary to the CRM and sends a recap message.

A Postgres log stores every transcript, intent, and action. A human review queue catches low confidence calls. A consent note starts the call when recording is active. The stack can run on one GPU for the speech model and one VPS for the orchestrator. The text model can run by API. The caller should never hear a raw model error. If a node fails, the orchestrator plays a fallback message and creates a human queue item.

The design constraint is narrow scope. The agent does not answer every question. It collects the fields required for the next action. If the caller asks outside scope, the agent records the question and routes it to a human queue.

Build steps

1. Register a dedicated phone number in Twilio. Point the voice webhook to your orchestrator endpoint. Set the greeting to state the service name, the consent line, and the expected call length. Keep the greeting under 12 seconds. Route only the intake number to the agent. Leave the main sales number human until the agent passes a test week. This isolates risk and gives you a clean audit trail.

2. Deploy n8n on a VPS. Create a workflow named voice intake. Add a webhook node to receive the Twilio call event. Add nodes to store the caller number, call start time, and recording status. Add a queue node for fallback cases. The workflow is the control plane. It decides when the speech model speaks, when the text model extracts data, and when a human gets the record.

3. Host NVIDIA Nemotron VoiceChat 11B on a GPU runner. Use a managed inference endpoint if you do not want to manage CUDA drivers. Configure streaming audio so the caller hears a reply within two seconds. Set a short system prompt that limits the agent to intake. It must ask for name, service, preferred time, and urgency. It must not give medical, legal, or pricing advice.

4. Connect the DeepSeek V4 Flash API to n8n. Send the transcript after the caller finishes each answer. Ask the model to return one JSON object with five fields: intent, name, phone, service, and time window. Set a confidence threshold. If the model returns low confidence, route the call to the human review queue. This keeps extraction cheap and fast.

5. Connect Google Calendar API or Freshdesk to the orchestrator. For booking intents, query open slots for the next five business days. Offer two options only. For order status or ticket intents, create a Freshdesk ticket with the extracted fields. Lock the slot when the caller confirms. Do not let the model invent a time. The calendar system is the source of truth.

6. Build an allowed intent list in n8n. Use six intents: book, reschedule, status, pricing request, human request, and out of scope. Map each intent to one action. Book goes to the calendar node. Status goes to the ticket node. Human request goes to the queue. Out of scope goes to a recap message. This stops the agent from wandering.

7. Write a recap template in n8n. Send the recap through WhatsApp Business API or Twilio SMS. Include the confirmed action, the time, and the next step. Send the same summary to HubSpot or Pipedrive as a note. If the caller does not confirm, send a form request for details. Do not send a form in the voice call. Keep the record in the CRM.

8. Add a Postgres table for call logs. Store call ID, intent, extracted fields, confidence, and final action. Add a daily reconciliation job in n8n. Compare call logs against calendar events and tickets. Flag missing records. Review the flagged calls once per day for the first two weeks. This turns the agent into a measured system, not a black box.

9. Run a latency test before launch. Use five test numbers and measure reply delay, extraction accuracy, and booking success. Set thresholds: reply under two seconds, extraction above 90 percent, booking completion above 80 percent. Record each failed call in Postgres. Fix one failure at a time. Do not launch on a demo that passes once. Launch on ten clean test calls.

10. Build the human handover path. When the caller asks for a person, create a queue item in n8n and notify the duty phone through Slack or WhatsApp. Include the transcript, the extracted fields, and the reason for handover. Set a service level for human pickup. If no human answers in ten minutes, send the caller a recap form. This preserves control.

Tools and costs

Use a small stack. Each tool has one job.

• Twilio: phone number, call webhook, SMS. Cost is usage based. Estimate $1 to $20 per month for low volume, plus per minute rates.
• n8n: orchestrator. Self host on a VPS. The VPS costs about $20 to $80 per month depending on region and memory.
• NVIDIA Nemotron VoiceChat 11B: speech model. Run on one GPU. Estimate $0.50 to $1.50 per hour on a cloud GPU, or use a managed endpoint with per minute pricing.
• DeepSeek V4 Flash API: extraction and summary. Use API pricing. Estimate $0.05 to $0.20 per 1000 calls for light text, labeled estimate.
• Google Calendar API: availability and booking. Free for standard usage within workspace limits.
• Freshdesk or HubSpot: ticket and CRM record. Use your existing plan. A starter plan may cost $0 to $50 per user per month.
• Postgres: log store. Run on the same VPS or a managed database. Estimate $0 to $25 per month.

The first build should stay under $500 per month for a small team. If call volume rises, move the speech model to a dedicated GPU and cache common replies. Do not buy a full contact center suite for the first build. Start with one number, one workflow, and one report. Add channels only after the voice path passes for two weeks.

Failure modes

What breaks first

• The speech model mishears a name or number. Fix: ask the caller to confirm each field. Use the keypad or a recap message for phone numbers.
• The agent offers a slot that does not exist. Fix: query the calendar API before speaking. Lock the slot after confirmation.
• The transcript contains payment details. Fix: add a redaction rule in n8n. Do not store card numbers. Route payment requests to a secure form.
• Latency rises during peak hours. Fix: stream audio, cap the prompt, and keep the intent list small. If delay passes three seconds, send a brief hold message.
• The agent answers outside scope. Fix: restrict the system prompt and map unknown questions to the human queue.

Review the Postgres log once per day for two weeks. Count the failure type with the highest volume. Fix that one first. Do not add new intents until the current six pass the test thresholds.

What good looks like

The agent answers within three rings. It captures five fields before a human sees the record. It books two slot options from the live calendar. It sends a recap within 60 seconds. The CRM note arrives with intent, name, phone, service, time, and confidence. A human reviews only low confidence calls. The daily reconciliation shows no missing records. The cost stays under a fixed monthly ceiling. The team stops listening to voicemail. The calendar fills with qualified requests. The founder owns the intake engine. Coverage runs after hours without adding a night shift. The response time is a system property. The handoff rule is written. The next build can move to order status or payment forms. The owner can inspect the log without asking a vendor. The prompts live in the workflow. The thresholds live in the test sheet. The system is auditable.