AI phone agent: how it works and how to build one

An AI phone agent is software that talks on a real phone line. It hears the caller, decides what to say with a large language model (LLM), and speaks the answer back. It can answer calls to your number, place calls for you, and hand a call to a person when it should.

This guide explains the parts of an AI phone agent, the ways to build one, and the limits that matter before you put one on a real line.

What are AI voice agents?

AI voice agents are programs that hold a spoken conversation. An AI phone agent is a voice agent that works over the phone network, not only in a browser or an app.

Each turn of the conversation follows the same loop:

1. Speech to text (STT) turns the caller's words into text.
2. The LLM reads the text, the instructions, and the conversation so far. It writes a reply or calls a tool.
3. Text to speech (TTS) turns the reply into audio and plays it on the call.
4. Turn detection decides when the caller has finished, so the agent does not talk over them.

The instructions are usually one system prompt. It says who the agent is, what it may do, and when it must stop.

AI phone answering: inbound calls

For AI phone answering, you point a phone number at an agent. When someone calls, the agent picks up, says it is an AI, and follows its instructions.

Typical inbound jobs:

• Answer questions about hours, prices, or an order.
• Take a message and send it to a person.
• Book, change, or cancel an appointment, when the agent has a tool for it.
• Pass the call to a person for anything the agent must not handle.

AI call bot: outbound calls

An AI call bot places calls. Your software starts each call with a request, and the agent runs the conversation.

Typical outbound jobs:

• Confirm or remind about an appointment.
• Call back a lead who asked for a call.
• Collect a short answer, such as a callback time.

Outbound calls need permission. Each call to a person must have a lawful basis, such as consent the person gave you. The first AI phone call tutorial shows one outbound call from start to finish.

The parts of an AI phone system

An AI phone system needs more than a model. Plan for each of these parts:

Part	What it does	What to decide
Phone number	Receives and places calls on the phone network.	Country, local or toll-free, and text support.
Speech	STT, TTS, and turn detection.	Language and voice.
LLM	Decides what to say and which tool to use.	Model, prompt, and what happens when it fails.
Tools	Actions during the call: hang up, press keys, send a text, transfer.	Which tools the agent may use.
Handover	Sends the call to a person.	Who, when, and what the agent says first.
Records	Status, events, and recordings after the call.	Webhooks, storage, and retention.

Voice AI agents: three ways to build one

There are three common ways to build voice AI agents for the phone.

	All-in agent platform	Communications API with agents	Do it yourself
Example	Vapi, Retell AI	Hail	Carrier SIP trunk, STT, LLM, and TTS vendors
Setup time	Short	Short	Long
Control	Dashboard, prompts, and APIs	API fields, your own prompt, optional own LLM	Full
Vendors to manage	One	One	Four or more
Other channels	Varies	SMS and email in the same account	You add them
Best for	Teams that want a visual builder	Developers and AI agents that need calls, texts, and email from code	Teams with special latency or hosting needs

Both platforms also support API setup: Vapi docs and Retell AI docs. Sources checked in October 2026.

Read Hail vs Vapi and Hail vs Retell AI for side-by-side details.

How to build an AI phone agent with Hail

Hail saves the agent once. Then you route a number to it. The same agent can also place outbound calls.

1. Save the agent with POST /v1/agents. Only name and system_prompt are required.

curl --fail-with-body --silent --show-error \
  https://api.hail.so/v1/agents \
  -H "Authorization: Bearer $HAIL_API_KEY" \
  -H 'Content-Type: application/json' \
  --data '{
    "name": "Front desk",
    "system_prompt": "You answer calls for Acme Dental. Give opening hours. Take a name and a callback number for anything else. Never give medical advice.",
    "first_message": "Thanks for calling Acme Dental. How can I help?"
  }'

2. Route your number to the agent with PATCH /v1/numbers/{id}. Use the agent id from step 1. Setting voice_agent_id registers the number for inbound calls. Setting sms_agent_id lets the same agent reply to texts.

curl --fail-with-body --silent --show-error -X PATCH \
  "https://api.hail.so/v1/numbers/$NUMBER_ID" \
  -H "Authorization: Bearer $HAIL_API_KEY" \
  -H 'Content-Type: application/json' \
  --data '{"voice_agent_id": "'"$AGENT_ID"'", "sms_agent_id": "'"$AGENT_ID"'"}'

3. Call the number. Hail adds an AI disclosure to your opening greeting. Then the agent follows your prompt.

4. Listen for events. Hail sends call.received when a call comes in, then call.answered and call.completed. See the webhooks docs.

To place an outbound call with the same agent, send agent_id to POST /v1/calls instead of a system_prompt. The agents docs list every field.

You can also do each step from an AI assistant over MCP (create_agent, then route_number) or from the CLI.

Write the prompt before you pick the vendor

The prompt decides most of the result. Start from this template, then fill in your facts. The voice agent prompt builder makes longer versions.

Limits to plan for

• Consent. Every outbound call request needs recipient_consent: true. It is your statement that you have a lawful basis to call. Hail does not check it for you.
• AI disclosure. The agent says it is an AI at the start of the call. This is on by default. If you turn it off, you take responsibility, and the agent still says it is an AI when asked.
• Voicemail. On outbound calls, Hail detects answering machines. If a machine answers, the agent hangs up without a word and leaves no message.
• Handover. You can list up to 10 people. The agent calls the right one, says who is calling and why, then goes silent. If nobody answers in 30 seconds, the agent comes back and offers to take a message.
• Length. Set max_duration_seconds on the agent (60 to 3600) so a stuck call ends.
• Cost. Inbound and outbound minutes cost the same on Hail. Estimate a month with the voice agent cost calculator, and check pricing.
• Testing. A completed call is not a finished task. Listen to test calls, and try rude, silent, and confused callers.

AI phone agent guides

• Make your first AI phone call with a voice API
• AI outbound calls: make your first call