# Hail: full docs corpus

> Let your AI assistant make and receive calls, send texts, and manage email. Keep track of conversations across calls and texts. Get phone numbers in 120+ countries. Choose a voice or bring your own. Open-source or cloud priced on usage.

This file combines all prose docs into one plain-text document for AI agents. Pricing data lives at /llms.txt and /costs.md.

<!-- source: https://hail.so/docs/mcp -->

# MCP clients

Hail exposes MCP as a **remote server**. Hail Cloud uses OAuth. Paste the URL into your client. Click Allow in the browser. Your agent then gets the call, SMS, email, agent, number and verification tools. There are no keys to manage and no installation.

> For the easy onboarding path, refer to the [client picker on hail.so/mcp](https://hail.so/mcp). It has copy-paste setup for the 8 most common clients (Claude.ai, ChatGPT, Cursor, Gemini, …). This page is the technical reference behind those snippets.

## URL

- **Hail Cloud**: `https://mcp.hail.so`
- **Self-hosted**: `http://<your-host>:8081` — refer to [Self-host](#self-host) below.

The Streamable HTTP transport serves the MCP root path. There is no `/mcp` suffix and no SSE.

> For web-based clients (Claude.ai, ChatGPT), the URL must be reachable from the client's servers — public DNS + TLS. If you want web clients to reach a self-hosted instance, tunnel it via cloudflared / tailscale funnel.

## Tools

Example. Ask your agent: "Find me a US number that can call and text." It calls `quote_numbers` with:

```json
{ "country_code": "US", "capabilities": ["voice", "sms"] }
```

The server exposes 36 tools. Schemas (args, validation, return shapes) are the source of truth — refer to [`mcp/hailhq/mcp/tools.py`](https://github.com/hail-hq/hail/blob/main/mcp/hailhq/mcp/tools.py).

Every tool sets the MCP hints `readOnlyHint`, `destructiveHint`, `idempotentHint` and `openWorldHint`, so clients can ask for approval before a write. `release_number`, `delete_agent`, `acquire_number`, `submit_verification` and `cancel_verification` are marked destructive. The first three also need a confirmation argument (`confirm_e164`, `confirm_name`, `confirm_total_cents`). `submit_verification` needs `attest_authorized: true`.

`place_call`, `send_sms` and `send_email` need `recipient_consent` set to the JSON boolean `true`. Strings such as `"true"` and numbers such as `1` are refused before any request is sent.

| Tool                            | Does                                                                                               |
| ------------------------------- | -------------------------------------------------------------------------------------------------- |
| `place_call`                    | Originate an outbound phone call.                                                                  |
| `get_call`                      | Fetch the current state of one call.                                                               |
| `list_calls`                    | List recent calls (cursor-paginated).                                                              |
| `send_sms`                      | Send an outbound SMS (recipient consent is required).                                              |
| `get_sms`                       | Fetch the current state of one SMS.                                                                |
| `list_sms`                      | List recent SMS messages (cursor-paginated).                                                       |
| `send_email`                    | Send an outbound email (supports `attachment_ids`).                                                |
| `upload_email_attachment`       | Upload a file, get back a reusable id.                                                             |
| `get_email`                     | Fetch one email's full record (body + inbound headers).                                            |
| `list_emails`                   | List emails (`direction="inbound"` for replies).                                                   |
| `get_email_raw`                 | Presigned URL for an inbound email's raw MIME.                                                     |
| `get_email_attachment`          | Presigned URL for one inbound attachment.                                                          |
| `get_email_events`              | Page through one email's event history.                                                            |
| `get_email_stats`               | Aggregate email counts for a time window.                                                          |
| `get_events`                    | Page through the event stream.                                                                     |
| `list_email_domains`            | List sending identities + the default `from` address.                                              |
| `whoami`                        | Identify the human behind the session (for `reply_to`).                                            |
| `list_contacts`                 | List the org's contacts (members + manual contacts).                                               |
| `lookup_contact`                | Find one contact by name, email, or phone fragment.                                                |
| `create_contact`                | Add a manual contact.                                                                              |
| `list_agents`                   | List saved agents.                                                                                 |
| `get_agent`                     | Fetch one saved agent.                                                                             |
| `create_agent`                  | Save an agent that answers calls and texts.                                                        |
| `update_agent`                  | Change an agent. `clear_fields` removes optional values.                                           |
| `delete_agent`                  | Delete an agent. Destructive. Needs `confirm_name`.                                                |
| `route_number`                  | Choose which agent answers a number.                                                               |
| `quote_numbers`                 | Compare live carrier offers for a new number (read-only).                                          |
| `list_number_regions`           | List the US/CA state codes `quote_numbers` takes as optional `region` (read-only).                 |
| `acquire_number`                | Buy a quoted number. Spends money. Needs `confirm_total_cents`.                                    |
| `list_numbers`                  | List the org's numbers (cursor-paginated).                                                         |
| `get_number`                    | Fetch one number.                                                                                  |
| `release_number`                | Release a number. Destructive and permanent. Needs `confirm_e164`.                                 |
| `get_verification_requirements` | Read what a carrier needs to verify the org for a number type (read-only).                         |
| `submit_verification`           | Send details and documents to the carrier. Irreversible. Needs `attest_authorized`.                |
| `list_verifications`            | List the org's verifications and their states (read-only).                                         |
| `get_verification`              | Fetch one verification; poll it for `approved` (read-only).                                        |
| `cancel_verification`           | Cancel an `awaiting_review` one, dismiss a `rejected` one, or end an approved US SMS registration. |
| `confirm_verification`          | Enter the code the carrier texted a person (`confirmation: "code"`).                               |

### Getting a number over MCP

1. `quote_numbers`: `{"country_code": "US", "capabilities": ["voice", "sms"]}`. Offers list `monthly_cents`, `setup_cents` and `quote_id`. Money is in cents. Quotes expire in 10 minutes.
2. `acquire_number`: copy `quote_id`, `country_code` and `number_type` from the offer. Set `confirm_total_cents` to `monthly_cents + setup_cents`. For an offer with `monthly_cents: 115` and `setup_cents: 50`:

   ```json
   {
     "quote_id": "8b1f0c2e-0000-4000-8000-000000000001",
     "country_code": "US",
     "number_type": "local",
     "confirm_total_cents": 165
   }
   ```

   The server checks the total. If it differs from the quote, the call fails with 409 and nothing is charged. The result is the number plus an `idempotency_key`. To retry after a timeout, pass that key as `idempotency_key`.

3. `route_number`: `{"number_id": "<id>", "voice_agent_id": "<agent-id>"}`.

> **Warning:** `acquire_number` buys a real number. It charges the setup fee and the first month now, then renews monthly. The account balance is the only spending limit. Keep the human approval prompt on for this tool.

### Buying a number that needs verification

Some numbers, for example a UK mobile, need a carrier check of the organization first. `quote_numbers` marks those offers `readiness: "verification_required"`. The whole flow works over MCP:

1. `quote_numbers`: `{"country_code": "GB", "number_type": "mobile"}`. The offer has `readiness: "verification_required"`, its `quote_id`, and `subject_types`: who may verify for it (`person`, `business`, or both).
2. Ask the user: "you or your company?". That is `subject_type` (`person` or `business`). It is required; never guess it.
3. `get_verification_requirements`: `{"country_code": "GB", "number_type": "mobile", "quote_id": "<quote_id>", "subject_type": "person"}`. The result lists `fields`, `documents` (skip a slot whose `needs_input` is false) and `address_required`.
4. Ask the user for every field and file. The agent must not invent or guess any value.
5. `submit_verification`. `files[].content_base64` is the file's bytes in base64. Slot names in `files` must be keys in `documents`:

   ```json
   {
     "country_code": "GB",
     "number_type": "mobile",
     "quote_id": "<quote_id>",
     "subject_type": "person",
     "fields": { "first_name": "Ada", "last_name": "Lovelace" },
     "address": {
       "customer_name": "Ada Lovelace",
       "street": "1 Example Road",
       "city": "London",
       "region": "London",
       "postal_code": "N1 1AA",
       "country_code": "GB"
     },
     "documents": { "identity": { "option": "passport", "fields": {} } },
     "files": [
       {
         "slot": "identity",
         "content_type": "image/jpeg",
         "content_base64": "<base64 of the passport photo>"
       }
     ],
     "attest_authorized": true
   }
   ```

   The result holds the verification `id` and `state`, never the details. If the carrier finds a problem, the result lists `problems` (`loc`, `msg`) and nothing is stored. Fix them with the user and submit again.

6. `get_verification`: `{"verification_id": "<id>"}`. Poll every few minutes to hours, not every few seconds. Review takes business days. Hail sends no email or webhook when it ends.
7. On `approved`: `quote_numbers` again. The offer is now `ready`. Then `acquire_number` and `route_number` as above. The organization may hold several approved verifications (several people or companies). Each entry in the offer's `verifications` has a `subject_name`: the person's or company's name from the form. When more than one is approved, show the user the names, ask which one, and pass its `id` as `verification_id` to `acquire_number`.
8. On `rejected`: read `rejection_reason`, ask the user, call `cancel_verification` to dismiss it, and submit a new one.

A person's US texting registration may wait on the person. `confirmation: "code"`: ask them for the code the carrier texted and call `confirm_verification` with `{"verification_id": "<id>", "code": "<code>"}`. `confirmation: "reply"`: they answer the carrier's text; nothing to call.

To register the organization to text US numbers from a US number, pass `purpose: "sms"` (default `"number"`) with `country_code: "US"` and `number_type: "local"` to `get_verification_requirements`, `submit_verification` and `list_verifications`. The carrier fee is debited from credit at filing:

```json
{
  "country_code": "US",
  "number_type": "local",
  "subject_type": "business",
  "purpose": "sms",
  "fields": { "legal_name": "Acme Inc" },
  "documents": {},
  "files": [],
  "attest_authorized": true
}
```

Limits: JPG, PNG or PDF files, 10 MiB each, 29 MiB in total, at most 20 files and 20 fields. Compress photos to under about 7 MiB. Slot names use letters, digits, `-` and `_` (at most 64). Each file must match its type. A second submission for the same country, number type, `subject_type` and name returns 409 while one exists.

> **Warning:** The user's details and documents pass through the model's context and are kept in the client's transcript. Keep the human approval prompt on for `submit_verification`, and have the user hand over each value and file in the conversation. A submitted verification cannot be cancelled, and the carrier keeps the documents. Only `awaiting_review` and `rejected` ones can be cancelled, plus an approved US SMS registration.

## Claude.ai (web)

1. **Settings → Connectors → Add custom connector**
2. Server URL: `https://mcp.hail.so`
3. Save. Claude prompts you to authorize on first use. Click **Allow** in the browser tab that opens.

That is all. There is no API key field.

## ChatGPT (web)

Custom MCP connectors are behind Developer Mode on ChatGPT today:

1. **Settings → Connectors → Advanced → Developer mode**
2. **Create** → paste `https://mcp.hail.so`
3. Save. ChatGPT guides you through the OAuth consent on the first call.

## Other clients

The [client picker](https://hail.so/mcp) has setup snippets for Cursor, Gemini, Windsurf, Copilot, Zed, Raycast, and Claude Desktop. All follow the same shape: paste the URL, then click Allow.

## Authorized apps

Each cloud client that you connect appears as a row at [`hail.so/console/apps`](https://hail.so/signin?next=%2Fconsole%2Fapps). Revoke deletes the consent and every active access and refresh token for that client. The client's next tool call returns 401 and runs OAuth again from the start.

Access tokens last 30 days, and refresh tokens last 180 days. In practice, you authorize each client again approximately every six months. The consent screen opens, you click **Allow**, and the client continues.

## Self-host

Self-hosted deployments do not use OAuth. Set `HAIL_API_KEY` and send it as a bearer token:

```sh
curl -H "Authorization: Bearer ${HAIL_API_KEY}" http://localhost:8081/
```

The MCP service selects its auth mode from env at boot. For the full env-var contract, refer to the [MCP modes table in the operations runbook](./self-host/operations.md#mcp-modes) (`HAIL_AUTH_URL` for cloud, `HAIL_API_KEY` for self-host, mutually exclusive).

## Stdio

For hosts that require stdio, run the entry point in the self-hosted MCP container. From your Hail checkout:

```sh
docker compose run --rm -T mcp python -u -m hailhq.mcp.stdio
```

The container reads `HAIL_API_KEY` from `.env` and calls the API at `http://api:8080`.
`HAIL_AUTH_URL` must be empty in `.env`, not only in your shell: stdio reads `.env` and refuses to start when it is set.
Stdio uses the same tools as the HTTP service.

---

<!-- source: https://hail.so/docs/agents -->

# Agents

An agent is a saved brain: instructions, greeting, AI line, voice, tools and
limits. A number routes its incoming calls to one agent and its incoming texts
to one agent. `POST /calls` can place outbound calls with the same agent.

```bash
# 1. Save an agent.
curl -X POST "$HAIL_API_URL/v1/agents" -H "Authorization: Bearer $HAIL_API_KEY" \
  -H 'Content-Type: application/json' -d '{
    "name": "Front desk",
    "system_prompt": "You are the front desk of Acme Dental. Book, change or cancel appointments.",
    "first_message": "Thanks for calling. Are you booking, changing, or cancelling an appointment?",
    "voice_config": {"language": "en"}
  }'
# → {"id": "a1b2...", ...}

# 2. Point a number at it. Calls and texts to the number are now answered.
curl -X PATCH "$HAIL_API_URL/v1/numbers/$NUMBER_ID" -H "Authorization: Bearer $HAIL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"voice_agent_id": "a1b2...", "sms_agent_id": "a1b2..."}'

# 3. Or place an outbound call with it.
curl -X POST "$HAIL_API_URL/v1/calls" -H "Authorization: Bearer $HAIL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"to": "+14155550100", "agent_id": "a1b2...", "recipient_consent": true}'
```

CLI: `hail agents create "Front desk" --prompt-file ./front-desk.md`, then
`hail numbers route <number-id> --calls <agent-id> --texts <agent-id>`.
Restrict tools with `hail agents update <id> --tools end_call,send_sms`; allow
all again with `hail agents update <id> --all-tools`.
MCP: `list_agents`, `get_agent`, `create_agent`, `update_agent`, `delete_agent`, `route_number`, and `place_call(agent_id=...)`.
SDK: `client.agents.create(...)`, `client.numbers.route(...)`.

Schemas: `AgentCreate`, `AgentUpdate`, `AgentResponse`, `PhoneNumberRoutingUpdate`
in [`openapi/openapi.yaml`](../../openapi/openapi.yaml).

## Hand over to a person

Let the agent pass a live call to a person you picked: a team member with a
phone number, or a contact. Use the `id` that `GET /contacts` (or
`hail contacts list`) shows: a contact's uuid, or `member:<user id>` for a
team member.

```bash
curl -X PATCH "$HAIL_API_URL/v1/agents/$AGENT_ID" -H "Authorization: Bearer $HAIL_API_KEY" \
  -H 'Content-Type: application/json' -d '{
    "handover_contacts": [
      {"contact_id": "member:6f1c2a9e-3b7d-4c51-9a8e-2d4f6b1e0c37", "note": "sales questions"},
      {"contact_id": "'"$CONTACT_ID"'", "note": "billing questions or an angry caller"}
    ]
  }'
```

CLI: `hail agents update <id> --handover "member:<user-id>=sales questions"`
(repeat the flag for more people; `--no-handover` clears the list).
MCP: `create_agent(handover_contacts=[{"contact_id": "...", "note": "..."}])` sets the list;
`update_agent(agent_id, handover_contacts=[...])` replaces it, and `[]` removes every contact.

- Up to 10 people. Each needs a phone number in a country Hail sells
  numbers in. A team member must belong to the organization; one who leaves
  it is skipped and cannot be dialed. It must pass the do-not-call and premium-rate checks, on save
  and again before dialing. `note` is 1-200 characters and tells the agent
  when to hand over.
- The agent gets the `transfer_call` tool. If `tools` is a list, saving
  contacts adds `transfer_call` to it and clearing them removes it; `null`
  (all tools) stays `null`. The agent never sees a phone number.
- The contact sees your Hail number as caller ID.
- The contact has 30 seconds to answer. If not, the agent comes back and
  offers to take a message.
- On answer, the agent says who is calling and why, then goes silent. If
  either side hangs up, the call ends.
- `max_duration_seconds` stops applying once the contact answers.
  `handover_max_duration_seconds` (60..3600, default 1800 = 30 minutes;
  CLI `--handover-max-minutes`) counts from the answer. When it is reached
  the agent says goodbye and the call ends for both people. Either person
  can hang up sooner.
- Billed as one call at the normal voice rate.
- Sends a [`call.transferred`](webhooks.md) webhook when the contact answers.
- `PATCH` with `[]` clears the list; omit the field (or send `null`) to leave it.

## What happens on an inbound call

1. The carrier sends the call to LiveKit. LiveKit creates a room and
   dispatches the voicebot ([setup](./self-host/livekit-cloud.md#4-inbound-calls)).
2. The voicebot reads the dialed number and the caller from the SIP
   participant and asks [`hailhq.core.inbound_calls`](../../core/hailhq/core/inbound_calls.py)
   who answers:
   - number not yours, a pool number, or a number on the wrong trunk: the call
     is dropped, no record;
   - no live voice agent (paused, or `voice_enabled: false`): the call is
     recorded as `failed`, `end_reason` `no_agent`, and `call.failed` is sent;
   - no credits, or voice suspended: `failed` with `insufficient_funds`
     (or `user_rejected`), and `call.failed`;
   - otherwise a `ringing` call, `call.received`, then the agent speaks.
3. The agent speaks the AI line, then `first_message` (or waits), then
   follows `system_prompt` with the agent's tools. With a `first_message`
   and no custom AI line, the agent speaks one greeting instead: the first
   sentence of `first_message`, then `I'm an AI assistant for {org}.`, then
   the rest. `call.answered` and
   `call.completed` follow as on outbound calls. Each answered minute costs
   the same as an outbound minute.

The AI line on inbound calls defaults to
`Hi, this is an AI assistant answering on behalf of {org}.` (or the short
line inside `first_message`, above); the workspace
default lives in Call settings (`organization_call_settings.ai_disclosure_line`)
and an agent can set its own `ai_disclosure_line`. `ai_disclosure: false` on the
agent skips it; the responsibility for that is yours.

## What happens on an inbound text

1. Hail stores the text and sends `sms.received` as before.
2. `STOP`, `CANCEL`, `END`, `QUIT`, `UNSUBSCRIBE` and `HELP`/`INFO` are answered by Hail and never reach
   the agent. `YES`, `START` and `UNSTOP` are an opt-in only from a person who opted out; Hail
   turns texts back on and replies. From anyone else they are an answer, and the agent gets them.
3. Any other text to a number with `sms_agent_id` set (agent `live`,
   `sms_enabled: true`) is answered by the agent: one reply, written from
   `system_prompt` and the last 20 texts of the last 24 hours with the caller
   (plus texts between that number and the caller that have no agent: sent
   through `POST /sms`, or received before the number had a text agent). Older
   texts and calls it reads with the `thread_history` tool (see [Threads](#threads)).
   The reply is sent from the same number through its carrier. The agent sends at most 20
   replies per thread in any 24 hours; past that it stays quiet until older
   replies leave the window. Replies bill as outbound SMS. Inbound rows carry `agent_reply_state`
   (`pending`, `processing`, `done`, `skipped`, `failed`) in the database;
   replies carry `agent_id`.

The text worker runs inside the voicebot service
([`hailhq.voicebot.textbot`](../../voicebot/hailhq/voicebot/textbot.py)) and
needs `HAIL_INTERNAL_SECRET` and `HAIL_API_URL`. It answers
`HAIL_TEXT_REPLY_CONCURRENCY` texts at once. A failed model or API call is
retried up to 3 times with backoff; a text still unanswered after
`HAIL_TEXT_REPLY_MAX_AGE_SECONDS` is skipped.

## Threads

A caller texts an order number to the agent's SMS number, then calls the agent.
When the caller asks about it, the agent looks it up with the `thread_history`
tool. A text sent during a call goes to the voice agent and is then marked
`done`. A text sent from the moment the call row is created (ringing included)
counts. If the voice agent never got it, the text agent answers it when the
call ends, or when the stale-call sweep closes the call. The reply age limit
(`HAIL_TEXT_REPLY_MAX_AGE_SECONDS`) restarts then.

- Scope: agent + caller number. Texts and call turns both count.
- Prompts carry no history. Both agents read it with the `thread_history` tool:
  `source` is `sms` (texts), `voice` (call turns) or `all`; `before` pages back
  over the last 7 days; `item_id` returns one full message. The voice prompt
  gets one line telling the agent to use the tool when the call has it; the
  text prompt always has it. Why: history in the prompt also held the agent's
  own past "I don't have that" lines, and the model repeated them.
- The text agent's chat holds the last 20 texts of the last 24 hours, no call
  turns. `thread_history` is its only tool.
- `thread_history` also returns texts between the number and the caller that
  have no agent.
- Hidden or invalid caller numbers get no history.
- Caller ID on a phone call can be faked. Do not put secrets in an agent's
  instructions or in texts it sends.
- History the tool returns is sent to the agent's LLM, including a bring-your-own endpoint.
- `send_sms` picks the number in this order:
  1. the dialed number, if it can text and is not bound to another agent.
     Bound to this agent: used. Free: it is bound to this agent when the agent
     has `sms_enabled`, else used unbound;
  2. the oldest org SMS number bound to this agent;
  3. the oldest free org SMS number: bound to this agent when it has
     `sms_enabled`, else used unbound;
  4. none: the agent says it cannot text.

  It never uses a number bound to another agent. A number bound this way
  makes the agent answer every text sent to it (audit entry `number.route`).
  A number used unbound stays unrouted.

Code: [`threads.py`](../../core/hailhq/core/threads.py),
[`text_watch.py`](../../voicebot/hailhq/voicebot/text_watch.py),
[`thread_history.py`](../../core/hailhq/core/agent_tools/thread_history.py).

## Routing rules

- `voice_agent_id` needs the `voice` capability and an agent with
  `voice_enabled: true`; otherwise the request returns 422. Setting it attaches the
  number for inbound at the carrier and lists it on Hail's LiveKit inbound
  trunk for that carrier ([`hailhq.core.inbound_routing`](../../core/hailhq/core/inbound_routing.py));
  `null` undoes both. `PhoneNumberResponse.inbound_registered` says where it stands.
- `sms_agent_id` needs the `sms` capability and an agent with
  `sms_enabled: true`; otherwise the request returns 422. No carrier work: inbound texts
  already reach Hail.
- Releasing a number, or deleting its agent, unregisters it first.
- A paused agent (`status: paused`) answers nothing; calls to its numbers
  fail with `no_agent`, and `POST /calls` with its `agent_id` returns 409. `POST /calls` with an agent
  that has `voice_enabled: false` also returns 409.

---

<!-- source: https://hail.so/docs/webhooks -->

# Webhooks

When a subscribed event occurs — inbound mail or SMS, a delivery report, a
call outcome — Hail `POST`s a signed JSON event to your URL. Verify the
`X-Hail-Signature` header against the once-shown secret. Return any `2xx`,
and you are done. Hail retries a non-2xx response (or a timeout) on a fixed
ladder.

## Verify the signature (Python)

The signature is HMAC-SHA256 over `f"{t}.{body}"` — the timestamp from the
header, a literal `.`, then the **raw request bytes**. This is a runnable
example. The `assert` passes (the real signer in
[`core/hailhq/core/webhooks.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/webhooks.py) produced
the fixture):

```python
import hashlib, hmac

def verify(secret: str, signature_header: str, raw_body: bytes) -> bool:
    # signature_header looks like "t=1700000000,v1=<hex>"
    parts = dict(p.split("=", 1) for p in signature_header.split(","))
    t, v1 = parts["t"], parts["v1"]
    mac = hmac.new(secret.encode(), f"{t}.".encode() + raw_body, hashlib.sha256)
    return hmac.compare_digest(mac.hexdigest(), v1)

# Worked example — prints True:
assert verify(
    "whsec_example",
    "t=1700000000,v1=72f7940d13ca8528cf655a04e82dabfe90a531e239241fda9bdff72980d33a4a",
    b'{"id":"evt_123","type":"email.received","data":{"id":"em_1"}}',
)
```

## Verify the signature (Node)

Hash the **raw request body bytes**, not a re-serialized object — JSON
key order and whitespace would differ and the HMAC would not match.

```js
import crypto from "node:crypto";

function verify(secret, signatureHeader, rawBody) {
  // rawBody is a Buffer of the exact bytes Hail sent.
  const parts = Object.fromEntries(
    signatureHeader.split(",").map((p) => p.split(/=(.*)/s).slice(0, 2)),
  );
  const mac = crypto
    .createHmac("sha256", secret)
    .update(`${parts.t}.`)
    .update(rawBody)
    .digest("hex");
  return crypto.timingSafeEqual(Buffer.from(mac), Buffer.from(parts.v1));
}
```

In Express, capture the raw bytes with `express.raw({ type: "application/json" })`.
In Next.js route handlers, use `await req.text()` / `req.arrayBuffer()` and pass
those bytes, not `await req.json()`.

## Headers

Every delivery carries these headers (refer to
[`core/hailhq/core/webhook_worker.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/webhook_worker.py)):

| Header                | Meaning                                                                                                                     |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `X-Hail-Signature`    | `t=<unix>,v1=<hex hmac_sha256>` — verify this.                                                                              |
| `X-Hail-Event`        | Event type, for example `email.received`.                                                                                   |
| `X-Hail-Delivery`     | Unique delivery id (stable across retries; use to dedupe).                                                                  |
| `X-Hail-Subscription` | Always present — identifies the subscription that produced this delivery.                                                   |
| `X-Hail-Email-Domain` | Informational: the source email domain for inbound events (when known). Branch on this to route per-domain in your handler. |

## Event types

Read supported event names, labels, and channels from the public catalog:

```bash
curl https://api.hail.so/v1/webhooks/event-types
```

The response contains an `items` array. Use each item's `type` in a subscription's `event_types`.
This endpoint needs no API key. Creating or changing a subscription still requires authentication.

The catalog and subscription validation share the `WebhookEventType` registry in
[`core/hailhq/core/webhook_events.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/webhook_events.py):

- **`email.received`** — a message arrived, and Hail accepted it.
- **`email.received.suppressed`** — a message arrived, but Hail held back
  fan-out/forwarding. `data.reason` is one of `forward_loop`, `forward_rate_limit`,
  `inbound_rate_limit`, `insufficient_funds`. One event fires _per reason_
  (a single message can produce more than one suppressed event).
- **`email.delivered`** — SES accepted the message for delivery.
- **`email.delivery_delayed`** — SES reports a transient delay.
- **`email.bounced`** — the recipient mail server rejected the message (permanent or soft bounce).
- **`email.complained`** — the recipient marked the message as spam.
- **`email.opened`** — the recipient opened the message (image tracked, approximate).
  Only fires for emails sent with an HTML body; plain-text-only emails are never
  tracked for opens.
- **`email.clicked`** — the recipient clicked a tracked link. Only fires for
  emails sent with an HTML body; plain-text-only emails are never tracked for
  clicks.
- **`email.send_failed`** — an outbound email failed to send.
- **`sms.received`** — an inbound SMS arrived, and Hail accepted it. Hail delivers
  it through the same signed, retried webhook worker as the email events
  (`X-Hail-Signature`, `X-Hail-Event`, `X-Hail-Delivery`). Hail omits the
  `X-Hail-Email-Domain` header.
- **`sms.delivered`** — the carrier confirmed delivery (requires Twilio delivery receipts).
- **`sms.undelivered`** — the carrier reported the message was not delivered.
- **`sms.failed`** — the send failed (transport error or carrier rejection).

**Call lifecycle** — `received` (inbound only), `answered`, `completed`,
`failed`, `busy`, `no_answer`, `transferred` (there is no `canceled` event; it has no data source):

- **`call.received`** — an inbound call reached one of your numbers and an
  agent is about to answer ([Agents](agents.md)). Fired before the answer.
- **`call.answered`** — the other side is on the line (call entered in-progress).
- **`call.completed`** — the call ended normally.
- **`call.failed`** — the call failed (setup error, trunk/media failure,
  force-closed, or an inbound call Hail refused: `end_reason` `no_agent` or
  `insufficient_funds`).
- **`call.busy`** — the callee was busy or rejected the call.
- **`call.no_answer`** — the callee did not answer.
- **`call.transferred`** — the agent handed the live call to a person
  ([Agents](agents.md#hand-over-to-a-person)). Sent once, when that person
  answers. `data` is the call object plus `transfer`:
  `{"contact_id": "…", "contact_name": "Sam"}`. `contact_id` is the id
  `GET /contacts` returns: a contact's uuid, or `member:<user id>` for a
  team member. Never a phone number.

Every `call.*` payload's `data` carries `id`, `status`, `direction`
(`inbound` or `outbound`), `from`, `to`, `agent_id` (null for calls placed
with an inline prompt) and, once terminal, `end_reason`
([`call_event_data`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/webhook_fanout.py)):

Failed call responses and `call.failed` events also carry `failure_message`.
For a confirmed geographic-permission block, it explains that calling the
destination is disabled. Raw carrier error text is never returned.

```json
{
  "type": "call.received",
  "data": {
    "id": "0d9c…",
    "status": "ringing",
    "direction": "inbound",
    "from": "+33612345678",
    "to": "+14155550100",
    "agent_id": "a1b2…"
  }
}
```

## Payload

Hail wraps every event in this envelope (`build_event_payload` in
[`webhooks.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/webhooks.py) assembles it). The `data`
shape comes from the event type. Inbound events use [`build_event_data`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/webhook_fanout.py) for `email.received*`. Delivery events use [`build_delivery_event_data`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/email_delivery_events.py) for the lifecycle events.

**Inbound example** (`email.received`):

```json
{
  "id": "9f2c…",
  "type": "email.received",
  "api_version": "2026-06-06",
  "created_at": "2026-06-14T12:00:00+00:00",
  "organization_id": "org-uuid",
  "data": {
    "id": "em-uuid",
    "direction": "inbound",
    "from_address": "sender@example.com",
    "to_addresses": ["agent@yourorg.hail.so"],
    "subject": "Re: invoice",
    "message_id": "<abc@example.com>",
    "in_reply_to": "<def@yourorg.hail.so>",
    "spam_verdict": "PASS",
    "virus_verdict": "PASS",
    "spf_verdict": "PASS",
    "dkim_verdict": "PASS",
    "dmarc_verdict": "PASS",
    "raw_url": "https://api.hail.so/v1/emails/em-uuid/raw",
    "attachments": [
      {
        "id": "att-uuid",
        "filename": "invoice.pdf",
        "content_type": "application/pdf",
        "size_bytes": 12345,
        "url": "https://…"
      }
    ]
  }
}
```

**Inbound example** (`sms.received`):

```json
{
  "id": "7a1b…",
  "type": "sms.received",
  "api_version": "2026-06-06",
  "created_at": "2026-07-10T12:00:00+00:00",
  "organization_id": "org-uuid",
  "data": {
    "id": "sms-uuid",
    "from": "+14155551234",
    "to": "+14155559999",
    "body": "hello back"
  }
}
```

`raw_url` and each attachment `url` are Hail API endpoints that 302-redirect to a
presigned S3 URL on access. `email.received.suppressed` carries a trimmed `data`
(`id`, `direction`, `from_address`, `to_addresses`, `subject`, `message_id`,
`reason`) — no verdicts, `raw_url`, or attachments.

**Outbound delivery example** (`email.bounced`):

```json
{
  "id": "9f2c…",
  "type": "email.bounced",
  "api_version": "2026-06-06",
  "created_at": "2026-07-02T12:00:05+00:00",
  "organization_id": "org-uuid",
  "data": {
    "id": "em-uuid",
    "kind": "bounced",
    "occurred_at": "2026-07-01T12:00:05+00:00",
    "from_address": "noreply@acme.com",
    "to_addresses": ["bob@example.com"],
    "subject": "Welcome",
    "detail": {
      "hard": true,
      "bounce_type": "Permanent",
      "bounce_sub_type": "General",
      "recipients": ["bob@example.com"],
      "diagnostic_code": "smtp; 550 5.1.1 user unknown"
    }
  }
}
```

The `detail` field varies by event type. `bounced` and `complained` carry SES metadata. `delivered` and `delivery_delayed` carry SES status details. `opened` and `clicked` carry `ip_address` and `user_agent`; `clicked` adds `link` (the event time is the sibling `occurred_at` field). Both only occur for emails sent with an HTML body: the tracking pixel and link rewriting live in the HTML part, so plain-text-only emails never produce them. For `bounced`, the provider-neutral `hard` flag distinguishes hard bounces from soft bounces. Only hard bounces move the email to `status=bounced` and count toward `bounced_hard` in `GET /emails/stats`.

## Retries

Hail retries a delivery that does not get a `2xx` on this fixed ladder
(`RETRY_SCHEDULE_SECONDS` in [`webhooks.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/webhooks.py)):

```
0s → 30s → 2m → 10m → 1h → 6h → 24h
```

After the 7th attempt fails, Hail marks the delivery **dead**. When an org-wide
subscription accrues **50 consecutive dead** deliveries, it auto-disables. To
re-enable it, set its status back to `active`. Replay a single delivery from the
console or through the API:

```bash
curl -X POST "$HAIL_API_URL/v1/webhooks/<subscription-id>/deliveries/<delivery-id>/redeliver" \
  -H "Authorization: Bearer $HAIL_API_KEY"
```

## Subscribe

Create a subscription. The response returns the signing secret **once**, at create:

```bash
POST /webhooks   {"target_url": "https://example.com/hooks/hail",
                  "event_types": ["email.received", "email.received.suppressed"]}
```

The `event_types` enum and request/response schemas are in
[`openapi/openapi.yaml`](https://github.com/hail-hq/hail/blob/main/openapi/openapi.yaml) (`WebhookSubscriptionCreate`).
The CLI has no `webhooks` command group — for the full endpoint list, refer to
[the CLI reference](./cli.md#webhooks).

---

<!-- source: https://hail.so/docs/byo-llm -->

# Bring your own LLM

Point a Hail voice call at your own OpenAI-compatible endpoint. Your agent
becomes the brain of the call: Hail handles telephony, speech-to-text,
text-to-speech, turn detection, and tools, and asks your endpoint what to
say on every turn.

Two ways to do it. Per call, by passing an `llm` block to `POST /calls` —
different brains for different calls, or a different tenant's endpoint each
time. Or standing, by saving an endpoint once in the console — every call
your organization places uses it.

## Run one in five minutes

This is a complete endpoint. It has no dependency on Hail or on any model
SDK, and it runs as written.

```python
# byo_endpoint.py
import json, os, time, uuid

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse, StreamingResponse

app = FastAPI()


def generate_reply(messages: list[dict]) -> str:
    """Replace this with your agent. Return plain text.

    `messages` is the OpenAI-format history for the call so far: Hail's
    composed instructions as the leading system message, then alternating
    user and assistant turns. This stub echoes the last user turn, which is
    enough to prove the wiring without involving a real model.
    """
    last_user = next(
        (m["content"] for m in reversed(messages) if m.get("role") == "user"), ""
    )
    return f"You said: {last_user}"


@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
    # Hail sends the api_key you configured as a bearer token. Check it —
    # your endpoint is reachable from the public internet.
    expected = os.environ.get("BYO_LLM_SECRET", "")
    if not expected or request.headers.get("authorization") != f"Bearer {expected}":
        return JSONResponse(status_code=401, content={"error": "invalid api key"})

    body = await request.json()
    model = body.get("model", "byo-example")
    reply = generate_reply(body.get("messages", []))

    chunk_id = f"chatcmpl-{uuid.uuid4().hex[:12]}"
    created = int(time.time())

    def sse(**fields) -> str:
        frame = {
            "id": chunk_id,
            "object": "chat.completion.chunk",
            "created": created,
            "model": model,
        }
        frame.update(fields)
        return f"data: {json.dumps(frame)}\n\n"

    async def stream():
        yield sse(
            choices=[
                {
                    "index": 0,
                    "delta": {"role": "assistant", "content": reply},
                    "finish_reason": None,
                }
            ]
        )
        yield sse(choices=[{"index": 0, "delta": {}, "finish_reason": "stop"}])
        yield "data: [DONE]\n\n"

    return StreamingResponse(stream(), media_type="text/event-stream")
```

Start it:

```bash
pip install fastapi uvicorn
export BYO_LLM_SECRET=demo-secret
uvicorn byo_endpoint:app --port 8000
```

Confirm it answers before you involve Hail:

```bash
curl -sN http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer demo-secret" \
  -H "Content-Type: application/json" \
  -d '{"model":"demo","messages":[{"role":"user","content":"What is the weather today?"}]}'
```

```
data: {"id": "chatcmpl-f5cd09e477fc", "object": "chat.completion.chunk", "created": 1786038174, "model": "demo", "choices": [{"index": 0, "delta": {"role": "assistant", "content": "You said: What is the weather today?"}, "finish_reason": null}]}

data: {"id": "chatcmpl-f5cd09e477fc", "object": "chat.completion.chunk", "created": 1786038174, "model": "demo", "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}]}

data: [DONE]
```

Hail requires a public `https` URL, so expose the port through a tunnel —
`cloudflared tunnel --url http://localhost:8000` or `ngrok http 8000` — and
use the `https` address it prints. Then place the call:

```bash
hail call +15551234567 \
  --llm-url https://your-tunnel.example.com/v1 \
  --llm-key demo-secret \
  --llm-model demo \
  --recipient-consent
```

Answer the phone, say something, and the agent repeats it back. Your
endpoint is now driving a live phone call.

Add `--prompt "…"` alongside the `--llm-*` flags to send your task prompt
too: Hail composes its voice preamble plus your prompt into the leading
`system` message your endpoint receives. Prompt and endpoint are
independent choices — at least one is required, both together is fine.

## The contract

Hail calls your endpoint once per voice turn. The request below is a real
capture, taken by pointing Hail's own client at a recording server.

```
POST /v1/chat/completions HTTP/1.1
authorization: Bearer <api_key>
content-type: application/json
accept: application/json
user-agent: LiveKit Agents/1.6.6 (python 3.13.12)
```

```json
{
  "messages": [
    {
      "role": "system",
      "content": "<Hail's voice preamble, then your system_prompt>"
    },
    {
      "role": "user",
      "content": "What's the weather in Paris, and can you end the call after?"
    }
  ],
  "model": "my-model",
  "stream": true,
  "stream_options": { "include_usage": true },
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
          "type": "object",
          "properties": {
            "city": { "type": "string", "description": "City name" }
          },
          "required": ["city"]
        }
      }
    }
  ]
}
```

Notes that follow from the capture, not from the spec:

- **The path is `{base_url}/chat/completions`.** A `base_url` of
  `https://you.example.com/v1` becomes
  `https://you.example.com/v1/chat/completions`. A trailing slash on
  `base_url` is safe — it does not produce a double slash.
- **`stream` is always `true`**, and `stream_options.include_usage` is
  always sent. Ignore the latter if you do not report usage.
- **`tools` carries Hail's agent tools** — `end_call`, `send_dtmf`,
  `send_sms`, and whatever else the call enabled — in OpenAI function
  format. Return a `tool_calls` delta to invoke one. Hail executes it and
  sends the result back on the very next request, as a `role: "tool"`
  message keyed by `tool_call_id`:

  ```json
  {
    "role": "tool",
    "tool_call_id": "call_1",
    "content": "The weather in Paris is sunny."
  }
  ```

  Restrict which tools a call may use with `--tools` / the `tools` field.

- **`tool_choice` is absent on the first turn**, and appears as
  `"auto"` on the request that follows a tool result. Treat it as optional
  and do not require it.
- Hail's client sends `x-stainless-*` headers. Do not reject unrecognized
  headers.

### What you must return

`text/event-stream`, one `data:` line per frame, each frame a JSON
`chat.completion.chunk`. The minimum that works is a single content chunk:

```
data: {"id":"chatcmpl-1","object":"chat.completion.chunk","created":1730000000,"model":"m","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello"},"finish_reason":null}]}
```

`data: [DONE]` and the `finish_reason: "stop"` frame are both **optional** —
closing the response body ends the turn cleanly. Send them anyway: they are
the OpenAI-compatible contract, and a future client may be stricter. Emit
many small content chunks rather than one large one; each chunk is spoken as
it arrives, so streaming token-by-token is what makes the agent sound
responsive instead of stalled.

## Rules and failure modes

**URLs must be `https` and publicly resolvable.** Hail checks the scheme
when you submit the call, then resolves the host and rejects private,
loopback, and link-local addresses — twice: once in the API, once again in
the voicebot at call time. A URL that resolves to a private address by the
time the call runs ends the call with `end_reason=provider_key_error`.

**There is no fallback.** You chose this brain deliberately, so Hail does
not silently substitute its own models when your endpoint fails. A standing
console endpoint can opt into fallback; a per-call endpoint cannot.

**Three consecutive failures end a per-call endpoint's call.** After three
non-recoverable errors in a row with no successful turn between them, Hail
speaks a short goodbye and hangs up with `end_reason=llm_endpoint_failed`,
rather than letting a dead endpoint burn the caller's minutes. A turn that
succeeds resets the count. An interruption is not a failure — barge-in
cancels the in-flight request and never counts against you. This give-up
is armed for per-call `llm` blocks only: a standing endpoint with fallback
off that keeps failing ends the call as `end_reason=agent_error` instead.

**A non-200 costs about four seconds.** Hail retries the turn three times
before giving up on it. The caller hears silence for that whole window, so
return a fast, valid reply on your own error paths rather than a 500.

**Your key is encrypted at rest.** The `api_key` is Fernet-encrypted before
it is written to call metadata, and decrypted only inside the voicebot.
Reads never return it.

## Per call

Every surface takes the same three fields.

```bash
curl -X POST https://api.hail.so/v1/calls \
  -H "Authorization: Bearer $HAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "to": "+15551234567",
    "llm": {
      "base_url": "https://you.example.com/v1",
      "api_key": "demo-secret",
      "model": "demo"
    },
    "recipient_consent": true
  }'
```

```bash
hail call +15551234567 \
  --llm-url https://you.example.com/v1 \
  --llm-key demo-secret \
  --llm-model demo \
  --recipient-consent
```

```python
from hail import Client, LLMConfig

async with Client(api_key="sk-...") as client:
    call = await client.calls.create(
        to="+15551234567",
        llm=LLMConfig(
            base_url="https://you.example.com/v1",
            api_key="demo-secret",
            model="demo",
        ),
        recipient_consent=True,
    )
```

The MCP `place_call` tool takes the same `llm` object.

## Standing, for every call

Save an endpoint once in the console instead of sending it per call:
**Console → Calls → Providers → LLM → Configure**. Choose the
`OpenAI-compatible` provider, enter the base URL, model, and key, and use
**Test** to validate the key against the endpoint before you save it. Keys
are write-only — after saving, the console shows only the last four
characters and the date it was set.

The same page configures speech-to-text and text-to-speech. Transport stays
Hail's.

Standing config also offers **fallback**: when enabled, a failure of your
endpoint falls through to Hail's own models rather than ending the call.
Off by default, because silently billing Hail's models would defeat the
point of bringing your own.

### Configure it as code

The console is one client of `/providers`; the CLI and the SDK are the
others. The organization always comes from your API key — it is never a
path or body field, so a key can only reach its own config.

```bash
# Save the endpoint and activate it. '--key -' reads the key from stdin,
# so it never lands in your shell history.
printf '%s' "$MY_PROVIDER_KEY" | hail providers set llm \
  --provider openai-compatible \
  --base-url https://you.example.com/v1 \
  --model demo \
  --key -

hail providers list                     # all layers; keys show as …ABCD
hail providers test llm                 # probe the stored key, live
hail providers activate llm --provider anthropic
hail providers delete llm anthropic
```

```python
from hail import Client

async with Client(api_key="sk-...") as client:
    await client.providers.set(
        "llm",
        provider="openai-compatible",
        api_key="demo-secret",
        params={"base_url": "https://you.example.com/v1", "model": "demo"},
        fallback_enabled=False,
    )
    result = await client.providers.test("llm")
    print(result.status)  # "valid" | "invalid"
```

`tts` and `stt` take the same five verbs — `params` is what differs per
layer (`voice_id`/`model` for `tts`, `model` for `stt`). The canonical
per-layer schemas are `LLMParams` / `TTSParams` / `STTParams` in
[`core/hailhq/core/provider_config.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/provider_config.py);
the routes are `/providers` in
[`openapi/openapi.yaml`](https://github.com/hail-hq/hail/blob/main/openapi/openapi.yaml).

Keys are write-only on every one of these paths: `GET /providers` returns
`key_last4` and `key_set_at` and nothing else key-shaped. To rotate a key,
`set` the layer again with the new one; to edit the model or base URL
without resending the key, omit `--key` (SDK: omit `api_key`).

`set` is a partial write: it changes only the fields you send and leaves
every other saved field alone. So this swaps the model and nothing else —

```bash
hail providers set tts --provider cartesia --model sonic-3
```

— the row's `voice_id` and its fallback setting survive untouched. Send
`--voice-id` (SDK: `params={"voice_id": ...}`) when you do want to change
it, and `--fallback` / `--fallback=false` (SDK: `fallback_enabled=True` /
`False`) when you want to move the fallback flag; omit them to leave both
as they are. What the server validates is the merged result, so a partial
write can never leave an invalid config behind — it 422s instead.

Config is per provider: rows are keyed by `(organization, layer, provider)`,
so pointing a layer at a different `--provider` starts a fresh row rather
than inheriting the old provider's params. `fallback_enabled` is `false` on
a new row.

> The console writes the same rows through `/internal`, which keeps
> full-replace semantics — that is what lets a console user clear a field by
> emptying it. Merging is specific to the public `/providers` routes the CLI
> and SDK use.

## Which brain runs a call

| Mode                      | Source                           | Fallback                    |
| ------------------------- | -------------------------------- | --------------------------- |
| **B — per-call endpoint** | `llm` on `POST /calls`           | none                        |
| **C — standing endpoint** | `/providers` (console, CLI, SDK) | opt-in                      |
| **A — Hail's models**     | no `llm` block                   | Anthropic → OpenAI → Google |

Precedence is B, then C, then A. A per-call `llm` block overrides your
standing config for that one call.

The resolution logic is in
[`voicebot/hailhq/voicebot/pipeline.py`](https://github.com/hail-hq/hail/blob/main/voicebot/hailhq/voicebot/pipeline.py)
(`build_llm`), and the request schema is
[`LLMConfig`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/schemas.py).

---

<!-- source: https://hail.so/docs/react-email -->

# react-email

Hail doesn't render React or store templates — you render a
[react-email](https://react.email) component to a string and send that
string. The body goes out exactly as given: a full
`<!DOCTYPE html>…</html>` document passes through unchanged.

## Render the template

```bash
npm install react-email react react-dom -E
```

```tsx
// Welcome.tsx
import * as React from "react";
import { Html, Body, Text, Button } from "react-email";

export function Welcome({ name }: { name: string }) {
  return (
    <Html lang="en">
      <Body>
        <Text>Hi {name}, welcome aboard.</Text>
        <Button href="https://example.com/start">Get started</Button>
      </Body>
    </Html>
  );
}
```

`render()` is async — it returns a `Promise<string>`. Call it twice: once for
the HTML body, once with `{ plainText: true }` for the text body.

## Send it (Node)

```tsx
// send.tsx
import * as React from "react";
import { render } from "react-email";
import { Welcome } from "./Welcome";

const html = await render(<Welcome name="Ada" />);
const text = await render(<Welcome name="Ada" />, { plainText: true });

const res = await fetch(`${process.env.HAIL_API_URL}/v1/emails`, {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.HAIL_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    to: ["ada@example.com"],
    from: "onboarding@yourdomain.com",
    subject: "Welcome aboard",
    body_html: html,
    body_text: text,
    recipient_consent: true,
  }),
});
```

Run it with [`tsx`](https://github.com/privatenumber/tsx) (`npm install -D tsx`; needs `"type": "module"` in `package.json`):

```bash
npx tsx send.tsx
```

## Send it (Python SDK)

There is no JS/TS SDK. Render in Node — at build time, or in a small script —
write the output to a file, and send the string with
[`hail-sdk`](https://pypi.org/project/hail-sdk/):

```python
from pathlib import Path
from hail import Client

html = Path("welcome.html").read_text()

async with Client() as client:  # reads HAIL_API_KEY / HAIL_API_URL from env
    await client.emails.create(
        to=["ada@example.com"],
        from_="onboarding@yourdomain.com",
        subject="Welcome aboard",
        body_html=html,
        recipient_consent=True,
    )
```

## Notes

- Field names are `body_html` / `body_text`. `EmailCreate` in
  [`openapi/openapi.yaml`](https://github.com/hail-hq/hail/blob/main/openapi/openapi.yaml)
  forbids extra fields — `html` / `text` return `422`.
- Open and click tracking only work with an HTML body. A plain-text-only
  email still sends, but opens and clicks are never tracked.
- Hail adds nothing to the body — no footer, no tracking notice. Disclose
  AI use yourself where the law requires it.
- An email with attachments goes out as raw MIME
  ([`core/hailhq/core/providers/email/ses.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/providers/email/ses.py)),
  which adds one trailing line break to each text part — no visible effect.

---

<!-- source: https://hail.so/docs/cli -->

# CLI reference

`hail` is the Go CLI. It codegens its client from [`openapi/openapi.yaml`](../../openapi/openapi.yaml) — that spec is the canonical contract. This page is a brief summary of each command group. Run `hail <cmd> --help` for the full, authoritative flag list.

## Install

Homebrew is the recommended install path on macOS and Linux:

```bash
brew install hail-hq/tap/hail
hail version
```

The fully qualified formula makes Homebrew add the `hail-hq/tap` tap
automatically. Upgrade or remove the CLI with:

```bash
brew upgrade hail-hq/tap/hail
brew uninstall hail
```

Alternatively, download the `darwin` or `linux` archive for your architecture
from [GitHub Releases](https://github.com/hail-hq/hail/releases), extract it,
and move the `hail` binary to a directory on your `PATH`. Release builds support
Intel (`amd64`) and ARM64.

After installation, choose the authentication mode:

```bash
# Hail Cloud: opens the device authorization flow and saves credentials.
hail login

# Self-host: point the CLI at your deployment; do not run hail login.
export HAIL_API_URL=http://localhost:8080
export HAIL_API_KEY='<the HAIL_API_KEY value from your server .env>'
```

Global flags (any command): `--api-url`, `--api-key`, `--json`. Auth resolves `--api-key > $HAIL_API_KEY > ~/.hail/credentials.json` (run `hail login`).

## Calls

Outbound phone calls only.

```bash
# Place an outbound call (consent attestation required)
hail call +15551234567 --prompt "You are a scheduling assistant." --recipient-consent

hail call status <id>     # one call (full UUID or 4+ char prefix)
hail call list
hail call tail <id>       # follow the event stream for one call
```

`hail call` flags: `--prompt` (mode A) and/or `--llm-url`/`--llm-key`/`--llm-model` (mode B) — at least one is required, and passing both runs your prompt on your own endpoint — `--from`, `--first-message`, `--ai-disclosure`, `--tools`, `--idempotency-key`, plus the consent flags.

To point a call at your own OpenAI-compatible endpoint, see [Bring your own LLM](./byo-llm.md) — it has the wire contract and an endpoint you can run in five minutes.

Language support:

- `--language` — one of 39 lowercase ISO 639-1 codes (e.g. `en`, `da`, `hi`). Hail auto-routes STT and turn detection per language (see [docs/languages.md](../../docs/languages.md) for the full table). STT provider selection is console-BYO-only — there is no per-call flag to pin one.

## SMS

```bash
# Send an outbound SMS (consent attestation required)
hail sms +15551234567 --body "Hello" --recipient-consent

hail sms status <id>
hail sms list --status delivered

hail sms suppressions list     # the opt-out list
hail sms sender-id get         # the org's custom sender ID
```

`hail sms` flags: `--body` (required), `--from`, `--idempotency-key`, plus the consent flags. `list` takes `--status` (`queued|sent|delivered|failed|undelivered|received`), `--to`, `--limit`, `--cursor`.

## Numbers

Dedicated phone numbers for voice and SMS.

```bash
hail numbers acquire --country US --type local          # buys the cheapest ready live offer
hail numbers acquire --country US --quote-id <quote-id>  # buys one exact offer from POST /numbers/quotes
hail numbers quotes --country GB --type mobile           # lists live offers and the verifications that can buy each one
hail numbers acquire --country GB --quote-id <quote-id> --verification-id <id>  # buys with one approved verification
hail numbers list
hail numbers get <id>
hail numbers enable-sms <id>   # repair SMS setup; Hail does this itself when a number becomes active
hail numbers register-sms --subject-type business --field legal_name="Acme Inc" ...  # US SMS registration
```

`acquire` requests live quotes ([`POST /numbers/quotes`](../../openapi/openapi.yaml)), then buys the cheapest ready offer (monthly plus setup) with its `quote_id`. Flags: `--country` (required), `--type` (`local|mobile|toll_free|national`), `--area-code` (area code or first digits as dialed in the country, e.g. `020`, `480`), `--city` (e.g. `London`), `--region` (optional with `--city`: US/CA state code, e.g. `AZ`), `--quote-id` (skip the quote and buy that offer), `--voice-only`, `--sms-only` (default: voice and SMS), `--expected-total-cents` (refuse with 409 if the quoted monthly plus setup differs), `--subject-type` (`person` or `business`: which approved verification buys a number that needs one; needed only when you are verified as both), `--verification-id` (which approved verification buys the number; needed only when more than one is approved), `--idempotency-key`.

`quotes` takes the same `--country`, `--type`, `--area-code`, `--city`, `--region`, `--voice-only` and `--sms-only` flags and buys nothing. Each offer prints its number, type, monthly and setup price, readiness and `quote_id`. Under it are the verifications that can buy it: id, name (or `(no name)`), person or company, and state. Pass an approved one's id as `--verification-id`. `--json` prints the raw [`POST /numbers/quotes`](../../openapi/openapi.yaml) response.

`register-sms` files a US SMS registration (`purpose=sms` on [`POST /verifications`](../../openapi/openapi.yaml)). `--subject-type` (`person` or `business`) is required. Hail picks the carrier from your US numbers. Pass `--number-id <id>` when your US numbers are with more than one carrier.

## Email

```bash
# List inbound mail (cursor-paginated)
hail email list --direction inbound

# List outbound mail, only failures
hail email list --direction outbound --status failed

# Fetch one email (full UUID or 4+ char prefix)
hail email get 1a2b

# Send (subject + at least one body flag required)
hail email send --to alice@example.com --subject "Hi" --body "Hello"
```

`hail email list` flags: `--direction` (`inbound|outbound`), `--status` (`queued|sent|failed|bounced|complained|received`), `--limit` (default 50), `--cursor`, `--all` (walk every page). Alias: `hail email ls`.

`hail email get <id>` prints headers, auth verdicts (SPF/DKIM/DMARC/spam/virus), the raw-MIME URL, and attachment metadata for inbound rows.

`hail email send` flags: `--to` (repeatable / comma-separated), `--cc`, `--bcc`, `--from`, `--from-name`, `--reply-to`, `--subject` (required), `--body`, `--body-html`, `--body-file`, `--body-html-file` (`-` reads stdin), `--idempotency-key`.

More email subcommands: `tail <id>`, `raw <id>`, `events <id>`, `stats`, `attachment`, `attachment-upload`. Run `hail email --help` for the list.

### Email domains

The identities that send and receive email. There are two kinds: `hail_mail` (operator-managed parent domain, verified immediately) and `custom` (your DNS). For `custom`, the register call returns DKIM CNAMEs. Publish them, then run `verify`.

```bash
# Register a hail-mail identity (uses server prefix defaults)
hail email domain register --kind hail_mail

# Register a custom domain (prints DKIM CNAMEs to publish)
hail email domain register --kind custom --domain acme.com

hail email domain list
hail email domain get <id>
hail email domain verify <id>      # re-poll the provider for DKIM status
hail email domain dns-check <id>   # detect the DNS host, confirm records + DMARC are published
hail email domain delete <id>      # also drops the SES identity for custom rows
```

`register` flags: `--kind` (`hail_mail|custom`, required), `--domain` (required for `custom`), `--local-prefix-user`, `--local-prefix-org` (for `hail_mail`), `--idempotency-key`. `list` takes `--limit` / `--cursor`. Aliases: `list`→`ls`, `delete`→`rm`.

`dns-check <id>` is read-only: it detects the tenant's DNS host and checks whether DKIM/MX/SPF and DMARC are live in public DNS. It never changes `verification_status` (run `verify` for that). `kind='hail_mail'` rows do no lookups, so the host line reads `not recognised`, and there's no table or DMARC line — that zone is Hail's, not the customer's. Otherwise, output is a `DNS host: <name> — <url>` line, a TYPE/NAME/VALUE/SEEN table (`SEEN` = seen in public DNS, not verified; `?` = that record's lookup failed, so it could not be checked), and a `DMARC: found` or `DMARC: not found — add TXT <name> "<value>"` line. If the zone or DMARC lookup fails, or the check times out, it prints one line — `DNS check could not be completed. Try again in a minute.` — instead of guessing at missing records.

`hail email domain list` closes with the address a send without `--from` goes out as. Own two or more verified identities and there is no default: the line reads `No default sender — a send without --from is rejected.`, and a `--from`-less `hail email send` fails with a 422 listing them.

> Renamed: `hail sender-domain ...` is now `hail email domain ...`. The old name no longer exists.

## Whoami

```bash
hail whoami          # human-readable
hail whoami --json   # for scripts
```

Prints the organization and the user the API key belongs to, plus how the
request authenticated (`apikey`, `jwt`, or `shared`). A shared operator key
carries no user, so the email and name come back empty. Use the email as
`hail email send --reply-to` so replies reach the person rather than the
sending domain.

## Contacts

The org's contact directory (members + manual contacts).

```bash
hail contacts list --q alice
hail contacts create "Alice" --phone +15551234567
hail contacts update <id> --email alice@example.com
hail contacts delete <id>
hail contacts set-phone me --phone +15551234567
hail contacts clear-phone me
```

`create` requires one of `--phone` / `--email`. `list` takes `--q`, `--limit`, `--cursor`, `--all`.

## Providers

Standing BYO provider config for the org's `llm`, `tts`, and `stt` layers — the same rows the console's Providers page writes. Applies to every call unless that call carries its own `--llm-*` block. See [byo-llm.md](./byo-llm.md).

```bash
# Save a provider and make it active ('--key -' reads the key from stdin,
# keeping it out of shell history)
printf '%s' "$MY_KEY" | hail providers set llm \
  --provider openai-compatible --base-url https://you.example.com/v1 \
  --model demo --key -

hail providers list                          # all layers; keys show as …ABCD
hail providers test llm                      # probe the stored key, live
hail providers activate llm --provider anthropic
hail providers delete llm anthropic
```

`set` requires only `--provider`. Params are per layer: `--model` (required by the `llm` layer, optional for `tts`/`stt`), `--base-url` (required by the `openai-compatible` LLM provider, rejected by the others), `--voice-id` (`tts` only). `--fallback` lets a failure of your provider fall through to Hail's own keys; `--fallback=false` turns it back off.

`set` is a partial write — it changes only the flags you pass and preserves everything else on the row, so `hail providers set tts --provider cartesia --model sonic-3` keeps the saved `voice_id` and the saved fallback setting. Omitting `--key` likewise keeps the stored key. Config is per provider, so a different `--provider` for the same layer starts a fresh row instead of inheriting the previous one's params. `test` takes `--provider` to probe a saved-but-inactive provider instead of the active one. Aliases: `list`→`ls`, `delete`→`rm`.

Keys are write-only: no command can print one back, only the last four characters and when it was set. The org comes from your API key — it is never an argument.

## Events

```bash
hail tail                      # stream events from across the org
hail tail --id call:1a2b       # narrow to one resource
```

`hail tail` flags: `--id`, `--kind`, `--interval`, `--from-start`, `--no-follow`.

## Webhooks

Org-wide outbound subscriptions. Each subscription fires an HMAC-signed POST for each matching event. Hail retries failed deliveries on a fixed ladder. Refer to [setup/webhooks.md](./webhooks.md) for the payload shape, the event-type list, and signature verification.

The CLI has no `webhooks` command group. Manage subscriptions through the HTTP API:

```bash
# Register a subscription (the response shows the signing secret ONCE — store it)
curl -X POST "$HAIL_API_URL/v1/webhooks" \
  -H "Authorization: Bearer $HAIL_API_KEY" \
  -d '{"target_url":"https://example.com/hooks/hail","event_types":["email.received","sms.received"]}'

# List subscriptions
curl "$HAIL_API_URL/v1/webhooks" -H "Authorization: Bearer $HAIL_API_KEY"

# Delivery attempts for one subscription
curl "$HAIL_API_URL/v1/webhooks/<sub-id>/deliveries" -H "Authorization: Bearer $HAIL_API_KEY"

# Replay one delivery
curl -X POST "$HAIL_API_URL/v1/webhooks/<sub-id>/deliveries/<delivery-id>/redeliver" \
  -H "Authorization: Bearer $HAIL_API_KEY"
```

Other endpoints: `PATCH /webhooks/{id}` (update URL, events, or status), `DELETE /webhooks/{id}`, and `POST /webhooks/{id}/rotate-secret`. Event types cover email, SMS, and call events — the canonical list is `WebhookEventType` in [`core/hailhq/core/schemas.py`](../../core/hailhq/core/schemas.py).

## Auth and utilities

```bash
hail login             # browser auth; saves an API key to ~/.hail/credentials.json
hail auth token        # print the bare API key (for scripting)
hail auth logout       # remove the local credentials file
hail mcp endpoint      # print the MCP server's Streamable HTTP URL
hail completion zsh    # shell completion script
hail version
```

---

<!-- source: https://hail.so/docs/architecture -->

# Architecture

Hail v1 is three Python services plus a Go CLI, built around LiveKit Cloud.

```
 AI agent                                     Hail                                LiveKit Cloud         PSTN
(caller)  ─────MCP URL──►  Hail MCP ─HTTP─►  Hail API  ◄────►  SIP+WebRTC  ◄────► carrier ◄────► 📞
                          (HTTP :8081)     (FastAPI :8080)
                                                 │
                                                 └─dispatch──►  Hail voicebot  (LiveKit Agents worker)
                                                                     │
                                                                     ├─ VAD:   Silero
                                                                     ├─ STT:   Deepgram
                                                                     ├─ LLM:   fallback(Anthropic → OpenAI → Gemini)
                                                                     │        or caller-provided endpoint
                                                                     └─ TTS:   Cartesia (→ ElevenLabs fallback)
```

## Services

- **api** (`:8080`, FastAPI) — the REST surface; it accepts `POST /calls` and the other routes. It is the source of truth for OpenAPI.
- **mcp** (`:8081`, Streamable HTTP; legacy SSE during the transition) — the MCP server that wraps the API. Agent clients (Claude.ai, ChatGPT, Claude Code, Cursor) connect to it. Refer to [MCP setup](./mcp.md).
- **voicebot** (LiveKit Agents worker) — registers with LiveKit Cloud. Hail dispatches it into a room for each call.
- **postgres** — call records, phone numbers, API keys.

LiveKit Cloud is external. The `hail` Go CLI is a scriptable tool for humans, not a service.

## Outbound call flow

1. The caller (an agent via MCP, the CLI, or direct HTTP) sends `POST /calls` with `{to, from, first_message?, …llm}` or with `agent_id` ([Agents](./agents.md)).
2. The Hail API creates a LiveKit room and dispatches the voicebot into it.
3. The voicebot joins the room. LiveKit places an outbound SIP call to `to` through the trunk of the carrier that owns `from` (Twilio, Telnyx or DIDWW; [`carrier_routing.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/carrier_routing.py)).
4. On pickup, the voicebot classifies who answered before it speaks (refer to the section below). Then it speaks the AI disclosure and the `first_message` (if set), and runs the STT → LLM → TTS loop. If the call set `ai_disclosure: false`, the voicebot skips the disclosure; Hail audit-logs the opt-out, and the opt-out is the caller's responsibility.
5. On hangup, the voicebot writes the call record and the text transcript to Postgres. Call audio is not recorded or stored.

### Answering machine detection and DTMF

Every outbound call runs LiveKit's AMD ([`voicebot/hailhq/voicebot/amd.py`](https://github.com/hail-hq/hail/blob/main/voicebot/hailhq/voicebot/amd.py)) against the greeting. The house pipeline uses independent greeting transcription and its own classifiers, in order: `ANTHROPIC_BACKUP_MODEL`, `OPENAI_MODEL`, then `GOOGLE_AMD_MODEL`. `ANTHROPIC_MODEL` is used only when none of these can be built. A classifier without its model id or key is skipped. With none left, AMD and the yes/no checks are skipped and the call proceeds normally. Each gets one try with a three-second timeout, and the next one runs after a failure. A check stops after five seconds in total, AMD's included. The failed provider is not retried. Detection releases speech as soon as a person's greeting is classified, after three seconds of no speech, or at the latest 20 seconds after SIP answer (`POST_ANSWER_TIMEOUT_SECONDS`), even when classification fails or speech continues. Two rechecks can follow, one after the other, each capped at five seconds, so up to 10 seconds in total. The pickup recheck runs only when detection timed out after at least one second of speech (`MIN_SPEECH_SECONDS`) and the recheck transcription is set up; the unavailable recheck runs on a `machine-unavailable` verdict. Automatic replies remain blocked until the explicit introduction finishes, so a pickup hello cannot queue a second greeting ahead of it. Words said while replies are blocked are kept in the chat and the transcript, and later replies answer them. A customer's own LLM classifies alone; its AMD check has no five-second total, only the 20-second detection limit. With fallback on, every usable house classifier backs it up; org STT selections are respected. Providers use direct house or customer keys, not LiveKit Inference. On `machine-vm` and `machine-unavailable`, the voicebot hangs up without a word. It never leaves a message. It records `status=no_answer` with `end_reason=voicemail_reached` / `machine_unavailable`. On `human` and `uncertain`, the call proceeds normally. One exception: after a `machine-unavailable` verdict, the voicebot checks the greeting again. If that check is unsure, it hangs up. If the check fails, it keeps listening as a phone menu.

On `machine-ivr`, the voicebot does **not** speak the greeting. A menu cannot hear it, and `session.say` is TTS-only, so the LLM would get no turn in which to press a key. Instead, the agent takes a real LLM turn (`generate_reply`) with the captured menu text and the `send_dtmf` tool. Hail defers the disclosure until the first person speaks. Hail writes the verdict to `call_events` as an `amd_result` row on every call. A detection failure is non-fatal: the call proceeds as if a human answered.

Billing no longer requires a `completed` status. Hail bills a call when `answered_at` is set (the SIP leg went active) **or** when the call completed normally. The first clause bills a machine-answered call and a call that failed mid-conversation. The second clause keeps billing for a completed call whose answer signal never arrived.

The agent can press keypad digits at any point with the `send_dtmf` tool ([`core/hailhq/core/agent_tools/send_dtmf.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/agent_tools/send_dtmf.py)) — not only after an IVR verdict. Thus the agent can navigate phone trees that it reaches mid-call.

## Inbound call flow

1. The carrier sends the INVITE to LiveKit (Twilio origination URI, Telnyx FQDN connection, DIDWW voice IN trunk; [LiveKit Cloud §4](./self-host/livekit-cloud.md#4-inbound-calls)). LiveKit matches it to the carrier's inbound trunk, which lists the numbers Hail registered ([`inbound_routing.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/inbound_routing.py)), creates a room and dispatches the voicebot with the dispatch rule's static metadata `{"direction": "inbound"}`.
2. The voicebot reads the SIP participant's attributes (`sip.trunkPhoneNumber`, `sip.phoneNumber`, `sip.trunkID`) and calls [`inbound_calls.open_inbound_call`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/inbound_calls.py): it resolves the number and its voice agent, checks suspension and funds, writes the `ringing` call and fires `call.received`. A refused call gets a `failed` row (`no_agent`, `insufficient_funds`) and `call.failed`; an unknown number is dropped and the room deleted.
3. The rest is the outbound path without AMD: AI line, `first_message`, the STT → LLM → TTS loop with the agent's tools, `call.answered` on `sip.callStatus = active`, `call.completed` and billing on hangup.

## Inbound texts answered by an agent

A text to a number with `sms_agent_id` set is stored and `sms.received` fires as before; then the row is marked `agent_reply_state = pending` (never for `STOP`/`START`/`HELP`). The text worker in the voicebot service ([`textbot.py`](https://github.com/hail-hq/hail/blob/main/voicebot/hailhq/voicebot/textbot.py)) claims it, builds the chat from the agent's instructions and the last 20 texts of the last 24 hours ([`text_agent.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/text_agent.py)), runs the same LLM chain calls use with the `thread_history` tool for older texts and calls (no history goes in any prompt), and sends the reply through `POST /internal/agent/reply-sms` (thread cap, funds, suppression, velocity, billing, delivery). A text that arrives during an active call (`dialing`, `ringing` or `in_progress`) is marked `skipped` instead and the voice agent gets it; if the voice agent did not get it, it is set back to `pending` when the call ends or the stale-call sweep closes the call.

## LLM modes

**A — system prompt (default).** The caller supplies `system_prompt`. The voicebot uses LiveKit's `FallbackAdapter`, which chains `ANTHROPIC_MODEL` → `ANTHROPIC_BACKUP_MODEL` → `OPENAI_MODEL` → `GOOGLE_MODEL`. Each model gets one try, and a model without its id or key is skipped. See [`pipeline.py`](https://github.com/hail-hq/hail/blob/main/voicebot/hailhq/voicebot/pipeline.py).

**B — BYO endpoint.** The caller supplies `llm: { base_url, api_key, model }`. The voicebot points `openai.LLM` at that endpoint. There is no fallback.

**C — standing BYO endpoint.** The organization saves an endpoint once on the console Providers page. Every call uses it, with opt-in fallback to Hail's models. A per-call mode B block overrides it.

Precedence is B, then C, then A. See [Bring your own LLM](./byo-llm.md) for the wire contract and a runnable endpoint.

## Data

- **Postgres** — call, SMS, and email records; phone numbers; email domains; contacts; webhook subscriptions; API keys.
- **S3** — email content (MIME) and attachments. No call audio.
- **LiveKit Cloud** — transient media (ephemeral).

## SMS

The `SmsProvider` adapter in [`core/hailhq/core/providers/sms/`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/providers/sms) sends SMS through the carrier that owns the sending number (Twilio or Telnyx).

**Outbound.** `POST /sms` with no `from` sends from the org's sender ID (or the platform default `HAIL`) wherever the carrier that sends sender-ID SMS (`sender_id_sms_carrier`, Twilio by default) delivers a dynamic alphanumeric sender ID unchanged — about 110 countries; one table per carrier (Twilio, Telnyx) in [`core/hailhq/core/sender_id_corridors.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/sender_id_corridors.py), generated from each carrier's per-country documentation by `scripts/sender_id_corridors.py`; Australia (+61) always gets `HAIL` when that carrier is Twilio; every other destination, including the US and Canada, uses the org's oldest active SMS-capable number and 422s when there is none (see [`core/hailhq/core/sender_id.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/sender_id.py)). There is no pool fallback — refer to `hail numbers` for number acquisition. Twilio posts delivery-status callbacks. These callbacks move `Sms.status` and fan out the `sms.delivered` / `sms.undelivered` / `sms.failed` webhook events.

**Inbound.** Twilio posts each incoming message to `POST /sms/inbound` (Hail verifies the `X-Twilio-Signature` header); Telnyx posts to `POST /sms/telnyx` (Ed25519 signature). It matches the destination number to an org, stores the message, and fires the `sms.received` webhook event. Messages to unknown or pool numbers are dropped. An opt-out reply (`STOP`) adds the sender to the org's suppression list; `START` removes it.

## Outbound email

The `EmailProvider` adapter in `core/hailhq/core/providers/email/` sends outbound mail through AWS SES. Hail stores two kinds of sender identity in `email_domains`:

- **`kind='custom'`** — tenant-controlled DNS (for example `acme.com`). `POST /email-domains` registers the identity with SES and auto-configures a custom MAIL FROM on `send.<domain>`. The response surfaces three DKIM CNAMEs **plus** the MAIL FROM MX/SPF records (each with an optional `priority`). The tenant publishes them, then calls `POST /email-domains/{id}/verify` to re-poll SES for the DKIM and MAIL FROM status. Verified custom domains can also **receive** — inbound matches by identity, with one row and one webhook per matched domain.
- **`kind='hail_mail'`** — a per-org address under an operator-managed parent domain. The full sender is `<user>+<org>@<HAIL_MAIL_BASE_DOMAIN>` (for example `alice+acme@mail.hail.so`). The operator pre-verifies the parent domain once, out of band. Thus Hail creates per-org rows as already verified, without a call to SES.

### Self-hosted vs managed

The two surfaces differ in the source of the prefixes and in the place where you edit them:

|                       | Self-hosted Hail                                                                                                         | Managed Hail (hail.so)                                                                              |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- |
| Auth                  | Shared `HAIL_API_KEY`; one sentinel "Self-hosted" org                                                                    | Per-user `hl_live_*` keys via the website's auth backend                                            |
| Org concept           | None — single sentinel org                                                                                               | Real orgs with members                                                                              |
| Hail-mail base domain | `HAIL_MAIL_BASE_DOMAIN` (operator's `.env`)                                                                              | `HAIL_MAIL_BASE_DOMAIN` (operator's deploy env)                                                     |
| Hail-mail prefixes    | User prefix from `HAIL_MAIL_FROM` / `HAIL_MAIL_DEFAULT_USER_PREFIX` (`.env`); org prefix derived per-org from the org id | Same: user prefix from env, org prefix derived per-org; org admins override per-org via the console |
| Where edits land      | Restart with new `.env` values                                                                                           | `PATCH /email-domains/{id}` (console writes this)                                                   |
| SES production access | Operator's AWS account                                                                                                   | Operator's AWS account                                                                              |
| Billing               | Off — `usage_events` accumulates as raw analytics                                                                        | Cloud rater applies cents/unit, debits `account_credits`                                            |

Hail validates both prefixes against `^[a-z0-9]([a-z0-9-]{0,18}[a-z0-9])?$` (1–20 chars, lowercase alphanumeric + hyphen, no leading or trailing hyphen). Thus the full local part stays well under the RFC-5321 64-char budget.

### Send-time resolution

`POST /emails` picks a sender in this order:

1. An explicit `from` — it must match a `verified` row that the caller's org owns.
2. The org's single verified domain. With two or more, there is no default: the request returns `422` listing them, and the caller must pass `from`.
3. An auto-minted hail-mail row with the configured prefixes, if `HAIL_MAIL_BASE_DOMAIN` is set.

If none of those resolve, the request returns `503` with instructions on how to register a domain.

`GET /email-domains` answers the same question ahead of a send: its `default_from` field holds the address a `from`-less send would use, and is `null` when the caller must choose.

Refer to [AWS SES setup](./self-host/aws-ses.md) for the operator-side setup. Refer to [`docs/superpowers/plans/2026-05-17-hail-mail-addressing.md`](https://github.com/hail-hq/hail/blob/main/docs/superpowers/plans/2026-05-17-hail-mail-addressing.md) for the addressing/configurability plan.

## Inbound email

Operators on AWS enable inbound email when they apply `infra/terraform/`.
The Terraform provisions an S3 bucket, an SES Receipt Rule + Rule Set,
and a small Lambda. The Lambda signs SES events and forwards them to
Hail's `POST /internal/ses-events` endpoint. The API parses the raw MIME
from S3 and routes the message to the owning org by the hail-mail
local-part (`<user>+<org>@mail.hail.so`). It persists an `Email` row
with `direction='inbound'`. The background delivery worker fans out
events to per-domain webhooks and org-wide subscriptions.

```
inbound SMTP ──► SES Receipt Rule
                 ├─ Action: S3       → s3://hail-inbound/raw/<msgid>
                 └─ Action: Lambda   → POST /internal/ses-events (HMAC-signed)
                                          │
                                          ▼
                                    Hail API:
                                      • verify HMAC
                                      • fetch raw MIME from S3
                                      • parse MIME, route to org
                                      • write Email row (direction='inbound')
                                      • enqueue email_attachments to S3
                                      • fan out webhook deliveries
                                      • enqueue forwarding sends (header rewrite)
```

The cloud-agnostic SMTP path is a stub
([`SmtpInboundProvider`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/providers/email/inbound/smtp.py)).
[SMTP inbound](./self-host/smtp-inbound.md) tracks it.

### Per-domain routing — forward and/or webhook

Each `email_domains` row carries `inbound_enabled`, `forward_to`, and `webhook_url`. When `inbound_enabled` is true:

- `forward_to` (a list of addresses, at most 10) triggers one outbound `Email` row per target through the existing send loop. The forward rewrites headers: envelope `From:` = `noreply+<org>@mail.hail.so`, `Reply-To:` = the original sender, `References:` preserved for threading, and `X-Hail-Forward-Hops` plus `Auto-Submitted: auto-forwarded` for loop suppression.
- `noreply` and `forwarder` are reserved user prefixes: no tenant can mint `noreply+<org>@mail.hail.so` and receive another org's bounces.
- Every forward address must prove it wants the mail (`email_forward_targets`, one row per org + address). A verified member's login email is verified on save. A member's login email whose account is not verified yet waits (no confirm mail) and goes live on the account verify click. Any other address gets one confirm mail from `noreply+<org>@mail.hail.so` with a link to `GET /v1/forward-targets/confirm?token=…`; the page has one button that POSTs, so link scanners cannot confirm. Caps: 10 unconfirmed addresses with a live link per org, 20 confirm mails per org per day, and 3 confirm mails per recipient address per day across all orgs (an org's first mail to an address always goes through; the website caps orgs per account at 5). Until the click the inbound row carries `suppressed_reasons: ["forward_target_pending"]` and nothing is forwarded. A spam complaint on a forward sets the address to `stopped` (`forward_target_stopped`), queues a notice to the org's verified owners, and only a new link (`POST /v1/forward-targets/{address}/resend`) restarts it. `GET /v1/forward-targets` lists every address with its status. Code: [`core/hailhq/core/forward_targets.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/forward_targets.py).
- `webhook_url` triggers a signed POST through the webhook delivery worker.

A separate `inbound_routes` table, for per-mailbox routing in custom domains, is deferred to a future milestone (when tenants point their own MX at SES).

### Org-wide subscriptions

The `/webhooks` CRUD surface is the firehose pattern — one subscription covers multiple event types (`email.received`, `email.bounced`, `email.complained`). Signatures are Stripe-style: `X-Hail-Signature: t=<unix>,v1=<hex>`. Hail retries deliveries on a fixed `0/30s/2m/10m/1h/6h/24h` ladder. After the last retry, Hail marks the delivery `dead`. After 50 consecutive dead deliveries, the subscription auto-disables.

Refer to [AWS SES setup](./self-host/aws-ses.md) §10 for the operator runbook. Refer to [`docs/superpowers/specs/2026-06-06-inbound-email-design.md`](https://github.com/hail-hq/hail/blob/main/docs/superpowers/specs/2026-06-06-inbound-email-design.md) for the full spec.

---

<!-- source: https://hail.so/docs/self-host -->

# Self-hosting

Run Hail's API, voicebot, MCP server, Postgres, and object storage with Docker
Compose. LiveKit Cloud and the providers for the channels you enable remain
external. The code is AGPLv3.

## Local evaluation

Prerequisites: Git, Docker Engine, and Docker Compose v2.

```bash
git clone https://github.com/hail-hq/hail
cd hail
cp .env.example .env

# Generate a key and put it in .env as HAIL_API_KEY.
printf 'hk_%s\n' "$(openssl rand -base64 32 | tr -d '/+=' | head -c 40)"

docker compose -f docker-compose.yml -f docker-compose.local.yml \
  run --rm api alembic upgrade head
docker compose -f docker-compose.yml -f docker-compose.local.yml up -d
curl --fail http://localhost:8080/healthz
```

The local overlay supplies Postgres. Plain `docker compose up` is only for a
managed database after `DATABASE_URL` has been changed from its bundled
`postgres` default. Compose reads `.env` for containers but does not export it
to your shell; set `HAIL_API_URL` and `HAIL_API_KEY` before using the CLI or SDK.

## Order

1. **[Operations](./operations.md#deployment-self-host)** — required credentials, migrations, authentication, phone-number binding, and troubleshooting.
2. **[LiveKit Cloud](./livekit-cloud.md)** — required media and SIP bridge for voice calls.
3. **[Twilio](./twilio.md)** — required phone numbers and SIP trunk for voice/SMS.
   **[Telnyx](./telnyx.md)** — optional second carrier for numbers, calls and SMS; quotes compare it with Twilio.
   **[DIDWW](./didww.md)** — optional voice carrier (outbound calls only; numbers bought and registered through Hail).
4. **[AWS SES](./aws-ses.md)** — optional outbound/inbound email. To receive without AWS, see [SMTP inbound](./smtp-inbound.md).
5. **[VM deployment](./vm-deploy.md)** — production deployment on Ubuntu with managed Postgres and HTTPS.

Local MCP clients use `http://localhost:8081`. Web-based clients need a public
HTTPS endpoint; see [MCP clients](../mcp.md#self-host).

## Day 2

- **[Operations runbook](./operations.md)** — develop, deploy, migrate, release. The single source of truth for operating a Hail deployment.

---

<!-- source: https://hail.so/docs/self-host/vm-deploy -->

# VM deployment

Self-host Hail on one Ubuntu VM behind HTTPS. GitHub Actions deploys every commit on `main`. The database is **managed** (Neon, Supabase, RDS, or your choice). The VM runs only the stateless services.

```
        ┌───────────────────────────────────────────────────────┐
        │  GitHub Actions  ──build──▶  ghcr.io/hail-hq/hail-*   │
        │       │                                               │
        │       └──ssh──▶  VM: docker compose pull && up -d     │
        └───────────────────────────────────────────────────────┘
                                  │
                                  ▼
   ┌──────────────────────────────────────────────────────────────┐
   │  Ubuntu VM                                                   │
   │                                                              │
   │   :80 / :443  ──▶  Caddy  ──▶  api  (api.<domain>)           │
   │                            └─▶  mcp  (mcp.<domain>)          │
   │                                                              │
   │                   voicebot ──outbound──▶ LiveKit Cloud       │
   └──────────────────────────────────────────────────────────────┘
                                  │
                                  └──▶ managed Postgres (Neon / RDS / …)
```

The voicebot is a worker. It dials out to LiveKit Cloud and needs no inbound port. Only `api.<domain>` and `mcp.<domain>` are public.

## Prerequisites

- An Ubuntu 22.04+ VM with a public IPv4 address. 2 vCPU / 4 GB RAM is sufficient to start.
- A managed Postgres that the VM can reach. Copy its connection string (typically `postgresql://USER:PASS@HOST/DB?sslmode=require`).
- A domain that you control. You point `api.<domain>` and `mcp.<domain>` at the VM's IP.

## 1. Prepare the VM

Connect with SSH as a sudo-capable user. Then run:

```bash
# Docker engine + compose plugin.
curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker "$USER"
# Re-login or `newgrp docker` so the group change takes effect.

# Firewall: only 22, 80, 443 reach the host.
sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 443/udp   # HTTP/3
sudo ufw enable

# Working directory.
sudo mkdir -p /opt/hail
sudo chown "$USER:$USER" /opt/hail
git clone https://github.com/hail-hq/hail /opt/hail
cd /opt/hail
```

If your fork is private, clone via SSH (`git@github.com:<owner>/hail.git`) and authorize a deploy key on the VM. The CI deploy step runs `git fetch origin` on every push and gets a 403 error otherwise.

> **UFW + Docker caveat:** Docker writes its own iptables rules that _bypass_ UFW. The compose file binds the api/mcp ports to `127.0.0.1`. Thus external traffic cannot reach them, even if UFW permits it locally. If you add a new published port, keep the `127.0.0.1:` prefix, unless you intend the port to be public.

## 2. DNS

Create two A records (or one wildcard) that point at the VM:

| record         | type | value     |
| -------------- | ---- | --------- |
| `api.<domain>` | A    | _VM IPv4_ |
| `mcp.<domain>` | A    | _VM IPv4_ |

If either name does not resolve to the VM yet, Caddy cannot get Let's Encrypt certs. Verify the records before you continue.

```bash
dig +short api.<domain>
dig +short mcp.<domain>
```

## 3. Configure `.env` on the VM

```bash
cd /opt/hail
cp .env.example .env
```

Edit `/opt/hail/.env` and fill in:

- All provider secrets (Twilio, LiveKit, Deepgram, Cartesia, optional ElevenLabs fallback, and at least one LLM key); see [Twilio](./twilio.md) and [LiveKit Cloud](./livekit-cloud.md).
- `HAIL_API_KEY`: generate with `openssl rand -base64 32 | tr -d '/+=' | head -c 40 | sed 's/^/hk_/'`.
- `DATABASE_URL`: the managed Postgres connection string. Most providers require `?sslmode=require`.
- `HAIL_DOMAIN`: your apex (for example, `hail.example.com`). Both `api.` and `mcp.` subdomains derive from it.

The `.env` file is already gitignored. Never commit it.

## 4. First boot

The compose invocations below are verbose because the prod overlay is not the default. If you use the VM often, add a shell alias to the deploy user's `~/.bashrc`:

```bash
echo "alias dcprod='docker compose -f /opt/hail/docker-compose.yml -f /opt/hail/docker-compose.prod.yml'" >> ~/.bashrc
```

```bash
cd /opt/hail
# Log in to GHCR so docker can pull (one-time; the CI deploy will re-login
# each run). Create a Personal Access Token with `read:packages`.
echo "$GHCR_PAT" | docker login ghcr.io -u "$GHCR_USER" --password-stdin

docker compose -f docker-compose.yml -f docker-compose.prod.yml pull
docker compose -f docker-compose.yml -f docker-compose.prod.yml run --rm api alembic upgrade head
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
docker compose -f docker-compose.yml -f docker-compose.prod.yml ps
```

To skip the manual GHCR login, trigger the GitHub Actions workflow (step 6) first. Its `docker login` step writes credentials that the VM can use again.

Bind your first phone number using [Self-host: first-run setup](./operations.md#self-host-first-run-setup). Because this deployment uses managed Postgres, replace `docker compose exec postgres …` with `psql "$DATABASE_URL" -c …`.

Send a request to `https://api.<domain>/healthz` from your laptop. You must see `{"status":"ok"}`. Caddy gets a Let's Encrypt cert on the first request. The first call after boot can take 10 to 20 seconds.

## 5. Configure GitHub Actions

Set these under **Settings → Secrets and variables → Actions**:

| name                     | type   | value                                                 |
| ------------------------ | ------ | ----------------------------------------------------- |
| `DEPLOY_SSH_HOST`        | secret | VM hostname or IP                                     |
| `DEPLOY_SSH_USER`        | secret | SSH user on the VM (in the `docker` group)            |
| `DEPLOY_SSH_PRIVATE_KEY` | secret | Private key matching an authorized public key         |
| `DEPLOY_SSH_PORT`        | secret | _(optional)_ SSH port; omit to default to 22          |
| `DEPLOY_PATH`            | _var_  | _(optional)_ repo checkout path; defaults `/opt/hail` |

Generate a deploy key dedicated to CI. Do not use a personal key again:

```bash
ssh-keygen -t ed25519 -f hail-deploy -C "github-actions@hail" -N ""
# On the VM: append hail-deploy.pub to ~/.ssh/authorized_keys
# In GitHub: paste hail-deploy (the private key) into DEPLOY_SSH_PRIVATE_KEY.
```

If every deploy must gate on a manual approval, create a `production` GitHub Environment (Settings → Environments → New → `production`) with required reviewers. The deploy job targets `environment: production` and inherits any protection rules automatically.

The GHCR push uses the workflow's `GITHUB_TOKEN` (already scoped `packages: write`). No extra PAT is necessary.

## 6. Trigger the first deploy

Push any commit to `main`, or run **Actions → Deploy → Run workflow** in the GitHub UI. The workflow:

1. Builds `api`, `voicebot`, `mcp` images in parallel and pushes both `sha-<commit>` and `latest` to ghcr.io/hail-hq/hail-\*.
2. Connects to the VM with SSH, fetches the new commit, and runs `alembic upgrade head`. Then `docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d` rolls every service except the voicebot onto the newly pulled `:latest`.
3. Rolls the voicebot with [`scripts/deploy/roll-voicebot.sh`](../../../scripts/deploy/roll-voicebot.sh): the new voicebot starts next to the old one, and once it has registered with LiveKit, the old one finishes its live calls (up to about 1 hour) and stops. A deploy no longer drops calls.

`concurrency: deploy-prod` makes simultaneous merges queue. They do not race.

## Day-2

- **Tail logs**: `docker compose -f docker-compose.yml -f docker-compose.prod.yml logs -f api`
- **Restart one service**: `docker compose -f docker-compose.yml -f docker-compose.prod.yml restart api`
- **Roll back**: every build pushes a `sha-<commit>` tag. To pin an older build:

  ```bash
  docker pull ghcr.io/hail-hq/hail-api:sha-<previous-commit>
  docker tag  ghcr.io/hail-hq/hail-api:sha-<previous-commit> ghcr.io/hail-hq/hail-api:latest
  docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d api
  ```

  Repeat for each service. The cleanest rollback is to revert the commit on `main` and let CI deploy again.

- **Update Caddy config**: edit `/opt/hail/Caddyfile`, then `docker compose -f docker-compose.yml -f docker-compose.prod.yml exec caddy caddy reload --config /etc/caddy/Caddyfile`.
- **Reclaim disk**: deploys remove unused Hail images older than 48 hours before pulling and after rollout, preserving all container-referenced images. A pull requires at least 20 GiB free. To recover manually: `docker image prune -af --filter "until=48h" --filter "label=org.opencontainers.image.source=https://github.com/hail-hq/hail"`. A dangling-only prune misses tagged `sha-<commit>` images. Older rollback tags remain in GHCR.
- **Bound logs**: the production overlay rotates each service’s Docker JSON logs at 10 MB with three files. The limit takes effect when containers are recreated.

## Footguns

| symptom                                                      | fix                                                                                                                                                                                                                                                                                               |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Live calls drop when you run `up -d` by hand during a deploy | A deploy can leave 2 voicebots running while the old one drains. A plain `up -d` scales the voicebot back to 1 and stops one with a 10 s limit. Wait until `docker compose -f docker-compose.yml -f docker-compose.prod.yml ps voicebot` shows one container, or pass the other services by name. |
| Caddy logs `tls: no certificates configured` on first start  | DNS has not propagated yet, or :80/:443 is not open. Confirm that `dig api.<domain>` returns the VM IP and `sudo ufw status` shows both ports open.                                                                                                                                               |
| `docker pull` 401 / "denied" from GHCR                       | The package is private by default. Set its visibility to public under GitHub → Packages → hail-\_ → Package settings, or rely on the workflow's `docker login` step.                                                                                                                              |
| `permission denied while trying to connect to docker daemon` | The deploy user must be in the `docker` group. Run `sudo usermod -aG docker $USER` on the VM and re-establish the SSH session.                                                                                                                                                                    |
| Caddy 502s for api.<domain>                                  | Caddy resolves `api` and `mcp` via Compose's internal DNS: all services share the `hail` project network. Do not split the project name across files.                                                                                                                                             |
| api/mcp port reachable from the public internet              | A published port without the `127.0.0.1:` prefix bypasses UFW. Check `docker compose -f docker-compose.yml -f docker-compose.prod.yml port api 8080` returns `127.0.0.1:8080`, not `0.0.0.0:8080`.                                                                                                |

---

<!-- source: https://hail.so/docs/self-host/livekit-cloud -->

# LiveKit Cloud

LiveKit Cloud supplies the media (SIP bridge + WebRTC) in v1. A self-hosted SFU is a later milestone.

## 1. Project + keys

1. Sign up at [cloud.livekit.io](https://cloud.livekit.io).
2. Create a project.
3. From **Settings → Keys**, copy these values into `.env`:
   - `LIVEKIT_URL` — `wss://<project>-<region>.livekit.cloud`
   - `LIVEKIT_API_KEY`
   - `LIVEKIT_API_SECRET`

## 2. SIP outbound trunk

First create the Twilio trunk and credentials in the [Twilio guide](./twilio.md).
Then, in LiveKit Cloud:

1. Open **Telephony → SIP trunks → Create new trunk**.
2. Select **Outbound** and use the Twilio termination domain
   (`<name>.pstn.twilio.com`) as the address.
3. Add the Twilio number in E.164 format and enter the same username/password
   configured on the Twilio trunk.
4. Create the trunk and copy its ID into `.env` as
   `LIVEKIT_TWILIO_SIP_OUTBOUND_TRUNK_ID`.

Hail passes this ID to LiveKit for every outbound call from a Twilio number.
The older `LIVEKIT_SIP_OUTBOUND_TRUNK_ID` name remains supported as a
compatibility alias; the explicit Twilio name takes precedence. Numbers from
another carrier use their own trunk: see [DIDWW](./didww.md) and
`LIVEKIT_TELNYX_SIP_OUTBOUND_TRUNK_ID`. Hail does not read a Twilio
trunk-domain environment variable. See LiveKit's
[outbound trunk reference](https://docs.livekit.io/telephony/making-calls/outbound-trunk/)
for the current UI and JSON forms.

## 3. Voicebot worker

With the local Compose overlay, run:

```bash
docker compose -f docker-compose.yml -f docker-compose.local.yml up -d voicebot
```

At startup, the worker registers with LiveKit as a dispatchable agent. The
Hail API dispatches it into a room for each call.

For the full flow, refer to [Architecture](../architecture.md).

Noise filtering, interruption handling, and call recovery are enabled automatically.
Run the API and voicebot worker from the same release.

## 4. Inbound calls

Hail answers calls on a number once the number routes calls to an agent
([Agents](../agents.md)). LiveKit needs **one inbound trunk** and **one
dispatch rule**. Hail adds and removes numbers on the trunk itself.

1. Inbound trunk. One trunk for every carrier: LiveKit allows a single
   wildcard (empty `numbers`) inbound trunk per project. Without `numbers`,
   LiveKit needs `allowed_addresses` (or `auth_username`/`auth_password`,
   which Twilio Elastic SIP does not send). The carrier pages say what each
   carrier sends: [Twilio](./twilio.md#6-inbound-calls),
   [Telnyx](./telnyx.md#inbound-calls), [DIDWW](./didww.md#inbound-calls).

   ```bash
   cat > inbound.json <<'JSON'
   {"trunk": {"name": "hail-inbound", "numbers": [], "allowed_addresses": ["0.0.0.0/0"]}}
   JSON
   lk sip inbound create inbound.json
   # → ST_...  → LIVEKIT_SIP_INBOUND_TRUNK_ID
   ```

   Hail reads the dialed number from the call and finds its carrier in
   `phone_numbers`; the trunk does not need to know it. To lock each carrier
   to its own trunk instead, create one trunk per carrier with that
   carrier's signaling IPs in `allowed_addresses` and set
   `LIVEKIT_<CARRIER>_SIP_INBOUND_TRUNK_ID` (an override not listed in
   `.env.example`) to that trunk's id. Narrow `0.0.0.0/0` to the carriers' IP ranges when you can.

2. Dispatch rule. One rule, bound to the inbound trunk, that puts each
   caller in its own room and dispatches the voicebot with the static
   metadata it expects:

   ```bash
   cat > dispatch-rule.json <<'JSON'
   {
     "dispatch_rule": {
       "name": "hail-inbound",
       "trunk_ids": ["<inbound-trunk-id>"],
       "rule": {"dispatchRuleIndividual": {"roomPrefix": "hail-in-"}},
       "roomConfig": {
         "agents": [{"agentName": "hail-voicebot", "metadata": "{\"direction\":\"inbound\"}"}]
       }
     }
   }
   JSON
   lk sip dispatch create dispatch-rule.json
   ```

   Room names include the caller's number (LiveKit's individual rule);
   `calls.from_e164` records it anyway.

3. Set the trunk id in `.env` and recreate `api` and `voicebot`. Then
   `PATCH /v1/numbers/{id}` with `voice_agent_id` registers a number
   ([Agents](../agents.md#routing-rules)).

Reference: LiveKit [accepting calls](https://docs.livekit.io/telephony/accepting-calls/),
[inbound trunk](https://docs.livekit.io/telephony/accepting-calls/inbound-trunk/),
[dispatch rule](https://docs.livekit.io/telephony/accepting-calls/dispatch-rule/).

---

<!-- source: https://hail.so/docs/self-host/twilio -->

# Twilio

You need a Twilio account, a phone number, and a SIP trunk that bridges to LiveKit Cloud.

## 1. Account credentials

From [console.twilio.com](https://console.twilio.com), copy these values:

- `TWILIO_ACCOUNT_SID` — starts with `AC…`
- `TWILIO_AUTH_TOKEN` — click "Show" to see the value

Put them in `.env`.

## 2. Phone number

Go to **Phone Numbers → Buy a number**. Select a number with the Voice capability. For outbound SMS, select a number that also has the SMS capability. The carrier fixes these capabilities when you buy the number; you cannot add SMS to a voice-only number later. Note the E.164 format (`+1…`).

## 3. SIP trunk

1. **Elastic SIP Trunking → Trunks → Create new Trunk**.
2. Under **Termination**, choose a unique termination domain such as
   `your-trunk.pstn.twilio.com`.
3. Add a credential list with a SIP username and password. LiveKit's outbound
   trunk must use the same credentials.
4. Under **Numbers**, attach the phone number from step 2.
5. Continue with [LiveKit Cloud setup](./livekit-cloud.md), using the Twilio
   termination domain and credentials to create a LiveKit outbound trunk.

Do not add the termination domain to Hail's `.env`: Hail uses the resulting
LiveKit trunk ID (`LIVEKIT_TWILIO_SIP_OUTBOUND_TRUNK_ID`) at runtime. Twilio calls
traffic from LiveKit to the PSTN “termination.” Its “origination” settings
carry inbound calls to LiveKit: see [§6](#6-inbound-calls).

Twilio requires outbound destinations and caller IDs in E.164 format. Trial
accounts can call only verified destination numbers. See Twilio's
[Elastic SIP Trunking reference](https://www.twilio.com/docs/sip-trunking).

## 4. Outbound SMS

Hail sends SMS from a dedicated number.

1. **Acquire an SMS-capable number** (step 2 above). Only numbers with the SMS
   capability can send.
2. **Hail sets the number up.** When the number becomes active, Hail attaches
   it to your organization's Twilio Messaging Service and creates that service
   the first time, with its inbound webhook pointed at
   `<HAIL_API_URL>/sms/inbound`. There is one Messaging Service per
   organization; every number joins the same shared sender pool. If that step
   failed (the API log says why), assigning a texts agent runs it again, and
   `POST /numbers/{id}/enable-sms` runs it by hand. Both are idempotent.
3. **Send.** Call `POST /sms` with the recipient and body. Hail sends from your
   organization's dedicated number.

**A2P 10DLC (United States).** Each organization registers its own brand and
campaign from the console (Numbers → Register for US texting) or with
`POST /verifications` and `purpose=sms`. Set `TWILIO_PRIMARY_PROFILE_SID` to
your approved Trust Hub primary profile (business identity ISV Reseller or
Partner); without it, SMS registration through Twilio is off. Texts from an
organization's US long codes to US numbers return 422 until it is approved.

**Sender ID (optional, rest-of-world).** For destinations that allow it, set a
custom alphanumeric sender with `PATCH /sms/sender-id` (read the current value
with `GET /sms/sender-id`). Clear it by sending `null`. When you set none, Hail
falls back to the platform default sender. The United States and Canada do not
allow alphanumeric sender IDs — messages there always send from the dedicated
number, regardless of this setting.

**Rate limits.** Per-organization send velocity is capped by
`HAIL_VELOCITY_SMS_PER_HOUR` (default 100) and `HAIL_VELOCITY_SMS_PER_DAY`
(default 1000). An abuse monitor suspends an organization's SMS channel when its
opt-out rate is too high — see [operations](./operations.md) for the
`HAIL_SMS_ABUSE_*` variables and how to lift a suspension.

## 5. Inbound SMS & opt-out

Inbound texts arrive through the Messaging Service's inbound webhook, which
Hail sets to `<HAIL_API_URL>/sms/inbound` (HTTP POST) when it creates or reuses
the service. Nothing to set on the number itself. Hail verifies Twilio's
`X-Twilio-Signature` against `HAIL_API_URL`, so that value must be the public
URL that Twilio posts to.

**Recognized keywords** (Hail matches them on the message body, case-insensitive):

- Opt out (STOP): `STOP`, `STOPALL`, `UNSUBSCRIBE`, `CANCEL`, `END`, `QUIT`
- Opt in (START): `START`, `YES`, `UNSTOP`
- Help: `HELP`, `INFO`

Hail records opt-outs in its own suppression list, regardless of the Twilio
configuration. Hail checks this list before every send.

**Opt-out replies:** By default, **Twilio** replies automatically to
STOP/HELP/START and carrier-blocks opted-out numbers. In that setup, keep
`HAIL_SMS_COMPLIANCE_REPLIES_ENABLED=false`. If you want **Hail** to own the
replies (for example, a non-Twilio provider, or a custom keyword experience),
do the two steps that follow. **Caution: if you disable Twilio's default
opt-out handling, the change is account-wide and requires a Twilio Support
request; there is no API for it.** First, disable Twilio's default opt-out
handling. Then set `HAIL_SMS_COMPLIANCE_REPLIES_ENABLED=true`.

## 6. Inbound calls

Twilio sends a call to LiveKit when the number sits on an Elastic SIP trunk
whose origination URI is LiveKit. Hail attaches the number when it routes
calls to an agent; you set up the trunk once.

1. On the trunk from step 3, open **Origination** and add the origination
   URI `sip:<project>.sip.livekit.cloud;transport=tcp` (your LiveKit SIP
   endpoint is on the project's settings page).
2. Copy the trunk SID (`TK...`) into `.env` as `TWILIO_SIP_TRUNK_SID`.
3. Create the LiveKit inbound trunk and dispatch rule
   ([LiveKit Cloud §4](./livekit-cloud.md#4-inbound-calls)) and set
   `LIVEKIT_SIP_INBOUND_TRUNK_ID`.

Twilio Elastic SIP Trunking has no inbound username/password, so the LiveKit
trunk lists the numbers instead: `PATCH /v1/numbers/{id}` with
`voice_agent_id` adds the number to the Twilio trunk
(`trunking.v1.trunks(TK...).phone_numbers`) and to the LiveKit trunk; `null`
removes it from both ([Agents](../agents.md)).

---

<!-- source: https://hail.so/docs/self-host/telnyx -->

# Telnyx

Second carrier for numbers, calls and SMS, next to [Twilio](./twilio.md).
When both are configured, `POST /numbers/quotes` compares live inventory,
prices and verification requirements from each and ranks the offers; `POST
/numbers` buys one by `quote_id`. A number never changes carrier: calls from a
Telnyx number go out through the Telnyx trunk, SMS through Telnyx messaging.

```bash
# 1. Live offers for this organization (Twilio + Telnyx)
curl -X POST "$HAIL_API_URL/v1/numbers/quotes" -H "Authorization: Bearer $HAIL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"country_code":"PT","capabilities":["voice"]}'
# 2. Buy the recommended offer
curl -X POST "$HAIL_API_URL/v1/numbers" -H "Authorization: Bearer $HAIL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"country_code":"PT","quote_id":"<recommended_quote_id>"}'
```

Or `hail numbers acquire --country PT`. The API does not say which carrier
backs an offer; the stored quote does (`offer->>'provider'` in the
`number_offers` table). Schemas:
[`openapi/openapi.yaml`](../../../openapi/openapi.yaml). Code:
[`carrier_routing.py`](../../../core/hailhq/core/carrier_routing.py),
[`number_offers.py`](../../../core/hailhq/core/number_offers.py),
[`number_orders.py`](../../../api/hailhq/api/number_orders.py).

Telnyx stays off until all five values below are set. With only
`TELNYX_API_KEY` set, quotes list Twilio offers only.

| `.env`                                 | Where it comes from                                                                                                                 |
| -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `TELNYX_API_KEY`                       | Mission Control → **Account → Keys & Credentials → API Keys**                                                                       |
| `TELNYX_CONNECTION_ID`                 | The SIP connection from step 2 (numbers Hail buys are attached to it)                                                               |
| `TELNYX_SIP_USERNAME`                  | The SIP connection's username (sent as `X-Telnyx-Username` on every INVITE)                                                         |
| `LIVEKIT_TELNYX_SIP_OUTBOUND_TRUNK_ID` | The LiveKit trunk from step 3                                                                                                       |
| `TELNYX_PUBLIC_KEY`                    | `GET https://api.telnyx.com/v2/public_key` → `data.public` (also under **Keys & Credentials → Public Key**). Verifies SMS webhooks. |

## 1. Outbound voice profile

Every SIP connection needs one. Set the countries you will call; the default
allows only US and CA.

```bash
curl -X POST https://api.telnyx.com/v2/outbound_voice_profiles \
  -H "Authorization: Bearer $TELNYX_API_KEY" -H 'Content-Type: application/json' \
  -d '{"name":"hail","traffic_type":"conversational","service_plan":"global",
       "whitelisted_destinations":["US","CA","GB","PT","SE"],
       "daily_spend_limit":"50","daily_spend_limit_enabled":true}'
```

## 2. SIP connection

An FQDN connection with credentials, as in
[LiveKit's Telnyx guide](https://docs.livekit.io/sip/quickstarts/configuring-telnyx-trunk/).
Pick a username (4–32 letters and digits) and a password (8–128 characters).

```bash
curl -X POST https://api.telnyx.com/v2/fqdn_connections \
  -H "Authorization: Bearer $TELNYX_API_KEY" -H 'Content-Type: application/json' \
  -d '{"active":true,"anchorsite_override":"Latency","connection_name":"hail",
       "user_name":"<username>","password":"<password>","transport_protocol":"TCP",
       "outbound":{"outbound_voice_profile_id":"<profile id from step 1>"}}'
```

The response `data.id` is `TELNYX_CONNECTION_ID`; the username is
`TELNYX_SIP_USERNAME`. The password goes only into the LiveKit trunk below.

## 3. LiveKit outbound trunk

```bash
cat > telnyx-trunk.json <<'EOF'
{"trunk": {"name": "telnyx", "address": "sip.telnyx.com", "numbers": ["*"],
  "auth_username": "<username>", "auth_password": "<password>"}}
EOF
lk sip outbound create telnyx-trunk.json
```

Copy the trunk ID (`ST_…`) into `.env` as `LIVEKIT_TELNYX_SIP_OUTBOUND_TRUNK_ID`
and restart `api`. A call from a `telnyx` number with this value empty fails
with `end_reason = carrier_route_failed` before any LiveKit room exists.

## 4. SMS

Nothing to create by hand. When an SMS-capable number becomes active, Hail
creates one Telnyx messaging profile per organization, with its webhook at
`<HAIL_API_URL>/sms/telnyx`, and attaches the number to it (assigning a texts
agent, or `POST /numbers/{id}/enable-sms`, runs the same step again). Delivery status
and inbound messages (including `STOP`/`START`/`HELP`) arrive on that route;
Hail checks the Ed25519 signature with `TELNYX_PUBLIC_KEY` and rejects
requests older than five minutes. `HAIL_API_URL` must be the public URL Telnyx
can reach.

**A2P 10DLC (United States).** Each organization registers its own brand and
campaign from the console (Numbers → Register for US texting) or with
`POST /verifications` and `purpose=sms`, as a company or a person. Telnyx
texts a person a PIN; they enter it in the console or with
`POST /verifications/{id}/confirm`. Until an organization is approved, texts
from its US long codes to US numbers return 422.

## 5. Countries that need verification

Telnyx will not sell some numbers (Sweden, Portugal, Italy, and others)
until a requirement group is approved. Hail files it: `GET
/v1/verifications/requirements?quote_id=<quote_id>&country_code=SE&number_type=mobile&subject_type=business`,
with the `quote_id` of a Telnyx offer, lists the fields and documents Telnyx
asks for. `POST /v1/verifications` (with the same `quote_id`) creates the
requirement group under
`customer_reference = hail-<organization UUID>`, uploads the files, and a
superadmin approval submits it to Telnyx. Once Telnyx approves, quotes for
that country and type turn `ready` and the order carries the group of the
Hail verification it uses (a group approved at Telnyx with no Hail row is not
used). The
console does all of this from the number picker. Plug-in:
[`core/hailhq/core/providers/verification/telnyx.py`](../../../core/hailhq/core/providers/verification/telnyx.py).

## Orders

Telnyx orders complete asynchronously. `POST /numbers` reserves setup plus the
first month from credits, places the order and returns the number as
`pending`. A sweeper polls the order every 15 seconds; on success the number
becomes `active` and the reservation turns into the monthly fee. An order still
pending after two hours, or one Telnyx has no record of, is marked `failed` and
refunded once. If Telnyx completes it later, release the number in Mission
Control by hand; the API log names the order.

## Inbound calls

Telnyx sends a call to LiveKit through the FQDN connection
(`TELNYX_CONNECTION_ID`) when that connection has inbound enabled and its
FQDN is LiveKit's SIP endpoint. Hail sets the number's voice connection when
it routes calls to an agent; you set up the connection once.

1. On the connection, **Inbound**: Destination number format `+E.164`,
   Origination number format `+E.164`, transport TCP.
2. Add an FQDN: `<project>.sip.livekit.cloud` (port 5060, no username).
3. Create the LiveKit inbound trunk and dispatch rule
   ([LiveKit Cloud §4](./livekit-cloud.md#4-inbound-calls)) and set
   `LIVEKIT_SIP_INBOUND_TRUNK_ID`.

`PATCH /v1/numbers/{id}` with `voice_agent_id` then sets
`connection_id` on the number (`PATCH /v2/phone_numbers/{id}/voice`) and
lists it on the LiveKit trunk; `null` clears both ([Agents](../agents.md)).
Reference: LiveKit [Telnyx guide](https://docs.livekit.io/telephony/start/providers/telnyx/).

---

<!-- source: https://hail.so/docs/self-host/didww -->

# DIDWW

Second voice carrier, next to [Twilio](./twilio.md). Calls in and out.
A number's `provider` column picks its trunk
([`core/hailhq/core/carrier_routing.py`](../../../core/hailhq/core/carrier_routing.py)).

```bash
# A call from a DIDWW number goes out through LIVEKIT_DIDWW_SIP_OUTBOUND_TRUNK_ID.
curl -X POST "$HAIL_API_URL/calls" -H "Authorization: Bearer $HAIL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"from":"+351300000000","to":"+14155550100","system_prompt":"Be brief.","recipient_consent":true}'
```

Buying goes through the normal quote flow once `DIDWW_API_KEY` and
`LIVEKIT_DIDWW_SIP_OUTBOUND_TRUNK_ID` are both set and DIDWW has enabled
inventory search on the account ([§1](#1-didww-account), step 3). With any
of the three missing, no DIDWW number is quoted ([§6](#6-no-didww-offer-shows-up)).

Not supported on DIDWW: SMS. Inbound calls: see [below](#inbound-calls).
Hail sends no SMS through
DIDWW (`CARRIERS["didww"].sms_route` is `None`), so a quote with `"sms"` in
`capabilities` never gets a DIDWW offer. DIDWW itself sells SMS-capable
numbers only in US, CA, MX and PR (number groups with the `sms_in` feature);
Portugal and the rest are voice only on their side too.

## 1. DIDWW account

1. Sign up at [my.didww.com](https://my.didww.com), billed in **USD**. Hail
   reads DIDWW's SKU prices as USD cents with no currency conversion or
   check; a non-USD account will silently mis-price every DIDWW number.
2. Outbound trunks are off by default. Ask `support@didww.com` to enable
   "Outbound Trunks" on the account.
3. Inventory search over the API is off by default too. Ask
   `support@didww.com` to enable `GET /v3/available_dids` on the account
   (and on the sandbox account, if you have one). Check:

   ```bash
   curl -s -H "Api-Key: $DIDWW_API_KEY" -H "Accept: application/vnd.api+json" \
     "https://api.didww.com/v3/available_dids?page[size]=1"
   # 403 {"errors":[{"detail":"Access for Customer is Denied"}]} → not enabled yet
   # 200 {"data":[...]}                                           → enabled
   ```

   Every other call works without it (`balance`, `countries`, `did_groups`,
   `address_requirements`), so the key looks fine while quotes stay empty.

4. Numbers are bought through Hail ([§5](#5-buying-a-number)); the DIDWW
   panel is only needed for the trunk and the API key.

## 2. DIDWW outbound trunk

**Voice → Outbound Trunks → Create** ([guide](https://doc.didww.com/voice/outbound-trunks/how-to-guides/create-outbound-trunk.html)):

- Authentication: **Credentials & IP-Based**.
- Allowed SIP IPs and Allowed RTP IPs: LiveKit Cloud's static ranges
  `143.223.88.0/21`, `161.115.160.0/19`, `153.57.128.0/18`
  ([LiveKit static IPs](https://docs.livekit.io/deploy/admin/regions/endpoints/)).
  If the form rejects a range, ask DIDWW support to add it.
- CLI settings: add the number to the allowed Caller IDs. On mismatch:
  **Reject call**.
- Save. Copy the SIP username and password for step 3.

DIDWW uses SIP digest on INVITE (realm `out.didww.com`), no REGISTER.
Signaling hosts: `fra.eu.out.didww.com` (EU), `nyc.us.out.didww.com` (US),
`any.out.didww.com` (anycast)
([SIP information](https://doc.didww.com/voice/outbound-trunks/outbound-sip-information.html)).

## 3. LiveKit outbound trunk

In LiveKit Cloud, **Telephony → SIP trunks → Create new trunk → Outbound**:

- Address: `fra.eu.out.didww.com`
- Numbers: `*` (any DIDWW number; the Twilio trunk uses the same). Hail picks
  the trunk by the number's `provider`, so no per-number list is needed.
- Username and password: from step 2

Or with the CLI:

```bash
cat > didww-trunk.json <<'EOF'
{"trunk": {"name": "didww", "address": "fra.eu.out.didww.com",
  "numbers": ["*"], "auth_username": "<username>", "auth_password": "<password>"}}
EOF
lk sip outbound create didww-trunk.json
```

Copy the trunk ID (`ST_…`) into `.env` as `LIVEKIT_DIDWW_SIP_OUTBOUND_TRUNK_ID`
and restart `api`.

## 4. API key

my.didww.com → **API** → create a key. Put it in `.env`:

```
DIDWW_API_KEY=<key>
DIDWW_ENVIRONMENT=production
```

`DIDWW_ENVIRONMENT=sandbox` points every call at `sandbox-api.didww.com`.
The sandbox is a separate site with its own login and key:
[my-sandbox.didww.com](https://my-sandbox.didww.com) → **API → DIDWW API 3 →
Create new API Key**. No sandbox login? Ask `support@didww.com` for a sandbox
account tied to your production account. The sandbox needs
`available_dids` enabled as well (step 3 of [§1](#1-didww-account)).

For a local smoke test keep the production values in `.env` and override on
the command line:

```bash
DIDWW_API_KEY=<sandbox key> DIDWW_ENVIRONMENT=sandbox \
  uv run --directory api uvicorn hailhq.api.main:app --port 8080
```

## 5. Buying a number

```bash
curl -X POST "$HAIL_API_URL/numbers/quotes" -H "Authorization: Bearer $HAIL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"country_code":"PT","number_type":"national","capabilities":["voice"]}'
# → offers[] with readiness "ready" or "verification_required"
curl -X POST "$HAIL_API_URL/numbers" -H "Authorization: Bearer $HAIL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"country_code":"PT","number_type":"national","quote_id":"<quote_id>"}'
```

The API does not say which carrier backs an offer. The stored quote does:

```bash
psql "$DATABASE_URL" -c "SELECT id, offer->>'provider' AS carrier, offer->>'e164' AS e164 FROM number_offers ORDER BY expires_at DESC LIMIT 10;"
```

Order of events for a country that needs end-user registration:

1. `readiness: verification_required` → the customer fills `/verifications`
   (console wizard). Hail creates the identity, address and proofs at DIDWW and
   validates them (`address_requirement_validations`). Papers that pass are
   approved at once. No person reviews them
   ([`create_verification`](../../../api/hailhq/api/routes/verifications.py)).
2. `POST /numbers` reserves setup + first month, orders the DID
   (`provisioning_state: pending`).
3. The reconciler files the registration (`address_verifications`) once the
   DID exists and polls it every 15 minutes. DIDWW approves in 1–3 days →
   `active`.
4. Rejected → `failed`, the DID is terminated, the monthly fee is refunded,
   the setup fee is charged (DIDWW billed it and does not refund).
5. Still pending after 7 days → `failed`. DID exists: terminated, monthly fee
   refunded, setup fee charged. No DID: setup and monthly fee refunded.

If the terminate call fails, the DID id is kept in
`provisioning_metadata.unreleased_resource_id` and the reconciler retries every
minute until DIDWW accepts it.

Schemas: [`openapi/openapi.yaml`](../../../openapi/openapi.yaml). Code:
[`providers/voice/didww.py`](../../../core/hailhq/core/providers/voice/didww.py),
[`providers/verification/didww.py`](../../../core/hailhq/core/providers/verification/didww.py),
[`number_orders.py`](../../../api/hailhq/api/number_orders.py), the `didww`
entry of `CARRIERS` in
[`carrier_routing.py`](../../../core/hailhq/core/carrier_routing.py).

A call from a `didww` number with `LIVEKIT_DIDWW_SIP_OUTBOUND_TRUNK_ID` empty
fails with `end_reason = carrier_route_failed` before any LiveKit room exists.
The same applies to Twilio numbers when `LIVEKIT_TWILIO_SIP_OUTBOUND_TRUNK_ID` is empty.

## 6. No DIDWW offer shows up

Ask for a quote, then read the `api` log and the stored quotes:

```bash
curl -s -X POST "$HAIL_API_URL/numbers/quotes" -H "Authorization: Bearer $HAIL_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"country_code":"PT","number_type":"national","capabilities":["voice"]}' \
  | python3 -c 'import json,sys; q=json.load(sys.stdin); print(q["incomplete"], len(q["offers"]))'
docker compose logs api | grep "number quotes incomplete"
psql "$DATABASE_URL" -c "SELECT offer->>'provider' AS carrier, count(*) FROM number_offers WHERE expires_at > now() GROUP BY 1;"
```

| You see                                                  | Cause                                                                                 | Fix                                                                                     |
| -------------------------------------------------------- | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `incomplete` true, log line `... no answer from didww`   | The DIDWW lookup failed. Almost always `available_dids` is not enabled (§1, step 3).  | Run the `curl` in §1 step 3. 403 → ask DIDWW support. Other error → see the `api` log.  |
| `incomplete` false and no `didww` row in `number_offers` | `DIDWW_API_KEY` or `LIVEKIT_DIDWW_SIP_OUTBOUND_TRUNK_ID` empty, or `"sms"` requested. | Set both in `.env`, restart `api`. Quote with `"capabilities":["voice"]`.               |
| 422 `we don't offer a national number…`                  | The pair (country, type) is not in `costs/didww.json`.                                | Wait for the weekly catalog sync, or run it (`docs/operations/number-catalog-sync.md`). |
| 422 for a pair `costs/didww.json` lists                  | The row has `by_request` or `receive_only` ([below](#numbers-hail-cannot-sell)).      | Nothing to fix: Hail does not sell it. For `by_request`, support orders it by hand.     |

The console's "both jobs" card (calls + SMS) never lists DIDWW; only
"calls-only numbers" does.

## Numbers Hail cannot sell

The catalog ([`costs/didww.json`](../../../costs/didww.json)) marks two kinds
of DIDWW number that the console and `POST /numbers/quotes` never offer:

```bash
jq -r '.numbers[] | select(.by_request) | "\(.country_code):\(.number_type)"' costs/didww.json
jq -r '.numbers[] | select(.receive_only) | "\(.country_code):\(.number_type)"' costs/didww.json
```

- `by_request: true` (40 rows on 2026-10-07): DIDWW picks the number when
  the order is placed, so there is no number to show and buy. Examples:
  PT national, DE local, GB local, FR local, ES local. The console shows
  "set up by our team, not bought here" with an **Ask support for a number**
  button that emails hi@hail.so. Twilio or Telnyx sell most of these
  country and type pairs; only these 8 have no other carrier: AL local,
  CH toll-free, DE national, DK national, HK national, IE national,
  MT national, NG local. Fulfilment:
  [`docs/operations/didww-by-request.md`](../../operations/didww-by-request.md).
- `receive_only: true` (86 rows, mostly toll-free): the number takes calls
  but cannot place them (DIDWW feature `voice_in` without `voice_out`).

The weekly catalog sync sets both from DIDWW's number groups
([runbook](../../operations/number-catalog-sync.md)); the API purchase gate
skips marked rows
([`telephony_catalog.py`](../../../core/hailhq/core/telephony_catalog.py)).

## Inbound calls

A DID takes calls through the voice IN trunk it is assigned to. Hail assigns
the DID when it routes calls to an agent; you create the trunk once.

1. **Voice → Inbound Trunks → Create**: type SIP, host
   `<project>.sip.livekit.cloud`, port 5060, transport TCP. Copy the trunk
   id into `.env` as `DIDWW_VOICE_IN_TRUNK_ID`. Set `DIDWW_API_KEY`
   (**my.didww.com → API**).
2. Create the LiveKit inbound trunk and dispatch rule
   ([LiveKit Cloud §4](./livekit-cloud.md#4-inbound-calls)) and set
   `LIVEKIT_SIP_INBOUND_TRUNK_ID`.

`PATCH /v1/numbers/{id}` with `voice_agent_id` then assigns the DID to the
trunk (`PATCH /v3/dids/{id}`, relationship `voice_in_trunk`) and lists the
number on the LiveKit trunk; `null` clears both ([Agents](../agents.md)).
A DID added by hand without a stored DID id is looked up by number.
Reference: DIDWW [voice IN trunks](https://doc.didww.com/voice/inbound-trunks/index.html).

---

<!-- source: https://hail.so/docs/self-host/aws-ses -->

# AWS SES (email)

Outbound and inbound email go through [Amazon SES](https://aws.amazon.com/ses/) ([SESv2 API](https://docs.aws.amazon.com/ses/latest/APIReference-V2/API_Operations.html)). You need an AWS account and the SES service enabled in one region. You also need IAM-role credentials (recommended for EC2/ECS/EKS deployments) or a long-lived access key.

## 1. Credentials

The Python SDK uses the standard [boto3 credential chain](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html). If you run on AWS infrastructure with an attached IAM role, leave the keys empty in `.env`. Otherwise, set:

```bash
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=AKIA…
AWS_SECRET_ACCESS_KEY=…
```

Minimal IAM policy (one statement is sufficient for the v1 surface):

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ses:SendEmail",
        "ses:CreateEmailIdentity",
        "ses:GetEmailIdentity",
        "ses:DeleteEmailIdentity"
      ],
      "Resource": "*"
    }
  ]
}
```

## 2. Sandbox vs production

New SES accounts start in the [sandbox](https://docs.aws.amazon.com/ses/latest/dg/request-production-access.html): 200 messages/day, 1 message/second, and you can send only **to verified addresses**. When you are ready to send to arbitrary recipients, request production access from **SES → Account dashboard → Request production access**.

## 3. Why `mail.hail.so` (a subdomain), not `hail.so`

Always send transactional mail from a **dedicated subdomain**, not from your apex domain. There are two reasons:

- **Reputation isolation.** If a bounce spike or a spam-report cluster damages the sender reputation, only the subdomain takes the damage. Your apex domain (website, marketing email if any) stays clean.
- **Standard practice.** Postmark uses `mtasv.net`, and SendGrid uses `sendgrid.net`. Resend has tenants verify their own subdomains. Do not mix transactional volume with brand traffic on the apex domain — it is a known risk.

The recommended subdomain for the Hail public cloud is `mail.hail.so`. Self-hosters: select a subdomain that you own, for example `mail.<your-domain>`.

## 4. Set up the SES identity

This is operator setup, done once per deployment. Hail does not configure SES for the parent `HAIL_MAIL_BASE_DOMAIN`. That identity must exist, and be verified, before the first `POST /emails`. The operator also configures the MAIL FROM subdomain. Custom tenant domains follow a separate, fully automated flow: Hail calls `CreateEmailIdentity` **and** configures their MAIL FROM (refer to §7).

1. Open **AWS Console → SES → Verified identities → Create identity → Domain**. Enter the bare subdomain (`mail.hail.so`). Enable **DKIM** (the default; keep the bit-length at 2048).
2. SES returns three CNAMEs of the form `<token>._domainkey.mail.hail.so → <token>.dkim.amazonses.com`. **Publish all three** at your DNS provider. Wait until SES sets the status to **Verified** (usually less than 1 hour).
3. **Configure a custom MAIL FROM domain** — this step is **mandatory** for production deliverability (refer to §5 below). On the identity detail page, select **Edit MAIL FROM** and set the value to `bounces.mail.hail.so`. SES returns one MX record and one TXT record. Publish both at DNS:
   ```
   bounces.mail.hail.so  MX   10  feedback-smtp.us-east-1.amazonses.com
   bounces.mail.hail.so  TXT  "v=spf1 include:amazonses.com ~all"
   ```
4. Wait until the MAIL FROM domain status is **Success**.

> This MAIL FROM subdomain — `bounces.mail.hail.so` on the **operator parent** — is operator-managed: you configure it once, manually, in the steps above. **Custom tenant domains are different**: Hail configures their MAIL FROM automatically. `POST /email-domains` (kind=`custom`) calls `PutEmailIdentityMailFromAttributes` for `send.<domain>` and returns the MX + SPF records to publish together with the DKIM CNAMEs. `POST /email-domains/{id}/verify` then re-polls both the DKIM status and the MAIL FROM status. Refer to §7.

## 5. DMARC alignment (required for inbox delivery)

Without an aligned MAIL FROM, SES uses `<random>@amazonses.com` as the Return-Path. SPF then authenticates against `amazonses.com`, not against your domain. This breaks DMARC alignment, and the DMARC policies of recipients push your mail to spam.

With `bounces.mail.hail.so` as MAIL FROM:

- **SPF** authenticates against `bounces.mail.hail.so` → aligns with the From-domain `mail.hail.so`.
- **DKIM** signs with `mail.hail.so` → aligns.
- Both aligned → DMARC `pass` → inbox.

Publish a DMARC record. Start at `p=none` (monitor-only). Increase the policy after some weeks of clean reports:

```
_dmarc.mail.hail.so  TXT  "v=DMARC1; p=none; rua=mailto:dmarc-reports@hail.so; adkim=s; aspf=s"
```

Field summary: `p=none` reports but does not block (start here). `quarantine` moves unaligned mail to spam. `reject` drops it. `adkim=s` and `aspf=s` require strict alignment between the DKIM/SPF authentication domain and the From domain.

> **TODO(dmarc-ratchet):** Hail's public `mail.hail.so` is currently at `p=none`. After 30 or more days of clean DMARC aggregate reports (no unauthenticated mail in `rua=` feeds), step the policy to `p=quarantine`. Monitor for 30 more days, then move to `p=reject`. Self-hosters must follow the same staged rollout on their own subdomain.

## 6. Configure Hail

```bash
HAIL_MAIL_BASE_DOMAIN=mail.hail.so

# Single-variable form (self-hosters running a SINGLE org):
HAIL_MAIL_FROM=admin+selfhost@mail.hail.so

# Multi-tenant: leave HAIL_MAIL_FROM unset. The org prefix is derived
# per-org from the organization id; set only the default user prefix:
# HAIL_MAIL_DEFAULT_USER_PREFIX=admin

AWS_REGION=us-east-1
```

Hail-mail addresses always have the shape `<user>+<org>@<HAIL_MAIL_BASE_DOMAIN>` — for example `alice+acme@mail.hail.so`. Hail validates both `<user>` and `<org>` against `^[a-z0-9]([a-z0-9-]{0,18}[a-z0-9])?$` (1–20 characters, lowercase alphanumeric plus hyphens, no leading or trailing hyphen).

Precedence at send time (highest wins):

- **User prefix:** explicit `local_prefix_user` → `HAIL_MAIL_FROM` (user part) → `HAIL_MAIL_DEFAULT_USER_PREFIX`.
- **Org prefix:** explicit `local_prefix_org` → `HAIL_MAIL_FROM` (org part, single-tenant) → derived per-org from the organization id. The org prefix is never a deploy-wide constant — that constant would make every org collide on one address.

### Self-host vs managed

**Self-hosters**: there is no console, so the env vars _are_ the configuration. Set them in `.env` once, then restart. `POST /emails` then works without a prior `POST /email-domains` — the server auto-mints a hail-mail row from the env defaults on the first send.

**Managed cloud**: the website provisions the hail-mail row of each org at signup. It calls `POST /email-domains` with prefixes derived from the org slug and the user identity. Org admins then change the visible address through the console, which writes via `PATCH /email-domains/{id}`. The env vars provide deploy-time defaults but rarely surface to tenants directly.

## 7. Custom (tenant) domains

Tenants can register their own DNS-controlled domain:

```bash
curl -X POST $HAIL_API_URL/v1/email-domains \
  -H "Authorization: Bearer $HAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"kind":"custom","domain":"acme.com"}'
```

The response returns the full DNS record set to publish:

- **three DKIM `_domainkey` CNAMEs** — `<token>._domainkey.acme.com → <token>.dkim.amazonses.com`;
- **a MAIL FROM MX + SPF TXT on `send.acme.com`** — Hail configures the custom MAIL FROM automatically, so there is no AWS-console step for the tenant:
  ```
  send.acme.com  MX   10  feedback-smtp.<region>.amazonses.com
  send.acme.com  TXT      "v=spf1 include:amazonses.com ~all"
  ```

After the tenant publishes all records, the tenant calls `POST /email-domains/{id}/verify` to re-poll SES for **both** the DKIM status and the MAIL FROM status.

```bash
hail email domain register --kind custom --domain acme.com
# → prints the DKIM CNAMEs + the send.acme.com MAIL FROM records in a copy-pastable table
hail email domain verify <id>
# → re-polls SES; flips the row to verified once the records are live
```

> Inbound on a custom domain: after the row is `verified`, enable inbound (`forward_to` and/or a webhook) to receive mail. Matching is by identity, so each receiving domain yields its own inbound row + webhook. Receiving still relies on the operator's region-wide SES receipt rule (§10).

## 8. Send

```bash
hail email send --to alice@example.com --subject "hi" --body "hello"
```

Or via HTTP:

```bash
curl -X POST $HAIL_API_URL/v1/emails \
  -H "Authorization: Bearer $HAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "to": ["alice@example.com"],
    "subject": "hi from Hail",
    "body_text": "this just works"
  }'
```

The `from` field is optional. Resolution order:

1. Explicit `from` — must match a `verified` email_domain row owned by the caller's org.
2. The org's single verified domain. Own two or more and there is no default — the call returns `422` listing them, so `from` must be explicit.
3. The auto-minted hail-mail row, if `HAIL_MAIL_BASE_DOMAIN` and the prefixes are configured.

If none of these resolve, the call returns `503` with instructions to register a domain.

To see which address a `from`-less send would use, read `default_from` on `GET /email-domains` (`hail email domain list`). It is `null` when the org owns several verified identities.

The optional `from_name` field sets a display name on the `From:` header (`"Acme Billing" <billing@acme.com>`). Non-ASCII names are RFC-2047-encoded automatically; control characters are rejected with `422`.

## 8a. Attachments

Upload a file once and attach it to as many sends as you want. The limit is 25MB per upload and per send (body + all attachments combined, measured before base64 encoding). SES caps the encoded wire message at 40MB and bandwidth-throttles messages over 10MB.

### Upload a file

```bash
curl -s -X POST $HAIL_API_URL/v1/email-attachments \
  -H "Authorization: Bearer $HAIL_API_KEY" \
  -F "file=@invoice.pdf" | jq -r .id
# → "3fa85f64-5717-4562-b3fc-2c963f66afa6"
```

The response is a JSON object with an `id` field (UUID). Store this id to reference the attachment in sends.

### Attach to a send

Pass `attachment_ids` (a list of UUIDs) in the `POST /emails` payload:

```bash
curl -X POST $HAIL_API_URL/v1/emails \
  -H "Authorization: Bearer $HAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "to": ["alice@example.com"],
    "subject": "Invoice",
    "body_text": "See attached.",
    "recipient_consent": true,
    "attachment_ids": ["3fa85f64-5717-4562-b3fc-2c963f66afa6"]
  }'
```

You can attach the same uploaded file to multiple sends without a new upload. CLI shortcut — upload and attach in one step:

```bash
hail email send --to alice@example.com --subject "Invoice" --body "See attached." --attach invoice.pdf
```

### Lifecycle

Hail garbage-collects unused uploads (not attached to any send) 24 hours after upload. After you attach the file to a send, Hail retains it indefinitely. You can reuse it across as many messages as you want.

## 9. What v1 does not do

Skip these until later milestones. This list names them so that you do not use SES features that are not wired yet:

- **Templates** — the API takes raw `body_text` / `body_html`. SES templates are a v2 request.
- **Cloud-agnostic inbound** — the SMTP listener is described in [SMTP inbound](./smtp-inbound.md); inbound currently runs on AWS only.

## 10. Inbound email

To receive mail at `<user>+<org>@<HAIL_MAIL_BASE_DOMAIN>`, you need four things:

1. An MX record on `mail.hail.so` that points at SES inbound.
2. An S3 bucket that SES can write raw MIME into.
3. A SES Receipt Rule that writes the object and invokes a Lambda.
4. A small Lambda that signs the SES event and POSTs it to Hail.

A Terragrunt wrapper at `infra/terragrunt.hcl` automates provisioning
around the bare Terraform module in `infra/terraform/`. The wrapper
configures an S3-backed remote state with a DynamoDB lock table. It
pulls every input from the repo's `.env` — no parallel `tfvars` file.

### 10.1 Terragrunt apply

```bash
# .env must contain at minimum: AWS_PROFILE, AWS_REGION,
# HAIL_TERRAFORM_STATE_BUCKET, HAIL_TERRAFORM_LOCK_TABLE, HAIL_API_URL,
# HAIL_INBOUND_HMAC_SECRET (generate: openssl rand -hex 32),
# HAIL_MAIL_BASE_DOMAIN.

cd infra
terragrunt init                # Terragrunt sources .env automatically via
terragrunt plan                # `run_cmd`; no manual export step.
terragrunt apply
```

Do a one-time bootstrap per AWS account before the first `terragrunt init`:
the state bucket + lock table do not auto-create. Refer to the comment block
at the top of [`infra/terragrunt.hcl`](https://github.com/hail-hq/hail/blob/main/infra/terragrunt.hcl) for
the AWS CLI one-liners. Refer to [operations](./operations.md) →
"Inbound email rollout → Stage 4" for the full sequence.

Outputs:

- `inbound_mx_record` — publish at DNS for `HAIL_MAIL_BASE_DOMAIN`.
- `inbound_bucket` — this is `${HAIL_MAIL_NAME_PREFIX}-mail`. Set `HAIL_MAIL_NAME_PREFIX` in the API `.env` to match the Terraform `name_prefix` var. The bucket name is not settable directly — there is no `HAIL_MAIL_BUCKET` var.
- `activate_command` — the `aws ses set-active-receipt-rule-set ...` to run once.

The bare Terraform module at `infra/terraform/` is provider-vanilla. If
you prefer to skip Terragrunt, it still works with
`terraform apply -var=...`.

### 10.2 Activate the receipt rule set (manual)

SES has **one active receipt rule set per region per AWS account.** The module
creates the rule set but does **not** activate it. If an account already has
another rule set active, activation is destructive.

- **Greenfield AWS account**: run the `activate_command` output verbatim.
- **Account with existing rules**: import the existing rule set into Terraform
  state and merge Hail's rule into it. As an alternative, skip the module's rule
  resource and add Hail's rule manually via the AWS console.

### 10.3 Publish the MX record

At your DNS provider, publish what the Terraform output prints, for example:

```
mail.hail.so  MX  10  inbound-smtp.us-east-1.amazonaws.com
```

### 10.4 Configure Hail

In the API service `.env`:

```bash
HAIL_INBOUND_ENABLED=true
HAIL_MAIL_NAME_PREFIX=hail-inbound-prod         # matches Terraform `name_prefix`; bucket = ${prefix}-mail
HAIL_INBOUND_HMAC_SECRET=<same as Terraform var>
```

Restart `api`. Send a test mail to a hail-mail address and confirm:

```bash
curl "$HAIL_API_URL/v1/emails?direction=inbound" \
  -H "Authorization: Bearer $HAIL_API_KEY"
```

Or, with the Python SDK:

```python
emails = await client.emails.list(direction="inbound")
```

If the API is down beyond the Lambda's async retries, failed deliveries land in
the `<name_prefix>-ingest-dlq` SQS queue (`ingest_dlq_url` terraform output). To
replay, re-drive each message's body at `POST /internal/ses-events`. The raw
MIME is still in S3.

## Delivery & engagement events

Outbound sends carry the SES configuration set named by
`HAIL_SES_CONFIGURATION_SET` (Terraform default: `hail-events`). SES
publishes Delivery / Bounce / Complaint / Reject / DeliveryDelay / Open /
Click events to SNS. The ingest Lambda relays them to
`POST /internal/ses-events`, which records them in `email_events`,
advances `emails.status`, and fans out webhooks.

Check a single email's timeline:

```bash
hail email events <email-id>
```

Account-level stats:

```bash
hail email stats --from 2026-06-01T00:00:00Z --bucket day
```

Notes:

- Open/Click tracking rewrites links through the default SES tracking
  domain (`awstrack.me`) unless you set up a [tracking domain](#tracking-domain).
- Open counts are approximate (mail clients that proxy images inflate them).
- Hail acknowledges and drops events for mail sent outside Hail from the
  same SES account (`status: unmatched` in the API log).
- Hail re-sends forwarded inbound mail (refer to §10.5) as normal outbound.
  It carries the config set and writes a synthetic `sent` event, so it
  counts in `/emails/stats` like any other send.

### Tracking domain

Tracked links and the open pixel point at `awstrack.me` by default. Ad
blockers skip or block that host, so clicks from those recipients never
reach SES and no `clicked` event is stored. To use your own host name
(example: `go.example.com`), do these steps **in this order**:

```bash
# 1. DNS: point the host at the VM, same target as api.<HAIL_DOMAIN>.
#    go.example.com  CNAME  api.example.com

# 2. VM .env, then redeploy (Caddy gets a certificate and starts proxying):
HAIL_TRACKING_DOMAIN=go.example.com

# 3. Check the proxy. Both x-amz-ses-* headers must be present, the region
#    must be your AWS_REGION and the protocol must be https.
curl --head https://go.example.com/favicon.ico

# 4. Create the SES identity for the host and print its DKIM records.
#    In the .env that Terragrunt reads:
HAIL_SES_TRACKING_DOMAIN=go.example.com
cd infra && terragrunt apply
terragrunt output ses_tracking_domain_dkim_records

# 5. Publish the 3 CNAME records. Wait until the identity is verified.
#    Use your AWS_REGION (and AWS_PROFILE) or the lookup returns NotFound.
aws sesv2 get-email-identity --email-identity go.example.com \
  --region us-east-1 --query VerifiedForSendingStatus

# 6. Switch the configuration set. New emails use the host from now on.
#    In the same .env:
HAIL_SES_TRACKING_ENABLED=true
terragrunt apply
```

Step 4 only creates the identity, so other applies keep working while the DKIM
records propagate. SES refuses step 6 until step 5 reports `true`. Do not run step 6 before
step 3 passes: SES would write a host into every link that does not answer.
Links in emails sent earlier keep their `awstrack.me` host and keep working.
While the VM is down, links on your tracking domain do not open.
To go back, empty `HAIL_SES_TRACKING_ENABLED` and run `terragrunt apply`. Keep
`HAIL_TRACKING_DOMAIN` and the DNS record: links in emails already sent with
your host stop working the moment the proxy goes away.
The proxy lives in [`Caddyfile`](https://github.com/hail-hq/hail/blob/main/Caddyfile);
the SES side in [`infra/terraform/ses_events.tf`](https://github.com/hail-hq/hail/blob/main/infra/terraform/ses_events.tf).

### 10.5 Forwarding and webhooks

Tenants configure routing per `email_domains` row:

```bash
# Forward every inbound on the org's hail-mail address to a real inbox
curl -X PATCH $HAIL_API_URL/v1/email-domains/$DOMAIN_ID \
  -H "Authorization: Bearer $HAIL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"inbound_enabled":true,"forward_to":["team@acme.com"]}'

# Or POST inbound events to a webhook URL — the response carries the secret once
curl -X PATCH $HAIL_API_URL/v1/email-domains/$DOMAIN_ID \
  -H "Authorization: Bearer $HAIL_API_KEY" \
  -d '{"inbound_enabled":true,"webhook_url":"https://hooks.acme.com/hail"}'
```

For org-wide multi-event delivery (firehose pattern):

```bash
curl -X POST $HAIL_API_URL/v1/webhooks \
  -H "Authorization: Bearer $HAIL_API_KEY" \
  -d '{"target_url":"https://hooks.acme.com/all","event_types":["email.received","email.bounced","email.complained"]}'
```

### 10.6 Webhook secrets at rest

Hail encrypts webhook signing secrets **at rest** with a deployment-scoped
[Fernet](https://cryptography.io/en/latest/fernet/) key
([`core/hailhq/core/secret_cipher.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/secret_cipher.py)).
The worker decrypts on each delivery, so deliveries survive API restarts and
work across multi-process deployments. If `HAIL_WEBHOOK_SECRET_KEY` is unset,
webhook creation returns `500`. Generate and set the key before you enable
webhooks:

```bash
# Generate a key (run once; store in .env, never commit). Run inside the
# project venv so the `cryptography` package is on PYTHONPATH.
uv run --directory core python -c "from hailhq.core.secret_cipher import generate_key; print(generate_key())"

# In .env:
HAIL_WEBHOOK_SECRET_KEY=<output above>
```

## Reference

- [SES sending limits](https://docs.aws.amazon.com/ses/latest/dg/manage-sending-quotas.html)
- [SESv2 API](https://docs.aws.amazon.com/ses/latest/APIReference-V2/Welcome.html)
- [DKIM in SES](https://docs.aws.amazon.com/ses/latest/dg/send-email-authentication-dkim.html)
- [Custom MAIL FROM](https://docs.aws.amazon.com/ses/latest/dg/mail-from.html)
- [DMARC overview](https://dmarc.org/overview/)
- OpenAPI: [`openapi/openapi.yaml`](https://github.com/hail-hq/hail/blob/main/openapi/openapi.yaml) → `/emails`, `/email-domains`, `/webhooks` tags
- Code paths: [`api/hailhq/api/routes/emails.py`](https://github.com/hail-hq/hail/blob/main/api/hailhq/api/routes/emails.py), [`api/hailhq/api/routes/email_domains.py`](https://github.com/hail-hq/hail/blob/main/api/hailhq/api/routes/email_domains.py), [`api/hailhq/api/routes/webhooks.py`](https://github.com/hail-hq/hail/blob/main/api/hailhq/api/routes/webhooks.py), [`core/hailhq/core/providers/email/ses.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/providers/email/ses.py)
- Inbound infra: [`infra/terraform/`](https://github.com/hail-hq/hail/blob/main/infra/terraform), [`infra/ses-ingest-lambda/`](https://github.com/hail-hq/hail/blob/main/infra/ses-ingest-lambda)
- Design spec: [`docs/superpowers/specs/2026-06-06-inbound-email-design.md`](https://github.com/hail-hq/hail/blob/main/docs/superpowers/specs/2026-06-06-inbound-email-design.md)

---

<!-- source: https://hail.so/docs/self-host/smtp-inbound -->

# SMTP inbound — not yet implemented

The `SmtpInboundProvider` interface exists in
[`core/hailhq/core/providers/email/inbound/smtp.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/providers/email/inbound/smtp.py)
but is not implemented. It is the cloud-agnostic / OSS-only path.
We defer it to a follow-up milestone.

When it is released, this page will describe:

- the `mailbot/` container (`aiosmtpd`-backed), parallel to `voicebot/`
- listen ports + TLS configuration
- the "front me with Maddy or Postfix" production recipe for SPF / DKIM /
  DMARC verification and flood resistance
- self-host quickstart

Until then, use the SES-backed inbound path documented in
[AWS SES](./aws-ses.md). If you must avoid AWS, file an
issue that tracks your need. Then we can give the SMTP listener priority.

## References

- Design spec: [`docs/superpowers/specs/2026-06-06-inbound-email-design.md`](https://github.com/hail-hq/hail/blob/main/docs/superpowers/specs/2026-06-06-inbound-email-design.md) §2
- Provider interface: [`core/hailhq/core/providers/email/inbound/base.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/providers/email/inbound/base.py)
- Placeholder stub: [`core/hailhq/core/providers/email/inbound/smtp.py`](https://github.com/hail-hq/hail/blob/main/core/hailhq/core/providers/email/inbound/smtp.py)

---

<!-- source: https://hail.so/docs/self-host/operations -->

# Operations runbook

This document is the single source of truth for how to develop, deploy, migrate, and release Hail. If you are an AI agent that starts work on this codebase, **read this document first** together with `CLAUDE.md`.

## Quick reference

| task                           | command                                                                                                                                   |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Bring up stack (bundled DB)    | `docker compose -f docker-compose.yml -f docker-compose.local.yml up -d`                                                                  |
| Bring up stack (managed DB)    | `docker compose up -d`                                                                                                                    |
| Bring up stack (prod VM)       | `docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d` — pulls images from GHCR; refer to the [VM guide](./vm-deploy.md) |
| Tail one service               | `docker compose logs -f <api\|voicebot\|mcp\|postgres>`                                                                                   |
| Run all tests                  | `cd <core\|api\|voicebot\|mcp\|sdk> && uv run pytest` (per suite, **from each dir**)                                                      |
| Lint                           | `uvx ruff check .` then `uvx black --check .` (repo root)                                                                                 |
| Apply DB migrations            | `docker compose run --rm api alembic upgrade head`                                                                                        |
| Regenerate OpenAPI + Go client | refer to _Development → Regenerating OpenAPI_ below                                                                                       |
| Publish SDK                    | tag `sdk-v<X.Y.Z>` and push (fires `release-sdk.yml`)                                                                                     |
| Publish CLI                    | tag `cli-v<X.Y.Z>` and push (fires `release-cli.yml`)                                                                                     |

## Local development

### Bringing up the stack

```bash
cp .env.example .env                                       # then fill in keys
pnpm install                                               # husky pre-commit hooks
docker compose \
  -f docker-compose.yml -f docker-compose.local.yml \
  up -d                                                    # postgres + api + voicebot + mcp
docker compose run --rm api alembic upgrade head           # apply schema
# bind a phone number to the self-host sentinel (see first-run setup below)
```

### Per-service dev loops (host-side, no Docker)

```bash
cd api      && uv run uvicorn hailhq.api.main:app --reload --port 8080
cd voicebot && uv run python -m hailhq.voicebot.main start
cd mcp      && uv run uvicorn hailhq.mcp.server:app --reload --port 8081
cd cli      && go run . <args>
```

Before host-side development, export `.env` into the shell: `set -a; source .env; set +a`. Pydantic Settings reads `.env` into Settings attributes, but plugin SDKs read `os.environ` directly.

### Tests

CI runs each suite from its own directory. Match that locally:

```bash
cd core     && uv run pytest -v
cd api      && uv run pytest -v
cd voicebot && uv run pytest -v
cd mcp      && uv run pytest -v
cd sdk      && uv run pytest -v
cd cli      && go test ./... && go vet ./...
```

Python tests use **testcontainers/postgres** locally (this starts a Postgres container automatically). When the `DATABASE_URL` env var is set, the tests use it instead (this is the CI path).

### Lint + format

Pre-commit runs `ruff check --fix`, `black`, `gofmt -w`, and `prettier --write` on staged files via husky + lint-staged. To run the checks manually:

```bash
uvx ruff check .            # at repo root
uvx black --check .         # at repo root
cd cli && gofmt -l . && go vet ./...
```

### Regenerating OpenAPI + Go CLI client

When API routes change:

```bash
# 1. Boot the API (or just import the app) and dump the spec
cd api && uv run python -c "from hailhq.api.main import app; import sys, yaml; yaml.safe_dump(app.openapi(), sys.stdout, sort_keys=False)" > ../openapi/openapi.yaml

# 2. Regenerate the Go client (consumes openapi.yaml via a build-tagged
#    preprocessor that downgrades to OpenAPI 3.0.3 — oapi-codegen v2
#    doesn't yet parse 3.1's anyOf:[type,null] nullable idiom).
cd cli && make codegen
```

Commit `openapi/openapi.yaml` and `cli/internal/client/client.gen.go` together with the route change.

### Adding a new env var

1. Add the field to `core/hailhq/core/config.py` `Settings` class.
2. Add the env line to `.env.example` under the right provider section (provider-grouped convention).
3. If code consumes the value, reference `settings.<field>`. If a LiveKit plugin consumes the value implicitly through `os.environ`, the Settings declaration is **documentation only**. In that case, the runtime path is `docker compose env_file: .env`, which exports the value to the container.

### Adding a new provider adapter

Put new adapters under `core/hailhq/core/providers/<channel>/<name>.py`. Each adapter implements the adapter interface of its channel (for example, `VoiceProvider` in `providers/voice/base.py`). `api/` and `voicebot/` must **not** import provider SDKs directly. Go through `core`.

## Deployment (self-host)

### Required external accounts

- **Twilio**: account SID + auth token + a phone number with voice capability + an Elastic SIP Trunk (Origination URI → LiveKit's inbound, Termination → Twilio's PSTN).
- **Telnyx** (optional second carrier): API key + outbound voice profile + SIP connection + LiveKit trunk + webhook public key ([Telnyx](./telnyx.md)). Quotes compare it with Twilio.
- **LiveKit Cloud**: project + URL + API key + secret + an outbound SIP trunk (`LIVEKIT_TWILIO_SIP_OUTBOUND_TRUNK_ID`) + one inbound trunk shared by every carrier (`LIVEKIT_SIP_INBOUND_TRUNK_ID`, [LiveKit Cloud §4](./livekit-cloud.md#4-inbound-calls)). Optional per-carrier trunks: `LIVEKIT_TELNYX_SIP_OUTBOUND_TRUNK_ID`, `LIVEKIT_DIDWW_SIP_OUTBOUND_TRUNK_ID` ([DIDWW](./didww.md)).
- **Deepgram** (STT): API key. Required; used for semantic turn detection and as the fallback when Speechmatics is unavailable.
- **Speechmatics** (STT, optional): API key. Enables language-specific STT routing and end-of-utterance detection for 22 languages. Deepgram-only self-hosts keep working; if absent, calls fall back to Deepgram with VAD turn detection.
- **Cartesia** (primary TTS): API key + a voice ID from the Cartesia voice library.
- **ElevenLabs** (fallback TTS, optional): API key + a voice ID. If `ELEVEN_API_KEY` is set, the system uses it automatically when Cartesia fails.
- **At least one LLM provider**: OpenAI / Gemini / Anthropic API key. The voicebot's mode-A FallbackAdapter chains all three. Mode-B uses a caller-provided OpenAI-compatible endpoint for each call.

For detailed setup walkthroughs, see [Twilio](./twilio.md),
[LiveKit Cloud](./livekit-cloud.md), and [MCP](../mcp.md). To run the stack on a
single Ubuntu VM with HTTPS and automatic deployment from `main`, see the
[VM deployment guide](./vm-deploy.md).

### Authentication

Self-host and managed cloud share the same FastAPI binary, but the contents of the env decide the auth mode implicitly:

| Mode                    | Trigger                                                                | What hail/api checks                                                                                                                                                                              |
| ----------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Self-host** (default) | Operator sets `HAIL_API_KEY` in `.env`                                 | Constant-time compare against the env var. All shared-key requests resolve to the sentinel `organization_id = 00000000-0000-0000-0000-000000000000` (the nil UUID) — no DB row, no member lookup. |
| **Managed cloud**       | The auth backend's `apikey` table is migrated into the shared Postgres | Hashes the bearer with `base64url(sha256())` and looks it up; resolves the org via `members.user_id = api_keys.reference_id → members.organization_id`.                                           |

Both modes can be active at the same time. If `HAIL_API_KEY` is set in managed cloud, it operates as a master/admin override that always works.

A managed-cloud user with no `member` row gets a **403 "user not provisioned"**, not a fabricated org. Provisioning is the responsibility of the website (through its `user.create.after` hook).

#### Self-host: first-run setup

```bash
# 1) Generate a shared API key — used for BOTH directions:
#    inbound (API checks bearer) + outbound (CLI/MCP/voicebot send it).
HAIL_API_KEY="hk_$(openssl rand -base64 32 | tr -d '/+=' | head -c 40)"
printf '%s\n' "$HAIL_API_KEY"
# Replace the blank HAIL_API_KEY= line in .env with this value.

# 2) Add a phone number bound to the self-host sentinel org id.
export TWILIO_E164='+1XXXXXXXXXX' TWILIO_PN_SID='PNxxxxxxxxxxxxxxxx'
docker compose exec postgres psql -U hail -d hail -c "INSERT INTO phone_numbers (organization_id, e164, country_code, number_type, capabilities, provider, provider_resource_id, provisioning_state, acquired_at) VALUES ('00000000-0000-0000-0000-000000000000', '${TWILIO_E164}', 'US', 'local', ARRAY['voice','sms'], 'twilio', '${TWILIO_PN_SID}', 'active', now());"
```

There is no recovery path for a lost key, so save the value of `HAIL_API_KEY`. Then run `export HAIL_API_KEY=…` in the shell that runs `hail`, or pass the key with `--api-key`.

> **`hail login` is managed-cloud only.** It runs the auth backend's device flow against `hail-website` and writes the resulting `hl_live_*` key to `~/.hail/credentials.json`. In self-host, there is no website to authorize against. Set `HAIL_API_KEY` directly, and you are done.

#### `hail auth` subcommands

For interactive sessions on a managed Hail deployment:

- `hail login` — runs the device-authorization flow and persists `~/.hail/credentials.json`.
- `hail auth logout` — deletes the local credentials file (idempotent).
- `hail auth token` — prints the bare API key. Use it in scripts as
  `export HAIL_API_KEY=$(hail auth token)`.

Self-hosters usually skip the device flow and set `HAIL_API_KEY`
directly, as the bootstrap section above shows.

#### Phone number pool

Pool numbers are unowned `phone_numbers` rows (`is_pool=TRUE`, `organization_id IS NULL`). An org without its own active number falls back to them on outbound calls. The claim is atomic (`SELECT … FOR UPDATE SKIP LOCKED`, in randomized order to spread carrier wear). Each number binds to one call at a time through `reserved_call_id`; the system releases it when the call ends. Implementation: `core/hailhq/core/pool.py`. The sweeper backstop window is `HAIL_POOL_RELEASE_GRACE_SECONDS`.

Add a Twilio number to the pool with `organization_id` NULL and `is_pool=TRUE` (the CHECK constraint enforces the pairing):

```bash
psql "$DATABASE_URL" -c "INSERT INTO phone_numbers (organization_id, e164, country_code, number_type, capabilities, provider, provider_resource_id, provisioning_state, is_pool, acquired_at) VALUES (NULL, '+1XXXXXXXXXX', 'US', 'local', ARRAY['voice','sms'], 'twilio', 'PNxxxxxxxxxxxxxxxx', 'active', TRUE, now());"
```

Attach the number to the same Twilio SIP trunk that you wired in [Twilio setup](./twilio.md). The trunk is chosen per carrier (`provider` column: `twilio` or `didww`, see [DIDWW](./didww.md)), not per number. To grow the pool, repeat the INSERT with a different `e164` / `PN_SID`. To quarantine a bad pool number without deletion, run `UPDATE phone_numbers SET provisioning_state='failed' WHERE e164=...`. The claim query skips non-`active` rows.

Callers cannot address a pool number explicitly with the `from` field of `POST /calls`. The number is shared, so a caller that names one would cross tenants. The fallback fires only when an org has zero active numbers of its own.

#### Managed cloud

Run `hail login`. The CLI opens `/device` on the website. You approve, and the CLI exchanges the device-flow session for a long-lived `hl_live_*` key, which the auth backend mints into the `apikey` table. `hail/api` reads the same table, so keys minted in the console work everywhere (CLI, MCP, direct API calls).

### MCP modes

The MCP service (`hail/mcp`) picks one of two modes at boot from env:

| Mode           | Env                                                            | Behaviour                                                                                                                                                                                  |
| -------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **oauth-rs**   | `HAIL_AUTH_URL` + `MCP_RESOURCE_URL` set, `HAIL_API_KEY` empty | FastMCP rejects unauth requests with `401 WWW-Authenticate: Bearer resource_metadata=…`; tools forward each request's JWT to the API; `.well-known/oauth-protected-resource` is published. |
| **static-key** | `HAIL_API_KEY` set, `HAIL_AUTH_URL` empty                      | No inbound auth; tools use the singleton `HailClient(api_key=HAIL_API_KEY)`; no protected-resource route.                                                                                  |

If both are set, boot fails with `ambiguous MCP auth config`. If neither is set, boot fails with `MCP auth not configured`. The service decides the mode once; a restart is necessary to change it.

The MCP service does not validate JWT signatures. `hail/api` is the single source of JWT-validation truth (`HAIL_AUTH_URL`, `HAIL_AUTH_AUDIENCES`). MCP forwards the bearer on the outbound call. The API validates the token and resolves the org.

## Database migrations

Schema lives in `api/migrations/versions/`. Alembic config: `api/alembic.ini`. The `DATABASE_URL` env var overrides `sqlalchemy.url`.

```bash
# Apply all pending
docker compose run --rm api alembic upgrade head
# OR host-side (needs DATABASE_URL exported):
cd api && uv run alembic upgrade head

# Author a new migration (hand-written raw SQL via op.execute)
cd api && uv run alembic revision -m "add foo column"

# Revert one revision
cd api && uv run alembic downgrade -1
```

For v1, write migrations as raw SQL by hand (`env.py` has no SQLAlchemy `target_metadata` wired in). Switch to `--autogenerate` when models become the source of truth.

### Migration discipline

**The deploy workflow runs `alembic upgrade head` against prod on every push to `main`.** There is no human review gate between the merge of a migration and its execution. Three rules keep this safe. The workflow enforces one; the other two depend on you:

1. **`api/migrations/env.py` caps statement runtime at 120s and lock-acquisition at 5s** via `SET LOCAL statement_timeout` / `SET LOCAL lock_timeout`. It issues them inside the migration transaction, so they apply to every statement that the migration runs. The choice of `SET LOCAL` (not `PGOPTIONS` or session-level `SET`) is deliberate. Neon's `-pooler` endpoint and other PgBouncer transaction-pooled connections reject the libpq startup option and reset session state between transactions. The per-transaction form is the only one that works on all of pooled / unpooled / direct.

   The deploy log also prints `alembic current` and `alembic upgrade head --sql` before it applies the migration. Thus a problem migration is visible in CI output before it touches data.

   If a legitimate migration needs more time (a backfill, `CREATE INDEX CONCURRENTLY`), bypass the guard: put its own `SET LOCAL statement_timeout = '<bigger>'` at the top of the migration body. Make the change explicit for that migration; do not raise the global cap. Bigger caps on every migration let a runaway migration use more wall-clock time before it aborts.

2. **Split all destructive shape changes into expand → backfill → contract across separate releases.** A column drop, a type narrowing, or a `NOT NULL` on existing data must not go in the same release as the code that depends on the new shape. The old containers still run when the migration starts. For example:

   | bad: one release                                | good: three releases                                                                                                            |
   | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
   | drop `phone_numbers.legacy_field` + remove code | release 1: stop writing `legacy_field` (still readable) → release 2: backfill / verify → release 3: `ALTER TABLE … DROP COLUMN` |
   | rename `calls.requested_at` → `calls.queued_at` | release 1: add new column, dual-write → release 2: backfill, switch reads → release 3: drop old column                          |

3. **Never edit, renumber, or remove a migration that prod has applied.** Author a new revision instead. Treat each revision at or below the previous deploy's `alembic current` as immutable. A renumber or history rewrite of a deployed migration is the same footgun as an edit. Prod's `alembic_version` still holds the old revision id. The next `alembic upgrade head` then cannot locate it (`Can't locate revision '<id>'`), or it silently skips the real DDL and collides downstream (`relation "…" already exists`).

   CI never catches this, because test fixtures build the schema via `Base.metadata.create_all`, not alembic. Recovery is manual. Apply the migrations that never ran on prod, then reconcile the pointer with `alembic stamp <head>` (or `UPDATE alembic_version`) once the schema matches. Refer to the Footguns table.

The expand/contract rule is the one that the workflow cannot enforce. If you break it, auto-deploy releases the footgun. The rollback is then "revert the code, write a new migration to restore the column, re-deploy" — hours of incident, not minutes.

### Cross-migration table ownership

Two services migrate the same Postgres database independently:

| Owner            | Migration tool                                                  | Tables                                                                                                                  |
| ---------------- | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| **hail-website** | `pnpm dlx @better-auth/cli migrate` (introspects `lib/auth.ts`) | `users`, `accounts`, `sessions`, `verifications`, `device_codes`, `api_keys`, `organizations`, `members`, `invitations` |
| **hail/api**     | alembic                                                         | `account_credits`, `usage_events`, `phone_numbers`, `calls`, `call_events`, `idempotency_keys`, `audit_log`             |

The schema source of truth for the website is `lib/auth.ts`. After each change to that file, run `pnpm dlx @better-auth/cli generate -y` to emit the resulting SQL into `better-auth_migrations/`, and commit it (audit trail). Then run `pnpm dlx @better-auth/cli migrate -y` to apply the diff against the target DB. Both subcommands introspect the auth config. `generate` writes the SQL file; `migrate` runs the same diff against the DB.

Columns in hail/api-owned tables that reference a website-owned table — `organization_id` everywhere, `audit_log.api_key_id` — carry **no foreign-key constraint**. This is deliberate.

**Why no FK:** the CLI regenerates the website schema from a TypeScript config on every relevant version bump. If alembic held a hard FK into `organizations(id)`, each rename or shape change on the website side would need a coordinated alembic migration in the same release, or it would break the next `alembic upgrade head`. Cross-tool referential integrity is a coordination tax we do not want to pay on every dependency bump.

**What we trade off:**

- No `ON DELETE CASCADE` from the auth side. If you delete an `organization` row in Postgres, orphaned rows stay in `account_credits`, `calls`, etc. v1 does not hard-delete orgs. If you start to do so, write a sweep query or a soft-delete.
- No DB-level guarantee that `organization_id` points at a real row. The auth flow in `api/hailhq/api/deps.py` validates that the org exists on every authenticated request, so application-layer integrity holds for the live path. Bulk inserts from migrations or fixtures must keep this integrity themselves.

**When to break the rule:** if you add a new hail/api-owned table that joins to another hail/api-owned table (for example, `call_events.call_id → calls.id`), keep the FK. Both ends are alembic-owned, so the constraint is safe.

### Shared-key sentinel

Shared-key (`HAIL_API_KEY`) requests resolve to `organization_id = 00000000-0000-0000-0000-000000000000` (the nil UUID) — a sentinel, not a real row. Nothing seeds it; nothing reads from `organizations` for that path. Self-host operators can attach `phone_numbers`, `account_credits`, etc. to the sentinel. To do so, pass the nil UUID as the org id directly (refer to _Self-host: first-run setup_ above for the phone-number example).

### Switching the database

Compose comes as two files. `docker-compose.yml` is the deployable base and assumes that `DATABASE_URL` reaches a Postgres you bring. `docker-compose.local.yml` is a thin overlay that adds a bundled `postgres` container and merges a `depends_on: postgres` into `api` and `voicebot`. Pick a mode:

**Bundled local Postgres** — layer both files:

```bash
docker compose -f docker-compose.yml -f docker-compose.local.yml up -d
```

`.env` must keep the default `DATABASE_URL=postgresql://hail:hail@postgres:5432/hail` (the in-network compose hostname).

**Reset the bundled DB** (this removes all data for a fresh start):

```bash
docker compose -f docker-compose.yml -f docker-compose.local.yml down -v   # -v removes volumes
docker compose -f docker-compose.yml -f docker-compose.local.yml up -d postgres
docker compose -f docker-compose.yml -f docker-compose.local.yml run --rm api alembic upgrade head
# re-seed (Phase above)
docker compose -f docker-compose.yml -f docker-compose.local.yml up -d
```

**Managed Postgres (Neon / Supabase / RDS / …)** — use only the base file:

```bash
# 1. Provision the DB; grab the postgres URL (sslmode=require for most hosted providers)
# 2. Edit .env:
#      DATABASE_URL=postgresql://USER:PASSWORD@HOST/DBNAME?sslmode=require
#    Comment out the bundled-local line.
# 3. Apply migrations
docker compose run --rm api alembic upgrade head
# 4. Re-seed (same SQL as above; replace `docker compose exec -T postgres
#    psql -U hail -d hail -c` with `psql "$DATABASE_URL" -c`)
# 5. Bring up the stack — no `-f docker-compose.local.yml`, so no postgres
#    container is started:
docker compose up -d
```

The migration `0001_initial.py` issues `CREATE EXTENSION IF NOT EXISTS pgcrypto;`. If a hosted provider gates extensions, allow `pgcrypto` on it.

## Releases

Tag conventions:

| tag prefix     | what fires                          | what it produces                                                                                                                                                |
| -------------- | ----------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sdk-v<X.Y.Z>` | `.github/workflows/release-sdk.yml` | `hail-sdk` on PyPI (trusted publishing — no token)                                                                                                              |
| `cli-v<X.Y.Z>` | `.github/workflows/release-cli.yml` | GoReleaser → multi-arch binaries on GitHub Releases + Homebrew formula push to `hail-hq/homebrew-tap`                                                           |
| `v<X.Y.Z>`     | nothing directly                    | **owned by the CLI release.** GoReleaser strips the `cli-` prefix, so `cli-v<X.Y.Z>` publishes its release page under `v<X.Y.Z>`. Never tag `v<X.Y.Z>` by hand. |

There is no umbrella release tag. `CHANGELOG.md` is the record of what each repo
version contains; the merge commit is the pointer. A hand-made `v<X.Y.Z>` tag
collides with the CLI's release namespace — it did once (2026-08-13), and the CLI
release then published its binaries onto a page whose tag pointed at older code.
`release-cli.yml` now fails loudly if that tag already exists somewhere else.

Hail does **not** release the service images (`hail-api`, `hail-voicebot`, `hail-mcp`) as versioned artifacts. The deploy workflow (`.github/workflows/deploy.yml`) pushes `latest` and `sha-<commit>` tags to GHCR on each push to `main` — the production VM pulls these. Self-hosters build from source via `docker compose up`. Versioned image releases are on the v1.x list.

The images install exactly the versions in `uv.lock` (`uv sync --frozen`), never newer ones at build time. To move to newer releases, run `uv lock --upgrade` in a PR; CI tests the new lock before it reaches an image.

### Releasing the SDK (Python)

```bash
# 1. Bump version in sdk/pyproject.toml (e.g., 0.1.0 → 0.2.0)
# 2. Commit
# 3. Tag and push
git tag sdk-v0.2.0
git push origin sdk-v0.2.0
# 4. Watch https://github.com/hail-hq/hail/actions
```

Pre-flight (one-time): configure a PyPI Trusted Publisher for `hail-sdk` that points at the `hail-hq/hail` repo + the `release-sdk.yml` workflow. After the project exists on PyPI, promote the _Pending_ publisher to a normal one (PyPI web UI).

### Releasing the CLI (Go)

```bash
# 1. Commit any CLI / config changes
# 2. Tag and push
git tag cli-v0.2.0
git push origin cli-v0.2.0
# 3. Watch the workflow; on green:
brew update && brew upgrade hail-hq/tap/hail
```

Required secret: `HOMEBREW_TAP_TOKEN` — a fine-grained PAT with **Contents: read+write** on `hail-hq/homebrew-tap`. Set it under repo Settings → Secrets → Actions. Without it, the binary build + GitHub Release succeed, but the formula push fails.

GoReleaser quirks to know:

- OSS GoReleaser does not have the Pro `monorepo:` block. The workflow strips the `cli-` prefix into the `GORELEASER_CURRENT_TAG` / `GORELEASER_PREVIOUS_TAG` env vars. It also passes `--skip=validate`, so GoReleaser does not reject the env-var-overridden tag. The manual `git rev-parse` and `git diff --exit-code HEAD` steps before GoReleaser keep the validate checks that _do_ matter.
- The snapshot version template is the literal `0.0.0-snapshot-{{ .ShortCommit }}` (not `incpatch`), because the repo carries non-semver tags like `sdk-v0.0.1` that confuse the parser.

### Cutting a repo release

A repo release is a CHANGELOG section, not a tag. Only components carry tags.

```bash
# 1. Move CHANGELOG.md's [Unreleased] block under the new version heading
# 2. Tick newly-shipped milestones in README.md
# 3. Bump sdk/pyproject.toml + sdk/hail/__init__.py if the SDK ships, then `uv lock`
# 4. Commit as `chore(release): <X.Y.Z> — <summary>`, merge to main
# 5. Tag the components that actually changed
git tag sdk-v0.15.0 cli-v0.20.0
git push origin sdk-v0.15.0 cli-v0.20.0
```

Do **not** tag a bare `v<X.Y.Z>`. That name belongs to the CLI release: GoReleaser
strips the `cli-` prefix, so `cli-v0.20.0` publishes its page under `v0.20.0`. A
hand-made `v<X.Y.Z>` makes the next CLI release upload its binaries onto whatever
commit that tag already points at. `release-cli.yml` fails the release when it
detects this.

The CLI release page carries the CHANGELOG prose for the matching repo version, so
edit its body after GoReleaser publishes.

## Conventions

- **Commits**: [Conventional Commits](https://www.conventionalcommits.org). `feat(scope): …`, `fix(scope): …`, `chore: …`, `docs(scope): …`, etc.
- **Markdown**: GitHub-flavored only. Use the binary task-list states `[ ]` / `[x]`; do not use the non-GFM `[~]` / `[-]`.
- **Python namespaces**:
  - Internal monorepo: `hailhq.*` (PEP 420 implicit namespace; no `hailhq/__init__.py` at the namespace root).
  - External SDK: `hail` — published as `hail-sdk` on PyPI, imports as `import hail`. Standalone — does **not** depend on any `hailhq.*` package.
- **Provider model IDs**: live only in `.env.example`; `Settings` fields default to empty strings. Do not set `Settings.<provider>_model = "literal"` — that is wrong.
- **Tag prefix grammar**: `<package>-v<semver>` for component releases (`sdk-v…`, `cli-v…`). Bare `v<semver>` is not a tag anyone creates by hand — GoReleaser mints it from `cli-v<semver>`.
- **No Opero references** in any committed file.

## Footguns (every one of these has bitten us)

| symptom                                                                                                     | root cause + fix                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ----------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Deploy dies mid-pull with `no space left on device`                                                         | Each deploy pulls `:latest` **and** `sha-<commit>` for three services; the `sha-` tags are tagged images, so a dangling-only prune never reclaims them. `deploy.yml` prunes unused Hail images older than 48 hours before pulling and after rollout, and requires 20 GiB free before a pull. Two weeks of retained builds filled the 150 GB VM on 2026-10-19; voice workers could not create `/tmp/pymp-*` startup files. Recover with `docker image prune -af --filter "until=48h" --filter "label=org.opencontainers.image.source=https://github.com/hail-hq/hail"`, then check free space and worker initialization. Running images are preserved; older rollback tags can be pulled from GHCR. |
| `voicebot` container exits showing the typer help menu                                                      | The CMD lacks the `start` subcommand. The Dockerfile fixes this; if you fork it, keep `["python", "-m", "hailhq.voicebot.main", "start"]`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| `ModuleNotFoundError: No module named 'hailhq'` after a Docker build                                        | Hatchling wheel config used `packages = ["hailhq/<service>"]` — strips the `hailhq/` prefix. Must be `packages = ["hailhq"]`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| `exec /opt/venv/bin/uvicorn: no such file or directory`                                                     | Renaming `/app/.venv` → `/opt/venv` between builder and runtime breaks shebangs. Keep the same path in both stages (we use `/app/.venv`).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| `RuntimeError: no running event loop` from `aiohttp.ClientSession()`                                        | A FastAPI dep that constructs LiveKit/aiohttp must be `async def`. Sync deps run in a threadpool worker thread with no loop.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| `google.auth.exceptions.DefaultCredentialsError: File  was not found.`                                      | `GOOGLE_GENAI_USE_VERTEXAI=true` with empty `GOOGLE_APPLICATION_CREDENTIALS`. Default is `false`; opt into Vertex by flipping it AND providing creds.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `failed to parse tag 'cli-v0.1.0' as semver` from GoReleaser                                                | OSS GoReleaser does not handle tag prefixes. The workflow's `Compute GoReleaser current/previous tags` step + `--skip=validate` flag handle it; do not remove either.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `hailhq-core` references a workspace member, but is not one                                                 | The Docker build is missing the repo-root `pyproject.toml`, `uv.lock` or a workspace member's `pyproject.toml`. Each Dockerfile copies all of them, then runs `uv sync --frozen`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
| `pytest` from repo root fails with `ImportPathMismatchError`                                                | Run each suite from its own directory. CI does this; replicate locally.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| First `pip install hail-sdk` from a venv with `hailhq.*` already installed shadows imports                  | The SDK is standalone by design. If you mix it with internal packages in the same venv, the `hail` package wins for `import hail` (intended); do not co-install for production.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| Deploy fails at `alembic upgrade head` with `Can't locate revision 'NNNN'` or `relation "…" already exists` | A branch reshuffle / history rewrite renumbered or removed a migration that prod already applied, so prod's `alembic_version` diverges from the code (refer to Migration discipline rule 3). Recover by hand: apply only the migrations that never ran on prod, then `UPDATE alembic_version SET version_num='<head>'` (or `alembic stamp <head>`) once the schema matches — do **not** re-run `upgrade head`, it re-hits the collision.                                                                                                                                                                                                                                                           |

## Inbound email rollout

This section records the end-to-end deployment of the inbound-email
milestone (umbrella `v0.5.0`, `sdk-v0.3.0`, `cli-v0.5.0`). It covers
five DB stages + infra + tag-driven component releases. Stage 1 needs
a brief maintenance window (the table rename is not online-safe).
Everything else is online.

### Full deployment order (copy-paste cheat-sheet)

```bash
# ── 0. Preflight on main ───────────────────────────────────────────────────
git checkout main && git pull --ff-only origin main
git log -1 --oneline                                  # confirm merge SHA

# ── 1. Stage-1 cutover: stop API, migrate, deploy new image ────────────────
docker compose stop api
docker compose run --rm api alembic upgrade 0006      # rename (DOWNTIME)
docker compose up -d api                              # new image w/ new code
hail email domain list                                # 200 smoke

# ── 2. Stage-2 + Stage-3 migrations (online) ───────────────────────────────
docker compose run --rm api alembic upgrade 0007      # inbound columns + email_attachments
docker compose run --rm api alembic upgrade 0008      # webhook tables
docker compose run --rm api alembic current           # → 0008 (head)

# ── 3. Provision AWS infra (Terragrunt reads .env directly) ───────────────
cd infra
terragrunt init
terragrunt plan
terragrunt apply
# Capture outputs: inbound_mx_record, inbound_bucket, activate_command, lambda_function_arn
#   (inbound_bucket confirms ${HAIL_MAIL_NAME_PREFIX}-mail — not independently settable)
#
# First time only: pre-create the state bucket + DynamoDB lock table once
# per AWS account — terragrunt can't bootstrap them. See the comment
# block at the top of infra/terragrunt.hcl for the AWS CLI one-liners.

# ── 4. Activate the SES Receipt Rule Set (manual; one-time per region) ─────
aws ses set-active-receipt-rule-set --rule-set-name hail-inbound-prod-rules
#  ↑ ONLY safe on accounts with no other active rule set; see Stage 4 below.

# ── 5. Publish the MX record (manual at your DNS provider) ─────────────────
#  e.g. mail.hail.so  MX  10  inbound-smtp.us-east-1.amazonaws.com
dig MX mail.hail.so                                   # wait for propagation

# ── 6. Flip the inbound flag in API .env, restart ──────────────────────────
# Add: HAIL_INBOUND_ENABLED=true
#      HAIL_MAIL_NAME_PREFIX=<terraform var name_prefix — same value, bucket derives as ${prefix}-mail>
#      HAIL_INBOUND_HMAC_SECRET=<same as tfvars>
#      HAIL_WEBHOOK_SECRET_KEY=$(uv run --directory core python -c "from hailhq.core.secret_cipher import generate_key; print(generate_key())")
docker compose up -d api                              # picks up new env

# ── 7. Release SDK 0.3.0 (fires release-sdk.yml → PyPI) ────────────────────
git tag -a sdk-v0.3.0 -m "SDK 0.3.0 — inbound email + webhooks"
git push origin sdk-v0.3.0

# ── 8. Release CLI 0.5.0 (fires release-cli.yml → GitHub Releases + brew) ──
git tag -a cli-v0.5.0 -m "CLI 0.5.0 — hail email list/get; webhooks; domain rename"
git push origin cli-v0.5.0

# ── 9. Umbrella tag marker (no workflow fires) ─────────────────────────────
git tag -a v0.5.0 -m "Hail 0.5.0 — inbound email milestone"
git push origin v0.5.0
gh release create v0.5.0 --notes-file CHANGELOG.md    # optional release page

# ── 10. Smoke test (see "Smoke test sequence" subsection below) ────────────
hail email domain list
dig MX mail.hail.so
# … send a test mail, expect status=received within 30s
hail email list --direction inbound
```

**Decoupling note.** You can release steps 1–6 days before steps 7–9.
Until the flag in step 6 flips, the system is outbound-only, and the
new CLI and SDK are not yet public. If something looks wrong after
step 6, roll back: set `HAIL_INBOUND_ENABLED=false` and restart. No
migration revert is necessary; no tag re-spin is necessary.

Detailed nuance per stage follows.

### Stage 1 — schema rename + new code (coordinated cutover, ~30s downtime)

Migration `0006` renames `sender_domains` → `email_domains`. The new
app code references only the new name, so you must release the rename
and the deploy together. Each incremental path adds view-shim
complexity for marginal gain.

```bash
# Take the API offline.
docker compose stop api

# Apply 0006 only.
DATABASE_URL=$DATABASE_URL uv run --directory api alembic upgrade 0006

# Deploy the new image and start.
docker compose up -d api

# Smoke: should 200, list returns same rows under the new name.
hail email domain list
```

Rollback: `alembic downgrade 0005` reverses the rename. The new code
breaks against the old name, so this works only if you also roll back
the image.

### Stage 2 — additive inbound schema (online, anytime after stage 1)

Migration `0007` is purely additive: nullable columns on `emails`, a new
`email_attachments` table, and action columns on `email_domains`. Old and
new code both run against it cleanly.

```bash
DATABASE_URL=$DATABASE_URL uv run --directory api alembic upgrade 0007
```

**Footgun on large tables.** If `emails` has more than a few million
rows, edit the migration to `CREATE INDEX CONCURRENTLY` via
`op.execute` before you apply it. `emails_inbound_message_id_uq` is a
partial unique index whose `WHERE` is false for every existing row,
but Postgres still scans the whole table to build it. Alembic's
`create_index` runs in a transaction, which blocks the concurrent
flag.

### Stage 3 — webhook tables (online, anytime after stage 2)

Migration `0008` adds `webhook_subscriptions` + `webhook_deliveries`.
Pure new tables, no locks on existing tables.

```bash
DATABASE_URL=$DATABASE_URL uv run --directory api alembic upgrade 0008
```

At this point, the API is ready to receive inbound mail and fire
webhooks. But nothing is wired up yet on the AWS side, and the inbound
flag is off, so the system stays effectively outbound-only.

### Stage 4 — provision AWS infrastructure

The bare Terraform module under `infra/terraform/` provisions S3, the SES
Receipt Rule + Rule Set, the Lambda, and IAM. A Terragrunt wrapper at
`infra/terragrunt.hcl` adds an S3 remote backend with a DynamoDB lock
table, and pulls every variable from the repo's `.env`, so there is no
parallel `tfvars` file to keep in sync.

**One-time bootstrap** per AWS account. Terragrunt's state backend
cannot bootstrap itself. Create the bucket + lock table once with the
AWS CLI. The AWS CLI does not read `.env` like Terragrunt does, so
this one step needs the env exported into your shell. Note: the region
of the state backend is `HAIL_TERRAFORM_STATE_REGION` — often shared
across deployments and **separate from `AWS_REGION`** (where the SES +
Lambda + raw-MIME bucket get provisioned). It falls back to
`AWS_REGION` when unset.

```bash
set -a; source .env; set +a   # exports AWS_PROFILE, AWS_REGION, HAIL_TERRAFORM_*
STATE_REGION=${HAIL_TERRAFORM_STATE_REGION:-$AWS_REGION}

aws --profile $AWS_PROFILE s3api create-bucket \
    --bucket $HAIL_TERRAFORM_STATE_BUCKET \
    --region $STATE_REGION
aws --profile $AWS_PROFILE s3api put-bucket-versioning \
    --bucket $HAIL_TERRAFORM_STATE_BUCKET \
    --versioning-configuration Status=Enabled
aws --profile $AWS_PROFILE dynamodb create-table \
    --table-name $HAIL_TERRAFORM_LOCK_TABLE \
    --attribute-definitions AttributeName=LockID,AttributeType=S \
    --key-schema AttributeName=LockID,KeyType=HASH \
    --billing-mode PAY_PER_REQUEST \
    --region $STATE_REGION
```

**Each deploy.** Generate (or rotate) the shared HMAC secret in `.env`
before the deploy. The API uses the same value to verify Lambda
POSTs:

```bash
# In .env:
#   HAIL_INBOUND_HMAC_SECRET=<openssl rand -hex 32 output>
#   HAIL_API_URL=https://api.hail.so          # the URL the Lambda calls

cd infra
terragrunt init                # Terragrunt reads .env via run_cmd internally;
terragrunt plan                # no manual `source .env` needed here.
terragrunt apply
```

Outputs to capture:

- `inbound_mx_record` — publish at DNS for `HAIL_MAIL_BASE_DOMAIN`.
- `inbound_bucket` — confirms `${HAIL_MAIL_NAME_PREFIX}-mail`; set `HAIL_MAIL_NAME_PREFIX` (not `inbound_bucket` itself) on the API.
- `activate_command` — the `aws ses set-active-receipt-rule-set ...`
  to run after apply.

**Plain Terraform alternative.** The bare module under `infra/terraform/`
still works with `terraform apply -var ...` if you prefer to skip
Terragrunt. The Terragrunt wrapper is opinionated about remote state +
`.env`-driven inputs; the underlying module is provider-vanilla.

**Manual step: activate the receipt rule set.** Terraform creates the
rule set but does not activate it (SES allows one active rule set per
region per account, and activation is destructive against any
existing one):

```bash
# Greenfield AWS account:
aws ses set-active-receipt-rule-set --rule-set-name hail-inbound-prod-rules

# Account with existing receipt rules: import the existing rule set
# into Terraform state, merge Hail's rule into it via the AWS console,
# or skip the module's aws_ses_receipt_rule resource entirely.
```

**Manual step: publish the MX record.** Publish the value that the
`inbound_mx_record` output prints, for example:

```
mail.hail.so  MX  10  inbound-smtp.us-east-1.amazonaws.com
```

DNS propagation usually takes minutes. SES does not deliver until the
record is live.

### Stage 5 — flip the inbound flag

Add to the API service `.env`:

```bash
HAIL_INBOUND_ENABLED=true
HAIL_MAIL_NAME_PREFIX=<terraform var name_prefix>
HAIL_INBOUND_HMAC_SECRET=<same value Terraform got>
HAIL_WEBHOOK_SECRET_KEY=<generate with `uv run --directory core python -c "from hailhq.core.secret_cipher import generate_key; print(generate_key())"`>
```

Restart the API. Until this step, `POST /internal/ses-events` returns
503 even if the Lambda is wired. Thus steps 1–4 can land days before
step 5, and you keep a free rollback window (no migration revert is
necessary to disable inbound).

### Smoke test sequence

After stage 5, in order:

1. **Routing exists.** Pick an org you control. From the console or
   the API, verify that the hail-mail row exists and `inbound_enabled=true`:

   ```bash
   hail email domain list
   ```

2. **MX is live.** From any host with `dig`:

   ```bash
   dig MX mail.hail.so
   # Should answer with inbound-smtp.<region>.amazonaws.com
   ```

3. **Round-trip a test mail.** Send to the org's hail-mail address
   from any external account. Within 30s:

   ```bash
   hail email list --direction inbound
   # Should show the new row with status=received.
   ```

4. **Raw + attachment access.** If the test mail had attachments,
   the listing must include their metadata. This fetch:

   ```bash
   hail email get <id> | jq .raw_url
   curl -L -H "Authorization: Bearer $HAIL_API_KEY" "$RAW_URL" > out.eml
   ```

   must download the raw MIME bytes.

5. **Forwarding** (only if `forward_to` is configured on the domain):
   the forward target must receive a copy from
   `noreply+<org>@mail.hail.so` with the original sender in
   `Reply-To:`. Check `hail email list --direction outbound`. A
   row with `metadata.forwarded_from = <inbound id>` must show
   `status=sent`.

6. **Webhook delivery** (only if a webhook is configured): the
   target must receive a signed POST. Verify that the signature header
   parses, that the body shape matches the documented event, and that
   the delivery row reaches `status=succeeded`:

   ```bash
   curl "$HAIL_API_URL/v1/webhooks/<subscription-id>/deliveries" \
     -H "Authorization: Bearer $HAIL_API_KEY"
   ```

7. **Bounce/complaint plumbing.** To send a test bounce, address a
   mail to `bounce@simulator.amazonses.com` from the verified sender.
   SES generates a bounce notification. The matching outbound row
   must change to `status=bounced` in a few seconds.

8. **Rate cap.** Set `email_domains.forward_rate_per_hour=1`, then
   send two inbounds with forwarding configured. The second one's
   `suppressed_reasons` must include `forward_rate_limit`.

### What can go wrong (failure modes we have seen first-hand or modeled)

- **DNS not propagated.** SES silently drops mail to a domain whose
  MX is wrong. `dig MX` is the first diagnostic.
- **Receipt rule set not activated.** SES accepts mail but does not
  route it; no S3 object appears. Check `aws sesv2
describe-active-receipt-rule-set` again.
- **SES sandbox.** New SES accounts accept mail only from verified
  addresses. During sandbox, the mail's `From:` must be on a verified
  SES identity.
- **Private webhook target rejected.** `httpx_post` blocks RFC-1918
  addresses by default. Self-hosters with internal webhook targets
  set `HAIL_WEBHOOK_ALLOW_PRIVATE_NETWORKS=true`.
- **Lambda → API connection refused.** Check the Lambda's
  CloudWatch logs (`/aws/lambda/hail-inbound-prod-ingest`). The
  POST URL must be reachable from AWS over the public internet.
- **Forward loops.** The system rejects a forward to a target on
  `HAIL_MAIL_BASE_DOMAIN`. If a target's auto-responder replies to
  the forwarder, the 3-hop counter catches the loop. If it does
  not, raise `HAIL_FORWARD_MAX_HOPS`, or blacklist the target:
  clear `forward_to`.

## SMS abuse monitor & channel suspensions

All orgs share one Hail-owned A2P 10DLC campaign, so one org's high opt-out
rate can cause carriers to throttle the whole platform. `AbuseMonitorWorker`
(hourly by default; `HAIL_ABUSE_MONITOR_POLL_SECONDS`) computes each org's
rolling opt-out rate. Past the threshold
(`HAIL_SMS_ABUSE_*` — window, min sends, max rate), it inserts a
`channel_suspensions` row. `check_sms_allowed` then blocks that org's
outbound SMS. The thresholds are unvalidated starting guesses. Expect to
tune them when real traffic arrives.

**You must lift a suspension manually for now — there is no CLI/route/expiry.**
An auto-suspended org stays blocked until an operator runs raw SQL. Inspect
and lift:

```bash
# See who is suspended and why
psql "$DATABASE_URL" -c \
  "SELECT organization_id, channel, reason, suspended_at FROM channel_suspensions;"

# Lift one org's SMS suspension (re-enables outbound immediately)
psql "$DATABASE_URL" -c \
  "DELETE FROM channel_suspensions WHERE organization_id = '<org-uuid>' AND channel = 'sms';"
```

Before you lift a suspension, confirm that the opt-out spike was benign (a
burst of legitimate STOPs, not abuse). Otherwise, the next hourly tick
suspends the org again. There is no cooldown/backoff yet, so a
persistently-abusive org trips again on the next run. If you see false
positives, tune `HAIL_SMS_ABUSE_MAX_OPT_OUT_RATE` up.

## Carry-forwards / open work

(Refer to `CHANGELOG.md` "Deferred to v1.x" for the canonical list.)

- LiveKit Egress recording → S3 (currently `recording.py` returns `None`)
- `idempotency_keys` GC sweeper
- SMTP-listener inbound email provider (`SmtpInboundProvider` is a stub)
- `hail bootstrap` admin CLI (closes the manual DB-seed step above)
- CallEvent dedupe across voicebot redispatch
- **Un-suspend tooling for `channel_suspensions`** — auto-suspend has no
  reverse path (refer to "SMS abuse monitor" above); add a `hail sms suspensions
lift <org>` command / operator route and/or an automatic cooldown column
  so recovery is not raw SQL.

---

<!-- source: https://hail.so/docs/contributing -->

# Contributing

For the full operational runbook (releases, deployment, DB switching, all the known problems), refer to [Operations](./self-host/operations.md). This page covers the contribution flow only.

## Setup

```bash
git clone <repo>
cd hail
cp .env.example .env.local
# fill in keys (see ./setup/)
pnpm install                      # installs husky + lint-staged + prettier
docker compose -f docker-compose.yml -f docker-compose.local.yml up postgres
                                  # just the data services for host-side dev
```

`pnpm install` installs the git pre-commit hook. The hook runs `ruff`/`black`/`gofmt`/`prettier` on staged files.

## Dev loops

- API: `cd api && uv run uvicorn hailhq.api.main:app --reload --port 8080`
- Voicebot: `cd voicebot && uv run python -m hailhq.voicebot.main start`
- MCP: `cd mcp && uv run uvicorn hailhq.mcp.server:app --reload --port 8081`
- CLI: `cd cli && go run . <args>`

Full stack in Docker:

- Bundled Postgres: `docker compose -f docker-compose.yml -f docker-compose.local.yml up`
- Managed Postgres (set `DATABASE_URL` to your hosted URL first): `docker compose up`

## Database migrations

The schema lives in [`api/migrations/versions/`](https://github.com/hail-hq/hail/tree/main/api/migrations/versions). The Alembic config is in [`api/alembic.ini`](https://github.com/hail-hq/hail/blob/main/api/alembic.ini); `DATABASE_URL` overrides the config default.

```bash
cd api
uv run alembic upgrade head       # apply all pending
uv run alembic revision -m "add foo"   # create a new revision (hand-edit the SQL)
uv run alembic downgrade -1       # revert the last revision
```

Migrations are hand-written raw SQL for v1 (no ORM models yet). When SQLAlchemy models are released, switch to `--autogenerate`.

## Regenerating openapi.yaml

After you change API routes, dump the spec:

```bash
curl -s http://localhost:8080/openapi.json \
  | python -c "import json, sys, yaml; yaml.safe_dump(json.load(sys.stdin), sys.stdout, sort_keys=False)" \
  > openapi/openapi.yaml
```

The Go CLI codegens its client from this file, so commit the update in the same PR as the route change.

Do not hand-edit `openapi/openapi.yaml`. CI regenerates it from the live app
and compares (refer to `.github/workflows/openapi-check.yml`). Any manual
change that the app does not also produce fails as "stale". A status raised
via `raise HTTPException(...)` does **not** appear in the spec unless the
route decorator declares it (for example `responses={429: {...}}`). Add it
there, then regenerate.

## Commit style

[Conventional Commits](https://www.conventionalcommits.org):

- `feat(api): add POST /calls`
- `fix(voicebot): handle SIP disconnect during greeting`
- `docs(setup): clarify Twilio trunk origination URI`

## Adding a provider

Put new adapters under `core/hailhq/core/providers/<channel>/<name>.py`. Each adapter implements that channel's adapter interface. Add config keys to `.env.example` in the same provider-grouped format.

## Model costs contributions

Public AI model costs live in [`costs/`](https://github.com/hail-hq/hail/tree/main/costs) under CC-BY-4.0. The JSON files at the top of that directory are the source of truth. CI validates them against the schemas in `costs/schema/` on every PR.

To update a price:

1. Edit `costs/<category>.json` (for example `costs/llm.json`).
2. Set `last_verified` to today (`YYYY-MM-DD`). Set `verified_by` to your GitHub handle.
3. Update `source_url` if it has changed.
4. Run `pnpm costs:validate` locally before you push.

A weekly cron opens a tracking issue that lists rows older than 30 days — refer to [`costs-stale.yml`](https://github.com/hail-hq/hail/blob/main/.github/workflows/costs-stale.yml).

## What we will not merge (v1)

- Code that hard-codes a provider in `api/` or `voicebot/` — route through `core/`.
- New env vars missing from `.env.example`.
- Features without a milestone in README.
- Web UI code (no dashboards in v1).
- Docs that paraphrase the OpenAPI spec or MCP tool schemas instead of a link to the canonical source.
- Non-GFM Markdown in docs.