> ## Documentation Index > Fetch the complete documentation index at: https://docs.getdialed.ai/llms.txt > Use this file to discover all available pages before exploring further. # Bulk ingestion > Upload millions of records as a record set — create a manifest, push chunks, seal, then trigger a flow by reference Past 10,000 records, don't send an array. Upload a **record set** — a manifest you push records into in chunks, seal, and then hand to a trigger by id. The records never travel inside the trigger request, so a 1,000,000-row upload is a hundred small POSTs and one trigger call that returns immediately. ## When to use it | Volume | Path | | ------------------------------ | ------------------------------------------------------------------------------------------------ | | Up to 10,000 records | Inline `records` array on the trigger — see [Trigger with records](/guides/trigger-with-records) | | 10,001 up to 5,000,000 records | A record set, triggered with `record_set_id` | Inline is simpler and stays the right answer for small loads. Everything on this page is for the case where the array would be too big to send at all. ## The three-step flow 1. **`POST /record-sets`** — create the manifest. Declares `chunk_size` and returns the record set's `id`. 2. **`POST /record-sets/{record_set_id}/records`** — once per chunk, until every record has been sent. 3. **`POST /record-sets/{record_set_id}/seal`** — declare `expected_count`. The API checks it against what it actually staged. Then trigger the flow with `record_set_id`. All three upload steps require admin access; reading a record set's status does not. There is no `GET /record-sets` list endpoint. A record set is a short-lived upload handle whose id you already hold — read one with `GET /record-sets/{record_set_id}` to check progress. ## Step 1 — Create the record set `chunk_size` is fixed for the whole upload and defaults to the largest chunk the API accepts (10,000). `metadata` is free-form and comes back on every read. ```bash curl theme={null} curl -X POST https://api.getdialed.ai/flows/record-sets \ -H "X-API-Key: $API_KEY" \ -H "Content-Type: application/json" \ -d '{ "chunk_size": 10000, "metadata": {"campaign": "spring-2026", "source": "crm-export.csv"} }' ``` ```python Python theme={null} import os import httpx client = httpx.Client( base_url="https://api.getdialed.ai/flows/", headers={"X-API-Key": os.environ["API_KEY"]}, ) res = client.post( "record-sets", json={ "chunk_size": 10000, "metadata": {"campaign": "spring-2026", "source": "crm-export.csv"}, }, ) record_set = res.json() record_set_id = record_set["id"] chunk_size = record_set["chunk_size"] ``` ```javascript JavaScript theme={null} const res = await fetch("https://api.getdialed.ai/flows/record-sets", { method: "POST", headers: { "X-API-Key": process.env.API_KEY, "Content-Type": "application/json", }, body: JSON.stringify({ chunk_size: 10000, metadata: { campaign: "spring-2026", source: "crm-export.csv" }, }), }); const { id: recordSetId, chunk_size: chunkSize } = await res.json(); ``` The `201` response is the manifest: ```json theme={null} { "id": "rs_0123456789abcdef", "status": "open", "chunk_size": 10000, "received_count": 0, "chunk_count": 0, "expected_count": null, "content_hash": null, "batch_id": null, "metadata": {"campaign": "spring-2026", "source": "crm-export.csv"}, "created_at": "2026-07-27T12:00:00Z", "updated_at": "2026-07-27T12:00:00Z", "sealed_at": null, "consumed_at": null, "staging_expires_at": "2026-08-03T12:00:00Z" } ``` `status` moves `open` → `sealed` → `consumed`. Only an `open` record set accepts chunks. ## Reading CSV files **Parsing is client-side in this release.** Chunk bodies are JSON only — there is no `text/csv` upload. You read the file, turn each row into an object, and POST batches of those objects. Given a CSV like this: ```csv theme={null} first_name,last_name,phone Alice,Nguyen,+15555550100 Bob,Marsh,+15555550101 Carla,Diaz,+15555550102 ``` each row becomes one object, and your flow reads its columns through `{{input.first_name}}`, `{{input.last_name}}` and `{{input.phone}}` — exactly as on the inline path. Combining several files into one upload works the same way: the manifest *is* the "combine files" primitive. Push each file's rows as additional chunks into the same record set, keeping a single running chunk index across all of them. ```python theme={null} import csv def rows(paths): """Yield one dict per row across several CSVs, header-mapped.""" for path in paths: with open(path, newline="") as fh: yield from csv.DictReader(fh) def chunks(iterable, size): batch = [] for item in iterable: batch.append(item) if len(batch) == size: yield batch batch = [] if batch: yield batch ``` ## Step 2 — Push the chunks Each chunk carries a **client-assigned `chunk_index`**, starting at 0, and the chunk's records in dial order: ```json theme={null} { "chunk_index": 0, "records": [ {"first_name": "Alice", "last_name": "Nguyen", "phone": "+15555550100"}, {"first_name": "Bob", "last_name": "Marsh", "phone": "+15555550101"}, {"first_name": "Carla", "last_name": "Diaz", "phone": "+15555550102"} ] } ``` Chunks may be sent in any order. Re-POSTing the same `chunk_index` with the same rows is a safe no-op: the response reports `replay: true` and the record set's counters do not move. That is what makes an interrupted upload recoverable — on any error you do not know whether the chunk landed, so you simply send it again. A row's position in the set is `chunk_index * chunk_size + offset`, where `offset` is its position within the chunk. Keep `chunk_size` constant for every chunk except the final one. A short chunk in the *middle* is allowed — it just leaves a harmless gap in the row numbering, because rows are processed by range and sealing validates the totals. ### Pacing and `429` The chunk endpoint has **its own rate limit**, separate from (and far above) the standard per-principal limit. Even so, honour `Retry-After` on a `429` and back off — behind multiple API replicas this is the difference between an upload that completes and one that fails intermittently. ```python Python theme={null} import time import httpx def post_chunk(client, record_set_id, chunk_index, records, attempts=6): """POST one chunk, honouring Retry-After on 429 and retrying safely. Retrying is safe at any point: the same chunk_index with the same rows is recognised as a replay and inserts nothing. """ backoff = 1.0 for attempt in range(attempts): res = client.post( f"record-sets/{record_set_id}/records", json={"chunk_index": chunk_index, "records": records}, ) if res.status_code == 429: wait = float(res.headers.get("Retry-After", backoff)) time.sleep(wait) backoff = min(backoff * 2, 60) continue res.raise_for_status() return res.json() raise RuntimeError(f"chunk {chunk_index} did not complete after {attempts} attempts") total = 0 for index, batch in enumerate(chunks(rows(["a.csv", "b.csv"]), chunk_size)): body = post_chunk(client, record_set_id, index, batch) total = body["received_count"] print(f"chunk {index}: +{body['inserted']} replay={body['replay']} total={total}") ``` ```javascript JavaScript theme={null} const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); async function postChunk(recordSetId, chunkIndex, records, attempts = 6) { let backoff = 1000; for (let attempt = 0; attempt < attempts; attempt++) { const res = await fetch( `https://api.getdialed.ai/flows/record-sets/${recordSetId}/records`, { method: "POST", headers: { "X-API-Key": process.env.API_KEY, "Content-Type": "application/json", }, body: JSON.stringify({ chunk_index: chunkIndex, records }), }, ); if (res.status === 429) { const retryAfter = res.headers.get("Retry-After"); await sleep(retryAfter ? Number(retryAfter) * 1000 : backoff); backoff = Math.min(backoff * 2, 60000); continue; } if (!res.ok) throw new Error(`chunk ${chunkIndex} failed: ${res.status}`); return res.json(); } throw new Error(`chunk ${chunkIndex} did not complete`); } ``` Each chunk responds with its own outcome: ```json theme={null} { "record_set_id": "rs_0123456789abcdef", "chunk_index": 0, "records_received": 10000, "inserted": 10000, "replay": false, "received_count": 10000 } ``` `received_count` is the running total across the whole record set. The value from your **last** chunk is what you pass as `expected_count` when you seal. | Status | Meaning | | ------ | ----------------------------------------------------------------------------------------- | | `200` | Chunk staged, or recognised as a replay (`replay: true`) | | `404` | Unknown record set — also returned for another tenant's record set, never `403` | | `409` | The record set is no longer `open`, so it cannot accept records | | `413` | Request body too large — split the upload into more chunks | | `422` | More records than the manifest's `chunk_size`, or a record over the per-record size limit | | `429` | Chunk rate limit exceeded — back off per `Retry-After` | An oversized-record `422` names the offending **row index**, never the row itself. No error body ever echoes your record data back. ## Step 3 — Seal `expected_count` is the total number of records you sent. The API compares it with the number it actually staged and refuses the seal if they disagree — **this is how a chunk that was silently lost in transit is caught**, before anything is dialled. You don't need the total up front, only at seal time, which is what lets you stream several files in without counting them first. ```bash curl theme={null} curl -X POST "https://api.getdialed.ai/flows/record-sets/rs_0123456789abcdef/seal" \ -H "X-API-Key: $API_KEY" \ -H "Content-Type: application/json" \ -d '{"expected_count": 1000000}' ``` ```python Python theme={null} res = client.post( f"record-sets/{record_set_id}/seal", json={"expected_count": total}, ) res.raise_for_status() sealed = res.json() ``` ```json theme={null} { "id": "rs_0123456789abcdef", "status": "sealed", "chunk_size": 10000, "received_count": 1000000, "chunk_count": 100, "expected_count": 1000000, "content_hash": "9f2c4e1b7a35d80c6e4f1a92b3c7d5e08f6a1b2c3d4e5f60718293a4b5c6d7e8", "batch_id": null, "metadata": {"campaign": "spring-2026"}, "created_at": "2026-07-27T12:00:00Z", "updated_at": "2026-07-27T12:04:31Z", "sealed_at": "2026-07-27T12:04:31Z", "consumed_at": null, "staging_expires_at": "2026-08-03T12:00:00Z" } ``` A `409` on seal names both numbers — what you expected and what was staged — so a missing chunk is distinguishable from a duplicate seal. Re-sealing with the same `expected_count` returns the same result, so the call is safe to retry. ## Step 4 — Trigger the flow Pass `record_set_id` where you would have passed `records`: ```bash curl theme={null} curl -X POST "https://api.getdialed.ai/flows/definitions/def_a1b2c3d4/trigger" \ -H "X-API-Key: $API_KEY" \ -H "Content-Type: application/json" \ -d '{ "source": "crm_export", "record_set_id": "rs_0123456789abcdef" }' ``` ```python Python theme={null} res = client.post( "definitions/def_a1b2c3d4/trigger", json={"source": "crm_export", "record_set_id": record_set_id}, ) batch = res.json() ``` ```json theme={null} { "batch_id": "batch_55667788", "definition_id": "def_a1b2c3d4", "status": "pending", "deduplicated": false } ``` `records` and `record_set_id` are **mutually exclusive** — sending both is a `422`. **The definition must use batched delivery.** A definition with per-record (direct-delivery) steps refuses a record set with `422`, pointing you back at the inline `records` path (up to 10,000), which works today. There is no silent coercion between the two delivery modes. A record set is **single-use** — exactly one trigger consumes it. The two `409`s here are distinct and mean different things: | Status | Meaning | | ------ | ------------------------------------------------------------------------------------ | | `404` | Unknown record set, or another tenant's — never `403` | | `409` | The record set is still `open` — seal it first | | `409` | The record set is already `consumed` — a trigger has claimed it | | `422` | Both `records` and `record_set_id` supplied, or the definition uses per-record steps | ## Idempotency Identical content re-submitted **within 7 days** is not ingested twice. The second call returns `200` with `deduplicated: true` and the **original** `batch_id`: ```json theme={null} { "batch_id": "batch_55667788", "definition_id": "def_a1b2c3d4", "status": "queued", "deduplicated": true } ``` A deduplicated submission is a **`200`, not an error**. A client that treats any non-fresh response as a failure will mis-handle it. The `batch_id` you get back is a usable handle to the original batch — poll it as normal. Content identity is computed by the server from the sealed records and the definition, so uploading the same rows against a *different* definition is not a duplicate. To deliberately re-run identical content — a genuine second dial of the same list — supply an `Idempotency-Key` header on the trigger, which forces a new batch: ```bash theme={null} curl -X POST "https://api.getdialed.ai/flows/definitions/def_a1b2c3d4/trigger" \ -H "X-API-Key: $API_KEY" \ -H "Idempotency-Key: redial-2026-07-27-morning" \ -H "Content-Type: application/json" \ -d '{"record_set_id": "rs_fedcba9876543210"}' ``` The key must be non-blank and at most 255 characters. It applies to the record-set path only and is ignored on the inline `records` path. After 7 days the window closes and identical content is accepted again without a key. ## What happens next The trigger returns as soon as the batch is minted — the per-record work happens after the response, which is why the call is fast regardless of record count. Every uploaded row becomes one [job](/concepts/batches-and-executions#jobs) and one execution. The batch is `pending` while its rows are being bound, and reaches **`queued`** once every row has a job and an execution. Watch it with `GET /batches/{batch_id}`. **Nothing is sent to your platform during ingestion.** A `queued` batch is fully staged and idle — dispatch to the contact-centre platform arrives in a later release. Uploading is safe to do ahead of time. Read the individual rows with `GET /jobs?batch_id=`, exactly as for an inline trigger. ## Retention | What | Reclaimed when | | -------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | An uploaded row (its job) | 30 days after the job reaches a terminal state, or your account's configured retention | | An unsealed or never-triggered record set, and its staged rows | Automatically, 7 days after the record set was created | The second row is the important one for abandoned uploads: a record set you create and never trigger does not sit around holding contact data indefinitely. Changing an account's retention affects **future batches only**. Rows already staged keep the retention that was in force when their batch was created. ## Limits | Limit | Value | | ---------------------------- | --------- | | Inline `records` per trigger | 10,000 | | Records per chunk | 10,000 | | Records per record set | 5,000,000 | | Per-record size | 8 KB | | Chunk request body | 16 MB | Records per chunk and chunk body size bind independently — at roughly 300 bytes per record the 16 MB body ceiling is reached well above the 10,000-record chunk cap, so the record count is normally the binding limit. ## Next steps The inline path for loads up to 10,000 records. Every status in the chain, including `staged` and `queued`.