> ## Documentation Index > Fetch the complete documentation index at: https://docs.getdialed.ai/llms.txt > Use this file to discover all available pages before exploring further. # Record transforms > Reshape a whole set of records inside a flow — deduplicate, filter, sort, summarise, split, combine and parse — with each step producing a new set the next one reads by reference A **record transform** is a step that takes a set of records, does one thing to all of them, and produces a new set. Remove the duplicates from an upload. Keep only the rows that matter. Put a list in order and take the top few. Tidy every phone number into its international form. Eleven of these ship today, plus a twelfth step that hands the result to another flow. They need no credential and call nothing outside the platform. ```json theme={null} { "task_id": "task_dedupe", "task_name": "Remove duplicate numbers", "platform_id": "getdialed", "service_id": "getdialed__records", "action_id": "getdialed__records__dedupe", "parameters": { "record_set_id": "{{ step_upload.output.record_set_id }}", "key_fields": ["phone"] } } ``` ## Why these are steps and not formatters A [pipe](/concepts/expressions#pipes) formats **one value in one record** — it trims a name, rounds a cost, normalizes a number. It only ever sees the record it is standing in. Removing duplicates across an entire upload is a different kind of question. So is sorting a whole list, or working out a total per region. Those need the **whole set at once**, which is why they are steps rather than something you can write inside a pair of braces. The split follows the shape of the work, not its difficulty: | You want to | Use | | ------------------------------------------------- | -------------------------------------------------------------- | | Reformat a value on the record in front of you | A [pipe](/concepts/expressions#pipes), inline in the parameter | | Reshape, reduce or reorder a whole set of records | A record transform step | ## How the records travel **By reference, never by value.** A transform step is given the *identifier* of a record set, reads it as the step runs, and hands back the identifier of the set it produced. The records themselves never travel through the flow. That is what makes a set of any size workable: the flow carries an id and a handful of counts, not fifty thousand rows. It is also why steps chain the way they do — each step's output id is bound into the next step's input exactly like any other step output: ```json theme={null} { "steps": [ { "step_id": "step_clean", "tasks": [ { "task_id": "task_dedupe", "task_name": "Remove duplicate numbers", "platform_id": "getdialed", "service_id": "getdialed__records", "action_id": "getdialed__records__dedupe", "parameters": { "record_set_id": "{{ input.record_set_id }}", "key_fields": ["phone"] } } ] }, { "step_id": "step_narrow", "tasks": [ { "task_id": "task_filter", "task_name": "Keep the active records", "platform_id": "getdialed", "service_id": "getdialed__records", "action_id": "getdialed__records__filter", "parameters": { "record_set_id": "{{ step_clean.output.record_set_id }}", "conditions": [{ "field": "status", "op": "==", "value": "active" }] } } ] } ] } ``` The set a step reads is left exactly as it was. A transform never edits its input — it derives a new set beside it. Every task also carries `task_id`, `task_name`, `platform_id` and `service_id`, as any task does. The step snippets further down show only `action_id` and `parameters`, which are the parts that differ between them — see [Flow definitions](/concepts/flow-definitions) for a task's full shape. ## The steps Every step below except the three parse steps takes a `record_set_id` to read; the parse steps take `text` and create the first set. All of them report a `record_set_id` for the set they produced, along with counts. What follows covers what each step does and its main settings — `GET /catalog/actions/{action_id}` publishes the complete schema for any of them. ### Reshaping records Builds a new set with exactly the columns you name. `field_map` is one entry per output column: the entry's **name** is the column that appears on the new records, and its **value** describes what goes in it. ```json theme={null} { "action_id": "getdialed__records__map_fields", "parameters": { "record_set_id": "{{ step_upload.output.record_set_id }}", "field_map": { "number1": "{{ row.phone_number | trim | toE164 }}", "first_name": "{{ row.given_name }}", "email": "{{ row.email_address | trim | lower }}", "external_ref": "ref-{{ row.account_id }}-{{ row.region }}", "source": "imported" } } } ``` Four shapes, all in that one example: a **rename** is the column on its own, a **fixed value** is text with no braces, a **combination** puts several columns in one entry with text around them, and a **formatter** is any [pipe](/concepts/expressions#pipes) applied to the record being processed. By default the new records hold **only** the columns you named and everything else is dropped — which is usually what a reshaping step is for. Set `keep_unmapped` to `true` to carry the rest across as well. Reads a value out of one column, translates it through a table of yours, and writes the result into another column. A state code into a time zone, a lead source into a campaign, a status word into the one your dialler expects. Every other column is carried across unchanged, so this step **adds** information rather than reshaping the record. ```json theme={null} { "action_id": "getdialed__records__lookup_table", "parameters": { "record_set_id": "{{ step_clean.output.record_set_id }}", "source_field": "state", "target_field": "timezone", "table": { "CA": "America/Los_Angeles", "NY": "America/New_York" }, "default_value": "America/Chicago" } } ``` `default_value` is the setting to think about. **Leave it out** and a value your table does not cover is reported as skipped — the record is not carried forward, which is what you want when an unmapped code means the record is not ready. **Supply one, including an empty string**, and it is written instead and the record is kept. A table typed in directly holds up to 10,000 entries. A longer table, or one several flows share, belongs in an [account variable set](/concepts/account-variables) — write `{{ account.lookups.tz_by_state }}` in the `table` field instead. Both behave identically once the step runs. ### Reducing and reordering Keeps the records that match and leaves the rest behind. Each condition names one of your columns, the comparison, and what to compare against. **Every condition must hold** — there is no "either this or that" in this version; run two filters and combine the results instead. ```json theme={null} { "action_id": "getdialed__records__filter", "parameters": { "record_set_id": "{{ step_clean.output.record_set_id }}", "conditions": [ { "field": "status", "op": "==", "value": "active" }, { "field": "score", "op": ">=", "value": 40 }, { "field": "phone", "op": "not_empty" } ] } } ``` The comparisons are `==`, `!=`, `>`, `<`, `>=`, `<=`, `contains`, `is_empty` and `not_empty`. The last two take no value. Keeps one record per distinct value and drops the rest. You choose the columns that decide whether two records are the same, and the **first** record with each value is the one kept, so your set's order survives. ```json theme={null} { "action_id": "getdialed__records__dedupe", "parameters": { "record_set_id": "{{ step_upload.output.record_set_id }}", "key_fields": ["first_name", "last_name"] } } ``` Several columns make **one combined key**: records match only when they agree on every column listed, so listing more columns keeps *more* records, not fewer. To deduplicate on a normalized value rather than what was typed, run `map_fields` first: `{{ row.phone | toE164 }}` writes the normalized number onto the record, then `dedupe` keys on that column — and the normalized value is still there for the step that sends the records on. Puts the set in order, optionally keeping only the first few once it is ordered. ```json theme={null} { "action_id": "getdialed__records__sort", "parameters": { "record_set_id": "{{ step_narrow.output.record_set_id }}", "sort_by": [ { "field": "score", "direction": "desc" }, { "field": "created_at", "direction": "asc" } ], "limit": 500 } } ``` Entries in `sort_by` run most-important-first: the second breaks ties left by the first, the third breaks ties left by the second. `limit` is applied **after** the ordering, so this example takes the five hundred highest-scoring records rather than the first five hundred of the set. Counts, totals, smallest, largest, average, first or last — over the whole set, or once per group. The result is a **new set with one record per group**, so the next step can filter or sort it exactly like any other set. ```json theme={null} { "action_id": "getdialed__records__aggregate", "parameters": { "record_set_id": "{{ step_clean.output.record_set_id }}", "group_by": ["region"], "aggregations": [ { "output_field": "leads", "function": "count" }, { "output_field": "total_value", "function": "sum", "field": "amount" }, { "output_field": "best_score", "function": "max", "field": "score" } ] } } ``` The summaries are `count`, `sum`, `min`, `max`, `avg`, `first` and `last`. `count` needs no source column and must not be given one; every other summary requires one. **Leave `group_by` empty** to summarise the whole set as a single group — that is the ordinary "give me one summary row" case and it needs no placeholder column. ### Splitting and combining Splits one set into several, grouped by the value of a column — the western records one way, the eastern records another. Each output is a full record set a later step can use on its own. ```json theme={null} { "action_id": "getdialed__records__split", "parameters": { "record_set_id": "{{ step_clean.output.record_set_id }}", "split_by": "region" } } ``` The outputs are **named after the values they were grouped by**, and a later step picks one out by that name: `{{ step_split.output.outputs.west }}` for the records whose `region` held `west`. Names rather than positions, so a name still means the same group the next time the flow runs. Choose a column that sorts records into a handful of groups — a region, a status, a campaign. A split produces **at most 50 outputs** and is refused, not shortened, if the column holds more distinct values than that. A record whose value is longer than 16 characters is reported as skipped, because a name that long is not something a later step can reasonably refer to. Combines several sets into one new set. **The order of the list is the order of the result**: all of the first set's records, then all of the second's, and so on. ```json theme={null} { "action_id": "getdialed__records__merge", "parameters": { "record_set_ids": [ "{{ step_priority.output.record_set_id }}", "{{ step_remainder.output.record_set_id }}" ], "mode": "union_by_key", "key_field": "phone" } } ``` `concat` keeps every record from every set. `union_by_key` keeps only the **first** record for each value of `key_field` and drops the later ones — which is how one clean list is built out of several overlapping ones, with the earlier set winning. `key_field` is required for `union_by_key` and refused for `concat`. At least two sets are required, and the same set may not be listed twice. The sets that were combined are left untouched. ### Reading text into records The three parse steps read text the flow is **already holding** — the body of an incoming webhook, a field of one, or the output of an earlier step. See [Parsing text versus uploading a file](#parsing-text-versus-uploading-a-file) below for where the line falls. ```json theme={null} { "action_id": "getdialed__records__parse_csv", "parameters": { "text": "{{ input.body }}", "has_header": true, "delimiter": "," } } ``` Quoted values are understood, including ones containing the separator, a line break or a doubled quote. `delimiter` must be exactly one character — a longer value is refused rather than having its first character used, because a separator that changed silently would misread every row. With `has_header` off, columns are named `column_1`, `column_2` and so on. A row whose number of values does not match its columns is reported as skipped and the rest of the text is still read. ```json theme={null} { "action_id": "getdialed__records__parse_json", "parameters": { "text": "{{ input.payload }}", "root_path": "results.items" } } ``` The document may be a list of records outright — leave `root_path` empty for that — or the list may sit somewhere inside it, in which case say where with names joined by single dots. A path that is not there, or that leads to something other than a list, is refused rather than treated as empty. An entry that is not an object is reported as skipped and the rest of the list is still read. ```json theme={null} { "action_id": "getdialed__records__parse_xml", "parameters": { "text": "{{ input.body }}", "row_element": "row", "attributes_as_fields": true } } ``` `row_element` names the element that repeats once per record. Every occurrence of it anywhere in the document becomes one record, and its child elements become that record's fields. Attributes become fields too by default, named `attr_x` for an attribute `x`, so an attribute and a child element sharing a name can never overwrite one another. A document that declares its own entities, or points at an external file, address or document type, is **rejected without being read**, and nothing it points at is fetched. That is deliberate: XML that can name an external resource is XML that can be used to make the platform fetch one on the author's behalf. Elements nested inside a record are flattened into names containing dots — `contact.email` for an `email` inside a `contact`. A later transform addresses columns by splitting on those dots, so it cannot reach a flattened name and will report every record as skipped. Where a following step needs to read a value, keep the repeating element's fields **flat**, or map the value across with `map_fields` before the rest of the chain. ## The `row` reference Inside `map_fields`, `{{ row. }}` refers to **the record being processed**, and pipes work on it exactly as they do anywhere else: ```text theme={null} {{ row.phone_number | trim | toE164 }} {{ row.email_address | trim | lower }} ref-{{ row.account_id }}-{{ row.region }} ``` `row` is available **only** in that step. Writing it anywhere else is rejected when you save the definition, with an error listing the roots that were actually available where you wrote it. Nothing but the current record is in reach inside those braces, which is what keeps one piece of text meaning the same thing on every record. Every other transform addresses columns by **plain name** rather than by expression — `key_fields`, `sort_by`, `group_by`, `split_by`, `source_field` and a condition's `field` all take a column name, with a dot for a column nested inside another (`contact.state`). If you need a computed key, compute it with `map_fields` first and then key on the column it wrote. The computed value is then visible on the output records too, rather than existing only inside the step that used it. ## When a record cannot be processed **One bad record does not fail the step.** A record that cannot be processed is left out with a reason, and the rest of the set carries on — one unparseable phone number in fifty thousand is not a reason to lose the other forty-nine thousand. Every transform reports the skips the same way afterwards: | Output | What it tells you | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `rejected_count` | How many were skipped, in total. This is the true number. | | `rejected_rows` | The first few, each by its **row number in the source** and why. The list is capped, so a set full of unusable values reports a count rather than a list. | | `records_seen` | How many records the step looked at, skipped ones included — so it reconciles against the source set's own row count. Reported by every step that reads a record set; the parse steps report `row_count` instead, since they are reading text rather than a set. | The row numbers are what to look up in the source — the set a transform read is still there, untouched, so a skipped record can be inspected after the fact. Reasons name the *kind* of problem: a required column was empty, a value was not a number where a comparison needed one, a formatter refused a value. They never quote the value itself, so a phone number or an email address in a bad row is not copied into an error report. ## When nothing comes out **A filter that matches nothing is a legitimate result, not a failure.** So is a parse of text that held no records, and a step whose every record was skipped. In all of those cases the step reports zero and **no new set is created** — `record_set_id` comes back empty. So a step that needs records should check the count first rather than assume a set exists. Guard it on the step's own count with an [exit condition](/concepts/flow-definitions): ```json theme={null} { "step_id": "step_narrow", "tasks": [ { "action_id": "getdialed__records__filter", "parameters": { "record_set_id": "{{ step_clean.output.record_set_id }}", "conditions": [{ "field": "status", "op": "==", "value": "active" }] } } ], "exit_conditions": [ { "condition": "{{ step_narrow.output.matched_count }} == 0", "action": "abort_execution", "reason": "No active records to send" } ] } ``` An exit condition fires when its comparison is **true**, so read that one as "if nothing matched, stop here" — the following step never runs on a set that was never created. Use `skip_next_step` instead where the rest of the flow should carry on regardless. The count to guard on differs by step, and each one says which in its own output schema — `matched_count` for a filter, `group_count` for a summary, `output_count` for a split, `row_count` for the rest. ## Handing a set to another flow `getdialed__utils__trigger_flow` starts another flow with a set of records. It is what makes the preparation worth doing in its own flow: one flow cleans a list, and this step hands the clean list to the flow that does the work. The full composition, end to end — upload a list, remove the duplicates, keep the rows that matter, then start the dialing flow with what is left: ```json theme={null} { "steps": [ { "step_id": "step_dedupe", "tasks": [ { "task_id": "task_dedupe", "task_name": "Remove duplicate numbers", "platform_id": "getdialed", "service_id": "getdialed__records", "action_id": "getdialed__records__dedupe", "parameters": { "record_set_id": "{{ input.record_set_id }}", "key_fields": ["phone"] } } ] }, { "step_id": "step_filter", "tasks": [ { "task_id": "task_filter", "task_name": "Keep the reachable, active records", "platform_id": "getdialed", "service_id": "getdialed__records", "action_id": "getdialed__records__filter", "parameters": { "record_set_id": "{{ step_dedupe.output.record_set_id }}", "conditions": [ { "field": "status", "op": "==", "value": "active" }, { "field": "phone", "op": "not_empty" } ] } } ] }, { "step_id": "step_handoff", "tasks": [ { "task_id": "task_trigger", "task_name": "Start the dialing flow", "platform_id": "getdialed", "service_id": "getdialed__utils", "action_id": "getdialed__utils__trigger_flow", "parameters": { "definition_id": "def_a1b2c3d4", "mode": "bulk", "record_set_id": "{{ step_filter.output.record_set_id }}" } } ] } ] } ``` The dialing flow's batch, its quota and its per-record accounting are all created from the **clean** set, because that is the set it was started with. Nothing is rebound mid-flight. ### The two modes | Mode | What it starts | Use it for | | ------------ | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | `bulk` | **One** run of the named flow, given the whole set, which then works through the records in batches. | Dialing, list uploads — anything that expects a list. | | `per_record` | **One separate run per record**, with that record as the run's input. | Per-lead work: enrichment, a follow-up message, a single record's round trip. | ```json theme={null} { "action_id": "getdialed__utils__trigger_flow", "parameters": { "definition_id": "def_b2c3d4e5", "mode": "per_record", "record_set_id": "{{ step_filter.output.record_set_id }}", "input_data": { "campaign": "Q4 Winback" } } } ``` In `per_record` mode, `input_data` is merged **underneath** the record: where a name appears in both, the record's value is the one the started flow sees. **Per-record mode is limited to 1,000 records, and a larger set is refused outright.** It is not shortened, not sampled and not partly processed — a set of 5,000 records starts nothing at all and reports why. Starting a thousand flows is already a substantial amount of work, and a limit that silently ran the first thousand would leave the other four thousand unaccounted for with nothing to show it happened. To act on a larger set, reduce it first — filter it, or sort it and take a `limit` — or use `bulk` mode with a flow that batches. The step reports `started_count` and `execution_count`, plus `failed_rows`: the first few records whose run could not be started, each by its row number in the set. In `bulk` mode there is one `batch_id`; in `per_record` mode there is a batch per record, so the step reports a capped sample in `batch_ids` and the counts are the true totals. The flow you name must be **active**. An archived or draft flow is refused rather than started, so a flow retired after this step was configured stops producing runs instead of producing them quietly. ## Parsing text versus uploading a file There are two doors, and each does what its size class needs: | What you have | Door | | ----------------------------------------------------------------------------------- | --------------------------------------------------- | | Text already in the flow — a webhook body, a field of one, an earlier step's output | A parse step | | A file | The [record set upload API](/guides/bulk-ingestion) | The parse steps accept **up to 1 MB of text**. Anything larger is refused with a message pointing at the upload API rather than being partly read — a half-read import is worse than a refused one, because nothing downstream can tell it apart from a complete one. ## Tracing what a step produced A derived set is an ordinary record set. Fetch it by id: ```bash theme={null} curl "https://api.getdialed.ai/v1/data/record-sets/rs_0123456789abcdef" \ -H "X-API-Key: $GETDIALED_API_KEY" ``` `GET /v1/data/record-sets/{record_set_id}` returns the set's manifest, and a derived set carries a lineage stamp in its metadata saying where it came from: ```json theme={null} { "metadata": { "derived_from": { "record_set_id": "rs_fedcba9876543210", "action_id": "getdialed__records__filter", "execution_id": "exec_4d5e6f70", "step_id": "task_filter" } } } ``` Four ids and an action name — which set it came from, what derived it, in which run and at which step. That is enough to answer "what did the filter actually output" after the fact, and it holds none of your records' data. There is **no list endpoint** for record sets. A set is read by an id you already hold — the one the step that produced it reported. Derived sets expire on the same schedule as any other record set, so trace a run while it is recent. ## Next steps The pipes that format one value, and the `row` reference these steps use. Upload a file as the record set these steps read. Every step's complete parameter and output reference. Steps, tasks, and the exit conditions that guard an empty result.