Record transforms

A record transform is a step that takes a set of records, does one thing to all of them, and produces a new set. Remove the duplicates from an upload. Keep only the rows that matter. Put a list in order and take the top few. Tidy every phone number into its international form. Eleven of these ship today, plus a twelfth step that hands the result to another flow. They need no credential and call nothing outside the platform.

{
  "task_id": "task_dedupe",
  "task_name": "Remove duplicate numbers",
  "platform_id": "getdialed",
  "service_id": "getdialed__records",
  "action_id": "getdialed__records__dedupe",
  "parameters": {
    "record_set_id": "{{ step_upload.output.record_set_id }}",
    "key_fields": ["phone"]
  }
}

Why these are steps and not formatters

A pipe formats one value in one record — it trims a name, rounds a cost, normalizes a number. It only ever sees the record it is standing in. Removing duplicates across an entire upload is a different kind of question. So is sorting a whole list, or working out a total per region. Those need the whole set at once, which is why they are steps rather than something you can write inside a pair of braces. The split follows the shape of the work, not its difficulty:

You want to	Use
Reformat a value on the record in front of you	A pipe, inline in the parameter
Reshape, reduce or reorder a whole set of records	A record transform step

How the records travel

By reference, never by value. A transform step is given the identifier of a record set, reads it as the step runs, and hands back the identifier of the set it produced. The records themselves never travel through the flow. That is what makes a set of any size workable: the flow carries an id and a handful of counts, not fifty thousand rows. It is also why steps chain the way they do — each step’s output id is bound into the next step’s input exactly like any other step output:

{
  "steps": [
    {
      "step_id": "step_clean",
      "tasks": [
        {
          "task_id": "task_dedupe",
          "task_name": "Remove duplicate numbers",
          "platform_id": "getdialed",
          "service_id": "getdialed__records",
          "action_id": "getdialed__records__dedupe",
          "parameters": {
            "record_set_id": "{{ input.record_set_id }}",
            "key_fields": ["phone"]
          }
        }
      ]
    },
    {
      "step_id": "step_narrow",
      "tasks": [
        {
          "task_id": "task_filter",
          "task_name": "Keep the active records",
          "platform_id": "getdialed",
          "service_id": "getdialed__records",
          "action_id": "getdialed__records__filter",
          "parameters": {
            "record_set_id": "{{ step_clean.output.record_set_id }}",
            "conditions": [{ "field": "status", "op": "==", "value": "active" }]
          }
        }
      ]
    }
  ]
}

The set a step reads is left exactly as it was. A transform never edits its input — it derives a new set beside it.

Every task also carries task_id, task_name, platform_id and service_id, as any task does. The step snippets further down show only action_id and parameters, which are the parts that differ between them — see Flow definitions for a task’s full shape.

The steps

Every step below except the three parse steps takes a record_set_id to read; the parse steps take text and create the first set. All of them report a record_set_id for the set they produced, along with counts. What follows covers what each step does and its main settings — GET /catalog/actions/{action_id} publishes the complete schema for any of them.

Reshaping records

Rename and compute fields — getdialed__records__map_fields

Builds a new set with exactly the columns you name. field_map is one entry per output column: the entry’s name is the column that appears on the new records, and its value describes what goes in it.

{
  "action_id": "getdialed__records__map_fields",
  "parameters": {
    "record_set_id": "{{ step_upload.output.record_set_id }}",
    "field_map": {
      "number1": "{{ row.phone_number | trim | toE164 }}",
      "first_name": "{{ row.given_name }}",
      "email": "{{ row.email_address | trim | lower }}",
      "external_ref": "ref-{{ row.account_id }}-{{ row.region }}",
      "source": "imported"
    }
  }
}

Four shapes, all in that one example: a rename is the column on its own, a fixed value is text with no braces, a combination puts several columns in one entry with text around them, and a formatter is any pipe applied to the record being processed.By default the new records hold only the columns you named and everything else is dropped — which is usually what a reshaping step is for. Set keep_unmapped to true to carry the rest across as well.

Map values through a table — getdialed__records__lookup_table

Reads a value out of one column, translates it through a table of yours, and writes the result into another column. A state code into a time zone, a lead source into a campaign, a status word into the one your dialler expects. Every other column is carried across unchanged, so this step adds information rather than reshaping the record.

{
  "action_id": "getdialed__records__lookup_table",
  "parameters": {
    "record_set_id": "{{ step_clean.output.record_set_id }}",
    "source_field": "state",
    "target_field": "timezone",
    "table": { "CA": "America/Los_Angeles", "NY": "America/New_York" },
    "default_value": "America/Chicago"
  }
}

default_value is the setting to think about. Leave it out and a value your table does not cover is reported as skipped — the record is not carried forward, which is what you want when an unmapped code means the record is not ready. Supply one, including an empty string, and it is written instead and the record is kept.A table typed in directly holds up to 10,000 entries. A longer table, or one several flows share, belongs in an account variable set — write {{ account.lookups.tz_by_state }} in the table field instead. Both behave identically once the step runs.

Reducing and reordering

Keep matching records — getdialed__records__filter

Keeps the records that match and leaves the rest behind. Each condition names one of your columns, the comparison, and what to compare against. Every condition must hold — there is no “either this or that” in this version; run two filters and combine the results instead.

{
  "action_id": "getdialed__records__filter",
  "parameters": {
    "record_set_id": "{{ step_clean.output.record_set_id }}",
    "conditions": [
      { "field": "status", "op": "==", "value": "active" },
      { "field": "score", "op": ">=", "value": 40 },
      { "field": "phone", "op": "not_empty" }
    ]
  }
}

The comparisons are ==, !=, >, <, >=, <=, contains, is_empty and not_empty. The last two take no value.

Remove duplicate records — getdialed__records__dedupe

Keeps one record per distinct value and drops the rest. You choose the columns that decide whether two records are the same, and the first record with each value is the one kept, so your set’s order survives.

{
  "action_id": "getdialed__records__dedupe",
  "parameters": {
    "record_set_id": "{{ step_upload.output.record_set_id }}",
    "key_fields": ["first_name", "last_name"]
  }
}

Several columns make one combined key: records match only when they agree on every column listed, so listing more columns keeps more records, not fewer.

To deduplicate on a normalized value rather than what was typed, run map_fields first: {{ row.phone | toE164 }} writes the normalized number onto the record, then dedupe keys on that column — and the normalized value is still there for the step that sends the records on.

Sort records — getdialed__records__sort

Puts the set in order, optionally keeping only the first few once it is ordered.

{
  "action_id": "getdialed__records__sort",
  "parameters": {
    "record_set_id": "{{ step_narrow.output.record_set_id }}",
    "sort_by": [
      { "field": "score", "direction": "desc" },
      { "field": "created_at", "direction": "asc" }
    ],
    "limit": 500
  }
}

Entries in sort_by run most-important-first: the second breaks ties left by the first, the third breaks ties left by the second. limit is applied after the ordering, so this example takes the five hundred highest-scoring records rather than the first five hundred of the set.

Summarise records — getdialed__records__aggregate

Counts, totals, smallest, largest, average, first or last — over the whole set, or once per group. The result is a new set with one record per group, so the next step can filter or sort it exactly like any other set.

{
  "action_id": "getdialed__records__aggregate",
  "parameters": {
    "record_set_id": "{{ step_clean.output.record_set_id }}",
    "group_by": ["region"],
    "aggregations": [
      { "output_field": "leads", "function": "count" },
      { "output_field": "total_value", "function": "sum", "field": "amount" },
      { "output_field": "best_score", "function": "max", "field": "score" }
    ]
  }
}

The summaries are count, sum, min, max, avg, first and last. count needs no source column and must not be given one; every other summary requires one. Leave group_by empty to summarise the whole set as a single group — that is the ordinary “give me one summary row” case and it needs no placeholder column.

Splitting and combining

Split records — getdialed__records__split

Splits one set into several, grouped by the value of a column — the western records one way, the eastern records another. Each output is a full record set a later step can use on its own.

{
  "action_id": "getdialed__records__split",
  "parameters": {
    "record_set_id": "{{ step_clean.output.record_set_id }}",
    "split_by": "region"
  }
}

The outputs are named after the values they were grouped by, and a later step picks one out by that name: {{ step_split.output.outputs.west }} for the records whose region held west. Names rather than positions, so a name still means the same group the next time the flow runs.Choose a column that sorts records into a handful of groups — a region, a status, a campaign. A split produces at most 50 outputs and is refused, not shortened, if the column holds more distinct values than that. A record whose value is longer than 16 characters is reported as skipped, because a name that long is not something a later step can reasonably refer to.

Combine record sets — getdialed__records__merge

Combines several sets into one new set. The order of the list is the order of the result: all of the first set’s records, then all of the second’s, and so on.

{
  "action_id": "getdialed__records__merge",
  "parameters": {
    "record_set_ids": [
      "{{ step_priority.output.record_set_id }}",
      "{{ step_remainder.output.record_set_id }}"
    ],
    "mode": "union_by_key",
    "key_field": "phone"
  }
}

concat keeps every record from every set. union_by_key keeps only the first record for each value of key_field and drops the later ones — which is how one clean list is built out of several overlapping ones, with the earlier set winning. key_field is required for union_by_key and refused for concat.At least two sets are required, and the same set may not be listed twice. The sets that were combined are left untouched.

Reading text into records

The three parse steps read text the flow is already holding — the body of an incoming webhook, a field of one, or the output of an earlier step. See Parsing text versus uploading a file below for where the line falls.

Read records from CSV text — getdialed__records__parse_csv

{
  "action_id": "getdialed__records__parse_csv",
  "parameters": {
    "text": "{{ input.body }}",
    "has_header": true,
    "delimiter": ","
  }
}

Quoted values are understood, including ones containing the separator, a line break or a doubled quote. delimiter must be exactly one character — a longer value is refused rather than having its first character used, because a separator that changed silently would misread every row. With has_header off, columns are named column_1, column_2 and so on. A row whose number of values does not match its columns is reported as skipped and the rest of the text is still read.

Read records from JSON text — getdialed__records__parse_json

{
  "action_id": "getdialed__records__parse_json",
  "parameters": {
    "text": "{{ input.payload }}",
    "root_path": "results.items"
  }
}

The document may be a list of records outright — leave root_path empty for that — or the list may sit somewhere inside it, in which case say where with names joined by single dots. A path that is not there, or that leads to something other than a list, is refused rather than treated as empty. An entry that is not an object is reported as skipped and the rest of the list is still read.

Read records from XML text — getdialed__records__parse_xml

{
  "action_id": "getdialed__records__parse_xml",
  "parameters": {
    "text": "{{ input.body }}",
    "row_element": "row",
    "attributes_as_fields": true
  }
}

row_element names the element that repeats once per record. Every occurrence of it anywhere in the document becomes one record, and its child elements become that record’s fields. Attributes become fields too by default, named attr_x for an attribute x, so an attribute and a child element sharing a name can never overwrite one another.

A document that declares its own entities, or points at an external file, address or document type, is rejected without being read, and nothing it points at is fetched. That is deliberate: XML that can name an external resource is XML that can be used to make the platform fetch one on the author’s behalf.

Elements nested inside a record are flattened into names containing dots — contact.email for an email inside a contact. A later transform addresses columns by splitting on those dots, so it cannot reach a flattened name and will report every record as skipped. Where a following step needs to read a value, keep the repeating element’s fields flat, or map the value across with map_fields before the rest of the chain.

The `row` reference

Inside map_fields, {{ row.<column> }} refers to the record being processed, and pipes work on it exactly as they do anywhere else:

{{ row.phone_number | trim | toE164 }}
{{ row.email_address | trim | lower }}
ref-{{ row.account_id }}-{{ row.region }}

row is available only in that step. Writing it anywhere else is rejected when you save the definition, with an error listing the roots that were actually available where you wrote it. Nothing but the current record is in reach inside those braces, which is what keeps one piece of text meaning the same thing on every record. Every other transform addresses columns by plain name rather than by expression — key_fields, sort_by, group_by, split_by, source_field and a condition’s field all take a column name, with a dot for a column nested inside another (contact.state). If you need a computed key, compute it with map_fields first and then key on the column it wrote. The computed value is then visible on the output records too, rather than existing only inside the step that used it.

When a record cannot be processed

One bad record does not fail the step. A record that cannot be processed is left out with a reason, and the rest of the set carries on — one unparseable phone number in fifty thousand is not a reason to lose the other forty-nine thousand. Every transform reports the skips the same way afterwards:

Output	What it tells you
`rejected_count`	How many were skipped, in total. This is the true number.
`rejected_rows`	The first few, each by its row number in the source and why. The list is capped, so a set full of unusable values reports a count rather than a list.
`records_seen`	How many records the step looked at, skipped ones included — so it reconciles against the source set’s own row count. Reported by every step that reads a record set; the parse steps report `row_count` instead, since they are reading text rather than a set.

The row numbers are what to look up in the source — the set a transform read is still there, untouched, so a skipped record can be inspected after the fact. Reasons name the kind of problem: a required column was empty, a value was not a number where a comparison needed one, a formatter refused a value. They never quote the value itself, so a phone number or an email address in a bad row is not copied into an error report.

When nothing comes out

A filter that matches nothing is a legitimate result, not a failure. So is a parse of text that held no records, and a step whose every record was skipped. In all of those cases the step reports zero and no new set is created — record_set_id comes back empty. So a step that needs records should check the count first rather than assume a set exists. Guard it on the step’s own count with an exit condition:

{
  "step_id": "step_narrow",
  "tasks": [
    {
      "action_id": "getdialed__records__filter",
      "parameters": {
        "record_set_id": "{{ step_clean.output.record_set_id }}",
        "conditions": [{ "field": "status", "op": "==", "value": "active" }]
      }
    }
  ],
  "exit_conditions": [
    {
      "condition": "{{ step_narrow.output.matched_count }} == 0",
      "action": "abort_execution",
      "reason": "No active records to send"
    }
  ]
}

An exit condition fires when its comparison is true, so read that one as “if nothing matched, stop here” — the following step never runs on a set that was never created. Use skip_next_step instead where the rest of the flow should carry on regardless. The count to guard on differs by step, and each one says which in its own output schema — matched_count for a filter, group_count for a summary, output_count for a split, row_count for the rest.

Handing a set to another flow

getdialed__utils__trigger_flow starts another flow with a set of records. It is what makes the preparation worth doing in its own flow: one flow cleans a list, and this step hands the clean list to the flow that does the work. The full composition, end to end — upload a list, remove the duplicates, keep the rows that matter, then start the dialing flow with what is left:

{
  "steps": [
    {
      "step_id": "step_dedupe",
      "tasks": [
        {
          "task_id": "task_dedupe",
          "task_name": "Remove duplicate numbers",
          "platform_id": "getdialed",
          "service_id": "getdialed__records",
          "action_id": "getdialed__records__dedupe",
          "parameters": {
            "record_set_id": "{{ input.record_set_id }}",
            "key_fields": ["phone"]
          }
        }
      ]
    },
    {
      "step_id": "step_filter",
      "tasks": [
        {
          "task_id": "task_filter",
          "task_name": "Keep the reachable, active records",
          "platform_id": "getdialed",
          "service_id": "getdialed__records",
          "action_id": "getdialed__records__filter",
          "parameters": {
            "record_set_id": "{{ step_dedupe.output.record_set_id }}",
            "conditions": [
              { "field": "status", "op": "==", "value": "active" },
              { "field": "phone", "op": "not_empty" }
            ]
          }
        }
      ]
    },
    {
      "step_id": "step_handoff",
      "tasks": [
        {
          "task_id": "task_trigger",
          "task_name": "Start the dialing flow",
          "platform_id": "getdialed",
          "service_id": "getdialed__utils",
          "action_id": "getdialed__utils__trigger_flow",
          "parameters": {
            "definition_id": "def_a1b2c3d4",
            "mode": "bulk",
            "record_set_id": "{{ step_filter.output.record_set_id }}"
          }
        }
      ]
    }
  ]
}

The dialing flow’s batch, its quota and its per-record accounting are all created from the clean set, because that is the set it was started with. Nothing is rebound mid-flight.

The two modes

Mode	What it starts	Use it for
`bulk`	One run of the named flow, given the whole set, which then works through the records in batches.	Dialing, list uploads — anything that expects a list.
`per_record`	One separate run per record, with that record as the run’s input.	Per-lead work: enrichment, a follow-up message, a single record’s round trip.

{
  "action_id": "getdialed__utils__trigger_flow",
  "parameters": {
    "definition_id": "def_b2c3d4e5",
    "mode": "per_record",
    "record_set_id": "{{ step_filter.output.record_set_id }}",
    "input_data": { "campaign": "Q4 Winback" }
  }
}

In per_record mode, input_data is merged underneath the record: where a name appears in both, the record’s value is the one the started flow sees.

Per-record mode is limited to 1,000 records, and a larger set is refused outright. It is not shortened, not sampled and not partly processed — a set of 5,000 records starts nothing at all and reports why. Starting a thousand flows is already a substantial amount of work, and a limit that silently ran the first thousand would leave the other four thousand unaccounted for with nothing to show it happened.To act on a larger set, reduce it first — filter it, or sort it and take a limit — or use bulk mode with a flow that batches.

The step reports started_count and execution_count, plus failed_rows: the first few records whose run could not be started, each by its row number in the set. In bulk mode there is one batch_id; in per_record mode there is a batch per record, so the step reports a capped sample in batch_ids and the counts are the true totals. The flow you name must be active. An archived or draft flow is refused rather than started, so a flow retired after this step was configured stops producing runs instead of producing them quietly.

Parsing text versus uploading a file

There are two doors, and each does what its size class needs:

What you have	Door
Text already in the flow — a webhook body, a field of one, an earlier step’s output	A parse step
A file	The record set upload API

The parse steps accept up to 1 MB of text. Anything larger is refused with a message pointing at the upload API rather than being partly read — a half-read import is worse than a refused one, because nothing downstream can tell it apart from a complete one.

Tracing what a step produced

A derived set is an ordinary record set. Fetch it by id:

curl "https://api.getdialed.ai/v1/data/record-sets/rs_0123456789abcdef" \
  -H "X-API-Key: $GETDIALED_API_KEY"

GET /v1/data/record-sets/{record_set_id} returns the set’s manifest, and a derived set carries a lineage stamp in its metadata saying where it came from:

{
  "metadata": {
    "derived_from": {
      "record_set_id": "rs_fedcba9876543210",
      "action_id": "getdialed__records__filter",
      "execution_id": "exec_4d5e6f70",
      "step_id": "task_filter"
    }
  }
}

Four ids and an action name — which set it came from, what derived it, in which run and at which step. That is enough to answer “what did the filter actually output” after the fact, and it holds none of your records’ data.

There is no list endpoint for record sets. A set is read by an id you already hold — the one the step that produced it reported. Derived sets expire on the same schedule as any other record set, so trace a run while it is recent.

Next steps

Expressions

The pipes that format one value, and the row reference these steps use.

Bulk ingestion

Upload a file as the record set these steps read.

Catalog

Every step’s complete parameter and output reference.

Flow definitions

Steps, tasks, and the exit conditions that guard an empty result.

Getting Started

Core Concepts

Building Flows

Triggers & Streaming

Why these are steps and not formatters

How the records travel

The steps

Reshaping records

Reducing and reordering

Splitting and combining

Reading text into records

The `row` reference

When a record cannot be processed

When nothing comes out

Handing a set to another flow

The two modes

Parsing text versus uploading a file

Tracing what a step produced

Next steps

Expressions

Bulk ingestion

Catalog

Flow definitions

​Why these are steps and not formatters

​How the records travel

​The steps

​Reshaping records

​Reducing and reordering

​Splitting and combining

​Reading text into records

​The row reference

​When a record cannot be processed

​When nothing comes out

​Handing a set to another flow

​The two modes

​Parsing text versus uploading a file

​Tracing what a step produced

​Next steps

Expressions

Bulk ingestion

Catalog

Flow definitions

Why these are steps and not formatters

How the records travel

The steps

Reshaping records

Reducing and reordering

Splitting and combining

Reading text into records

The `row` reference

When a record cannot be processed

When nothing comes out

Handing a set to another flow

The two modes

Parsing text versus uploading a file

Tracing what a step produced

Next steps