Skip to main content

Items and data flow

Every step in a workflow receives an array and emits an array. The elements are called items, and an item is a plain JSON object: {"scope_id": 1234, "ami": "ami-0abc"}. There is no envelope, no wrapper, no special binary type.

Keep that rule in mind and the rest of the page follows from it: why a step sometimes runs five times, why a summary step sees everything at once, and how a fan-out over a thousand scopes behaves.

Read a step's results with items or outputs​

When a step finishes, the engine records its items. You read them with two expressions:

  • steps.detect.items returns the full array.
  • steps.detect.outputs.total returns the total field of the first item. It's shorthand for steps.detect.items[0].total.

Use outputs when a step produces a single result, which is the common case. Use items when a step returns a collection: outputs still works there, but it silently gives you only the first item.

inputs:
  total:    "${{ steps.detect.outputs.total }}"   # one field of the first item
  findings: "${{ steps.detect.items }}"           # every item

How many times a step runs: per item or once per batch​

Each plugin declares one of two modes, and the engine invokes it accordingly.

In per-item mode the engine calls the plugin once per input item. Three items in means three invocations, and the three results are collected back into one array. Inside each invocation the step sees $item (the object it's working on), $itemIndex (its position) and $itemsLength (how many items are in the batch).

In whole-batch mode the engine calls the plugin once and hands it the entire array as $items. One invocation, one result. Use this mode for anything that aggregates, counts, or summarizes.

The defaults cover the common case without any configuration:

PluginModeWhy
code-exec, log, set-variablewhole batchAggregating is the usual reason to reach for them
sub-workflowper itemOne child run per item is nearly always the intent
http-request, and most othersper itemOne call per item is usually correct

To change the mode on one step, set metadata.fanOutPerItem: true makes a whole-batch plugin run once per item, and false makes a per-item plugin run once with the whole batch.

- id: mark_processed
  type: module
  plugin_type: code-exec
  metadata:
    fanOutPerItem: true      # code-exec takes the whole batch by default
  config:
    code: |
      var r = $item || {};
      return { ...r, processed: true };

Why $items is unavailable in a per-item step​

A per-item invocation gets $item, $itemIndex and $itemsLength, and never $items. If a per-item step references $items, graph validation fails with FANOUT_ITEMS_UNAVAILABLE and the workflow can't be saved.

The reason is size. Copying the whole batch into every per-item invocation multiplies the payload by the number of items, so a hundred-item page of scopes becomes a hundred copies of that page. That is what ended a production run mid-sweep, and it's why the engine now rejects it before the run starts.

If a step needs the whole batch, run it in whole-batch mode: put the aggregation in a set-variable or code-exec step, which already work that way.

Automatic fan-out, and what turns it off​

A step doesn't need a forEach block to run many times. When the step before it hands over several items and its plugin runs per item, the engine dispatches it once per item automatically, up to 5 at a time by default.

Four things turn that automatic fan-out off. Check them first when a step you expected to run per item ran only once:

  • More than one upstream edge carries items. That is a join, not a fan-out: the plugin runs once with every upstream item concatenated. This is also what keeps re-entry loops from firing forever.
  • The step declares forEach. You asked for explicit iteration, so the engine doesn't add its own.
  • error_handling.retry_policy.max_attempts is greater than 1. A retry policy on a step disables its automatic fan-out. Put the retry on the steps inside a sub-workflow instead.
  • metadata.fanOutPerItem: false. The explicit switch.

To control how many items run at once, set metadata.maxParallelism on the step. If it's not set, the engine uses the plugin's config.maxParallelism, and failing that the default of 5.

Run independent branches in parallel​

A branch is a path that leaves a step: two steps connected to the same predecessor, or the true and false ports of a decider. Nothing in the graph forces an order between branches, but by default the engine runs them one after another anyway. It starts one ready step, waits for it to finish, and only then starts the next.

That default is safe rather than fast. Branches written expecting to run in series often share a rate-limited API, or write the same variable, and when two branches write the same variable the last write wins. Running them side by side could change what the run produces.

To run branches in parallel, first check that they are independent: no shared variable, no API that would throttle two callers at once. Then set maxStepParallelism on the workflow, which caps how many ready steps run at the same time:

metadata:
  maxStepParallelism: 3   # 1 is the default, 32 the maximum

Whatever the value, ready steps always start in step id order, and when several fail at once the run reports the failure from the lowest step id. A run stays predictable no matter which branch happens to finish first.

This setting is for branches only. To run the items of one step in parallel, use metadata.maxParallelism on that step, as described in the previous section, or a forEach block with parallel: true.

How deciders split items between ports​

A decider in per-item mode evaluates its expression once per item and sends each item out the matching port, so one input batch becomes two output batches:

in:  [{value: 5}, {value: 15}, {value: 8}, {value: 20}]

conditional  (value > 10)
  true  →  [{value: 15}, {value: 20}]
  false →  [{value: 5},  {value: 8}]

Both ports always carry an array. A branch that matched nothing receives an empty array rather than nothing at all.

Join several steps into one with a join strategy​

When a step has more than one incoming connection, the engine needs a rule for when to start it. That rule is the join strategy, set with join_strategy on the step:

StrategyFires when
all (default)Every predecessor has reached a terminal state. A failed predecessor counts as terminal
anyThe first predecessor completes. Failed, skipped, and cancelled ones never satisfy it
allSettledEvery predecessor is terminal, success or failure alike. Skipped ones count as settled
countAt least join_count predecessors have completed. Failed and skipped ones don't count toward the quorum

Whatever arrives from the predecessors is concatenated into one input array.

Two cases come up in almost every workflow that joins:

A loop back into a node needs join_strategy: any. A node that a later step loops back into has two incoming edges: the first entry and the re-entry. With the default all it waits on both forever and the loop deadlocks. Every published suite that loops sets any on that node.

Give success and failure paths separate nodes. A node fed by one completed edge and one failure edge can't start under any strategy: any ignores the failure edge and waits for a success that never comes, and all waits on the edge that was neither settled nor skipped. Point each path at its own node with a single predecessor, where the default all is satisfied as soon as that predecessor finishes:

- id: resolve_ok        # single edge from classify_ok:true
- id: resolve_failed    # single edge from the failure lane

This is why error handling uses one resolve node per lane instead of a shared final node.

Next steps​