Loops and iteration

There are five ways to repeat work in a workflow, and each one fits a different case. Picking the wrong one shows up late, as a run that deadlocks or a fan-out that exceeds a size budget, so choose before wiring anything.

Everything here builds on items and data flow: a step that receives many items may already run many times without any loop construct at all.

Choose the right kind of loop

You want to	Use
Page through a large source and act on every item, with a final total	The streaming pagination pattern
Process an array you already have, without spawning child runs	forEach on the step
Give each item its own run, its own state budget, and its own failure	sub-workflow
Batch items and control the loop yourself with explicit ports	split-in-batches
Poll until something becomes true	loop.while on the step

If all you need is to run a step once per item it receives, you may not need any of them: the engine already runs per-item plugins once per item.

Repeat a step with forEach

The simplest explicit loop. Point it at an array, name the variable, and the step runs once per element:

- id: ensure_items
  type: module
  plugin_type: sub-workflow
  forEach:
    expression: "${{ steps.detect_drift.outputs.findings }}"
    itemVariable: finding
    parallel: true
    maxConcurrency: 5
  config:
    workflowId: wf_ensure_action_item
    waitForCompletion: true

Each element arrives as inputs.finding. Iterations retry independently, and in parallel mode one failed iteration doesn't stop the rest.

Iterations run one at a time by default. Set parallel: true to run up to maxConcurrency of them at once, 5 unless you say otherwise. On a large fan-out the difference is large: a thousand sequential sub-workflow items took about 50 minutes, and running 5 at a time was roughly three times faster.

maxConcurrency must be a literal number. It accepts no expressions, so changing it means saving a new version.

To fold every element into one value instead of producing one result per element, add a reduce block. It always runs sequentially and can't be combined with parallel: true:

- id: total_savings
  type: module
  plugin_type: code-exec
  forEach:
    expression: "${{ steps.price_scopes.items }}"
    reduce:
      initialValue: 0
      accumulator: acc
  config:
    code: |
      return inputs.acc + inputs.item.monthly_saving;

Run a child workflow per item with sub-workflow

sub-workflow iterates by default: hand it N items and it starts N child runs, 5 at a time.

- id: scan_scopes
  type: module
  plugin_type: sub-workflow
  config:
    workflowId: cost_scope_collect
    iterateItems: true      # the default
    maxParallelism: 5

Set iterateItems: false and you get one child run receiving every item as inputs.items instead.

This is the heavier option, and often the right one. Each child is a real execution with its own id, its own logs, and its own state budget, so a large sweep doesn't accumulate everything into one parent run. When a fan-out is big enough to hit HISTORY_LIMIT_EXCEEDED, this is the fix.

Process items in batches with split-in-batches

Use it when you want the batching visible on the canvas and under your control. The node has two output ports: loop fires once per batch, and done fires when no batches are left. Connect the last step of your processing back into the node to request the next batch.

Config	Default
batchSize	1	Items per batch
reset	false	Restart the iteration state on re-entry

Its outputs carry items (the current batch, or everything processed once done), batchIndex, totalBatches, and done.

Poll with loop.while

A step can repeat itself until a condition becomes false or it reaches a cap:

- id: poll_status
  type: module
  plugin_type: http-request
  loop:
    while: "steps.poll_status.outputs.status != 'ready'"
    maxIterations: 50
  config:
    url: "https://api.example.com/status"

It's the least used of the five, and no published suite uses it. When you're waiting on something external, prefer signal-wait or one of the *-wait nodes, which park the run at no cost instead of spending iterations. Keep loop.while for polling a source that offers no callback.

Page through a large source

This is the standard shape for scanning everything and aggregating a result, and the one to copy for any organization-wide sweep. Four visible nodes, no hidden iteration:

variables:
  app_results:
    initialValue: []

steps:
  # 1. One page per invocation.
  - id: fetch_applications
    type: module
    plugin_type: np-entity-paginated-fetch
    config:
      entity: application
      mode: stream

  # 2. One child run per item (sub-workflow iterates by default).
  - id: scan_application
    type: module
    plugin_type: sub-workflow
    config: { workflowId: cost_action_items }
    inputs: { application_id: "${{ $item.id }}" }

  # 3. Append this page to a variable. Runs once per page, because
  #    set-variable takes the whole batch.
  - id: acc_apps
    type: module
    plugin_type: set-variable
    metadata: { flipPorts: true }
    config:
      path: app_results
      value: "${{ concat(variables.app_results, $items) }}"

  # 4. Summary reads the accumulator, never $items.
  - id: summary
    type: module
    plugin_type: code-exec
    inputs:
      results: "${{ variables.app_results }}"
    config:
      code: |
        var items = inputs.results || [];
        return { apps_scanned: items.length };

connections:
  - { from: start,              to: fetch_applications }
  - { from: fetch_applications, to: scan_application,   source_port: loop }
  - { from: scan_application,   to: acc_apps }
  - { from: acc_apps,           to: fetch_applications, target_port: callback }
  - { from: fetch_applications, to: summary,            source_port: done }

Four rules make it work, and breaking any of them produces a real failure:

• Items come out of the loop port on every page, including the last. The done port carries an empty array.
• The summary connects to done but reads variables.*. Reading $items there gets you an empty array, every time.
• Put the accumulator between the work and the back-edge. Without it, each iteration's data is lost when the loop fires again.
• Grow the list with concat(). concat(variables.results, $items) appends this page to what's already there.

Cycles in the graph and the 1000 execution cap

The graph is allowed to contain cycles. A connection back to an earlier node simply runs that node again, and nodes that keep state between runs (like split-in-batches) recognize the re-entry.

Two rules apply. A node may run at most 1000 times in one run; past that the run fails with LOOP_LIMIT_EXCEEDED. And a node that a loop comes back into needs join_strategy: any, because it has two incoming edges and the default all would wait on both forever.

Common mistakes

• A retry policy disables automatic fan-out. max_attempts above 1 on a step that would otherwise dispatch per item turns it into a single whole-batch call. Put the retry inside the sub-workflow instead, with jitter so parallel children don't retry at the same moment.
• $items is unavailable in a per-item step. It fails validation with FANOUT_ITEMS_UNAVAILABLE. Aggregate in a whole-batch step.
• error_handling.fallback_step is ignored on a forEach step. Only continueOnError is honored there. See error handling.
• A re-entry loop without an accumulator loses its data. Add set-variable on the loop tail.
• The forEach block is YAML only. The editor's node panel doesn't expose it yet.

Next steps

• Items and data flow: why a step runs once or many times
• Error handling: isolating a failed iteration
• Limits: what a large fan-out costs
• Right-size Kubernetes costs: the pagination pattern in a working suite