Skip to main content

Production setup

A single agent is enough to get started, but in production it's a single point of failure. This page shows you how to keep the agent available on Kubernetes when nodes fail, get drained, or get upgraded, with the Helm chart or with the nullplatform/agent OpenTofu module.

info

Everything on this page works with version 3.6.0 or later of the nullplatform-agent Helm chart. With the OpenTofu module, set nullplatform_agent_helm_version to 3.6.0 or later.

Run several agents​

When your only agent restarts, gets evicted, or loses its node, nothing runs your actions until it comes back. Two or more agents close that gap.

You don't need a load balancer or any coordination between agents. Every replica registers with the same tags, so your notification channel selector matches all of them. Nullplatform then distributes commands among the agents that are available.

Set the number of replicas:

# values.yaml
replicaCount: 2

Check that every replica registered​

Each replica shows up as its own agent, with its own ID and heartbeat. List the active agents that carry your tags:

np agent list \
--organization-id 1 \
--status active \
--tags environment:prod,team:platform

With two replicas, you get two results with the same tags (response trimmed):

{
  "paging": { "limit": 30, "offset": 0 },
  "results": [
    {
      "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
      "status": "active",
      "runtime": "k8s",
      "tags": { "environment": "prod", "team": "platform" },
      "last_heartbeat": "2026-11-11T17:36:42.585Z"
    },
    {
      "id": "9b2e4c71-3d5a-4f8e-b1c6-7a0d2e9f4b13",
      "status": "active",
      "runtime": "k8s",
      "tags": { "environment": "prod", "team": "platform" },
      "last_heartbeat": "2026-11-11T17:37:10.840Z"
    }
  ]
}

Spread agents across nodes​

Replicas only protect you if they don't fail together. By default, Kubernetes can schedule every agent on the same node. If that node goes down, all your agents go down with it. Topology spread constraints tell the scheduler to distribute the agents across nodes or zones.

Default spread​

When you turn it on with no other settings, the agents spread across nodes. It's a soft rule: if the cluster can't satisfy it, Kubernetes still schedules the agents instead of leaving them Pending.

# values.yaml
replicaCount: 2

topologySpreadConstraints:
  enabled: true

That renders this constraint on the agent pod:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: nullplatform-agent
        app.kubernetes.io/instance: <release-name>

To check where the agents landed, list their pods with the node each one runs on:

kubectl get pods -n nullplatform-tools -o wide \
  -l app.kubernetes.io/name=nullplatform-agent

Custom constraints​

To spread by another topology, such as availability zones, pass your own list in constraints (topology_spread_constraints in the OpenTofu module). Your list replaces the default.

You can leave out the labelSelector. The chart fills it in with the agent's labels, so the constraint only counts pods from this agent release. If you set your own labelSelector, the chart keeps it as is.

# values.yaml
replicaCount: 3

topologySpreadConstraints:
  enabled: true
  constraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway
note

whenUnsatisfiable: DoNotSchedule is a hard rule. If your cluster doesn't have enough nodes or zones to satisfy it, the extra agents stay Pending. Use ScheduleAnyway unless you're sure the capacity is there.

Requirements​

Topology spread needs more than one agent. With a single replica, the install fails instead of ignoring the setting: the Helm chart refuses to render, and the OpenTofu module fails at plan. If you enable the chart's autoscaling, the chart checks autoscaling.maxReplicas instead of replicaCount.

The OpenTofu module also fails at plan when you set topology_spread_constraints but leave topology_spread_constraints_enabled off, since nothing would apply them. It checks each constraint too: whenUnsatisfiable must be DoNotSchedule or ScheduleAnyway, and maxSkew a whole number of 1 or more.

Keep agents up during node drains​

Spreading protects you from a node that fails. Node drains are different: cluster upgrades, maintenance, and autoscaler scale-downs evict pods on purpose, and nothing stops them from evicting every agent at once. A PodDisruptionBudget limits how many agents Kubernetes can evict at the same time.

# values.yaml
replicaCount: 2

podDisruptionBudget:
  enabled: true
  maxUnavailable: 1

Set either minAvailable or maxUnavailable, not both. If you set neither, the budget uses maxUnavailable: 1.

The chart refuses a budget that would never allow an eviction, because it would hang every node drain and cluster upgrade on the agent's node. That means maxUnavailable: 0 fails, and so does a minAvailable that isn't lower than the replica count (autoscaling.minReplicas when autoscaling is on).

tip

Node autoscalers are already covered. The chart adds two annotations to the agent pod by default so autoscalers don't evict it to consolidate or scale down its node: karpenter.sh/do-not-disrupt for Karpenter and cluster-autoscaler.kubernetes.io/safe-to-evict for Cluster Autoscaler. With Karpenter, the agent's node is then never consolidated or expired, so give the NodePool a terminationGracePeriod if its nodes still need to rotate.

Let running commands finish on shutdown​

Even with a budget, each agent eventually stops: during a rollout, a drain, or a restart. When Kubernetes sends it SIGTERM, the agent waits for the commands it's running to finish and report their result. Then it disconnects. Nullplatform re-delivers any command that's still running after the wait to the next available agent.

Two settings control this:

  • Shutdown drain (shutdownDrainTimeoutSeconds, default 90): how long the agent waits for running commands. Size it to your longest action. For reference, a start-blue-green takes about 70 seconds. Set it to 0 to skip the wait.
  • Grace period (terminationGracePeriodSeconds, default 120): how long Kubernetes gives the agent before it kills it. It has to be at least 10 seconds above the drain, so the agent has time to disconnect. The chart refuses a value that doesn't leave that margin.

The defaults fit most setups. If your actions take longer, raise both:

# values.yaml
shutdownDrainTimeoutSeconds: 150
terminationGracePeriodSeconds: 180
note

The shutdown drain needs agent 0.14.0 or later. The chart uses the latest agent image by default, so this only matters if you pin the image tag.

Put it all together​

A production baseline with two agents spread across nodes, a disruption budget, and the default shutdown drain:

# values.yaml
replicaCount: 2

topologySpreadConstraints:
  enabled: true

podDisruptionBudget:
  enabled: true
  maxUnavailable: 1

Next steps​