--- sidebar_label: Production setup toc_max_heading_level: 3 doc_id: 1beb4ce1-2499-4b6b-9d6a-ca4eca84c054 description: >- Configure the nullplatform agent for production: run several replicas, spread them across nodes or zones, and keep running commands safe during node drains and restarts. keywords: - nullplatform agent - production - high availability - replicas - topology spread constraints - pod disruption budget - graceful shutdown - helm - opentofu title: Production setup canonical: 'https://docs.nullplatform.com/docs/agent/production-setup' --- # Production setup A single agent is enough to get started, but in production it's a single point of failure. This page shows you how to keep the agent available on Kubernetes when nodes fail, get drained, or get upgraded, with the Helm chart or with the `nullplatform/agent` OpenTofu module. :::info Everything on this page works with version 3.6.0 or later of the `nullplatform-agent` Helm chart. With the OpenTofu module, set `nullplatform_agent_helm_version` to 3.6.0 or later. ::: ## Run several agents When your only agent restarts, gets evicted, or loses its node, nothing runs your actions until it comes back. Two or more agents close that gap. You don't need a load balancer or any coordination between agents. Every replica registers with the same tags, so your [notification channel](agent-channel.md) selector matches all of them. Nullplatform then distributes commands among the agents that are available. Set the number of replicas: **replicas-helm** ```yaml # values.yaml replicaCount: 2 ``` **replicas-tofu** ```hcl module "agent" { source = "git::https://github.com/nullplatform/tofu-modules.git//nullplatform/agent?ref=" # ...your existing inputs replicas = 2 } ``` ### Check that every replica registered Each replica shows up as its own agent, with its own ID and heartbeat. List the active agents that carry your tags: **replicas-verify-cli** ```bash np agent list \ --organization-id 1 \ --status active \ --tags environment:prod,team:platform ``` **replicas-verify-curl** ```bash curl -L 'https://api.nullplatform.com/controlplane/agent?organization_id=1&status=active&tags=environment:prod,team:platform' \ -H 'Accept: application/json' \ -H 'Authorization: Bearer ' ``` With two replicas, you get two results with the same tags (response trimmed): ```json { "paging": { "limit": 30, "offset": 0 }, "results": [ { "id": "f47ac10b-58cc-4372-a567-0e02b2c3d479", "status": "active", "runtime": "k8s", "tags": { "environment": "prod", "team": "platform" }, "last_heartbeat": "2026-11-11T17:36:42.585Z" }, { "id": "9b2e4c71-3d5a-4f8e-b1c6-7a0d2e9f4b13", "status": "active", "runtime": "k8s", "tags": { "environment": "prod", "team": "platform" }, "last_heartbeat": "2026-11-11T17:37:10.840Z" } ] } ``` ## Spread agents across nodes Replicas only protect you if they don't fail together. By default, Kubernetes can schedule every agent on the same node. If that node goes down, all your agents go down with it. Topology spread constraints tell the scheduler to distribute the agents across nodes or zones. ### Default spread When you turn it on with no other settings, the agents spread across nodes. It's a soft rule: if the cluster can't satisfy it, Kubernetes still schedules the agents instead of leaving them `Pending`. **spread-default-helm** ```yaml # values.yaml replicaCount: 2 topologySpreadConstraints: enabled: true ``` **spread-default-tofu** ```hcl module "agent" { # ... replicas = 2 topology_spread_constraints_enabled = true } ``` That renders this constraint on the agent pod: ```yaml topologySpreadConstraints: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway labelSelector: matchLabels: app.kubernetes.io/name: nullplatform-agent app.kubernetes.io/instance: ``` To check where the agents landed, list their pods with the node each one runs on: ```bash kubectl get pods -n nullplatform-tools -o wide \ -l app.kubernetes.io/name=nullplatform-agent ``` ### Custom constraints To spread by another topology, such as availability zones, pass your own list in `constraints` (`topology_spread_constraints` in the OpenTofu module). Your list replaces the default. You can leave out the `labelSelector`. The chart fills it in with the agent's labels, so the constraint only counts pods from this agent release. If you set your own `labelSelector`, the chart keeps it as is. **spread-custom-helm** ```yaml # values.yaml replicaCount: 3 topologySpreadConstraints: enabled: true constraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway ``` **spread-custom-tofu** ```hcl module "agent" { # ... replicas = 3 topology_spread_constraints_enabled = true topology_spread_constraints = [ { maxSkew = 1, topologyKey = "topology.kubernetes.io/zone", whenUnsatisfiable = "DoNotSchedule" }, { maxSkew = 1, topologyKey = "kubernetes.io/hostname", whenUnsatisfiable = "ScheduleAnyway" }, ] } ``` :::note `whenUnsatisfiable: DoNotSchedule` is a hard rule. If your cluster doesn't have enough nodes or zones to satisfy it, the extra agents stay `Pending`. Use `ScheduleAnyway` unless you're sure the capacity is there. ::: ### Requirements Topology spread needs more than one agent. With a single replica, the install fails instead of ignoring the setting: the Helm chart refuses to render, and the OpenTofu module fails at `plan`. If you enable the chart's autoscaling, the chart checks `autoscaling.maxReplicas` instead of `replicaCount`. The OpenTofu module also fails at `plan` when you set `topology_spread_constraints` but leave `topology_spread_constraints_enabled` off, since nothing would apply them. It checks each constraint too: `whenUnsatisfiable` must be `DoNotSchedule` or `ScheduleAnyway`, and `maxSkew` a whole number of 1 or more. ## Keep agents up during node drains Spreading protects you from a node that fails. Node drains are different: cluster upgrades, maintenance, and autoscaler scale-downs evict pods on purpose, and nothing stops them from evicting every agent at once. A PodDisruptionBudget limits how many agents Kubernetes can evict at the same time. **pdb-helm** ```yaml # values.yaml replicaCount: 2 podDisruptionBudget: enabled: true maxUnavailable: 1 ``` **pdb-tofu** ```hcl module "agent" { # ... replicas = 2 pod_disruption_budget = { enabled = true, maxUnavailable = 1 } } ``` Set either `minAvailable` or `maxUnavailable`, not both. If you set neither, the budget uses `maxUnavailable: 1`. The chart refuses a budget that would never allow an eviction, because it would hang every node drain and cluster upgrade on the agent's node. That means `maxUnavailable: 0` fails, and so does a `minAvailable` that isn't lower than the replica count (`autoscaling.minReplicas` when autoscaling is on). :::tip Node autoscalers are already covered. The chart adds two annotations to the agent pod by default so autoscalers don't evict it to consolidate or scale down its node: `karpenter.sh/do-not-disrupt` for Karpenter and `cluster-autoscaler.kubernetes.io/safe-to-evict` for Cluster Autoscaler. With Karpenter, the agent's node is then never consolidated or expired, so give the NodePool a `terminationGracePeriod` if its nodes still need to rotate. ::: ## Let running commands finish on shutdown Even with a budget, each agent eventually stops: during a rollout, a drain, or a restart. When Kubernetes sends it `SIGTERM`, the agent waits for the commands it's running to finish and report their result. Then it disconnects. Nullplatform re-delivers any command that's still running after the wait to the next available agent. Two settings control this: - **Shutdown drain** (`shutdownDrainTimeoutSeconds`, default `90`): how long the agent waits for running commands. Size it to your longest action. For reference, a `start-blue-green` takes about 70 seconds. Set it to `0` to skip the wait. - **Grace period** (`terminationGracePeriodSeconds`, default `120`): how long Kubernetes gives the agent before it kills it. It has to be at least 10 seconds above the drain, so the agent has time to disconnect. The chart refuses a value that doesn't leave that margin. The defaults fit most setups. If your actions take longer, raise both: **drain-helm** ```yaml # values.yaml shutdownDrainTimeoutSeconds: 150 terminationGracePeriodSeconds: 180 ``` **drain-tofu** ```hcl module "agent" { # ... shutdown_drain_timeout_seconds = 150 termination_grace_period_seconds = 180 } ``` :::note The shutdown drain needs agent 0.14.0 or later. The chart uses the latest agent image by default, so this only matters if you pin the image tag. ::: ## Put it all together A production baseline with two agents spread across nodes, a disruption budget, and the default shutdown drain: **baseline-helm** ```yaml # values.yaml replicaCount: 2 topologySpreadConstraints: enabled: true podDisruptionBudget: enabled: true maxUnavailable: 1 ``` **baseline-tofu** ```hcl module "agent" { source = "git::https://github.com/nullplatform/tofu-modules.git//nullplatform/agent?ref=" # ...your existing inputs nullplatform_agent_helm_version = "3.6.0" replicas = 2 topology_spread_constraints_enabled = true pod_disruption_budget = { enabled = true, maxUnavailable = 1 } } ``` ## Next steps - [Install with Helm](install-helm.md): install the agent on a Kubernetes cluster - [Create a notification channel](agent-channel.md): connect the agent to nullplatform events - [Agent security model](security.md): how the agent handles transport, execution scoping, and credentials