Batch workload optimization

Batch workload optimization addresses three patterns that leave batch nodes only partly used:

• Mixed long and short jobs: short jobs finish first, leaving long jobs scattered across mostly idle nodes.
• Large bursts of short jobs: scaling immediately creates many nodes even when the jobs could accept some waiting and reuse slots as earlier jobs finish.
• Small arrivals after a burst: spreading new jobs across the fleet keeps surplus nodes busy and delays their release.

CloudPilot provides two controls to address these patterns:

• Provisioning delay lets eligible jobs wait for running work to free slots before adding capacity. Jobs can use available slots immediately, and demand below the configured active minimum can still create the missing nodes without waiting.
• Minimum active nodes gives new jobs a soft preference for a smaller Active set after the pool has been stable for 10 minutes. Surplus nodes finish their existing jobs and can become empty. They remain available when Active nodes are full; running jobs are not moved, and idle nodes are not reserved.

The aim is to use allocated capacity more fully. Extra waiting can extend job completion when capacity remains busy. Each control is optional; leaving both fields empty disables Batch workload optimization.

Explore the principles, timelines, and illustrative utilization examples in the interactive Batch workload optimization tool .

Availability and scope

Batch workload optimization is available for AWS, GKE, and AKS NodePools with CloudPilot Agent v1.21.0 or later. The per-Pod delay and serial growth behavior described below also require an optimizer build containing these provisioning improvements; the Agent version alone does not enable them. Use current Agent and optimizer releases together. It is not available for Alibaba Cloud NodePools.

Use this feature for NodePools dedicated to run-to-completion workloads. The provisioning delay and serial scale-out apply only when all three conditions are met:

1. The Pod has restartPolicy: Never or restartPolicy: OnFailure.
2. The Pod explicitly selects exactly one NodePool through nodeSelector["karpenter.sh/nodepool"] or required node affinity that unambiguously selects one NodePool across all satisfiable alternatives.
3. That NodePool has a valid, positive provisioningDelay.

An example eligible Pod specification is:

spec:
  restartPolicy: Never
  nodeSelector:
    karpenter.sh/nodepool: batch-workers
  containers:
    - name: task
      image: busybox:1.37
      command: ["sh", "-c", "sleep 30"]
      resources:
        requests:
          cpu: "1"
          memory: 128Mi

Eligible workloads include Job and CronJob Pods, one-time bare Pods, and run-to-completion Pods from systems such as Argo Workflows or Spark operators. The controller type alone does not determine eligibility.

Pods with restartPolicy: Always, Pods without a unique explicit NodePool binding, and Pods targeting a pool with no positive delay continue to use normal provisioning. Replacement of deleting nodes and disruption simulations bypass the Job provisioning gate. Minimum-active placement preferences are a separate pool-level Agent behavior; they do not make ordinary Pods eligible for delayed or serial provisioning.

Configure a NodePool

In the CloudPilot console:

1. Open Node Autoscaler and select Node Pools.
2. Add a NodePool or edit an existing NodePool.
3. Open Batch workload optimization.
4. Configure either or both fields, then save the NodePool.

The equivalent NodePool wrapper fields are:

jobNodePoolOptimization:
  minActiveNodes: 2
  provisioningDelay: 30s

jobNodePoolOptimization is a CloudPilot NodePool setting alongside nodePoolSpec; it is not part of the native Karpenter NodePoolSpec.

Minimum active nodes

minActiveNodes must be a positive whole number. It controls a node count, not a reservation of CPU or memory. It does not proactively create or keep idle nodes, and it does not prevent Karpenter from deleting empty nodes.

The setting has two related uses:

• Agent placement: after the pool has been stable for 10 minutes, the Agent periodically ranks eligible nodes, keeps the first minActiveNodes active, and soft-drains the rest.
• Optimizer provisioning: with a positive provisioning delay, when eligible unschedulable batch Pods need capacity and the pool is below its active minimum, the optimizer can create nodes immediately, up to the missing count. Excess demand waits for the delay and growth rules below.

For example, a pool with a minimum of two and no nodes can create up to two NodeClaims for pending eligible work immediately. A burst that would need six nodes does not immediately create all six. With no pending work, the same setting creates no nodes.

The Agent’s soft-drain taint is:

node.cloudpilot.ai/job-nodepool-draining=draining:PreferNoSchedule

It changes placement preference for new Pods without evicting running Pods, cordoning nodes, deleting NodeClaims directly, or overriding Karpenter disruption budgets. Existing work continues until it completes. When a soft-draining node becomes empty, the NodePool’s normal Karpenter consolidation settings determine when it is deleted.

For safe natural drain behavior:

• Keep the NodePool dedicated to run-to-completion Pods and DaemonSets.
• Add karpenter.sh/do-not-disrupt: "true" to active batch Pods that must finish before normal Karpenter voluntary disruption. This is an operational protection, not a prerequisite for Agent soft-draining in v1.21.1 and later.
• Avoid broad tolerations that tolerate node.cloudpilot.ai/job-nodepool-draining; they weaken Kubernetes placement preference.

In Agent v1.21.1 and later, the controller associates nodes with NodePools through karpenter.sh/nodepool. It does not require node.cloudpilot.ai/managed=true or inspect Pod annotations before applying the placement preference. Nodes with the node-level annotation karpenter.sh/do-not-disrupt: "true" are excluded from soft-draining.

Provisioning delay and serial growth

provisioningDelay must be a positive, finite Go duration such as 30s, 1.5m, or 2m. When the active minimum is met, each eligible Pod must reach its own unschedulable delay before it can drive new provisioning. An older Pod does not unlock younger Pods in the same pool.

The optimizer follows this sequence:

1. Reuse suitable existing active capacity where possible.
2. Below the minimum, permit immediate new NodeClaims only up to the missing count. Count both eligible active nodes and in-flight capacity, without double-counting a registered NodeClaim and its node.
3. At or above the minimum, admit only eligible Pods whose own unschedulable age has reached the delay.
4. If a previously created Job-budget NodeClaim has not initialized, defer further eligible Job growth.
5. Otherwise, allow at most one new Job-budget NodeClaim for the pool in that provisioning round.
6. Once that node initializes, re-evaluate pending Pods and reuse its capacity before allowing another growth NodeClaim.

The delay is measured from each Pod’s PodScheduled=False, Reason=Unschedulable transition time. If that transition is unavailable, the optimizer falls back to the Pod’s creation time. An older Pod does not start or complete the timer for younger Pods. Node initialization releases the growth budget; it does not restart another full delay for Pods that have already waited long enough. The optimizer re-evaluates remaining demand on subsequent rounds.

A NodeClaim created for ordinary work or a replacement can contribute to the minimum, but does not consume the eligible Job growth budget. These claims retain their exemption across provisioning rounds. Below-minimum catch-up may create more than one node; the one-at-a-time growth rule applies after the minimum is reached.

The optimizer excludes cordoned and deleting nodes, and nodes with the Job-draining taint set to NoSchedule, both from the active minimum and from capacity reuse for eligible Job Pods, even if the Pods have broad tolerations. The Agent’s natural-drain taint uses PreferNoSchedule, which is a soft preference: these nodes remain available for capacity reuse and Kubernetes scheduling. Provisioning delay gates optimizer scale-out; it does not hold Pods that can already schedule on existing capacity.

Examples of the provisioning rules

All values below are illustrative.

Why utilization improves

Batch workload optimization improves the opportunity to reuse and concentrate capacity in three ways:

• Short tasks can reuse a node while another node is starting. Serial growth gives newly available slots time to serve more of the queue before the next capacity decision.
• Later arrivals can reuse free slots on an earlier node. Avoiding surplus nodes for short work can let later jobs reuse the capacity already available, rather than extending a scattered tail. This depends on placement and arrival timing; CloudPilot does not predict task duration or guarantee that long tasks share a node.
• An active-set preference can let surplus nodes finish and empty. After Agent stabilization, subsequent small batches prefer the active set instead of repeatedly keeping every node busy. Existing tasks remain where they are until completion.

These are resource-utilization benefits, with a latency trade-off. Provisioning every required node immediately can finish a large burst sooner. Compare the work completed and completion time alongside allocated node-time.

Three illustrative workload comparisons

The following scenarios use synthetic tasks and a simplified model. They are examples of the mechanism, not production measurements, CPU benchmarks, or guaranteed savings. The interactive Batch workload optimization tool  shows their node occupancy and event timelines.

Every task requests one of four identical slots per node. Both strategies receive the same work and use the same 30-second node startup time. Within each comparison, empty-node deletion uses the same idle-tail assumption: 30 seconds for the short-task stream, and 60 seconds for the mixed-duration and natural-drain examples. All empty nodes can retire, including the active node: the minimum is not an idle-capacity reservation. Each comparison uses one common observation window long enough to include both strategies’ completed work and modeled node deletion.

Occupied-slot utilization is useful task-slot-seconds divided by allocated node-seconds times four slots per node. Startup and idle time count in the denominator. This metric describes modeled task occupancy, not observed CPU or memory use. Real instance sizes, packing, startup, consolidation settings, and disruption budgets change the result.

1. A stream of short tasks

Workload: at time zero there are no nodes or tasks. From time 1 through 30 seconds, 64 tasks arrive at a rate of two to three tasks per second. Each task runs for 10 seconds. Batch workload optimization uses minimum one and provisioning delay 30 seconds.

Immediate provisioning responds to the growing queue by allocating up to 16 nodes. Batch workload optimization creates its first node at 1 second to fill the minimum deficit. That node initializes at 31 seconds. Its short tasks repeatedly finish and release slots while later growth nodes are still starting, so subsequent tasks can reuse ready capacity.

Both strategies perform 64 × 10 = 640 task-slot-seconds of useful work over a common 152-second window.

The optimized run’s fourth claim initializes at 122 seconds, just after the last task finishes, so it serves no tasks. The model still charges its startup and full idle tail through 152 seconds. It does not assume perfect capacity foresight.

In this modeled run, the optimized queue uses 64.3% fewer allocated node-seconds, but finishes 62 seconds later. This demonstrates the reuse opportunity when tasks are short relative to node startup, together with the waiting cost of growing gradually.

2. Short and long tasks arrive together over time

Workload: at time zero there are no nodes or tasks. Sixty-four ten-second tasks arrive gradually from time 1 through 158 seconds, approximately one every two to three seconds. Four 300-second tasks arrive at 60, 80, 100, and 120 seconds, while the short-task stream continues. Batch workload optimization uses minimum one and delay 30 seconds; both strategies use 30-second startup and a 60-second empty-node idle tail.

Both strategies place tasks on the least-occupied ready node. Occupancy ties prefer the node that least recently received a task, then use a stable node identifier. Tasks are considered in arrival order with identical times and requests for both strategies; neither uses task duration to choose placement.

Immediate provisioning creates four nodes as the initial queue accumulates. Each long task lands on a different node while short tasks continue arriving and completing. After the short work finishes, the four long tasks leave one occupied slot on each of four nodes.

Batch workload optimization starts with one node and reuses slots released by short tasks. Continuing demand eventually adds a second node and then a temporary third node for short work. The first two long tasks start on the first node at 91 and 111 seconds; the next two start on the second node at 130 and 150 seconds. Short tasks continue arriving after all four long tasks have arrived, and the long tasks share capacity with ongoing short work. The long work spans two nodes in this particular run.

Both strategies perform 64 × 10 + 4 × 300 = 1,840 task-slot-seconds of useful work over a common 510-second window.

In this modeled run, allocated node-time is 42.4% lower, with a 30-second increase in final completion time. Short work releases slots for later arrivals; after the temporary node drains, the remaining long work occupies two nodes instead of four. These placements arise from available capacity and shared duration-unaware rules. Different arrival timing, placement, or consolidation behavior can change the result; concentrating long-running work is not guaranteed.

3. Natural drain during recurring small batches

Workload: six existing nodes have been stable for at least 10 minutes. Each node has one protected task with 30 seconds of work remaining. Twenty-four further tasks arrive one every 10 seconds from time 20 through 250 seconds, each running for 20 seconds. With Batch workload optimization enabled, the Agent keeps one node active and applies the soft-drain preference to the other five.

The immediate comparison continues placing small batches across available nodes under the same least-occupied placement policy. When occupancy is tied, both strategies prefer the node that least recently received a task, then use a stable node identifier. The optimized comparison first prefers the active node. Existing protected tasks on surplus nodes are not migrated or evicted; they complete where they started. Without more work placed there, those nodes become empty and can consolidate under the same idle-tail rule.

The five optimized surplus nodes become empty at 30 seconds and retire at 90 seconds after the 60-second idle tail. Without the active-set preference, recurring tasks cycle across all six nodes; the first surplus node retires at 290 seconds and the last at 330 seconds. The optimized active node also retires at 330 seconds after all work completes; it is not reserved by the minimum.

Both strategies perform 6 × 30 + 24 × 20 = 660 task-slot-seconds of useful work over a common 330-second window.

In this modeled run, natural drain uses 57.4% fewer allocated node-seconds while keeping the same completion time and zero new-task waiting.

The Agent’s natural-drain behavior is independent of the Pod provisioning gate. This scenario needs no extra nodes and illustrates placement and consolidation rather than per-Pod waiting or serial growth. PreferNoSchedule remains a preference: when the active set is full, compatible new work can still use a soft-draining node. A minimum of one does not reserve that node once all work is finished.

Use the tool’s timelines to distinguish tasks waiting for capacity, nodes starting, occupied slots, and nodes in their idle tail. For each scenario, compare completed work, allocated node-seconds, occupied-slot utilization, and final completion time together.

Choose values and verify the result

Start conservatively and measure both scheduling latency and node utilization.

Use a shorter delay or leave it empty for workloads with strict start-time objectives. A finite duration has no product-level maximum, so avoid values that could leave work waiting longer than the business process allows.

Measure comparable workload runs over the same observation window. Record completed work, Pending and completion latency, node/NodeClaim creation and initialization times, and allocated resource-seconds. CPU and memory utilization also depend on requests, workload behavior, and instance size; the occupied-slot example above is not a CPU utilization metric.

Operational considerations

• The Agent’s fixed 10-minute stabilization window restarts when the NodePool gains a node. This concerns active-set ranking and soft drain; it is separate from the optimizer’s per-Pod delay and immediate below-minimum catch-up.
• The combined controls do not evict running Jobs. Normal Karpenter disruption behavior still depends on Pod protections, consolidation settings, and disruption budgets.
• Actual results depend on arrival patterns, job duration, cloud instance startup time, packing, consolidateAfter, and disruption budgets. No fixed utilization increase or savings percentage is guaranteed.
• Provisioning delay does not remove the cost of listing and validating pending Pods.
• Clearing one field disables that behavior. Clearing both fields removes the CloudPilot optimization annotations and owned soft-drain taints.