Allow Jobs to Use Spot with PodMutation

Starting with agent v1.21.0, the webhook evaluates Pod-level workload policies after applying PodMutation, including for Job Pods. A Pod annotation of karpenter.sh/do-not-disrupt: "true" causes CloudPilot to inject required node affinity that excludes Spot nodes. A Spot-only NodePool cannot satisfy that constraint.

If a selected Job can tolerate voluntary disruption, use PodMutation to overwrite this annotation with "false" before the webhook chooses its affinity. This also removes the Pod’s do-not-disrupt protection. It does not preserve that protection while allowing Spot.

Apply the annotation override

Save this rule as allow-spot-job.yaml. It matches only the report-job Job in the batch-demo namespace. names contains the Job name, not a generated Pod name.

apiVersion: agent.cloudpilot.ai/v1alpha1
kind: PodMutation
metadata:
  name: allow-spot-report-job
spec:
  enable: true
  priority: 100
  workloadSelector:
    namespaces:
      - batch-demo
    kinds:
      - Job
    names:
      - report-job
  mutationAdd:
    annotations:
      karpenter.sh/do-not-disrupt: "false"

PodMutation does not support deleting annotation keys through mutationRemove. The override replaces the value instead. If another matching rule writes the same annotation, give this rule a higher priority so it runs last.

Create a Job and inspect its Pod

Save this manifest as report-job.yaml. The Job template deliberately retains "true" so you can compare the template with the Pod created by admission.

apiVersion: batch/v1
kind: Job
metadata:
  name: report-job
  namespace: batch-demo
spec:
  backoffLimit: 0
  template:
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "true"
    spec:
      restartPolicy: Never
      containers:
        - name: example
          image: busybox:1.36
          command: ["sh", "-c", "echo Ready for inspection; sleep 3600"]
          resources:
            requests:
              cpu: 5m
              memory: 8Mi
            limits:
              cpu: 50m
              memory: 32Mi

With a healthy CloudPilot webhook and an enabled NodePool, apply the rule before creating the Job:

kubectl create namespace batch-demo
kubectl apply -f allow-spot-job.yaml
sleep 35
kubectl apply -f report-job.yaml
 
kubectl get job report-job -n batch-demo -o yaml
kubectl get pods -n batch-demo -l job-name=report-job -o yaml

For an otherwise unrestricted Job, the expected difference is:

On AWS, the webhook also adds the corresponding eks.amazonaws.com/capacityType requirement or preference. Other scheduling constraints still apply. The override cannot bypass a separate workload.cloudpilot.ai/spot-friendly: "false" label or workload.cloudpilot.ai/rebalanceable: "false" annotation.

Spot preference is a soft constraint: the Pod can still run on an available OnDemand node. Inspect the annotation and affinity to verify the mutation; the node’s capacity type alone does not prove whether it applied.

The Job template remains unchanged. Running Pods keep their existing affinity; changing an annotation does not remove it in place. Eligible unscheduled Pods that have been Pending for more than five minutes may be recreated by the PodMutation fallback so their replacements receive the rule.

This example has no TTL cleanup, so its Job and Pod remain available for inspection after the container exits. Removing the PodMutation later only changes future admission; it does not restore protection on existing Pods.

For the complete field reference, see PodMutation API Reference.