Use PodMutation During a NodePool Merge

This guide explains how to use PodMutation to move workloads from multiple existing NodePools to one new NodePool during a merge.

The goal is straightforward:

Keep the workload manifests unchanged at first
Define the old scheduling rules in workloadSelector
Remove those old rules in mutationRemove
Add the new merged NodePool rules in mutationAdd
Recreate the Pods so the webhook applies the new scheduling constraints

In practice, PodMutation acts as a transition layer for the merge. Instead of editing every workload immediately, you let the admission webhook rewrite Pod scheduling rules when replacement Pods are created.

How the merge works

In this example:

the first old NodePool is selected by example.com/node-pool=pool-a
the second old NodePool is selected by example.com/node-role=pool-b
both old NodePools use the taint key dedicated, but with different values
the merged NodePool is selected by example.com/node-pool=pool-merged

The merge is implemented with two PodMutation objects:

one matches workloads that still point to pool-a
one matches workloads that still point to pool-b

Each mutation removes the old nodeSelector and toleration, then adds the new merged NodePool selector and toleration.

Important behavior

This workflow depends on the current webhook behavior in the agent:

The mutating webhook applies PodMutation only on Pod CREATE.
Existing Pods are not rewritten in place.
DaemonSet-owned Pods are skipped by the webhook.
Inside one PodMutation, mutationRemove is applied before mutationAdd.
The webhook refreshes its PodMutation cache every 30 seconds, so wait briefly after creating or updating a PodMutation.

Prerequisites

The cloudpilot-webhook deployment is healthy.
The PodMutation CRD is installed.
The old and new NodePools already exist.
The merged NodePool is schedulable before you restart workloads.

For this example, the cluster must have nodes that use these labels and taints:

example.com/node-pool=pool-a with dedicated=pool-a:NoSchedule
example.com/node-role=pool-b with dedicated=pool-b:NoSchedule
example.com/node-pool=pool-merged with dedicated=pool-merged:NoSchedule

Procedure

The command examples below assume you save the two YAML snippets from this page as:

merge-nodepools-workloads.yaml
merge-nodepools-podmutation.yaml

Apply workloads that still target the old NodePools.
Confirm that the current Pods are using the old selectors and tolerations.
Create the PodMutation rules for the merge.
Wait for the webhook cache to refresh.
Restart the workloads so that new Pods pass through admission.
Confirm that the replacement Pods now target the merged NodePool.


# 1. Apply the example workloads
kubectl apply -f merge-nodepools-workloads.yaml
 
# 2. Verify the current scheduling rules
kubectl get pods -n merge-demo -o wide
kubectl get pod -n merge-demo -l app=app-pool-a -o yaml | grep -A12 "nodeSelector:"
kubectl get pod -n merge-demo -l app=app-pool-b -o yaml | grep -A12 "nodeSelector:"
 
# 3. Apply the PodMutation rules for the merge
kubectl apply -f merge-nodepools-podmutation.yaml
 
# 4. Wait for the webhook cache refresh
sleep 35
 
# 5. Recreate Pods so the webhook can mutate them
kubectl rollout restart deployment/app-pool-a -n merge-demo
kubectl rollout restart deployment/app-pool-b -n merge-demo
kubectl rollout status deployment/app-pool-a -n merge-demo --timeout=120s
kubectl rollout status deployment/app-pool-b -n merge-demo --timeout=120s
 
# 6. Verify that the workloads now target the merged NodePool
kubectl get pods -n merge-demo -o wide
kubectl get pod -n merge-demo -l app=app-pool-a -o yaml | grep -A12 "nodeSelector:"
kubectl get pod -n merge-demo -l app=app-pool-b -o yaml | grep -A12 "nodeSelector:"

Expected result

Before the merge:

app-pool-a Pods target example.com/node-pool=pool-a
app-pool-b Pods target example.com/node-role=pool-b

After the restart:

both workloads create new Pods with example.com/node-pool=pool-merged
both workloads tolerate dedicated=pool-merged:NoSchedule
unrelated selectors and tolerations remain unchanged

Example workloads


apiVersion: v1
kind: Namespace
metadata:
  name: merge-demo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-pool-a
  namespace: merge-demo
  labels:
    app: app-pool-a
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app-pool-a
  template:
    metadata:
      labels:
        app: app-pool-a
    spec:
      nodeSelector:
        example.com/node-pool: pool-a
      tolerations:
        - key: dedicated
          value: pool-a
          effect: NoSchedule
      containers:
        - name: nginx
          image: nginx:1.25-alpine
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-pool-b
  namespace: merge-demo
  labels:
    app: app-pool-b
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app-pool-b
  template:
    metadata:
      labels:
        app: app-pool-b
    spec:
      nodeSelector:
        example.com/node-role: pool-b
      tolerations:
        - key: dedicated
          value: pool-b
          effect: NoSchedule
      containers:
        - name: nginx
          image: nginx:1.25-alpine
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 200m
              memory: 128Mi

Example PodMutation rules


apiVersion: agent.cloudpilot.ai/v1alpha1
kind: PodMutation
metadata:
  name: merge-pool-a
spec:
  enable: true
  priority: 100
  workloadSelector:
    nodeSelectorMatch:
      example.com/node-pool: pool-a
    tolerationMatch:
      - key: dedicated
        value: pool-a
        effect: NoSchedule
  mutationRemove:
    nodeSelectorKeys:
      - example.com/node-pool
    tolerationKeys:
      - dedicated
  mutationAdd:
    nodeSelector:
      example.com/node-pool: pool-merged
    tolerations:
      - key: dedicated
        value: pool-merged
        effect: NoSchedule
---
apiVersion: agent.cloudpilot.ai/v1alpha1
kind: PodMutation
metadata:
  name: merge-pool-b
spec:
  enable: true
  priority: 100
  workloadSelector:
    nodeSelectorMatch:
      example.com/node-role: pool-b
    tolerationMatch:
      - key: dedicated
        value: pool-b
        effect: NoSchedule
  mutationRemove:
    nodeSelectorKeys:
      - example.com/node-role
    tolerationKeys:
      - dedicated
  mutationAdd:
    nodeSelector:
      example.com/node-pool: pool-merged
    tolerations:
      - key: dedicated
        value: pool-merged
        effect: NoSchedule

Notes

If you apply the PodMutation and restart the workload immediately, the new Pods may still use the old rules because the webhook cache has not refreshed yet.
If you do not recreate the Pods, the merge does not take effect because existing Pods are not mutated in place.
If the old NodePools use different label keys, create one PodMutation per source rule set, as shown above.

Use node affinity during a NodePool merge

Use nodeAffinityMatch, mutationRemove.nodeAffinity, and mutationAdd.nodeAffinity when workloads select their old NodePools with spec.affinity.nodeAffinity instead of spec.nodeSelector.

The example below matches Pods that still require pool-a, removes that old requirement, and adds a requirement for the merged NodePool:


apiVersion: agent.cloudpilot.ai/v1alpha1
kind: PodMutation
metadata:
  name: merge-pool-a-affinity
spec:
  enable: true
  priority: 100
  workloadSelector:
    nodeAffinityMatch:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: example.com/node-pool
                operator: In
                values:
                  - pool-a
  mutationRemove:
    nodeAffinity:
      keys:
        - example.com/node-pool
  mutationAdd:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: example.com/node-pool
                operator: In
                values:
                  - pool-merged

The webhook evaluates nodeAffinityMatch against the original incoming Pod. All matching PodMutation objects are selected before any mutation is applied, so an affinity added by one rule cannot cause another rule to start matching.

Required node selector terms keep their Kubernetes semantics:

terms are ORed;
requirements inside one term are ANDed;
a Pod term may contain additional requirements and still match;
requirement values are compared as a set, so their order does not matter.

Within one PodMutation, removal runs before addition. To replace a requirement that uses the same key, list the key under mutationRemove.nodeAffinity.keys and add the new requirement under mutationAdd.nodeAffinity.

When required affinity is added to an existing required selector, CloudPilot AI combines them with logical AND. This may produce multiple required terms because the webhook distributes both sets of OR terms. Preferred terms are appended and deduplicated.

To remove every customer-provided node affinity constraint, use:


mutationRemove:
  nodeAffinity:
    all: true

all: true removes only spec.affinity.nodeAffinity. It preserves podAffinity and podAntiAffinity. CloudPilot-managed Spot, NonSpot, and Diversity affinity is appended after PodMutation processing, so those system constraints remain on the final Pod.

PodMutation configuration supports matchExpressions but not matchFields. CloudPilot-managed affinity keys are reserved and cannot be matched, added, or removed by key. As with other PodMutation fields, changes affect only newly created Pods; restart or roll out the workload to apply them.

See the PodMutation API Reference for the complete field schema, operator rules, and safety limits.