OOM Auto-Remediation

Overview

CloudPilot AI Workload Autoscaler automatically detects, classifies, and remediates Out-of-Memory (OOM) events. When an eligible, auto-remediable OOM is processed after the initial onboarding data gate is satisfied or disabled, the system first creates a short-term active boost, then carries the retained OOM history into the normal recommendation floor. This prevents an expired boost from immediately returning the workload below a recently failed resource level.

The OOM handler does not directly restart, evict, or resize Pods. Instead, it records boosted resource values into the AutoscalingPolicyConfiguration (APC) status. The recommender preserves the retained floor, while the updater, Pod admission webhook, and OOM recovery controller apply resources through their normal flows.

Controller Responsibilities

OOM remediation is split across several components:

How OOM Events Are Detected

The system uses two detection rules based on Kubernetes Pod Status:

1. Cgroup OOM Kill

When a container’s total memory usage exceeds its cgroup limit, the Linux kernel’s OOM killer terminates the main process. Kubernetes reports this as:

lastState.terminated.reason: OOMKilled
lastState.terminated.exitCode: 137

This covers all container types (Java, Go, Python, Node.js, etc.) and is the most common OOM scenario in Kubernetes.

2. JVM Exit on OutOfMemoryError

When a Java container is configured with -XX:+ExitOnOutOfMemoryError and the JVM’s internal heap is exhausted, the JVM exits with code 3:

lastState.terminated.reason: Error
lastState.terminated.exitCode: 3

This catches Java heap OOM even when the container’s cgroup limit is not exceeded (i.e., the JVM heap is the bottleneck, not the container total memory).

Note: For the exit-code-3 detection to work, the JVM must have -XX:+ExitOnOutOfMemoryError configured. The Workload Autoscaler automatically injects this flag for Java containers it directly manages. For containers using the env-var integration path, you should add this flag to your startup scripts. See Java Workload Optimization for details.

Detection and classification timing

The controller waits until an OOM termination is at least 90 seconds old before it processes and classifies the event. This delay is intentional and unchanged: it gives cloudpilot-node-agent time to persist the OOM details and Prometheus time to scrape them. The event is processed on the first 15-second OOM-handler scan after that minimum delay, so the APC status normally changes shortly after the 90-second point rather than immediately when the container exits.

How OOM Events Are Classified (Java)

CgroupOOMKill is the Kubernetes-level oomType recorded when reason=OOMKilled; it is not a Java OOM classification. Only Java containers receive the javaOOMType classifications below. A non-Java cgroup OOM has no javaOOMType and receives memory-only remediation.

For Java containers, the system queries the cloudpilot-node-agent metrics in Prometheus to determine the specific OOM sub-type:

When classification data is unavailable (for example, node-agent is not deployed or the data is not available in Prometheus), the system treats the Java event as a potential heap OOM. It still raises JVM heap only when both CLOUDPILOT_WORKLOAD_AUTOSCALER_JVM_XMS and CLOUDPILOT_WORKLOAD_AUTOSCALER_JVM_XMX environment values can be read and parsed from the failed Pod; otherwise, remediation is intentionally memory-only.

What Happens After Detection

1. One-Shot Boost Computation

The system computes each event from the resources that were actually applied to the failed Pod:

• Memory: the maximum of memory request at OOM × 1.5, memory request at OOM + 200Mi, and the current adjusted recommendation
• Heap (Java heap OOM only): for each of the failed Pod’s parsed CLOUDPILOT_WORKLOAD_AUTOSCALER_JVM_XMS and CLOUDPILOT_WORKLOAD_AUTOSCALER_JVM_XMX environment values, the maximum of parsed value × 1.5 and the corresponding current heap recommendation

The shared active boost keeps the highest result. It is not multiplied from the previous shared boost, so multiple replicas that OOM while still running the same old heap do not compound the heap target merely because their events are processed in different controller scans. A later OOM can raise the target when the failed Pod was actually running a higher applied value.

Both named environment values must be present and valid for a heap boost. The handler does not use arbitrary -Xms or -Xmx flags from the Pod command, arguments, or other environment values as the OOM baseline. If either named value is missing or invalid, the controller applies memory-only remediation. For a valid heap boost, memory is raised to at least Xmx. A KeepLimit memory limit remains a hard ceiling for both memory and heap.

2. APC Status Update

The OOM event and computed boost are recorded in the APC status:

• OOMRecords: history of OOM events (capped at 10, FIFO), which also supplies the retained recommendation floor
• ActiveOOMBoosts: per-container memory/heap floor, initially with a 36-hour expiry, only for an eligible auto-remediable event after the initial onboarding gate is satisfied or disabled
• OOMRemediation condition: records the latest handled event outcome (OOMBoostApplied or Skipped); it is not cleared merely because a boost expires
• Controller-owned floor metadata: stored in the existing evpa.cloudpilot.ai/oom-floor-state APC annotation; the handler records trusted heap provenance and the recommender writes the exact recommendation acknowledgement, without a CRD change

A native_thread OOM or an OOM processed while the initial onboarding data gate is pending is recorded for visibility but does not create an active boost; the latest condition outcome is Skipped.

The annotation is internal controller state and should not be edited manually.

3. Resource Application

The boosted values are applied through existing mechanisms — the OOM handler itself does not mutate Pods:

• Updater: drift detection sees the Pod’s resources are below the boosted floor → triggers InPlace resize or recreate
• Webhook: when a new Pod is admitted, uses the safe current recommendation when available or status.originalRequests otherwise, then overlays the active boost on that baseline
• OOM Recovery (safety net): if the updater cannot act (e.g., OnCreate mode, recommendation not ready), the recovery controller evicts the Pod so a new one is created with boosted resources

OOM Recovery only acts when the APC still has an active OOM boost, the update mode is OnCreate, ReCreate, or InPlace, proactive updates are not disabled, and the target Pod is not preempted, deleting, gone for scheduling, or still inside the Startup Boost window. For single-replica Deployments without PVCs, recovery can trigger a workload rollout instead of directly evicting the Pod to reduce disruption.

OnCreate behavior: Ordinary recommendation changes do not cause the recommender or admission webhook to restart an existing Pod. OOM Recovery is a separate safety path: while an OOM boost is active, it may evict an OOM Pod or trigger an eligible Deployment rollout so that the replacement receives the boost. An OnCreate workload can therefore have Pods replaced after OOM even though normal recommendation updates remain non-disruptive.

4. Retained Recommendation Floor

For a workload that manages memory, the recommender derives a floor from the OOM records still retained in APC status:

• Memory floor: the highest positive memoryRequestAtOOM for that current workload container
• Java heap floor: the highest recorded boosted Xms/Xmx from records whose one-shot heap provenance is trusted by the controller
• Legacy Java records: when existing records predate heap provenance metadata, only the earliest valid heap remediation per container is used as the migration seed; all valid records can still contribute to the memory floor

The floor can change adjusted memory in adjustedRecommendation and trusted Java heap values in jvmRecommendations; rawRecommendation continues to show the metrics-derived result. An OOM floor never lowers a recommendation: when the normal adjusted recommendation is already higher, that higher value is preserved. The recommendation pipeline applies the normal minimum first, then the retained OOM floor, and finally reapplies RequestMax and KeepLimit. As a result, explicit policy ceilings continue to win even when they are lower than an OOM-derived floor.

A skipped record, such as a native_thread event or an event captured while the initial onboarding gate is pending, can still contribute its failed memory request to the retained memory floor; it does not create an active boost or an immediate Pod action. A gate-pending floor is applied only after the gate is satisfied. Java heap floors still require trusted one-shot heap remediation.

The floor lasts as long as its source record remains in the 10-record FIFO. When an old maximum record ages out, the floor is recalculated from the remaining records. An omitted updateResources value follows the API default of [cpu, memory], so memory remains managed. An explicitly empty list means no resources are managed, and updateResources: [cpu] manages CPU only; in both cases OOM history does not add a retained memory floor. Disabling runtime optimization prevents a Java heap floor while preserving an eligible memory floor.

5. Safe Boost Expiry and Handoff

An active boost initially lasts 36 hours. Starting five minutes before it expires, the controller verifies that the current, policy-matched, complete recommendation status contains the exact retained floor and that this revision has been acknowledged.

If that proof is not yet available, the controller extends only the matching existing boost with a renewable 15-minute handoff lease. The lease does not restart the full 36-hour window, and retained history alone never creates a new active boost. Once the exact safe recommendation is acknowledged, the active boost expires naturally; adjusted memory and any eligible JVM heap recommendation continue to enforce the retained floor, subject to RequestMax and KeepLimit.

When an Active Boost or Pod Action Does NOT Trigger

• Normal application crashes (exit code 1 without OOM): not treated as OOM
• Kubernetes eviction (reason=Evicted): node memory pressure, not container OOM
• Liveness probe failure (exit code 137 but reason≠OOMKilled): not OOM
• Java native-thread OOM (native_thread): recorded for visibility, but no automatic resource boost is applied because adding memory or heap does not fix thread exhaustion
• Initial onboarding data gate pending: the event can be recorded with a Skipped outcome, but no active boost is created
• UpdateMode=Off: automatic update and recovery actions are skipped; OOM status may still be recorded for visibility

Status fields to check

Use the APC status to verify what the system decided:

Determine whether remediation is currently active from a non-expired activeOOMBoosts entry, not from the OOMRemediation condition alone. If an OOM termination is less than 90 seconds old, wait for the minimum delay. After 90 seconds, allow for the next 15-second handler scan; if no record appears, check node-agent and Prometheus health. If activeOOMBoosts exists but a Pod is not updated immediately, check the Pod’s update mode, Startup Boost annotation/window, proactive-update disable annotation, and whether the Pod is already deleting or preempted.