OOM Auto-Remediation

Overview

CloudPilot AI Workload Autoscaler detects and classifies Out-of-Memory (OOM) events, then combines each retained event’s independent recovery target with normal recommendations. A record can influence recommendations for at most seven days from the actual occurrence. The ten-record FIFO can remove it earlier. Resource and JVM targets remain subject to the current applicable policy bounds.

The OOM handler records facts and recovery targets; it does not directly resize or restart Pods. Recommendation, admission and update paths use the same effective OOM projection. There is no separate temporary OOM Boost or handoff lease.

Controller Responsibilities

OOM remediation is split across several components:

How OOM Events Are Detected

The system scans both regular and init-container statuses. It checks a termination that is still in state.terminated as well as the most recent termination in lastState.terminated, and uses two detection rules:

1. Cgroup OOM Kill

When a container’s total memory usage exceeds its cgroup limit, the Linux kernel’s OOM killer terminates the main process. Kubernetes reports this in the current or last termination state as:

(state|lastState).terminated.reason: OOMKilled
(state|lastState).terminated.exitCode: 137

This covers all container types (Java, Go, Python, Node.js, etc.) and is the most common OOM scenario in Kubernetes.

2. JVM Exit on OutOfMemoryError

When a Java container is configured with -XX:+ExitOnOutOfMemoryError and the JVM raises an OutOfMemoryError, the JVM exits with code 3:

(state|lastState).terminated.reason: Error
(state|lastState).terminated.exitCode: 3

This catches JVM-level heap, GC-overhead, metaspace, direct-buffer, and native-thread OOMs even when the container’s cgroup limit is not exceeded.

Note: For the exit-code-3 detection to work, the JVM must have -XX:+ExitOnOutOfMemoryError configured. The Workload Autoscaler automatically injects this flag for Java containers it directly manages. For containers using the env-var integration path, you should add this flag to your startup scripts. See Java Workload Optimization for details.

Detection and classification timing

An event-driven Pod-status observer can record a newly visible OOMDetected fact as soon as it can attribute the transition to a managed workload. Java classification and remediation wait until the termination is at least 5 seconds old, giving the container runtime time to finalize the selected current or previous log. The OOM handler also scans workload state about every 15 seconds to recover transitions that the immediate observer did not record.

Pod Status retains only limited termination history. If a Pod is deleted before a scan, or several rapid restarts overwrite both the current and previous termination slots, the controller cannot reconstruct every older occurrence. Captured occurrences are recorded individually, but OOM detection and Audit Log remain best effort.

Each successfully captured occurrence produces an OOMDetected Audit Log fact and, when Java classification applies, a later OOMClassified fact; the Console groups those facts into one OOM occurrence. Init-container OOMs are included for Audit Log visibility, but init containers are not managed by APC recommendations, so they do not create a status.oomRecords entry, recommendation floor, or active boost.

How OOM Events Are Classified (Java)

CgroupOOMKill is the Kubernetes-level oomType recorded when reason=OOMKilled; it is not a Java OOM classification. Only Java containers receive the javaOOMType classifications below. A non-Java cgroup OOM has no javaOOMType and receives memory-only remediation.

For Java containers, the Workload Autoscaler reads the log slot that still owns the detected occurrence: current logs for state.terminated and previous logs for lastState.terminated. It verifies the Pod and exact termination before and after the read, then recognizes these subtypes:

The log read is deliberately bounded. The controller requests at most 200 timestamped lines, rejects a response larger than 256 KiB, retains only the last 10 KiB in memory for parsing, and gives each occurrence-level attempt two seconds. The dedicated Kubernetes client is limited to QPS 2 with burst 12; a classification run uses at most four workers, 32 attempts, and five seconds. If the log is missing, too large, truncated before the relevant message, changed during verification, ambiguous, unsupported, or outside that budget, the subtype is unknown rather than guessed from cumulative metrics.

Raw container logs are never uploaded or persisted by this path. Only the resulting subtype is written to APC status and Audit Log. Retained records created by older releases can still show legacy heap_inferred or non_heap_inferred values; new occurrence-bound classifications do not generate them.

When the subtype is unknown, the system treats a known Java event conservatively. It still raises JVM heap only when both CLOUDPILOT_WORKLOAD_AUTOSCALER_JVM_XMS and CLOUDPILOT_WORKLOAD_AUTOSCALER_JVM_XMX values can be read and parsed from the Pod; otherwise, remediation is intentionally memory-only.

What Happens After Detection

Independent Event Targets

For an eligible event, memory recovery starts from the larger of the failed Pod’s observed memory Request multiplied by 1.5 and that Request plus 200Mi. Eligible Java heap recovery starts from the failed Pod’s valid named Xms/Xmx values multiplied by 1.5. Both named values must be available for a heap contribution. Ordinary recommendations are combined later; the handler does not copy an old shared recommendation into a new event.

These inputs are desired PodSpec snapshots observed during detection, not proof of the exact resources enacted at OOM time. A new event can legitimately produce a larger target when its own Pod snapshot is larger. It cannot renew a different event’s old high contribution.

The existing classification and onboarding rules remain. A skipped occurrence can preserve its observed original memory Request without gaining automatic heap remediation. Heap contributions require trusted event provenance. Native-thread JVM exits do not gain a new resource-remediation algorithm.

Seven-Day Recommendation Influence

The effective target combines the ordinary recommendation with still-retained, unexpired contributions for the same workload container. Each contribution expires 168 hours after occurrence; legacy records lacking occurrence time may use detection time. Restart, upgrade or repeated observation does not restart the clock. Expired or FIFO-evicted records cannot be revived by compatibility fields.

A projection that declines or becomes empty triggers full normal recommendation recalculation, including when the RP evaluation interval has not elapsed. A calculation spanning an expiry must be checked again before publication or use. Missing fresh data does not authorize continued use of an expired elevated target. Expiry itself does not authorize an immediate shrink: normal action gates and drift thresholds still apply.

Current Request Max, Limit Policy, QoS rules and applicable JVM bounds constrain the result. Absolute workload budgets are allocated across containers and rounded within the budget. Request and Limit remain distinct. Explicitly contradictory policy minima can prevent a valid target; a normal feasible target is fitted within its hard ceiling.

Startup Boost remains a separate feature with its existing policy scope and normal completion. Removing the temporary OOM mechanism does not remove Startup Boost.

Normal Updates and Fault Recovery

Ordinary updates follow Schedule and update mode. OnCreate applies targets when a new Pod is created and does not independently authorize replacement after a single OOM.

All update entries share one in-flight operation per workload. The current object must complete its actual resource update, or its bound replacement must become Ready, before another object is selected. Healthy updates require all workload Pods currently Ready and their current business containers running for at least five minutes. This is a running-age check: it does not require five minutes of continuous Ready or another five-minute wait after InPlace.

While the workload is not fully Ready, only Pending, resize-pending and current confirmed OOM fault candidates may proceed. A changed valid target can be retried for the current Pending object. Healthy siblings remain protected. Fault recovery retains the same single-operation ordering; it does not wait for every other faulted replica to recover first.

Frequent-OOM Emergency ReCreate

At least two distinct confirmed OOM occurrences for the same workload identity and container within the rolling preceding hour permit operation-local ReCreate, overriding Schedule and policy update mode, including policy Off. Events may come from different replicas or replacement Pods of that same workload. Duplicate reports, unrelated crashes, restart-count deltas and another workload’s history do not count.

The original Drift Thresholds still apply. Any qualifying Pod in the workload may participate, including a healthy sibling or CPU-only drift, subject to readiness, running age, Startup protection, resource scope, current Max/JVM bounds, applicable PDB and action quota. The effective business-disable switch always stops new mutations, including emergency and replacement-admission mutations.

One hour controls permission to start an emergency operation; seven days controls a record’s recommendation contribution. Neither extends the other. After qualification expires, no new emergency operation starts. A previously issued, identity-bound operation may finish, including replacement admission after expiry, using current enablement and policy constraints. A fixed target that has converged must not cause repeated replacements.

OOM observation is best effort. Rapid terminations can overwrite status before collection, so the controller requires observed evidence instead of inventing missing occurrences.

When No New Pod Action Is Authorized

• The effective workload optimization switch is disabled.
• Schedule or mode blocks the ordinary operation, without current frequent-OOM qualification.
• No original drift threshold is met, or no valid current target is available.
• A different operation for the same workload is unfinished.
• A healthy candidate fails whole-workload readiness or running-age checks.
• Applicable Startup, JVM, Max, PDB or quota checks block the action.

Kubernetes may naturally finish an API action accepted before closure. That is separate from authorizing another mutation after closure. Requesting more memory cannot guarantee recovery from an application defect.

On HPA-aware releases, OOM observation continues, but retained OOM protection and frequent-OOM recovery do not override HPA-protected Memory. Policy ceilings do not replace the protected template baseline. Retained history does not authorize remediation of a protected resource.

Status fields to check

Use the APC status to verify what the system decided:

Check occurrence time and retained history, effective workload enablement, Schedule/mode or emergency qualification, current policy revision, the in-flight object and actual Pod/JVM resources. An outcome condition or operation marker alone does not prove target application. Audit Log retention is separate from the ten-record remediation FIFO; see Audit Log.