Node Status Metrics

The CloudPilot Agent exposes a Prometheus-compatible endpoint that reports the current node status and disruption state shown by CloudPilot AI. You can scrape this endpoint directly with Prometheus or use the Prometheus receiver in an OpenTelemetry Collector.

The endpoint is available inside the cluster at:

http://cloudpilot-agent.cloudpilot.svc.cluster.local/metrics

Metric reference

All metrics are gauges. A value of 1 indicates the current state represented by the metric’s labels.

node_pool is empty for nodes that are not associated with a Karpenter NodePool. Some OpenTelemetry backends omit attributes with an empty value.

Effective node status

cloudpilot_node_status reports one of the following values:

Kubernetes node readiness remains the primary status, matching the CloudPilot Node List. Use cloudpilot_node_disruption_status when you need the independent disruption lifecycle.

Disruption status

cloudpilot_node_disruption_status reports one of the following values:

Migration-blocked observations cover the most recent ten minutes and are not permanent history. Use the CloudPilot Console and Karpenter Events for the full diagnostic message and lifecycle details.

Collection health

cloudpilot_node_status_collection_success uses the following source values:

The nodeclaim and event source metrics are emitted only for supported Karpenter integrations. On other clusters, the Agent emits only source="node" collection health and reports disruption_status="not_applicable".

Verify the endpoint

Port-forward the cluster-local Agent service:

kubectl -n cloudpilot port-forward \
  --address 127.0.0.1 \
  svc/cloudpilot-agent 24909:80

In another terminal, query the endpoint:

curl -fsS http://127.0.0.1:24909/metrics \
  | grep -E '^cloudpilot_node_(status|disruption_status|status_collection_success)'

The port-forward is attached to the current Agent Pod. Re-create it after an Agent rollout or restart.

OpenTelemetry Collector configuration

Use the Prometheus receiver to scrape the Agent service, then export the data to your OpenTelemetry-compatible backend:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: cloudpilot-agent
          scrape_interval: 60s
          static_configs:
            - targets:
                - cloudpilot-agent.cloudpilot.svc.cluster.local:80
 
exporters:
  otlp:
    endpoint: ${env:OTEL_EXPORTER_OTLP_ENDPOINT}
 
service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [otlp]

Configure TLS and authentication on the OTLP exporter as required by your observability backend.

Query examples

Find nodes whose effective status is not Ready:

cloudpilot_node_status{status!="ready"} == 1

Find nodes with an actionable disruption state:

cloudpilot_node_disruption_status{status=~"migration_blocked|protected|terminating"} == 1

Detect an unavailable collection source:

cloudpilot_node_status_collection_success == 0

Freshness and performance

The Agent calculates metrics from its existing informer caches and does not query the Kubernetes API for each scrape. A calculated snapshot is cached for one minute. Scraping more frequently than once per minute does not increase data freshness, so a 60-second scrape interval is recommended.

After a node or NodeClaim change, allow up to approximately one minute for the new state to appear. Results produced before the Node informer has synchronized are not cached.

Security considerations

The Agent service is a Kubernetes ClusterIP service and is not exposed outside the cluster by default. The metrics include cluster IDs, node names, NodePool names, and status labels. Do not expose /metrics publicly. Use Kubernetes network policies when access must be limited to a dedicated monitoring namespace.

For the Kubernetes permissions used to collect Node and NodeClaim state, see Permissions Required to Use CloudPilot AI.