Observability and Cost
Use this reference to interpret CloudPilot Console and MCP readings without
confusing configuration, observed state, estimates, and historic events. Paths
are under the MCP response data unless explicitly identified as Console-only.
Check the deployed tool catalog before requesting a tool; fields available in
newer implementations are not a guarantee of release to every environment.
Read the envelope first
| Response field | Meaning |
|---|---|
observedAt | Time the MCP response was assembled, not necessarily the collection time of every underlying metric. |
status | Overall read status. partial can mean a stale source, missing series, or bounded/downsampled history; inspect warnings and sources. It is not the cluster’s health status. |
warnings[], sources[].status | Collection/availability qualifications. Preserve these when summarizing a result. |
consoleUrl | Relevant Console view; it does not grant additional access. |
data.complete, data.pageInfo | For ordinary list tools, pagination state: returned, matching, hasMore, and nextCursor. A complete filtered list is not a complete cluster inventory. |
Follow the next cursor with the same scope, filters, sort, detail, and limit when all matches are needed. Do not sum a cluster-wide summary once per page; pagination bounds returned rows, not the meaning of aggregate totals. Start a new traversal after changing the query. A resource inventory can change between pages, so do not describe ordinary list pagination as an atomic snapshot.
Ordinary summary lists default to 50 rows and allow up to 100; tools with
detail: full default to 20 and allow up to 50. A server can apply a lower
configured maximum. Use a list for discovery and the relevant single-resource
tool for detail instead of requesting every full configuration unnecessarily.
Cluster overview and current resources
MCP: get_cluster_overview; resource-only view: get_cluster_resources.
| Overview field group | Important paths | Interpretation |
|---|---|---|
| Identity | cluster.clusterId, name, provider, region, kubernetesVersion, joinedAt, lastActiveTime, demo | Cluster identity and most recently observed heartbeat. Demo data is simulated and unsuitable for a customer savings assessment. |
| Connectivity | cluster.status | Online/offline/demo presentation; not a claim that every node or application is healthy. |
| Feature eligibility | capabilities.<feature>.supported, .reason | Supported can be true, false, or null (unknown). Capability is not installation, enablement, or recent activity. |
| Pod readiness | health.pods.ready, notReady, total | Count of Pods represented by the source snapshot; not an application SLA. |
| Nodes and zones | health.nodes.total, onDemand, spot, other; health.zones[].name, nodes | Counts, with unclassified capacity retained under other. Zone counts can be a different snapshot from a separately fetched node page. |
| Instance distribution | health.instanceTypeFamilyNodes | Optional family-to-count map, not an exhaustive instance-type catalog. |
| Alerts | health.alerts.total, highestSeverity | Active alert summary; a null summary means unavailable, not zero alerts. Fetch alert rows for reasons. |
| Freshness | health.freshnessStatus, resources.freshnessStatus | Whether the underlying current resource snapshot is fresh or qualified. Inspect warnings before concluding recovery. |
| CPU / memory | resources.cpu, resources.memory, resources.managedCores | The resource series expose unit, provisioned, allocatable, requested, and used. CPU is cores, memory is GiB. Managed cores is a separate management quantity. |
| Optimization state | nodeAutoscaler.enabled, state, message, lastActiveTime, progress; workloadAutoscaler.allowed, installed, enabled, version | Policy switches, component allowance, and runtime state are different. Enabled alone does not prove optimization is making progress. progress.targetReady, pods.total/remaining, and sourceNodes.total/remaining are operation progress counts, not readiness for every cluster Pod/node. |
| Versions | version.agentVersion, onboardManifestVersion, needUpgrade | Agent and onboard release identifiers. Do not use either as the Kubernetes version or every component’s image version. |
| Costs | cost.monthlyCost, onDemandMonthlyCost, spotMonthlyCost, potentialMonthlySaving, currency, symbol | Monthly estimates, not an invoiced monthly bill. Optional initial/NA/WA estimate fields retain their separate source meanings. |
| Storage | storage | PVC collection and capacity summary; see below. Null means unavailable. |
Provisioned is total node capacity; Allocatable is capacity available for Pod scheduling after reservations; Requested is declared workload requests; Used is measured consumption. Requested can exceed used without being safely removable: consider peaks, availability, scheduling constraints, and policy minima. A null reading must not be converted to zero.
get_cluster_resources returns these same four resource values under cpu
and memory, without the overview’s surrounding health/cost state.
Resource history and time ranges
MCP: get_cluster_resource_history.
| Input or result | Contract |
|---|---|
period | 1h (default), 1d, 7d, or 30d. Presets cannot be combined with start, end, or step. |
Custom start, end | Unix seconds; positive and ordered, at most 32 days apart. Omitted end defaults to now; omitted start defaults to one hour before end. |
Custom step | Positive duration such as 5m or 1h; effective custom step is at least five minutes and may grow to honor the point limit. Preset steps follow the preset contract instead. |
max_points | Default 300, range 1–1000 per series. A bound on output, not a retention setting. |
range.start, end, step, points, downsampled | Returned effective range, step, point count, and whether output samples were reduced. Use this metadata, not an assumed browser date range. |
cpu / memory | unit, avgAllocatable, avgRequested, avgUsed, and samples[] with timestamp, allocatable, requested, used. CPU cores and GiB; averages are resource quantities, not CPU-hours or GiB-hours. |
Missing source series and an empty range produce warnings. Do not infer zero usage from an empty chart. Returned averages and retained display samples can be based on different aggregation/downsampling steps; recomputing an average from a reduced set of points need not reproduce the server’s aggregate.
Active alerts, nodes, and events
Active alerts
list_cluster_alerts returns items[].id, message, and severity.
Use detail: full for provider and type. Severity is numeric:
1 Low, 2 Medium, 3 High, 4 Critical; larger means more severe.
Filters are providers, types, and severities; sort fields are severity
(default descending), provider, type, or id.
The tool lists active alerts, not a notification delivery log or an alert history. An empty list means no matching active alerts in that successful read, not that there were no past incidents. Notification suppression is separate; see organization configuration.
Nodes
list_cluster_nodes identifies each row using id, name, instanceType,
capacityType, zone, nodeClaim, and nodePool. Read status,
optimizationBlockedReason, readyPods, notReadyPods, and optional
statusLastTransitionTime before interpreting a node as removable.
rebalanced indicates the source’s optimization classification, not current
Kubernetes readiness. price is accompanied by currency and symbol; do not
sum raw values across currencies. With detail: full, inspect message,
ipAddress, tags, labels, usedResources, requestResources,
allocatableResources, provisionedResources, diskInfos, and nodeClaimStatus.
Resource map keys and disk details are source/provider-specific; keep their
units and identities rather than treating every number as GiB.
Filters support instance_types, capacity_types, zones, node_claims,
node_pools, and at most one statuses value per request. Sort by name,
creation_time, price, age, pod_number, or node_pool. Use the exact
source status value returned by the inventory, not a translated Console label.
Controller and NodeClaim events
| Tool | Fields and use |
|---|---|
list_controller_events, get_controller_event | id, type, operationType, provider, reason, nodePoolNames, current/deleted NodeClaim names and IDs, schedulableFailedPods, startTimestamp, endTimestamp, processTime, succeeded, errors. Read full data or the single-event tool for the operation’s evidence. |
list_node_claim_events | id, nodeClaimId, nodeClaimName, nodeName, provider, type, owner, start/end timestamps, processTime, succeeded, errors; full detail adds source data. |
Controller event filters include RFC3339 start_time/end_time, operations,
boolean status, and node_pool_names. NodeClaim filters include types,
boolean statuses, and node_claim_names. succeeded is the outcome of that
event, not a live readiness flag. Correlate IDs/names and timing with the current
node/pool state rather than treating a historic error as a continuing outage.
Event startTimestamp and endTimestamp preserve Unix nanoseconds from the
source; processTime is a duration string. Do not parse these numeric event
times using the seconds contract of history query inputs. Optional
createdAt, updatedAt, and deletedAt are timestamp strings instead.
WA activity is a different stream
The Console’s WA Activity view records actions such as Recommend, PlanUpdate,
InPlaceResize, EvictPod, Rollout, StartupBoost, and OOMRemediation. Outcomes are
Running, Succeeded, Failed, or Skipped. Its event identity/sequence,
workload reference, event time, reason, and before/target/observed resource
evidence explain optimization decisions. Retained sequences and dropped-event
boundaries matter: a missing event does not prove no action occurred.
WA Activity is not the SaaS organization audit log and is not returned by the NA controller event tools. No WA Activity read tool is registered in the referenced MCP catalog. TODO: expose this stream after its release, pagination, access, and retention contract is confirmed. Until then, use the Console; do not invent a tool name or reconstruct an activity timeline from resource charts.
WA resource/runtime history is separately available through
get_workload_metrics and get_workload_runtime_metrics. Their series include
explicit units; pressure series use percent, JVM heap series GiB, and JVM GC
duration seconds. An available pressure metric does not imply the policy uses
that signal for recommendations. Workload versus Pod scope must match before
comparing resource curves.
Image Accelerator metrics
get_image_accelerator_status.data.metrics contains cluster aggregate counters.
list_image_accelerator_nodes returns per-node items[] and clusterTotals.
| Metric | Unit / meaning |
|---|---|
receivedBytesTotal | Cumulative bytes obtained through peers. |
servedBytesTotal | Cumulative bytes served to peers. Do not add to received bytes as “unique accelerated data”: one transfer has a sender and receiver. |
remotePulledBytesTotal | Cumulative bytes obtained from a remote registry. |
mirrorRequestsHit, mirrorRequestsMiss | Request counters, not bytes. A request hit rate is different from a byte-weighted data hit rate. |
Per-node mirrorLastSuccessTimestamp | Unix seconds of last successful mirror operation; absent/zero history is not proof a pull just failed. |
The Console’s data hit rate is
receivedBytesTotal / (receivedBytesTotal + remotePulledBytesTotal), displayed
as a percentage. Do not substitute hit/miss request counters into that formula.
With no transferred data there is no observed traffic from which to establish
acceleration for a new workload.
Cluster totals include historical nodes and can exceed the sum of visible node
rows. Filters and pagination do not turn them into page totals. Counters are
not an hourly throughput rate or a measured end-to-end startup-time saving.
The list supports node-name search and a single sort field: node_name or
the snake_case spelling of one returned metric, such as received_bytes_total.
PVC coverage and units
list_cluster_pvcs.data.collection carries collection state, coverage, and
cluster summary independently of items[].
| Path | Meaning |
|---|---|
collection.collectionStatus | collecting, complete, partial, failed, or disabled. This is collection completeness, not list pagination. |
collection.freshnessStatus, collectedAt | Snapshot freshness/time; inspect per-row metric times as well. |
collection.disabledReason | diskMonitorSetting or dataProtectionMode, when collection is disabled. |
collection.coverage.expectedNodes, successfulNodes | Node collection coverage, not measured PVC count. |
collection.summary.totalPVCs, boundPVCs, measuredPVCs | Inventory/binding/measurement counts. Unmeasured PVCs are not counted as zero utilization. |
collection.summary.provisionedCapacityBytes | Capacity of bound claims, distinct from measured filesystems. |
collection.summary.measuredUsedBytes, measuredCapacityBytes | Used/capacity totals for measured filesystems. Their ratio is the aggregate measured utilization. |
Full row filesystemCapacityBytes, usedBytes, availableBytes | Kubelet-reported filesystem bytes, not GiB and not the storage pool’s capacity. Available space need not equal capacity minus used because of filesystem reservations. |
items[].usageRate | Fraction, not percentage points: illustrative 0.75 displays as 75%. Null is unavailable; a real zero is valid. |
Full row dataReason, metricCollectedAt | Missing-data reason (notBound, notMounted, unsupported, collectionFailed, or unknown) and sample time. |
Sort by usage_rate, namespace, name, storage_class, lifecycle_state,
data_state, or metric_collected_at; filter search, lifecycle_states,
and data_states. Request full detail for bytes, storage class, and collection
time. An unavailable metric cannot establish that a PVC is empty or safe to
delete. See PVC Utilization.
Cost history, allocation, and currency
| Question / tool | Main returned paths | Interpretation |
|---|---|---|
Organization cost: get_organization_cost_history | range; history.clustersCostsAggregationEstimates keyed by currency | Estimated aggregate history. Do not add converted versions of the same aggregate together. |
Cluster cost: get_cluster_cost_history | history.currency, symbol, averageMonthlyCost, latestMonthCost, latestHourlyCost, items[] | Keep monthly totals/estimates separate from hourly figures and bucket values. Optional input currency is USD or CNY. |
Capacity split: get_cluster_capacity_cost_history | history.clustersCostsByCapacity, clustersCostsByCapacityAggregationEstimates | On-Demand/Spot series with currency-specific aggregates. Do not add an aggregate to the same source split again. |
Stored records: list_cluster_cost_records | items[].recordTime, currency, average cores/GiB, resource-time totals, On-Demand/Spot/total compute costs | Averages, integrated resource-time, and monetary costs are different dimensions. Filter exact stored currency and record-time range. |
Workload allocation: list_workload_costs | summary, items, trend, coverage, supportedDimensions, unsupportedDimensions | Read availability and coverage before ranking workloads. Allocated compute cost is estimated attribution, not a workload invoice. |
One workload: get_workload_cost_trend | trend.items, poolCosts, currency, coveragePercent, availability, agentUpgradeRequired | Query an exact workload_id returned by the list, with the same range and NodePool scope. |
Custom unit prices: get_cluster_custom_price | currency, price.cpuPricePerHour, price.memoryPricePerHour | USD per CPU core-hour and per GiB-hour. A pricing assumption, not measured consumption or a cloud purchase. |
Organization/cluster cost history defaults to 30d, supports 7d/30d/90d,
and custom spans up to 100 days. Capacity history supports 7d/30d and up to
32 days; workload cost supports 7d (default)/30d and up to 31 days. Cost
range endpoints must be full-hour Unix seconds, ordered and not in the future;
omitted end uses the current completed hour. max_points is 2–500 (default
100). Effective steps can grow to respect it; workload cost uses whole-hour
steps that divide the selected range. Read returned range instead of assuming
that all tools used the same resolution.
Capacity samples retain the source keys ON_DEMAND and SPOT; corresponding
monthly/hourly aggregates are maps by capacity type. Cost record resource-time
totals such as totalCPUCoreTime and totalMemoryGiBTime are core-hours and
GiB-hours, whereas avgTotalCPUCore and avgTotalMemoryGiB are daily average
cores and GiB. Record costs are onDemandComputeCost, spotComputeCost, and
totalComputeCost, with recordTime identifying the stored daily record.
Workload cost queries filter namespaces, kinds, node_pools, and literal
name search. Sort fields are total_cost (default descending), cpu_cost,
memory_cost, requested_cpu, requested_memory, cpu_utilization,
memory_utilization, wasted_cpu, wasted_memory, wasted_cost, name,
namespace, or kind. Pin explicit full-hour start/end values for a long
pagination traversal; do not let a moving preset resolve to a different range
on a subsequent page.
| Workload cost row / summary | Meaning |
|---|---|
items[].workloadID, name, namespace, kind, nodePools | Workload identity and allocation dimensions. Keep the returned workload ID for trend requests. |
metadataAvailable, currentResourcesAvailable | Whether current metadata/resource fields are populated; historical cost can survive a missing current workload. |
currentReplicas, currentRequestedCPU, currentRequestedMemoryGiB, currentUsedCPU, currentUsedMemoryGiB | Current replica/resource state, in cores and GiB; optional usage is not zero. |
cpuCost, memoryCost, totalCost | Estimated cost allocated over the selected range. |
currentHourlyEstimate, currentWastedHourlyCost | Current hourly cost and idle-allocation estimate, not immediately realizable savings or the range’s accumulated cost. |
cpuUtilizationPercent, memoryUtilizationPercent | Percent scale (25 means 25%), not a fraction. |
summary.totalWorkloadCost, allocatedCost, averageDailyCost, currentHourlyEstimate, mostExpensiveWorkload | Distinct range cost, allocation summary, daily average, current run rate, and ranked workload. Keep scope and currency attached. |
Trend points contain timestamp, cpuCost, memoryCost, totalCost, and
optional requestedCPU, requestedMemoryGiB, usedCPU, usedMemoryGiB.
Single-workload poolCosts separates the selected range’s allocation by
nodePool; do not add this split to the same workload total a second time.
Workload cost summary.availability can be available, collecting, partial,
or unsupported; accuracy is estimated. Inspect calculationMode,
coveragePercent, coverage.observedPercent, unmappedWorkloadCount,
historyStartedAt, and agentUpgradeRequired. A null cost is unavailable, not
free. Current resource fields and currentHourlyEstimate are not historical
range totals. Support for newer workload-cost reads and dimensions must be
confirmed in the deployed catalog; do not treat main-branch schema presence as
release confirmation.
Timestamp encodings also differ: range inputs use Unix seconds, ordinary
resource/WA metric samples use seconds, and cost history.items[].timestamp
uses the source’s Prometheus timestamp encoding (seconds, including fractions).
Do not apply a blanket milliseconds conversion to every numeric timestamp.
history.step is the raw API duration value, while the surrounding MCP
range.step is a duration string. Use the latter for human-readable resolution.
Saving Report and baseline selection
MCP: get_saving_report. Check get_cluster_overview.capabilities.savingReport
and the deployed Console provider support before offering the report; it is not
available for Universal clusters. A capability flag is not proof that baseline
data exists.
The report range and baseline range are separate. period is 7d or 30d
(default). A custom report uses paired positive start/end Unix seconds,
no period, no future end, and a maximum 100-day span. Custom step defaults
to 24h and must be at least 1h. The existing cost API rounds report endpoints
to full hours; the returned range records that effective request.
An optional baseline_start/baseline_end pair selects a read-time baseline
range (ordered, not future, at most 100 days). It does not save a cluster
setting. Without that override, the server uses a saved default baseline range
when configured, otherwise the initial captured baseline. Inspect the returned
baselineMode (custom_range, default_range, or initial),
baselineRangeStart, baselineRangeEnd, and baselineDate; do not assume every
report uses the first onboard date. Current and range-baseline resources are
averages over their respective ranges, not a pair of instant node inventories.
| Report path | Unit / meaning |
|---|---|
baseline, current | Monthly cost; provisionedCpu, requestedCpu, usedCpu in cores; provisionedMemoryGiB, requestedMemoryGiB, usedMemoryGiB in GiB. |
baseline.cpuUtilization, memoryUtilization; corresponding current fields | Used/requested fractions, not used/provisioned. Illustrative 0.25 is 25%. |
savings.estimatedMonthlyAmount, savings.percent | Legacy savings amount and percentage; percent uses the percent scale (25 means 25%). |
consoleSavings.amount, percent, baselineCost, currentCost, source | Optional Console-equivalent headline calculation. source identifies estimated_original_cost or cluster_cost. Prefer this group when reproducing that headline; do not mix its basis with legacy savings fields. |
estimatedOriginalProvisionedCPU, estimatedOriginalProvisionedMem, estimatedOriginalCost | Optional counterfactual estimates using the baseline operating pattern; not measured current capacity or an invoice. |
costPerCPUBaseline, costPerCPUCurrent | Currency per provisioned CPU core-hour. |
costPerCPUSavingsPercent, provisionedCPUGrowthRate, costGrowthRate | Source fractions (0.25 means 25%), unlike savings.percent and consoleSavings.percent. |
savings.cpuUtilizationImprovement, memoryUtilizationImprovement | Difference between utilization fractions. Illustrative 0.1 is an increase of 10 percentage points, not 10% relative growth. |
currency, symbol | Currency for monetary report fields. Keep them with every displayed amount. |
Read Saving Report for calculation details. Resource efficiency, changed workload demand, cheaper capacity, custom prices, and discounts can all affect a comparison. Do not attribute the entire monthly difference to one policy change without evidence.
Investigation scenarios
Optimization is enabled, but the cluster looks unhealthy
Read overview freshness, Pod counts, active alerts, and autoscaler state first. Then filter nodes by pool/status and read the relevant controller event detail. For WA, inspect workload readiness, assigned policy, recommendations, and matching-scope history; use Console WA Activity for action evidence. Report what is configured, what was observed, and what is missing separately. Do not recommend Rebalance or Upgrade merely to repair an unknown/stale read.
Compare savings before and after a configuration change
Pin explicit report and baseline ranges, currency, pricing assumptions, and workload scope. Read returned effective ranges and baseline mode; retain warnings and coverage. Compare resource usage and requested/provisioned capacity alongside cost. Use the Console headline basis for Console parity and label counterfactual savings as estimated. Never sum monthly estimates as daily spend.
Verify Image Accelerator or PVC monitoring
For acceleration, check support/install state and NodePool/NodeClass settings, then compare peer/remote counters around an actual authorized image-pull test; historical totals alone cannot establish a new pull’s path. For PVCs, check collection status, measured PVC/node coverage, per-row data state and metric time before interpreting usage. An unsupported or disabled measurement remains unknown, even if its list page is complete.