Resource Observation

Overview

EXAMPLEmonitoring questions
Question Useful evidence
Which Pod currently uses the most CPU? kubectl top pod
Which container inside a Pod uses the memory? kubectl top pod --containers
Why did a container restart? Pod status, last termination state, and events
Was CPU usage high fifteen minutes ago? A time-series monitoring system
Is the container being CPU-throttled? cgroup or Prometheus container metrics
Is Metrics Server itself unhealthy? APIService status, Deployment status, and logs

To find answers to these questions, the monitoring system depends on collection, transport, storage, query, and interpretation capabilities.

Start the Monitoring Workload

Step 1: Create an isolated namespace 


1
2

 
kubectl create namespace rke2-monitoring
kubectl get namespace rke2-monitoring

All workload resources used in this lecture belong in this namespace.

Step 2: Create the multi-container Pod

Create resource-observer.yaml:



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

 
apiVersion: v1
kind: Pod
metadata:
  name: resource-observer
  namespace: rke2-monitoring
  labels:
    app: resource-observer
spec:
  restartPolicy: Always
  containers:
    - name: web
      image: nginx:1.27-alpine
      ports:
        - containerPort: 80
      resources:
        requests:
          cpu: 20m
          memory: 24Mi
        limits:
          cpu: 100m
          memory: 64Mi

    - name: cpu-worker
      image: busybox:1.36
      command:
        - /bin/sh
        - -c
        - |
          echo "Starting bounded CPU workload"
          while true; do
            i=0
            while [ "$i" -lt 50000 ]; do
              i=$((i + 1))
            done
          done
      resources:
        requests:
          cpu: 50m
          memory: 8Mi
        limits:
          cpu: 200m
          memory: 32Mi

    - name: memory-holder
      image: busybox:1.36
      command:
        - /bin/sh
        - -c
        - |
          echo "Allocating a file in a memory-backed volume"
          dd if=/dev/zero of=/memory/payload bs=1M count=48
          sleep 36000
      volumeMounts:
        - name: memory-data
          mountPath: /memory
      resources:
        requests:
          cpu: 5m
          memory: 16Mi
        limits:
          cpu: 50m
          memory: 96Mi
  volumes:
    - name: memory-data
      emptyDir:
        medium: Memory
        sizeLimit: 64Mi

Deploy the pod



1
2
3

 
kubectl apply -f resource-observer.yaml
kubectl -n rke2-monitoring wait --for=condition=Ready pod/resource-observer --timeout=120s
kubectl -n rke2-monitoring get pod resource-observer -o wide

The containers provide three different behaviors:

  • web provides a low-usage application container;
  • cpu-worker continuously requests CPU time but is limited to 200m;
  • memory-holder writes 48Mi to a memory-backed volume.

The resource limits bound the load generated by this hands-on exercise.

The Kubernetes resource metrics pipeline 
flowchart TB
    CG["Linux cgroups / CRI stats"] --> KL["kubelet on each node"]
    KL -->|"resource endpoint"| MS["RKE2 packaged Metrics Server"]
    MS --> MA["Kubernetes Metrics API"]
    MA --> TOP["kubectl top"]
    MA --> HPA["Horizontal Pod Autoscaler"]
  1. Linux cgroups and the container runtime account for container usage.
  2. The kubelet exposes node, Pod, and container resource metrics.
  3. Metrics Server periodically collects those metrics from each kubelet.
  4. Metrics Server publishes them through the Kubernetes API aggregation layer.
  5. kubectl top and autoscalers query the Metrics API.

This pipeline primarily exposes:

  • CPU usage;
  • memory working-set usage;
  • the measurement timestamp and window.

It does not provide a long-term history, application request rates, traces, or logs.

How to interpret CPU and memory

CPU

  • 1 CPU is one Kubernetes CPU unit.
  • 1000m is one CPU; 250m is one quarter of a CPU.
  • A value from kubectl top represents recent consumption.
  • CPU is measured over a time window.
    • A short usage spike may be averaged out.

Memory

  • Memory is normally displayed in Ki, Mi, or Gi.
  • Metrics Server reports a working-set-oriented value.
  • Memory usage is not the same as a container’s request or limit.
  • Reaching a memory limit can cause the kernel to terminate a process with an out-of-memory event.
Four relevant resource values
  • Requests and limits are configuration.
  • Current and historical usage are measurements.
Value Source Purpose
Request resources.requests in the Pod specification The scheduler uses it when choosing a node
Limit resources.limits in the Pod specification The runtime enforces it with Linux cgroups
Current usage Runtime/cgroup statistics collected through the kubelet and Metrics Server Shows recent CPU and memory use
Historical usage Samples stored by Prometheus or another monitoring system Shows changes over time
INFORequest and limit

Inspect the resource configuration of all three containers:



1

 
kubectl -n rke2-monitoring get pod resource-observer -o jsonpath='{range .spec.containers[*]}{.name}{"\n  requests: "}{.resources.requests}{"\n  limits:   "}{.resources.limits}{"\n\n"}{end}'
  • The scheduler used these requests when placing resource-observer.
  • The runtime applies the limits separately to each container.
  • The cpu-worker CPU limit is 200m.
  • The memory-holder memory limit is 96Mi.
  • Neither value reports actual consumption.
INFOCurrent usage

The node measures container usage. The kubelet exposes the measurements, Metrics Server collects them, and kubectl top displays them:



1

 
kubectl top pod resource-observer -n rke2-monitoring --containers

Illustrative output:



1
2
3
4

 
POD                 NAME            CPU(cores)   MEMORY(bytes)
resource-observer   web             1m           8Mi
resource-observer   cpu-worker      198m         1Mi
resource-observer   memory-holder   1m           49Mi

Your values will differ. The expected pattern is that cpu-worker uses the most CPU and memory-holder uses the most memory. Metrics Server does not preserve earlier samples.

INFOHistorical usage

Prometheus or another monitoring system can collect and store repeated samples:



1
2
3
4
5

 
TIME    CONTAINER       CPU    MEMORY
10:00   cpu-worker      195m   1Mi
10:00   memory-holder   1m     49Mi
10:05   cpu-worker      199m   1Mi
10:05   memory-holder   1m     49Mi

This history can show whether usage is stable, increasing, or temporary.

Example interpretation

For the illustrative output above:

  • cpu-worker uses more than its 50m request. This is allowed.
  • Its usage is close to, but does not directly prove throttling at, its 200m limit.
  • memory-holder uses more than its 16Mi request.

  • Its usage remains below its 96Mi limit.

  • Historical measurements are needed before changing the request or limit.

RKE2 Metrics Server

RKE2 packaged component

RKE2 distributes Metrics Server as a packaged component named rke2-metrics-server.

On an RKE2 server, packaged component manifests are managed below:



1

 
/var/lib/rancher/rke2/server/manifests/

RKE2 rewrites packaged manifests when the service starts. Do not directly edit those generated files. RKE2 configuration or the supported chart customization mechanism should be used when a packaged component must be changed.

The component can be disabled through RKE2 server configuration:



1
2

 
disable:
  - rke2-metrics-server

Disabling a packaged component actively removes it. This hands-on exercise assumes it is not disabled.

Resource metrics are deliberately narrow

Metrics Server supplies current CPU and memory measurements for:

  • kubectl top;
  • Horizontal Pod Autoscaler;
  • Vertical Pod Autoscaler;
  • basic current-state inspection.

It is not a replacement for Prometheus or another monitoring platform.

Capability Metrics Server Prometheus-style monitoring
Current node CPU and memory Yes Yes
Current Pod/container CPU and memory Yes Yes
Historical queries No Yes
Dashboards No Yes
Alert rules No Yes
Application metrics No Yes, when exported and scraped
Detailed throttling/restart series No Yes, when collected
RKE2 scheduling caveat

In an HA design, server nodes may be tainted so ordinary workloads do not run on them. If every available node is restricted to critical add-ons, Metrics Server can remain Pending until an eligible agent node is added.

Always inspect the actual scheduling event:



1

 
kubectl -n kube-system describe pod -l app.kubernetes.io/name=metrics-server

Validate the Resource Metrics Pipeline

Step 1: Record the cluster versions

Resource metrics behavior depends on the RKE2, Kubernetes, and Metrics Server versions. Record them before troubleshooting.



1
2
3

 
kubectl version
kubectl get nodes -o wide
kubectl -n kube-system get deployment rke2-metrics-server -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
Step 2: Locate the RKE2 add-on 


1
2
3

 
kubectl -n kube-system get addons.k3s.cattle.io | grep metrics
kubectl -n kube-system get deployment rke2-metrics-server
kubectl -n kube-system get pods -l app.kubernetes.io/name=metrics-server -o wide

The expected state is:

  • the add-on exists;
  • the Deployment has an available replica;
  • the Pod is Running and ready;
  • the Pod has been scheduled on a node.

If the label query returns no Pod, inspect the Deployment’s selector instead of guessing:



1
2

 
kubectl -n kube-system get deployment rke2-metrics-server -o jsonpath='{.spec.selector.matchLabels}{"\n"}'
kubectl -n kube-system get pods --show-labels | grep metrics-server
Step 3: Validate the aggregated API 


1
2

 
kubectl get apiservice | grep metrics.k8s.io
kubectl api-versions | grep metrics.k8s.io

Inspect the registered APIService:



1

 
kubectl describe apiservice $(kubectl get apiservice -o name | grep metrics.k8s.io | head -n 1)

Look for an Available condition with status True.

Query API discovery directly:



1

 
kubectl get --raw /apis/metrics.k8s.io/

This tests the API path without relying on the formatting performed by kubectl top.

Confirm that the running hands-on Pod has metrics:



1

 
kubectl top pod resource-observer -n rke2-monitoring --containers
Step 4: Inspect Metrics Server logs 


1

 
kubectl -n kube-system logs deployment/rke2-metrics-server --tail=100

Common failure categories include:

  • the Metrics API has not finished registering;
  • Metrics Server cannot reach a kubelet;
  • a kubelet certificate cannot be validated;
  • node addressing selects an unreachable address;
  • a firewall blocks the path;
  • the Pod is unschedulable or repeatedly restarting.

Record the exact message. Avoid applying broad insecure TLS workarounds before identifying the certificate or addressing problem.

Observe Node and Pod Usage

Establish a baseline

Metrics may take a short time to appear after a node or Pod starts.



1
2
3
4

 
kubectl top nodes
kubectl top pods -A
kubectl top pods -A --sort-by=cpu
kubectl top pods -A --sort-by=memory

Questions:

  1. Which node reports the greatest CPU usage?
  2. Which system Pod reports the greatest memory usage?
  3. Does one number alone prove that either resource is overloaded?
  4. What additional capacity or historical evidence would be needed?
Inspect allocatable capacity

kubectl top node reports recent usage. Node objects report capacity and allocatable resources.



1

 
kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.capacity.cpu,CPU-ALLOCATABLE:.status.allocatable.cpu,MEMORY:.status.capacity.memory,MEMORY-ALLOCATABLE:.status.allocatable.memory'

allocatable is the resource amount available for Pods after Kubernetes accounts for resources reserved for the node and system.

Do not calculate production headroom from a single kubectl top sample. Capacity planning normally uses historical percentiles, workload growth, and failure scenarios.

Observe by namespace and label 


1
2
3

 
kubectl top pod -n kube-system
kubectl top pod -n rke2-monitoring
kubectl top pod -n rke2-monitoring -l app=resource-observer --containers

Useful filters include:

  • namespace for team or environment ownership;
  • labels for one application;
  • --containers for attribution inside multi-container Pods;
  • sorting for quick triage.

Hands-on: Observe the Multi-Container Pod

Step 3: Observe the Pod and each container

Allow the metrics pipeline to collect samples, then run:



1
2

 
kubectl top pod -n rke2-monitoring
kubectl top pod resource-observer -n rke2-monitoring --containers

Repeat the container view several times:



1
2
3
4
5

 
for sample in 1 2 3 4 5; do
  date
  kubectl top pod resource-observer -n rke2-monitoring --containers
  sleep 10
done

Expected observations:

  • cpu-worker should dominate CPU consumption;
  • memory-holder should dominate memory consumption;
  • the Pod total combines its containers;
  • measurements vary between samples;
  • usage is not identical to requests or limits.
Step 4: Compare configuration with observation

Display each container’s requests and limits:



1

 
kubectl -n rke2-monitoring get pod resource-observer -o jsonpath='{range .spec.containers[*]}{.name}{"\n  requests: "}{.resources.requests}{"\n  limits:   "}{.resources.limits}{"\n\n"}{end}'

Complete the following for each container:

Container CPU request CPU limit Observed CPU Memory request Memory limit Observed memory
web            
cpu-worker            
memory-holder            

Interpret the results:

  1. Which container is using more than its CPU request?
  2. Which container is closest to its memory limit?
  3. Would one sample justify lowering a request?
  4. What time range and workload conditions would make that decision safer?
Step 5: Observe CPU limiting without overclaiming

The cpu-worker has a 200m CPU limit and continuously asks for CPU time. Its observed usage should remain near the configured boundary, subject to measurement windows and node conditions.



1

 
kubectl top pod resource-observer -n rke2-monitoring --containers

This is evidence that usage is bounded, but kubectl top does not directly report a throttling counter. To prove CPU throttling, use cgroup statistics or a monitoring system that collects metrics such as throttled CFS periods.

Diagnose Memory Failure

Step 1: Create a Pod that exceeds its memory limit

Create memory-failure.yaml:



1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

 
apiVersion: v1
kind: Pod
metadata:
  name: memory-failure
  namespace: rke2-monitoring
  labels:
    app: memory-failure
spec:
  restartPolicy: Never
  containers:
    - name: allocator
      image: python:3.12-alpine
      command:
        - python3
        - -c
        - |
          import time
          print("Allocating 96 MiB with a 64 MiB container limit", flush=True)
          payload = bytearray(96 * 1024 * 1024)
          print(len(payload), flush=True)
          time.sleep(3600)
      resources:
        requests:
          cpu: 10m
          memory: 32Mi
        limits:
          cpu: 100m
          memory: 64Mi

Apply and watch it:



1
2

 
kubectl apply -f memory-failure.yaml
kubectl -n rke2-monitoring get pod memory-failure -w

Exit the watch with Ctrl-C after the container terminates.

Step 2: Build an evidence chain

Inspect status and events:



1
2
3

 
kubectl -n rke2-monitoring describe pod memory-failure
kubectl -n rke2-monitoring get pod memory-failure -o jsonpath='{.status.containerStatuses[0].state.terminated.reason}{"\n"}'
kubectl -n rke2-monitoring get events --sort-by=.lastTimestamp

Attempt to inspect current metrics:



1

 
kubectl top pod memory-failure -n rke2-monitoring --containers

The terminated container may disappear before Metrics Server exposes a useful sample. This demonstrates an important limitation:

Current-state metrics alone may not preserve the evidence needed after a short failure.

The Pod’s termination reason and events are the stronger evidence here. Historical monitoring would make it possible to correlate memory growth with the termination.

Step 3: Explain, do not merely label, the failure

A complete diagnosis includes:

  1. the configured memory limit;
  2. the attempted allocation;
  3. the observed termination reason;
  4. the restart policy;
  5. why kubectl top may not retain a sample;
  6. a proposed correction.

Possible corrections depend on the cause:

  • reduce or bound the application’s memory behavior;
  • process data in smaller chunks;
  • correct a memory leak;
  • raise the request and limit based on measured evidence;
  • add historical monitoring and an alert before the limit is reached.

Simply increasing the limit can postpone a leak and move the failure to the node level.

Raw Metrics and Runtime Evidence

Query the Metrics API directly

Discover the version served by the cluster:



1

 
kubectl api-versions | grep metrics.k8s.io

Save the first advertised version:



1
2
3
4
5

 
METRICS_VERSION=$(
  kubectl api-versions |
  awk -F/ '/^metrics.k8s.io\// {print $2; exit}'
)
echo "$METRICS_VERSION"

Query resource-observer through the raw Metrics API:



1

 
kubectl get --raw "/apis/metrics.k8s.io/${METRICS_VERSION}/namespaces/rke2-monitoring/pods/resource-observer" | python3 -m json.tool

Locate:

  • timestamp;
  • window;
  • Pod and container names;
  • CPU usage;
  • memory usage.

kubectl top is a convenient client for these API data, not the collector.

Compare with RKE2 container runtime statistics

On an RKE2 node, the bundled crictl can query containerd through the CRI socket:



1

 
sudo /var/lib/rancher/rke2/bin/crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock stats

Locate the resource-observer Pod sandbox and its containers:



1
2
3

 
POD_ID=$(sudo /var/lib/rancher/rke2/bin/crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock pods --name resource-observer -q)

sudo /var/lib/rancher/rke2/bin/crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock ps --pod "$POD_ID"

The runtime view and kubectl top may differ because:

  • they may use different sampling windows;
  • commands may run at different times;
  • Metrics Server measurements are intended for autoscaling and basic inspection;
  • display units and aggregation differ.

Agreement in broad behavior is more meaningful than exact equality between two asynchronous samples.

Inspect node processes safely

On the node hosting resource-observer:



1

 
kubectl -n rke2-monitoring get pod resource-observer -o wide

Then, on that node:



1

 
sudo systemd-cgtop

Exit with q.

This is a host-level view. It is useful when the Kubernetes API or Metrics Server is unavailable, but translating host cgroups back to Kubernetes workloads requires care.

From Live Metrics to Historical Monitoring

When Metrics Server is not enough

Use a historical monitoring system when the question includes:

  • when did usage increase?
  • how long did the condition last?
  • is consumption trending upward?
  • what was the 95th percentile over one week?
  • should an alert fire before an OOM kill?
  • did an application deployment change resource behavior?

A common RKE2/Rancher monitoring stack uses:

  • Prometheus for collection and time-series storage;
  • Prometheus Operator for declarative configuration;
  • ServiceMonitor and PodMonitor resources for scrape discovery;
  • Grafana for visualization;
  • Alertmanager for notification routing.

Monitoring stacks consume meaningful CPU, memory, and storage themselves. Check the supported Rancher/RKE2 release documentation and cluster capacity before installing one. Do not add a full stack merely to make kubectl top work.

References