ClusterIP, NodePort, and Multi-Service Communication

Overview

• We already know how to create Pods and how Deployments keep the desired number of Pods running.
• The next problem is connectivity:
  • Pod IP addresses are ephemeral.
  • A restarted Pod may receive a new IP address.
  • A scaled Deployment may have several interchangeable Pods.
• Applications therefore should not normally connect directly to a particular Pod IP.
• Kubernetes Services provide stable network identities in front of changing sets of Pods.
• In this lecture we focus on two Service types:
  • ClusterIP: stable access from inside the cluster.
  • NodePort: access through a port on a Kubernetes node.

A useful mental model is:

1
2
 
Deployment  -> keeps Pods running
Service     -> keeps applications connected to those Pods
 

Services overview

Pods are intentionally replaceable. Consider a Deployment with two replicas:

1
2
3
4
 
backend Deployment
    |
    +-- backend Pod A   10.42.0.8
    +-- backend Pod B   10.42.1.4
 

If Pod A is recreated, its replacement may become 10.42.0.12. A frontend that hard-codes 10.42.0.8 will break.

A Service provides a stable abstraction:

1
2
3
4
5
 
backend-svc
ClusterIP: 10.43.140.207
        |
        +--> backend Pod A
        +--> backend Pod B
 

The Service selects Pods using labels.

Common Service types

Type	Reachability	Typical use
ClusterIP	Inside the cluster	Backend APIs, databases, internal services
NodePort	Through a port on node IPs	Labs, simple external exposure, debugging
LoadBalancer	External load balancer	Cloud-provider production exposure

ClusterIP is the default Service type. A NodePort Service also receives a ClusterIP; NodePort adds another way to reach the same Service.

flowchart TB
    subgraph Cluster["Kubernetes Cluster"]
        subgraph Pods["Backend Pods"]
            B1["Backend Pod 1\n10.42.0.x:80"]
            B2["Backend Pod 2\n10.42.1.x:80"]
        end

        C1["ClusterIP Service\n10.43.x.x:80"]
        N1["NodePort\nNodeIP:30080"]
    end

    Internal["Internal Pod"] -->|"backend-svc:80"| C1
    External["External client"] -->|"NodeIP:30080"| N1
    N1 --> C1
    C1 --> B1
    C1 --> B2

Suppose we have:

1
2
3
4
 
containers:
  - name: backend
    ports:
      - containerPort: 8080
 

and:

1
2
3
4
 
ports:
  - port: 80
    targetPort: 8080
    nodePort: 30080
 

Then:

1
2
3
4
5
6
7
 
containerPort = 8080   Documentation of the application's container port

targetPort    = 8080   Port on the selected Pods to which the Service forwards

port          = 80     Port exposed by the Service / ClusterIP

nodePort      = 30080  Port exposed on eligible Kubernetes node addresses
 

The traffic path is:

1
 
NodeIP:30080 -> Service:80 -> Pod:8080
 

Deployment Validation

Kubernetes YAML is unforgiving about indentation and field placement. Before applying a manifest, validate it.

INFOYAML syntax

If yamllint is installed:

1
2
3
 
sudo apt update
sudo apt install -y yamllint
yamllint echo-deployment.yaml
 

INFOKubernetes client-side validation

1
 
kubectl apply --dry-run=client -f echo-deployment.yaml
 

This parses the YAML and checks the resource structure known to kubectl.

INFOAPI server validation

Once the cluster is running:

1
 
kubectl apply --dry-run=server -f echo-deployment.yaml
 

This asks the API server to validate the object without storing it.

A useful workflow is:

1
2
3
 
kubectl apply --dry-run=client -f app.yaml
kubectl apply --dry-run=server -f app.yaml
kubectl apply -f app.yaml
 

INFOFinal validation

After applying, do not assume success simply because kubectl apply returned without an error:

1
2
3
 
kubectl get deployments
kubectl get pods -o wide
kubectl describe deployment backend
 

ClusterIP Services

Concept

A ClusterIP Service provides:

• a stable virtual IP address;
• a stable DNS name;
• a logical endpoint in front of one or more Pods;
• traffic distribution among healthy endpoints.

Applications normally use the Service DNS name, not the ClusterIP itself.

For a Service named backend-svc in the default namespace:

1
2
3
4
 
backend-svc
backend-svc.default
backend-svc.default.svc
backend-svc.default.svc.cluster.local
 

are progressively more explicit DNS names.

ClusterIP Demo

INFOStep 1: Create the backend Deployment

Create echo-deployment.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
        - name: backend
          image: nginxdemos/hello:plain-text
          ports:
            - containerPort: 80
 

Validate it first:

1
 
kubectl apply --dry-run=client -f echo-deployment.yaml
 

Then deploy:

1
 
kubectl apply -f echo-deployment.yaml
 

Observe the Pods:

1
 
kubectl get pods -l app=backend -o wide
 

Record the Pod IP addresses for debugging purposes.

INFOStep 2: Create a ClusterIP Service

Create echo-svc.yaml:

1
2
3
4
5
6
7
8
9
10
 
apiVersion: v1
kind: Service
metadata:
  name: backend-svc
spec:
  selector:
    app: backend
  ports:
    - port: 80
      targetPort: 80
 

Validate and deploy:

1
2
3
 
kubectl apply --dry-run=client -f echo-svc.yaml
kubectl apply --dry-run=server -f echo-svc.yaml
kubectl apply -f echo-svc.yaml
 

Inspect the Service:

1
2
 
kubectl get svc backend-svc -o wide
kubectl describe svc backend-svc
 

INFOStep 3: Verify that the Service found the Pods

Modern Kubernetes represents Service backends using EndpointSlices:

1
 
kubectl get endpointslices -l kubernetes.io/service-name=backend-svc -o wide
 

Compare those addresses with:

1
 
kubectl get pods -l app=backend -o wide
 

If the Service has no endpoints, the most common cause is a mismatch between:

1
2
3
 
spec:
  selector:
    app: backend
 

and the Pod labels:

1
2
3
 
metadata:
  labels:
    app: backend
 

Inspect labels with:

1
 
kubectl get pods --show-labels
 

INFOStep 4: Test Service DNS from inside the cluster

Create a temporary curl Pod:

1
 
kubectl run curl-test --image=curlimages/curl --restart=Never -it --rm -- sh
 

Inside the Pod:

1
2
3
 
nslookup backend-svc
curl http://backend-svc
curl http://backend-svc.default.svc.cluster.local
 

Exit when finished:

1
 
exit
 

INFOStep 5: Compare Service and direct-Pod connectivity

Get the Pod addresses:

1
 
kubectl get pods -l app=backend -o wide
 

Launch the temporary curl pod above again and compare:

1
2
 
curl http://<POD-IP>:80
curl http://backend-svc:80
 

The first tests CNI Pod networking directly. The second also tests Service routing and DNS.

NodePort Services

Concept

A NodePort Service exposes a Service through a port on eligible node addresses.

The default NodePort range is:

1
 
30000-32767
 

A NodePort Service still has a ClusterIP. Conceptually:

1
2
3
4
5
6
7
8
9
10
 
client
  |
  v
NodeIP:NodePort
  |
  v
ClusterIP:port
  |
  v
selected Pod:targetPort
 

For our FABRIC RKE2 environment, test NodePort against the deliberately configured dataplane node IP, such as 192.168.1.1, rather than assuming that 127.0.0.1 is equivalent. Loopback NodePort behavior depends on kube-proxy mode and configuration.

NodePort Hands-On

Option 1: Create a NodePort imperatively

1
 
kubectl expose deployment backend --name=backend-nodeport --type=NodePort --port=80 --target-port=80
 

Inspect the assigned port:

1
 
kubectl get svc backend-nodeport
 

Example:

1
2
 
NAME               TYPE       CLUSTER-IP      PORT(S)
backend-nodeport   NodePort   10.43.120.50    80:31642/TCP
 

Here:

1
2
 
ClusterIP port = 80
NodePort       = 31642
 

Test from a node or another reachable host:

1
 
curl http://192.168.1.1:31642
 

Option 2: Specify a fixed NodePort

For repeatable labs, a fixed port can be easier. Create the following file called backend-nodeport.yaml.

1
2
3
4
5
6
7
8
9
10
11
12
 
apiVersion: v1
kind: Service
metadata:
  name: backend-nodeport
spec:
  type: NodePort
  selector:
    app: backend
  ports:
    - port: 80
      targetPort: 80
      nodePort: 38080
 

Validate before deployment:

1
2
 
kubectl apply --dry-run=server -f backend-nodeport.yaml
kubectl apply -f backend-nodeport.yaml
 

Then:

1
 
curl -v --connect-timeout 5 http://192.168.1.1:38088/
 

Inspect what backs the Service

1
2
 
kubectl get svc backend-nodeport -o wide
kubectl get endpointslices -l kubernetes.io/service-name=backend-nodeport -o wide
 

Multi-Service Communication

Concept

Real applications usually contain multiple services. The important design principle is:

Internal components communicate through Service names, not through Pod IP addresses.

We will build three components:

• Quote Service: provides text.
• Time Service: provides the current server time.
• Frontend Service: calls both internal Services and combines their responses.

Only the frontend needs external exposure.

Quote of the Day App

INFOArchitecture

flowchart TB
    User["External client"] -->|"NodeIP:NodePort"| FS["frontend-svc\nNodePort"]
    FS --> FE["Frontend Pod\n:8080"]
    FE -->|"http://quote-svc"| QS["quote-svc\nClusterIP :80"]
    FE -->|"http://time-svc"| TS["time-svc\nClusterIP :80"]
    QS --> Q1["Quote Pod 1 :80"]
    QS --> Q2["Quote Pod 2 :80"]
    TS --> T1["Time Pod :8080"]

Notice that the frontend does not know the Pod IPs for quote or time. It only knows their Service names.

INFOQuote Service

Create quote.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: quote
spec:
  replicas: 2
  selector:
    matchLabels:
      app: quote
  template:
    metadata:
      labels:
        app: quote
    spec:
      containers:
        - name: quote
          image: nginxdemos/hello:plain-text
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: quote-svc
spec:
  selector:
    app: quote
  ports:
    - port: 80
      targetPort: 80
 

1
2
 
kubectl apply --dry-run=server -f quote.yaml
kubectl apply -f quote.yaml
 

INFOTime Service

Create time.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: time
spec:
  replicas: 1
  selector:
    matchLabels:
      app: time
  template:
    metadata:
      labels:
        app: time
    spec:
      containers:
        - name: time
          image: busybox
          command: ["sh", "-c"]
          args:
            - |
              while true; do
                printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nConnection: close\r\n\r\n$(date)\n" \
                | nc -l -p 8080 -s 0.0.0.0;
              done
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: time-svc
spec:
  selector:
    app: time
  ports:
    - port: 80
      targetPort: 8080
 

1
2
 
kubectl apply --dry-run=server -f time.yaml
kubectl apply -f time.yaml
 

INFOFrontend Service

Create frontend.yaml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
        - name: frontend
          image: busybox
          command: ["sh", "-c"]
          args:
            - |
              while true; do
                Q=$(wget -qO- http://quote-svc);
                T=$(wget -qO- http://time-svc);
                printf "HTTP/1.1 200 OK\r\nContent-Type: text/plain\r\nConnection: close\r\n\r\nQuote service:\n$Q\nTime: $T\n" \
                | nc -l -p 8080 -s 0.0.0.0;
              done
          ports:
            - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: frontend-svc
spec:
  type: NodePort
  selector:
    app: frontend
  ports:
    - port: 80
      targetPort: 8080
      nodePort: 30080
 

Validate and deploy:

1
2
 
kubectl apply --dry-run=server -f frontend.yaml
kubectl apply -f frontend.yaml
 

Inspect everything:

1
2
3
4
 
kubectl get deployments
kubectl get pods -o wide
kubectl get svc
kubectl get endpointslices
 

Test internal Services first:

1
2
3
4
 
kubectl run curl-test \
  --image=curlimages/curl \
  --restart=Never \
  -it --rm -- sh
 

Inside the temporary curl pod

1
2
3
 
curl http://quote-svc
curl http://time-svc
curl http://frontend-svc
 

Then test the NodePort from a FABRIC node:

1
 
curl -v --connect-timeout 5 http://192.168.1.1:30080/
 

A Layered Troubleshooting Workflow

When a Service fails, do not begin by changing random firewall, MTU, or offload settings. Test one layer at a time.

Suppose:

1
2
3
4
 
Pod IP:      10.42.0.10
ClusterIP:   10.43.140.207
Node IP:     192.168.1.1
NodePort:    30080
 

Layer 1: Is the application actually running?

1
2
3
 
kubectl get pods -o wide
kubectl logs <pod-name>
kubectl exec <pod-name> -- wget -qO- http://127.0.0.1:80/
 

If localhost inside the Pod fails, the problem is the application/container, not Kubernetes networking.

Layer 2: Can the node reach a Pod directly?

1
 
curl -v --connect-timeout 5 http://10.42.0.10:80/
 

If this fails, investigate CNI/routing before looking at Services.

Layer 3: Can the ClusterIP Service be reached?

1
 
curl -v --connect-timeout 5 http://10.43.140.207:80/
 

Also verify Service endpoints:

1
2
3
 
kubectl get endpointslices \
  -l kubernetes.io/service-name=backend-svc \
  -o wide
 

If direct Pod access works but ClusterIP fails, investigate kube-proxy / Service rules.

Layer 4: Can the NodePort be reached?

1
 
curl -v --connect-timeout 5 http://192.168.1.1:30080/
 

If Pod and ClusterIP work but NodePort fails, investigate NodePort rules, eligible node addresses, and host firewall behavior.

Do not confuse ports

For:

1
 
80:30080/TCP
 

correct tests are:

1
2
3
 
curl http://<POD-IP>:80
curl http://<CLUSTER-IP>:80
curl http://<NODE-IP>:30080
 

These are not equivalent:

1
2
 
curl http://<POD-IP>:30080       # usually wrong
curl http://<CLUSTER-IP>:30080   # usually wrong
 

curl HTTP code 000

A command such as:

1
 
curl -s -o /dev/null -w '%{http_code}\n' http://192.168.1.1:30080/
 

returns 000 when curl never receives an HTTP response. During debugging, remove -s and use:

1
 
curl -v --connect-timeout 5 http://192.168.1.1:30080/
 

Distinguish:

• Connection refused
• Connection timed out
• Network is unreachable

They point to different layers.

Useful inspection commands

1
2
3
4
5
6
 
kubectl get pods -o wide
kubectl get svc -o wide
kubectl get endpointslices -o wide
kubectl get nodes -o wide
ip route
ip addr
 

For kube-proxy rules on an iptables-based cluster:

1
2
 
sudo iptables-save | grep KUBE
sudo iptables-save | grep 30080
 

Kubernetes Networking Theory

Review: NAT

Network Address Translation rewrites network-layer addressing as packets cross a boundary.

Common forms:

• SNAT: change the source address.
• DNAT: change the destination address.
• PAT / masquerading: combine address and port translation so multiple clients can share an address.

Example:

flowchart LR
    A["Host\n192.168.1.10"] -->|"Src=192.168.1.10"| SNAT["SNAT / masquerade"]
    SNAT -->|"Src=203.0.113.5"| S["Internet server"]
    S -->|"Dst=203.0.113.5"| DNAT["Connection tracking / reverse translation"]
    DNAT -->|"Dst=192.168.1.10"| A

In Kubernetes, NAT commonly appears in Service and egress handling. The Pod networking model itself aims to make Pod addresses directly meaningful within the cluster rather than placing every Pod behind a separate NAT boundary.

The Kubernetes Network Model

At a conceptual level, Kubernetes expects:

• Pods can communicate with other Pods across nodes.
• Nodes can communicate with Pods.
• A Pod sees its own IP as the same IP other cluster participants use to address it.
• Services provide stable virtual identities in front of changing Pod endpoints.

That model allows application developers to think in terms of:

1
 
frontend -> backend-svc
 

rather than:

1
 
frontend -> discover Pod -> inspect host -> translate address -> connect
 

How Services Work Under the Hood

A Service object does not normally create a process that listens on the ClusterIP.

Instead:

1. The Service selector identifies matching Pods.
2. Kubernetes maintains EndpointSlices containing backend addresses.
3. kube-proxy (or another implementation of Service routing) programs the node dataplane.
4. Packets addressed to the Service are redirected to one of the endpoints.

For a NodePort Service:

1
2
3
4
5
6
7
 
NodeIP:30080
      |
      v
Service virtual IP:80
      |
      v
PodIP:targetPort
 

Depending on the kube-proxy mode, rules may be implemented with iptables, nftables, or another dataplane mechanism.

Inspect an iptables-based node with:

1
2
3
 
sudo iptables-save | grep KUBE-SERVICES
sudo iptables-save | grep KUBE-NODEPORTS
sudo iptables-save | grep 30080
 

Do not depend on 127.0.0.1:<NodePort> as your primary NodePort test. Loopback NodePort support depends on kube-proxy mode and configuration. In this course, test the known FABRIC dataplane IP instead.

Container Network Interface (CNI)

Kubernetes delegates Pod network setup to CNI plugins.

When a Pod is created, the networking stack typically must:

1. create a network namespace;
2. create a veth pair;
3. place one end in the Pod namespace;
4. assign the Pod an IP address;
5. install routes;
6. connect the Pod to local and inter-node networking;
7. remove that configuration when the Pod is deleted.

A useful troubleshooting distinction is:

1
 
Pod is Running
 

versus:

1
 
Pod networking is actually functional
 

A container can be running while return routing, Service routing, or the overlay network is broken.

If Pod deletion hangs with an error such as:

1
2
3
 
KillPodSandboxError
failed to destroy network for sandbox
cni plugin not initialized
 

inspect CNI configuration and the CNI DaemonSet before assuming the workload itself is responsible.

RKE2 Canal

For the RKE2 clusters used in this course, Canal combines:

• Flannel for the inter-node overlay;
• Calico for workload networking and policy.

Useful commands:

1
2
3
 
kubectl -n kube-system get pods -o wide
ip link | grep -E 'cali|flannel'
ip route | grep 10.42
 

On a healthy two-node example:

1
2
 
node1 local Pod subnet:  10.42.0.0/24
node2 local Pod subnet:  10.42.1.0/24
 

Node1 may contain direct routes such as:

1
 
10.42.0.10 dev caliXXXXXXXX scope link
 

and a remote route such as:

1
 
10.42.1.0/24 via 10.42.1.0 dev flannel.1 onlink
 

The two route types represent different pieces of the path:

1
2
 
cali*      -> local Pod attachment
flannel.1  -> remote Pod subnet over the overlay
 

Popular CNI Implementations

• Flannel: simple overlay networking, commonly VXLAN.
• Calico: routed networking and network policy; may use several dataplane modes depending on configuration.
• Cilium: eBPF-based networking, observability, and policy.
• Canal: combines Flannel networking with Calico policy/workload integration; this is the default CNI in RKE2.

CAREERProduction networking work is structured troubleshooting

Service incidents often arrive as a vague report—“the application is down”— but the cause may be the process, Pod readiness, selector, EndpointSlice, DNS, Service dataplane, CNI route, NetworkPolicy, or external load balancer.

The official Kubernetes Debug Services guide follows the same layered approach used in this lecture: verify the Service, test DNS and IP connectivity, inspect EndpointSlices and Pods, and then examine the Service proxy implementation.

During an incident, record:

• the failing request and timestamp;
• each layer tested and its result;
• the first layer where expected behavior changes;
• the mitigation and its risk;
• the root cause and preventive follow-up.

Being able to produce that evidence is more valuable professionally than memorizing a list of networking commands.