I wanted to see if I could move the MCP server somewhere else. Somewhere I did not have to think about.

Nebius recently shipped something called Endpoints as part of their new serverless compute offering. You give it a container image and it runs the container for you. You get back a public URL. No server to set up, no drivers to install, no cluster to configure. I thought about my mcp-gpu-server project, where I built an MCP server that talks directly to NVIDIA hardware using NVML. That one has to run near the hardware. But most MCP servers do not. Most of them are just HTTP services. So why do they need to live on a machine I manage?

This is the story of what happened when I tried to run one on Nebius instead.

How the pieces fit together

The setup has two parts. On your Mac, Claude Desktop talks to a small local Python script called bridge.py over the stdio protocol that MCP uses. That script takes every tool call Claude makes and forwards it as an HTTP request to a container running on Nebius. That container then calls Nebius Token Factory to produce an embedding and sends the result back. The model itself, Qwen3-Embedding-8B, lives in Token Factory. My container is just a thin layer that receives requests and forwards them.

The machine I used

I did this on a Nebius GPU VM called nebius-tarantula, which I already had running for other work. Ubuntu, eu-north1 region. I checked two things before writing a single line of code.

python3 --version
Python 3.12.3

docker --version
Docker version 29.2.1, build a5c7197

Python 3.12 and Docker 29. That was enough to start.

Setting up the project

I created a separate directory and a virtual env. The reason for the virtual env is simple: I did not want whatever I install here to affect anything else running on the machine. This VM has other projects on it.

mkdir mcp-serverless && cd mcp-serverless
python3 -m venv venv
source venv/bin/activate
pip install fastapi uvicorn requests

FastAPI handles the HTTP routing. Uvicorn is the server that runs it. Requests is how the container calls Token Factory. That is the whole dependency list. There is no ML framework here because this server does not run any models. It receives a request, forwards it to Token Factory, and returns the result.

The server itself

The server needs three routes. The first one, /tools, is how MCP clients discover what a server can do. Without it, Claude has no idea what tools are available. The second, /call, is what gets hit when Claude actually wants to use a tool. The third, /health, is specifically for Nebius. When the container starts, Nebius pings /health before it routes any real traffic. If that route does not exist, the platform assumes the container is broken.

from fastapi import FastAPI
from pydantic import BaseModel
from typing import Any
import requests
import os
import time

app = FastAPI()

NEBIUS_API_KEY = os.getenv("NEBIUS_API_KEY", "")
NEBIUS_EMBED_URL = "https://api.studio.nebius.ai/v1/embeddings"

@app.get("/tools")
def list_tools():
    return {
        "tools": [
            {
                "name": "embed_text",
                "description": "Generate embeddings via Nebius Token Factory",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "text": {
                            "type": "string",
                            "description": "Text to embed"
                        }
                    },
                    "required": ["text"]
                }
            }
        ]
    }

class CallRequest(BaseModel):
    tool: str
    parameters: dict[str, Any]

@app.post("/call")
def call_tool(req: CallRequest):
    if req.tool == "embed_text":
        text = req.parameters.get("text", "")
        start = time.time()
        response = requests.post(
            NEBIUS_EMBED_URL,
            headers={
                "Authorization": f"Bearer {NEBIUS_API_KEY}",
                "Content-Type": "application/json"
            },
            json={
                "model": "Qwen/Qwen3-Embedding-8B",
                "input": text
            }
        )
        elapsed = time.time() - start
        data = response.json()
        embedding = data["data"][0]["embedding"]
        return {
            "tool": "embed_text",
            "result": {
                "embedding_dim": len(embedding),
                "first_5_values": embedding[:5],
                "latency_seconds": round(elapsed, 3)
            }
        }
    return {"error": f"Unknown tool: {req.tool}"}

@app.get("/health")
def health():
    return {"status": "ok"}

Notice that the API key comes from an env variable. It is never written into the code or baked into the container image. The image will end up on Docker Hub, and I do not want credentials there.

Before touching Docker I tested this directly on the VM. The reason is that debugging a broken container on a remote platform is much harder than debugging a broken script on a machine where you can see what is happening. Test first, containerize second.

export NEBIUS_API_KEY="your_token_factory_key"
nohup uvicorn server:app --host 0.0.0.0 --port 8000 &

I used nohup and the ampersand so the server runs in the background. That way I can keep using the same terminal to run curl.

curl http://localhost:8000/tools

The tool description came back. Then the actual embedding call:

curl -X POST http://localhost:8000/call \
  -H "Content-Type: application/json" \
  -d '{"tool": "embed_text", "parameters": {"text": "Merhaba dünya"}}'

4096-dimensional vector. 85ms. That 85ms is the floor for everything that follows. It is what a Token Factory call costs when you are already inside the Nebius network in the same region.

Packaging as a container

The Dockerfile is deliberately small. No GPU base image, no CUDA toolkit, no large ML libraries. This server does not need any of that. A slim Python image is everything it needs.

FROM python:3.12-slim

WORKDIR /app

COPY server.py .

RUN pip install fastapi uvicorn requests

EXPOSE 8000

CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8000"]

I built the image and tested it on port 8001 so it would not clash with the server already running on 8000.

docker build -t mcp-server .

docker run -d -p 8001:8000 \
  -e NEBIUS_API_KEY=$NEBIUS_API_KEY \
  --name mcp-test \
  mcp-server

curl http://localhost:8001/tools

The tool list came back from the container. The image works.

The Container Registry detour

My original plan was to push the image to Nebius Container Registry. I went to Storage in the Nebius console and created a registry called mcp-registry.

Then I tried to authenticate from the VM. The Nebius CLI was installed but had no config. Setting it up requires a service account and a PEM-encoded private key. I created the service account, added it to the editors group, generated a key, and then realized the key I got was an S3-compatible access key, not an IAM token. The registry needed a different kind of authentication.

At that point I switched to Docker Hub. Nebius Endpoints accepts images from Docker Hub without any extra setup, and I had already spent enough time on authentication.

docker login -u mesutoezdil
docker tag mcp-server mesutoezdil/mcp-server:latest
docker push mesutoezdil/mcp-server:latest

If you want to use Nebius Container Registry properly you can, once you have the Nebius CLI configured with a service account key. For this experiment I moved on.

Creating the endpoint

In the Nebius console, AI Services in the left sidebar, then Endpoints. The page was empty.

I clicked Configure it yourself. The quick start option pre-fills an nginx image and does not show env variable settings until later in the flow. I needed to set the API key from the start, so the manual form made more sense.

The form has several fields. For name I typed mcp-server-endpoint. For image path I typed docker.io/mesutoezdil/mcp-server:latest. For port I changed the default 8080 to 8000, which is the port uvicorn listens on inside the container.

Under env variables I added NEBIUS_API_KEY with the Token Factory key as the value. This is what the container reads when it starts up.

The default compute selection is GPU, which costs around 1.59 per hour. This server does not run any models locally, so GPU is waste of money. I clicked Without GPU, selected Non-GPU AMD Epyc Genoa, and chose 4 vCPUs with 16 GiB of RAM. Cost: 0.14 per hour.

I clicked Create. The status showed Provisioning. About two minutes later it changed to Running and a public endpoint address appeared in the Network section: 89.xxx.yyy.cc:8000.

This was the moment that made the whole thing real. A container I had built on a VM was now running on a platform I did not configure, reachable from the public internet, and I had not touched a single network setting.

I tested it immediately from the VM:

curl http://89.169.111.36:8000/tools

Tool list. Then from my Mac:

curl -X POST http://89.169.111.36:8000/call \
  -H "Content-Type: application/json" \
  -d '{"tool": "embed_text", "parameters": {"text": "Hello from Nebius Serverless MCP"}}'

4096 dimensions. 139ms from Germany to eu-north1 and back. The endpoint was working.

The bridge

Claude Desktop speaks MCP over stdio, which is a local process protocol. The Nebius Endpoint speaks HTTP. They cannot talk to each other directly, so I wrote a bridge script that runs locally on my Mac. It speaks stdio to Claude and HTTP to the endpoint. One small script that translates between the two.

This is the same kind of separation I keep seeing in HAMi work. One layer decides, another layer applies. Here one layer translates, another layer executes.

My Mac had Python 3.9. The MCP library needs 3.10 or above, so I installed 3.11 first.

brew install python@3.11
python3.11 -m venv ~/mcp-bridge-env
source ~/mcp-bridge-env/bin/activate
pip install mcp requests

Then the bridge script at ~/mcp-bridge-env/bridge.py:

import asyncio
import requests
from mcp.server import Server
from mcp.server.stdio import stdio_server
from mcp import types

NEBIUS_ENDPOINT = "http://89.169.111.36:8000"

app = Server("nebius-mcp-bridge")

@app.list_tools()
async def list_tools() -> list[types.Tool]:
    return [
        types.Tool(
            name="embed_text",
            description="Generate embeddings via Nebius Serverless Endpoint (Token Factory)",
            inputSchema={
                "type": "object",
                "properties": {
                    "text": {"type": "string", "description": "Text to embed"}
                },
                "required": ["text"]
            }
        )
    ]

@app.call_tool()
async def call_tool(name: str, arguments: dict) -> list[types.TextContent]:
    if name == "embed_text":
        response = requests.post(
            f"{NEBIUS_ENDPOINT}/call",
            json={"tool": "embed_text", "parameters": arguments}
        )
        result = response.json()
        return [types.TextContent(type="text", text=str(result))]
    raise ValueError(f"Unknown tool: {name}")

async def main():
    async with stdio_server() as (read, write):
        await app.run(read, write, app.create_initialization_options())

if __name__ == "__main__":
    asyncio.run(main())

To tell Claude Desktop about this script I edited ~/Library/Application Support/Claude/claude_desktop_config.json and added the mcpServers section. The preferences that were already in the file stayed untouched.

{
  "mcpServers": {
    "nebius-serverless": {
      "command": "/Users/mesutoezdil/mcp-bridge-env/bin/python3",
      "args": ["/Users/mesutoezdil/mcp-bridge-env/bridge.py"]
    }
  },
  "preferences": {
    "coworkScheduledTasksEnabled": true,
    "ccdScheduledTasksEnabled": true
  }
}

Then I restarted Claude Desktop so it would pick up the new config.

pkill -a Claude && sleep 2 && open -a Claude

I checked the log file to confirm the server had loaded correctly.

cat ~/Library/Logs/Claude/mcp-server-nebius-serverless.log

The log showed the bridge initializing, completing a handshake with Claude, and responding to a tools/list request with the embed_text tool. Connected, no errors.

Claude calls the tool

I opened a new chat in Claude Desktop and typed one message.

Use the embed_text tool to embed this sentence: Nebius serverless MCP is working

Claude called the tool. No prompting, no extra config.

4096 dimensions. 564ms end to end.

That 564ms covered the MCP protocol on my Mac, the bridge script, the network from Germany to eu-north1, the container receiving the request, the Token Factory call, and everything back. For a tool running on a server I never configured, on a platform that handled all the infrastructure, that is a number I am comfortable with.

What the numbers mean

The Token Factory call from inside the Nebius VM was 85ms. From my laptop to the endpoint and back was 139ms. The full Claude Desktop round trip was 564ms. The difference between 139ms and 564ms is the MCP protocol and bridge overhead. The Nebius infrastructure itself is fast.

The endpoint costs 0.14 per hour. You stop it from the console when you do not need it. There is no state to worry about, no teardown procedure.

What this actually changes

In my HAMi work I keep coming back to the same point. The problem in GPU scheduling is not the scheduler itself. It is how the GPU is modeled in the first place. You cannot fix a modeling problem by tuning a scheduler. The model has to change.

The same idea applies here. Most MCP tool work focuses on what the tool does. But underneath that question is another one that rarely gets asked: where does the tool live, and who keeps it alive? If that answer is always a server you manage, it puts a ceiling on what you can build with MCP. Every tool becomes infrastructure debt.

Running the server on Nebius Endpoints lifts that ceiling. I still write the tool. I still decide what it does. I just do not manage the machine anymore.

The one thing I would add before using this in production is authentication. The endpoint URL is currently public. Nebius Endpoints has a token authentication toggle in the settings. One click, then one extra header in the bridge script.

The code from this article is at github.com/mesutoezdil/mcp-serverless. Server, Dockerfile, bridge script, all there. If you hit the Container Registry authentication wall, use Docker Hub instead.

The missing bit was a translator. HAMi already publishes everything I need over Prometheus on port 31993. An LLM does not speak Prometheus. So I wrote a small server in Go that sits in between and speaks the Model Context Protocol. The full source, with build and run instructions, is in my repo.

My GitHub

This article walks through what I built, what bit me on the way, and how to reproduce it on your own cluster.

Why I bother

It is curiosity, mostly, plus an old habit. I keep finding pairs of projects that do not know about each other yet, and I want to see what happens when they meet. HAMi was already on the cluster I was using. MCP was on every other tab in my browser this month. Adapting the one to the other made for a better afternoon than reading another tutorial on either.

The work is mostly trial and error. The bugs along the way are not a tax I paid, they are half the reason I write any of this down.

Before diving into the details, I also reproduced the same setup on Nebius GPU instances to validate how this behaves in a cloud environment. The goal was to compare my on-prem NVIDIA L40S setup with an on-demand GPU instance and verify that provisioning, NVIDIA runtime configuration, and GPU observability behave consistently without additional adjustments. The instance was ready within minutes with drivers and runtime support available out of the box, allowing me to follow the exact same steps and confirm consistent behavior across environments.

What we are building, in one picture

Two clients touch HAMi in this post. One is a Go program that runs the chain end to end from the command line. The other is MCP Inspector, an official browser based debugger for MCP servers. They share the same hami-mcp-server binary in the middle and they read from the same :31993/metrics endpoint that HAMi already exposes. The shape looks like this.

Subscribe now

The Go binary in the middle is the same in both paths. The thing that changes is who launches it. In the command line path it is our own test client. In the browser path it is MCP Inspector. The protocol it speaks (stdio JSON-RPC) does not change either way.

Why two paths. The command line one is the proof that an LLM can read HAMi without anyone wiring it by hand. The browser one is the proof that the server is a generic MCP server, useful to tools we did not write.

What was on the box

I started by checking the GPU and the HAMi pods. The first command asks the NVIDIA driver for free and total memory on every GPU.

$ nvidia-smi --query-gpu=name,memory.free,memory.total --format=csv
name, memory.free [MiB], memory.total [MiB]
NVIDIA L40S, 45458 MiB, 46068 MiB

One L40S, 46 GiB total, 45 GiB free. Nothing was using the card.

The second command lists k8s pods in the system namespace and filters to the ones HAMi runs.

$ kgp -n kube-system | grep hami

hami-device-plugin-5n2mv           2/2     Running   0   15m
hami-scheduler-7bbbfc4f7f-d6n29    2/2     Running   2   40h

Two pods, both up. The device plugin is the one that exposes the metrics endpoint we are about to scrape.

The first look at HAMi metrics

This command pulls the raw Prometheus text from the device plugin and prints the first ten lines.

$ curl -s http://localhost:31993/metrics | head -10
# HELP GPUDeviceCoreAllocated Device core allocated for a certain GPU
# TYPE GPUDeviceCoreAllocated gauge
GPUDeviceCoreAllocated{deviceidx=”0”,deviceuuid=”GPU-4cdfc14a...”,nodeid=”nebius-tarantula”,zone=”vGPU”} 0
# HELP GPUDeviceCoreLimit Device memory core limit for a certain GPU
# TYPE GPUDeviceCoreLimit gauge
GPUDeviceCoreLimit{deviceidx=”0”,deviceuuid=”GPU-4cdfc14a...”,nodeid=”nebius-tarantula”,zone=”vGPU”} 100

Every line is a metric. The names are reasonable once you read them twice. There are about eight metric families on a quiet cluster. The labels are where things get sneaky, but I will come back to that.

If your cluster is not local, replace localhost:31993 with the right host or run kubectl port-forward against the device plugin pod.

The four “tools”

I wanted four tool calls.

(i) A cluster summary. How many nodes, how many GPUs, how much vGPU memory is checked out, how much is free, who is using the most.

(ii) Per device numbers. The same data broken down by GPU, with an optional node filter.

(iii) Per pod attribution. Which pod is sitting on which GPU and how much of it.

(vi) A free form metric query. Give me one metric with a label filter.

Each tool is just a Go function. It scrapes the HAMi endpoint, parses the Prometheus text, returns JSON. The MCP library on top handles the JSON-RPC framing over stdio. That is the whole architecture.

The full implementation is in main.go in the repo. Around 500 lines, four handlers, no caching.

Two bugs that bit me

I did not get this right on the first try. Two things are worth flagging because they will likely bite anyone else who tries.

The Prometheus parser panicked on a clean construction

I wrote the obvious code. Declare a TextParser, hand it the response body, get back metric families. On the first call, the program panicked with: Invalid name validation scheme requested: unset.

The cause is in prometheus/common v0.67. The parser keeps a private validation scheme that defaults to a useless zero value. There is a global called model.NameValidationScheme that looks like it would help. It does not, the parser does not read it.

The fix is a one liner. Build the parser through expfmt.NewTextParser(model.UTF8Validation) instead of zero initialising it. The repo has the working version in main.go near the top of scrape() in main.go.

Two label families that look the same but are not

This one was sneakier. The first version of my cluster summary keyed devices on (nodeid, deviceuuid). Made sense. Then I deployed a pod that asked HAMi for a vGPU, scraped again, and the device count doubled.

"device_count": 2,
"devices": [
  {"node": "nebius-tarantula", "device_uuid": "GPU-4cdfc14a...", "memory_allocated_mb": 8192},
  {"node": "",                 "device_uuid": "GPU-4cdfc14a...", "memory_allocated_mb": 0}
]

The phantom device has an empty node. That was the clue. When I diffed the raw metrics from before and after the pod landed, two new series had appeared.

vGPUCoreAllocated{containeridx="0",deviceuuid="...",nodename="nebius-tarantula",podname="hami-demo-workload",...} 30
vGPUMemoryAllocated{containeridx="0",deviceuuid="...",nodename="nebius-tarantula",podname="hami-demo-workload",...} 8.589934592e+09

Look at the label keys. The device level metrics use nodeid. The new per pod metrics use nodename. Same logical thing, different key. My code read s.Labels["nodeid"] against a per pod sample, got an empty string, and the empty string went into the dedup tuple as a new device.

The fix is small. Whitelist the metric names that the cluster summary cares about, and let the per pod metrics flow through to a different tool. The whitelist and the rest of the fix are in handleGetClusterSummary in main.go.

The lesson I will keep. When a Prometheus exporter speaks two label dialects, never write a single dedup key that spans both.

A bonus, the per pod series was also the first place I realised HAMi already publishes pod attribution. The pod label is what an operator actually wants when something is melting on a shared GPU.

A workload to make the metrics interesting

A clean cluster reports zero everywhere. To see the tools do real work, you need a pod that holds a HAMi reservation. The repo has a manifest at k8s-vgpu-workload.yaml that requests one GPU, 8 GiB of memory, and 30 percent of the cores, then sleeps for half an hour.

The first command tells Kubernetes to create the pod from that manifest. The second command checks that the pod is actually running.

$ k apply -f k8s-vgpu-workload.yaml
pod/hami-demo-workload created

$ kgp hami-demo-workload
NAME                 READY   STATUS    RESTARTS   AGE
hami-demo-workload   1/1     Running   0          8s

Now check what HAMi recorded on the pod. The next command prints the pod’s full spec and filters to the lines starting with the HAMi annotation prefix.

$ k describe pod hami-demo-workload | grep hami.io
hami.io/bind-phase:              success
hami.io/vgpu-devices-allocated:  GPU-4cdfc14a-c633-e61e-9235-56118c547d80,NVIDIA,8192,30:;
hami.io/vgpu-node:               nebius-tarantula

The middle line is the one with the most information in it. Reading the comma separated fields, HAMi allocated GPU UUID GPU-4cdfc14a... of type NVIDIA, with 8192 MiB of memory and 30 percent of cores. The annotation is how HAMi keeps the device plugin and the scheduler in sync.

When you are done with the pod, remove it with k delete -f k8s-vgpu-workload.yaml.

Hand testing one tool

The repo includes a small shell helper at test_stdio.sh. It sends the MCP initialise handshake, the notifications/initialized follow up, and one tools/call, then prints the JSON-RPC responses on stdout.

The next command runs the helper for get_cluster_summary, takes the last response line, and uses jq to pull out the human readable text payload.

$ ./test_stdio.sh ./hami-mcp-server get_cluster_summary '{}' \
    | tail -1 \
    | jq -r '.result.content[0].text'

{
  "core_utilization_pct": 30,
  "device_count": 1,
  "devices": [
    {
      "node": "nebius-tarantula",
      "device_uuid": "GPU-4cdfc14a-c633-e61e-9235-8888888888",
      "device_type": "NVIDIA L40S",
      "memory_allocated_mb": 8192,
      "memory_limit_mb": 46068,
      "memory_percent": 0.17782408613354173,
      "core_allocated": 30,
      "core_limit": 100,
      "shared_containers": 1
    }
  ],
  "hami_build_date": "20260417-04:22:41",
  "hami_version": "v2.8.1",
  "memory_utilization_pct": 17.78240861335417,
  "node_count": 1,
  "total_memory_allocated_mb": 8192,
  "total_memory_free_mb": 37876,
  "total_memory_limit_mb": 46068,
  "total_shared_containers": 1
}

Numbers match the demo pod request. 8 GiB out of 46, 30 percent of the cores, one shared container. HAMi version is right there in the JSON, which is handy when you need to file a bug report.

Picking a local LLM

I started with vLLM and NVIDIA’s Nemotron Nano 8B. Two problems showed up before I got the image on disk. First, vllm/vllm-openai is around 10 GiB compressed, and decompresses to about twice that. The host had 17 GiB free. Second, even if it had fit, the L40S is now sliced by HAMi, and a second inference engine running outside HAMi accounting needs --gpu-memory-utilization trimmed to leave room for the demo pod’s reservation.

I switched to Ollama with llama3.2:3b quantised to Q4. Image around 6 GiB, model around 2 GiB. Ollama exposes an OpenAI compatible chat completions endpoint, so the test client does not care which engine is on the other side.

The next command starts Ollama as a Docker container with full GPU access. We bind it to 127.0.0.1:11434 on purpose. Ollama has no authentication, so a public bind would let any internet host call your local model. The e2e client only needs to reach Ollama from the same host, so localhost is fine.

$ docker run -d --name ollama --gpus all --network host -e OLLAMA_HOST=127.0.0.1:11434 -v ollama-models:/root/.ollama ollama/ollama:latest

The hex string is the new container ID. Docker prints it because we ran with -d (detached). The next command tells the running container to pull the model weights.

$ docker exec ollama ollama pull llama3.2:3b
pulling dde5aa3fc5ff: 100%  2.0 GB
...
success

The 2.0 GB layer is the model weights. The smaller blobs are the tokenizer, the chat template, and so on. After success, Ollama has the model on disk and is ready to serve it.

The last command asks the OpenAI compatible endpoint which models it can serve, to confirm the pull worked.

$ curl -s http://localhost:11434/v1/models
{"object":"list","data":[{"id":"llama3.2:3b","object":"model",...}]}

The Ollama logs confirm it sees the L40S directly through the NVIDIA container runtime, with the full 45 GiB available. That is because Docker bypassed HAMi entirely. HAMi only accounts for what the Kubernetes scheduler hands out. The demo pod that requested 8 GiB shows up in HAMi metrics. The Ollama container does not. That is correct behaviour, but it confuses people the first time.

The end to end run

The repo has a small Go program at cmd/e2e/main.go that launches hami-mcp-server as a subprocess, performs the MCP handshake, calls two tools, builds a prompt out of their JSON, and asks the LLM to summarise. Run it.

$ ./e2e --server ./hami-mcp-server --llm-url http://localhost:11434 --model llama3.2:3b

==> Starting hami-mcp-server
==> tools/call get_cluster_summary
{
  "core_utilization_pct": 30,
  "device_count": 1,
  "devices": [...],
  "memory_utilization_pct": 17.78240861335417,
  "total_memory_allocated_mb": 8192,
  "total_memory_free_mb": 37876,
  "total_memory_limit_mb": 46068
}

==> tools/call get_vgpu_allocation
{
  "allocations": [
    {
      "namespace": "default",
      "pod_name": "hami-demo-workload",
      "node": "nebius-tarantula",
      "device_uuid": "GPU-4cdfc14a-c633-e61e-9235-7878787878787878",
      "container_index": "0",
      "memory_allocated_mb": 8192,
      "core_allocated": 30
    }
  ],
  "match_count": 1
}

==> Asking llama3.2:3b at http://localhost:11434

==> Model answer
The HAMi cluster has a utilization rate of 30% for its single NVIDIA L40S
device, with 30 cores allocated out of a limit of 100. The total memory
allocation is relatively low at 8192 MB, leaving 37876 MB free. There are
no concerns based on the provided metrics, as the utilization rates and
memory allocations appear to be within expected ranges for this type of
GPU. However, it's worth noting that the cluster has only one device,
which may limit its scalability. Overall, the cluster appears to be
running within a healthy range.

The numbers in the answer match the numbers in the JSON. The scalability comment is fair, a real on call engineer would say the same thing.

I did not write a special prompt to coax this out. The JSON had clear field names, and a 3B Q4 model was enough to write something readable on top of it.

The same plumbing, this time in a browser

The end to end Go client is a good smoke test, but it bakes in one LLM and one prompt. For day to day debugging you want to poke at the tools by hand and see the raw JSON. MCP Inspector is the official browser based MCP debugger that lets you do exactly that. It is a small Node.js app. You point it at the binary, it opens a browser tab, every tool gets a button you can click.

The next command starts Inspector and points it at our binary. HOST=0.0.0.0 makes it listen on every network interface, which is what lets a browser on a different machine reach it. ALLOWED_ORIGINS is its CORS allow list, the URL the browser loads from. MCP_AUTO_OPEN_ENABLED=false skips the local browser auto open, since this VM has no graphical session. npx -y downloads the package and runs it without asking for confirmation.

A note before you run this. Binding Inspector to 0.0.0.0 puts a debug tool on the public internet. Token authentication is on by default and it helps, but a real deployment should sit behind a VPN or an SSH tunnel. The form below is fine for a one off demo on a throwaway VM.

$ HOST=0.0.0.0 \
  ALLOWED_ORIGINS="http://YOUR_VM_IP:6274" \
  MCP_AUTO_OPEN_ENABLED=false \
  npx -y @modelcontextprotocol/inspector ./hami-mcp-server

Starting MCP inspector...
✓ Proxy server listening on 0.0.0.0:6277
✓ Session token: <64 hex chars, regenerated on every restart>
✓ MCP Inspector is up and running at:
   http://0.0.0.0:6274/?MCP_PROXY_AUTH_TOKEN=<token>

(You do not think I’m sharing my personal info here, right?)

Two ports, one role each. 6274 is the React UI the browser loads. 6277 is the proxy the UI then calls into. The proxy spawns hami-mcp-server as a subprocess and talks to it over stdio. The session token is a one shot secret printed on startup, treat it like a password and do not paste it into a public chat.

Open the printed URL in your browser, with your VM’s IP in place of 0.0.0.0. You see a form with a Command field and an Arguments field. Paste the absolute path of the binary into Command, leave Arguments empty, click Connect. The right hand pane lists the four tools the server registers.

Click on get_cluster_summary, leave the metadata pairs empty, click Run Tool. Inspector forwards a JSON-RPC tools/call down to the binary. The binary scrapes HAMi and returns JSON. The panel renders it. With our demo pod running, the result has memory_allocated_mb: 8192, core_allocated: 30, shared_containers: 1. Run k delete pod hami-demo-workload, click Run Tool again, and every number goes back to zero.

get_vgpu_allocation is the more interesting one. With the pod running it returns a single allocation entry. The entry names the namespace, the pod, the device UUID, the 8 GiB of memory, and the 30 percent of cores. Pass pod_name: hami-demo-workload to the call and the same row comes back. Pass a name that does not exist and you get an empty list with a note saying no matching pod has an active vGPU reservation.

run_promql lets you write a metric name plus optional label matchers. Put nodeGPUMemoryPercentage{nodeid="nebius-tarantula"} into the query field and the response is one sample with value: 0.17782..., the same 17.78 percent the cluster summary reports, just with more decimal places.

What this exercise proves is small but useful. The MCP server we wrote is not tied to one client. Whatever MCP capable tool you point at this binary will see the same four tools and the same JSON shapes.

I work this way on purpose. Adapting two real systems to play together is more interesting to me than building greenfield, and a real bug story is more useful than a marketing pitch. The post you just read is one of those exercises. The rest of them are at github.com/mesutoezdil.

Subscribe now

Closing

Any MCP capable client can now read this cluster. The clients do not need to know how HAMi works, they just call get_cluster_summary or one of the other three and get back small JSON.

On a one node home lab this is more machinery than the problem deserves. The value shows up when you have several GPU nodes shared across teams and you want a uniform way to ask about them. The server itself is around 500 lines of Go. The hard parts were the two label families and the parser footgun. The rest was mechanical.

You wait weeks for review and it can still get closed. Doing a tiny cleanup feels too small. The maintainer might reply “we didn’t need this” and shut it.

So where’s the right middle?

I’ll talk about this through two small refactor PRs I opened to modelcontextprotocol/registry.

Both are still in review as I write this. Whichever way they go, I wanted to write down why I opened them and where their limits are.

PR #1240: https://github.com/modelcontextprotocol/registry/pull/1240

PR #1241: https://github.com/modelcontextprotocol/registry/pull/1241

First PR: Delete One Line

Inside internal/database/postgres.go there was this line in ListServers:

_ = argIndex // Silence unused variable warning

This is a classic Go pattern. You assign a value to argIndex somewhere and then don’t use it later. Go compiler complains “unused variable”.

To silence the linter you write _ = argIndex and move on.

The thing is, argIndex is already used above (for cursor pagination). It only looks “unused” because the last increment doesn’t go anywhere.

So the warning suppression isn’t needed. The variable does get used. Just the final assignment is dead.

The fix is one line:

        whereConditions = append(whereConditions, cursorCondition)
        args = append(args, cursorArgs...)
    }
-   _ = argIndex // Silence unused variable warning

    // Build the WHERE clause
    whereClause := ""

The diff:

$ gh pr diff 1240 --repo modelcontextprotocol/registry

 1 file changed, 1 deletion(-)

What’s the risk here? The maintainer might say “we wanted to keep that comment around for context” and close it. Or maybe they’re planning to use argIndex for a future flow I don’t know about. I see it as dead in the current code but I don’t know the history.

Second PR: Pull a Repeated Pattern Into a Helper

In the auth handlers I noticed the same pattern in three different files:

if resp.StatusCode != http.StatusOK {
    body, _ := io.ReadAll(resp.Body)
    return nil, fmt.Errorf("GitHub API error (status %d): %s", resp.StatusCode, body)
}

Same exact thing in three places. Twice in github_at.go (getGitHubUser, getGitHubUserOrgs). Once in github_oidc.go (fetchJWKS).

All three read the response body, swallow the error, and stuff the body into the error message.

I added a helper to common.go:

func readBody(r io.Reader) string {
    b, _ := io.ReadAll(r)
    return string(b)
}

Then I cut each call site down to one line:

 if resp.StatusCode != http.StatusOK {
-    body, _ := io.ReadAll(resp.Body)
-    return nil, fmt.Errorf("GitHub API error (status %d): %s", resp.StatusCode, body)
+    return nil, fmt.Errorf("GitHub API error (status %d): %s", resp.StatusCode, readBody(resp.Body))
 }

I also removed the "io" import from all three files. The package owns it through common.go now. Diff:

$ gh pr diff 1241 --repo modelcontextprotocol/registry

 4 files changed, 8 insertions(+), 9 deletions(-)

This Helper Has a Known Weakness

readBody swallows the error. b, _ := io.ReadAll(r). If the body can’t be read (broken connection, malformed response), b ends up empty and the error gets lost.

I left it that way on purpose because the original code did the same thing.

The PR is a refactor, not a behavior change. But the “right” version would return (string, error) and make every caller deal with it:

func readBody(r io.Reader) (string, error) {
    b, err := io.ReadAll(r)
    if err != nil {
        return "", err
    }
    return string(b), nil
}

I didn’t do that because the scope would balloon. You’d have to add error handling at three call sites. That stops being a small cleanup.

If a maintainer asks for it in review I’ll do it as a follow-up PR.

For now the PR keeps the existing behavior and just trims the duplicate lines.

This is a trade-off you make on every PR like this. Keep it small and you have a better chance of merge but it’s not “fully right”.

Make it bigger and it’s right but a maintainer might call it scope creep and close it. You weigh this for each one.

What Does This Look Like to a Maintainer?

I learned these the hard way over a few merged and a few rejected PRs. 2 ways it can go.

The good way. “Clean cleanup, code reads a bit better, merging.” Single-line fixes like #1240 and dedup helpers like #1241 usually get through. Especially if the change really doesn’t break behavior and there’s no big counter-argument.

The bad way. “Why does this PR exist? Did we need a review cycle for one line?” Some maintainers see PRs this small as noise. If the repo isn’t very active or the maintainer has no bandwidth, small PRs sit in a queue and rot.

I don’t know which way these will go. As I write this both are in “Review required”. If they get closed, that’s also useful. If they get merged, the repo is a tiny bit cleaner.

When Does This Kind of PR Make Sense?

A few conditions.

If you’re learning the repo. A small PR forces you to read the codebase. What patterns are here, what’s the lint config, how are tests written, what’s the naming style. That work has value even if the PR doesn’t merge.

If you have a bigger PR coming. A small PR introduces you to the maintainer. A week later when a real feature PR shows up, “oh this person contributed before” is a useful memory for them.

If there’s actually a meaningful cleanup. “There’s a typo here” is meaningful. “There’s an unused line” is borderline. “Let’s reformat every line in the file” is not.

If the repo is active. If they merge PRs every week, your small PR has a fair shot. If it’s a half-dead repo, you’ll wait a long time and probably waste it.

modelcontextprotocol/registry is active. There are weekly merges. The duplicate auth pattern was really sitting there. When I opened the PRs they felt in tune with the way the repo moves. Even so, the result is up to review.

Both PRs are still open. Whichever way they go, I’ll update this post.

Most MCP tutorials stop at the config file. I wanted to see the HTTP requests, the JSON-RPC calls, the session lifecycle. So I dug in.

MCP is everywhere now. Anthropic launched it, everyone is building servers. But when you read the docs, you get quickstart guides. Install this, configure that, done.

What none of them tell you: how does the session actually start? Where does the session ID come from? What does a real tool call look like on the wire?

I needed to know this because I work with k8s. If I am going to run MCP servers in prod, I need to understand what is actually happening. Not just it works on my laptop.

The Setup

I used Nebius AI Cloud. They have GPU instances (L40S with 46GB VRAM). I did not actually use the GPU for this test. kagent runs fine on CPU. But in prod, if your MCP tools need to run image generation or video analysis, you will need GPU scheduling.

For multi-tenant GPU sharing, I contribute to Project HAMi (a CNCF sandbox project). It splits GPUs across pods. Different topic, but worth mentioning if you are thinking about AI workloads on k8s. The VM: 8 vCPUs, 32GB RAM, Ubuntu 24.04. Fresh install.

Installing k3s

k3s is lightweight k8s. One command, 15 seconds:

curl -sfL https://get.k3s.io | sh -

From my terminal:

[INFO]  systemd: Starting k3s
real    0m15.794s

I picked k3s because a 5-node cluster is overkill here. It gives me a working k8s API in under 20 seconds.

Installing kagent

kagent is a k8s-native agent framework that includes MCP servers. Version 0.9.0 just came out. First, the CRDs:

time helm install kagent-crds \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \
  --namespace kagent \
  --create-namespace

From my terminal:

Pulled: ghcr.io/kagent-dev/kagent/helm/kagent-crds:0.9.0
NAME: kagent-crds
LAST DEPLOYED: Sat Apr 25 15:53:23 2026
NAMESPACE: kagent
STATUS: deployed
REVISION: 1
real    0m5.476s

Check what got installed:

k get crds | grep kagent.dev

Output:

agents.kagent.dev                              2026-04-25T15:53:24Z
mcpservers.kagent.dev                          2026-04-25T15:53:24Z
memories.kagent.dev                            2026-04-25T15:53:24Z
modelconfigs.kagent.dev                        2026-04-25T15:53:24Z
modelproviderconfigs.kagent.dev                2026-04-25T15:53:24Z
remotemcpservers.kagent.dev                    2026-04-25T15:53:24Z
sandboxagents.kagent.dev                       2026-04-25T15:53:24Z
toolservers.kagent.dev                         2026-04-25T15:53:24Z

Eight CRDs. One new CRD in 0.9.0 is sandboxagents.kagent.dev. I have not dug into what that does yet, but it is new since the last version.

Then the main kagent install:

export NEBIUS_API_KEY="your-api-key-here"

time helm install kagent \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --namespace kagent \
  --set providers.default=openAI \
  --set providers.openAI.apiKey=$NEBIUS_API_KEY \
  --set providers.openAI.endpoint=https://api.studio.nebius.ai/v1 \
  --set providers.openAI.model=meta-llama/Llama-3.3-70B-Instruct \
  --wait --timeout=10m

I am using Nebius Token Factory (OpenAI-compatible API) with Llama 3.3 70B. Why not OpenAI directly? Because I wanted to test with an open model, and Nebius pricing is good ($2.56 for this entire test session).

The install timed out at 10 minutes:

Error: INSTALLATION FAILED: context deadline exceeded
real    10m2.775s

This is not actually a failure. Helm’s --wait flag expects all pods to be Ready. One pod (kmcp-controller-manager) was still pulling its image. After another minute, everything was running.

Check the pods:

kgp -n kagent

Output (17 pods total):

NAME                                              READY   STATUS    RESTARTS   AGE
argo-rollouts-conversion-agent-985c94668-4zm2t    1/1     Running   0          13m
cilium-debug-agent-5bdfbbf8b5-8qw5z               1/1     Running   0          13m
cilium-manager-agent-7c85d7b7df-9qrlh             1/1     Running   0          13m
cilium-policy-agent-5dd67dd9fc-skf4m              1/1     Running   0          13m
helm-agent-589bf45db8-j48tr                       1/1     Running   0          13m
istio-agent-6b44cd4fd-vcd9r                       1/1     Running   0          13m
k8s-agent-895864cb6-756sb                         1/1     Running   0          13m
kagent-controller-87fb569dc-wv48f                 1/1     Running   4          26m
kagent-grafana-mcp-dc4d9d79d-4f8tx                1/1     Running   0          26m
kagent-kmcp-controller-manager-777746db7c-kvcpx   1/1     Running   0          26m
kagent-postgresql-68f97986df-g9xqs                1/1     Running   0          26m
kagent-querydoc-7dbd595b58-db4cb                  1/1     Running   0          26m
kagent-tools-77c6f575c8-t7cwt                     1/1     Running   0          26m
kagent-ui-df66c64bd-pzht2                         1/1     Running   0          26m
kgateway-agent-5cc8d7fdc6-sfx9d                   1/1     Running   0          13m
observability-agent-78c9d59d54-5kck2              1/1     Running   0          13m
promql-agent-7685b45dc-h9cps                      1/1     Running   0          13m

All Running. Good.

The Manual Patch Nobody Tells You About

Something the docs do not mention. The Helm parameter --set providers.openAI.endpoint creates the API key secret correctly. But it does not populate the baseUrl field in the ModelConfig CRD.

Check it:

k get modelconfig default-model-config -n kagent -o yaml

You will see:

spec:
  apiKeySecret: kagent-openai
  apiKeySecretKey: OPENAI_API_KEY
  model: meta-llama/Llama-3.3-70B-Instruct
  provider: OpenAI

No openAI.baseUrl field. That means kagent will try to call https://api.openai.com/v1 instead of Nebius.

The fix:

k patch modelconfig default-model-config -n kagent --type=merge -p '{
  "spec": {
    "openAI": {
      "baseUrl": "https://api.studio.nebius.ai/v1"
    }
  }
}'

Output:

modelconfig.kagent.dev/default-model-config patched

Check again:

k get modelconfig default-model-config -n kagent -o jsonpath='{.spec.openAI.baseUrl}'

Output:

https://api.studio.nebius.ai/v1

Good. This is a known limitation in kagent 0.9.0. The Helm chart sets the secret but not the CRD field. You need the manual patch. I am mentioning this because if you deploy kagent with a non-OpenAI provider, you will hit this.

The MCP Wire Protocol

This is the part nobody writes about. What does MCP look like on the wire?

Here is a simple diagram showing the flow:

I port-forwarded the kagent-tools service to my laptop:

k port-forward -n kagent svc/kagent-tools 8084:8084 --address 127.0.0.1 &

kagent-tools is an MCP server. It exposes 124 k8s-related tools (we will get to that).

Step 1 > Initialize a Session

MCP uses JSON-RPC 2.0. First request: initialize.

curl -i -X POST http://localhost:8084/mcp \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "initialize",
    "params": {
      "protocolVersion": "2024-11-05",
      "capabilities": {},
      "clientInfo": {"name": "test", "version": "1.0"}
    }
  }'

The response from my terminal:

HTTP/1.1 200 OK
Content-Type: application/json
Mcp-Session-Id: mcp-session-bed70407-ca06-4689-a2a6-e1ab213add6a
Date: Sat, 25 Apr 2026 16:19:30 GMT
Content-Length: 175

{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":"2024-11-05","capabilities":{"tools":{"listChanged":true}},"serverInfo":{"name":"kagent-tools-server","version":"0.1.3"}}}

Look at the header: Mcp-Session-Id.

The session ID is in an HTTP header. Not in the JSON response. This matters because if you are building an MCP client, you cannot just parse the JSON and expect to find the session ID. You need to read headers.

The MCP docs mention sessions, but they do not show you the actual HTTP exchange.

Step 2 > List Available Tools

Now that I have a session ID, I can call tools/list:

curl -s -X POST http://localhost:8084/mcp \
  -H "Content-Type: application/json" \
  -H "Mcp-Session-Id: mcp-session-bed70407-ca06-4689-a2a6-e1ab213add6a" \
  -d '{
    "jsonrpc": "2.0",
    "id": 2,
    "method": "tools/list",
    "params": {}
  }' > /tmp/tools-list.json

cat /tmp/tools-list.json | jq '.result.tools | length'

Output:

124

124 tools. Let me show you what one looks like:

cat /tmp/tools-list.json | jq '.result.tools[0]'

Output:

{
  "annotations": {
    "readOnlyHint": false,
    "destructiveHint": true,
    "idempotentHint": false,
    "openWorldHint": true
  },
  "description": "Check the logs of the Argo Rollouts Gateway API plugin",
  "inputSchema": {
    "type": "object",
    "properties": {
      "namespace": {
        "description": "The namespace of the plugin resources",
        "type": "string"
      },
      "timeout": {
        "description": "Timeout for log collection in seconds",
        "type": "string"
      }
    }
  },
  "name": "argo_check_plugin_logs"
}

The annotations are interesting. MCP has a safety hint system. Each tool tells you:

• destructiveHint: true means this tool can change things

• idempotentHint: false means running it twice is not safe

• readOnlyHint: false confirms it is not read-only

These hints let LLMs (or orchestrators) make smarter decisions about when to call tools.

The 124 tools are organized by category:

Argo Rollouts:      3 tools
Cilium networking: 15 tools
Helm:               8 tools
Istio:             12 tools
Kubernetes core:   45 tools
Gateway API:        6 tools
Prometheus/PromQL: 35 tools

Step 3 > Call a Tool

Let me actually execute something. I will call k8s_get_resources to list namespaces:

curl -s -X POST http://localhost:8084/mcp \
  -H "Content-Type: application/json" \
  -H "Mcp-Session-Id: mcp-session-bed70407-ca06-4689-a2a6-e1ab213add6a" \
  -d '{
    "jsonrpc": "2.0",
    "id": 3,
    "method": "tools/call",
    "params": {
      "name": "k8s_get_resources",
      "arguments": {
        "resource_type": "namespace"
      }
    }
  }' | jq '.'

From my terminal:

{
  "jsonrpc": "2.0",
  "id": 3,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "NAME              STATUS   AGE\ndefault           Active   28m\nkagent            Active   27m\nkube-node-lease   Active   28m\nkube-public       Active   28m\nkube-system       Active   28m\n"
      }
    ]
  }
}

That is a real k get ns executed inside the MCP server pod. The tool server has cluster-admin RBAC, so it can read anything.

MCP itself is just JSON-RPC over HTTP. What feels like magic is the LLM picking the right tool and formatting arguments correctly.

The Transport: Server-Sent Events

One more thing I noticed. When I made a plain GET request to the MCP endpoint:

curl -v http://localhost:8084/mcp 2>&1 | head -25

Response headers:

HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: keep-alive
Content-Type: text/event-stream
Date: Sat, 25 Apr 2026 16:16:03 GMT
Transfer-Encoding: chunked

text/event-stream is Server-Sent Events (SSE). MCP’s STREAMABLE_HTTP transport uses SSE under the hood.

Makes sense for streaming. The MCP server can push tokens to the client as the LLM generates them.

Agent-to-Agent Communication

kagent has built-in support for agent-to-agent (A2A) delegation. I wanted to see how this works with MCP.

Here is what happens when one agent delegates to another:

I created a custom agent called sre-orchestrator:

cat <<'EOF' | k apply -f -
apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: sre-orchestrator
  namespace: kagent
spec:
  type: Declarative
  description: "SRE Orchestrator that delegates to specialized agents"
  declarative:
    modelConfig: default-model-config
    systemMessage: |
      You are an SRE orchestrator. When asked about metrics or PromQL,
      delegate to promql-agent. When asked about cluster state, use K8s tools directly.
    tools:
    - type: McpServer
      mcpServer:
        name: kagent-tool-server
        kind: RemoteMCPServer
        apiGroup: kagent.dev
        toolNames:
        - k8s_get_resources
        - k8s_describe_resource
        - k8s_get_pod_logs
    - type: Agent
      agent:
        name: promql-agent
EOF

Output:

agent.kagent.dev/sre-orchestrator created

Wait for it to be ready:

k wait --for=condition=Ready agent/sre-orchestrator -n kagent --timeout=120s

Output:

agent.kagent.dev/sre-orchestrator condition met

This agent has two types of tools:

1. Direct MCP tools (k8s_get_resources, k8s_describe_resource, k8s_get_pod_logs)

2. Another agent (promql-agent)

I gave it a task:

kagent invoke --agent sre-orchestrator \
  --task "First, list all pods in kagent namespace. Then ask promql-agent to write a PromQL query for CPU usage of those pods." \
  --stream

What happened (simplified from the JSON stream):

1. sre-orchestrator called k8s_get_resources with namespace=kagent, resource_type=pod

2. Got back 18 pod names

3. Called kagent__NS__promql_agent (agent delegation)

4. promql-agent generated this PromQL query:

sum(rate(container_cpu_usage_seconds_total{pod=~"argo-rollouts-conversion-agent-985c94668-4zm2t|cilium-debug-agent-5bdfbbf8b5-8qw5z|..."}[5m])) by (pod)

5. Response came back to sre-orchestrator

Two separate sessions. That surprised me.

From the JSON stream I saw:

Parent session: 2640382e-4884-417e-95ac-20d9af90085a (sre-orchestrator)
Sub-session:    b0354a88-bdef-4290-8442-c1f88138a9fa (promql-agent)

I checked the PostgreSQL database to confirm this:

k exec -n kagent deployment/kagent-postgresql -it -- psql -U kagent -d kagent -c "
SELECT 
  id,
  LEFT(id, 8) as short_id,
  agent_id,
  created_at,
  updated_at
FROM session 
WHERE id IN (
  '2640382e-4884-417e-95ac-20d9af90085a',
  'b0354a88-bdef-4290-8442-c1f88138a9fa'
)
ORDER BY created_at;
"

My output:

id | short_id | agent_id | created_at | updated_at           
--------------------------------------+----------+------------------------------+------
 2640382e-4884-417e-95ac-20d9af90085a | 2640382e | kagent__NS__sre_orchestrator | 2026-04-25 16:22:35.674079+00 | 2026-04-25 16:22:35.674079+00
 b0354a88-bdef-4290-8442-c1f88138a9fa | b0354a88 | kagent__NS__promql_agent     | 2026-04-25 16:22:49.573249+00 | 2026-04-25 16:22:49.573249+00

The sub-session was created 14 seconds after the parent. Two separate database records. Complete isolation.

Token accounting is also separate (from the JSON stream metadata):

• Parent session: 3,707 tokens total

• Sub-session: 2,407 tokens (prompt: 2,081, completion: 326)

When one agent delegates to another, the sessions do not share state. The parent only sees the final response from the child, not the internal reasoning or tool calls.

Worth knowing if you are debugging a multi-agent flow. You will need to dig into both sessions, not just the parent.

The Agent Card Protocol

A2A has a discovery mechanism. Every agent exposes a JSON file at /.well-known/agent-card.json.

I port-forwarded the k8s-agent service:

k port-forward -n kagent svc/k8s-agent 8080:8080 --address 127.0.0.1 &
sleep 2
curl -s http://localhost:8080/.well-known/agent-card.json | jq '.'

The response:

{
  "capabilities": {
    "pushNotifications": false,
    "stateTransitionHistory": true,
    "streaming": true
  },
  "defaultInputModes": ["text"],
  "defaultOutputModes": ["text"],
  "description": "An Kubernetes Expert AI Agent specializing in cluster operations, troubleshooting, and maintenance.",
  "name": "k8s_agent",
  "preferredTransport": "JSONRPC",
  "protocolVersion": "0.3.0",
  "skills": [
    {
      "description": "The ability to analyze and diagnose Kubernetes Cluster issues.",
      "examples": [
        "What is the status of my cluster?",
        "How can I troubleshoot a failing pod?",
        "What are the resource limits for my nodes?"
      ],
      "id": "cluster-diagnostics",
      "name": "Cluster Diagnostics",
      "tags": ["cluster", "diagnostics"]
    },
    {
      "description": "The ability to manage and optimize Kubernetes resources.",
      "examples": [
        "Scale my deployment X to 3 replicas.",
        "Optimize resource requests for my pods."
      ],
      "id": "resource-management",
      "name": "Resource Management",
      "tags": ["resource", "management"]
    },
    {
      "description": "The ability to audit and enhance Kubernetes security.",
      "examples": [
        "Check for RBAC misconfigurations.",
        "Audit my network policies."
      ],
      "id": "security-audit",
      "name": "Security Audit",
      "tags": ["security", "audit"]
    }
  ],
  "url": "http://k8s-agent.kagent:8080",
  "version": ""
}

The agent advertises:

• What it can do (three skills: cluster-diagnostics, resource-management, security-audit)

• Example queries it understands

• Protocol version (0.3.0)

• Preferred transport (JSONRPC)

Compare this to promql-agent:

k port-forward -n kagent svc/promql-agent 8082:8080 --address 127.0.0.1 &
sleep 2
curl -s http://localhost:8082/.well-known/agent-card.json | jq '.skills'

Output:

[
  {
    "description": "Translates a natural language description of monitoring needs into a precise and performant PromQL query, providing an explanation, assumptions, and alternatives.",
    "id": "generate-promql-query",
    "name": "Generate PromQL Query",
    "tags": ["promql", "prometheus", "query-generation", "metrics"]
  },
  {
    "description": "Explains how an existing PromQL query works, helps debug issues, or suggests refinements for better performance or accuracy.",
    "id": "explain-debug-promql-query",
    "name": "Explain or Debug PromQL Query",
    "tags": ["promql", "debug-query", "optimization"]
  },
  {
    "description": "Provides information about Prometheus data models, PromQL functions, syntax, common patterns, and best practices for writing effective queries.",
    "id": "promql-concepts-best-practices",
    "name": "PromQL Concepts and Best Practices",
    "tags": ["promql", "concepts", "best-practices", "tutorial"]
  }
]

And my custom sre-orchestrator:

k port-forward -n kagent svc/sre-orchestrator 8081:8080 --address 127.0.0.1 &
sleep 2
curl -s http://localhost:8081/.well-known/agent-card.json | jq '.skills'

Output:

[]

Empty. I did not define any skills for it. That is fine. Skill definitions are optional. But if you are building a multi-agent system, defining skills helps other agents understand what to delegate.

Things That Did Not Work

Not everything went smoothly.

Llama 3.3 70B and Tool Calling

I tried a general question first:

kagent invoke --agent k8s-agent \
  --task "What is the status of this cluster? How many nodes and namespaces?" \
  --stream

The agent called k8s_get_resources with these parameters (from the JSON stream):

{
  "namespace": "null",
  "resource_name": "null"
}

Not JSON null. The string "null".

The tool server tried to run k get node null -n null. From my terminal:

[Kubernetes] get node null -n null -o wide failed: exit status 1

The agent retried 9 times with the same parameters. Same error every time. Then it gave up.

But when I made the question specific:

kagent invoke --agent k8s-agent \
  --task "List all pods in the kagent namespace" \
  --stream

It worked perfectly. The parameters this time:

{
  "namespace": "kagent",
  "resource_type": "pod"
}

The tool executed successfully and returned 18 pods.

The problem is Llama 3.3 70B tool calling accuracy. It is not as sharp as GPT-4. When the query is vague, it guesses wrong. When the query is specific, it is fine.

If you run Llama in prod, write specific prompts. I learned that the hard way. Or just use Claude or GPT-4 and stop worrying about it.

Subscribe now

Wrapping up

MCP itself is HTTP, JSON-RPC, and a session header. Nothing exotic. The problem is that most tutorials stop at the config file, so you never see the actual exchange.

A few things stuck with me.

» Sessions live in the Mcp-Session-Id header, not the JSON body. If you are writing a client, parse headers. I almost missed this.

» Tool schemas have annotations like destructiveHint and readOnlyHint. That is genuinely useful. You can build guardrails so the LLM does not fire destructive tools without confirmation.

» Agent-to-agent delegation creates a new session under the hood. Parent and child are isolated. Separate database rows, separate token counts. Worth knowing if you are debugging a multi-agent flow.

» Tool calling is more LLM-dependent than I expected. Llama 3.3 70B handles specific prompts fine but trips on vague ones. For prod, I would probably reach for Claude or GPT-4 unless my prompts were really locked down.

» And the kagent Helm chart in 0.9.0 does not populate the baseUrl in ModelConfig. If you use Nebius or anything other than OpenAI, you have to patch it manually after install. Took me 20 minutes to figure that out.

Next?

I tested MCP with CPU-only agents. But what if your tools need GPUs? Imagine an MCP tool that runs Stable Diffusion, or does real-time video analysis.

On k8s, you will need GPU scheduling. I work on Project HAMi for this. It is a CNCF sandbox project that does multi-tenant GPU sharing. HAMi plus MCP could be interesting: MCP tools request GPU fractions, HAMi schedules them across pods.

Maybe a future article.

All the commands above are from a real session on a Nebius VM. Whole thing took 30 minutes on a k3s cluster. If you want to try it yourself, k3s and any OpenAI-compatible LLM API will work. I used Nebius for cost reasons, but OpenAI, Anthropic, or local Ollama all work.

The MCP Dev Summit is happening in Shanghai in September. I might submit this as a talk. If you are going, let me know.

When sth failed, I debugged it. When sth worked, I explained why. The errors in this article are real errors I hit. The fixes are things I actually tried. If you run these same commands on the same setup, you will get the same results. The full repo with all manifests and setup script is here:

See my repo on GitHub

A note on scope: This article covers the highlights. The full setup, all manifests, the complete troubleshooting guide, and the setup script live in the GitHub repo. If you want to run this yourself, start there.

If you are new to HAMi: I wrote a separate deep-dive on running HAMi in a real k8s env. That covers the installation from scratch and explains how the GPU virtualization layer works internally. Read that first if you have not set up HAMi before: https://medium.com/@mesutoezdil/hami-in-a-real-kubernetes-environment-e8eaa872f388.

If you want to see how I approach testing observability tool for GPU: https://mesutoezdil.substack.com/p/i-tested-every-feature-of-ingero

What this is about

kagent turns AI agents into k8s resources. Your system prompt, your tools, your model config: all of it lives as a CRD. You version it in Git, deploy it with Helm, and inspect it with kubectl just like everything else in your cluster.

HAMi virtualizes GPUs at the k8s scheduler level. One physical NVIDIA L40S becomes ten virtual GPUs in k8s, each with hard memory limits enforced at the CUDA driver level.

Nebius Token Factory is an OpenAI-compatible inference API. I used Llama 3.3 70B for all tests. No OpenAI account needed.

The question I wanted to answer: can an AI agent manage GPU-virtualized workloads from inside a k8s cluster, using only open-source models?

The answer is yes!

The machine

GPU:  1x NVIDIA L40S (46GB VRAM)
CPU:  8 vCPUs
RAM:  32GB
OS:   Ubuntu 24.04 LTS for NVIDIA GPUs (CUDA 13)

nvidia-smi

| NVIDIA-SMI 580.126.09    CUDA Version: 13.0      |
|   0  NVIDIA L40S    0MiB / 46068MiB    0%        |

46GB VRAM sitting completely idle. By the end of this, it runs as ten virtual GPUs.

1 > k3s and Helm

k3s is the right choice for a single-node setup. One binary, starts in seconds.

curl -sfL https://get.k3s.io | sh -

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chown ubuntu:ubuntu ~/.kube/config
export KUBECONFIG=~/.kube/config
alias k=kubectl
kubectl config set-context --current --namespace=kagent

k get node

NAME               STATUS   ROLES           AGE   VERSION
nebius-tarantula   Ready    control-plane   48s   v1.34.6+k3s1

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

2 > Installing kagent

kagent ships two Helm charts. CRDs go first, then the main chart. This is intentional. It lets you upgrade CRDs independently without touching running agents.

kubectl create namespace kagent

helm install kagent-crds \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent-crds \
  --namespace kagent

Pulled: ghcr.io/kagent-dev/kagent/helm/kagent-crds:0.8.6
STATUS: deployed

Now the main chart. This is where I point it at Nebius Token Factory instead of OpenAI:

helm install kagent \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --namespace kagent \
  --set providers.default=openAI \
  --set providers.openAI.apiKey=$NEBIUS_API_KEY \
  --set providers.openAI.model=meta-llama/Llama-3.3-70B-Instruct

providers.default=openAI does not mean I am using OpenAI. Nebius implements the OpenAI API specification. The provider field picks which API client to use, not which company to pay.

One thing the Helm chart does not handle automatically: setting the base URL in the ModelConfig. I had to patch it manually after install:

kubectl create secret generic kagent-openai \
  --from-literal=OPENAI_API_KEY=$NEBIUS_API_KEY \
  -n kagent --dry-run=client -o yaml | kubectl apply -f -

kubectl patch modelconfig default-model-config --type=merge -p '{
  "spec": {
    "openAI": {"baseUrl": "https://api.studio.nebius.ai/v1"},
    "model": "meta-llama/Llama-3.3-70B-Instruct"
  }
}'

kubectl rollout restart deployment -n kagent

After that, everything looks right:

k get agents,modelconfigs,remotemcpservers

NAME                                      TYPE          RUNTIME  READY  ACCEPTED
agent.kagent.dev/k8s-agent                Declarative   python   True   True
agent.kagent.dev/helm-agent               Declarative   python   True   True
agent.kagent.dev/istio-agent              Declarative   python   True   True
agent.kagent.dev/cilium-debug-agent       Declarative   python   True   True
... (10 agents total)

NAME                                         PROVIDER  MODEL
modelconfig.kagent.dev/default-model-config  OpenAI    meta-llama/Llama-3.3-70B-Instruct

NAME                                               PROTOCOL         ACCEPTED
remotemcpserver.kagent.dev/kagent-tool-server      STREAMABLE_HTTP  True
remotemcpserver.kagent.dev/kagent-grafana-mcp      STREAMABLE_HTTP  True

Ten agents, all ready. ModelConfig pointing at Nebius. Two MCP servers registered with 175 tools between them.

3 > Installing HAMi

Without HAMi, k8s cannot see the GPU at all:

k get node nebius-tarantula \
  -o jsonpath='{.status.allocatable}' | python3 -m json.tool

{"cpu": "8", "memory": "32865164Ki", "pods": "110"}

No nvidia.com/gpu. From k8s’ perspective, this is just a CPU node.

Before installing HAMi, there is one fix that the documentation does not make obvious. The nvidia-ctk tool writes a containerd config in version 1 format. k3s uses version 3. These two formats conflict, and it causes HAMi’s device plugin to crash on startup. The fix is to delete the conflicting file:

sudo rm -f /etc/containerd/conf.d/99-nvidia.toml
sudo systemctl restart containerd
sudo systemctl restart k3s
sleep 20

k3s already has the nvidia runtime in its own containerd config. The deleted file was silently overriding it with something incompatible. Now install:

helm repo add hami-charts https://project-hami.github.io/HAMi/
helm repo update

kubectl label node nebius-tarantula gpu=on

helm install hami hami-charts/hami \
  --set scheduler.kubeScheduler.imageTag=v1.34.6 \
  --set devicePlugin.nvidiaDriverRoot=/usr \
  -n kube-system

Two things matter here. The imageTag must exactly match your k8s version. HAMi extends the default scheduler, so version compatibility is strict. The nvidiaDriverRoot=/usr tells HAMi where to find NVIDIA libraries on the host.

k get pods -n kube-system | grep hami

hami-device-plugin-cw6t5          2/2   Running
hami-scheduler-84f8888fc5-fqrp8   2/2   Running

Both pods 2/2. Now check the node:

k get node nebius-tarantula \
  -o jsonpath='{.status.allocatable}' | python3 -m json.tool

{
  "cpu": "8",
  "memory": "32865164Ki",
  "nvidia.com/gpu": "10",
  "pods": "110"
}

"nvidia.com/gpu": "10". One physical GPU, ten virtual GPUs. The node annotation shows the full registration:

[{
  "id": "GPU-745c1c2c-56d7-168a-ca56-97e3dd8e0db7",
  "count": 10,
  "devmem": 46068,
  "type": "NVIDIA L40S",
  "mode": "hami-core",
  "health": true
}]

4 > First agent invocation

kubectl port-forward svc/kagent-controller 8083:8083 -n kagent &

kagent invoke --agent k8s-agent \
  --task "What is the status of this cluster? List all running pods." \
  --stream

Watching the stream, you can see exactly what happens. The LLM receives the task, decides to call k8s_get_resources, gets back the pod table, and summarizes:

"The cluster has 25 running pods across different namespaces,
including kagent and kube-system."

Total: 6,131 tokens.

5 > The GPU check

Before HAMi was installed, I asked the agent about GPU availability:

kagent invoke --agent k8s-agent \
  --task "What GPUs are available on the node?" --stream

"The node does not have any GPUs available."

After HAMi:

kagent invoke --agent k8s-agent \
  --task "Check nebius-tarantula allocatable resources. How many GPUs?" --stream

"The node nebius-tarantula has 10 GPUs available, type NVIDIA L40S."

The agent reads HAMi’s k8s annotations and understands what they mean.

6 > The self-inspection test

This one was genuinely interesting. I asked the agent to describe itself using the k8s API:

kagent invoke --agent k8s-agent \
  --task "Describe yourself using the Kubernetes API. What CRD defines you?" \
  --stream

The agent tried the default namespace first and failed. Then it listed all agents across namespaces, found itself in kagent, ran describe on k8s-agent -n kagent, read its own CRD including the system prompt and tool list, and explained its own architecture.

An agent reading its own k8s definition. Not from training data. From a live API call.

7 > Creating a custom agent

Beyond the ten built-in agents, you can define your own. I created an SRE orchestrator that delegates to the promql-agent for metrics queries:

apiVersion: kagent.dev/v1alpha2
kind: Agent
metadata:
  name: sre-orchestrator
  namespace: kagent
spec:
  type: Declarative
  declarative:
    modelConfig: default-model-config
    systemMessage: |
      You are an SRE orchestrator. Use k8s tools for cluster state.
      Delegate to promql-agent for metrics and PromQL queries.
    tools:
    - type: McpServer
      mcpServer:
        name: kagent-tool-server
        kind: RemoteMCPServer
        apiGroup: kagent.dev
        toolNames:
        - k8s_get_resources
        - k8s_describe_resource
        - k8s_get_pod_logs
    - type: Agent
      agent:
        name: promql-agent

kubectl apply -f sre-orchestrator.yaml
kubectl get agent sre-orchestrator

NAME               TYPE          RUNTIME  READY  ACCEPTED
sre-orchestrator   Declarative   python   True   True

Ready in under two minutes. The type: Agent entry is what makes A2A work. It lets the orchestrator call promql-agent as if it were a tool.

8 > Agent talking to agent

kagent invoke --agent sre-orchestrator \
  --task "Check pods in kagent namespace then ask promql-agent for CPU usage query." \
  --stream

Two different session IDs appear in the stream:

sre-orchestrator session:  6e4e0146-9a78-41a6-b2af-a8f3f2dd79db
promql-agent sub-session:  2ab4bf39-379d-404c-8fda-0e6c9e28c5da

Each agent ran in its own process with its own context window. The orchestrator only saw the final answer from promql-agent, not its internal reasoning. Both sessions were stored independently in PostgreSQL.

The actual tool call in the stream:

{
  "name": "kagent__NS__promql_agent",
  "args": {"request": "CPU usage query for pods in kagent namespace"},
  "metadata": {
    "kagent_subagent_session_id": "2ab4bf39-379d-404c-8fda-0e6c9e28c5da"
  }
}

9 > Agent creates a HAMi GPU pod

kagent invoke --agent k8s-agent \
  --task "Create a pod named gpu-test-1 requesting 1 nvidia.com/gpu with HAMi annotation nvidia.com/gpumem: 20000. Use ubuntu:22.04." \
  --stream

The agent generated and applied this:

metadata:
  name: gpu-test-1
  annotations:
    nvidia.com/gpumem: "20000"
spec:
  containers:
  - image: ubuntu:22.04
    resources:
      limits:
        nvidia.com/gpu: 1

HAMi picked it up immediately:

hami.io/bind-phase: success
hami.io/vgpu-devices-allocated: GPU-745c1c2c-56d7-168a-ca56-97e3dd8e0db7,NVIDIA,46068,100:;

I then created a second pod requesting 15,000 MiB. Both landed on the same physical GPU. 20,000 + 15,000 = 35,000 MiB, well under the 46,068 physical limit.

Then I asked the agent to explain what just happened:

kagent invoke --agent k8s-agent \
  --task "Look at gpu-test-1 and gpu-test-2. Explain how HAMi is sharing the GPU." \
  --stream

"GPU-745c1c2c-56d7-168a-ca56-97e3dd8e0db7 is shared between both pods.
gpu-test-1 has 20,000 MiB, gpu-test-2 has 15,000 MiB from the same NVIDIA L40S."

10 > Overcommit protection

I created a pod requesting 11 virtual GPUs when only 10 exist:

resources:
  limits:
    nvidia.com/gpu: 11

k describe pod gpu-overcommit-test | tail -5

Warning  FailedScheduling  hami-scheduler  0/1 nodes: 1 NodeUnfitPod
Warning  FilteringFailed   hami-scheduler  no available node, 1 nodes do not meet

Pod stays Pending forever. HAMi will not schedule what it cannot fulfill. No partial allocation, no silent failure.

11 > HAMi metrics

curl http://localhost:31992/metrics

HostCoreUtilization{devicetype="NVIDIA-NVIDIA L40S",zone="vGPU"} 0
HostGPUMemoryUsage{devicetype="NVIDIA-NVIDIA L40S",zone="vGPU"} 6.40090112e+08
hami_build_info{version="v2.8.1",go_version="go1.25.5"} 1

610 MiB consumed. That is the CUDA runtime overhead from idle pods. Utilization is 0% because sleep infinity does not run any GPU compute. Standard Prometheus format, works with any existing monitoring setup.

12 > kagent CLI

kagent get agent

+----+-----------------------------------+----------------------+------------------+
| #  | NAME                              | CREATED              | DEPLOYMENT_READY |
+----+-----------------------------------+----------------------+------------------+
| 1  | kagent/k8s-agent                  | 2026-04-18T07:27:18Z | true             |
| 4  | kagent/sre-orchestrator           | 2026-04-18T07:45:56Z | true             |
...

kagent get session

+----+------------------+------------------------------+----------------------+
| #  | ID               | AGENT                        | CREATED              |
+----+------------------+------------------------------+----------------------+
| 6  | 6e4e0146-...     | kagent__NS__sre_orchestrator | 2026-04-18T07:46:36Z |
| 7  | 2ab4bf39-...     | kagent__NS__promql_agent     | 2026-04-18T07:46:40Z |

Row 7 is the A2A sub-session created by row 6. The 4-second gap is the delegation latency. PostgreSQL stores everything.

The A2A agent card

Every kagent agent exposes its capabilities at /.well-known/agent-card.json:

curl http://10.42.0.68:8080/.well-known/agent-card.json | python3 -m json.tool

{
  "name": "k8s_agent",
  "protocolVersion": "0.3.0",
  "preferredTransport": "JSONRPC",
  "capabilities": {"streaming": true, "stateTransitionHistory": true},
  "skills": [
    {"id": "cluster-diagnostics", "name": "Cluster Diagnostics"},
    {"id": "resource-management", "name": "Resource Management"},
    {"id": "security-audit", "name": "Security Audit"}
  ],
  "url": "http://k8s-agent.kagent:8080"
}

This is how agents discover each other in multi-agent systems.

What did not work

Memory CRD only supports Pinecone as the vector database backend. There is no built-in option. Cross-session agent memory requires an external Pinecone account.

kmcp init runs without error and creates nothing. The project scaffolding feature is listed in the CLI help but does not actually work in v0.8.6.

Ubuntu + HAMi + sleep: HAMi injects libvgpu.so via /etc/ld.so.preload. If your image does not have CUDA libraries, even the sleep binary fails to start. Use CUDA-enabled base images for GPU workloads.

HAMi WebUI is not bundled in the main Helm chart. The Prometheus metrics endpoint works fine but the graphical dashboard requires a separate installation.

Why this setup makes sense

Your deployment specs live in Git. Your network policies live in Git. Your RBAC rules live in Git. There is no reason your AI agent system prompts should be any different. kagent makes that possible without building custom tooling.

HAMi fixes GPU waste without requiring any changes to your applications. It works at the CUDA driver level, invisible to the workload.

Together, they give you AI agents that can see and manage GPU-virtualized infrastructure from inside the cluster, using open-source models, with no dependency on any closed AI provider.

All manifests, the setup script, and eight specific troubleshooting cases with root causes and verified fixes are in the repo: https://github.com/mesutoezdil/kagentWithHami

One last thing: if any command here does not work for you, open an issue in the repo. I read them. These are not generated examples. They are things I actually ran, and I can help debug if something behaves differently on your setup.

I tested Ingero v0.9.1 on a real NVIDIA L40S GPU from start to finish.

I ran the commands, checked the outputs, and looked at how it behaves in real situations.

In this post, I cover:

• How it traces CUDA at kernel level with eBPF

• Why nvidia-smi can look fine while performance is not

• How CPU, disk I/O, and GPU latency are connected

• The difference between Driver API and Runtime API

• Real root causes and how to fix them

  

Everything is based on actual runs, not theory.

Read here: https://medium.com/@mesutoezdil/i-tested-every-feature-of-ingero-v0-9-1-94ae6de6eb7a

This made sense in the past. Back then, GPU workloads were mostly long training jobs that really needed the full power of a GPU. But things have changed.

While we’re at it, if you want to learn (GPU and) CUDA (starting from zero), you can star the repo I created for this and contribute as well. And you can follow me on LinkedIn if you’d like.

Today, inference workloads are smaller. Experiments are shorter. Teams share the same infrastructure.

In reality, a single pod often uses only about 15–20% of the GPU memory, and the rest just stays unused. But k8s still locks the whole GPU for that one pod, so nobody else can use it until the job is done.

The easy solution is to buy more GPUs. The smarter solution is to share what you already have.

Subscribe now

So what does HAMi actually do?

HAMi is a k8s extension that lets you share GPUs instead of giving the whole card to just one pod. Instead of treating a GPU as one big block, it allows you to decide exactly how much memory and compute each workload should get.

So, in a nutshell, that’s it. Of course, there’s more to it than that.

Inside the container, it still looks like the pod has its own GPU with the amount of memory you assigned. It doesn’t see other workloads running on the same physical GPU.

But from the cluster side, the rest of the GPU is still free and can be used by other pods.

Two resources make this possible:

nvidia.com/gpumem → This is the GPU memory in MiB. If a pod asks for 10240, it will see exactly 10240 MiB inside the container, no matter how big the actual GPU is.

nvidia.com/gpucores → This controls how much compute power the pod can use. If a pod requests 50, it can use up to 50% of the GPU.

This is not the same as GPU time-slicing. Time-slicing just shares the GPU over time and doesn’t really isolate memory.

HAMi is stricter. It sets hard limits. If a workload tries to use more memory than it was given, it simply fails. It doesn’t spill over into someone else’s share.

I think I’ve explained what we’re going to do today up to this point.

Why I set this up on my own server

I run AI workloads on my own setup: an NVIDIA L40S on k3s. The card has 48 GB of VRAM, and honestly, most of my daily workloads don’t even come close to using all of it.

I usually run multiple things at once, inference services, evaluation jobs, small fine-tuning experiments, all competing for the same GPU.

HAMi felt like the right solution. But the official docs assume you already know a lot, so it’s not that easy to follow in practice.

So I went through the setup myself, ran into the real issues, and wrote down what actually happens.

I tested everything step by step on my own setup: an NVIDIA L40S with k3s, driver 580.126.09, and CUDA 13.0.

This isn’t theory. Every command here was actually run on real hardware. If sth worked, I noted it. If it failed, I noted that too.

The goal is simple but strict: to make sure HAMi is not just installed, but actually working.

So what does “working” actually look like?

Before we start, it’s important to be clear about this. A lot of people say “it works” too early. A setup is not working just because:

• Pods are running

• helm install finished successfully

• The scheduler pod is up.

A setup is really working when:

• You can actually access the GPU inside a container

• K8s correctly shows the GPU resources

• The HAMi scheduler doesn’t block your workloads

• The limits you set are really enforced inside the container

Everything in this guide is about proving exactly that.

Environment

This guide was tested on a clean setup with k3s running on a recent stable k8s version, using an NVIDIA L40S GPU with driver 580.126.09 and CUDA 13.0.

Everything described here was actually executed on this env.

Note: Different k8s distributions behave differently, especially k3s vs. kubeadm. This guide confirms HAMi works in a lightweight k3s env. The official HAMi docs require kubectl v1.16+, Helm v3+, CUDA v10.2+, and NVIDIA Driver v440+.

Before starting, NVIDIA drivers must already be installed on your GPU nodes. The next section covers the container runtime config. This is the part most guides skip, and it’s where most failures actually happen.

Step 1 > Configure the NVIDIA container runtime

HAMi and GPU workloads in general need the NVIDIA container runtime to be set as the default runtime on each GPU node.

If this isn’t configured, containers won’t see the GPU at all, no matter what k8s says.

Run the following on every GPU node in your cluster.

Install nvidia-container-toolkit:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

Configure Docker (if using Docker as your container runtime)

Edit /etc/docker/daemon.json to set nvidia as the default runtime:

{
  “default-runtime”: “nvidia”,
  “runtimes”: {
    “nvidia”: {
      “path”: “/usr/bin/nvidia-container-runtime”,
      “runtimeArgs”: []
    }
  }
}

Then restart Docker:

sudo systemctl daemon-reload && systemctl restart docker

Configure containerd (if using containerd)

Edit /etc/containerd/config.toml:

version = 2
[plugins]
  [plugins.”io.containerd.grpc.v1.cri”]
    [plugins.”io.containerd.grpc.v1.cri”.containerd]
      default_runtime_name = “nvidia”

      [plugins.”io.containerd.grpc.v1.cri”.containerd.runtimes]
        [plugins.”io.containerd.grpc.v1.cri”.containerd.runtimes.nvidia]
          privileged_without_host_devices = false
          runtime_engine = “”
          runtime_root = “”
          runtime_type = “io.containerd.runc.v2”
          [plugins.”io.containerd.grpc.v1.cri”.containerd.runtimes.nvidia.options]
            BinaryName = “/usr/bin/nvidia-container-runtime”

Then restart containerd:

sudo systemctl daemon-reload && systemctl restart containerd

With the runtime in place, we can now verify that GPU workloads actually run before we add HAMi on top.

Step 2 > Validate the GPU stack

Before touching HAMi, you first need to make sure k8s can actually run a GPU workload.

This step is not optional. If it fails, no amount of HAMi setup will fix it and in most cases, what looks like a “HAMi issue” is really a problem with the GPU runtime or drivers underneath.

Create this test pod:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-test
spec:
  restartPolicy: Never
  containers:
    - name: cuda
      image: nvcr.io/nvidia/cuda:12.2.0-base-ubuntu22.04
      command: [”nvidia-smi”]
      resources:
        limits:
          nvidia.com/gpu: 1

Apply and verify:

kubectl delete pod cuda-test --ignore-not-found
kubectl apply -f cuda-test.yaml
kubectl wait --for=condition=Ready pod/cuda-test --timeout=60s || true
kubectl wait --for=condition=Succeeded pod/cuda-test --timeout=60s || true
kubectl logs cuda-test

What’s happening here is simple: K8s looks for a node that exposes nvidia.com/gpu, the device plugin reports the available GPUs, the container runtime makes the GPU accessible inside the container, and then nvidia-smi runs there.

If you can see GPU info in the logs, it means your whole GPU stack (drivers, container runtime, device plugin, and k8s scheduling) is working correctly.

Do not move forward until this step works.

Troubleshooting Step 2

As I mentioned earlier, I encountered a lot of errors in my own tests, so I’m sharing these solutions in the hope that they’ll be helpful to you as well. If nvidia-smi failed, work through these in order.

i. Remove conflicting NVIDIA device plugins

If you’ve tried multiple GPU setups before, stale device plugins may be interfering:

kubectl delete daemonset nvidia-device-plugin-daemonset -n kube-system --ignore-not-found

ii. Ensure the node is labeled correctly

Some setups require explicit GPU labeling on the node:

kubectl label node $(hostname) nvidia.com/gpu.present=true --overwrite

iii. Verify GPU access outside k8s

Isolate whether the problem is k8s or the runtime itself:

sudo ctr run --rm --gpus 0 docker.io/nvidia/cuda:12.2.0-base-ubuntu22.04 test nvidia-smi

If this works but the k8s pod fails, the issue is in the device plugin or container runtime config, not the driver.

iv. Re-run the test

kubectl delete pod cuda-test --ignore-not-found
kubectl apply -f cuda-test.yaml
kubectl logs cuda-test

Once you see clean nvidia-smi output, you’re ready to continue.

Step 3 > Verify GPU resources on the node

Now that we know the GPU works, let’s confirm that k8s is correctly advertising it as a schedulable resource.

kubectl get nodes -o jsonpath=’{.items[*].status.allocatable}’ | grep -i nvidia

Or without filtering, to see the full picture:

kubectl get nodes -o jsonpath=’{.items[*].status.allocatable}’

Expected output includes:

nvidia.com/gpu: 1

At this stage, you won’t see nvidia.com/gpucores or nvidia.com/gpumem yet. These are HAMi-specific resources and only show up after it’s installed. Right now, we’re just making sure the GPU is visible to the scheduler.

Once that baseline is confirmed, we’re ready to bring HAMi into the picture.

Step 4 > Label your GPU nodes

HAMi’s scheduler only works with nodes that have the gpu=on label. This is a specific requirement. If the label is missing, the scheduler will simply ignore the node, and your GPU workloads either won’t get scheduled or will run without HAMi’s resource control at all.

kubectl label nodes $(hostname) gpu=on

If you have multiple GPU nodes, run this for each one, replacing $(hostname) with the actual node name. It’s easy to skip this step and then spend an hour wondering why HAMi isn’t enforcing anything.

Step 5 > Install HAMi

HAMi builds on top of how k8s already handles GPUs. It adds its own scheduler logic, introduces new GPU resources like gpucores and gpumem, and makes it possible to share a single GPU across multiple workloads. It doesn’t replace the existing GPU stack, it depends on it.

Before you install it, first check your k8s server version. The HAMi scheduler image tag needs to match that version exactly.

kubectl version

Then add the HAMi Helm repository and install:

helm repo add hami-charts https://project-hami.github.io/HAMi/

helm install hami hami-charts/hami \
  --set scheduler.kubeScheduler.imageTag=v1.XX.X \
  -n kube-system

Replace v1.XX.X with your actual k8s server version. For example, if your cluster is running 1.28.4, use --set scheduler.kubeScheduler.imageTag=v1.28.4.

Getting this version wrong is one of the most common HAMi installation failures. The scheduler will start but refuse to bind to k8s correctly if there’s a mismatch and it won’t always fail loudly.

Step 6 > Verify HAMi components

Once the Helm install completes, check that both the device plugin and scheduler are running:

kubectl get pods -n kube-system | grep hami

You should see both vgpu-device-plugin and vgpu-scheduler in a Running state. If one of them is in CrashLoopBackOff, the most common reason is that the scheduler image version doesn’t match your k8s version from the previous step.

To make sure the scheduler is actually doing sth, not just sitting there, check its logs.

kubectl logs -n kube-system -l app=hami-scheduler

Look for signs that it’s actually making decisions. Things like scheduling actions, resource checks, and GPU allocation attempts in the logs.

If you want to go a bit deeper, you can also check the node-level resource state.

kubectl get nodes -o wide
kubectl describe node $(hostname)

With both components confirmed running, let’s prove the whole chain works end to end.

Step 7 > Re-run the GPU workload under HAMi

This step answers a direct question: did installing HAMi break anything that was already working?

kubectl delete pod cuda-test --ignore-not-found
kubectl apply -f cuda-test.yaml
kubectl wait --for=condition=Ready pod/cuda-test --timeout=60s || true
kubectl wait --for=condition=Succeeded pod/cuda-test --timeout=60s || true
kubectl logs cuda-test

You should see the same clean nvidia-smi output as before. If you do, the scheduling chain is intact, no regression was introduced, and HAMi is coexisting cleanly with your existing GPU setup.

Step 8 > Submit a vGPU workload and verify resource limits

This is the step that proves HAMi is actually doing sth. We’re going to submit a workload that requests a specific amount of GPU memory, then verify from inside the container that the limit is enforced.

Create the following pod:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
    - name: ubuntu-container
      image: ubuntu:18.04
      command: [”bash”, “-c”, “sleep 86400”]
      resources:
        limits:
          nvidia.com/gpu: 1          # requesting 1 vGPU
          nvidia.com/gpumem: 10240   # limiting to 10240 MiB of GPU memory

Apply it:

kubectl apply -f gpu-pod.yaml
kubectl get pod gpu-pod

Once it’s running, exec into it and check what the GPU looks like from inside:

kubectl exec -it gpu-pod -- nvidia-smi

If HAMi is working correctly, you’ll see output similar to this.

Note the memory cap at 10240 MiB, not the full GPU memory:

[HAMI-core Msg]: Initializing.....
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15    Driver Version: 550.54.15    CUDA Version: 12.4               |
|-----------------------------------------+------------------------+----------------------+
|   0  Tesla V100-PCIE-32GB          On   |   00000000:3E:00.0 Off |                   0 |
| N/A   29C    P0    24W / 250W           |       0MiB / 10240MiB  |      0%     Default |
+-----------------------------------------+------------------------+----------------------+

The key detail here is 0MiB / 10240MiB

The container only sees 10240 MiB of GPU memory, not the full 32 GB of the physical card. That means HAMi’s resource control is actually working.

This is exactly what we’ve been building toward. From the container’s point of view, it has its own dedicated slice of the GPU.

At the same time, other pods can use the remaining capacity on the same card, without any of them being aware of each other.

What you’ve actually verified

At this point, your understanding should have changed a bit.

Before, the thinking was simple: test GPU → install HAMi → done.

Now it’s more structured: set up the runtime → make sure the GPU really works → label the node → install HAMi with the correct version → check the scheduler → finally verify a real vGPU workload with enforced limits.

By going through this process, you’ve proven a few important things.

The NVIDIA container runtime is correctly set up. The GPU stack works end-to-end in k8s. Your GPU nodes are properly labeled and picked up by the HAMi scheduler.

HAMi itself is installed with the right version. It doesn’t break existing GPU workloads. And most importantly, resource limits like gpumem are actually enforced inside the container.

What this guide doesn’t cover is also important. We didn’t go into gpucores for limiting compute, or fairness and multi-tenant scheduling, or more advanced setups like multi-GPU and MIG.

All of those build on top of the foundation you’ve just validated.

Common failure scenarios

Not everything in life goes according to plan. Especially when it comes to such a complex issue, even if expectations are high, it’s only natural to encounter challenges in the real world.

GPU not visible in k8s

kubectl get nodes -o jsonpath=’{.items[*].status.allocatable}’

If nvidia.com/gpu is missing, the device plugin is not working or the container runtime is misconfigured. Revisit Step 1.

Pod stuck in ContainerCreating

Almost always means runtime hook issues or missing NVIDIA libraries. The pod was scheduled, it just can’t start. Revisit the containerd or Docker runtime config in Step 1.

nvidia-smi works outside k8s but not inside a pod

The container runtime is not wired into k8s correctly. The device plugin cannot pass GPU access into the pod. Fix Step 1 before continuing.

HAMi scheduler in CrashLoopBackOff

Almost certainly a version mismatch between the scheduler image tag and your k8s server version. Re-run kubectl version, then reinstall with the correct imageTag.

gpu-pod stays Pending after HAMi install

Check that the node has the gpu=on label. Without it, HAMi’s scheduler ignores the node entirely.

Common misconceptions

I’d like to touch on a few points that will prevent you from letting your imagination run wild.

“HAMi is not working.” If nvidia-smi runs in a plain GPU pod, the base stack is fine. HAMi is a scheduler extension, verify the scheduler image version and node labels before assuming HAMi itself is broken.

“The scheduler is running, so GPU sharing is active.” Scheduler presence confirms deployment. Actual resource enforcement is only confirmed by exec-ing into a running vGPU pod and checking memory limits, as shown in Step 8.

“I don’t need to set the imageTag. HAMi figures it out.” In most recent versions this is true, but version mismatches are still the top cause of silent failures. When in doubt, set it explicitly.

Troubleshooting order

When sth goes wrong, always debug in this sequence:

1. NVIDIA drivers and nvidia-container-toolkit

2. Container runtime config (Docker or containerd)

3. Device plugin status

4. Node resource advertisement and gpu=on label

5. HAMi scheduler version and logs

kubectl delete pod cuda-test
kubectl delete pod gpu-pod

End

At the end of the day, HAMi is not some magic layer you just install and forget. It only works as well as the foundation underneath it. If your GPU stack is solid, HAMi gives you a clean and practical way to actually use your hardware efficiently instead of letting it sit idle.

Start simple. Validate each layer. Don’t assume anything is working until you’ve seen it yourself inside the container.

I’d like to express my thanks to JetBrains for their support.

Haven’t posted articles anywhere. Not because I wasn’t making stuff. The opposite actually.

Shallow prompt posts and recycled “AI” stuff had turned my feed into noise.

People sharing prompt outputs like they just invented the wheel and used their own brain was tiring me out.

I’m back though, with one simple rule: real content, stuff that works, stuff you can actually use again.

So yeah, I started posting about OTel.

This is probably my first real article about it. Let’s just jump in.

I think we can all agree on this: when systems get messy, trying to figure out what’s wrong from one graph or one log line gets really hard.

You know that feeling during an outage when you open a bunch of dashboards and still can’t find a clear answer? Super familiar.

That’s why teams now look at three things together: traces, metrics, and logs. I posted the basic stuff about these before with the #OTelDay-2 tag.

But it’s not really about definitions.

The real deal is: what question does each one answer?

Subscribe now

When do you use which one? And how do they tell a story together?

I think that’s the best way to learn. Hard to find the right answer if you don’t ask the right question first.

A trace shows one request’s trip through your system.

The request goes through services. Sits in a queue. Hits the database.

Sometimes calls an outside API. Each step on that trip is a span.

Each span usually has the name of what it’s doing.

When it started, how long it took, did it work, did it break... All that lives on the span. And there’s also tags. They give you context. Like db.system = PostgreSQL or http.route = /checkout.

Now we can clearly see what traces do.

Traces answer these questions: “Which step was slow?” “Where exactly did it break?” “How did this mess up the whole thing?”

You look at traces when you want to see what caused what.

Same when you want to understand the order stuff happened. That’s what gives you the big picture.

In my #OTelDay-1 post I really pushed one thing: context propagation.

Basically the same trace ID following the request through all the services.

If that breaks, the trace breaks. The story stays half-told.

Another key thing is grabbing the right tags from the start. This is the info that actually helps when stuff breaks.

Customer type, region, order ID, stuff like that.

That way you don’t drown in hundreds of traces. You narrow it down fast.

So here’s what we’re learning: traces show you cause and effect and order.

Context needs to follow through. Important tags need to be grabbed early.

So instead of tracking every single line of code, focus on the key spots.

Incoming requests, outgoing calls, database stuff. Traces tell us one user’s trip.

But how do we know when something’s hitting lots of users at once?

That’s where metrics come in. The basic question metrics ask is: “Is this normal?”

Metrics are numbers over time. They tell you there’s a spike. Traces show you where.

One of the biggest mistakes with metrics is setting alerts based on averages.

Because averages hide user pain. A few users can have a terrible time while tons of fast requests keep the average looking fine.

Another mistake is making random metrics for each service. Way better to have a shared set of core metrics across services.

We’ve asked the right questions so far. But there’s still something missing: details.

When we want to know exactly what went down, logs come in. Logs answer: “What exactly happened?”

Logs have parameters, choices that were made, error dumps, security stuff.

When they’re organized and linked to trace and span IDs, they’re super powerful.

They give you clarity without all the noise.

Logging mistakes happen a lot too. Messy and super detailed logs are hard to search and expensive.

Another bad one is logging sensitive stuff. That’s why you gotta mask things where needed.

Now we know what these three things do.

But the real magic is when they work together. During an outage, it usually goes like this:

First, metrics sound the alarm. Like p95 delay for /checkout jumps from 300 milliseconds to 2.2 seconds.

Then you check traces. You filter for that same time and /checkout. You see most slow traces are spending 1.8 seconds on the payment step.

Next you pull up logs for those same trace IDs. You notice timeouts when calling the payment gateway when fraud_check = true.

Finally you check dependency metrics. You confirm outside API errors only went up in East US.

Each step clears things up a bit more.

Bottom line: metrics sound the alarm. Traces show where the problem is.

Logs confirm what happened. Together they make a messy situation something you can actually solve step by step.

For this to work smoothly across teams, you need standards. That’s exactly why OpenTelemetry rules matter. I talked about this more in my #OTelDay-3 post.

Let me wrap up with a quick rule:

Check metrics first. To understand how big it is.

Then go to traces. To find the bottleneck.

Dig into logs if you need details.

And one last thing: don’t trust averages for alerts. Averages hide user pain.

Percentiles show what your slowest users are really going through. Most of the time, that’s where the real story is.

Now let me quickly recap last week’s posts so everything really sticks.

#OTelDay-0

I wanted us to ask ourselves: “If you already have metrics and logs, why are you still blind in prod?” A reminder that OTel is awesome and will solve a ton of problems by giving us a common language for observability.

#OTelDay-1

Metrics check the system’s pulse. Response times and error rates show us the general health. Logs give clearer answers to “What happened?” They show what went down behind the scenes. Traces show a request’s journey between services. Where it slowed down, where it got stuck. And the most important piece connecting everything: context propagation. Keeps traces from breaking in distributed systems.

#OTelDay-2

OTel is an open-source standard for collecting telemetry data. Instead of every framework or vendor doing observability their own way, OTel gives us a shared model. Shows how events in the system connect to each other in a clear and consistent way. OTel isn’t a logging tool. Doesn’t replace your logs or monitoring system. Doesn’t store data. Doesn’t show dashboards. OTel is a standard. Defines what telemetry data should look like and how to collect it. So different tools can work together smoothly.

#OTelDay-3

If your measurement language isn’t clear, your decisions won’t be clear either. This is exactly where semantic conventions become critical. With semantic conventions you can put services side by side. Metric names don’t get messed up over time. You don’t have to rewrite everything when you switch vendors.

That’s the rundown.

Keep it simple, keep it real, and your systems will thank you.

Got questions? I’m here.

In this article I will give an introduction to OpenShift’s network design. Let me say at the outset that I have no (professional) affiliation with RedHat or OpenShift. I share these articles and related posts on LinkedIn with the intention of helping myself first and then others. I think that’s obvious, so let’s get to the point.

OpenShift’s network design helps containers, services, and users communicate smoothly inside and outside the cluster. It expands on k8s networking with extra tools for control and visibility. A lot of things we know from k8s we won’t repeat here, but we will still point out some reminders. Each Pod in OpenShift gets its own IP and communicates through a flat network called the Cluster Network. K8s Services hide these IP addresses behind stable IPs and DNS names.

OpenShift handles internal networking with CNI plugins. OpenShift SDN used to be the default, but from OpenShift 4.6 onward, OVN-Kubernetes is recommended. Both plugins meet basic networking needs, but OVN-Kubernetes provides better scalability, security, and flexibility. Let’s take a closer look at what this all means.

Internal Communication (Pod-to-Pod Communication)

OpenShift uses CNI plugins to manage internal pod networking. CNI plugins handle IP assignment, routing, and security for OpenShift pods. Each pod receives a unique IP address and communicates over the Cluster Network using K8s-native services. These services provide a stable Cluster IP to access pods across nodes.

Supported CNI plugins include OpenShift SDN (default in older versions) and OVN-Kubernetes (default in OpenShift 4.6+). These plugins facilitate pod networking, traffic policies, and service connectivity. Well, what is this SDN?

☞ Software-Defined Networking (SDN)

OpenShift SDN is a built-in CNI plugin that manages pod communication and enforces policies using software-defined overlays. It offers three operational modes:

Cluster Network Mode ⇢ All pods can communicate freely across the cluster

Multitenant Mode ⇢ Enforces isolation between projects (namespaces)

Subnet Mode ⇢ Each node has its own subnet for pod IPs

OpenShift SDN is now deprecated in favor of OVN-K8s, which is the default CNI plugin since OpenShift 4.6. SDN is simple to use and good for small-to-medium systems but might struggle in larger setups. OpenShift SDN’s centralised control plane can become a bottleneck in clusters with 1000+ nodes.

Open vSwitch (OVS) powers SDN, creating virtual bridges and supporting overlay technologies like VXLAN or GENEVE. Features of OVS include:

⇥ Virtual bridging of pod traffic

⇥ Flow control using OpenFlow rules

⇥ MAC address management

⇥ Overlay networking using VXLAN or GENEVE

VXLAN is used with OpenShift SDN to expand virtual networks. GENEVE is supported by OVN-K8s and offers enhanced flexibility compared to VXLAN.

☞ OVN-Kubernetes

OVN-K8s has replaced OpenShift SDN as the default CNI provider, offering enhanced scalability, security, and flexibility for modern OpenShift deployments. Built on Open Virtual Network (OVN), this plugin introduces a logical networking architecture that leverages:

⇥ Logical switches/routers (managed through OVSDB)

⇥ Distributed control plane (improves scalability vs SDN’s centralised model)

⇥ GENEVE encapsulation (more flexible than VXLAN)

The traditional OpenShift SDN modes (Cluster/Multitenant/Subnet) are superseded by OVN’s unified architecture, which provides:

⇥ Automated IP address management (no manual subnet allocation)

⇥ Distributed east-west routing (no single chokepoint)

⇥ Integration with OpenStack/VM workloads (via OVN provider)

While OVN introduces minimal overhead (~5-8% vs SDN), its distributed architecture scales better beyond 500 nodes.

External Communication

I hope that the internal communication has become clear in a nutshell. In fact, for those familiar with the k8s structure, the external structure will be similarly obvious. OpenShift exposes services to external users using three main mechanisms:

NodePort: Binds services to a static port on each worker node. Suitable for development or testing but not recommended for production. Binds services to a static port (30000-32767) on every worker node.

LoadBalancer: Integrates with cloud or on-prem load balancers (e.g., AWS ELB, Azure LB). Distributes external traffic to internal services.

Ingress Controller: A HAProxy-based controller that handles HTTP/HTTPS routing via OpenShift Routes. It’s the default and most flexible option for exposing web applications. And OpenShift’s HAProxy-based Ingress Controller (default) enables advanced L7 routing via Route objects.

The Ingress Controller receives incoming requests and maps them to the appropriate service within the cluster using DNS-based routing and predefined rules.

Routing in OpenShift

OpenShift leverages DNS-based service discovery and Route objects to manage both internal and external application access. Routes (custom resources in route.openshift.io/v1) expose services externally via the Ingress Controller. Each namespace automatically receives internal DNS addresses in the following format:

my-service.my-namespace.svc.cluster.local

OpenShift Routes provide a mechanism to expose services externally through hostnames (e.g., myapp.apps.example.com). When a Route resource is created, OpenShift’s Ingress Controller (typically the HAProxy-based default ingress controller) processes hostname and path-based routing rules specified in the Route definition and forwards external traffic to the appropriate backend service. However, the OpenShift DNS Operator itself typically manages internal cluster DNS resolution (*.cluster.local) rather than external DNS records. External DNS management (like creating a DNS entry such as myapp.apps.example.com) usually requires:

⇥ Manual DNS configuration (if external DNS is managed separately)

⇥ Integration with the OpenShift ExternalDNS Operator (if automation is desired)

By default, OpenShift doesn’t automatically create external DNS entries unless you specifically configure it using an additional operator like the ExternalDNS Operator. Last but not least, when a Route resource is created:

⇥ The Ingress Controller (default is HAProxy-based) inspects the Route definition,

⇥ Based on the specified hostname and optional path rules, it routes incoming external traffic to the correct backend service within the cluster,

⇥ External DNS records (e.g., myapp.apps.example.com) are typically managed separately, either manually or via OpenShift’s ExternalDNS Operator.

Service Mesh

It is also important to address this point. OpenShift provides robust support for Service Mesh, enabling secure and observable communication between microservices within the cluster. The primary implementation is the OpenShift Service Mesh (OSSM), built upon the Istio service mesh framework.

Service Mesh capabilities include:

Traffic Management ⇢ Intelligent load balancing, advanced routing rules, circuit breaking, retries, and traffic shifting.

Security ⇢ Secure service-to-service communication enforced via mutual TLS encryption.

Observability ⇢ Integrated monitoring capabilities including metrics collection, distributed tracing, and logging.

And how it works? Each pod participating in the service mesh contains an Envoy sidecar proxy, which intercepts and manages all incoming and outgoing network traffic. The Istio Control Plane (primarily Istiod in modern Istio versions) centrally manages and distributes configuration and policy settings to the Envoy proxies. OpenShift Service Mesh leverages Kubernetes Operators to streamline the deployment, configuration, and lifecycle management of all mesh components, significantly simplifying operational complexity.

Egress (Outbound Traffic) Management

Egress defines how Pods within your OpenShift cluster access external networks such as the internet or third-party services. It determines which IP addresses are used for outbound traffic and controls which domains or IP ranges can be reached.

In some scenarios, it is necessary for Pods to consistently use a fixed external IP address—for example, to comply with firewall rules or access control lists enforced by external APIs or databases that only allow traffic from known IPs.

In OpenShift, egress IP management is handled by CNI plugins like OpenShift SDN or OVN-K8s. These plugins allow administrators to define which node handles outbound traffic and which external IP addresses are assigned. Egress firewall rules can also be applied to limit access to specific domains or IPs. For instance, you may want to allow traffic only to trusted domains such as *.mycompany.com while blocking all other external access. Common Use Cases Include:

⇥ Restricting traffic to specific external services or destinations

⇥ Allowing only approved IP ranges for outbound communication

⇥ Assigning fixed public IPs to Pods for predictable firewall rule configuration

OVN-Kubernetes offers built-in support for egress firewall policies, allowing fine-grained control over outbound traffic behavior per namespace.

DNS Management

OpenShift provides comprehensive DNS resolution capabilities, automatically managing both internal and external networking components such as Services, Pods, and Routes. This integrated DNS functionality is crucial for seamless service discovery, efficient internal routing, and secure external exposure of workloads.

Internally, OpenShift assigns a predictable DNS record to every Service within a namespace using a standardized naming convention:

service-name.namespace.svc.cluster.local

These internal fully qualified domain names (FQDNs) ensure that Pods can reliably discover and communicate with other services within the cluster. The OpenShift DNS Operator manages internal DNS resolution by leveraging system-level components like dnsmasq or bind. As a system-level operator, it handles internal DNS queries efficiently, maintaining consistent performance and reliability.

Externally, the exposure of applications is achieved through Route objects. When a Route is created, OpenShift automatically generates a domain name following a structured convention:

<route-name>-<namespace>.apps.<cluster-domain>

For example, creating a route named myapp in the dev namespace on a cluster with the base domain ocp4.example.com results in:

myapp-dev.apps.ocp4.example.com

However, although OpenShift internally generates this route, making it resolvable externally typically requires further action. External DNS entries need to be configured either manually or via OpenShift’s ExternalDNS Operator. The ExternalDNS Operator seamlessly integrates with external DNS providers, such as AWS Route 53 or Azure DNS, automating public DNS record management.

To accommodate enterprise-level requirements, OpenShift supports extensive DNS customization options. Organizations can implement custom root domains or wildcard DNS entries for enhanced control over application routing and branding. Custom DNS management can be carried out manually or through automated API-driven approaches, provided proper DNS delegation and TLS certificate management procedures are established.

Starting with OpenShift 4.x, DNS configurations have become more flexible and are managed natively through Kubernetes Custom Resource Definitions (CRDs). This advancement allows administrators to easily define DNS forwarding rules, create private DNS zones, specify upstream resolvers, and utilize sophisticated DNS capabilities such as split-horizon DNS or namespace-specific DNS policies. Internal DNS queries within the cluster rely exclusively on the reserved cluster.local domain, whereas external DNS resolutions depend upon the cluster’s configured base domain. All DNS operations strictly adhere to OpenShift’s RBAC policies and namespace isolation principles, ensuring secure, compliant, and reliable DNS resolution.

Network Structure in terms of Security

Network security within OpenShift is enforced primarily through NetworkPolicies, a native Kubernetes mechanism designed to manage and control Pod-to-Pod and external communication. By default, many Container Network Interface (CNI) implementations allow unrestricted communication between Pods across namespaces and nodes. NetworkPolicies provide administrators the capability to override this behavior, implementing fine-grained rules based on criteria such as Pod labels, namespaces, IP address blocks, ports, and protocols.

A common scenario involves securing database services to ensure only authorized application components can access them. For instance, consider an environment where you have an application Pod labeled app=backend and another administrative Pod labeled app=admin. To ensure that only the backend application Pods can access a database Pod labeled app=database on port 5432/TCP, a specific NetworkPolicy would be configured. This policy explicitly allows access from the backend Pods while denying all other traffic, effectively creating a robust network security boundary.

NetworkPolicies in OpenShift are expressed through standard k8s YAML manifests and can be deployed using the command-line interface (oc apply -f <yaml_file>). Enforcement of these policies depends on the networking layer’s support, with plugins such as OVN-K8s fully compliant with K8s NetworkPolicies. Additionally, administrators can manage NetworkPolicies visually through the OpenShift Web Console, streamlining the process of defining and reviewing Pod and service traffic rules.

It’s crucial to recognise that the effectiveness and default behavior of NetworkPolicies can vary significantly depending on the underlying CNI plugin in use. Certain plugins operate under a permissive allow all by default model, whereas others implement a more restrictive deny by default strategy. In the restrictive model, explicit rules must be defined to allow necessary traffic, significantly influencing the cluster’s default security posture. Understanding these distinctions is vital when planning secure deployments, especially in multi-tenant or regulated environments.

Conclusion

So, that’s OpenShift networking in a nutshell! We’ve seen how OpenShift builds on standard k8s networking, adding powerful features like OVN-K8s for scalability, Service Mesh for secure microservices communication, and flexible DNS management for internal and external service discovery. NetworkPolicies help you define clear security boundaries, while Egress management ensures controlled external access. At the end of the day, getting comfortable with these concepts will make managing your OpenShift cluster easier and more secure—letting you focus less on infrastructure headaches and more on building great apps.

OKD > The Open-Source Base of OpenShift

Let’s start with the core of OpenShift. OKD is the open-source, community-supported version of OpenShift. It was previously called “OpenShift Origin” and serves as the base for Red Hat’s commercial products. Red Hat first builds OKD, then adds extra security, management tools, and support to create its enterprise solutions.

Why is it called “OKD”? It’s not an abbreviation. Red Hat isn’t allowed to use “Kubernetes” in its product names because of Linux Foundation’s rules. So, they named the community version “OKD.” It runs on physical servers, virtual machines, and private cloud setups, relying completely on community support. It also receives experimental updates and features first. Later, if these features are stable, they often appear in Red Hat’s commercial versions. This means OKD is very flexible for testing new ideas, but it does not have official Red Hat support. Companies that need a strong Service Level Agreement usually pick a commercial version.

OCP > OpenShift Container Platform by Red Hat

If you want to use OpenShift in a business setting, OCP is the best choice. You can think of OCP as the professional version of OKD, provided by Red Hat. It comes with better security, enterprise support, and extra management tools—features that make many companies trust OCP for running securely on physical servers, virtual machines, and private cloud environments. All managed OpenShift services (OSD, ROSA, ARO) are built on OCP, solidifying it as the core of the enterprise OpenShift experience. Along with official support and regular updates, Red Hat includes special security features such as RBAC and Security Context Constraints, which let teams manage permissions precisely and help keep containers more secure.

Building on those strong enterprise capabilities, newer versions of OpenShift introduced an operator-based architecture. Operators automate tasks like installing software, monitoring services, and upgrading them inside the cluster, reducing manual work for admins. OpenShift offers Source-to-Image, a feature that can convert application code directly into a container image. This streamlines the process of moving from source code to a running container, allowing developers to focus more on coding and less on configuration.

To complete the picture, OpenShift seamlessly integrates with popular CI/CD tools such as Jenkins, Tekton, and ArgoCD. This means you can set up pipelines that automatically build and deploy code, so when developers push changes to a Git repository, OpenShift takes care of building new container images and rolling out updates. GitOps tools like ArgoCD help keep infrastructure in sync with the code, enhancing reliability. By combining these features, OpenShift provides a comprehensive ecosystem that supports both operational efficiency and developer productivity.

OSD, ROSA, and ARO > Managed OpenShift Services

If you don’t want to manage your own infrastructure and prefer a fully managed OpenShift experience, OSD, ROSA, and ARO are good options.

OSD (OpenShift Dedicated) → A managed OpenShift version that runs on AWS and GCP, fully handled by Red Hat. No need for internal management, Red Hat takes care of everything.

ROSA (Red Hat OpenShift Service on AWS) → A fully integrated OpenShift service provided by AWS and Red Hat together. It works closely with AWS services while keeping the OpenShift experience.

ARO (Azure Red Hat OpenShift) → A managed OpenShift service on Microsoft Azure, run by Red Hat and Microsoft. It integrates well with Azure resources and works smoothly with Microsoft services.

These services are great for companies that want to start using OpenShift quickly without dealing with infrastructure.

With these managed services, you can focus on developing and scaling your apps. Red Hat and the cloud providers handle security patches, upgrades, and hardware management. This saves time and resources for your team.

Edge Computing and Virtualization > Expanding OpenShift’s Capabilities

OpenShift is not just for big data centers and cloud environments. It also works well for edge computing and virtualization.

OpenShift Edge Computing → Allows small-scale OpenShift setups for IoT, 5G, and retail apps. It provides a lightweight and optimized OpenShift experience for remote locations.

OpenShift Virtualization → Lets you run virtual machines on K8s. This makes it easier to move traditional data center workloads to OpenShift and K8s environments.

These options are useful for businesses that want to support hybrid and multi-cloud setups. For example, a company with a main data center could run OCP for primary workloads, while using Edge clusters in different branch offices for local data processing. If the company still needs some old VMs, they can run those inside OpenShift too, thanks to OpenShift Virtualization.

How OpenShift Versions Connect

Let’s look at how OpenShift versions relate to each other. So Hierarchy is:

OKD → OCP → OSD, ROSA, ARO, Edge, Virtualization

This means:

✓ OKD is the foundation of all OpenShift versions

✓ OCP is the commercial, Red Hat-supported version of OKD

✓ OSD, ROSA, and ARO are fully managed cloud services built on OCP

✓ Edge and Virtualization expand OpenShift’s use cases

This structure lets you pick the version or service that fits your organization. Some people start with OKD and later switch to OCP. Others jump straight to a managed version like OSD or ARO. The choice depends on your team’s size, budget, and need for official support.

Many cloud providers offer their own managed K8s services, like Amazon EKS, Azure AKS, or Google GKE. OpenShift adds extra features on top of vanilla K8s, such as built-in security rules, S2I, and an operator-based architecture. If you need enterprise support, Red Hat’s OCP or the managed OpenShift services may be more suitable. But if you just want basic K8s with less overhead, EKS, AKS, or GKE might be enough.

Conclusion

OpenShift is a flexible K8s platform with many setup and management options. You can choose OKD, OCP, OSD, ROSA, ARO, Edge, or Virtualization depending on your goals.

✓ If you need a community version without official support, go with OKD

✓ If you need enterprise support and extra features, OCP is the best choice

✓ If you want a fully managed experience, go for OSD, ROSA, or ARO

✓ If you want a smaller setup or special use cases, consider Edge or Virtualization

No matter which one you pick, OpenShift can help you simplify container management and scale your applications. By combining Red Hat’s resources, a strong community, and K8s innovations, OpenShift provides a robust platform for modern software development.

1 > Prerequisites

Before starting, ensure:

☞ You have macOS (>= 13.x)

☞ You have at least 16GB RAM and 35GB disk space

☞ You have a Red Hat account to obtain the pull secret

Go to the URL in the image above and create a user account, making sure to save all the necessary information (name, address, phone number).

And you should see this page after you have made the relevant confirmations in your email.

On the same page, click on Cluster List on the left and see the info above.

To use OpenShift locally, download the necessary files according to the respective OS.

Don’t forget to download the secret right afterwards.

So far we have downloaded the OpenShift Local File and the pull secret file, now let’s go back to the terminal.

2 > Installation of CRC

Step 1 >> Download and Install CRC

Install via Installer Package

cd ~/Downloads # your files may have been downloaded elsewhere
sudo installer -pkg crc-macos-installer.pkg -target /

Or install manually (it is easiest and healthiest to start with the above method)

# skip this part if you downloaded the files from the official website
curl -L -o ~/Downloads/crc-macos.tar.xz <https://mirror.openshift.com/pub/openshift-v4/clients/crc/latest/crc-macos-amd64.tar.xz>
cd ~/Downloads
tar -xvf crc-macos.tar.xz
sudo mv crc-macos-*/crc /usr/local/bin

3 > Initial CRC Setup

crc setup -h # to take a look at some important info
crc setup

This command prepares the environment by configuring the hypervisor, networking, and security settings.

4 > Pull Secret Configuration (Avoid Entering Every Time)

Download the pull-secret from Red Hat OpenShift Console (If you did not do it!)

👉 Link

Move it to a fixed location:

# so it doesn't ask you for a secret every time you log in
mv ~/Downloads/pull-secret.txt ~/.crc/pull-secret.txt 
crc config set pull-secret-file ~/.crc/pull-secret.txt

Verify:

crc config view | grep pull-secret-file 
# you see your secret, for example: - pull-secret-file: /Users/yourUserName/Downloads/pull-secret.txt

5 > Start OpenShift Cluster

crc start

This will:

☞ Start the OpenShift Virtual Machine

☞ Apply the pull-secret

☞ Configure networking

☞ Deploy OpenShift components

❯ crc start
INFO Using bundle path /Users/yourUserName/.crc/cache/crc_vfkit_4.17.10_arm64.crcbundle
INFO Checking if running macOS version >= 13.x
INFO Checking if running as non-root
INFO Checking if crc-admin-helper executable is cached
INFO Checking if running on a supported CPU architecture
INFO Checking if crc executable symlink exists
INFO Checking minimum RAM requirements
INFO Check if Podman binary exists in: /Users/mesutoezdil/.crc/bin/oc
INFO Checking if running emulated on Apple silicon
INFO Checking if vfkit is installed
INFO Checking if old launchd config for tray and/or daemon exists
INFO Checking if crc daemon plist file is present and loaded
INFO Checking SSH port availability
INFO Loading bundle: crc_vfkit_4.17.10_arm64...
INFO Creating CRC VM for OpenShift 4.17.10...
INFO Generating new SSH key pair...
INFO Generating new password for the kubeadmin user
INFO Starting CRC VM for openshift 4.17.10...
INFO CRC instance is running with IP 127.0.0.1
INFO CRC VM is running
INFO Updating authorized keys...
INFO Configuring shared directories
INFO Check internal and public DNS query...
WARN Failed public DNS query from the cluster: ssh command error:
command : curl --head <https://quay.io>
err     : Process exited with status 60:
INFO Check DNS query from host...
INFO Verifying validity of the kubelet certificates...
INFO Starting kubelet service
INFO Waiting for kube-apiserver availability... [takes around 2min]
INFO Adding user's pull secret to the cluster...
INFO Updating SSH key to machine config resource...
INFO Waiting until the user's pull secret is written to the instance disk...
INFO Changing the password for the kubeadmin user
INFO Updating cluster ID...
INFO Updating root CA cert to admin-kubeconfig-client-ca configmap...
INFO Starting openshift instance... [waiting for the cluster to stabilize]
INFO All operators are available. Ensuring stability...
INFO Operator console is progressing
INFO Operator authentication is not yet available
INFO Operator authentication is not yet available
INFO Operator authentication is not yet available
INFO Operator authentication is degraded
INFO Operator authentication is degraded
INFO Operator authentication is degraded
INFO All operators are available. Ensuring stability...
INFO Operators are stable (2/3)...
INFO Operators are stable (3/3)...
INFO Adding crc-admin and crc-developer contexts to kubeconfig...
Started the OpenShift cluster.

The server is accessible via web console at:
  <https://console-openshift-console.apps-crc.testing>

Log in as administrator:
  Username: kubeadmin
  Password: FZIsX-XXXXX-YYYYY-ZZZZZ

Log in as user:
  Username: developer
  Password: developer

Use the 'oc' command line interface:
  $ eval $(crc oc-env)
  $ oc login -u developer <https://api.crc.testing>:XXXX

6 > Automate Terminal Commands in .zshrc (If you use bash, adapt the commands to it)

To avoid repeating commands, add these to ~/.zshrc:

echo 'eval $(crc oc-env)' >> ~/.zshrc
echo 'alias crcup="crc start"' >> ~/.zshrc
echo 'alias crcdown="crc stop"' >> ~/.zshrc
echo 'alias crcdel="crc delete && crc setup"' >> ~/.zshrc
echo 'alias crcoc="eval $(crc oc-env)"' >> ~/.zshrc
echo 'oc login -u kubeadmin -p FZIsX-97VhH-gZuRA-QQgkS <https://api.crc.testing:6443>' >> ~/.zshrc
echo 'oc login -u developer -p developer <https://api.crc.testing:6443>' >> ~/.zshrc
source ~/.zshrc

Now you can:

☞ crcup → Start OpenShift

☞ crcdown → Stop OpenShift

☞ crcdel → Reset CRC

☞ crcoc → Load OpenShift CLI

7 > Login to OpenShift (Admin & Developer)

Admin Login (Admin and Developer differences below)

oc login -u kubeadmin -p FZIsX-XXXXX-YYYYY-ZZZZZ <https://api.crc.testing>:XXXX

# output
Login successful.

You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".

Developer Login

oc login -u developer -p developer <https://api.crc.testing>:XXXX

# output
Login successful.

You don't have any projects. You can try to create a new project, by running

    oc new-project <projectname>

8 > OpenShift Web Console & Cluster Info

Access Web Console

🔗 Open

To get credentials:

crc console --credentials

List Nodes

oc get nodes

List Running Pods

oc get pods --all-namespaces

9 > Stopping and Restarting OpenShift

Stop CRC (Shutdown OpenShift)

crc stop

Delete and Reset CRC

crc delete
crc cleanup
crc setup

10 > Restarting OpenShift from Scratch (If needed)

After a restart:

crcup
oc login -u kubeadmin -p FZIsX-XXXXX-YYYYY-ZZZZZ <https://api.crc.testing>:XXXX

11 > Difference Between Admin & Developer Users

• Manage Nodes

  • Admin (kubeadmin): ✅ Yes ⇢ Developer: ❌ No

• Create Projects

  • Admin (kubeadmin): ✅ Yes ⇢ Developer: ✅ Yes

• Access Web UI

  • Admin (kubeadmin): ✅ Yes ⇢ Developer: ✅ Yes

• View Logs

  • Admin (kubeadmin): ✅ Yes ⇢ Developer: ❌ No

• Modify Cluster Configuration

  • Admin (kubeadmin): ✅ Yes ⇢ Developer: ❌ No

12 > Debugging & Troubleshooting

Check Cluster Logs

crc status
oc get pods --all-namespaces

If Cluster Fails to Start

crc cleanup
crc setup
crc start

Check Configs

crc config view

Stopping and Restarting CRC Like It's a New Day

Now, let’s test what happens when you stop or completely delete CRC and then restart it as if it were a new day.

✘ Stopping CRC (Shutdown)

To stop CRC without losing data:

crc stop

✓ This will shut down the OpenShift VM, but keep all configurations and data.

✓ The OpenShift web console will become inaccessible.

✓ You can start it again later without reinstalling anything.

Check in your browser: Go to here.

It should show a “site unreachable” error.

! Completely Deleting CRC

If you want to completely remove CRC, including all stored configurations and data:

crc delete
crc cleanup

This will wipe everything and require a full reinstallation next time.

Restarting CRC as If It’s a New Day

Let’s assume you turned off your computer and now want to start everything from scratch the next day.

Open a terminal and start CRC:

crcup

(This is an alias in .zshrc that runs crc start)

Login to OpenShift:

oc login -u kubeadmin -p FZIsX-XXXXX-YYYYY-ZZZZZ <https://api.crc.testing>:XXXX

(Or log in as developer if needed)

Open the OpenShift web console again.

The dashboard should now be accessible again!

Want a Faster Startup?

If you don’t want to reinstall everything, don’t delete CRC. Instead, just:

crc stop  # stop CRC
crc start  # start CRC again

This way, you don’t have to set up everything from scratch each time.

CRC Storage Usage Analysis & Growth Impact

Current Storage Usage

From your command:

du -sh ~/.crc

Breakdown:

du -sh ~/.crc/*

Will Storage Usage Increase Over Time?

Yes, as you create new pods, store images, and persist data, the storage usage will increase. Factors That Increase Storage:

☞ Pods & Containers

• New pods require space for logs, configs, and runtime storage

• Pods that pull new container images will increase the cache size

☞ Persistent Volumes & PVCs

• If you use Persistent Volume Claims (PVCs), CRC will allocate additional storage

• Large databases or applications using storage-backed PVCs consume significant space

☞ Downloaded Container Images

• When pulling images from quay.io, Docker Hub, or Red Hat Registry, they get stored inside CRC

• More images = higher storage consumption

☞ Logs & Metrics Data

• OpenShift generates logs from different services (like kube-apiserver, etcd)

• Continuous logging = gradual storage increase

How to Prevent Storage Bloat?

To check OpenShift storage usage:

oc adm top nodes
oc adm top pods

To list Persistent Volumes:

oc get pvc

How to Free Up Space?

i. Delete Unused Pods

oc delete pod <pod-name>

To delete all pods in a namespace:

oc delete pod --all -n <namespace>

ii. Remove Unused Container Images

podman images
podman rmi <image-id>

To prune all unused images:

podman system prune -a

iii. Delete Unused Persistent Volumes

oc delete pvc <pvc-name>

vi. Clean Logs & CRC Cache

rm -rf ~/.crc/logs/*
rm -rf ~/.crc/cache/*

v. Reset CRC to Free Up Disk Space

crc delete
crc cleanup
crc setup

WARNING: This will reset CRC and require reconfig.

Use aliases for quick cleanup & reset:

echo 'alias crcreset="crc delete && crc cleanup && crc setup"' >> ~/.zshrc

Conclusion

That’s it! Now you know how to install, set up, and manage OpenShift CRC on your Mac. With these simple steps, you’re ready to quickly create, test, and deploy applications locally.

🎶 I have successfully executed these commands in my own environment step by step. If you encounter any issues, please let me know in the comments.

1 > Why CIS Benchmark?

CIS Benchmark provides recommended rules for securing k8s. If you follow these rules, you reduce risks such as unauthorized access, data leaks, or misconfigs. The kube-bench tool checks your cluster against these rules and gives you a report. In that report, you’ll see:

✔︎ PASS: The item meets the recommendation

✔︎ FAIL: The item does not meet the recommendation

✔︎ WARN: The item might be risky or needs attention

✔︎ INFO: The item is just for info

2 > Prerequisites

☞ A working k8s cluster (for example, one control-plane node and two worker nodes).

☞ The kubectl command to interact with the cluster. It is my env (on iximiuz):

k get pods
No resources found in default namespace.

k get nodes
NAME        STATUS   ROLES           AGE   VERSION
cplane-01   Ready    control-plane   25s   v1.32.1
node-01     Ready    <none>          13s   v1.32.1
node-02     Ready    <none>          13s   v1.32.1

3 > Download the kube-bench YAML Files

We need two YAML files to run the checks:

wget -O ar-kube-control-plane.yaml <https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-master.yaml>

wget -O ar-kube-node.yaml <https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job-node.yaml>

ar-kube-control-plane.yaml tests the control-plane components and ar-kube-node.yaml tests the worker nodes. You can open these files (for example, cat ar-kube-control-plane.yaml) to see the details.

4 > Run the CIS Benchmark Tests

Use these commands to create the Jobs in your cluster:

kubectl create -f ar-kube-control-plane.yaml

kubectl create -f ar-kube-node.yaml

This starts two Jobs: one for the control-plane and one for the node checks. Run kubectl get pods to see the Pods created by these Jobs. They should have a Completed status once done.

5 > Save and View the Test Results

When the Pods are complete, get their logs:

kubectl logs <Control Plane Pod> > ar-kube-results-control-plane.log

kubectl logs <Node Pod> > ar-kube-results-node.log

Replace <Control Plane Pod> and <Node Pod> with the real Pod names. Then:

cat ar-kube-results-control-plane.log

cat ar-kube-results-node.log

Look for FAIL, WARN, or PASS. FAIL means you need to fix something. Output is too long to be included here.

6 > Example Findings

In this section, we look at common issues you might see in your kube-bench results. Each item shows a part of the CIS Benchmark and a hint about why it matters and how to fix it. Here are a few examples:

1.1.12: “Ensure etcd data directory is owned by etcd:etcd” → Why it matters: etcd stores cluster data (like states, secrets, or config details). If other users can change or read the etcd data directory, they could tamper with cluster info. But here is the solution: Make sure you run this command on the node where etcd is installed. After changing ownership, you may need to restart etcd. This ensures that etcd picks up the new file ownership correctly.

chown etcd:etcd /var/lib/etcd

systemctl daemon-reload

systemctl restart etcd

1.2.5: “Ensure the --kubelet-certificate-authority argument is set as appropriate” → Why it matters: The API server must verify kubelet certificates. If the CA (Certificate Authority) is not set, anyone could pose as a kubelet. Here is the solution: Identify the CA file (often found in /etc/kubernetes/pki/ca.crt). Then edit the API server manifest (usually /etc/kubernetes/manifests/kube-apiserver.yaml). And add or modify:

- --kubelet-certificate-authority=/etc/kubernetes/pki/ca.crt

After saving, the kubelet usually restarts the API server pod automatically.

4.1.1: “Ensure the kubelet service file permissions are 600” → Why it matters: If the kubelet service file is world-readable or writable, an attacker could modify the service and gain control. Here is the solution: These commands restrict access so that only the root user can edit the kubelet service file, and then restart the kubelet.

chmod 600 /lib/systemd/system/kubelet.service

systemctl daemon-reload

systemctl restart kubelet

4.1.9: “If the kubelet config.yaml configuration file is being used, validate permissions are set to 600” → Why it matters: Kubelet’s config.yaml can contain sensitive details, such as TLS settings or cluster server addresses. Here is the solution: This ensures that only the root user can read or change the file.

chmod 600 /var/lib/kubelet/config.yaml

systemctl daemon-reload

systemctl restart kubelet

These are just a few examples. Your actual report may contain many other checks with different IDs and messages. Always review each FAIL or WARN item and see the suggestions in the output. That way, you can fix or improve your cluster’s security.

7 > Re-run Tests After Fixes

Once you fix the items in the report, you can re-run the tests:

kubectl delete -f ar-kube-control-plane.yaml

kubectl delete -f ar-kube-node.yaml

Then:

kubectl create -f ar-kube-control-plane.yaml

kubectl create -f ar-kube-node.yaml

This lets you see if FAIL items are now PASS.

8 > Some Advanced Tips

Sometimes, you don’t need to run every single CIS check. You can focus on certain items or skip checks that do not apply to your setup. The kube-bench tool supports this flexibility. For example, if you only want to test the control plane, you can run kube-bench run --targets master, which checks only master-related items. If you want to test a specific control, such as “1.2.5,” just add --check 1.2.5. That way, you quickly see if your fix for that one issue is correct. If there are checks you don’t need, you can skip them with --skip. For instance, kube-bench run --targets node --skip 4.1.1,4.1.9 ignores checks 4.1.1 and 4.1.9. Because commands can change with different kube-bench versions, you should also check the official usage docs or run kube-bench --help to stay updated.

9 > Additional Tips

i- Backup configs ☞ Before changing any configuration, make a copy.

ii- Go step by step ☞ Fix a few items at a time and re-check.

iii- Keep reading docs ☞ Kubernetes Official Docs and CIS Benchmark Docs

iv- Automation ☞ Consider adding kube-bench to a CI/CD pipeline.

9 > Conclusion

Using kube-bench with the CIS Benchmark is a simple way to check and improve your k8s security. By following the recommendations, you lower the risk of attacks and make sure your cluster is set up with best practices in mind.

Special thanks to ArdanLabs for their amazing educational videos on k8s and Golang, and for their support in this journey. ArdanLabs is a well-known technology company specializing in software development, training, and consulting, particularly in Golang and Kubernetes. They provide high-quality educational content, workshops, and courses to help developers and enterprises build efficient and scalable apps.

This guide will walk you through the process of creating a default deny NetworkPolicy to block all network traffic to and from Pods in a namespace. Along the way, you will gain a deeper understanding of NetworkPolicies and how to fine-tune them for your security needs.

🎶 I have successfully executed these commands in my own environment step by step. If you encounter any issues, please let me know in the comments.

How to Apply This in Your Own Environment

If you want to try this setup in your own k8s cluster, follow these steps:

1. Ensure you have a Kubernetes cluster running (such as Minikube, Kind, or a managed cloud service like AKS, EKS, or GKE).

2. Install kubectl and configure it to connect to your cluster.

3. Verify that your cluster is working properly by running:

This should list all the nodes in your cluster.

kubectl get nodes

4. If your cluster does not have a CNI plugin that supports NetworkPolicy (e.g., Calico, Cilium), install one. For example, to install Calico:

kubectl apply -f <https://docs.projectcalico.org/manifests/calico.yaml>

4. Once your cluster is ready, proceed with the following steps.

Create a New Namespace

To maintain isolation, create a dedicated namespace for this test:

kubectl create namespace arkube-ns

This ensures that all the resources we create in this tutorial remain isolated from other workloads in the cluster.

Deploy a Simple Nginx Web Server

Now that we have a namespace, let's deploy an Nginx web server inside the arkube-ns namespace.

1. Create a file named arkube-nginx.yml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: arkube-nginx  # Name of the deployment
  namespace: arkube-ns  # Namespace where this deployment will reside
spec:
  replicas: 1  # Number of Nginx replicas
  selector:
    matchLabels:
      app: arkube-nginx  # Selector label for identifying pods
  template:
    metadata:
      labels:
        app: arkube-nginx  # Label applied to the created pods
    spec:
      containers:
      - name: nginx  # Name of the container
        image: nginx:1.14.2  # Image version to use for Nginx
        ports:
        - containerPort: 80  # Exposing port 80 inside the container

1. Apply the deployment:

kubectl apply -f arkube-nginx.yml

This deploys an Nginx pod, which we will use to test connectivity and NetworkPolicies.

Get the Cluster IP Address of the Nginx Pod

Since we need a target for network testing, retrieve the IP address of the Nginx Pod:

kubectl get pods -n arkube-ns -o wide

Take note of the IP address displayed, as we will use it in the next step.

Create a Test Client Pod

We’ll create a client Pod that continuously tries to access the Nginx server to test network connectivity.

1. Create a file named arkube-client.yml with the following content:

apiVersion: v1
kind: Pod
metadata:
  name: arkube-client  # Name of the client pod
  namespace: arkube-ns  # Namespace where the client pod is deployed
  labels:
    app: arkube-client  # Label for identifying the client pod
spec:
  containers:
  - name: busybox  # Name of the container
    image: radial/busyboxplus:curl  # Lightweight image with curl support
    command: ['sh', '-c', 'while true; do curl -m 3 <Nginx Pod Cluster IP address>; sleep 5; done']  # Looping curl request for connectivity testing

1. Replace <Nginx Pod Cluster IP address> with the IP address obtained in Step 3.

2. Apply the client Pod:

kubectl create -f arkube-client.yml

Verify Connectivity

Now, confirm that the client Pod can reach the Nginx server:

kubectl logs -n arkube-ns arkube-client

If the connection is successful, you should see HTTP response data from the Nginx server.

Create a Default Deny NetworkPolicy

Next, we’ll create a default deny NetworkPolicy to block all incoming and outgoing traffic in the arkube-ns namespace.

1. Create a file named arkube-deny-all.yml with the following content:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: arkube-deny-all  # Name of the NetworkPolicy
  namespace: arkube-ns  # Namespace where this policy will be applied
spec:
  podSelector: {}  # Apply to all pods in the namespace
  policyTypes:
  - Ingress  # Deny incoming traffic
  - Egress  # Deny outgoing traffic

1. Apply the NetworkPolicy:

kubectl create -f arkube-deny-all.yml

Verify the NetworkPolicy

Check the logs of the client Pod again. If the NetworkPolicy is working, you should see Connection timed out errors:

kubectl logs -n arkube-ns arkube-client

Troubleshooting NetworkPolicy Issues

If the client Pod can still reach the Nginx server after applying the arkube-deny-all policy, check the following:

1. Client still reaches Nginx:

   • Possible cause: No CNI plugin installed or it is misconfigured.

   • Solution: Install a NetworkPolicy-aware CNI like Calico:

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

1. Policy not applied:

   • Possible cause: The Pod needs to be restarted.

   • Solution: Restart the client Pod:

kubectl delete pod arkube-client -n arkube-ns && kubectl create -f arkube-client.yml

Allow Specific Traffic

To allow only the client Pod to access the Nginx server, create this policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: arkube-allow-client  # Name of the NetworkPolicy
  namespace: arkube-ns  # Namespace where this policy will be applied
spec:
  podSelector:
    matchLabels:
      app: arkube-nginx  # Apply this policy to Nginx pods
  policyTypes:
  - Ingress  # Allow incoming traffic
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: arkube-client  # Allow traffic only from client pods

Apply the policy:

kubectl create -f arkube-allow-client.yml

Key Takeaways:

• A default deny policy ensures that no traffic is allowed unless explicitly permitted

• NetworkPolicy enforcement requires a CNI that supports NetworkPolicies

• Use troubleshooting steps to diagnose misconfigurations

• Always test policies with a client-server setup before applying them to production

• Consider combining NetworkPolicies with RBAC for enhanced security

Stay safe!

Relevant Documentation

• Kubernetes Network Policies

In this article, we will explore k8s security from the perspective of the “Top 10 Web Hacking Techniques of 2024,” published by James Kettle on PortSwigger on February 4, 2025.

10 > Hijacking OAuth Flows via Cookie Tossing

OAuth is everywhere—k8s included. This hack shows how attackers can mess with cookies across subdomains to hijack sessions. In a k8s setup, where many services share subdomains, it could let someone sneak into your cluster or steal sensitive data. One way to tighten cookie policies is by adding an annotation like:

kubectl annotate ingress my-ingress nginx.ingress.kubernetes.io/proxy-cookie-domain="myapp.example.com"

which enforces stricter domain rules. You can also add the HttpOnly and Secure flags:

kubectl annotate ingress my-ingress nginx.ingress.kubernetes.io/proxy-cookie-path="/; HttpOnly; Secure"

to protect cookies from being accessed by client-side scripts or sent over non-HTTPS connections.

9 > ChatGPT Account Takeover – Wildcard Web Cache Deception

Web cache deception isn’t new, but this twist is next-level. By exploiting how caches handle paths, attackers can sneak into places they shouldn’t be. For k8s, where caching helps with speed, this could expose secrets or lead to account takeovers. Disabling caching for sensitive paths via:

kubectl annotate ingress my-ingress nginx.ingress.kubernetes.io/cache-control="no-store, no-cache, must-revalidate"

helps prevent caching of critical endpoints. If you’re using a tool like Varnish, you can specify certain paths not to cache at all by setting something like:

helm upgrade my-cache-chart stable/varnish --set varnish.no_cache="/login,/api/secret"

8 > OAuth Non-Happy Path to ATO

By manipulating the Referer header, attackers can trick OAuth into handing over access. In k8s, where OAuth is used for service authentication, this could let attackers impersonate services. You can enforce Referer validation with a snippet such as:

kubectl annotate ingress my-ingress nginx.ingress.kubernetes.io/configuration-snippet="if ($http_referer !~* '^https://trusteddomain.com') { return 403; }"

Also, rotating OAuth secrets and restricting callback URLs via:

kubectl create secret generic oauth-config --from-literal=client_id='NEW_CLIENT_ID' --from-literal=client_secret='NEW_SECRET'

reduces the risk of unauthorized redirection.

7 > CVE-2024-4367 – Arbitrary JavaScript Execution in PDF.js

PDF.js, often used in apps running on k8s, can let attackers run malicious JavaScript if left unpatched. Updating the PDF.js container image to a patched version (for example, 2.18.5 or newer) with:

helm upgrade my-app ./chart --set pdfjs.version=2.18.5

helps close known holes. Running as a non-root user with a read-only filesystem:

securityContext:
  runAsNonRoot: true
  readOnlyRootFilesystem: true

further limits potential damage.

6 > DoubleClickjacking: A New Era of UI Redressing

Clickjacking returns, and in k8s, where admins rely on web dashboards, it can trick you into unintended actions. Adding a Content Security Policy through an annotation such as:

kubectl annotate ingress dashboard-ingress nginx.ingress.kubernetes.io/configuration-snippet="add_header Content-Security-Policy \\"frame-ancestors 'none'\\";"

helps ensure your dashboard can’t be iframed by malicious sites. You can also lock down dashboard access with RBAC:

kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --group=admin

to limit who can access critical controls in the first place.

5 > Exploring the DOMPurify Library: Bypasses and Fixes

DOMPurify is a popular way to sanitize HTML and stop XSS, but new bypass techniques keep emerging. Upgrading DOMPurify regularly, for example via:

npm install dompurify@latest

helps ensure you’re running the latest patched version. Combining client-side sanitization with server-side protections, like adding:

kubectl annotate ingress my-ingress nginx.ingress.kubernetes.io/configuration-snippet="add_header X-XSS-Protection '1; mode=block';"

provides extra layers of defense.

4 > WorstFit: Unveiling Hidden Transformers in Windows ANSI

Charset conversion vulnerabilities are less obvious but can allow malicious payloads to sneak in. In Kubernetes, where multi-platform interactions are common, forcing UTF-8 can help:

kubectl annotate ingress my-ingress nginx.ingress.kubernetes.io/proxy-set-headers="Content-Type: application/json; charset=utf-8"

and validating encodings at the application level (rejecting non-UTF-8 requests) cuts down on hidden character attacks.

3 > Unveiling TE.0 HTTP Request Smuggling

HTTP request smuggling has a new variant, TE.0, which can let attackers bypass security or poison caches in k8s. Upgrading your ingress controller:

helm upgrade nginx-ingress ingress-nginx/ingress-nginx --version latest

ensures you have the latest patches. Enabling strict header parsing with:

kubectl annotate ingress my-ingress nginx.ingress.kubernetes.io/enable-strict-http-headers="true"

prevents ambiguous Transfer-Encoding or Content-Length headers from being exploited.

2 > SQL Injection Isn't Dead: Smuggling Queries at the Protocol Level

SQL injection keeps evolving. In k8s, where databases power many apps, parameterized queries and regular audits are essential. You can also use policies like Kyverno to enforce best practices:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-parameterized-queries
spec:
  validationFailureAction: enforce
  rules:
    - name: check-sql-queries
      match:
        resources:
          kinds:
            - Deployment
      validate:
        message: "Use parameterized queries for SQL calls."

Meanwhile, rotating database credentials and storing them as Secrets:

kubectl create secret generic db-credentials --from-literal=username='dbUser' --from-literal=password='S3cur3Pa$$'

further limits damage if an injection does occur.

1 > Confusion Attacks: Exploiting Hidden Semantic Ambiguity in Apache HTTP Server

Apache HTTP Server is common in k8s, often as a reverse proxy or ingress. Confusion attacks exploit hidden ambiguities to bypass security. Keeping Apache up to date (for instance, FROM httpd:2.4.57 in your Dockerfile) and enabling ModSecurity in httpd.conf:

LoadModule security2_module modules/mod_security2.so
<IfModule security2_module>
  SecRuleEngine On
</IfModule>

will help catch ambiguous or malformed requests before they can cause trouble.

What This Means for Kubernetes in 2025

The 2024 Top 10 Web Hacking Techniques list is a reminder that the threat landscape never stops evolving. For k8s teams, it means staying proactive and prepared. Patch everything, harden your configurations, and monitor for unusual behavior. Keep up with new research and adapt your defenses accordingly. K8s is powerful, but it’s also a big target—learning from these top 10 techniques will help you stay ahead of the curve in 2025 and beyond.

As important as building a strong infrastructure is, it’s crucial to remember that attackers are constantly developing more advanced techniques to infiltrate your systems. Ensuring k8s security requires staying up to date and continuously adapting.

This article shows exactly that risk: (i) Vulnerable Scenario: A ServiceAccount with cluster-admin rights, (ii) Secure Scenario: A safer approach, including minimal RBAC, non-root containers, and optional network restrictions. Let me give you a little spoiler beforehand:

Each step highlights what is happening and why it matters.

PART I – VULNERABLE SCENARIO

In this first part, we create resources in the default namespace, intentionally exposing them to a big security hole.

Over-Privileged ServiceAccount: debug-sa.yaml

A ServiceAccount is like an “identity” for pods. Giving it cluster-admin means your pods can do anything in the cluster—list secrets, modify nodes, etc.

############################################
# debug-sa.yaml
# over-privileged sa with cluster-admin role
############################################
apiVersion: v1
kind: ServiceAccount
metadata:
  name: debug-sa
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: debug-sa-cluster-admin-binding
subjects:
- kind: ServiceAccount
  name: debug-sa
  namespace: default
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

The ClusterRoleBinding references cluster-admin. This effectively gives root-like power to any pod using debug-sa. If an attacker gains access to the pod, they could control the entire cluster.

“Arkube” Deployment Using debug-sa: arkube-deployment.yaml

We show a normal nginx-based app, but it’s assigned this dangerously powerful serviceAccount. Now the nginx pod can manipulate cluster resources far beyond its legitimate need.

##################################################
# arkube-deployment.yaml
# basic nginx but with cluster-admin via debug-sa
##################################################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: arkube
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: arkube
  template:
    metadata:
      labels:
        app: arkube
    spec:
      serviceAccountName: debug-sa   # over-privileged sa
      containers:
      - name: arkube-container
        image: nginx:alpine
        ports:
        - containerPort: 80

A simple container now has the power to manage or read any resource in the cluster. Attackers can exploit any small vulnerability in the container or app to escalate privileges fully.

A Secret with a DB Password: arkube-db-secret.yaml

In real life, you store sensitive data (API keys, DB credentials) in k8s Secrets. The next step will show how these can be easily exposed if the serviceAccount is too permissive.

##################################################
# arkube-db-secret.yaml
# holds "supersecret" as a base64-encoded password
##################################################
apiVersion: v1
kind: Secret
metadata:
  name: arkube-db-secret
  namespace: default
type: Opaque
data:
  # echo -n 'supersecret' | base64 => c3VwZXJzZWNyZXQ=
  password: c3VwZXJzZWNyZXQ=

Because debug-sa is cluster-admin, any pod using it can read or even modify this secret.

Malicious Pod: malicious-pod.yaml

We simulate an attacker’s behavior: a pod that uses debug-sa to show how it can list secrets. This can be extended to reading the actual secret values, installing cryptominers, or deleting your entire environment.

##################################################
# malicious-pod.yaml
# attacker's pod that repeatedly lists secrets
##################################################
apiVersion: v1
kind: Pod
metadata:
  name: malicious-pod
  namespace: default
spec:
  serviceAccountName: debug-sa
  containers:
  - name: malicious-container
    image: bitnami/kubectl:latest
    command: ["/bin/sh"]
    args:
      - "-c"
      - |
        echo "Malicious pod running..."
        while true; do
          echo "[*] Listing secrets in default namespace:"
          kubectl get secrets -n default
          echo "Sleeping 20s..."
          sleep 20
        done

If you check logs (kubectl logs malicious-pod), you will see your arkube-db-secret among others. Could decode the base64-encoded password, pivot to your database, or read more critical data.

Cleaning Up the Vulnerable Setup

If you need to remove these insecure resources:

• kubectl delete pod malicious-pod

• kubectl delete deployment arkube

• kubectl delete -f debug-sa.yaml

• kubectl delete secret arkube-db-secret

PART II – SECURE SCENARIO

Next, we fix everything by following best practices: (i) Use a dedicated namespace for your app, (ii) Apply minimal RBAC (a Role + RoleBinding) to your ServiceAccount, (iii) Run your container as a non-root user to prevent host-level compromise, (iv) (Optional) Add a NetworkPolicy for restricting traffic within the cluster.

Create a Namespace: arkube

Using a separate namespace is a fundamental practice. It helps keep your resources organized and simplifies the scope of RBAC rules. Isolation and clarity. The “arkube” namespace is where we will deploy everything for this service.

• kubectl create namespace arkube

Minimal RBAC: arkube-sa.yaml

Now we define an ordinary ServiceAccount (arkube-sa), a Role that only lets it “get” and “list” pods, services, and secrets in the “arkube” namespace, and a RoleBinding that associates them.

########################################################
# arkube-sa.yaml
# a minimal role for reading resources, and a roleBinding
########################################################
apiVersion: v1
kind: ServiceAccount
metadata:
  name: arkube-sa
  namespace: arkube
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: arkube-role
  namespace: arkube
rules:
- apiGroups: [""]
  resources: ["pods", "services", "secrets"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: arkube-rolebinding
  namespace: arkube
subjects:
- kind: ServiceAccount
  name: arkube-sa
  namespace: arkube
roleRef:
  kind: Role
  name: arkube-role
  apiGroup: rbac.authorization.k8s.io

Key Differences from the Vulnerable Scenario: (i) No cluster-wide privileges (cluster-admin) and (ii) The ServiceAccount can only read certain resource types (pods, services, secrets) within the arkube namespace. Even if an attacker lands in your pod, they cannot see or alter resources outside this namespace (let alone the entire cluster).

Secure Deployment: arkube-deployment-secure.yaml

We re-deploy “arkube” (still a basic nginx app), but we do two major improvements: (i) Use the limited arkube-sa, and (ii) Run as a non-root user to reduce container breakouts.

#################################################################
# arkube-deployment-secure.yaml
# non-root user, minimal RBAC, safer environment for arkube
#################################################################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: arkube
  namespace: arkube
spec:
  replicas: 1
  selector:
    matchLabels:
      app: arkube
  template:
    metadata:
      labels:
        app: arkube
    spec:
      serviceAccountName: arkube-sa
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
      containers:
      - name: arkube-container
        image: bitnami/nginx:latest
        env:
          - name: NGINX_HTTP_PORT_NUMBER
            value: "8080"
        ports:
        - containerPort: 8080
        securityContext:
          allowPrivilegeEscalation: false

Why Non-Root? By default, Nginx tries port 80, which typically requires root privileges. Using runAsUser: 1000 + port 8080 ensures the container does not run as root. And allowPrivilegeEscalation: false further blocks any attempt to gain super-user powers from inside the container. Your application can function normally (serving on port 8080), but an attacker inside the container has far fewer escalation paths.

NetworkPolicy

If your cluster supports NetworkPolicy (e.g., using Calico, Cilium, etc.), you can limit how pods communicate. For instance, you may only want traffic from an Ingress controller or only allow egress to a DB namespace.

##################################################
# networkPolicy-secure.yaml
# restricts inbound/outbound traffic for arkube pods
##################################################
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-limited-ingress-egress
  namespace: arkube
spec:
  podSelector:
    matchLabels:
      app: arkube
  policyTypes:
  - Ingress
  - Egress
  ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            ingress: "allowed"
  egress:
    - to:
      - namespaceSelector:
          matchLabels:
            db: "allowed"

Even if an attacker gets into your arkube pod, they cannot pivot to other namespaces or services that are not labeled as allowed.

Testing a Malicious Pod Again: malicious-test.yaml

Finally, to confirm security, we replicate a malicious attempt in the arkube namespace, using the arkube-sa. We try to list secrets in the default namespace. It should fail now.

##########################################################
# malicious-test.yaml
# attacker in 'arkube' tries listing secrets in 'default'
# should get "forbidden" if minimal RBAC is working
##########################################################
apiVersion: v1
kind: Pod
metadata:
  name: malicious-test
  namespace: arkube
spec:
  serviceAccountName: arkube-sa
  containers:
  - name: test-container
    image: bitnami/kubectl:latest
    command: ["/bin/sh"]
    args:
      - "-c"
      - |
        echo "Follow me in linkedin..."
        kubectl get secrets -n default
        echo "We expect a Forbidden error here."
        sleep 300

When you run:

• kubectl apply -f malicious-test.yaml

• kubectl logs -n arkube malicious-test

You should see an error like:

Error from server (Forbidden): secrets is forbidden:
User "system:serviceaccount:arkube:arkube-sa" cannot list resource "secrets" 
in API group "" in the namespace "default"

This is exactly what we want—the attacker’s attempt fails because of minimal RBAC.

I feel like some readers are saying now, “No one makes such simple mistakes in the real world.” Here are a few extra (advanced) tips and hints:

• Whenever you finish debugging, remove or rotate any ServiceAccount tokens or credentials you used. This helps prevent “temporary” elevated privileges from lingering in your env.

• In many clusters, the default ServiceAccount can be used inadvertently by pods. Consider setting automountServiceAccountToken: false on pods unless explicitly needed.

• Use tools like Trivy or Gitleaks to detect whether any token, password, or certificate was accidentally committed to source code.

• Tools like OPA Gatekeeper or Kyverno can block pod deployments that request overly broad privileges (e.g., privileged: true or hostPath mounts).

• K8s can produce detailed audit logs about who did what, and when. Forward these logs to a SIEM system or use a tool like Falco in real-time to detect suspicious behaviours (e.g., someone spawning a shell in a container).

• Depending on your k8s version and setup, Pod Security Admission or a PSS alternative can enforce baseline, restricted, or privileged rules for pods. This helps ensure containers don’t run as root unless absolutely necessary, and can limit privilege escalation.

• For NetworkPolicy and more advanced resource governance, consistent labeling of namespaces (e.g., ingress=allowed or db=allowed) simplifies writing and maintaining policies.

Conclusion

In the vulnerable setup, one ServiceAccount with cluster-admin rights gave any pod full control of the cluster. As a result, a malicious pod could easily read secrets such as arkube-db-secret, which is extremely risky in a real environment.

In the secure setup, we created a dedicated arkube namespace, used a minimal Role/RoleBinding to limit access, ran containers as non-root to prevent easy privilege escalations, and (optionally) added a NetworkPolicy to control connections. These steps greatly reduce potential damage if someone breaks in.

To go even further, you can store secrets in an external vault, scan container images for known issues, add checks for weak configurations in your CI/CD pipeline, and use real-time monitoring tools like Falco. By following these guidelines, you significantly lower the chance of a major breach and keep your cluster safer.

Special thanks to ArdanLabs for their amazing educational videos on K8s and Golang, and for their support in this journey. ArdanLabs is a well-known technology company specializing in software development, training, and consulting, particularly in Go (Golang) and Kubernetes. They provide high-quality educational content, workshops, and courses to help developers and enterprises build efficient and scalable apps.