Kubernetes on Zack's Blog

Bedrock-Powered RAG on EKS

Mon, 06 Oct 2025 03:30:40 +0000

The Idea

I’ve previously built Local RAG and ChatBot applications using OpenWebUI, and after exploring AWS Bedrock by creating a knowledge base with an S3 data source, I saw a bigger opportunity. My existing OpenWebUI chatbot was running on AWS Fargate, and I decided it was time to level up the architecture.

The goal was to migrate the solution to an Amazon EKS cluster and build a custom, self-developed RAG pipeline that leverages core AWS services. This would allow me to host two distinct AI applications on a single, scalable platform:

EKS & VPC Lattice Integration for A/B Testing

Tue, 30 Sep 2025 23:56:25 +0000

For those experienced with Kubernetes, managing traffic between microservices often brings tools like Istio to mind. I had done some posts before on Istio for Distributed Tracing and Traffic Routing.

This PoC explores a different approach: using AWS VPC Lattice. While the goal—in this case, A/B testing between different service versions—is the same, the implementation differs. Instead of Istio’s VirtualService and DestinationRule resources, this setup leverages the vendor-neutral Kubernetes Gateway API. This allows for a more standardized way of defining routing within the cluster, while VPC Lattice provides the power to extend this networking seamlessly across different VPCs.

Claude Code with Kubernetes MCP Server

Tue, 23 Sep 2025 01:40:11 +0000

In this post, I’ll demonstrate how to install and use the Kubernetes MCP server GitHub repo with Claude Code , and then show how I migrated the previous AWS serverless OpenWebUI + Bedrock solution to run locally on Minikube. Finally, we’ll explore how to use this Kubernetes MCP server to inspect and troubleshoot the deployment.

Installation & Usage

Before getting started, make sure you have the following prerequisites installed:

kubectl installed and available in your PATH
A valid kubeconfig file with contexts configured
Access to a Kubernetes cluster (e.g., Minikube, Rancher Desktop, GKE)
Helm v3 installed and in your PATH (optional if you don’t plan to use Helm)

By default, the server loads kubeconfig from ~/.kube/config.

Kubernetes 1.33: In-Place Pod Vertical Scaling

Fri, 30 May 2025 02:01:03 +0000

Discover Kubernetes 1.33’s In-Place Vertical Scaling. Learn how to resize pod CPU and memory on the fly without restarts, eliminating downtime and optimizing resource costs.

Previously, adjusting the CPU or memory for Kubernetes pods necessitated a disruptive full restart, causing downtime particularly detrimental for critical and stateful applications.

However, Kubernetes 1.33 introduces “In-Place Pod Vertical Scaling” (K8s.io docs) as a default beta feature, revolutionizing this by allowing on-the-fly CPU and memory adjustments to running pods without any restarts. This game-changing capability eliminates downtime for resource changes, enables better cost optimization by avoiding over-provisioning, and significantly benefits stateful workloads like databases by allowing them to scale without interruption.

EKS - Debug Prometheus Metrics

Sun, 29 Sep 2024 04:32:21 +0000

AWS Managed Prometheus & Grafana is the “plug-and-play” choice for production workloads requiring minimal management, while on the other hand installing helm kube-prometheus-stack offers maximum control but requires more effort to maintain and scale effectively.

Hence for cost control and full customization, I decided to install kube-prometheus-stack on my local lab cluster.

Understand Prometheus Pull-based Monitoring Flow

Expose Metrics: Applications expose metrics in Prometheus format.
Discover Targets:
- Kubernetes-native targets: Discovered via the Kubernetes API.
- Non-cloud-native targets: Defined statically or exposed through exporters.
Scrape Metrics: Prometheus scrapes metrics periodically from /metrics endpoints.
Store Metrics: Metrics are stored in Prometheus’s time-series database.
Visualize Metrics: Grafana (in kube-prometheus-stack) is often used to query and visualize metrics.

Prometheus metrics issue

EKS - Cluster Upgrade

Fri, 27 Sep 2024 04:32:04 +0000

‘Kubernetes Release vs EKS EOL’

A Kubernetes version encompasses both the control plane and the data plane. While AWS manages and upgrades the control plane, we (cluster owner/customer) hold the responsibility for initiating upgrades for both cluster control plane as well as the data plane. When we initiate a cluster upgrade, AWS manages upgrading the control plane, and we are still responsible for initiating the upgrades of the data plane, which includes worker nodes provisioned via Self Managed node groups, Managed Node Groups, Fargate & other add-ons. If worker nodes are provisioned via Karpenter Controller, we can take advantage of Drift or Disruption Controller features (spec.expireAfter) for automatic node recycling and upgrade.

EKS - Get Started with Karpenter

Wed, 25 Sep 2024 04:31:49 +0000

‘Karpenter vs Cluster Autoscaler’

Karpenter is more modern, flexible, and cost-efficient, making it a better choice for dynamic, complex, or large-scale workloads on EKS.

Cluster Autoscaler is simpler and integrates seamlessly with AWS Managed Node Groups, making it suitable for basic scaling needs.

Transitioning to Karpenter from Cluster Autoscaler is a logical step when EKS cluster demands evolve toward more complex scaling with diverse workloads, cost optimization, fine-grained control over node provisioning.

EKS - Enable IAM role for Service Accounts (IRSA)

Thu, 19 Sep 2024 04:31:26 +0000

‘when EKS ConfigMap meet AWS Secret manager

Here I will demo a mysql as database and a wordpress deployment as backend in EKS to reference ConfigMap and Secret via environment variables during run time by reading the environment-specific configuration (like environment names or feature flags) from a ConfigMap and sensitive information (like a database password) from a Secret.

Create demo resources

# vim cf-demo.yaml to create demo namespace, configmap, secret and deployment

root@asb:~/cf# cat secret.yaml
apiVersion: v1
kind: Secret
metadata:
 name: mysql-root-secret
type: Opaque
data:
 MYSQL_ROOT_PASSWORD: c2VjdXJlcGFzc3dvcmQ=  # Base64 for "securepassword"

root@asb:~/cf# cat mysql.yaml
apiVersion: v1
kind: Service
metadata:
 name: mysql-service
spec:
 ports:
 - port: 3306
 selector:
 app: mysql
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mysql
spec:
 selector:
 matchLabels:
 app: mysql
 template:
 metadata:
 labels:
 app: mysql
 spec:
 containers:
 - name: mysql
 image: mysql:5.7
 env:
 - name: MYSQL_ROOT_PASSWORD
 valueFrom:
 secretKeyRef:
 name: mysql-root-secret
 key: MYSQL_ROOT_PASSWORD

root@asb:~/cf# cat wordpress.yaml

apiVersion: v1
kind: ConfigMap
metadata:
 name: wordpress-config
data:
 WORDPRESS_DB_HOST: "mysql-service:3306"
 WORDPRESS_DB_NAME: "wordpress"
---
apiVersion: v1
kind: Secret
metadata:
 name: wordpress-secret
type: Opaque
data:
 WORDPRESS_DB_USER: d29yZHByZXNz  # Base64 for "wordpress"
 WORDPRESS_DB_PASSWORD: c2VjdXJlcGFzc3dvcmQ=  # Base64 for "securepassword"
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: wordpress
spec:
 replicas: 1
 selector:
 matchLabels:
 app: wordpress
 template:
 metadata:
 labels:
 app: wordpress
 spec:
 containers:
 - name: wordpress
 image: wordpress:latest
 ports:
 - containerPort: 80
 env:
 - name: WORDPRESS_DB_HOST
 valueFrom:
 configMapKeyRef:
 name: wordpress-config
 key: WORDPRESS_DB_HOST
 - name: WORDPRESS_DB_NAME
 valueFrom:
 configMapKeyRef:
 name: wordpress-config
 key: WORDPRESS_DB_NAME
 - name: WORDPRESS_DB_USER
 valueFrom:
 secretKeyRef:
 name: wordpress-secret
 key: WORDPRESS_DB_USER
 - name: WORDPRESS_DB_PASSWORD
 valueFrom:
 secretKeyRef:
 name: wordpress-secret
 key: WORDPRESS_DB_PASSWORD
---
apiVersion: v1
kind: Service
metadata:
 name: wordpress-service
spec:
 selector:
 app: wordpress
 ports:
 - protocol: TCP
 port: 80
 targetPort: 80
 type: LoadBalancer  # Or NodePort if LoadBalancer is not available

Apply and verify the Configuration and Secrets in the Pod

root@asb:~/cf# kubectl create ns cf-test
root@asb:~/cf# kubectl apply -f . -n cf-test

configmap/wordpress-config created
secret/wordpress-secret created
deployment.apps/wordpress created
service/wordpress-service created
secret/mysql-root-secret created
service/mysql-service created
deployment.apps/mysql created

root@asb:~/cf# kubectl get all -n cf-test
NAME READY STATUS RESTARTS AGE
pod/mysql-fdff667f8-xzblb 1/1 Running 0 6s
pod/wordpress-6dff4575b9-sgn8t 1/1 Running 0 12m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mysql-service ClusterIP 10.106.191.27 <none> 3306/TCP 6s
service/wordpress-service LoadBalancer 10.97.169.213 pending 80:31208/TCP 12m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/mysql 1/1 1 1 6s
deployment.apps/wordpress 1/1 1 1 12m

NAME DESIRED CURRENT READY AGE
replicaset.apps/mysql-fdff667f8 1 1 1 6s
replicaset.apps/wordpress-6dff4575b9 1 1 1 12m

# Kubectl exec into pod to verify the configmap and secret

root@asb:~/cf# kubectl logs wordpress-6dff4575b9-sgn8t -n cf-test
WordPress not found in /var/www/html - copying now...
Complete! WordPress has been successfully copied to /var/www/html
No 'wp-config.php' found in /var/www/html, but 'WORDPRESS_...' variables supplied; copying 'wp-config-docker.php' (WORDPRESS_DB_HOST WORDPRESS_DB_NAME WORDPRESS_DB_PASSWORD WORDPRESS_DB_USER WORDPRESS_SERVICE_PORT WORDPRESS_SERVICE_PORT_80_TCP WORDPRESS_SERVICE_PORT_80_TCP_ADDR WORDPRESS_SERVICE_PORT_80_TCP_PORT WORDPRESS_SERVICE_PORT_80_TCP_PROTO WORDPRESS_SERVICE_SERVICE_HOST WORDPRESS_SERVICE_SERVICE_PORT)
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.48.245. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.48.245. Set the 'ServerName' directive globally to suppress this message
[Wed Nov 06 23:39:29.456836 2024] [mpm_prefork:notice] [pid 1:tid 1] AH00163: Apache/2.4.62 (Debian) PHP/8.2.25 configured -- resuming normal operations
[Wed Nov 06 23:39:29.456923 2024] [core:notice] [pid 1:tid 1] AH00094: Command line: 'apache2 -D FOREGROUND'

root@asb:~/cf# kubectl exec -it wordpress-6dff4575b9-sgn8t -n cf-test -- env | grep WORDPRESS_DB
WORDPRESS_DB_NAME=wordpress
WORDPRESS_DB_USER=wordpress
WORDPRESS_DB_PASSWORD=securepassword
WORDPRESS_DB_HOST=mysql-service:3306

IRSA with AWS Secret Manager to ensure security best practices in EKS

EKS - Cluster Backup with Velero

Sun, 15 Sep 2024 04:31:09 +0000

Valero is an open-source tool for storing, restoring, and migrating Kubernetes cluster resources and persistent volumes. Valero provides a way to hold the entire state of a Kubernetes cluster, all its objects and their consistent numbers, store backup files to Cloud storage like AWS S3, and then restore them to a previous state to ensure K8S data resilience, disaster recovery results and easy transport between clusters.

Velero for EKS

Backup EKS cluster using Velero, can be followed by the below path:

EKS - KubeBench and OPA Gatekeeper

Fri, 13 Sep 2024 04:30:16 +0000

“After scalling, let’s go EKS Security !”

In the last post, I was able to implement EKS cluster autoscaler and Horizontal Pod Autoscaler (HPA), in this post I will continue with EKS security practice with Kube-Bench and OPA Gatekeeper.

kube-bench:

kube-bench is a tool that checks Kubernetes clusters against the CIS (Center for Internet Security) benchmarks, a set of best practices for securing Kubernetes. It is critical to ensure that a cluster complies with these security guidelines, helping identify potential vulnerabilities and misconfigurations. Key features include generating detailed audit reports, performing automated compliance checks, and easily integrating into existing CI/CD pipelines for continuous security assessments.

EKS - Cluster Autoscaler and Horizontal Pod Autoscaler (HPA)

Thu, 12 Sep 2024 04:18:19 +0000

“Explore how pods and EC2 nodes can be scaled in EKS cluster "

Autoscaler and Horizontal Pod Autoscaler (HPA) Practice on EKS

In last post, I was able to deploy EKS cluster via Jenkins and Terraform:

Jenkins - Multi-Destnation Continuse Deployment with Terraform

This post walks through the process of setting up an Autoscaler and Horizontal Pod Autoscaler (HPA) in an Amazon EKS cluster. We will explore how to dynamically scale the number of nodes in EKS cluster and how to autoscale K8S pods based on CPU utilization using HPA.

Helm + Kustomize for ZackBlog

Tue, 12 Mar 2024 03:17:00 +0000

Kustomize is a configuration management tool for Kubernetes that allows users to customize application manifests without modifying the original YAML files, in this post I will explore Kustomize with overlays, bases, and transformers, then create simple kustomization.yaml files for different environments using Zackblog, then to practice using Kustomize’s built-in resources like configMapGenerator and secretGenerator.

Let’s take Zackblog k8s deployment manifest as an example and convert it into a Kustomize setup. We will start by organizing the files and gradually exploring key Kustomize features.

Istio: Distributed Tracing with Jaeger

Sun, 24 Dec 2023 03:08:40 +0000

Last post we were able to deploy Istio and manage traffic for a book review microservice application. This session we will dive deeper into Istio for its add-on Jaeger for Microservice tracing.

Jaeger is an open-source end-to-end distributed tracing tool to monitor and troubleshoot the performance of microservices-based distributed systems by providing insights into the latency and other performance metrics.

Trace

A trace represents the entire journey of a request or transaction as it propagates through various services and components of a distributed system. It captures the path the request takes, including all the microservices it interacts with, from start to finish. A trace is composed of multiple spans.

Istio: Traffic Routing

Fri, 22 Dec 2023 03:07:00 +0000

Here we use helm to install istio (istio-base, istiod, istio gateway), then deploy a sample online book store microservice “bookinfo”. Practise istio tasks include Traffic Management, Observability, Security.

Bookinfo Topology:

Helm install istio (istiod, istio-ingress)

kubectl create namespace istio-system
helm pull istio/base
helm install istio-base . -n istio-system --set defaultRevision=default

helm pull istio/istiod
helm install istiod . -n istio-system

kubectl create namespace istio-ingress
helm pull istio/gateway
helm install istio-ingress . -n istio-ingress

helm ls -n istio-system
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
istio-base istio-system 1 2023-12-17 08:14:06.943276388 +0800 CST deployed base-1.20.1 1.20.1
istiod istio-system 1 2023-12-17 08:15:40.370551503 +0800 CST deployed istiod-1.20.1 1.20.1

helm ls -n istio-ingress
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
istio-ingress istio-ingress 1 2023-12-17 08:25:07.111999373 +0800 CST deployed gateway-1.20.1 1.20.1

Deploy bookinfo microservice and istio ingressgateway and virtualservice

kubectl label namespace istio-system istio-injection=enabled

kubectl apply -f https://github.com/istio/istio/blob/master/samples/bookinfo/platform/kube/bookinfo.yaml -oyaml > bookinfo.yaml
kubectl apply -f bookinfo.yaml

kubectl get po
NAME READY STATUS RESTARTS AGE
details-v1-698d88b-wmfcb 2/2 Running 0 21m
ratings-v1-6484c4d9bb-cb6gx 2/2 Running 0 21m
reviews-v1-5b5d6494f4-jrsvc 2/2 Running 0 21m
reviews-v2-5b667bcbf8-jgfzj 2/2 Running 0 21m
reviews-v3-5b9bd44f4-tmmfz 2/2 Running 0 21m

kubectl apply -f https://github.com/istio/istio/blob/master/samples/bookinfo/networking/bookinfo-gateway.yaml -oyaml > bookinfo-gateway.yaml
kubectl apply -f bookinfo-gateway.yaml

Deploy Kiali, jaeger, grafana, prometheus

wget https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/prometheus.yaml
wget https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/jaeger.yaml
wget https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/grafana.yaml
kubectl create -f prometheus.yaml -f jaeger.yaml -f grafana.yaml

Visit http://book.istio:31000/productpage, with review (v1, v2, v3)

Move To Rancher

Mon, 26 Jun 2023 03:00:12 +0000

My current employer using Rancher, due to the vender prefers this??

So far many different ways I have used to deploy k8s cluster, each with its own pros and cons.

home lab build k8s components (etcd, keepalived, apiserver, scheduler, coreDNS, calico)
home lab k8s cluster with kubeadm
k8s on AWS using kops and eksctl
AWS self-managed k8s cluster directly on ec2 by ansible

Rancher Support matrix

Local docker Installation

Kubespary with Terraform on AWS

Wed, 24 May 2023 02:40:31 +0000

Kubespary is a powerful and highly configurable tool that automates and supports deployment on various cloud providers (AWS, GCE, Azure, etc.) and on-premises infrastructure. In this post I will run its build-in Terraform and ansible code to provision a k8s cluster based on AWS EC2.

Benefit of Kubespary

We can define specific settings in cluster-config.yaml for networking, authentication, and other Kubernetes features, it also supports for multiple networking plugins (Calico, Flannel, Weave Net, etc.), By managing hosts file and rerun the playbook to make add or remove nodes easily, Integrated CI/CD Pipelines and Infrastructure as Code (IaC) with Terraform and Ansible to achieve automation.

Redis cluster with Helm

Fri, 09 Dec 2022 23:23:27 +0000

Today we are playing around with Redis, Here we use helm to install Redis cluster, then validate statefulset storage and cluster availability.

Typical Redis cluster (3 master + 3 slave for slots) Topology:

Helm install bitnami/redis-cluster

helm repo add bitnami https://charts.bitnami.com/bitnami
helm pull bitnami/redis-cluster
kubectl create ns redis
helm install zz-redis bitnami/redis-cluster -n redis

Redis-cluster status

kubectl get po | grep zz-redis
zz-redis-redis-cluster-0 1/1 Running 3 (33m ago) 36m
zz-redis-redis-cluster-1 1/1 Running 1 (32m ago) 36m
zz-redis-redis-cluster-2 1/1 Running 1 (33m ago) 36m
zz-redis-redis-cluster-3 1/1 Running 1 (33m ago) 36m
zz-redis-redis-cluster-4 1/1 Running 1 (33m ago) 36m
zz-redis-redis-cluster-5 1/1 Running 1 (33m ago) 36m

kubectl get svc
zz-redis-redis-cluster ClusterIP 10.96.68.34 <none> 6379/TCP 37m
zz-redis-redis-cluster-headless ClusterIP None <none> 6379/TCP,16379/TCP 37m

kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
redis-data-zz-redis-redis-cluster-0 Bound pvc-81f69c15-2f59-4704-9fec-c3ab217ebca5 1Gi RWO rook-ceph-block 37m
redis-data-zz-redis-redis-cluster-1 Bound pvc-871dbb68-a78b-48a8-8feb-8726eb8a795e 1Gi RWO rook-ceph-block 37m
redis-data-zz-redis-redis-cluster-2 Bound pvc-afd8c82e-c314-426a-a3cd-3c8d10b42bb1 1Gi RWO rook-ceph-block 37m
redis-data-zz-redis-redis-cluster-3 Bound pvc-e8fd5d47-dd60-4358-8cf5-a17d6574bbe2 1Gi RWO rook-ceph-block 37m
redis-data-zz-redis-redis-cluster-4 Bound pvc-04d4f148-b7c1-407e-b9a8-b2fa911405f0 1Gi RWO rook-ceph-block 37m
redis-data-zz-redis-redis-cluster-5 Bound pvc-99e8b2cc-c5d5-4636-a19b-042922eca3cc 1Gi RWO rook-ceph-block 37m

Validate cluster by set key

Apache Kafka cluster with testing topics

Fri, 09 Dec 2022 22:09:32 +0000

Apache Kafka has proven to be an extremely popular event streaming platform, as its scalable distributed architecture, high performance, and use cases, some key terms and concepts as below:

Kafka clusters and Kafka brokers
Kafka clients and servers
Producers, and Consumers, and Consumer groups
Kafka topics & Kafka partitions, offsets
Kafka topic replication, leaders, and followers
ZooKeeper or not

Typical Kafka Topology:

Helm install bitnami/kafka

Here we use helm to install Kafka, then validate statefulset storage and cluster availability by creating a topic, producer, and consumer.

Ingress controller for Subdomain based Routing

Fri, 18 Nov 2022 22:01:55 +0000

In this post I will run the steps required to set up MetalLB for LoadBalancer IP allocation, the NGINX Ingress Controller for routing traffic based on subdomains, and how to configure local DNS to enable subdomain-based routing for Kubernetes services.

Ingress-nginx install

We will use Helm to install our ingress controller ingress-nginx, which will be used to route traffic based on subdomains.

# install ingress controller ingress-nginx
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace

# check for ingressclasses
kubectl get ingressclasses.networking.k8s.io
NAME CONTROLLER PARAMETERS AGE
nginx k8s.io/ingress-nginx <none> 149m

MetalLB for loadbalancing

Rook-Ceph for dynamic Persistent Volume

Wed, 09 Nov 2022 23:01:53 +0000

Today, I will explore how to install and configure Persistent Volume for all the deployment in K8S to consume.

Helm install Rook-Ceph for persistent storage

Configure local VM block storage to add 50Gb sdb to all k8s master and worker nodes
Install rook-ceph cluster

git clone --single-branch --branch master https://github.com/rook/rook.git
cd rook/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f operator.yaml
kubectl create -f cluster.yaml

Ceph toolbox to check cluster status

kubectl create -f toolbox.yaml
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
ceph status
ceph osd status
ceph df
rados df

Ceph Dashboard service for HTTPS login

kubectl create -f dashboard-external-https.yaml
kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

Create storage pool and storage class

kubectl create -f pool.yaml
cd csi/rbd
kubectl create -f storageclass.yaml

Set “rook-ceph-block” as the default storage class

kubectl patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
rook-ceph-block (default) rook-ceph.rbd.csi.ceph.com Delete Immediate true 8d

Check pool and OSDs in Ceph web UI