Posts on Zack's Blog

Can AI Actually Build a Complex Project? A Brutal RAG System Post-Mortem

Sun, 01 Mar 2026 05:27:11 +0000

Last month, I did something that, looking back, was probably the most educational practice of my year. I tried using Claude and GPT to build a complete multi-source RAG system from scratch. This wasn’t a “toy demo” for social media; it was a real internal tool for my lab designed to handle messy document data, query rewriting, hybrid search, and deployment for my team.

I set a strict rule for myself: Let the AI write almost all the code, while I handle the requirements and decision-making.

Improve Billing & Health Analyzer with Bedrock v2

Wed, 21 Jan 2026 04:50:08 +0000

I constantly rely on AWS Billing and Cost Management and Trusted Advisor to monitor costs and security. However, manual billing reviews are time-consuming, and raw Cost Explorer data doesn’t provide the full picture. What if an intelligent program could proactively call the Billing and Trusted Advisor APIs, retrieve the raw data, and pass it to an LLM for in-depth analysis of account health—then deliver actionable insights and AI-powered recommendations? That would save a significant amount of time and effort.

Serverless Billing & Health Analyzer with Bedrock

Sat, 18 Oct 2025 11:42:05 +0000

Boost EKS RAG with LangChain

Sat, 11 Oct 2025 05:21:03 +0000

After building a custom RAG system on Amazon EKS, my son asked why the application could only handle one question at a time, while ChatGPT allows users to continue conversations through follow-up questions with contextual memory. That made me realize there was an opportunity to leverage the power of the LangChain framework for better abstractions, conversational memory, and access to the broader LangChain ecosystem.

This post documents the journey of migrating to a LangChain-powered solution — the challenges faced, and the new capabilities and benefits gained.

Bedrock-Powered RAG on EKS

Mon, 06 Oct 2025 03:30:40 +0000

The Idea

I’ve previously built Local RAG and ChatBot applications using OpenWebUI, and after exploring AWS Bedrock by creating a knowledge base with an S3 data source, I saw a bigger opportunity. My existing OpenWebUI chatbot was running on AWS Fargate, and I decided it was time to level up the architecture.

The goal was to migrate the solution to an Amazon EKS cluster and build a custom, self-developed RAG pipeline that leverages core AWS services. This would allow me to host two distinct AI applications on a single, scalable platform:

EKS & VPC Lattice Integration for A/B Testing

Tue, 30 Sep 2025 23:56:25 +0000

For those experienced with Kubernetes, managing traffic between microservices often brings tools like Istio to mind. I had done some posts before on Istio for Distributed Tracing and Traffic Routing.

This PoC explores a different approach: using AWS VPC Lattice. While the goal—in this case, A/B testing between different service versions—is the same, the implementation differs. Instead of Istio’s VirtualService and DestinationRule resources, this setup leverages the vendor-neutral Kubernetes Gateway API. This allows for a more standardized way of defining routing within the cluster, while VPC Lattice provides the power to extend this networking seamlessly across different VPCs.

Claude Code with Kubernetes MCP Server

Tue, 23 Sep 2025 01:40:11 +0000

In this post, I’ll demonstrate how to install and use the Kubernetes MCP server GitHub repo with Claude Code , and then show how I migrated the previous AWS serverless OpenWebUI + Bedrock solution to run locally on Minikube. Finally, we’ll explore how to use this Kubernetes MCP server to inspect and troubleshoot the deployment.

Installation & Usage

Before getting started, make sure you have the following prerequisites installed:

kubectl installed and available in your PATH
A valid kubeconfig file with contexts configured
Access to a Kubernetes cluster (e.g., Minikube, Rancher Desktop, GKE)
Helm v3 installed and in your PATH (optional if you don’t plan to use Helm)

By default, the server loads kubeconfig from ~/.kube/config.

AWS Bedrock with Open WebUI

Tue, 16 Sep 2025 14:56:30 +0000

Recently Data team reached out, trying to build an EC2 running Open WebUI, connecting to AWS Bedrock, to offer team member AI application. This guide provides step-by-step instructions for deploying and integrating with AWS Bedrock models. .

The architecture consists of two Docker containers operating on a single EC2 instance: Open WebUI serves as the user-facing chat interface, while a Bedrock Access Gateway acts as middleware. This gateway securely forwards requests from Open WebUI to the AWS Bedrock API using the EC2 instance’s attached IAM role. The containers communicate over a private Docker network, isolating traffic between them.

Automate PR Review using Gemini CLI within GitHub Action

Wed, 03 Sep 2025 12:29:23 +0000

I recently found an interesting tool run-gemini-cli, a GitHub Action that integrates Gemini directly into your repositories. I was intrigued by its promise to perform pull request reviews, triage issues, and even modify code using conversational commands right inside GitHub action workflow. Let’s try it out.

My goal was to see if I could use this action to get instant, automated feedback on pull requests. Instead of waiting for a human to spot simple issues, I wanted an AI to do the first pass, check for common problems, and even explain code changes on demand.

AI Coding - Build an Electricity and Gas Analytic Platform

Thu, 14 Aug 2025 07:45:20 +0000

In the past few days, I got an idea from my recent electricity and gas bills—thinking about how to run data analysis to examine usage patterns and trends using the Python approach: employing Pandas for reading data, using Numpy for normalisation, and Matplotlib to create a few visualisations.

However, this time I wanted to try talking to Gemini CLI via the terminal to see if it could generate a web-based analytical platform for me, as someone with no developer background.

Exploratory Data Analysis (EDA) with Peppa Pig

Sun, 27 Jul 2025 05:56:03 +0000

This was a linguistic analysis project where the primary goal was not just to count words, but to evaluate the language, themes, and emotional tone of the children’s show “Peppa Pig” (specifically, the first four seasons) to determine its suitability for a pre-kindergarten audience. In this study, I will try to answer a broader question:

“Beyond a simple word list, what can a multi-faceted data analysis tell us about the show’s true educational and emotional value?”

AI-assisted Coding with Google Gemini Cli

Tue, 22 Jul 2025 14:08:01 +0000

In the last post, I used Claude Code with Kimi K2 to complete the Django web app layout optimization. One thing I wasn’t satisfied with was the pagination system on this Django blog. It used a numbered pagination interface at the bottom of the home page—functional, but a bit dated. I wanted to modernize it with automatic loading of more posts via infinite scroll.

However, I couldn’t get this done using Kimi K2. Despite spending a significant number of tokens (and money), the model couldn’t provide a working solution.

Migrating Blog to Django Web App

Mon, 21 Jul 2025 04:28:57 +0000

I originally used Jekyll, a straightforward static site generator, to build my blog. It was easy to set up and ideal for managing content in Markdown. However, it had its limitations: serving only static content meant a lack of dynamic features and essential tools like a built-in database and user management. That’s when I decided to migrate to Django. With its ability to handle dynamic content and a fully integrated database, it was the perfect solution and the clear choice for the future of my blog.

Claude Code + Kimi K2

Sat, 19 Jul 2025 00:40:01 +0000

Effortless Coding with Claude Code and Kimi K2: Features, Pricing, and Setup Guide

Recently Kimi K2 became one of the most popular model in HuggingFace, claimed its compatible with Claude Code API. With my new sign up and got a $15 free credit, Let’s see how it can intergrate with Claud Code and validate its coding performance!

🧠 Claude Code + Kimi K2?

Claude Code is command-line tool from Anthropic Claude, powered by its latest models (Opus, Sonnet, Haiku). It is highly effective but can be expensive.

Kubernetes 1.33: In-Place Pod Vertical Scaling

Fri, 30 May 2025 02:01:03 +0000

Discover Kubernetes 1.33’s In-Place Vertical Scaling. Learn how to resize pod CPU and memory on the fly without restarts, eliminating downtime and optimizing resource costs.

Previously, adjusting the CPU or memory for Kubernetes pods necessitated a disruptive full restart, causing downtime particularly detrimental for critical and stateful applications.

However, Kubernetes 1.33 introduces “In-Place Pod Vertical Scaling” (K8s.io docs) as a default beta feature, revolutionizing this by allowing on-the-fly CPU and memory adjustments to running pods without any restarts. This game-changing capability eliminates downtime for resource changes, enables better cost optimization by avoiding over-provisioning, and significantly benefits stateful workloads like databases by allowing them to scale without interruption.

Cost Optimization with Amazon Q & Cost Explorer MCP Server

Sun, 20 Apr 2025 08:36:56 +0000

Managing cloud costs effectively, especially on AWS, is crucial. Wasted resources can easily inflate bills. This post introduces the AWS Cost Explorer MCP Server, a tool designed to simplify analyzing your AWS spending using the Model Context Protocol (MCP).

GitHub Repository: https://github.com/awslabs/mcp/tree/main/src/cost-explorer-mcp-server

What this AWS Cost Explorer MCP Server Does

This specific MCP Server, provided by AWS Labs, acts as a specialized tool that connects an AI assistant like Amazon Q directly to the detailed AWS cost and usage data. Think of it as giving us AI assistant the specific knowledge and tools needed to understand and analyze our cloud spending.

MLOPS - Enhancing Oscar Model with LightGBM

Fri, 21 Mar 2025 13:36:27 +0000

In a previous post - MLOps - Build a Oscar Best Picture Winner Model, I was able to establish a baseline model using a RandomForestClassifier to predict the Oscar for Best Picture. This classic workflow involved data cleaning, training, and prediction, providing a solid starting point.

However, a deeper look at the results revealed critical weaknesses that an experienced machine learning engineer would immediately flag:

Inadequate Model Choice: The initial model wasn’t powerful enough for the task. The classification report showed a recall of 0.00 for the “winner” class. This is a major red flag, indicating the model completely failed to identify any actual winners, likely due to the severe class imbalance.
Misleading Evaluation Metrics: I think I relied too heavily on accuracy. On an imbalanced dataset, a model can achieve high accuracy simply by always predicting the majority class. Better to shift our focus to more robust metrics like the F1-score, ROC AUC, and Precision-Recall AUC.

This analysis led to idea to enhance this Oscar prediction with a more sophisticated LightGBM (LGBM) classifier model, known for its high performance, speed, and efficiency on tabular data.

MLOps - Build a Oscar Best Picture Winner Model

Sun, 23 Feb 2025 09:37:43 +0000

In this post, we will continue to build a basic machine learning model to predict the Best Picture winner at the Academy Awards (Oscar).

We will use our previous processed dataset that includes information about the nominees and winners from the 72nd to the 96th Oscar ceremonies. The goal is to predict the winner based on various features like IMDb ratings, Metascore, Tomatometer percentage, Golden Globe and BAFTA wins/nominations.

Step 1: Understand the features and model

MLOps - Data Processing for Oscar Winner Model

Sun, 23 Feb 2025 05:35:54 +0000

After downloading a dataset from Kaggle about historical Oscar nominations and winners, it is time to move on to data cleansing and enrichment.

The original data contained seven headers, but only a few were useful: the film name, ceremony year, and winner status. Additionally, the dataset spanned from 1927 to 2024 and included all Oscar categories, making it quite complex—perhaps too overwhelming for me as a beginner. So, I decided to focus on Best Picture as it is always the most important award among all the other categories.

MLOps - How About Predict Oscar Winner

Fri, 21 Feb 2025 11:50:12 +0000

The Idea

The 97th Academy Awards ceremony, presented by the Academy of Motion Picture Arts and Sciences (AMPAS), will take place on March 2, 2025, at the Dolby Theatre in Hollywood, Los Angeles. Last year 2024 I had some greate experience with some great movies like Dune Part2 and Wicked, not sure if some of my faviourate actors can grab a Oscar.

so Why not go and use historical Oscar nominations dataset for the past 20 years, to create and train a model by feeding categories and results, to predict the winners each year, then input this years’ nominations to get a prediction ??

Destruct AWS account using AWS-Nuke

Tue, 18 Feb 2025 08:16:01 +0000

Here’s a summary of the actions I took to delete an AWS account using an open source tool aws-nuke as it is at the end of free tier I donot need it anymore to aviod cost:

1. Install aws-nuke:

apt install aws-nuke

This command installs aws-nuke, a tool used to delete AWS resources in an account. It requires AWS credentials and the correct configuration file to specify what resources to remove.

MLOps - Get Started with KubeRay

Tue, 11 Feb 2025 04:35:22 +0000

Ray is an open-source framework designed for scalable and distributed ML workloads, including training, tuning, and inference. It provides a simple API for scaling Python applications across multiple nodes.

Distributed Training: Easily scale PyTorch, TensorFlow, and other ML jobs across multiple GPUs/instances.
Hyperparameter Tuning: Integrates with Optuna and Ray Tune for distributed hyperparameter optimization.
Parallel Inference: Supports inference pipelines that scale out dynamically based on demand.
Fault Tolerance: If a node fails, Ray can reschedule tasks on other available nodes.

Together with EKS, Karpenter, and Ray, a modern ML team can achieve dynamic Auto-Scaling and GPU resource allocation from local deployment to Running Distributed ML Jobs in Cloud Production environment:

MLOps - Deploy ML workload into EKS with Karpenter

Sat, 01 Feb 2025 04:35:11 +0000

Finally, machine learning workload into AWS EKS with Karpenter

In the previous post, I was able to complete both local Pytorch ML and AWS SageMaker practice, and containerize and deploy ML docker image locally.

In this post, I will explore the model deployment to EKS cluster with Karpenter to simulate a more scalable and real-life production-ready environment.

Challenges when moving to Cloud deployment

Here are key differences between local PyTorch model vs AWS SageMaker artifacts, and how they impact the Docker image size and Performance considerations for LLM images in EKS deployment.

MLOps - Deploy Classifier with AWS Serverless

Sat, 25 Jan 2025 04:34:58 +0000

Continue Pneumonia Classifier by moving to AWS Serverless

Go AWS Serverless Deployment

Deploying machine learning models in production requires additional considerations to address latency, scalability, cost-efficiency, and monitoring.

A modern approach to hosting an ML application in AWS can be considered as a serverless architecture.

This allows users to upload images via a S3 static web page and send them to API Gateway. The API Gateway receives the HTTP POST request and forwards it to the Lambda function, which handles the image preprocessing, inference, and postprocessing logic. The Lambda function sends the image payload to the SageMaker endpoint, by calling the SageMaker endpoint using the SageMaker Runtime SDK (invoke_endpoint) which hosts my trained model, and then retrieves the prediction. The prediction result is sent back to the frontend for display.

MLOps - Move Image Classifier to AWS SageMaker

Sat, 25 Jan 2025 04:34:41 +0000

Continue Pneumonia Classifier by moving to AWS SageMaker

In this post, we will continue Pneumonia Classifier practice by leveraging AWS managed AI service SageMaker. We will:

Provision AWS Sagemaker AI domain and workspace.
Run a JupiterLab to download, process, and upload chest X-ray dataset from Kaggle to S3 for SageMaker access.
Use SageMaker Pre-built image classification algorithm, to define SageMaker Estimator to configure the SageMaker training job, including compute resources, training duration, and data input method.
Count the number of training samples and set the hyperparameters, perform hyperparameter tuning to find the best configuration for the model.
Launching the Hyperparameter Tuning Job.
Using CloudWatch and SageMaker Train Job to monitor and troubleshoot.

In the AWS SageMaker AI JupiterLab

MLOps - Containerize Classifier Application

Sat, 25 Jan 2025 04:34:14 +0000

Last post we were able to build and train the pneumonia classifier model.
In this post we are moving a step forward to: Containerizing the model and creating a frontend-backend application to allow users to upload images and get predictions.

Overview of the Application Architecture

A high-level diagram or description of the system architecture:

Backend: A Dockerized Flask/FastAPI application serving the trained model.
Frontend: A Dockerized React/Streamlit app for image upload and displaying predictions.
Interaction: Frontend sends images to the backend, which processes them and returns predictions.

Step 1: Containerizing the Backend

MLOps - Build a Image Classifier Model

Sat, 25 Jan 2025 04:34:02 +0000

Pneumonia causes over 2.5 million deaths annually worldwide. Traditional diagnosis through chest X-ray analysis is time-consuming and requires expert radiologists. This project aims to develop an automated deep learning system to classify pneumonia from chest X-rays with high accuracy, potentially assisting healthcare professionals in faster diagnosis.

Purpose

Here I will use a local environment with Pytorch and Jupyter Notebook to:

Demonstrate end-to-end development of a medical image classifier
Using Kaggle dataset for NIH Chest X-Ray image processing
Create a baseline model for pneumonia detection (Normal vs Pneumonia)
Local GPU acceleration for model training and prediction

Typical ML steps to achieve image classifier:

MLOps - Deploy ML workload to K8S

Mon, 20 Jan 2025 04:33:48 +0000

Machine Learning workload deployed in K8S and minikube!!

So far the local ML practice is just the beginning, in a real world production environment, ML projects typically follow a structured lifecycle and often deployed in scalable, cloud-based environments. Cloud Providers like AWS with Managed Kubernetes Services (EKS) provide orchestration, scaling, and fault tolerance to handle ML workloads.

Path for deploying MLOps workload in K8S

Transition from local ML practice to a K8S-based deployment (This post)
Start with MLOps tools like Kubeflow.
Shift from local to cloud platforms (AWS) to deploy ML workload on EKS.
Practice deploying models using REST APIs (local) and APT Gateway (AWS).
Try local data engineering (ETL pipelines, data lakes, etc.), then move to AWS data services and solutions for ML workload.

Setting up Minikube with GPU on WSL Ubuntu

MLOps - Build a Knowledge Base with DeepSeek R1

Sun, 19 Jan 2025 04:33:34 +0000

‘Deep Seek is really hot at the moment!

In this post, I want to build a local Knowledge Base using WSL, Docker, Ollama, Open WebUI and DeepSeek R1 7b.

Ollama is a desktop application designed to run and interact with large language models (LLMs) locally on a machine. It provides an easy interface for downloading, managing, and using various LLMs, ensuring privacy and local execution.

Open WebUI is a web-based user interface for interacting with AI models. It is often used in conjunction with locally hosted or remote LLMs, providing a customizable and user-friendly platform to input queries and manage model interactions.

AWS Private Link and Cross-account Package deployment

Fri, 15 Nov 2024 04:33:19 +0000

Recently I got a task from the Company’s app team, to install an agent on EC2 instances across several AWS accounts and enable connection via AWS private link to a target AWS account where the management server is hosted. To achieve this task, here I will see how to use AWS Systems Manager for software distribution and installation for multiple AWS accounts and install and configure AWS private link using Terraform.

MLOps - Explore ML tools

Wed, 02 Oct 2024 04:32:58 +0000

In the last post MLOPS - Lab Setup, I was able to set the local ML lab environment, and run validation in Jupyter Notebook to test the CODA device and performance on my local PC.

Although Jupyter Notebooks can be user-friendly tools for ML practice, offering easy interaction and immediate feedback, which simplifies testing and debugging, it has limitations such as reproducibility issues, challenges in collaboration and version control, scalability concerns for larger projects, and a lack of automation for tasks like retraining.

MLOps - Setup a Home Machine Learning Lab

Tue, 01 Oct 2024 04:32:35 +0000

Transitioning from DevOps to MLOps can be achieved by leveraging existing DevOps expertise by adding new layers specific to machine learning.

Key Differences:
Model Lifecycle Management: MLOps handles model training, deployment, and retraining.
Data Versioning: Tools like DVC ensure dataset version control.
Experiment Tracking: MLflow and Weights & Biases track model training parameters and results.
Model Serving: Deploy models with TensorFlow Serving or TorchServe.
Model Drift: Monitor data changes over time to trigger retraining.

EKS - Debug Prometheus Metrics

Sun, 29 Sep 2024 04:32:21 +0000

AWS Managed Prometheus & Grafana is the “plug-and-play” choice for production workloads requiring minimal management, while on the other hand installing helm kube-prometheus-stack offers maximum control but requires more effort to maintain and scale effectively.

Hence for cost control and full customization, I decided to install kube-prometheus-stack on my local lab cluster.

Understand Prometheus Pull-based Monitoring Flow

Expose Metrics: Applications expose metrics in Prometheus format.
Discover Targets:
- Kubernetes-native targets: Discovered via the Kubernetes API.
- Non-cloud-native targets: Defined statically or exposed through exporters.
Scrape Metrics: Prometheus scrapes metrics periodically from /metrics endpoints.
Store Metrics: Metrics are stored in Prometheus’s time-series database.
Visualize Metrics: Grafana (in kube-prometheus-stack) is often used to query and visualize metrics.

Prometheus metrics issue

EKS - Cluster Upgrade

Fri, 27 Sep 2024 04:32:04 +0000

‘Kubernetes Release vs EKS EOL’

A Kubernetes version encompasses both the control plane and the data plane. While AWS manages and upgrades the control plane, we (cluster owner/customer) hold the responsibility for initiating upgrades for both cluster control plane as well as the data plane. When we initiate a cluster upgrade, AWS manages upgrading the control plane, and we are still responsible for initiating the upgrades of the data plane, which includes worker nodes provisioned via Self Managed node groups, Managed Node Groups, Fargate & other add-ons. If worker nodes are provisioned via Karpenter Controller, we can take advantage of Drift or Disruption Controller features (spec.expireAfter) for automatic node recycling and upgrade.

EKS - Get Started with Karpenter

Wed, 25 Sep 2024 04:31:49 +0000

‘Karpenter vs Cluster Autoscaler’

Karpenter is more modern, flexible, and cost-efficient, making it a better choice for dynamic, complex, or large-scale workloads on EKS.

Cluster Autoscaler is simpler and integrates seamlessly with AWS Managed Node Groups, making it suitable for basic scaling needs.

Transitioning to Karpenter from Cluster Autoscaler is a logical step when EKS cluster demands evolve toward more complex scaling with diverse workloads, cost optimization, fine-grained control over node provisioning.

EKS - Enable IAM role for Service Accounts (IRSA)

Thu, 19 Sep 2024 04:31:26 +0000

‘when EKS ConfigMap meet AWS Secret manager

Here I will demo a mysql as database and a wordpress deployment as backend in EKS to reference ConfigMap and Secret via environment variables during run time by reading the environment-specific configuration (like environment names or feature flags) from a ConfigMap and sensitive information (like a database password) from a Secret.

Create demo resources

# vim cf-demo.yaml to create demo namespace, configmap, secret and deployment

root@asb:~/cf# cat secret.yaml
apiVersion: v1
kind: Secret
metadata:
 name: mysql-root-secret
type: Opaque
data:
 MYSQL_ROOT_PASSWORD: c2VjdXJlcGFzc3dvcmQ=  # Base64 for "securepassword"

root@asb:~/cf# cat mysql.yaml
apiVersion: v1
kind: Service
metadata:
 name: mysql-service
spec:
 ports:
 - port: 3306
 selector:
 app: mysql
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: mysql
spec:
 selector:
 matchLabels:
 app: mysql
 template:
 metadata:
 labels:
 app: mysql
 spec:
 containers:
 - name: mysql
 image: mysql:5.7
 env:
 - name: MYSQL_ROOT_PASSWORD
 valueFrom:
 secretKeyRef:
 name: mysql-root-secret
 key: MYSQL_ROOT_PASSWORD

root@asb:~/cf# cat wordpress.yaml

apiVersion: v1
kind: ConfigMap
metadata:
 name: wordpress-config
data:
 WORDPRESS_DB_HOST: "mysql-service:3306"
 WORDPRESS_DB_NAME: "wordpress"
---
apiVersion: v1
kind: Secret
metadata:
 name: wordpress-secret
type: Opaque
data:
 WORDPRESS_DB_USER: d29yZHByZXNz  # Base64 for "wordpress"
 WORDPRESS_DB_PASSWORD: c2VjdXJlcGFzc3dvcmQ=  # Base64 for "securepassword"
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: wordpress
spec:
 replicas: 1
 selector:
 matchLabels:
 app: wordpress
 template:
 metadata:
 labels:
 app: wordpress
 spec:
 containers:
 - name: wordpress
 image: wordpress:latest
 ports:
 - containerPort: 80
 env:
 - name: WORDPRESS_DB_HOST
 valueFrom:
 configMapKeyRef:
 name: wordpress-config
 key: WORDPRESS_DB_HOST
 - name: WORDPRESS_DB_NAME
 valueFrom:
 configMapKeyRef:
 name: wordpress-config
 key: WORDPRESS_DB_NAME
 - name: WORDPRESS_DB_USER
 valueFrom:
 secretKeyRef:
 name: wordpress-secret
 key: WORDPRESS_DB_USER
 - name: WORDPRESS_DB_PASSWORD
 valueFrom:
 secretKeyRef:
 name: wordpress-secret
 key: WORDPRESS_DB_PASSWORD
---
apiVersion: v1
kind: Service
metadata:
 name: wordpress-service
spec:
 selector:
 app: wordpress
 ports:
 - protocol: TCP
 port: 80
 targetPort: 80
 type: LoadBalancer  # Or NodePort if LoadBalancer is not available

Apply and verify the Configuration and Secrets in the Pod

root@asb:~/cf# kubectl create ns cf-test
root@asb:~/cf# kubectl apply -f . -n cf-test

configmap/wordpress-config created
secret/wordpress-secret created
deployment.apps/wordpress created
service/wordpress-service created
secret/mysql-root-secret created
service/mysql-service created
deployment.apps/mysql created

root@asb:~/cf# kubectl get all -n cf-test
NAME READY STATUS RESTARTS AGE
pod/mysql-fdff667f8-xzblb 1/1 Running 0 6s
pod/wordpress-6dff4575b9-sgn8t 1/1 Running 0 12m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mysql-service ClusterIP 10.106.191.27 <none> 3306/TCP 6s
service/wordpress-service LoadBalancer 10.97.169.213 pending 80:31208/TCP 12m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/mysql 1/1 1 1 6s
deployment.apps/wordpress 1/1 1 1 12m

NAME DESIRED CURRENT READY AGE
replicaset.apps/mysql-fdff667f8 1 1 1 6s
replicaset.apps/wordpress-6dff4575b9 1 1 1 12m

# Kubectl exec into pod to verify the configmap and secret

root@asb:~/cf# kubectl logs wordpress-6dff4575b9-sgn8t -n cf-test
WordPress not found in /var/www/html - copying now...
Complete! WordPress has been successfully copied to /var/www/html
No 'wp-config.php' found in /var/www/html, but 'WORDPRESS_...' variables supplied; copying 'wp-config-docker.php' (WORDPRESS_DB_HOST WORDPRESS_DB_NAME WORDPRESS_DB_PASSWORD WORDPRESS_DB_USER WORDPRESS_SERVICE_PORT WORDPRESS_SERVICE_PORT_80_TCP WORDPRESS_SERVICE_PORT_80_TCP_ADDR WORDPRESS_SERVICE_PORT_80_TCP_PORT WORDPRESS_SERVICE_PORT_80_TCP_PROTO WORDPRESS_SERVICE_SERVICE_HOST WORDPRESS_SERVICE_SERVICE_PORT)
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.48.245. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.48.245. Set the 'ServerName' directive globally to suppress this message
[Wed Nov 06 23:39:29.456836 2024] [mpm_prefork:notice] [pid 1:tid 1] AH00163: Apache/2.4.62 (Debian) PHP/8.2.25 configured -- resuming normal operations
[Wed Nov 06 23:39:29.456923 2024] [core:notice] [pid 1:tid 1] AH00094: Command line: 'apache2 -D FOREGROUND'

root@asb:~/cf# kubectl exec -it wordpress-6dff4575b9-sgn8t -n cf-test -- env | grep WORDPRESS_DB
WORDPRESS_DB_NAME=wordpress
WORDPRESS_DB_USER=wordpress
WORDPRESS_DB_PASSWORD=securepassword
WORDPRESS_DB_HOST=mysql-service:3306

IRSA with AWS Secret Manager to ensure security best practices in EKS

EKS - Cluster Backup with Velero

Sun, 15 Sep 2024 04:31:09 +0000

Valero is an open-source tool for storing, restoring, and migrating Kubernetes cluster resources and persistent volumes. Valero provides a way to hold the entire state of a Kubernetes cluster, all its objects and their consistent numbers, store backup files to Cloud storage like AWS S3, and then restore them to a previous state to ensure K8S data resilience, disaster recovery results and easy transport between clusters.

Velero for EKS

Backup EKS cluster using Velero, can be followed by the below path:

EKS - KubeBench and OPA Gatekeeper

Fri, 13 Sep 2024 04:30:16 +0000

“After scalling, let’s go EKS Security !”

In the last post, I was able to implement EKS cluster autoscaler and Horizontal Pod Autoscaler (HPA), in this post I will continue with EKS security practice with Kube-Bench and OPA Gatekeeper.

kube-bench:

kube-bench is a tool that checks Kubernetes clusters against the CIS (Center for Internet Security) benchmarks, a set of best practices for securing Kubernetes. It is critical to ensure that a cluster complies with these security guidelines, helping identify potential vulnerabilities and misconfigurations. Key features include generating detailed audit reports, performing automated compliance checks, and easily integrating into existing CI/CD pipelines for continuous security assessments.

EKS - Cluster Autoscaler and Horizontal Pod Autoscaler (HPA)

Thu, 12 Sep 2024 04:18:19 +0000

“Explore how pods and EC2 nodes can be scaled in EKS cluster "

Autoscaler and Horizontal Pod Autoscaler (HPA) Practice on EKS

In last post, I was able to deploy EKS cluster via Jenkins and Terraform:

Jenkins - Multi-Destnation Continuse Deployment with Terraform

This post walks through the process of setting up an Autoscaler and Horizontal Pod Autoscaler (HPA) in an Amazon EKS cluster. We will explore how to dynamically scale the number of nodes in EKS cluster and how to autoscale K8S pods based on CPU utilization using HPA.

Jenkins - Multi-Destination CD pipeline with Terraform

Thu, 15 Aug 2024 04:17:55 +0000

“Design a Jenkins pipeline can handle multiple destination: EC2, EKS across different environments”

Jenkins CD Pipeline Design

In last post, I was able to create a Jenkins Universal CI Pipeline to create blog docker image and push to DockerHub:

Jenkins - Universal CI Pipeline with Ansible & Terraform

Now it is time to design the continuous deployment pipeline with Ansible and Terraform for infrastructure provision and application configuration and deployment.

Jenkins - Universal CI Pipeline with Ansible & Terraform

Fri, 02 Aug 2024 04:17:40 +0000

“Such a long time I nearly forgot how to use Jenkins..”

Jenkins recap

It has been some time since I adapted CI/CD pipelines from Jenkins to AWS CodePipeline and GitHub Actions workflows. Now, it’s time to recap and improve some of my previous Jenkins practices.

Universal Jenkins Docker image design

This time, instead of installing Jenkins on a server, I prefer to containerize a universal Jenkins Docker image with the necessary packages installed, so it provides consistency and reproducibility, portability, and easy updates and rollbacks.

RedHat Identity Management (IdM) with AD Integration

Sat, 13 Jul 2024 06:17:15 +0000

When a company faces challenge to manage its Linux environments across local and public cloud, RedHat Identity management can be the solution to achieve:

With Local AD and Azure AD (AAD) Integration
With AWS SSO Integration as external identity provider
LDAP, Kerberos and NTP
A web-based management front-end running on Apache

A Typical AD User Authentication Flow End-to-End:

User Creation and Management:

Azure AD / Local AD: Users are created in the Azure Active Directory or local Active Directory.
Synchronization to RedHat IdM: The users are synchronized from AD to RedHat IdM using the two-way trust established between AD and IdM.

Accessing EC2 Instances via SSH:

Serverless with AWS Fargate

Wed, 10 Jul 2024 04:11:38 +0000

‘if people tend to move to serverless, how ‘infrastructure engineer’ will end up’ Why go serverless

Some of the company’s applications recently moved from Rancher to Fargate, which is understandable as the cloud resource and traffic will be very intensive only during a certain period (HSC exam), hence AWS serverless with Fargate can be a better option for such business mode so the rest of the year without the exam we can save costs significantly.

Handling a RDS MySQL cluster CPU 100%

Mon, 17 Jun 2024 04:10:51 +0000

Today I got a performance issue from our analytic team, saying they experienced a Production MySQL cluster running on RDS very slow since yesterday morning.

I started to look into below areas for investigation:

AWS CloudWatch Metrics for RDS

AWS CloudWatch provides a wide range of metrics that can help diagnose resource usage for databases. So I started with:

CloudWatch - Metrics - All metrics - Add query - RDS - Top 10 RDS instances by highest CPU utilization

Python Microservice: K8S deployment

Wed, 22 May 2024 04:10:28 +0000

Now It is time to change from docker-compose to deploy into Kubernetes.

As this is not new to me to deploy microservice into K8S, also I already have a running Kubernetes cluster in hand, so here I will just create docker images for the 3 services: API gateway, user and order, then push them into the docker hub repository, then create Kubernetes manifest for deployment and service.

# Folder structure
/07-with-k8s
.
├── api_gateway.py
├── depolyment.yaml
├── Dockerfile_apigateway
├── Dockerfile_order
├── Dockerfile_user
├── order_service.py
└── user_service.py

# Build, tag and push the docker images
docker login

docker build -t zackz001/python-user:latest -f Dockerfile_user .
docker build -t zackz001/python-order:latest -f Dockerfile_order .
docker build -t zackz001/python-apigateway:latest -f Dockerfile_apigateway .
docker push zackz001/python-apigateway:latest
docker push zackz001/python-user:latest
docker push zackz001/python-order:latest

docker image ls

REPOSITORY                                      TAG       IMAGE ID       CREATED        SIZE
zackz001/python-apigateway                      latest    bc3db11f4be8   1 hours ago    138MB
zackz001/python-user                            latest    c93973bece33   1 hours ago    136MB
zackz001/python-order                           latest    e35d5de9254b   1 hours ago    136MB
prom/prometheus                                 latest    1bd2b9635267   8 days ago     271MB
grafana/grafana                                 latest    c42c21cd0ebc   3 weeks ago    453MB
consul                                          1.15.4    686495461132   4 months ago   155MB
docker.elastic.co/elasticsearch/elasticsearch   7.13.2    11a830014f7c   3 years ago    1.02GB
docker.elastic.co/logstash/logstash             7.13.2    8dc1af4dd662   3 years ago    965MB
docker.elastic.co/kibana/kibana                 7.13.2    6c4869a27be1   3 years ago    1.35GB

# k8s deployment Manifests

apiVersion: apps/v1
kind: Deployment
metadata:
 name: user-service
spec:
 replicas: 1
 selector:
 matchLabels:
 app: user-service
 template:
 metadata:
 labels:
 app: user-service
 spec:
 containers:
 - name: user-service
 image: zackz001/python-user:latest
 ports:
 - containerPort: 5001

---
apiVersion: v1
kind: Service
metadata:
 name: user-service
spec:
 selector:
 app: user-service
 ports:
 - protocol: TCP
 port: 5001
 targetPort: 5001

---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: order-service
spec:
 replicas: 1
 selector:
 matchLabels:
 app: order-service
 template:
 metadata:
 labels:
 app: order-service
 spec:
 containers:
 - name: order-service
 image: zackz001/python-order:latest
 ports:
 - containerPort: 5002

---
apiVersion: v1
kind: Service
metadata:
 name: order-service
spec:
 selector:
 app: order-service
 ports:
 - protocol: TCP
 port: 5002
 targetPort: 5002

---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: api-gateway
spec:
 replicas: 1
 selector:
 matchLabels:
 app: api-gateway
 template:
 metadata:
 labels:
 app: api-gateway
 spec:
 containers:
 - name: api-gateway
 image: zackz001/python-apigateway:latest
 ports:
 - containerPort: 5000

---
apiVersion: v1
kind: Service
metadata:
 name: api-gateway
spec:
 type: NodePort
 selector:
 app: api-gateway
 ports:
 - protocol: TCP
 port: 5000
 targetPort: 5000

Now Run kubectl apply -f to bring all deployments and services up and running. Should see all services in the Rancher console.

Python Microservice: Monitoring Stack

Tue, 21 May 2024 04:09:17 +0000

Both Prometheus and Grafana are compatible with microservice applications, integrating Prometheus with Flask is straightforward to provide performance and monitoring metrics.

Here I will update the docker-compose to add Prometheus and Grafana as services, then add Prometheus metrics in both order and user application code by importing PrometheusMetrics from the prometheus_flask_exporter module, which is used to expose Prometheus metrics for the Flask application. Then initialize Prometheus metrics with metrics = PrometheusMetrics(app). Use @metrics.counter('get_orders_count', 'Count of calls to the get_orders endpoint') to define a Prometheus counter metric that increments each time the get_orders endpoint is called.

Python Microservice: Logging with ELK

Mon, 20 May 2024 04:08:51 +0000

ELK (Elasticsearch, Logstash, Kibana) is a popular log management solution. We will use the ELK Stack to collect and analyze logs.

Here I need to extend the current configuration by adding services in the docker-compose file for Elasticsearch, Logstash, and Kibana, and configure the microservices to send logs to Logstash.

Also, I will need to configure the logging in both user and order Python application code to send logs to Logstash. By importing the built-in logging module and GelfUdpHandler from the pygelf module, to provide a flexible framework for emitting log messages from Python programs to send log messages in the GELF (Graylog Extended Log Format) to a remote Graylog server, which is typically part of the ELK stack.

Python Microservice: API Gateway & Consul

Sun, 19 May 2024 03:56:00 +0000

API Gateway acts as a single entry point for all clients and handles the request routing, composition, and protocol translation in a microservices architecture, here I will create an API Gateway using Python Flask and the requests library, to route both “user” and “order” services.

Here I will create an API Gateway to handle the 2 services (user and order). By importing the requests module, which allows us to send HTTP requests in Python. It’s used for making API calls to other services.

Python Microservice: Containerization

Sat, 18 May 2024 03:55:16 +0000

Flask stands out as one of Python’s most popular web frameworks. Designed for versatility and ease of use, Flask offers a robust starting point for crafting web apps.

By the following posts, I will use Flask to:

Create a series of Python web applications from simple app (Hello Zack)
Develop microservices applications (order and user), deploy using Docker compose
Create Python API gateway application
Integrate with Consul for service discovery and register
Enable logging with ELK, monitoring with Prometheus & Grafana
Lastly, I will create Kubernetes manifest for K8S deployment

I will skip ArgoCD and Github Action as I had done similar posts before.

Python: File Handling for AWS tagging

Fri, 17 May 2024 03:19:57 +0000

In the previous post I developed shell script + awscli to apply aws EC2 tags, since the last post we discovered Python Boto3 scripts for AWS resource automation and management, I think it is time to improve the EC2 tagging task with Python and boto3, together with file handling to achieve:

List and export EC2 information to a CSV file (instanceID, default instance name, Existing tags)
Define 4 mandatory tags in CSV header (Env, BizOwner, Technology, Project)
Validate exported tags against the 4 mandatory new tags, if any of the new mandatory tags exists, then keep the tag and value, if any of the new mandatory tags do not exist, add the key and leave the value blank
Get CSV file filled with mandatory tags input from Biz team (manual work)
Open the updated CSV file, apply the mandatory tags based on the input value
Create and trigger Lambda function with AWS config rules to enforce 4 mandatory tags whenever a new instance is launched

List and export EC2 information to a CSV

Python: Boto3 for AWS

Thu, 16 May 2024 03:19:36 +0000

Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python. It enables developers to build software that uses Amazon services like EC2, S3, RDS, etc.

I will build a portable Python3.9 + Boto3 Docker environment to test some AWS automation tasks.

Build and run a docker with Python3.9 + Boto3

As I do not want to install Python, Boto3, and AWScli on my local PC, creating a Docker image with all software ready as a portable env is the best way to start.

Modularizing AWS SSO User Assignment with Terraform

Sat, 13 Apr 2024 03:17:24 +0000

In the company, we use AWS SSO for user authentication; when a new user is created from Azure AD, it will automatically synced by AWS IAM Identity Center and Azure AD integration, and then our team will need to handle the SSO user assignment to put them in the required AWS accounts with requested permission sets, it became a pain when such requests coming more frequently and every time an individual user or a whole team with different accounts and permission requirements need to be fulfilled, so how to handle this efficiently become my recent topic.

Helm + Kustomize for ZackBlog

Tue, 12 Mar 2024 03:17:00 +0000

Kustomize is a configuration management tool for Kubernetes that allows users to customize application manifests without modifying the original YAML files, in this post I will explore Kustomize with overlays, bases, and transformers, then create simple kustomization.yaml files for different environments using Zackblog, then to practice using Kustomize’s built-in resources like configMapGenerator and secretGenerator.

Let’s take Zackblog k8s deployment manifest as an example and convert it into a Kustomize setup. We will start by organizing the files and gradually exploring key Kustomize features.

Automate AMI patching with Lambda and Cloudformation

Thu, 01 Feb 2024 03:14:05 +0000

The application team managing Rancher clusters in AWS EC2 faced a compliance challenge with their Rancher node template golden AMI. To meet security and compliance requirements, they needed to ensure that this AMI is patched regularly with the latest updates. This process had to be automated to guarantee that a newly patched AMI is available every month.

Overview of workflow with Lambda and Cloudformation

I developed this CloudFormation template to set up an automated process for patching Amazon Machine Images (AMIs) on a monthly schedule using AWS services such as Lambda, EventBridge, SNS, and Parameter Store. Here’s a workflow by design:

PostgreSQL: Prod-Grade with k8s Operator

Thu, 11 Jan 2024 03:13:44 +0000

Despite all the challenges, in the last 2 years, clever people still managed ways to deploy production-grade database within a Kubernetes cluster by using Kubernetes as a platform to develop custom resource definition (CRDs) like helm charts like bitnami/postgresql-ha, and PostgreSQL Operator like CrunchyData/postgres-operator or zalando/postgres-operator.

Last post I was able to deploy a single PostgreSQL in local k8s, but I had to manually create Kubernetes namespaces, define database creds, configuration and environment variables into k8s secret and configmap, also to define the statefulset yaml with volume claim template.

PostgreSQL: Deploy into K8S

Wed, 10 Jan 2024 03:13:24 +0000

PostgreSQL by default does not build for Kubernetes, and a database with StatefulSet workload in K8S can be brutal to manage. In my lab K8S cluster, here we create namespace, secret, configmap, PVC, and StatefulSet to run a single PostgreSQL.

# create namespace "postgresql"
[root@freeipa-server ~]# kubectl create ns postgresql
namespace/postgresql created

# create secret to store database creds
[root@freeipa-server ~]# kubectl -n postgresql create secret generic postgresql --from-literal POSTGRES_USER="postgresadmin" --from-literal POSTGRES_PASSWORD='admin123' --from-literal POSTGRES_DB="postgresdb" --from-literal REPLICATION_USER="replicationuser" --from-literal REPLICATION_PASSWORD='replicationPassword'
secret/postgresql created

# create configmap, pvc, statefulset with init container to run postgresql
[root@freeipa-server ~]# vim stateful.yaml

apiVersion: v1
kind: ConfigMap
metadata:
 name: postgres
data:
 pg_hba.conf: |+
 # TYPE DATABASE USER ADDRESS METHOD
 host replication replicationuser 0.0.0.0/0 md5
 # "local" is for Unix domain socket connections only
 local all all trust
 # IPv4 local connections:
 host all all 127.0.0.1/32 trust
 # IPv6 local connections:
 host all all ::1/128 trust
 # Allow replication connections from localhost, by a user with the
 # replication privilege.
 local replication all trust
 host replication all 127.0.0.1/32 trust
 host replication all ::1/128 trust

 host all all all scram-sha-256
 postgresql.conf: |+
 data_directory = '/data/pgdata'
 hba_file = '/config/pg_hba.conf'
 ident_file = '/config/pg_ident.conf'

 port = 5432
 listen_addresses = '*'
 max_connections = 100
 shared_buffers = 128MB
 dynamic_shared_memory_type = posix
 max_wal_size = 1GB
 min_wal_size = 80MB
 log_timezone = 'Etc/UTC'
 datestyle = 'iso, mdy'
 timezone = 'Etc/UTC'

 #locale settings
 lc_messages = 'en_US.utf8' # locale for system error message
 lc_monetary = 'en_US.utf8' # locale for monetary formatting
 lc_numeric = 'en_US.utf8' # locale for number formatting
 lc_time = 'en_US.utf8' # locale for time formatting

 default_text_search_config = 'pg_catalog.english'

 #replication
 wal_level = replica
 archive_mode = on
 archive_command = 'test ! -f /data/archive/%f && cp %p /data/archive/%f'
 max_wal_senders = 3
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: postgres
spec:
 selector:
 matchLabels:
 app: postgres
 serviceName: "postgres"
 replicas: 1
 template:
 metadata:
 labels:
 app: postgres
 spec:
 terminationGracePeriodSeconds: 30
 initContainers:
 - name: init
 image: postgres:15.0
 command: [ "bash", "-c" ]
 args:
 - |
 #create archive directory
 mkdir -p /data/archive && chown -R 999:999 /data/archive
 volumeMounts:
 - name: data
 mountPath: /data
 readOnly: false
 containers:
 - name: postgres
 image: postgres:15.0
 args: ["-c", "config_file=/config/postgresql.conf"]
 ports:
 - containerPort: 5432
 name: database
 env:
 - name: PGDATA
 value: "/data/pgdata"
 - name: POSTGRES_USER
 valueFrom:
 secretKeyRef:
 name: postgresql
 key: POSTGRES_USER
 optional: false
 - name: POSTGRES_PASSWORD
 valueFrom:
 secretKeyRef:
 name: postgresql
 key: POSTGRES_PASSWORD
 optional: false
 - name: POSTGRES_DB
 valueFrom:
 secretKeyRef:
 name: postgresql
 key: POSTGRES_DB
 optional: false
 volumeMounts:
 - name: config
 mountPath: /config
 readOnly: false
 - name: data
 mountPath: /data
 readOnly: false
 volumes:
 - name: config
 configMap:
 name: postgres
 defaultMode: 0755
 volumeClaimTemplates:
 - metadata:
 name: data
 spec:
 accessModes: [ "ReadWriteOnce" ]
 storageClassName: "standard"
 resources:
 requests:
 storage: 100Mi
---
apiVersion: v1
kind: Service
metadata:
 name: postgres
 labels:
 app: postgres
spec:
 ports:
 - port: 5432
 targetPort: 5432
 name: postgres
 clusterIP: None
 selector:
 app: postgres

[root@freeipa-server ~]# kubectl create -f stateful.yaml -n postgresql
configmap/postgres created
statefulset.apps/postgres created
service/postgres created

# validate for pvc, pods
[root@freeipa-server ~]# kubectl get pvc -n postgresql
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-postgres-0 Bound pvc-dd89fc0a-915f-40eb-b61f-917234074a61 100Mi RWO longhorn 19m

[root@freeipa-server ~]# kubectl get all -n postgresql
NAME READY STATUS RESTARTS AGE
pod/postgres-0 1/1 Running 0 6m29s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/postgres ClusterIP None <none> 5432/TCP 6m29s

NAME READY AGE
statefulset.apps/postgres 1/1 6m29s

# check container logs for database connection status
[root@freeipa-server ~]# kubectl logs -n postgresql postgres-0
Defaulted container "postgres" out of: postgres, init (init)
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /data/pgdata ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
syncing data to disk ... ok

Success. You can now start the database server using:

 pg_ctl -D /data/pgdata -l logfile start

waiting for server to start....2024-01-12 00:52:56.718 UTC [49] LOG: starting PostgreSQL 15.0 (Debian 15.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2024-01-12 00:52:56.719 UTC [49] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-01-12 00:52:56.730 UTC [49] LOG: could not open usermap file "/config/pg_ident.conf": No such file or directory
2024-01-12 00:52:56.733 UTC [52] LOG: database system was shut down at 2024-01-12 00:52:55 UTC
2024-01-12 00:52:56.744 UTC [49] LOG: database system is ready to accept connections
 done
server started
CREATE DATABASE

/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

2024-01-12 00:52:56.957 UTC [49] LOG: received fast shutdown request
waiting for server to shut down....2024-01-12 00:52:56.961 UTC [49] LOG: aborting any active transactions
2024-01-12 00:52:56.962 UTC [49] LOG: background worker "logical replication launcher" (PID 56) exited with exit code 1
2024-01-12 00:52:56.963 UTC [49] LOG: shutting down
2024-01-12 00:52:57.042 UTC [49] LOG: checkpoint starting: shutdown immediate
..2024-01-12 00:52:59.314 UTC [49] LOG: checkpoint complete: wrote 918 buffers (5.6%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.434 s, sync=0.014 s, total=2.279 s; sync files=250, longest=0.007 s, average=0.001 s; distance=11271 kB, estimate=11271 kB
2024-01-12 00:52:59.318 UTC [49] LOG: database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2024-01-12 00:52:59.385 UTC [1] LOG: starting PostgreSQL 15.0 (Debian 15.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2024-01-12 00:52:59.385 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2024-01-12 00:52:59.385 UTC [1] LOG: listening on IPv6 address "::", port 5432
2024-01-12 00:52:59.389 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-01-12 00:52:59.398 UTC [1] LOG: could not open usermap file "/config/pg_ident.conf": No such file or directory
2024-01-12 00:52:59.404 UTC [67] LOG: database system was shut down at 2024-01-12 00:52:59 UTC
2024-01-12 00:52:59.415 UTC [1] LOG: database system is ready to accept connections

Conclusion

PostgreSQL: Replication & Failover

Mon, 08 Jan 2024 03:10:54 +0000

Last post I started a hands-on session with PostgreSQL installation on both Docker and Docker Compose, explored important PostgreSQL configuration, and was able to mount persistent volumes and config files to customize PostgreSQL.

In this post, I will set up a second PostgreSQL instance to establish primary and standby replication for PostgreSQL HA, using some pg tools. Lastly, we will test failover by shutting down the primary and promoting the standby instance.

PostgreSQL: Get Started

Sat, 06 Jan 2024 03:10:34 +0000

PostgreSQL is a very popular open-source relational database management system (RDBMS), known for its extensibility and feature-rich capabilities, making it suitable for mission-critical applications. Not to mention its active PostgreSQL community.

In the upcoming posts, I will start a series of PostgreSQL studies to:

Explore PostgreSQL main features, installation, and basic administration tasks.
Deploy a PostgreSQL cluster onto K8S with PostgreSQL Operator, validate backup and rolling upgrades.
Create a simple Flash microservice application to connect to the PostgreSQL cluster and validate failover.
Integrate the whole deployment into a CICD pipeline for automation.
Create AWS RDS PostgreSQL, with S3 Block storage as a replica.

By the end of the series, we should be able to have a comprehensive understanding of PostgreSQL from a DevOps perspective.

Istio: Distributed Tracing with Jaeger

Sun, 24 Dec 2023 03:08:40 +0000

Last post we were able to deploy Istio and manage traffic for a book review microservice application. This session we will dive deeper into Istio for its add-on Jaeger for Microservice tracing.

Jaeger is an open-source end-to-end distributed tracing tool to monitor and troubleshoot the performance of microservices-based distributed systems by providing insights into the latency and other performance metrics.

Trace

A trace represents the entire journey of a request or transaction as it propagates through various services and components of a distributed system. It captures the path the request takes, including all the microservices it interacts with, from start to finish. A trace is composed of multiple spans.

Istio: Traffic Routing

Fri, 22 Dec 2023 03:07:00 +0000

Here we use helm to install istio (istio-base, istiod, istio gateway), then deploy a sample online book store microservice “bookinfo”. Practise istio tasks include Traffic Management, Observability, Security.

Bookinfo Topology:

Helm install istio (istiod, istio-ingress)

kubectl create namespace istio-system
helm pull istio/base
helm install istio-base . -n istio-system --set defaultRevision=default

helm pull istio/istiod
helm install istiod . -n istio-system

kubectl create namespace istio-ingress
helm pull istio/gateway
helm install istio-ingress . -n istio-ingress

helm ls -n istio-system
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
istio-base istio-system 1 2023-12-17 08:14:06.943276388 +0800 CST deployed base-1.20.1 1.20.1
istiod istio-system 1 2023-12-17 08:15:40.370551503 +0800 CST deployed istiod-1.20.1 1.20.1

helm ls -n istio-ingress
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
istio-ingress istio-ingress 1 2023-12-17 08:25:07.111999373 +0800 CST deployed gateway-1.20.1 1.20.1

Deploy bookinfo microservice and istio ingressgateway and virtualservice

kubectl label namespace istio-system istio-injection=enabled

kubectl apply -f https://github.com/istio/istio/blob/master/samples/bookinfo/platform/kube/bookinfo.yaml -oyaml > bookinfo.yaml
kubectl apply -f bookinfo.yaml

kubectl get po
NAME READY STATUS RESTARTS AGE
details-v1-698d88b-wmfcb 2/2 Running 0 21m
ratings-v1-6484c4d9bb-cb6gx 2/2 Running 0 21m
reviews-v1-5b5d6494f4-jrsvc 2/2 Running 0 21m
reviews-v2-5b667bcbf8-jgfzj 2/2 Running 0 21m
reviews-v3-5b9bd44f4-tmmfz 2/2 Running 0 21m

kubectl apply -f https://github.com/istio/istio/blob/master/samples/bookinfo/networking/bookinfo-gateway.yaml -oyaml > bookinfo-gateway.yaml
kubectl apply -f bookinfo-gateway.yaml

Deploy Kiali, jaeger, grafana, prometheus

wget https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/prometheus.yaml
wget https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/jaeger.yaml
wget https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/grafana.yaml
kubectl create -f prometheus.yaml -f jaeger.yaml -f grafana.yaml

Visit http://book.istio:31000/productpage, with review (v1, v2, v3)

Ubuntu 18.04 to 22.04 in-place upgrade

Mon, 28 Aug 2023 03:04:11 +0000

My manage said we have many ubuntu 16.04, can you believe?

Every single Ubuntu LTS comes with 5 years of standard support. During those five years, bug fixes and security patches will be provided. Ubuntu 18.04 ‘Bionic Beaver’ is reaching End of Standard Support this May. So today we are going to run in-place upgrade for Ubuntu 18.04 LTS to 22.04 LTS.

Pre-upgrade checklist

Validate current OS version and running service (nginx)

# current OS version
root@ubuntu-test:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

# nginx service status

root@ubuntu-test:~# echo "ubuntu-inplace-upgrade zack-testing-nginx-service!!" >> /var/www/html/index.html
root@ubuntu-test:~# systemctl restart nginx
root@ubuntu-test:~# curl localhost
ubuntu-inplace-upgrade zack-testing-nginx-service!!

Fully update the system

# update system
root@ubuntu-test:~# sudo apt update
Hit:1 http://au.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://au.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:3 http://au.archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:4 http://au.archive.ubuntu.com/ubuntu bionic-security InRelease
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
root@ubuntu-test:~# sudo apt upgrade -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

# reboot system before upgrade
root@ubuntu-test:~# sudo do-release-upgrade
Checking for a new Ubuntu release
You have not rebooted after updating a package which requires a reboot. Please reboot before upgrading.
root@ubuntu-test:~# reboot
Connection closing...Socket close.

Take full system backup

Here I took a VM snapshot before upgrade

Move To Rancher

Mon, 26 Jun 2023 03:00:12 +0000

My current employer using Rancher, due to the vender prefers this??

So far many different ways I have used to deploy k8s cluster, each with its own pros and cons.

home lab build k8s components (etcd, keepalived, apiserver, scheduler, coreDNS, calico)
home lab k8s cluster with kubeadm
k8s on AWS using kops and eksctl
AWS self-managed k8s cluster directly on ec2 by ansible

Rancher Support matrix

Local docker Installation

Kubespary with Terraform on AWS

Wed, 24 May 2023 02:40:31 +0000

Kubespary is a powerful and highly configurable tool that automates and supports deployment on various cloud providers (AWS, GCE, Azure, etc.) and on-premises infrastructure. In this post I will run its build-in Terraform and ansible code to provision a k8s cluster based on AWS EC2.

Benefit of Kubespary

We can define specific settings in cluster-config.yaml for networking, authentication, and other Kubernetes features, it also supports for multiple networking plugins (Calico, Flannel, Weave Net, etc.), By managing hosts file and rerun the playbook to make add or remove nodes easily, Integrated CI/CD Pipelines and Infrastructure as Code (IaC) with Terraform and Ansible to achieve automation.

ZackBlog AWS Serverless webhosting

Sun, 30 Apr 2023 03:02:02 +0000

In this article, I will see how to host “zackweb” as a static web application using the following AWS serverless options:

S3 static webhosting
AWS CDK + CloudFront

Prerequisite

Add one more step in the existing Github Action workflow to copy the static web content to the newly created S3 bucket

# edit github action workflow
aws s3 cp ~/zack-gitops-project/zack_blog/_site/* s3://zackweb-serverless/ --recursive

# validate content in s3 bucket
ubuntu@ip-172-31-26-78:~$ aws s3 ls s3://zackweb-serverless --summarize
 PRE aboutme/
 PRE assets/
 PRE certificate/
 PRE gitrepo/
 PRE jekyll/
 PRE pro/
 PRE skillroadmap/
2023-07-30 14:55:05 4455 404.html
2023-07-30 14:55:05 504 Dockerfile
2023-07-30 14:55:06 80555 feed.xml
2023-07-30 14:55:06 7760 index.html
2023-07-30 14:55:06 0 nginx.conf

Total Objects: 5
 Total Size: 93274

Option 1: S3 static webhosting

Enable Free SSL certificate for blog

Tue, 07 Feb 2023 23:28:37 +0000

An interviewer told me this blog is insecure!!

Generate free SSL certificate from https://zerossl.com/
Validate the Certificate with Private Key via https://www.sslshopper.com/certificate-key-matcher.html
Upload ‘certificate.crt’ and ‘private.key’ to web server /etc/nginx/ssl/
Setting up NGINX HTTPS Server by including the ssl parameter to the listen directive in the server block under ‘http’ in ’nginx.conf’:

http {
 server {
 listen 443 ssl;
 server_name zackdevops.online;

 ssl_certificate /etc/nginx/ssl/certificate.crt;
 ssl_certificate_key /etc/nginx/ssl/private.key;
 }
 ...
}

Fix 2 errors:

2023/02/07 10:29:33 [emerg] 73175#73175: "server" directive is not allowed here in /etc/nginx/nginx.conf:11

2023/02/07 10:30:25 [error] 73207#73207: *1 directory index of "/usr/share/nginx/html/" is forbidden,client: 163.53.144.82, server: zackdevops.online, request: "GET / HTTP/1.1", host: "zackdevops.online"
2023/02/07 10:30:36 [error] 73207#73207: *1 directory index of "/usr/share/nginx/html/" is forbidden, client: 163.53.144.82, server: zackdevops.online, request: "GET / HTTP/1.1", host: "zackdevops.online"

Bingo! https://zackdevops.online connection is secure!

Ansible for AWS Dynamic Inventory

Sun, 22 Jan 2023 23:26:07 +0000

When using Ansible with AWS, maintaining the inventory file will be a hectic task as AWS has frequently changed IPs, autoscaling instances, and much more.

Here we will install and apply ansible plugin for AWS dynamic inventory which makes an API call to AWS to get the instance information in the run time. It gives the EC2 instance details dynamically to manage the AWS infrastructure.

It supports most of the public and private cloud platforms not limited to just AWS.

Redis cluster with Helm

Fri, 09 Dec 2022 23:23:27 +0000

Today we are playing around with Redis, Here we use helm to install Redis cluster, then validate statefulset storage and cluster availability.

Typical Redis cluster (3 master + 3 slave for slots) Topology:

Helm install bitnami/redis-cluster

helm repo add bitnami https://charts.bitnami.com/bitnami
helm pull bitnami/redis-cluster
kubectl create ns redis
helm install zz-redis bitnami/redis-cluster -n redis

Redis-cluster status

kubectl get po | grep zz-redis
zz-redis-redis-cluster-0 1/1 Running 3 (33m ago) 36m
zz-redis-redis-cluster-1 1/1 Running 1 (32m ago) 36m
zz-redis-redis-cluster-2 1/1 Running 1 (33m ago) 36m
zz-redis-redis-cluster-3 1/1 Running 1 (33m ago) 36m
zz-redis-redis-cluster-4 1/1 Running 1 (33m ago) 36m
zz-redis-redis-cluster-5 1/1 Running 1 (33m ago) 36m

kubectl get svc
zz-redis-redis-cluster ClusterIP 10.96.68.34 <none> 6379/TCP 37m
zz-redis-redis-cluster-headless ClusterIP None <none> 6379/TCP,16379/TCP 37m

kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
redis-data-zz-redis-redis-cluster-0 Bound pvc-81f69c15-2f59-4704-9fec-c3ab217ebca5 1Gi RWO rook-ceph-block 37m
redis-data-zz-redis-redis-cluster-1 Bound pvc-871dbb68-a78b-48a8-8feb-8726eb8a795e 1Gi RWO rook-ceph-block 37m
redis-data-zz-redis-redis-cluster-2 Bound pvc-afd8c82e-c314-426a-a3cd-3c8d10b42bb1 1Gi RWO rook-ceph-block 37m
redis-data-zz-redis-redis-cluster-3 Bound pvc-e8fd5d47-dd60-4358-8cf5-a17d6574bbe2 1Gi RWO rook-ceph-block 37m
redis-data-zz-redis-redis-cluster-4 Bound pvc-04d4f148-b7c1-407e-b9a8-b2fa911405f0 1Gi RWO rook-ceph-block 37m
redis-data-zz-redis-redis-cluster-5 Bound pvc-99e8b2cc-c5d5-4636-a19b-042922eca3cc 1Gi RWO rook-ceph-block 37m

Validate cluster by set key

Apache Kafka cluster with testing topics

Fri, 09 Dec 2022 22:09:32 +0000

Apache Kafka has proven to be an extremely popular event streaming platform, as its scalable distributed architecture, high performance, and use cases, some key terms and concepts as below:

Kafka clusters and Kafka brokers
Kafka clients and servers
Producers, and Consumers, and Consumer groups
Kafka topics & Kafka partitions, offsets
Kafka topic replication, leaders, and followers
ZooKeeper or not

Typical Kafka Topology:

Helm install bitnami/kafka

Here we use helm to install Kafka, then validate statefulset storage and cluster availability by creating a topic, producer, and consumer.

Ingress controller for Subdomain based Routing

Fri, 18 Nov 2022 22:01:55 +0000

In this post I will run the steps required to set up MetalLB for LoadBalancer IP allocation, the NGINX Ingress Controller for routing traffic based on subdomains, and how to configure local DNS to enable subdomain-based routing for Kubernetes services.

Ingress-nginx install

We will use Helm to install our ingress controller ingress-nginx, which will be used to route traffic based on subdomains.

# install ingress controller ingress-nginx
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace

# check for ingressclasses
kubectl get ingressclasses.networking.k8s.io
NAME CONTROLLER PARAMETERS AGE
nginx k8s.io/ingress-nginx <none> 149m

MetalLB for loadbalancing

Rook-Ceph for dynamic Persistent Volume

Wed, 09 Nov 2022 23:01:53 +0000

Today, I will explore how to install and configure Persistent Volume for all the deployment in K8S to consume.

Helm install Rook-Ceph for persistent storage

Configure local VM block storage to add 50Gb sdb to all k8s master and worker nodes
Install rook-ceph cluster

git clone --single-branch --branch master https://github.com/rook/rook.git
cd rook/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f operator.yaml
kubectl create -f cluster.yaml

Ceph toolbox to check cluster status

kubectl create -f toolbox.yaml
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash
ceph status
ceph osd status
ceph df
rados df

Ceph Dashboard service for HTTPS login

kubectl create -f dashboard-external-https.yaml
kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

Create storage pool and storage class

kubectl create -f pool.yaml
cd csi/rbd
kubectl create -f storageclass.yaml

Set “rook-ceph-block” as the default storage class

kubectl patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
rook-ceph-block (default) rook-ceph.rbd.csi.ceph.com Delete Immediate true 8d

Check pool and OSDs in Ceph web UI

GitOps with Argo CD

Mon, 07 Nov 2022 21:46:56 +0000

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes applications. This is a post to show how to enable Argo CD on local k8s and AWS EKS, deploy applications and sync with GitHub manifests.

Argo CD install ingress controller ingress-nginx

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Download and install Argo CD CLI

curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
rm argocd-linux-amd64

Configure Argo CD API Server and deploy zackweb

Change the argocd-server service type to NodePort, initialize admin password, and deploy “zackweb” via Argo CD application manifests (argo-zackweb-application.yaml).

About Zack-web Gitops project!

Wed, 02 Nov 2022 14:22:08 +0000

Project Introduction

This is my first web blog using Jekyll, as a practical way by following Cloud Resume Challenge to build my cloud and devops concept and technical skillsets. ~~

The Design

By design, I will create:

a web blog Zack’s Blog: with content and details to introduce myself
a github repo zack-gitops-project: to source control all code that I build and run locally by “jekyll serve”, validate site and pages, then push the source code to github.
a Dockerfile to build jekyll code into a docker image running by nginx/alpine
3 folders with manifests for staging and prod webapp deploy:
- /terraform for creating AWS VPC and EKS to host the website as the production environment;
- /k8s-local-deploy for website image deployment to local K8S as testing;
- /eks-deploy for prod deployment with ArgoCD application manifest
a EC2 instance: as staging environment for AWS with Godaddy domain hosting test
a EKS cluster: as PROD environment to validate ArgoCD sync for web deployment

The Architecture