Posts

“Such a long time I nearly forgot how to use Jenkins..” Jenkins recap It has been some time since I adapted CI/CD pipelines from Jenkins to AWS CodePipeline and GitHub Actions workflows. Now, it’s time to recap and improve some of my previous Jenkins practices. Universal Jenkins Docker image design This time, instead of installing Jenkins on a server, I prefer to containerize a universal Jenkins Docker image with the necessary packages installed, so it provides consistency and reproducibility, portability, and easy updates and rollbacks. ...

When a company faces challenge to manage its Linux environments across local and public cloud, RedHat Identity management can be the solution to achieve: With Local AD and Azure AD (AAD) Integration With AWS SSO Integration as external identity provider LDAP, Kerberos and NTP A web-based management front-end running on Apache A Typical AD User Authentication Flow End-to-End: User Creation and Management: Azure AD / Local AD: Users are created in the Azure Active Directory or local Active Directory. Synchronization to RedHat IdM: The users are synchronized from AD to RedHat IdM using the two-way trust established between AD and IdM. Accessing EC2 Instances via SSH: ...

‘if people tend to move to serverless, how ‘infrastructure engineer’ will end up’ Why go serverless Some of the company’s applications recently moved from Rancher to Fargate, which is understandable as the cloud resource and traffic will be very intensive only during a certain period (HSC exam), hence AWS serverless with Fargate can be a better option for such business mode so the rest of the year without the exam we can save costs significantly. ...

Today I got a performance issue from our analytic team, saying they experienced a Production MySQL cluster running on RDS very slow since yesterday morning. I started to look into below areas for investigation: AWS CloudWatch Metrics for RDS AWS CloudWatch provides a wide range of metrics that can help diagnose resource usage for databases. So I started with: CloudWatch - Metrics - All metrics - Add query - RDS - Top 10 RDS instances by highest CPU utilization ...

Now It is time to change from docker-compose to deploy into Kubernetes. As this is not new to me to deploy microservice into K8S, also I already have a running Kubernetes cluster in hand, so here I will just create docker images for the 3 services: API gateway, user and order, then push them into the docker hub repository, then create Kubernetes manifest for deployment and service. # Folder structure /07-with-k8s . ├── api_gateway.py ├── depolyment.yaml ├── Dockerfile_apigateway ├── Dockerfile_order ├── Dockerfile_user ├── order_service.py └── user_service.py # Build, tag and push the docker images docker login docker build -t zackz001/python-user:latest -f Dockerfile_user . docker build -t zackz001/python-order:latest -f Dockerfile_order . docker build -t zackz001/python-apigateway:latest -f Dockerfile_apigateway . docker push zackz001/python-apigateway:latest docker push zackz001/python-user:latest docker push zackz001/python-order:latest docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE zackz001/python-apigateway latest bc3db11f4be8 1 hours ago 138MB zackz001/python-user latest c93973bece33 1 hours ago 136MB zackz001/python-order latest e35d5de9254b 1 hours ago 136MB prom/prometheus latest 1bd2b9635267 8 days ago 271MB grafana/grafana latest c42c21cd0ebc 3 weeks ago 453MB consul 1.15.4 686495461132 4 months ago 155MB docker.elastic.co/elasticsearch/elasticsearch 7.13.2 11a830014f7c 3 years ago 1.02GB docker.elastic.co/logstash/logstash 7.13.2 8dc1af4dd662 3 years ago 965MB docker.elastic.co/kibana/kibana 7.13.2 6c4869a27be1 3 years ago 1.35GB # k8s deployment Manifests apiVersion: apps/v1 kind: Deployment metadata: name: user-service spec: replicas: 1 selector: matchLabels: app: user-service template: metadata: labels: app: user-service spec: containers: - name: user-service image: zackz001/python-user:latest ports: - containerPort: 5001 --- apiVersion: v1 kind: Service metadata: name: user-service spec: selector: app: user-service ports: - protocol: TCP port: 5001 targetPort: 5001 --- apiVersion: apps/v1 kind: Deployment metadata: name: order-service spec: replicas: 1 selector: matchLabels: app: order-service template: metadata: labels: app: order-service spec: containers: - name: order-service image: zackz001/python-order:latest ports: - containerPort: 5002 --- apiVersion: v1 kind: Service metadata: name: order-service spec: selector: app: order-service ports: - protocol: TCP port: 5002 targetPort: 5002 --- apiVersion: apps/v1 kind: Deployment metadata: name: api-gateway spec: replicas: 1 selector: matchLabels: app: api-gateway template: metadata: labels: app: api-gateway spec: containers: - name: api-gateway image: zackz001/python-apigateway:latest ports: - containerPort: 5000 --- apiVersion: v1 kind: Service metadata: name: api-gateway spec: type: NodePort selector: app: api-gateway ports: - protocol: TCP port: 5000 targetPort: 5000 Now Run kubectl apply -f to bring all deployments and services up and running. Should see all services in the Rancher console. ...

Both Prometheus and Grafana are compatible with microservice applications, integrating Prometheus with Flask is straightforward to provide performance and monitoring metrics. Here I will update the docker-compose to add Prometheus and Grafana as services, then add Prometheus metrics in both order and user application code by importing PrometheusMetrics from the prometheus_flask_exporter module, which is used to expose Prometheus metrics for the Flask application. Then initialize Prometheus metrics with metrics = PrometheusMetrics(app). Use @metrics.counter('get_orders_count', 'Count of calls to the get_orders endpoint') to define a Prometheus counter metric that increments each time the get_orders endpoint is called. ...

ELK (Elasticsearch, Logstash, Kibana) is a popular log management solution. We will use the ELK Stack to collect and analyze logs. Here I need to extend the current configuration by adding services in the docker-compose file for Elasticsearch, Logstash, and Kibana, and configure the microservices to send logs to Logstash. Also, I will need to configure the logging in both user and order Python application code to send logs to Logstash. By importing the built-in logging module and GelfUdpHandler from the pygelf module, to provide a flexible framework for emitting log messages from Python programs to send log messages in the GELF (Graylog Extended Log Format) to a remote Graylog server, which is typically part of the ELK stack. ...

API Gateway acts as a single entry point for all clients and handles the request routing, composition, and protocol translation in a microservices architecture, here I will create an API Gateway using Python Flask and the requests library, to route both “user” and “order” services. Here I will create an API Gateway to handle the 2 services (user and order). By importing the requests module, which allows us to send HTTP requests in Python. It’s used for making API calls to other services. ...

Flask stands out as one of Python’s most popular web frameworks. Designed for versatility and ease of use, Flask offers a robust starting point for crafting web apps. By the following posts, I will use Flask to: Create a series of Python web applications from simple app (Hello Zack) Develop microservices applications (order and user), deploy using Docker compose Create Python API gateway application Integrate with Consul for service discovery and register Enable logging with ELK, monitoring with Prometheus & Grafana Lastly, I will create Kubernetes manifest for K8S deployment I will skip ArgoCD and Github Action as I had done similar posts before. ...

In the previous post I developed shell script + awscli to apply aws EC2 tags, since the last post we discovered Python Boto3 scripts for AWS resource automation and management, I think it is time to improve the EC2 tagging task with Python and boto3, together with file handling to achieve: List and export EC2 information to a CSV file (instanceID, default instance name, Existing tags) Define 4 mandatory tags in CSV header (Env, BizOwner, Technology, Project) Validate exported tags against the 4 mandatory new tags, if any of the new mandatory tags exists, then keep the tag and value, if any of the new mandatory tags do not exist, add the key and leave the value blank Get CSV file filled with mandatory tags input from Biz team (manual work) Open the updated CSV file, apply the mandatory tags based on the input value Create and trigger Lambda function with AWS config rules to enforce 4 mandatory tags whenever a new instance is launched List and export EC2 information to a CSV ...