Python on Zack's Blog

Python Microservice: K8S deployment

Wed, 22 May 2024 04:10:28 +0000

Now It is time to change from docker-compose to deploy into Kubernetes.

As this is not new to me to deploy microservice into K8S, also I already have a running Kubernetes cluster in hand, so here I will just create docker images for the 3 services: API gateway, user and order, then push them into the docker hub repository, then create Kubernetes manifest for deployment and service.

# Folder structure
/07-with-k8s
.
├── api_gateway.py
├── depolyment.yaml
├── Dockerfile_apigateway
├── Dockerfile_order
├── Dockerfile_user
├── order_service.py
└── user_service.py

# Build, tag and push the docker images
docker login

docker build -t zackz001/python-user:latest -f Dockerfile_user .
docker build -t zackz001/python-order:latest -f Dockerfile_order .
docker build -t zackz001/python-apigateway:latest -f Dockerfile_apigateway .
docker push zackz001/python-apigateway:latest
docker push zackz001/python-user:latest
docker push zackz001/python-order:latest

docker image ls

REPOSITORY                                      TAG       IMAGE ID       CREATED        SIZE
zackz001/python-apigateway                      latest    bc3db11f4be8   1 hours ago    138MB
zackz001/python-user                            latest    c93973bece33   1 hours ago    136MB
zackz001/python-order                           latest    e35d5de9254b   1 hours ago    136MB
prom/prometheus                                 latest    1bd2b9635267   8 days ago     271MB
grafana/grafana                                 latest    c42c21cd0ebc   3 weeks ago    453MB
consul                                          1.15.4    686495461132   4 months ago   155MB
docker.elastic.co/elasticsearch/elasticsearch   7.13.2    11a830014f7c   3 years ago    1.02GB
docker.elastic.co/logstash/logstash             7.13.2    8dc1af4dd662   3 years ago    965MB
docker.elastic.co/kibana/kibana                 7.13.2    6c4869a27be1   3 years ago    1.35GB

# k8s deployment Manifests

apiVersion: apps/v1
kind: Deployment
metadata:
 name: user-service
spec:
 replicas: 1
 selector:
 matchLabels:
 app: user-service
 template:
 metadata:
 labels:
 app: user-service
 spec:
 containers:
 - name: user-service
 image: zackz001/python-user:latest
 ports:
 - containerPort: 5001

---
apiVersion: v1
kind: Service
metadata:
 name: user-service
spec:
 selector:
 app: user-service
 ports:
 - protocol: TCP
 port: 5001
 targetPort: 5001

---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: order-service
spec:
 replicas: 1
 selector:
 matchLabels:
 app: order-service
 template:
 metadata:
 labels:
 app: order-service
 spec:
 containers:
 - name: order-service
 image: zackz001/python-order:latest
 ports:
 - containerPort: 5002

---
apiVersion: v1
kind: Service
metadata:
 name: order-service
spec:
 selector:
 app: order-service
 ports:
 - protocol: TCP
 port: 5002
 targetPort: 5002

---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: api-gateway
spec:
 replicas: 1
 selector:
 matchLabels:
 app: api-gateway
 template:
 metadata:
 labels:
 app: api-gateway
 spec:
 containers:
 - name: api-gateway
 image: zackz001/python-apigateway:latest
 ports:
 - containerPort: 5000

---
apiVersion: v1
kind: Service
metadata:
 name: api-gateway
spec:
 type: NodePort
 selector:
 app: api-gateway
 ports:
 - protocol: TCP
 port: 5000
 targetPort: 5000

Now Run kubectl apply -f to bring all deployments and services up and running. Should see all services in the Rancher console.

Python Microservice: Monitoring Stack

Tue, 21 May 2024 04:09:17 +0000

Both Prometheus and Grafana are compatible with microservice applications, integrating Prometheus with Flask is straightforward to provide performance and monitoring metrics.

Here I will update the docker-compose to add Prometheus and Grafana as services, then add Prometheus metrics in both order and user application code by importing PrometheusMetrics from the prometheus_flask_exporter module, which is used to expose Prometheus metrics for the Flask application. Then initialize Prometheus metrics with metrics = PrometheusMetrics(app). Use @metrics.counter('get_orders_count', 'Count of calls to the get_orders endpoint') to define a Prometheus counter metric that increments each time the get_orders endpoint is called.

Python Microservice: Logging with ELK

Mon, 20 May 2024 04:08:51 +0000

ELK (Elasticsearch, Logstash, Kibana) is a popular log management solution. We will use the ELK Stack to collect and analyze logs.

Here I need to extend the current configuration by adding services in the docker-compose file for Elasticsearch, Logstash, and Kibana, and configure the microservices to send logs to Logstash.

Also, I will need to configure the logging in both user and order Python application code to send logs to Logstash. By importing the built-in logging module and GelfUdpHandler from the pygelf module, to provide a flexible framework for emitting log messages from Python programs to send log messages in the GELF (Graylog Extended Log Format) to a remote Graylog server, which is typically part of the ELK stack.

Python Microservice: API Gateway & Consul

Sun, 19 May 2024 03:56:00 +0000

API Gateway acts as a single entry point for all clients and handles the request routing, composition, and protocol translation in a microservices architecture, here I will create an API Gateway using Python Flask and the requests library, to route both “user” and “order” services.

Here I will create an API Gateway to handle the 2 services (user and order). By importing the requests module, which allows us to send HTTP requests in Python. It’s used for making API calls to other services.

Python Microservice: Containerization

Sat, 18 May 2024 03:55:16 +0000

Flask stands out as one of Python’s most popular web frameworks. Designed for versatility and ease of use, Flask offers a robust starting point for crafting web apps.

By the following posts, I will use Flask to:

Create a series of Python web applications from simple app (Hello Zack)
Develop microservices applications (order and user), deploy using Docker compose
Create Python API gateway application
Integrate with Consul for service discovery and register
Enable logging with ELK, monitoring with Prometheus & Grafana
Lastly, I will create Kubernetes manifest for K8S deployment

I will skip ArgoCD and Github Action as I had done similar posts before.

Python: File Handling for AWS tagging

Fri, 17 May 2024 03:19:57 +0000

In the previous post I developed shell script + awscli to apply aws EC2 tags, since the last post we discovered Python Boto3 scripts for AWS resource automation and management, I think it is time to improve the EC2 tagging task with Python and boto3, together with file handling to achieve:

List and export EC2 information to a CSV file (instanceID, default instance name, Existing tags)
Define 4 mandatory tags in CSV header (Env, BizOwner, Technology, Project)
Validate exported tags against the 4 mandatory new tags, if any of the new mandatory tags exists, then keep the tag and value, if any of the new mandatory tags do not exist, add the key and leave the value blank
Get CSV file filled with mandatory tags input from Biz team (manual work)
Open the updated CSV file, apply the mandatory tags based on the input value
Create and trigger Lambda function with AWS config rules to enforce 4 mandatory tags whenever a new instance is launched

List and export EC2 information to a CSV

Python: Boto3 for AWS

Thu, 16 May 2024 03:19:36 +0000

Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python. It enables developers to build software that uses Amazon services like EC2, S3, RDS, etc.

I will build a portable Python3.9 + Boto3 Docker environment to test some AWS automation tasks.

Build and run a docker with Python3.9 + Boto3

As I do not want to install Python, Boto3, and AWScli on my local PC, creating a Docker image with all software ready as a portable env is the best way to start.