Python

Now It is time to change from docker-compose to deploy into Kubernetes. As this is not new to me to deploy microservice into K8S, also I already have a running Kubernetes cluster in hand, so here I will just create docker images for the 3 services: API gateway, user and order, then push them into the docker hub repository, then create Kubernetes manifest for deployment and service. # Folder structure /07-with-k8s . ├── api_gateway.py ├── depolyment.yaml ├── Dockerfile_apigateway ├── Dockerfile_order ├── Dockerfile_user ├── order_service.py └── user_service.py # Build, tag and push the docker images docker login docker build -t zackz001/python-user:latest -f Dockerfile_user . docker build -t zackz001/python-order:latest -f Dockerfile_order . docker build -t zackz001/python-apigateway:latest -f Dockerfile_apigateway . docker push zackz001/python-apigateway:latest docker push zackz001/python-user:latest docker push zackz001/python-order:latest docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE zackz001/python-apigateway latest bc3db11f4be8 1 hours ago 138MB zackz001/python-user latest c93973bece33 1 hours ago 136MB zackz001/python-order latest e35d5de9254b 1 hours ago 136MB prom/prometheus latest 1bd2b9635267 8 days ago 271MB grafana/grafana latest c42c21cd0ebc 3 weeks ago 453MB consul 1.15.4 686495461132 4 months ago 155MB docker.elastic.co/elasticsearch/elasticsearch 7.13.2 11a830014f7c 3 years ago 1.02GB docker.elastic.co/logstash/logstash 7.13.2 8dc1af4dd662 3 years ago 965MB docker.elastic.co/kibana/kibana 7.13.2 6c4869a27be1 3 years ago 1.35GB # k8s deployment Manifests apiVersion: apps/v1 kind: Deployment metadata: name: user-service spec: replicas: 1 selector: matchLabels: app: user-service template: metadata: labels: app: user-service spec: containers: - name: user-service image: zackz001/python-user:latest ports: - containerPort: 5001 --- apiVersion: v1 kind: Service metadata: name: user-service spec: selector: app: user-service ports: - protocol: TCP port: 5001 targetPort: 5001 --- apiVersion: apps/v1 kind: Deployment metadata: name: order-service spec: replicas: 1 selector: matchLabels: app: order-service template: metadata: labels: app: order-service spec: containers: - name: order-service image: zackz001/python-order:latest ports: - containerPort: 5002 --- apiVersion: v1 kind: Service metadata: name: order-service spec: selector: app: order-service ports: - protocol: TCP port: 5002 targetPort: 5002 --- apiVersion: apps/v1 kind: Deployment metadata: name: api-gateway spec: replicas: 1 selector: matchLabels: app: api-gateway template: metadata: labels: app: api-gateway spec: containers: - name: api-gateway image: zackz001/python-apigateway:latest ports: - containerPort: 5000 --- apiVersion: v1 kind: Service metadata: name: api-gateway spec: type: NodePort selector: app: api-gateway ports: - protocol: TCP port: 5000 targetPort: 5000 Now Run kubectl apply -f to bring all deployments and services up and running. Should see all services in the Rancher console. ...

Both Prometheus and Grafana are compatible with microservice applications, integrating Prometheus with Flask is straightforward to provide performance and monitoring metrics. Here I will update the docker-compose to add Prometheus and Grafana as services, then add Prometheus metrics in both order and user application code by importing PrometheusMetrics from the prometheus_flask_exporter module, which is used to expose Prometheus metrics for the Flask application. Then initialize Prometheus metrics with metrics = PrometheusMetrics(app). Use @metrics.counter('get_orders_count', 'Count of calls to the get_orders endpoint') to define a Prometheus counter metric that increments each time the get_orders endpoint is called. ...

ELK (Elasticsearch, Logstash, Kibana) is a popular log management solution. We will use the ELK Stack to collect and analyze logs. Here I need to extend the current configuration by adding services in the docker-compose file for Elasticsearch, Logstash, and Kibana, and configure the microservices to send logs to Logstash. Also, I will need to configure the logging in both user and order Python application code to send logs to Logstash. By importing the built-in logging module and GelfUdpHandler from the pygelf module, to provide a flexible framework for emitting log messages from Python programs to send log messages in the GELF (Graylog Extended Log Format) to a remote Graylog server, which is typically part of the ELK stack. ...

API Gateway acts as a single entry point for all clients and handles the request routing, composition, and protocol translation in a microservices architecture, here I will create an API Gateway using Python Flask and the requests library, to route both “user” and “order” services. Here I will create an API Gateway to handle the 2 services (user and order). By importing the requests module, which allows us to send HTTP requests in Python. It’s used for making API calls to other services. ...

Flask stands out as one of Python’s most popular web frameworks. Designed for versatility and ease of use, Flask offers a robust starting point for crafting web apps. By the following posts, I will use Flask to: Create a series of Python web applications from simple app (Hello Zack) Develop microservices applications (order and user), deploy using Docker compose Create Python API gateway application Integrate with Consul for service discovery and register Enable logging with ELK, monitoring with Prometheus & Grafana Lastly, I will create Kubernetes manifest for K8S deployment I will skip ArgoCD and Github Action as I had done similar posts before. ...

In the previous post I developed shell script + awscli to apply aws EC2 tags, since the last post we discovered Python Boto3 scripts for AWS resource automation and management, I think it is time to improve the EC2 tagging task with Python and boto3, together with file handling to achieve: List and export EC2 information to a CSV file (instanceID, default instance name, Existing tags) Define 4 mandatory tags in CSV header (Env, BizOwner, Technology, Project) Validate exported tags against the 4 mandatory new tags, if any of the new mandatory tags exists, then keep the tag and value, if any of the new mandatory tags do not exist, add the key and leave the value blank Get CSV file filled with mandatory tags input from Biz team (manual work) Open the updated CSV file, apply the mandatory tags based on the input value Create and trigger Lambda function with AWS config rules to enforce 4 mandatory tags whenever a new instance is launched List and export EC2 information to a CSV ...

Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python. It enables developers to build software that uses Amazon services like EC2, S3, RDS, etc. I will build a portable Python3.9 + Boto3 Docker environment to test some AWS automation tasks. Build and run a docker with Python3.9 + Boto3 As I do not want to install Python, Boto3, and AWScli on my local PC, creating a Docker image with all software ready as a portable env is the best way to start. ...