[{"content":"Last month, I did something that, looking back, was probably the most educational practice of my year. I tried using Claude and GPT to build a complete multi-source RAG system from scratch. This wasn\u0026rsquo;t a \u0026ldquo;toy demo\u0026rdquo; for social media; it was a real internal tool for my lab designed to handle messy document data, query rewriting, hybrid search, and deployment for my team.\nI set a strict rule for myself: Let the AI write almost all the code, while I handle the requirements and decision-making.\nThe Illusion of Productivity\nFor the first three hours, I felt like the world had changed. AI drafted the architecture—clean and beautiful. It wrote the backend code using FastAPI and LangChain, built the React frontend components one by one, and even configured the Dockerfile and Nginx reverse proxy settings perfectly.\nIt felt like having a partner who knows everything and never gets tired. I genuinely believed that at this efficiency, solo developers were about to take off. Then, the integration phase began, and I hit a brick wall for a solid week.\nThe Chasm Between \u0026ldquo;Complete\u0026rdquo; and \u0026ldquo;Production-Ready\u0026rdquo;\nAI is unparalleled at getting you from \u0026ldquo;0 to 0.6.\u0026rdquo; It nukes the boilerplate work. However, between a \u0026ldquo;seemingly complete\u0026rdquo; project and one that actually runs in production lies a chasm deeper than you imagine. That chasm contains:\nEdge cases and hidden coupling between modules. Tacit knowledge (e.g., business constraints that aren\u0026rsquo;t in the database schema but exist in human heads). Race conditions and a hundred ways users can break the system that AI cannot foresee. This final 20% of the project often takes three to five times longer than the initial 80%.\nThe Hardest Parts Aren\u0026rsquo;t About Writing Code\nComplexity in software doesn\u0026rsquo;t come from the volume of code, but from three specific areas:\nState Space Explosion: Systems have dozens of components with internal states. AI lacks the \u0026ldquo;gut feeling\u0026rdquo; of a seasoned engineer who knows which combinations are likely to fail. The Domino Effect of Decisions: Every technical choice (like choosing Redis over Memcached) impacts future consistency and scaling. AI can list pros and cons, but it cannot make a comprehensive trade-off based on team capability or future business direction. Fuzzy Requirements: Real projects are rarely well-defined. Navigating ambiguity and converging on real needs is a human capability that AI lacks. A Week of Hitting the Wall: My Diary\nDay 1: Smooth sailing. AI built the skeleton and basic API routes. Day 2: The pipeline failed. AI\u0026#39;s PDF code worked on clean papers but failed on lab scans and handwritten notes. Swapping libraries broke downstream modules. Day 3-4: AI implemented hybrid search but forgot to normalize scores. The BM25 scores drowned out the vector similarities. AI doesn\u0026#39;t think about how code behaves on real data distributions. Day 5-6: Deployment hell. CUDA version mismatches, OOM errors, and Nginx conflicts with SSE streaming. Every fix AI suggested was for an outdated library version. Day 7: Finally got it running, but I was exhausted. The Core Realization: Your Own Capability\nAI\u0026rsquo;s helpfulness is directly proportional to the skill of the user. It is an engine, and you are the driver.\nIf you are an expert: AI is a 3-5x multiplier. You spot bugs instantly and know exactly how to guide the engine. If you are a beginner: AI takes you to the \u0026ldquo;looks okay\u0026rdquo; stage, and then you get stuck. You enter a \u0026ldquo;death loop\u0026rdquo; of asking AI for fixes that create new, incomprehensible bugs. What AI Still Cannot Do\nCross-file logic consistency: AI lacks a persistent \u0026ldquo;mental model\u0026rdquo; of a project with dozens of files. Performance profiling: AI writes functional code, but it doesn\u0026rsquo;t naturally account for N+1 queries or memory deep copies. System-level debugging: When bugs involve the OS, kernel parameters, or third-party service interactions, AI is often useless. The Human Element: Negotiating requirements and understanding unstated concerns is 30% of a project\u0026rsquo;s success. The New Definition of \u0026ldquo;Doing a Project\u0026rdquo;\nIn the AI era, coding is being compressed. The new core skills are understanding the problem, evaluating AI-generated solutions, and integration. AI has removed the \u0026ldquo;manual labor\u0026rdquo; of programming, leaving behind only the \u0026ldquo;intellectual labor.\u0026rdquo;\nAI can help you finish a complex project, but it cannot do it well without you. It is your engine, but you must know where you are going. The stronger the AI becomes, the more you actually need to understand the underlying systems to stay in control.\n","permalink":"https://zackblog.work/posts/can-ai-actually-build-a-complex-project-a-brutal-rag-system-post-mortem/","summary":"\u003cp\u003eLast month, I did something that, looking back, was probably the most educational practice of my year. I tried using Claude and GPT to build a complete multi-source RAG system from scratch. This wasn\u0026rsquo;t a \u0026ldquo;toy demo\u0026rdquo; for social media; it was a real internal tool for my lab designed to handle messy document data, query rewriting, hybrid search, and deployment for my team.\u003c/p\u003e\n\u003cp\u003eI set a strict rule for myself: \u003cstrong\u003eLet the AI write almost all the code, while I handle the requirements and decision-making.\u003c/strong\u003e\u003c/p\u003e","title":"Can AI Actually Build a Complex Project? A Brutal RAG System Post-Mortem"},{"content":"I constantly rely on AWS Billing and Cost Management and Trusted Advisor to monitor costs and security. However, manual billing reviews are time-consuming, and raw Cost Explorer data doesn\u0026rsquo;t provide the full picture. What if an intelligent program could proactively call the Billing and Trusted Advisor APIs, retrieve the raw data, and pass it to an LLM for in-depth analysis of account health—then deliver actionable insights and AI-powered recommendations? That would save a significant amount of time and effort.\nThis post documents building a multi-account serverless billing and health analyzer powered by Amazon Bedrock — evolving from a single-account prototype to a production-ready solution that analyses multiple AWS accounts with a cost-optimised two-stage LLM pipeline.\nThe Evolution: From v1 to v2\nThe original v1 solution worked well for a single account, but faced challenges when scaling:\nLimited data coverage: Only 9 Trusted Advisor checks out of 537 available Single account only: No cross-account visibility Expensive for rich data: Sending all raw data to a powerful model is wasteful The v2 solution addresses these with a two-stage LLM pipeline and multi-account support:\n# v2 Architecture: Two-Stage LLM Pipeline EventBridge (Monthly 1st @ 9AM) │ ▼ Lambda (per account group) ├── PARALLEL DATA COLLECTION (up to 4 accounts simultaneously) │ ├── Cost Explorer (3 months + anomalies) │ ├── AWS Health API (maintenance, outages, deprecations) │ └── Trusted Advisor (ALL 537 checks per account) │ ├── STAGE 1: Claude Haiku 4.5 (cheap \u0026amp; fast) │ └── Filter raw data → extract only actionable items │ ├── STAGE 2: Claude Opus 4.5 (powerful) │ └── Generate executive report with prioritised recommendations │ └── SNS Email with [group-name] subject # Why Two Stages? ┌─────────────────────────────────────────────────────────────────┐ │ Raw Data (~200KB) → Haiku Filter → Filtered (~5KB) → Opus │ │ │ │ Cost: ~$0.80/run (single model) vs ~$0.40/run (two-stage) │ └─────────────────────────────────────────────────────────────────┘ Haiku handles the \u0026#34;grunt work\u0026#34; of filtering noise. Opus focuses on generating strategic insights from high-signal data. Key Improvements in v2\nMulti-Account Support: Analyse 2-10+ accounts per group with parallel data collection via cross-account IAM roles Full Trusted Advisor Coverage: All 537 checks instead of just 9 — no blind spots AWS Health API Integration: Scheduled maintenance, ongoing issues, EOL/deprecation notices Cost Anomaly Detection: Automatic detection of unusual spending patterns Account Groups: Deploy separate stacks per team/project with independent schedules Executive-Friendly Reports: ~2000 words, tables, per-account cost breakdown, prioritised actions # Account Groups Configuration (app.py) account_groups = { \u0026#34;platform-team\u0026#34;: { \u0026#34;accounts\u0026#34;: [ {\u0026#34;id\u0026#34;: \u0026#34;111111111111\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;dev\u0026#34;}, {\u0026#34;id\u0026#34;: \u0026#34;222222222222\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;staging\u0026#34;}, {\u0026#34;id\u0026#34;: \u0026#34;333333333333\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;prod\u0026#34;} ], \u0026#34;email\u0026#34;: \u0026#34;platform-leads@example.com\u0026#34; }, \u0026#34;data-team\u0026#34;: { \u0026#34;accounts\u0026#34;: [ {\u0026#34;id\u0026#34;: \u0026#34;444444444444\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;data-dev\u0026#34;}, {\u0026#34;id\u0026#34;: \u0026#34;555555555555\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;data-prod\u0026#34;} ], \u0026#34;email\u0026#34;: \u0026#34;data-leads@example.com\u0026#34; } } # Each group gets its own Lambda, SNS topic, and monthly schedule # Adding a new group = edit config + deploy IAM role to member accounts + cdk deploy Implementation Journey: Challenges \u0026amp; Solutions\nBedrock Marketplace Permissions New Bedrock models (Haiku 4.5, Opus 4.5) failed with AccessDeniedException on first Lambda invoke. Solution: Manually invoke each model once via CLI to enable account-wide access — Lambda role cannot be the \u0026ldquo;first invoker\u0026rdquo; for Marketplace models. Trusted Advisor Timeout with Multiple Accounts Sequential API calls for 537 checks × N accounts caused Lambda timeouts. Solution: Implemented parallel data collection using ThreadPoolExecutor with max 4 workers — 4 accounts now complete in ~160s (same as 1 account). Cost Drivers Combined Instead of Per-Account Stage 1 prompt was aggregating costs across all accounts. Solution: Updated prompt to explicitly preserve per-account top_5_services structure — now shows separate cost breakdown for each account. Model Selection: Sonnet vs Opus Tested both Claude Sonnet 4.5 and Opus 4.5 for Stage 2 analysis. Decision: Selected Opus for better formatting, effort estimates, and business impact statements — marginal cost increase (~$0.02/run) is negligible for monthly runs. Deployment with AWS CDK\n# 1. Deploy IAM role to each member account $ aws cloudformation deploy \\ --template-file member-role.yaml \\ --stack-name HealthAnalyzerRole \\ --capabilities CAPABILITY_NAMED_IAM \\ --parameter-overrides TrustedAccountId=\u0026lt;MAIN_ACCOUNT_ID\u0026gt; \\ --profile \u0026lt;MEMBER_ACCOUNT_PROFILE\u0026gt; # 2. Configure account groups in app.py # 3. Deploy all stacks $ cdk deploy --all --profile \u0026lt;YOUR_PROFILE\u0026gt; # 4. Confirm SNS email subscription # 5. Test manually $ aws lambda invoke \\ --function-name HealthAnalyzer-\u0026lt;GROUP_NAME\u0026gt;-HealthAnalyzer* \\ --invocation-type Event \\ --profile \u0026lt;YOUR_PROFILE\u0026gt; \\ /tmp/test.json # Check completion (~4 minutes later) $ aws logs filter-log-events \\ --log-group-name /aws/lambda/HealthAnalyzer-\u0026lt;GROUP\u0026gt;-* \\ --filter-pattern \u0026#34;\\\u0026#34;AWS Health Analyzer Complete\\\u0026#34;\u0026#34; \\ --profile \u0026lt;YOUR_PROFILE\u0026gt; Sample Report Output\nThe AI-powered report now covers multiple accounts with per-account breakdown:\nSubject: [platform-team] AWS Health Report - 2026-01-22 ## Executive Summary Brief overview of key findings across all 3 accounts... ## Cost Analysis | Account | Current Month | Previous Month | Change | |---------|---------------|----------------|--------| | dev | $1,234 | $1,100 | +12% | | staging | $567 | $590 | -4% | | prod | $2,100 | $2,050 | +2% | | TOTAL | $3,901 | $3,740 | +4% | ### Top 5 Cost Drivers - dev | Service | Cost | % of Total | |---------|------|------------| | EC2 | $500 | 40% | | S3 | $300 | 24% | ... ### Top 5 Cost Drivers - staging ... ## Platform Alerts | Priority | Service | Date | Action Required | |----------|---------|--------|--------------------------| | High | RDS | Feb 15 | MySQL 5.7 EOL migration | ... ## Security Findings | Severity | Count | Top Issue | |----------|-------|---------------------| | Critical | 2 | Public S3 buckets | | High | 5 | Open security groups| ... ## Top 5 Recommended Actions | Priority | Action | Owner | Timeline | |----------|-------------------------|----------|----------| | 1 | Fix public S3 buckets | Security | 24 hours | | 2 | Migrate RDS to MySQL 8 | DBA | 2 weeks | ... Cost Comparison\nMetric v1 (Single Account) v2 (Multi-Account) Accounts 1 2-10+ Trusted Advisor Checks 9 537 per account AWS Health API ❌ ✅ Cost Anomalies ❌ ✅ LLM Pipeline Single model Two-stage (Haiku→Opus) Cost per Run (2 accounts) ~$0.50 ~$0.41 Annual Cost (monthly runs) ~$6 ~$5-10 Lessons Learned\nTwo-stage LLM pipelines reduce costs while maintaining quality — use cheap models for filtering, expensive models for insights Parallel execution is essential for multi-account scaling — ThreadPoolExecutor makes 4 accounts as fast as 1 Account groups provide flexibility — different teams get independent reports and schedules Prompt engineering matters — explicit structure in prompts prevents unwanted aggregation Cross-account IAM with least-privilege enables secure multi-account access Conclusion\nBuilding a multi-account health analyzer with Amazon Bedrock demonstrates how modern cloud engineers can leverage serverless + AI to create intelligent automation that scales. The two-stage LLM pipeline (Haiku for filtering, Opus for analysis) provides enterprise-grade insights at ~$0.40 per run — less than a cup of coffee for comprehensive health reports across multiple AWS accounts.\nThe complete implementation — including CDK stack definitions, Lambda code, member account IAM role template, and comprehensive documentation — is available at my GitHub repository. Special thanks to Amazon Q for assistance throughout this journey.\n","permalink":"https://zackblog.work/posts/improve-billing-health-analyzer-with-bedrock-v2/","summary":"\u003cp\u003eI constantly rely on AWS Billing and Cost Management and Trusted Advisor to monitor costs and security. However, manual billing reviews are time-consuming, and raw Cost Explorer data doesn\u0026rsquo;t provide the full picture. What if an intelligent program could proactively call the Billing and Trusted Advisor APIs, retrieve the raw data, and pass it to an LLM for in-depth analysis of account health—then deliver actionable insights and AI-powered recommendations? That would save a significant amount of time and effort.\u003c/p\u003e","title":"Improve Billing \u0026 Health Analyzer with Bedrock v2"},{"content":"I constantly rely on AWS Billing and Cost Management and Trusted Advisor to monitor costs and security. However, manual billing reviews are time-consuming, and raw Cost Explorer data doesn’t provide the full picture. What if an intelligent program could proactively call the Billing and Trusted Advisor APIs, retrieve the raw data, and pass it to an LLM for in-depth analysis of account health—then deliver actionable insights and AI-powered recommendations? That would save a significant amount of time and effort.\nThis post documents building a serverless billing and health analyzer powered by Amazon Bedrock — the challenges faced, solutions implemented, and what this means for modern cloud operations.\nWhy Serverless + AI-Powered?\nServerless architecture combined with AI-powered analysis creates intelligent automation with zero infrastructure management, pay-per-execution pricing (~$0.50/month), automatic scaling and high availability, natural language insights beyond raw metrics, and proactive monitoring with minimal operational overhead.\n# Architecture ┌─────────────────────────────────────────────────────────────────────┐ │ EventBridge (Monthly) → Lambda Function │ │ ├─ Cost Explorer API (Billing Data) │ │ ├─ Trusted Advisor API (Recommendations) │ │ ├─ Bedrock API (AI Analysis) │ │ └─ SNS Topic (Email Delivery) │ └─────────────────────────────────────────────────────────────────────┘ # Tech Stack - Infrastructure as Code: AWS CDK (Python) - Compute: AWS Lambda (Python 3.12) - AI/ML: Amazon Bedrock (Claude Sonnet 4 via Inference Profile) - APIs: Cost Explorer, Trusted Advisor, Bedrock Runtime - Notifications: Amazon SNS - Scheduling: Amazon EventBridge # Folder Structure root@zack:/mnt/f/zack-gitops-project/mlops/aws-account-health-analyzer# tree . ├── README.md ├── SOLUTION.md ├── app.py ├── cdk.json ├── cdk.out │ ├── BillingAnalyzerStack.assets.json │ ├── BillingAnalyzerStack.template.json │ ├── cdk.out │ ├── manifest.json │ └── tree.json ├── lambda │ └── billing_analyzer.py ├── requirements-dev.txt ├── requirements.txt ├── serverless │ ├── __init__.py │ ├── __pycache__ │ │ ├── __init__.cpython-312.pyc │ │ └── billing_analyzer_stack.cpython-312.pyc │ ├── billing_analyzer_stack.py │ └── serverless_stack.py ├── source.bat └── tests ├── __init__.py └── unit ├── __init__.py └── test_serverless_stack.py # Monthly Operating Cost: ~$0.50 - Lambda: ~$0.10 (1 execution/month, 20s duration) - Bedrock: ~$0.30 (Claude Sonnet 4, ~2000 tokens) - SNS: ~$0.01 (1 email/month) Implementation Journey: Challenges \u0026amp; Solutions\nLambda Timeout and Bedrock Token Lambda timed out after 159 seconds when iterating through all 537 Trusted Advisor checks. Solution: Bedrock API tokens optimized to check only Trust Advisor action recommended checks (Security, Service limits and Fault tolerance) instead of 358 checks, reducing Bedrock API token and Lambda execution time to ~20 seconds. - Bedrock Inference Profile Access Needed to upgrade from Claude Haiku to Claude Sonnet 4 for better analysis quality. Solution: Used inference profile apac.anthropic.claude-sonnet-4-20250514-v1:0 with updated IAM permissions matching existing EKS workload patterns. Deployment with AWS CDK\n# Setup $ npm install -g aws-cdk $ cdk init app --language python $ source .venv/bin/activate $ pip install -r requirements.txt # Configure email in serverless/billing_analyzer_stack.py topic.add_subscription(subscriptions.EmailSubscription(\u0026#34;your-email@example.com\u0026#34;)) # Deploy $ cdk bootstrap aws://ACCOUNT-ID/ap-southeast-2 --profile YOUR-PROFILE $ cdk deploy --profile YOUR-PROFILE # Test manually $ aws lambda invoke \\ --function-name BillingAnalyzerStack-BillingAnalyzer* \\ --profile YOUR-PROFILE \\ /tmp/test.json # Cleanup $ cdk destroy --profile YOUR-PROFILE --force Sample Report Output\nThe AI-powered report combines billing data with Trusted Advisor recommendations:\nValidation \u0026amp; Accuracy\nTo ensure AI recommendations were accurate, I asked Amazon Q to validate Bedrock API generated findings against actual AWS resources and issues — achieving 100% accuracy. The AI-Powered solution correctly identified 1 Elastic IP, 1 EBS volume, 4 security group violations, and proper MFA configuration, with appropriate prioritization of the critical security risk.\nNext Step Enhencement\nAt this moment, Amazon Q is my best partner, as I copied the email findings and asked Amazon Q to cross-check and validate them, then requested Amazon Q to take appropriate actions to remove the EIP, EBS and SG ports.\nThis can also be achieved by integrating with AI tools and agents—for example, using a Bedrock model to generate scripts or AWS CLI commands directly from the email, or passing the task to a Lambda function to perform the mitigation. This enables a fully automated workflow. Additionally, I can later integrate Cost Anomaly Detection into the Lambda function to fetch and analyze cost data, allowing me to be notified immediately whenever an unusual cost spike pattern occurs.\nConclusion\nBuilding a serverless billing analyzer with Amazon Bedrock demonstrates how modern cloud engineers can leverage serverless + AI to create intelligent automation that provides real business value for ~$0.50/month.\nThe complete implementation — including CDK stack definitions, Lambda code, and comprehensive documentation — is available at my GitHub repository. Special thanks to Amazon Q for assistance throughout this journey.\n","permalink":"https://zackblog.work/posts/serverless-billing-health-analyzer-with-bedrock/","summary":"\u003cp\u003eI constantly rely on AWS Billing and Cost Management and Trusted Advisor to monitor costs and security. However, manual billing reviews are time-consuming, and raw Cost Explorer data doesn’t provide the full picture. What if an intelligent program could proactively call the Billing and Trusted Advisor APIs, retrieve the raw data, and pass it to an LLM for in-depth analysis of account health—then deliver actionable insights and AI-powered recommendations? That would save a significant amount of time and effort.\u003c/p\u003e","title":"Serverless Billing \u0026 Health Analyzer with Bedrock"},{"content":"After building a custom RAG system on Amazon EKS, my son asked why the application could only handle one question at a time, while ChatGPT allows users to continue conversations through follow-up questions with contextual memory. That made me realize there was an opportunity to leverage the power of the LangChain framework for better abstractions, conversational memory, and access to the broader LangChain ecosystem.\nThis post documents the journey of migrating to a LangChain-powered solution — the challenges faced, and the new capabilities and benefits gained.\nWhy LangChain?\nLangChain is an industry-standard framework offering standardized abstractions for RAG, chains, and memory; built-in conversational memory for context retention; advanced ConversationalRetrievalChain for multi-turn dialogue; seamless ecosystem integration with agents and tools; strong community support with active development and documentation; and continuous future-proofing through regular updates and new features.\n# Architecture Comparison ┌─────────────────────────────────────────────────────────────────────┐ │ ORIGINAL CUSTOM IMPLEMENTATION │ ├─────────────────────────────────────────────────────────────────────┤ │ User Query → Manual Vector Search → Context Assembly │ │ → Manual Prompt Construction → Bedrock API Call │ │ → Manual Response Parsing → Return Answer │ │ │ │ Pros: Full control, lightweight, no framework overhead │ │ Cons: Manual memory management, limited conversational context │ └─────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────┐ │ LANGCHAIN-POWERED IMPLEMENTATION │ ├─────────────────────────────────────────────────────────────────────┤ │ User Query → ConversationalRetrievalChain │ │ → Automatic Context Retrieval + Memory Integration │ │ → Optimized Prompt Templates → Custom DirectBedrockLLM│ │ → Structured Response with Sources → Return Answer │ │ │ │ Pros: Built-in memory, contextual conversation, extensible chains │ │ Cons: Framework dependency, requires custom LLM for profiles │ └─────────────────────────────────────────────────────────────────────┘ # Performance Comparison ┌──────────────────────┬─────────────────┬─────────────────────┐ │ Metric │ Original │ LangChain │ ├──────────────────────┼─────────────────┼─────────────────────┤ │ Query Time │ 6–8 seconds │ 6–8 seconds │ │ Embedding Dimensions │ 384 (V1) │ 1024 (V2) │ │ Chunk Size │ 1000 chars │ 1000 chars │ │ Chunk Overlap │ 200 chars │ 200 chars │ │ Source Attribution │ ✓ (manual) │ ✓ (automatic) │ │ Confidence Scores │ ✓ │ ✓ │ │ Conversation Memory │ ✗ (manual) │ ✓ (built-in) │ │ Context Awareness │ Limited │ Full multi-turn │ │ Code Maintainability │ Custom logic │ Framework patterns │ │ Extensibility │ Manual work │ Plugin ecosystem │ └──────────────────────┴─────────────────┴─────────────────────┘ Technology Stack Evolution\n# Core Technologies (Maintained) - **Orchestration:** Amazon EKS (Elastic Kubernetes Service) - **Vector Database:** Weaviate (deployed as a StatefulSet) - **LLM:** Amazon Bedrock (Claude 4.0 Sonnet via Inference Profile) - **Embeddings:** Amazon Titan Text Embeddings V2 (1024 dimensions) - **Frontend:** React SPA (served via Nginx) - **Storage:** AWS S3 for documents, AWS EFS for chat history - **Infrastructure as Code:** Terraform # New Additions (LangChain Integration) - **Framework:** LangChain (Python) - **Backend:** FastAPI + LangChain RAG Components - **Chains:** ConversationalRetrievalChain with Memory - **Custom Components:** DirectBedrockLLM for inference profile support - **Document Processing:** RecursiveCharacterTextSplitter - **Memory Management:** ConversationBufferMemory # The deployment architecture remains cloud-native and fully scalable ┌─────────────────────────────────────────────────────────────────────┐ │ EKS Cluster (langchain namespace) │ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────┐ │ │ │ Frontend │ │ Backend │ │ Weaviate Vector │ │ │ │ - Nginx │ │ - FastAPI │ │ - StatefulSet │ │ │ │ - React SPA │ │ - LangChain │ │ - 1024-dim │ │ │ │ - LoadBalancer│ │ - Custom LLM │ │ - Persistence │ │ │ │ │ │ - Chains │ │ - HNSW Index │ │ │ └─────────────────┘ │ - Memory │ └─────────────────────┘ │ │ └─────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘ │ ┌──────────────┼──────────────┐ ▼ ▼ ▼ ┌─────────────────┐ ┌─────────────┐ ┌─────────────┐ │ S3 │ │ Bedrock │ │ EFS │ │ Document Store │ │ Inference │ │ Chat History│ │ (LangChain) │ │ Profile │ │ (Memory) │ └─────────────────┘ └─────────────┘ └─────────────┘ The Migration Journey: Key Improvements\nDirect Bedrock Integration with LangChain Developed a custom DirectBedrockLLM class to directly invoke Bedrock inference profile models, achieving full compatibility with Anthropic Claude inference profiles and smoother LangChain integration. Also adopted Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0) for 1024-dimensional embeddings and improved performance.\nBackend Updated for LangChain Framework Alignment Updated backend code, imports, and implemented the _call method to align with the latest LangChain architecture, ensuring compatibility across document upload, chat, and memory functions.\nImproved Conversation Experience Added a NEW CHAT button to let users start new conversations while preserving previous threads in Shared Conversation History. The conversation interface was moved to the top of the QUERY tab for better accessibility.\nEnsure Vector Database with LangChain Updated the UI to display detailed vector statistics after document processing, including document counts, embedding dimensions, and chunk totals for improved observability after LangChain update.\nWorkflow: LangChain Edition\nThe LangChain-powered system maintains the same user experience while adding sophisticated conversation management under the hood:\n┌─────────────────────────────────────────────────────────────────────┐ │ DOCUMENT INGESTION FLOW │ ├─────────────────────────────────────────────────────────────────────┤ │ User Upload → FastAPI → S3 Storage │ │ → LangChain Document Creation (with metadata) │ │ → RecursiveCharacterTextSplitter (1000/200) │ │ → BedrockEmbeddings (Titan V2) │ │ → Weaviate.add_documents() → Index Complete │ └─────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────┐ │ CONVERSATIONAL QUERY FLOW │ ├─────────────────────────────────────────────────────────────────────┤ │ User Query → ConversationalRetrievalChain │ │ ├─ Memory: Load conversation history │ │ ├─ Retriever: Vector search in Weaviate │ │ ├─ Context: Combine history + retrieved docs │ │ └─ LLM: DirectBedrockLLM generates answer │ │ → Memory: Store Q\u0026amp;A pair │ │ → Response: Answer + Sources + Confidence + Time │ └─────────────────────────────────────────────────────────────────────┘ Final Look\nThe frontend now supports context-aware conversations and follow-up questions while still displaying source attribution, confidence scores, and execution time.\nConclusion\nMigrating from a custom RAG implementation to LangChain was a journey of discovery, problem-solving, and learning. It preserved all original features while adding sophisticated conversation management and future-proofing for advanced AI capabilities using LangChain as the framework.\nLangChain also unlocks powerful future capabilities such as multi-agent systems for specialized tasks, seamless tool integration with APIs and databases, advanced memory using entity tracking and knowledge graphs, hybrid search combining vector and keyword retrieval, streaming responses for real-time UX, multi-modal RAG for text and images, built-in evaluation frameworks for testing, and smart cost optimization through caching and token control.\nThe complete LangChain implementation — including all custom components, deployment manifests, and comprehensive documentation — is available at my GitHub repository. Special thanks to Amazon Q and Claude for their assistance in debugging, problem-solving, and providing architectural guidance throughout this migration journey.\n","permalink":"https://zackblog.work/posts/boost-eks-rag-with-langchain/","summary":"\u003cp\u003eAfter building a \u003ca href=\"/posts/bedrock-powered-rag-on-eks/\"\u003ecustom RAG system on Amazon EKS\u003c/a\u003e, my son asked why the application could only handle one question at a time, while ChatGPT allows users to continue conversations through follow-up questions with contextual memory. That made me realize there was an opportunity to leverage the power of the LangChain framework for better abstractions, conversational memory, and access to the broader LangChain ecosystem.\u003c/p\u003e\n\u003cp\u003eThis post documents the journey of migrating to a LangChain-powered solution — the challenges faced, and the new capabilities and benefits gained.\u003c/p\u003e","title":"Boost EKS RAG with LangChain"},{"content":"The Idea\nI\u0026rsquo;ve previously built Local RAG and ChatBot applications using OpenWebUI, and after exploring AWS Bedrock by creating a knowledge base with an S3 data source, I saw a bigger opportunity. My existing OpenWebUI chatbot was running on AWS Fargate, and I decided it was time to level up the architecture.\nThe goal was to migrate the solution to an Amazon EKS cluster and build a custom, self-developed RAG pipeline that leverages core AWS services. This would allow me to host two distinct AI applications on a single, scalable platform:\nA General Chatbot: An OpenWebUI instance for general-purpose conversation, powered directly by a serverless LLM from Amazon Bedrock. A Custom RAG System: A new pipeline where internal documents are embedded, stored in a vector database (Weaviate), and retrieved at query time to provide accurate, document-grounded answers. System Design\nMy objective was to deploy a robust Retrieval-Augmented Generation (RAG) system with a decoupled, microservices-based architecture on a scalable, cloud-managed infrastructure. The platform supports document uploads, automated text extraction, vectorization, and a conversational interface where users can ask questions about their own documents. To achieve scalability, reliability, and operational efficiency, I adopted a containerized architecture centered around AWS managed services, fully defined using Infrastructure as Code (IaC).\nTechnology Stack\n# Core Technologies - **Orchestration:** Amazon EKS (Elastic Kubernetes Service) - **Vector Database:** Weaviate (deployed as a StatefulSet) - **LLM:** Amazon Bedrock (Anthropic Claude Sonnet 4.0) - **Backend:** FastAPI (Python) - **Frontend:** React SPA (served via Nginx) - **Storage:** AWS S3 for documents, AWS EFS for chat history - **Container Registry:** AWS ECR - **Infrastructure as Code:** Terraform Vector Database: Weaviate on EKS\nWeaviate was chosen over AWS Kendra for its open-source flexibility, cost efficiency, and Kubernetes-native deployment. Running directly inside the EKS cluster minimizes latency between the backend and the database. Weaviate provides native vector search using the HNSW algorithm, supports multi-modal data (text, images, etc.), and exposes a powerful GraphQL API for flexible queries. Its modular architecture allows pluggable vectorizers such as text2vec-transformers, making it ideal for scalable, cloud-native applications.\n┌─────────────────┐ ┌──────────────────┐ ┌─────────────────────┐ │ Frontend │ │ EKS Cluster │ │ AWS Services │ │ (React SPA) │──▶│ Backend API │──▶│ Bedrock Claude │ │ LoadBalancer │ │ Weaviate DB │ │ S3 / EFS Storage │ └─────────────────┘ └──────────────────┘ └─────────────────────┘ ┌─────────────────────────────────────────────────────────────────────┐ │ EKS Cluster │ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────────┐ │ │ │ Frontend │ │ Backend │ │ Weaviate Vector │ │ │ │ - Nginx │ │ - FastAPI │ │ - Vector Store │ │ │ │ - Static SPA │ │ - Doc Service │ │ - Transformer │ │ │ │ - LoadBalancer│ │ - Chat API │ │ - Text2Vec │ │ │ └─────────────────┘ │ - Weaviate │ │ - HNSW Index │ │ │ │ Client │ │ - Persistence │ │ │ └─────────────────┘ └─────────────────────┘ │ └─────────────────────────────────────────────────────────────────────┘ │ ┌──────────────┼──────────────┐ ▼ ▼ ▼ ┌─────────────────┐ ┌─────────────┐ ┌─────────────┐ │ S3 │ │ Bedrock │ │ EFS │ │ Document Store │ │ Claude 4.0 │ │ Chat History│ │ File Uploads │ │ Sonnet Model│ │ Persistence │ └─────────────────┘ └─────────────┘ └─────────────┘ System Workflow\nThe RAG system delivers a seamless flow from document ingestion to conversational querying. The React single-page frontend is served through Nginx and communicates with the backend via proxied /api/* calls, eliminating CORS issues. External access and health checks are managed through an AWS Application Load Balancer (ALB). To maintain lightweight security for internal users, I implemented a session-based authentication system that displays an access modal when the app first loads. The access code is validated and stored in the browser’s sessionStorage, enforcing per-session protection without complex user management.\nThe backend, built with FastAPI, handles concurrent requests efficiently through async I/O. It connects to Weaviate for vector operations via the Python SDK, integrates with AWS Bedrock and S3 through Boto3, and supports document ingestion in formats such as PDF, DOCX, and XLSX. When users upload a document, the pipeline stores the file in S3, extracts text content, embeds it into vector representations, and indexes it in Weaviate. Progress tracking and UI updates are managed in real time through the FastAPI backend.\nUser Upload → FastAPI → S3 Storage → Text Extraction → Vector Embedding → Weaviate Index │ │ │ │ │ │ └─ Metadata ─────────┘ │ │ └─ Progress Tracking ──────────────────────────────────────┘ │ └─ Frontend Update ←─────────────────────────────────────────────────────┘ For question answering, user queries go through a vector search pipeline that retrieves the most relevant document chunks from Weaviate using HNSW similarity search. These context snippets are passed to Claude Sonnet on Amazon Bedrock, which generates the final answer enriched with referenced sources. Each query and response pair is persisted in AWS EFS to maintain per-session chat history.\nUser Query → Vector Search → Context Retrieval → LLM Generation → Response + Sources │ │ │ │ │ │ │ └─ Top-K Documents ──┘ │ │ └─ Similarity Search (HNSW) ────────────────────────┘ └─ Chat History Update ←─────────────────────────────────────────────┘ Enable Tracing, Monitoring, and Dashboards\nEven with only two namespaces deployed for the Chatbot and RAG systems, it’s essential to have foundational Kubernetes observability stacks in place. Proper tracing, monitoring, and dashboard visualization ensure visibility into system performance, simplify debugging, and support proactive maintenance as the deployment scales.\nKiali Dashboard:\nGrafana Dashboard:\nJaeger Tracing:\nChallenges and Debugging\nWith great support using ClaudeCode and AmazonQ, I was able to fix the following issues:\nFixing inference parameter mismatches for the selected Bedrock model. Resolving a client initiation failure in the backend\u0026rsquo;s connection to Weaviate. Solving a port conflict (8080) between the two containers in the Weaviate StatefulSet. Implementing frontend optimizations to fix document DELETE button pop-up. Designing functions to properly manage, retrieve, and display document lists, indexed and chunk status, and chat histories. Enabling secure document downloads from the web interface. Tuning Weaviate\u0026rsquo;s resource request and limits to resolve OOM (Out of Memory) kills. Optimized resource request to have a single t3.large (2C8G) ec2 can handle: Chatbot, RAG System, Add-on (CoreDNS, EFS driver, AWS Loadbalancer Controller), Istio, Monitoring Stack (Prometheus, Grafana, Kiali and Jaeger). Conclusion\nI think this is a well-structured, powerful, scalable, and cost-effective AI application for intelligent document processing that can grow with organizational needs. Using EKS, with its native support for scalability and namespaces, this solution can be easily replicated to serve different teams, ensuring complete security and data isolation for documents and chat histories.\nThe full application, Terraform and EKS manifests are now available at my GitHub repo.\n","permalink":"https://zackblog.work/posts/bedrock-powered-rag-on-eks/","summary":"\u003cp\u003e\u003cstrong\u003eThe Idea\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eI\u0026rsquo;ve previously built \u003ca href=\"/posts/mlops-build-a-knowledge-base-with-deepseek-r1/\"\u003eLocal RAG\u003c/a\u003e  and \u003ca href=\"/posts/aws-bedrock-with-open-webui/\"\u003eChatBot applications\u003c/a\u003e  using OpenWebUI, and after exploring AWS Bedrock by creating a knowledge base with an S3 data source, I saw a bigger opportunity. My existing OpenWebUI chatbot was running on AWS Fargate, and I decided it was time to level up the architecture.\u003c/p\u003e\n\u003cp\u003eThe goal was to migrate the solution to an Amazon EKS cluster and build a custom, self-developed RAG pipeline that leverages core AWS services. This would allow me to host two distinct AI applications on a single, scalable platform:\u003c/p\u003e","title":"Bedrock-Powered RAG on EKS"},{"content":"For those experienced with Kubernetes, managing traffic between microservices often brings tools like Istio to mind. I had done some posts before on Istio for Distributed Tracing and Traffic Routing.\nThis PoC explores a different approach: using AWS VPC Lattice. While the goal—in this case, A/B testing between different service versions—is the same, the implementation differs. Instead of Istio\u0026rsquo;s VirtualService and DestinationRule resources, this setup leverages the vendor-neutral Kubernetes Gateway API. This allows for a more standardized way of defining routing within the cluster, while VPC Lattice provides the power to extend this networking seamlessly across different VPCs.\nThis post is about integrating AWS EKS cluster with VPC Lattice to enable advanced traffic management and A/B testing for microservices. The setup leverages the standard Kubernetes Gateway API to define routing rules, which are implemented by the AWS Gateway Controller to provision and configure VPC Lattice resources automatically.\nThe Architecture\nThe resulting architecture allows internet traffic to be routed through a Network Load Balancer to a UI service. The UI then communicates with backend services through VPC Lattice, which manages traffic splitting for A/B testing between two different versions of the checkout service.\nInternet Traffic ↓ Network Load Balancer (ui-nlb) ↓ UI Service (updated to use VPC Lattice) ↓ VPC Lattice Gateway (managed by AWS) ↓ HTTPRoute (splits traffic 75% / 25%) ├───────────↓ │ ↓ Checkout v1 (25%) Checkout v2 (75%) Phase 1: Infrastructure Setup (EKS Security Group)\nGoal: Allow the EKS cluster to securely receive traffic from the VPC Lattice service network.\nProcess: The cluster\u0026rsquo;s primary security group was modified to allow ingress traffic from the AWS-managed prefix lists for VPC Lattice. This opens a secure communication channel without exposing ports to the public internet.\nKey Commands:\n# 1. Get the EKS cluster\u0026#39;s security group ID CLUSTER_SG=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --output json| jq -r \u0026#39;.cluster.resourcesVpcConfig.clusterSecurityGroupId\u0026#39;) # 2. Find the AWS-managed prefix list for VPC Lattice PREFIX_LIST_ID=$(aws ec2 describe-managed-prefix-lists --query \u0026#34;PrefixLists[?PrefixListName==\u0026#39;com.amazonaws.$AWS_REGION.vpc-lattice\u0026#39;].PrefixListId\u0026#34; | jq -r \u0026#39;.[]\u0026#39;) # 3. Authorize ingress traffic from the prefix list to the cluster\u0026#39;s security group aws ec2 authorize-security-group-ingress --group-id $CLUSTER_SG --ip-permissions \u0026#34;PrefixListIds=[{PrefixListId=${PREFIX_LIST_ID}}],IpProtocol=-1\u0026#34; Outcome: ✅ Network connectivity established between VPC Lattice and the EKS cluster.\nPhase 2: Gateway API Installation\nGoal: Install the Kubernetes components required to understand and manage Gateway API resources.\nProcess: This involved two parts: first, applying the standard Gateway API Custom Resource Definitions (CRDs), which provide the Gateway and HTTPRoute resource types. Second, installing the AWS Gateway Controller using Helm, which acts as the \u0026ldquo;translator\u0026rdquo; between the Kubernetes API and VPC Lattice.\nKey Commands:\n# 1. Install Gateway API CRDs (the \u0026#34;language\u0026#34;) kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml # 2. Install the AWS Gateway Controller (the \u0026#34;translator\u0026#34;) helm install gateway-api-controller oci://public.ecr.aws/aws-application-networking-k8s/aws-gateway-controller-chart --version=v1.0.5 --namespace gateway-api-controller Outcome: ✅ The AWS Gateway Controller is running and ready to provision VPC Lattice resources based on Gateway API objects.\nPhase 3: Gateway Configuration\nGoal: Define and create the VPC Lattice Service Network.\nProcess: A GatewayClass resource was applied to define \u0026ldquo;VPC Lattice\u0026rdquo; as a gateway provider. Then, a Gateway resource was created, which prompted the controller to provision the actual VPC Lattice service network and associate it with the cluster.\nKey Commands:\n# 1. Define VPC Lattice as a gateway provider kubectl apply -f gatewayclass.yaml # 2. Create the Gateway, which provisions the VPC Lattice Service Network cat eks-workshop-gw.yaml | envsubst | kubectl apply -f - # 3. Wait for the gateway to be programmed and ready kubectl wait --for=condition=Programmed gateway/${EKS_CLUSTER_NAME} -n checkout Outcome: ✅ A VPC Lattice service network is created and associated with the EKS cluster.\nPhase 4: Application Deployment for A/B Testing\nGoal: Deploy two distinct versions of the checkout application to serve as targets for traffic splitting.\nProcess: Using Kustomize, two versions of the checkout service were deployed into separate namespaces (checkout and checkoutv2) to simulate a real-world A/B testing scenario.\nKey Commands:\n# Deploy the two application versions kubectl apply -k ~/environment/eks-workshop/modules/networking/vpc-lattice/abtesting/ # Verify the rollout status of the v2 deployment kubectl rollout status deployment/checkout -n checkoutv2 Outcome: ✅ Two versions of the checkout service are running independently.\nPhase 5: Traffic Routing \u0026amp; Health Checks\nGoal: Define the traffic splitting rules and configure health checks for the application targets.\nProcess: An HTTPRoute resource was created to define the core routing logic, splitting traffic 25% to the original checkout service and 75% to the checkoutv2 service. A TargetGroupPolicy was also applied to configure detailed health checks for the VPC Lattice target groups.\nKey Resources:\nHTTPRoute for Traffic Splitting:\napiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: checkoutroute namespace: checkout spec: parentRefs: - name: eks-workshop sectionName: http rules: - backendRefs: - name: checkout namespace: checkout port: 80 weight: 25 - name: checkout namespace: checkoutv2 port: 80 weight: 75 matches: - path: type: PathPrefix value: / TargetGroupPolicy for Health Checks:\napiVersion: application-networking.k8s.aws/v1alpha1 kind: TargetGroupPolicy metadata: name: checkout-policy namespace: checkout spec: targetRef: kind: Service name: checkout healthCheck: enabled: true path: \u0026#34;/health\u0026#34; Outcome: ✅ Traffic is actively being split between the two service versions with proper health checks in place.\nPhase 6: UI Integration\nGoal: Reconfigure the frontend UI application to send traffic to the new VPC Lattice endpoint.\nProcess: The UI deployment was updated to use the DNS name assigned by VPC Lattice to the HTTPRoute. This directs all backend calls from the UI through the managed VPC Lattice gateway instead of using internal Kubernetes service DNS.\nKey Commands:\n# 1. Get the VPC Lattice assigned DNS name export CHECKOUT_ROUTE_DNS=\u0026#34;http://$(kubectl get httproute checkoutroute -n checkout -o json | jq -r \u0026#39;.metadata.annotations[\u0026#34;application-networking.k8s.aws/lattice-assigned-domain-name\u0026#34;]\u0026#39;)\u0026#34; # 2. Update and redeploy the UI to use the new DNS name kubectl kustomize ~/environment/eks-workshop/modules/networking/vpc-lattice/ui/ | envsubst | kubectl apply -f - kubectl rollout restart deployment/ui -n ui Outcome: ✅ The end-to-end traffic flow is complete and fully functional.\nIn browser and try to checkout multiple times (with different items in the cart), we notice that the checkout now uses the \u0026ldquo;Lattice checkout\u0026rdquo; pods about 75% of the time.\nVPC Lattice vs. Istio Service Mesh\nSimilarities\nBoth handle traffic routing (blue/green, canary), service discovery, load balancing, and enforce auth policies. Differences\nArchitecture: Istio uses sidecars; VPC Lattice is AWS-managed (no sidecars). Ops: Istio needs you to manage control plane \u0026amp; certs; VPC Lattice is fully managed. Performance: Sidecars can add latency; Lattice uses AWS networking (lower latency). Scope: Istio works anywhere; Lattice is AWS-only (EKS, ECS, EC2, Lambda). Complexity: Istio is harder to learn; Lattice is simpler with AWS tools. When to Use\nVPC Lattice: Best if you’re all-in on AWS, want simplicity, no sidecars, and cross-service AWS connectivity.\nIstio: Best for multi-cloud/on-prem, advanced mesh features, vendor-neutral, and deep traffic control.\nVPC Lattice essentially provides a \u0026ldquo;service mesh as a service\u0026rdquo; for AWS workloads, reducing the operational complexity that comes with self-managed solutions like Istio. The full source code and guide can be found at GitHub repo.\n","permalink":"https://zackblog.work/posts/eks-vpc-lattice-integration-for-a-b-testing/","summary":"\u003cp\u003eFor those experienced with Kubernetes, managing traffic between microservices often brings tools like Istio to mind. I had done some posts before on Istio for \u003ca href=\"/posts/istio-traffic-routing/\"\u003eDistributed Tracing and Traffic Routing\u003c/a\u003e.\u003c/p\u003e\n\u003cp\u003eThis PoC explores a different approach: using \u003cstrong\u003eAWS VPC Lattice\u003c/strong\u003e. While the goal—in this case, A/B testing between different service versions—is the same, the implementation differs. Instead of Istio\u0026rsquo;s \u003ccode\u003eVirtualService\u003c/code\u003e and \u003ccode\u003eDestinationRule\u003c/code\u003e resources, this setup leverages the vendor-neutral Kubernetes Gateway API. This allows for a more standardized way of defining routing within the cluster, while VPC Lattice provides the power to extend this networking seamlessly across different VPCs.\u003c/p\u003e","title":"EKS \u0026 VPC Lattice Integration for A/B Testing"},{"content":"In this post, I’ll demonstrate how to install and use the Kubernetes MCP server GitHub repo with Claude Code , and then show how I migrated the previous AWS serverless OpenWebUI + Bedrock solution to run locally on Minikube. Finally, we’ll explore how to use this Kubernetes MCP server to inspect and troubleshoot the deployment.\nInstallation \u0026amp; Usage\nBefore getting started, make sure you have the following prerequisites installed:\nkubectl installed and available in your PATH A valid kubeconfig file with contexts configured Access to a Kubernetes cluster (e.g., Minikube, Rancher Desktop, GKE) Helm v3 installed and in your PATH (optional if you don’t plan to use Helm) By default, the server loads kubeconfig from ~/.kube/config.\nAdding the MCP Server to Claude Code\nAdd the Kubernetes MCP server with the built-in command:\nclaude mcp add kubernetes -- npx mcp-server-kubernetes This will automatically configure the MCP server in the Claude Code settings.\nWe can now test basic operations such as listing and creating resources using our connected Kubernetes cluster.\nMigrating AWS Serverless OpenWebUI + Bedrock to Minikube\nNext, I converted my AWS-based deployment to run locally on Minikube. After migration, the Kubernetes MCP server provided an easy way to monitor and troubleshoot the setup.\nroot@zack:/mnt/f/zack-gitops-project/mlops/terraform-fargate-bedrock-openwebui/k8s-manifests# tree . ├── README.md ├── aws-credentials-secret.yaml ├── deploy.sh ├── openwebui-deployment.yaml ├── openwebui-service.yaml ├── persistent-volume-claim.yaml └── validate-deployment.sh 1 directory, 7 files root@zack:/mnt/f/zack-gitops-project/mlops/terraform-fargate-bedrock-openwebui/k8s-manifests# cat openwebui-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: openwebui-deployment namespace: default spec: replicas: 1 selector: matchLabels: app: openwebui template: metadata: labels: app: openwebui spec: containers: - name: openwebui image: ghcr.io/open-webui/open-webui:main ports: - containerPort: 8080 volumeMounts: - name: data-volume mountPath: /app/backend/data env: - name: DATA_DIR value: \u0026#34;/app/backend/data\u0026#34; - name: WEBUI_SECRET_KEY_FILE value: \u0026#34;/app/backend/data/.webui_secret_key\u0026#34; resources: requests: memory: \u0026#34;1Gi\u0026#34; cpu: \u0026#34;500m\u0026#34; limits: memory: \u0026#34;2Gi\u0026#34; cpu: \u0026#34;1000m\u0026#34; - name: bedrock-gateway image: zackz001/openwebui-bedrock-gateway:v1 # Update with my DockerHub image ports: - containerPort: 80 envFrom: - secretRef: name: aws-credentials resources: requests: memory: \u0026#34;512Mi\u0026#34; cpu: \u0026#34;250m\u0026#34; limits: memory: \u0026#34;1Gi\u0026#34; cpu: \u0026#34;500m\u0026#34; volumes: - name: data-volume persistentVolumeClaim: claimName: openwebui-data Now the web console is running locally—migration achieved 🎉. A local deployed openwebui using AWS Bedrock apac.anthropic.claude-sonnet-4-20250514-v1:0 as model, connecting via AWS Bedrock Gateway via AWS credentials from K8S secret, persistent volume to store chat history using k8s pv, expose service to local potal access\nKey Migration Mappings\nAWS IAM Role → Kubernetes Secret (for credentials) AWS EFS → PersistentVolumeClaim (for data persistence) ECS Task → Kubernetes Pod (for container orchestration) ALB/ECS Service → Kubernetes Service (for network access) Architecture Comparison\nSide-by-side Terraform vs Kubernetes diagrams Detailed component mapping table Preserved networking behavior (localhost communication) Migration Benefits\nDevelopment advantages: local, faster, offline-friendly Architecture preservation: same containers, ports, and communication Production readiness: clear cloud migration path with Kubernetes-native tooling The full manifests are now available at my GitHub repo.\n","permalink":"https://zackblog.work/posts/claude-code-with-kubernetes-mcp-server/","summary":"\u003cp\u003eIn this post, I’ll demonstrate how to install and use the Kubernetes MCP server \u003ca href=\"https://github.com/Flux159/mcp-server-kubernetes\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eGitHub repo\u003c/a\u003e with Claude Code , and then show how I migrated the previous \u003ca href=\"/posts/aws-bedrock-with-open-webui/\"\u003eAWS serverless OpenWebUI + Bedrock solution\u003c/a\u003e to run locally on Minikube. Finally, we’ll explore how to use this Kubernetes MCP server to inspect and troubleshoot the deployment.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eInstallation \u0026amp; Usage\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eBefore getting started, make sure you have the following prerequisites installed:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003ekubectl\u003c/code\u003e installed and available in your \u003ccode\u003ePATH\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003eA valid \u003ccode\u003ekubeconfig\u003c/code\u003e file with contexts configured\u003c/li\u003e\n\u003cli\u003eAccess to a Kubernetes cluster (e.g., Minikube, Rancher Desktop, GKE)\u003c/li\u003e\n\u003cli\u003eHelm v3 installed and in your \u003ccode\u003ePATH\u003c/code\u003e (optional if you don’t plan to use Helm)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eBy default, the server loads \u003ccode\u003ekubeconfig\u003c/code\u003e from \u003ccode\u003e~/.kube/config\u003c/code\u003e.\u003c/p\u003e","title":"Claude Code with Kubernetes MCP Server"},{"content":"Recently Data team reached out, trying to build an EC2 running Open WebUI, connecting to AWS Bedrock, to offer team member AI application. This guide provides step-by-step instructions for deploying and integrating with AWS Bedrock models. .\nThe architecture consists of two Docker containers operating on a single EC2 instance: Open WebUI serves as the user-facing chat interface, while a Bedrock Access Gateway acts as middleware. This gateway securely forwards requests from Open WebUI to the AWS Bedrock API using the EC2 instance\u0026rsquo;s attached IAM role. The containers communicate over a private Docker network, isolating traffic between them.\nPrerequisites\nBefore proceeding, ensure the following requirements are met:\nEnsure access granted for desired model via AWS console Bedrock model catelogy Launch an EC2 (e.g., Ubuntu 24.04 LTS) and create an IAM role with permissions to access AWS Bedrock (e.g., bedrock:\\*), and attach to the instance. Ensure Security group allows inbound TCP traffic on port 80 for Open WebUI and port 8000 for bedrock-gateway. Also Docker is installed. Step 1: Get Bedrock model access granted\nGo AWS console Bedrock, choose the model and request for access.\nStep 2: Launch EC2 and Deploy the OpenWebUI Container\nRun the OpenWebUI container, attach it to the previously created network, and map the instance\u0026rsquo;s port 80 to the container\u0026rsquo;s port 8080 for public access.\n# Create a custom network for container communication docker network create bedrock-net # Run the OpenWebUI container docker run -d \\ --name openwebui \\ --network bedrock-net \\ --restart unless-stopped \\ -p 80:8080 \\ ghcr.io/open-webui/open-webui:main Step 3: Build Bedrock Access Gateway Docker Image\nClone the official GitHub repo, modify the Dockerfile\\_ecs to add the AWS CLI package, then Build the custom Docker image and tag it as bedrock-gateway:\ngit clone https://github.com/aws-samples/bedrock-access-gateway.git cd bedrock-access-gateway/src vim Dockerfile_ecs # add bellow RUN pip install --no-cache-dir awscli docker build -f Dockerfile_ecs -t bedrock-gateway . Step 4: Launch the Bedrock Gateway Securely\nTo avoid hardcoding AWS credentials, dynamically fetch temporary credentials from the EC2 instance metadata service and pass them to the Docker container as environment variables. This script fetches the credentials and starts the container.\n# Get a session token from the instance metadata service TOKEN=$(curl -X PUT \u0026#34;http://169.254.169.254/latest/api/token\u0026#34; -H \u0026#34;X-aws-ec2-metadata-token-ttl-seconds: 21600\u0026#34;) # Get the IAM role name attached to the instance ROLE_NAME=$(curl -H \u0026#34;X-aws-ec2-metadata-token: $TOKEN\u0026#34; http://169.254.169.254/latest/meta-data/iam/security-credentials/) # Fetch the full credential set for the role (requires jq) CREDS=$(curl -H \u0026#34;X-aws-ec2-metadata-token: $TOKEN\u0026#34; http://169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE_NAME) export AWS_ACCESS_KEY_ID=$(echo $CREDS | jq -r \u0026#39;.AccessKeyId\u0026#39;) export AWS_SECRET_ACCESS_KEY=$(echo $CREDS | jq -r \u0026#39;.SecretAccessKey\u0026#39;) export AWS_SESSION_TOKEN=$(echo $CREDS | jq -r \u0026#39;.Token\u0026#39;) # Run the gateway container, injecting the credentials and region docker run \\ -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \\ -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \\ -e AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN \\ -e AWS_REGION=ap-southeast-2 \\ -d --name bedrock-gateway \\ --network bedrock-net \\ -p 8000:80 \\ bedrock-gateway Step 5: Configure OpenWebUI connection to the Gateway and set up the Bedrock Model\nThe final step is to configure OpenWebUI to use the gateway, this can be followed by the offical Doc.\nNavigate to the OpenWebUI interface at your EC2 instance\u0026rsquo;s public IP address and sign up for a local account.\nGo to Settings → Connections.\nEnter the following API details:\nAPI URL: http://bedrock-gateway:80/api/v1. This address resolves correctly because both containers are on the same Docker network. API Key: bedrock. This is a placeholder value, as the gateway relies on the IAM role for authentication. Add the desired Bedrock model IDs, such as anthropic.claude-3-sonnet-20240229-v1:0. Now we can now start a new chat and select a Bedrock model from the dropdown list. After addressing a few issues, now we have a fully functional and secure chat interface powered by AWS Bedrock, running in your own environment with no hardcoded credentials.\nEvolution Summary: From Manual to Automated\nOur initial approach involved manually deploying two Docker containers on a single EC2 instance: Open WebUI as the chat interface and a Bedrock Access Gateway as middleware. While functional, this manual setup had limitations:\nManual EC2 provisioning and Docker container management No persistent storage - data lost on container restarts Single point of failure with one EC2 instance Manual credential management via instance metadata No infrastructure as code or version control The manual approach served as our proof of concept, validating the architecture and integration patterns. However, for production use, we needed something more robust and maintainable.\nThe Serverless Transformation\nOur new architecture leverages modern AWS services to address every limitation of the manual approach:\nInternet → ALB → ECS Service (Fargate) ├── Task 1: OpenWebUI (ECR) + Bedrock Gateway (ECR) └── Task 2: OpenWebUI (ECR) + Bedrock Gateway (ECR) ↓ EFS Filesystem (Persistent Storage) Key Improvements Achieved\n1. Infrastructure as Code with Terraform\nEverything is now defined in Terraform, enabling version control, reproducible deployments, and easy environment management:\n# Deploy entire infrastructure with one command terraform init terraform apply # Creates: VPC, subnets, ALB, ECS cluster, ECR repositories, # EFS filesystem, security groups, IAM roles, and more 2. Serverless with ECS Fargate\nNo more EC2 instances to manage. Fargate handles all the underlying infrastructure:\nAutomatic scaling based on demand Built-in high availability across multiple AZs No server patching or maintenance Pay only for actual container runtime 3. Container Images in ECR\nBoth OpenWebUI and Bedrock Gateway images are now stored in Amazon ECR with automated build and push:\n# Automated image build and deployment ./build-and-push.sh # Builds custom images and pushes to ECR # Updates ECS service with new images automatically 4. Persistent Storage with EFS\nThe biggest improvement: all user data, chat history, and configurations now persist across deployments:\nChat conversations survive container restarts User uploads and settings maintained Shared storage across multiple tasks for session consistency Automatic backups and cross-AZ replication 5. High Availability and Load Balancing\nApplication Load Balancer distributes traffic across multiple Fargate tasks:\nZero-downtime deployments with rolling updates Health checks ensure only healthy containers receive traffic Automatic failover if a task becomes unhealthy SSL termination at the load balancer Step 1: Infrastructure Deployment\nClone the GitHub repo and deploy the complete infrastructure:\ncd terraform-fargate-bedrock-openwebui # Initialize and deploy infrastructure terraform init terraform apply # Creates: ECR repositories, ECS cluster, VPC, ALB, EFS, security groups Step 2: Build and Push Container Images\nAutomated script handles the entire image build and deployment process:\n# Build and push both images to ECR ./build-and-push.sh # This script: # 1. Authenticates with ECR # 2. Pulls/builds OpenWebUI image # 3. Builds custom Bedrock Gateway with AWS CLI # 4. Pushes both images to ECR repositories # 5. Triggers ECS service update Step 3: Service Deployment\nECS automatically deploys the containers with the new images:\n# Deploy ECS service with ECR images terraform apply # ECS handles: # - Task definition updates # - Rolling deployment # - Health checks # - Load balancer registration Configuration and Usage\nOpenWebUI Configuration\nWith EFS persistence, configuration only needs to be done once:\nAPI URL: http://localhost:11434/api/v1 (internal container communication) API Key: Any value (e.g., bedrock-key) Models: Auto-populated from Bedrock Gateway or manually add anthropic.claude-3-5-haiku-20241022-v1:0 All settings persist across deployments thanks to EFS storage mounted at /app/backend/data.\nOperational Benefits\nMonitoring and Logging\nCloudWatch logs for both containers with 7-day retention ECS service metrics and health monitoring ALB access logs and target group health checks Security Improvements\nNo hardcoded credentials - IAM roles for service authentication VPC isolation with security groups controlling access ECR image vulnerability scanning Corporate IP restrictions via security groups Cost Optimization\nFargate: ~$44/month for 2 tasks (vs. EC2 instance costs) EFS: ~$3-5/month for typical usage ECR: ~$0.07/month for image storage No idle EC2 costs - pay only for actual usage Troubleshooting and Maintenance\nFuture Enhancements\nThe Terraform-based architecture provides a foundation for additional improvements:\nAuto-scaling based on CPU/memory metrics Multi-environment deployments (dev/staging/prod) CI/CD pipeline integration with GitHub Actions Custom domain with Route 53 and ACM certificates Enhanced monitoring with CloudWatch dashboards This evolution from manual EC2 deployment to automated serverless architecture demonstrates how modern AWS services can transform a proof of concept into a production-ready solution. The combination of Terraform, ECS Fargate, ECR, and EFS provides a robust, scalable, and maintainable platform for AI applications.\n","permalink":"https://zackblog.work/posts/aws-bedrock-with-open-webui/","summary":"\u003cp\u003eRecently Data team reached out, trying to build an EC2 running Open WebUI, connecting to AWS Bedrock, to offer team member AI application. This guide provides step-by-step instructions for deploying and integrating with AWS Bedrock models. .\u003c/p\u003e\n\u003cp\u003eThe architecture consists of two Docker containers operating on a single EC2 instance: Open WebUI serves as the user-facing chat interface, while a Bedrock Access Gateway acts as middleware. This gateway securely forwards requests from Open WebUI to the AWS Bedrock API using the EC2 instance\u0026rsquo;s attached IAM role. The containers communicate over a private Docker network, isolating traffic between them.\u003c/p\u003e","title":"AWS Bedrock with Open WebUI"},{"content":"I recently found an interesting tool run-gemini-cli, a GitHub Action that integrates Gemini directly into your repositories. I was intrigued by its promise to perform pull request reviews, triage issues, and even modify code using conversational commands right inside GitHub action workflow. Let\u0026rsquo;s try it out.\nMy goal was to see if I could use this action to get instant, automated feedback on pull requests. Instead of waiting for a human to spot simple issues, I wanted an AI to do the first pass, check for common problems, and even explain code changes on demand.\nQuick Start: Getting it Running\n1. Get a Gemini API Key First, This one needs an API key. I grabbed one from Google AI Studio, which has a pretty generous free tier.\n2. Add it as a GitHub Secret Next, we need to save the API key in git repository\u0026rsquo;s secrets (under Settings \u0026gt; Secrets and variables \u0026gt; Actions). Just go and create a new secret named GEMINI_API_KEY and pasted API key there.\n3. Update the .gitignore The CLI creates a local settings folder and temporary credentials, so I added them to my .gitignore file to keep the repo clean.\ngemini-cli settings .gemini/ GitHub App credentials gha-creds-*.json 4. Set up the Workflow This was the fun part. Gemini official Git repo provided a list of example action workflows, which can be copied directly into my local repo .workflow folder with a simple command in GEMINI Cli:\n/setup-github This automatically pulled the example workflow YAML files in my .github/workflows directory.\n5. Try it out! Now everything is set up, it\u0026rsquo;s time for a test drive. Let\u0026rsquo;s create a testing branch, add a test markdown file, and open a pull request. To trigger the review, I need to leave a comment and @gemini-cli in the PR description:\n@gemini-cli Please explain what the test-for-gemini.md file does. As soon as I posted the comment, it automatically triggered a Gemini GitHub Action. The Gemini Dispatch workflow correctly identified my request in the comment and triggered the review workflow later.\nYou see it installed Gemini Cli in a runner, using both GitHub key and the Gemini API key, to run an AI-based PR review based on what I said in the comment. It analysed the new markdown file I created in the test branch, provided a recommendation in the workflow output, and put its opinion right in the PR conversation with suggested change reflected in the PR ready to update and merge.\nOther exampe Gemini Workflows\nThe action comes with several pre-built workflows that can be useful:\nGemini Dispatch: This acts as the central router. It listens for comments and triggers the right workflow, whether it\u0026rsquo;s for reviewing a PR or triaging an issue. Issue Triage: This can automatically label, comment on, and manage new GitHub Issues. I can see this being a huge time-saver for open-source projects. Pull Request Review: This is the one I used. It automatically reviews PRs when they\u0026rsquo;re opened or when you manually trigger it with a comment. Gemini CLI Assistant: This gives you a general-purpose, conversational AI assistant right in your PRs and issues for a whole range of tasks. Final Thoughts \u0026amp; What\u0026rsquo;s Next\nMy key takeaway is that the run-gemini-cli action is a fantastic AI collaborator. It doesn’t replace human review, but it excels at being a tireless assistant that handles the repetitive first-pass checks, provides instant context on demand, and enforces a consistent quality bar on every pull request.\nThis immediately speeds up the feedback loop and frees up our team to focus on what really matters—architecture, logic, and design—instead of getting bogged down in minor details. It\u0026rsquo;s a powerful foundation, and I\u0026rsquo;m already thinking about ways to push it further, like letting it auto-fix trivial errors or integrating it more deeply with other static analysis tools.\nExplore more fun, features or examples from its official GitHub repo\n","permalink":"https://zackblog.work/posts/automate-pr-review-using-gemini-cli-within-github-action/","summary":"\u003cp\u003eI recently found an interesting tool \u003cstrong\u003erun-gemini-cli\u003c/strong\u003e, a GitHub Action that integrates Gemini directly into your repositories. I was intrigued by its promise to perform pull request reviews, triage issues, and even modify code using conversational commands right inside GitHub action workflow. Let\u0026rsquo;s try it out.\u003c/p\u003e\n\u003cp\u003eMy goal was to see if I could use this action to get instant, automated feedback on pull requests. Instead of waiting for a human to spot simple issues, I wanted an AI to do the first pass, check for common problems, and even explain code changes on demand.\u003c/p\u003e","title":"Automate PR Review using Gemini CLI within GitHub Action"},{"content":"In the past few days, I got an idea from my recent electricity and gas bills—thinking about how to run data analysis to examine usage patterns and trends using the Python approach: employing Pandas for reading data, using Numpy for normalisation, and Matplotlib to create a few visualisations.\nHowever, this time I wanted to try talking to Gemini CLI via the terminal to see if it could generate a web-based analytical platform for me, as someone with no developer background.\nObjective: Zack\u0026rsquo;s Super Cool Electricity and Gas Consumption Analysis Platform\nInitially, I got two datasets in CSV format based on energy consumption, which contained electricity and gas hourly usage data for the recent 2 months. I told Gemini CLI via the terminal.\nPlease check my power and gas usage from @electricity_consumption_2months_patterned.csv and @gas_consumption_2months_patterned.csv, acting like a real professional analysis to see the usage pattern and trends insight It understood my request, created a requirement document, chose the tech stack and framework, created scripts to perform the required data analyses, and then built the web app to enable uploading CSVs to analyse trends, with insights into peak usage periods, and to predict future consumption patterns.\nPlease generate a professional requirements and design doc, as I need a web platform to allow users to upload such usage CSV to have power and gas usage and trends analysed # Project: Energy Consumption Analysis and Visualization ## 1. Overview The goal of this project is to analyze hourly electricity and gas consumption data from CSV files to understand usage patterns, identify trends, and visualize the findings in a web-based format. The system will also provide a professional report with optimization suggestions and allow for data export. ## 2. Data Input The system will accept two CSV files as input: - `electricity_consumption_2months_patterned.csv`: Containing electricity usage data. - `gas_consumption_2months_patterned.csv`: Containing gas usage data. ### Data Format Each CSV file is expected to have the following columns: - `Consumption (kwh)`: Energy consumption in kilowatt-hours. - `Estimated Cost Inc. Tax (p)`: Estimated cost in pence. - `Start`: The start time of the consumption period. - `End`: The end time of the consumption period. ## 3. Data Processing and Analysis The backend of the application will perform the following steps: 1. **Data Loading:** Load the electricity and gas consumption data from the provided CSV files. 2. **Data Cleaning and Preparation:** - Parse the `Start` and `End` columns into datetime objects. - Handle any missing or erroneous data. - Create new features from the timestamps, such as: - Hour of the day - Day of the week - Week of the year - Month 3. **Data Analysis:** - **Aggregate Data:** Calculate daily, weekly, and monthly consumption and cost for both electricity and gas. - **Trend Analysis:** Analyze the consumption and cost trends over time. - **Peak/Valley Usage Analysis:** Determine the times of day and days of the week with the highest and lowest energy consumption. - **Cost Analysis:** - **Cost Composition:** Analyze the proportion of electricity and gas costs. - **Cost Comparison:** Compare costs across different time periods. - **Savings Potential Assessment:** Evaluate potential savings based on usage patterns. - **Usage Pattern Analysis:** - **Usage Habit Analysis:** Identify and analyze the user\u0026#39;s energy consumption habits. - **Anomaly Detection:** Detect and flag any unusual consumption data points. - **Pattern Recognition:** Recognize and highlight recurring consumption patterns. ## 4. Visualization and Reporting The web interface will be divided into the following pages: ### 4.1. Data Overview Page - **Key Metrics:** Display key indicators such as total consumption, total cost, and average consumption. - **Overall Statistics:** Provide a summary of the data. ### 4.2. Consumption Trend Analysis Page - **Time Series Chart:** Interactive line chart for visualizing hourly consumption data. - **Trend Charts:** Bar charts for daily, weekly, and monthly consumption and cost trends. - **Peak/Valley Analysis:** Visualizations to highlight peak and off-peak usage. ### 4.3. Cost Analysis Page - **Cost Composition Chart:** Pie or donut chart showing the breakdown of costs between electricity and gas. - **Cost Comparison View:** A view to compare costs over different periods. - **Savings Potential:** Display potential savings with actionable insights. ### 4.4. Usage Pattern Analysis Page - **Habit Analysis:** Visualizations to show consumption habits (e.g., heatmap of hourly usage). - **Anomaly Highlighting:** Highlight anomalies directly on the time series charts. - **Pattern Display:** Show identified recurring patterns. ### 4.5. Professional Report Page - **Comprehensive Report:** A downloadable summary report of the analysis. - **Optimization Suggestions:** Provide a list of recommendations for reducing consumption and costs. - **Data Export:** Buttons to export the raw and processed data to CSV. ## 5. Technology Stack - **Backend:** Python with Flask or FastAPI. - **Data Analysis:** Pandas. - **Frontend:** React or Vue.js. - **Charting Library:** Chart.js or D3.js. - **Styling:** Bootstrap or Tailwind CSS. ## 6. Deployment and Execution The application will be designed to run locally. The user will be able to: 1. Place the consumption CSV files in a designated `data` directory. 2. Run a single command to start the web server. 3. Access the visualizations and reports through a web browser. Following the design above, it took about 30 minutes with 3 rounds of debugging to fix issues like the time attribute in the dataset and some npm package import issues. The web page looked pretty nice and modern:\nWhen looking at weekly trends, the AI models highlight interesting patterns, such as peak energy usage at different times during the day and over the week, which correlates with more time spent at home. This level of insight was once reserved for professional analysts but is now accessible to anyone with the right dataset.\nEven with a professional summary report.\nThe Transformation of Programming\nMy typical approach involves using Python as my preferred and familiar toolset for one-off analyses. However, this AI experiment helped me build a reusable React application where data could be easily uploaded and re-analysed. It chose a stack of technical frameworks that I haven’t mastered, completing in 30 minutes what would have taken me days manually.\nThis level of efficiency is both unsettling and strangely reassuring. If AI can compress days of work into minutes, what is our true value? In the future, the key to competition may no longer lie in mastering every framework, but in the vision we apply to them.\nConclusion: A Future Built with AI\nClaude Code or Gemini CLI won’t replace most programmers, but they will empower those who master them to become far more effective. The real change isn’t just technical; it’s a fundamental shift in our roles.\nIn the past, developers’ value was in translating requirements into code. Now, it lies in proposing good ideas and solving real-world problems. Code is just a tool, and AI helps us wield it. AI can execute the plan, but we must provide the vision.\nThis is why labels like \u0026ldquo;Python programmer\u0026rdquo; or \u0026ldquo;front-end engineer\u0026rdquo; are becoming obsolete. The only title that matters now is \u0026ldquo;problem-solver using AI.\u0026rdquo; The greatest threat isn’t that AI will take your job, but that refusing to embrace it will make your skills irrelevant.\nThe full source code and dataset are now available at my GitHub repo.\n","permalink":"https://zackblog.work/posts/ai-coding-build-an-electricity-and-gas-analytic-platform/","summary":"\u003cp\u003eIn the past few days, I got an idea from my recent electricity and gas bills—thinking about how to run data analysis to examine usage patterns and trends using the Python approach: employing Pandas for reading data, using Numpy for normalisation, and Matplotlib to create a few visualisations.\u003c/p\u003e\n\u003cp\u003eHowever, this time I wanted to try talking to Gemini CLI via the terminal to see if it could generate a web-based analytical platform for me, as someone with no developer background.\u003c/p\u003e","title":"AI Coding - Build an Electricity and Gas Analytic Platform"},{"content":"This was a linguistic analysis project where the primary goal was not just to count words, but to evaluate the language, themes, and emotional tone of the children\u0026rsquo;s show \u0026ldquo;Peppa Pig\u0026rdquo; (specifically, the first four seasons) to determine its suitability for a pre-kindergarten audience. In this study, I will try to answer a broader question:\n\u0026ldquo;Beyond a simple word list, what can a multi-faceted data analysis tell us about the show\u0026rsquo;s true educational and emotional value?\u0026rdquo;\nPhase 1 \u0026amp; 2: Data Acquisition and Extraction\nThe original dataset is a 220-page PDF of the show\u0026rsquo;s transcripts of first 4 seansons. The first step involves cleaning and structuring the data using PyPDF2 to trim irrelevant introductory pages.\nDue to the original pdf page has a left and right two vertical context, I need to switch to pdfplumber which is another pdfplumber Python library designed specifically for extracting structured data from PDFs — especially tables and well-formatted text to manipulation (like merging/splitting pages) to handle the complex two-column layout. In addition, I noticed there are headers/footers also need to be removed, now we have a clean, structured dataset saved as a single CSV file: season1_4_all_pages_cleaned.csv.\nText pre-processing complete. Total words before filtering: 88497 Total words after filtering: 42566 Phase 3: Initial NLP and Exploratory Data Analysis (EDA)\nNow we need to ensure the data to be ready for next step, some extra pre-processing jobs include: converting all text to lowercase, removing punctuation, and using nltk for tokenization. Due to the nature of this show, there are many words we need to build a custom stop word list to filter out character names like (\u0026lsquo;peppa\u0026rsquo;, \u0026lsquo;george\u0026rsquo;) and sounds (\u0026lsquo;oink\u0026rsquo;, \u0026lsquo;woof\u0026rsquo;), which could improve the signal-to-noise ratio. We can use a WordCloud to have a visual sense of the most prominent terms.\nPhase 4: Readability Analysis – How Complex is the Language?\nNow, what about the overall language? I\u0026rsquo;ll calculate readability scores to assess the text\u0026rsquo;s complexity and determine the appropriate grade level for the audience. I will use two standard metrics:- Flesch-Kincaid Grade Level: Estimates the U.S. school grade level required to understand the text.\nFlesch Reading Ease: Rates text on a 100-point scale. Higher scores indicate easier-to-read material.: Flesch-Kincaid Grade Level: 2.58 Flesch Reading Ease Score: 88.85 Interpretation: Easy to read. A grade level of ~2.5 means the language is simple enough for a second-grader to read. For a preschooler\u0026rsquo;s *listening* comprehension—which is always several levels higher than their reading ability—this is the sweet spot. The language is easy to follow but still models proper sentence structure.\nPhase 5: Sentiment Analysis – What is the Emotional Tone?\nNext, I will analyze the emotional tone of the dialogue using VADER (Valence Aware Dictionary and sEntiment Reasoner), a lexicon and rule-based sentiment analysis tool specifically tuned for social media text, but it also works well on many kinds of English text. It\u0026rsquo;s part of the nltk (Natural Language Toolkit) library in Python. The results a 216-to-1 positive-to-negative ratio and a complete absence of flat, neutral dialogue, the data proves this show creates an overwhelmingly positive, safe, and emotionally engaging environment for its viewers:\nsentiment_type positive 216 negative 1 Name: count, dtype: int64 Phase 6: Topic Modeling – What is the Show Actually About?\nLDA (Latent Dirichlet Allocation ) is a topic modeling technique — an unsupervised machine learning algorithm used to discover hidden thematic structures (topics) in a collection of documents. It assumes that:- Each document is a mixture of topics.\nEach topic is a mixture of words. Here I will use LDA to help answer: “What topics and themes of the show are present in this text data, and how are they distributed?” After refining the model to filter out noise, five distinct topics emerged: Topic 0 (Travel \u0026amp; Excursions): car, look, mr, everyone... Topic 1 (Outdoor Play): muddy, boat, little, good... Topic 2 (Toys \u0026amp; Imaginative Play): dinosaur, teddy, play, box... Topic 3 (General Activities): ball, game, find, house... Topic 4 (Social Interactions): rabbit, please, friends, hello... This confirmed that the show\u0026rsquo;s narrative is consistently focused on core childhood experiences: family trips, playing outside, imaginative play with toys, and polite social interaction with friends.\nPhase 7: Benchmark Analysis – How Does it Compare to a Standard?\nFinally, I returned to the classic benchmark: the Dolch Sight Words list, which is a set of 220 frequently used English words (plus 95 common nouns) that children are encouraged to recognize by sight, without needing to sound them out used in early childhood literacy (Pre-K to Grade 3). This analysis provides a more traditional academic measure.\nPeppa Pig season 1-4 use 208 out of 315 Dolch words. That\u0026#39;s an overlap of 66.03%. An overlap of 66% is substantial, showing a strong alignment with foundational vocabulary for early readers. The analysis also revealed that the most common words in Peppa Pig not on the list are social words (\u0026lsquo;hello\u0026rsquo;, \u0026lsquo;mr\u0026rsquo;, \u0026rsquo;everyone\u0026rsquo;) and play-related words (\u0026lsquo;dinosaur\u0026rsquo;), which reinforces the findings from the topic modeling.\nConclusion: A Data-Driven Verdict\nThis multi-faceted analysis went far beyond a simple word count. By combining readability scores, sentiment analysis, topic modeling, and a benchmark comparison, I was able to construct a complete profile of the show. The data provides a clear and conclusive answer: with its simple sentence structures, overwhelmingly positive emotional tone, and consistent focus on developmentally appropriate themes, \u0026ldquo;Peppa Pig\u0026rdquo; is an exceptionally well-suited and beneficial program for its target pre-kindergarten audience.\nThe full notebook and dataset are now available at my GitHub repo.\n","permalink":"https://zackblog.work/posts/exploratory-data-analysis-eda-with-peppa-pig/","summary":"\u003cp\u003eThis was a linguistic analysis project where the primary goal was not just to count words, but to evaluate the language, themes, and emotional tone of the children\u0026rsquo;s show \u0026ldquo;Peppa Pig\u0026rdquo; (specifically, the first four seasons) to determine its suitability for a pre-kindergarten audience. In this study, I will try to answer a broader question:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u0026ldquo;Beyond a simple word list, what can a multi-faceted data analysis tell us about the show\u0026rsquo;s true educational and emotional value?\u0026rdquo;\u003c/strong\u003e\u003c/p\u003e","title":"Exploratory Data Analysis (EDA) with Peppa Pig"},{"content":"In the last post, I used Claude Code with Kimi K2 to complete the Django web app layout optimization. One thing I wasn’t satisfied with was the pagination system on this Django blog. It used a numbered pagination interface at the bottom of the home page—functional, but a bit dated. I wanted to modernize it with automatic loading of more posts via infinite scroll.\nHowever, I couldn’t get this done using Kimi K2. Despite spending a significant number of tokens (and money), the model couldn’t provide a working solution.\nSo, I decided to try Google Gemini CLI to see how it would handle the task.\nObjective: Add Infinite Scroll\nThe goal was simple: show the initial 12 posts, but instead of making users click “Next” or choose a page number, automatically load 5 more posts as they scroll towards the bottom of the page.\nGetting Started: Installing Gemini CLI\nEnsure Node.js v18 or higher is installed, and install Google Gemini CLI globally:\n# Install curl, Node.js \u0026amp; npm (v20.x) RUN apt-get update \\ \u0026amp;\u0026amp; apt-get install -y curl ca-certificates \\ \u0026amp;\u0026amp; curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \\ \u0026amp;\u0026amp; apt-get install -y nodejs \\ \u0026amp;\u0026amp; rm -rf /var/lib/apt/lists/* # Install Google Gemini CLI globally (non-interactive) RUN npm install -g @google/gemini-cli Launching Gemini and Authenticating\nRun gemini in terminal and follow the OAuth flow to log in with Google account. This unlocks free access to Gemini 2.5 Pro.\nDescribing the Goal to Gemini CLI\nTalk to Gemini in plain English:\n“Can you validate the Django web app I\u0026rsquo;m running in the current folder? I think on the home page it lists 12 posts by default, sorted from newest. There are pagination buttons at the bottom to navigate to older posts. I don\u0026rsquo;t want users to click pagination buttons anymore. Instead, can Django automatically show 12 posts by default, then load 5 more as the user scrolls down? Is that achievable?”\nGemini immediately understood. It read the local Django files and settings, identified the feature as infinite scroll and laid out a clear plan:\nBackend (Django): Keep existing paginator logic but add a new view to handle asynchronous requests. This view will return a small HTML snippet with the next batch of posts. Frontend (JavaScript): Modify home.html to detect when users scroll to the bottom. Then use fetch() to request new posts and append them to the DOM. Remove old pagination. ✔ ReadManyFiles will attempt to read and concatenate files using patterns: `blog/views.py`, `blog/templates/blog/home.html` (within target…) ### ReadManyFiles Result (Target Dir: `/mnt/f/zack-gitops-project/django_project`) Successfully read and concatenated content from **2 file(s)**. **Processed Files:** - `blog/templates/blog/home.html` - `blog/views.py` I gave allow so Gemini can now create and edit files under the Django folder for me.\nStep 1: Modifying the Backend\nCreating the New View\nGemini added a new view in blog/views.py to serve the extra posts:\n# blog/views.py from django.core.paginator import Paginator from django.http import HttpResponse from django.shortcuts import render from .models import Post def infinite_scroll_posts(request): page_number = request.GET.get(\u0026#34;page\u0026#34;) posts_list = Post.objects.all().order_by(\u0026#39;-date_posted\u0026#39;) paginator = Paginator(posts_list, 5) page_obj = paginator.get_page(page_number) return render(request, \u0026#34;blog/post_list_partial.html\u0026#34;, {\u0026#34;posts\u0026#34;: page_obj}) Creating the Partial Template\nThis template (post_list_partial.html) includes only the post loop, making the response lightweight:\n\u0026lt;!-- blog/templates/blog/post_list_partial.html --\u0026gt; {% for post in posts %} \u0026lt;article class=\u0026#34;media content-section\u0026#34;\u0026gt; \u0026lt;img class=\u0026#34;rounded-circle article-img\u0026#34; src=\u0026#34;{{ post.author.profile.image.url }}\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;media-body\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;article-metadata\u0026#34;\u0026gt; \u0026lt;a class=\u0026#34;mr-2\u0026#34; href=\u0026#34;{% url \u0026#39;user-posts\u0026#39; post.author.username %}\u0026#34;\u0026gt;{{ post.author }}\u0026lt;/a\u0026gt; \u0026lt;small class=\u0026#34;text-muted\u0026#34;\u0026gt;{{ post.date_posted|date:\u0026#34;F d, Y\u0026#34; }}\u0026lt;/small\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;h2\u0026gt;\u0026lt;a class=\u0026#34;article-title\u0026#34; href=\u0026#34;{% url \u0026#39;post-detail\u0026#39; post.id %}\u0026#34;\u0026gt;{{ post.title }}\u0026lt;/a\u0026gt;\u0026lt;/h2\u0026gt; \u0026lt;p class=\u0026#34;article-content\u0026#34;\u0026gt;{{ post.content|striptags|slice:\u0026#34;:200\u0026#34; }}\u0026lt;/p\u0026gt; \u0026lt;a href=\u0026#34;{% url \u0026#39;post-detail\u0026#39; post.id %}\u0026#34; class=\u0026#34;read-more-link\u0026#34;\u0026gt;Read more\u0026lt;/a\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;/article\u0026gt; {% endfor %} Adding the URL\nGemini updated blog/urls.py:\n# blog/urls.py urlpatterns = [ # ... other paths path(\u0026#34;infinite-scroll-posts/\u0026#34;, views.infinite_scroll_posts, name=\u0026#34;infinite-scroll-posts\u0026#34;), ] Step 2: Updating the Frontend\nGemini then modified home.html to remove the pagination and include infinite scroll logic:\n\u0026lt;!-- blog/templates/blog/home.html --\u0026gt; {% extends \u0026#34;blog/base.html\u0026#34; %} {% block content %} \u0026lt;div id=\u0026#34;post-container\u0026#34;\u0026gt; {% for post in posts %} \u0026lt;article class=\u0026#34;media content-section\u0026#34;\u0026gt; \u0026lt;!-- ... post content ... --\u0026gt; \u0026lt;/article\u0026gt; {% endfor %} \u0026lt;/div\u0026gt; \u0026lt;div id=\u0026#34;loading\u0026#34; style=\u0026#34;display:none;\u0026#34;\u0026gt; \u0026lt;p\u0026gt;Loading...\u0026lt;/p\u0026gt; \u0026lt;/div\u0026gt; \u0026lt;script\u0026gt; let page = 2; let isLoading = false; const loading = document.getElementById(\u0026#39;loading\u0026#39;); const postContainer = document.getElementById(\u0026#39;post-container\u0026#39;); function loadMorePosts() { if (isLoading) return; isLoading = true; loading.style.display = \u0026#39;block\u0026#39;; fetch(`/infinite-scroll-posts/?page=${page}`) .then(response =\u0026gt; response.text()) .then(data =\u0026gt; { if (data.trim().length \u0026gt; 0) { postContainer.innerHTML += data; page++; isLoading = false; loading.style.display = \u0026#39;none\u0026#39;; } else { loading.style.display = \u0026#39;none\u0026#39;; window.removeEventListener(\u0026#39;scroll\u0026#39;, handleScroll); } }) .catch(error =\u0026gt; { console.error(\u0026#39;Error loading more posts:\u0026#39;, error); isLoading = false; loading.style.display = \u0026#39;none\u0026#39;; }); } function handleScroll() { if (window.innerHeight + window.scrollY \u0026gt;= document.body.offsetHeight - 200) { loadMorePosts(); } } window.addEventListener(\u0026#39;scroll\u0026#39;, handleScroll); \u0026lt;/script\u0026gt; {% endblock content %} Conclusion\nAnd just like that, the feature was complete. Now, the home page loads 12 posts initially, and as I scroll, new posts appear automatically.\nThis was an exciting example of AI-assisted programming and development. Gemini CLI didn’t just generate code—it understood the requirement, outlined a clear plan, and executed it smoothly by accessing local file system and manage everything for me.\nIt’s like having a senior full-stack developer knowing every type of programming language or application on standby—someone who understands your ideas in plain language and coding them to life. This is truely a game-changer for people without a programming background, being able to optimize a Django web app—handling backend views, URL routing, and frontend JavaScript—feels wild.\nWe’re truly in the AI era. Every day there are new or enhanced AI model launching, they chasing to be the top. Imagine two years ago, how people were excited by the first version of ChatGPT. Now crazy in AI coding race: Claude Code, Gemini, GPT-4o, Kimi K2, Qwen Code, and others\n","permalink":"https://zackblog.work/posts/ai-assisted-coding-with-google-gemini-cli/","summary":"\u003cp\u003eIn the last post, I used Claude Code with Kimi K2 to complete the Django web app layout optimization. One thing I wasn’t satisfied with was the pagination system on this Django blog. It used a numbered pagination interface at the bottom of the home page—functional, but a bit dated. I wanted to modernize it with automatic loading of more posts via infinite scroll.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"/images/gemini6.png\"\u003e\u003cimg alt=\"[Image Placeholder 01: Introduction Graphic]\" loading=\"lazy\" src=\"/images/gemini6.png\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eHowever, I couldn’t get this done using Kimi K2. Despite spending a significant number of tokens (and money), the model couldn’t provide a working solution.\u003c/p\u003e","title":"AI-assisted Coding with Google Gemini Cli"},{"content":"I originally used Jekyll, a straightforward static site generator, to build my blog. It was easy to set up and ideal for managing content in Markdown. However, it had its limitations: serving only static content meant a lack of dynamic features and essential tools like a built-in database and user management. That\u0026rsquo;s when I decided to migrate to Django. With its ability to handle dynamic content and a fully integrated database, it was the perfect solution and the clear choice for the future of my blog.\n1. Setting Up the Local Django Environment\nI decided to build a Docker image to set up a portable Python environment. By mounting the working directory to my local Git repository, I ensured consistency across platforms. Using VSCode\u0026rsquo;s container integration, I could launch directly into the development environment, making the setup seamless.\n2. Creating the Django Application\nThe next step was to build the Django application. This involved structuring the project, creating models for blog posts, setting up user management, and managing content with Django\u0026rsquo;s powerful ORM. A key task was writing a script to convert my old Jekyll Markdown posts into the new Django database format.\n3. Cloud Hosting, Domain, and SSL\nI purchased a domain through Cloudflare, configured the DNS, and implemented SSL. This finally got rid of the tedious process of manually renewing a free SSL certificate every three months.\n4. Upgrading Django to the Latest Version\nLooking at the gap between Django 2.1 and 5.2 felt like staring up at a mountain. I knew a direct jump was risky, so I opted for an incremental approach, hopping between Long-Term Support (LTS) versions to ensure a secure and manageable upgrade.\n# Navigate to the project directory root@zack:~# vim Dockerfile # Update requirements in both Django project and Docker context vim requirements.txt # Example change for one of the upgrade steps Django==2.2 # Changed from 2.1 to 2.2 django-crispy-forms==2.3 # Updated crispy-bootstrap4==2024.1 # Updated # ... and other dependencies # Install the upgraded packages pip install -U -r requirements.txt # Run checks and the test suite to find regressions python manage.py check python manage.py test # Verify the new Django version python3 -m django --version This path felt much more manageable: 2.1 → 2.2 → 3.2 → 4.2 → 5.2.\n5. Modernizing the Layout and Fixing Dependencies\nUpgrade runtime (Python) from 3.8 slim to 3.13 slim in docker base image to be compatitable with Django 4.0+- Using a dedicated upgrade branch, merge to editing branch only ensure success of each version upgrade- Adjusted and updated the CSS to create a more modern layout.\nFixed TemplateDoesNotExist error where upgraded dependencies needed to be explicitly registered in INSTALLED\\_APPS in settings.py. Resolved CSRF errors by adding trusted origins to settings.py, a requirement for Django 4+ to handle flexible production IPs. 6. Updating CI/CD with GitHub Actions\nI updated the existing GitHub Actions workflow from the Jekyll project to automate the deployment process for the new Django app, fixed the CI/CD pipeline and aligned all environments, so can be confident that what tested locally is what is running on target EC2. This reproducibility is the ultimate goal of DevOps practices.\nBefore Django version upgrade and Layout Modernization\nA Look at the New Design after upgrade\nAchievements and Reflections\nAfter days of focused work, I leaned back and looked at what I had built. Here is a summary of the final achievements:\n✅ A blazing-fast app running on Django 5.2 and Python 3.13. ✅ A robust, automated CI/CD pipeline that ensure reproducibility. ✅ Consistent environments across local, testing, and production, eliminating deployment surprises. ✅ A fresh, modern CSS layout that makes the whole app feel new again. Conclusion\nMigrating from Jekyll to Django wasn’t just about solving a technical problem—it was a demonstration of best DevOps practices. The journey was insightful, and I enjoyed the debugging challenges. It reaffirmed the importance of using the right tools for the job.\n","permalink":"https://zackblog.work/posts/migrating-blog-to-django-web-app/","summary":"\u003cp\u003eI originally used Jekyll, a straightforward static site generator, to build my blog. It was easy to set up and ideal for managing content in Markdown. However, it had its limitations: serving only static content meant a lack of dynamic features and essential tools like a built-in database and user management. That\u0026rsquo;s when I decided to migrate to Django. With its ability to handle dynamic content and a fully integrated database, it was the perfect solution and the clear choice for the future of my blog.\u003c/p\u003e","title":"Migrating Blog to Django Web App"},{"content":"Effortless Coding with Claude Code and Kimi K2: Features, Pricing, and Setup Guide\nRecently Kimi K2 became one of the most popular model in HuggingFace, claimed its compatible with Claude Code API. With my new sign up and got a $15 free credit, Let\u0026rsquo;s see how it can intergrate with Claud Code and validate its coding performance!\n🧠 Claude Code + Kimi K2?\nClaude Code is command-line tool from Anthropic Claude, powered by its latest models (Opus, Sonnet, Haiku). It is highly effective but can be expensive.\nKimi K2 is a recently released model from Moonshot AI. It offers similar AI coding capabilities at a much lower price.\nKey Features:\nClaude-compatible prompt API Low latency and stable access in China Affordable pricing 📈 Pricing Comparison (Per 1 Million Tokens)\nModel/API Input Cost Output Cost Total Claude Opus $15 $75 $90 Claude Sonnet $3 $15 $18 Claude Haiku $0.80 $4 $4.80 Kimi K2 ~0.50 ~$2.50 ~$3.05 Kimi K2 is over 90% cheaper than Claude Opus and significantly less expensive than Sonnet or Haiku, making it ideal for budget-conscious developers (like me).\n✨ Installation Guide (Node.js Setup)\nStep 1: Environment Preparation Ensure you have Node.js 18+ installed:\n# Check versions root@zack:~# node -v v20.19.3 root@zack:~# npm -v 11.4.2 Step 2: Install Claude Code CLI\n# Global install npm install -g @anthropic-ai/claude-code # Launch the interface claude 🚀 Kimi K2 API Integration\n1. Get the Kimi API Key Register at the Moonshot Console and generate an API Key.\n2. Configure Environment Variables For Bash (Linux/WSL):\n# Add variables to your shell profile echo \u0026#39;export ANTHROPIC_BASE_URL=\u0026#34;https://api.moonshot.cn/anthropic/\u0026#34;\u0026#39; \u0026gt;\u0026gt; ~/.bashrc echo \u0026#39;export ANTHROPIC_API_KEY=\u0026#34;your_Kimi_API_Key\u0026#34;\u0026#39; \u0026gt;\u0026gt; ~/.bashrc # Reload the profile to apply changes source ~/.bashrc Final Step: Verify the Launch Run the tool again:\nclaude BINGO! We see the API source listed as API Base URL: https://api.moonshot.cn/anthropic/ . This confirms that Claude Code is now using the Kimi K2 API via the overridden environment variables.\nFixing My Blog\u0026rsquo;s CSS Overflow Issue\nFor a long time, I had been bothered by a persistent content and code snippet overflow issue on my blog. Due to my limited frontend expertise, I couldn\u0026rsquo;t fix the CSS, even after trying several other AI tools (like GPT-3o, Gemini 2.5 Pro, and DeepSeek R1). I decided to see if the Claude Code + Kimi K2 combo could solve the problem.\nOnce I explained the issue, you see it correctly started to analyze project\u0026rsquo;s folder structure, identifying the key CSS files which handle the media and view, and suggesting what needed to be adjusted.\nThen requested it to update the code file on my behalf.\nSeems the initial fix only worked for the vertical orientation but not horizontal, so a little more iteration was needed.\nWell done, the issue was completely resolved! All together toke less than 5 minutes via API with less the $0.77 spending!!\n✨ Conclusion\nThe combination of the Claude Code CLI with the Kimi K2 API is a powerful and cost-effective setup for AI-assisted coding. If you\u0026rsquo;re after the absolute best performance and don\u0026rsquo;t mind the cost, Claude Code is your go-to. But for excellent value with similar features, Kimi K2 is a clear winner.\n","permalink":"https://zackblog.work/posts/claude-code-kimi-k2/","summary":"\u003cp\u003e\u003cstrong\u003eEffortless Coding with Claude Code and Kimi K2: Features, Pricing, and Setup Guide\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eRecently Kimi K2 became one of the most popular model in HuggingFace, claimed its compatible with Claude Code API. With my new sign up and got a $15 free credit, Let\u0026rsquo;s see how it can intergrate with Claud Code and validate its coding performance!\u003c/p\u003e\n\u003chr\u003e\n\u003cp\u003e\u003cstrong\u003e🧠 Claude Code + Kimi K2?\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eClaude Code\u003c/strong\u003e is command-line tool from Anthropic Claude, powered by its latest models (Opus, Sonnet, Haiku). It is highly effective but can be expensive.\u003c/p\u003e","title":"Claude Code + Kimi K2"},{"content":"Discover Kubernetes 1.33\u0026rsquo;s In-Place Vertical Scaling. Learn how to resize pod CPU and memory on the fly without restarts, eliminating downtime and optimizing resource costs.\nPreviously, adjusting the CPU or memory for Kubernetes pods necessitated a disruptive full restart, causing downtime particularly detrimental for critical and stateful applications.\nHowever, Kubernetes 1.33 introduces \u0026ldquo;In-Place Pod Vertical Scaling\u0026rdquo; (K8s.io docs) as a default beta feature, revolutionizing this by allowing on-the-fly CPU and memory adjustments to running pods without any restarts. This game-changing capability eliminates downtime for resource changes, enables better cost optimization by avoiding over-provisioning, and significantly benefits stateful workloads like databases by allowing them to scale without interruption.\nReal-World Use Cases:\nDatabases (e.g., PostgreSQL): Give it more RAM for a heavy query without stopping transactions. Node.js API Services: Handle traffic spikes by giving them more CPU/memory on the fly. ML Inference Services (e.g., TensorFlow Serving): Allocate more resources for larger models or batch sizes without disrupting requests. Service Mesh Sidecars (e.g., Envoy): Dynamically adjust resources based on traffic without affecting the main application. What’s Really Happening\nWhen we submit a patch, the kubelet quickly checks if the node has enough allocatable capacity to handle the new request. If it does, the kubelet communicates with the container runtime (containerd or CRI-O) via the Container Runtime Interface (CRI) to adjust CPU and memory resources on the fly—no container restarts needed. This update process is asynchronous and non-blocking, with clear status updates available to keep us informed.\nKey Points:\nResources.requests and resources.limits are now mutable on the fly(KEP-1287). Kubelet verifies node capacity before applying resource changes. Kubelet uses CRI to instruct container runtimes to adjust cgroups without restarting containers. The resizing process is asynchronous and non-blocking. New pod conditions in kubectl describe pod: PodResizePending — node is busy, retry later. PodResizeInProgress — resizing is underway. Hands-On:\nEnsuer to have a k8s cluster with version 1.33 in hand, here I am going to create a test pod, with initial requested CPU and Memory resource, then try to patch the CPU and memory to see if this feature will work without pod restart.\nroot@133-m1:~# kubectl get node NAME STATUS ROLES AGE VERSION 133-m1 Ready control-plane 3m55s v1.33.1 133-w1 Ready worker 3m28s v1.33.1 root@133-m1:~# vim test.yaml apiVersion: v1 kind: Pod metadata: name: resize-demo spec: containers: - name: resource-watcher image: ubuntu:22.04 command: - \u0026#34;/bin/bash\u0026#34; - \u0026#34;-c\u0026#34; - | apt-get update \u0026amp;\u0026amp; apt-get install -y procps bc echo \u0026#34;=== Pod Started: $(date) ===\u0026#34; # Functions to read container resource limits get_cpu_limit() { if [ -f /sys/fs/cgroup/cpu.max ]; then # cgroup v2 local cpu_data=$(cat /sys/fs/cgroup/cpu.max) local quota=$(echo $cpu_data | awk \u0026#39;{print $1}\u0026#39;) local period=$(echo $cpu_data | awk \u0026#39;{print $2}\u0026#39;) if [ \u0026#34;$quota\u0026#34; = \u0026#34;max\u0026#34; ]; then echo \u0026#34;unlimited\u0026#34; else echo \u0026#34;$(echo \u0026#34;scale=3; $quota / $period\u0026#34; | bc) cores\u0026#34; fi else # cgroup v1 local quota=$(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us) local period=$(cat /sys/fs/cgroup/cpu/cpu.cfs_period_us) if [ \u0026#34;$quota\u0026#34; = \u0026#34;-1\u0026#34; ]; then echo \u0026#34;unlimited\u0026#34; else echo \u0026#34;$(echo \u0026#34;scale=3; $quota / $period\u0026#34; | bc) cores\u0026#34; fi fi } get_memory_limit() { if [ -f /sys/fs/cgroup/memory.max ]; then # cgroup v2 local mem=$(cat /sys/fs/cgroup/memory.max) if [ \u0026#34;$mem\u0026#34; = \u0026#34;max\u0026#34; ]; then echo \u0026#34;unlimited\u0026#34; else echo \u0026#34;$((mem / 1048576)) MiB\u0026#34; fi else # cgroup v1 local mem=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes) echo \u0026#34;$((mem / 1048576)) MiB\u0026#34; fi } # Print resource info every 5 seconds while true; do echo \u0026#34;---------- Resource Check: $(date) ----------\u0026#34; echo \u0026#34;CPU limit: $(get_cpu_limit)\u0026#34; echo \u0026#34;Memory limit: $(get_memory_limit)\u0026#34; echo \u0026#34;Available memory: $(free -h | grep Mem | awk \u0026#39;{print $7}\u0026#39;)\u0026#34; sleep 5 done resizePolicy: - resourceName: cpu restartPolicy: NotRequired - resourceName: memory restartPolicy: NotRequired resources: requests: memory: \u0026#34;128Mi\u0026#34; cpu: \u0026#34;100m\u0026#34; limits: memory: \u0026#34;128Mi\u0026#34; cpu: \u0026#34;100m\u0026#34; Explore the Pod’s Initial State:\nkubectl describe pod resize-demo | grep -A8 Limits: root@133-m1:~# kubectl describe pod resize-demo | grep -A8 Limits: Limits: cpu: 100m memory: 128Mi Requests: cpu: 100m memory: 128Mi kubectl logs resize-demo --tail=8 === Pod Started: Sat May 31 01:48:50 UTC 2025 === ---------- Resource Check: Sat May 31 01:48:50 UTC 2025 ---------- CPU limit: .100 cores Memory limit: 128 MiB Available memory: 2.8Gi Resize CPU:\nkubectl patch pod resize-demo --subresource resize --patch \\ \u0026#39;{\u0026#34;spec\u0026#34;:{\u0026#34;containers\u0026#34;:[{\u0026#34;name\u0026#34;:\u0026#34;resource-watcher\u0026#34;, \\ \u0026#34;resources\u0026#34;:{\u0026#34;requests\u0026#34;:{\u0026#34;cpu\u0026#34;:\u0026#34;200m\u0026#34;}, \u0026#34;limits\u0026#34;:{\u0026#34;cpu\u0026#34;:\u0026#34;200m\u0026#34;}}}]}}\u0026#39; root@133-m1:~# kubectl describe pod resize-demo | grep -A8 Limits: Limits: cpu: 200m memory: 128Mi Requests: cpu: 200m memory: 128Mi kubectl logs resize-demo --tail=8 ---------- Resource Check: Sat May 31 01:49:16 UTC 2025 ---------- CPU limit: .200 cores Memory limit: 128 MiB Available memory: 2.8Gi Resize Memory:\nkubectl patch pod resize-demo --subresource resize --patch \\ \u0026#39;{\u0026#34;spec\u0026#34;:{\u0026#34;containers\u0026#34;:[{\u0026#34;name\u0026#34;:\u0026#34;resource-watcher\u0026#34;, \u0026#34;\\ resources\u0026#34;:{\u0026#34;requests\u0026#34;: {\u0026#34;memory\u0026#34;:\u0026#34;256Mi\u0026#34;}, \u0026#34;limits\u0026#34;:{\u0026#34;memory\u0026#34;:\u0026#34;256Mi\u0026#34;}}}]}}\u0026#39; root@133-m1:~# kubectl describe pod resize-demo | grep -A8 Limits: Limits: cpu: 200m memory: 256Mi Requests: cpu: 200m memory: 256Mi root@133-m1:~# kubectl logs resize-demo --tail=8 ---------- Resource Check: Sat May 31 01:50:16 UTC 2025 ---------- CPU limit: .200 cores Memory limit: 256 MiB Available memory: 2.8Gi Verify No Container Restarts Occurred:\nkubectl get pod resize-demo -o jsonpath=\u0026#39;{.status.containerStatuses[0].restartCount}\u0026#39; 0 Cloud Provider Support 🌩️\nBefore rush to try this in production, let’s look at support across major Kubernetes providers:\nGoogle Kubernetes Engine (GKE): Available on the Rapid channel in GKE (GKE docs). Amazon EKS: Kubernetes 1.33 version is available since May 2025. Azure AKS: Kubernetes 1.33 version is now available for Preview (AKS Release Notes). Limits with Default K8S VPA\nCurrent Status (K8s 1.33): VPA does not yet support in-place resizing — it still recreates pods when adjusting resources. This limitation is explicitly noted in the Kubernetes documentation: “As of Kubernetes 1.33, VPA does not support resizing pods in-place, but this integration is being worked on.”\nActive development is happening in kubernetes/autoscaler PR 7673 to integrate VPA with in-place resizing capability.\nThe Future Integration We Need:\nKubernetes 1.33’s in-place pod resize marks a major step toward making vertical scaling as smooth and non-disruptive as horizontal autoscaling, but there’s more to come. Future improvements include deeper Vertical Pod Autoscaler (VPA) integration to minimize pod evictions, expansion beyond CPU and memory to resources like GPUs and ephemeral storage, better scheduler awareness to prevent unexpected evictions, integration with the Cluster Autoscaler for smarter node scaling, and advanced metrics-based resizing using application-level signals. Together, these developments aim to make vertical scaling fully dynamic, efficient, and interruption-free—inviting users to experiment and help shape this evolving capability.\n","permalink":"https://zackblog.work/posts/kubernetes-1-33-in-place-pod-vertical-scaling/","summary":"\u003cp\u003e\u003cstrong\u003eDiscover Kubernetes 1.33\u0026rsquo;s In-Place Vertical Scaling. Learn how to resize pod CPU and memory on the fly without restarts, eliminating downtime and optimizing resource costs.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003ePreviously, adjusting the CPU or memory for Kubernetes pods necessitated a disruptive full restart, causing downtime particularly detrimental for critical and stateful applications.\u003c/p\u003e\n\u003cp\u003eHowever, Kubernetes 1.33 introduces \u0026ldquo;\u003cstrong\u003eIn-Place Pod Vertical Scaling\u003c/strong\u003e\u0026rdquo; (\u003ca href=\"https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eK8s.io docs\u003c/a\u003e) as a default beta feature, revolutionizing this by allowing on-the-fly CPU and memory adjustments to running pods without any restarts. This game-changing capability eliminates downtime for resource changes, enables better cost optimization by avoiding over-provisioning, and significantly benefits stateful workloads like databases by allowing them to scale without interruption.\u003c/p\u003e","title":"Kubernetes 1.33: In-Place Pod Vertical Scaling"},{"content":"Managing cloud costs effectively, especially on AWS, is crucial. Wasted resources can easily inflate bills. This post introduces the AWS Cost Explorer MCP Server, a tool designed to simplify analyzing your AWS spending using the Model Context Protocol (MCP).\nGitHub Repository: https://github.com/awslabs/mcp/tree/main/src/cost-explorer-mcp-server\nWhat this AWS Cost Explorer MCP Server Does\nThis specific MCP Server, provided by AWS Labs, acts as a specialized tool that connects an AI assistant like Amazon Q directly to the detailed AWS cost and usage data. Think of it as giving us AI assistant the specific knowledge and tools needed to understand and analyze our cloud spending.\nLeveraging this server through an AI like Amazon Q offers significant advantages over manually navigating the AWS Cost Explorer console or writing complex API queries:\nDeeper, Actionable Cost Insights: Go beyond standard console reports. The server allows the AI to break down costs granularly (by service, region, tag, etc.) and identify specific drivers of spending changes. Conversational Cost Querying: Ask complex cost questions in plain English directly within your chat interface (like Amazon Q). For example, query \u0026ldquo;Why did my S3 costs increase last Tuesday?\u0026rdquo; or \u0026ldquo;Which EC2 instances in the \u0026lsquo;dev\u0026rsquo; environment are costing the most?\u0026rdquo; without needing specialized query languages. Automated, Context-Aware Optimization: This is a key benefit. The server can analyze your Infrastructure as Code (IaC) definitions and current resource usage patterns to provide tailored optimization suggestions. It might recommend switching specific instances to Reserved Instances, identify idle resources, or suggest downsizing opportunities, directly within your AI chat. Real-time Pricing Checks: The server enables the AI to fetch current AWS pricing information on demand, helping validate costs or explore pricing for different configurations. In essence, this MCP server transforms how engineers interact with AWS cost data, making analysis more intuitive, proactive, and integrated into their workflow when paired with an AI assistant like Amazon Q.\nQuick Start: How to Install and Run AWS Cost Explorer MCP Server\nPrerequisites\nInstall uv, Python, AWSCli Configure AWS credentials with permissions to access AWS services. Ensure you have: An AWS account with appropriate permissions. AWS credentials configured using aws configure or environment variables. Your IAM role or user must have permission to access the AWS Pricing API. Install AmazonQ Create AWS Builder ID and sign in using AWS Builder ID in AmazonQ Installation Steps\nInstall AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html\nInstall Amazon Q:https://docs.aws.amazon.com/amazonq/latest/qdeveloper-ug/command-line-installing.html#command-line-installing-appimage- Register an AWS Builder ID.\nLogin AmazonQ using the AWS Builder ID.\nSet up MCP server Configuration File: Create a file at local AmazonQ folder with the following content, so we can work with MCP across AWS using defined MCP server and interact with AWS account profiles, like here I defined my own AWS profile \u0026ldquo;zack\u0026rdquo;, The MCP Server will use the AWS profile specified in the AWS_PROFILE environment variable. If this variable is not set, it defaults to the \u0026ldquo;default\u0026rdquo; profile in our AWS configuration, Ensure the AWS profile used has permissions to access the AWS Pricing API.\nvim ~/.aws/amazonq/mcp.json { \u0026#34;mcpServers\u0026#34;: { \u0026#34;awslabs.cost-analysis-mcp-server\u0026#34;: { \u0026#34;command\u0026#34;: \u0026#34;uvx\u0026#34;, \u0026#34;args\u0026#34;: [\u0026#34;awslabs.cost-explorer-mcp-server@latest\u0026#34;], \u0026#34;env\u0026#34;: { \u0026#34;FASTMCP_LOG_LEVEL\u0026#34;: \u0026#34;ERROR\u0026#34;, \u0026#34;AWS_PROFILE\u0026#34;: \u0026#34;zack\u0026#34; }, \u0026#34;disabled\u0026#34;: false, \u0026#34;autoApprove\u0026#34;: [] }, \u0026#34;awslabs.aws-pricing-mcp-server\u0026#34;: { \u0026#34;command\u0026#34;: \u0026#34;uvx\u0026#34;, \u0026#34;args\u0026#34;: [ \u0026#34;awslabs.aws-pricing-mcp-server@latest\u0026#34; ], \u0026#34;env\u0026#34;: { \u0026#34;AWS_PROFILE\u0026#34;: \u0026#34;zack\u0026#34;, \u0026#34;FASTMCP_LOG_LEVEL\u0026#34;: \u0026#34;ERROR\u0026#34; } }, \u0026#34;awslabs.cdk-mcp-server\u0026#34;: { \u0026#34;command\u0026#34;: \u0026#34;uvx\u0026#34;, \u0026#34;args\u0026#34;: [ \u0026#34;awslabs.cdk-mcp-server@latest\u0026#34; ], \u0026#34;env\u0026#34;: { \u0026#34;FASTMCP_LOG_LEVEL\u0026#34;: \u0026#34;ERROR\u0026#34; } }, \u0026#34;awslabs.aws-documentation-mcp-server\u0026#34;: { \u0026#34;command\u0026#34;: \u0026#34;uvx\u0026#34;, \u0026#34;args\u0026#34;: [ \u0026#34;awslabs.aws-documentation-mcp-server@latest\u0026#34; ], \u0026#34;env\u0026#34;: { \u0026#34;FASTMCP_LOG_LEVEL\u0026#34;: \u0026#34;ERROR\u0026#34; }, \u0026#34;disabled\u0026#34;: false, \u0026#34;autoApprove\u0026#34;: [] }, \u0026#34;awslabs.terraform-mcp-server\u0026#34;: { \u0026#34;command\u0026#34;: \u0026#34;uvx\u0026#34;, \u0026#34;args\u0026#34;: [ \u0026#34;awslabs.terraform-mcp-server@latest\u0026#34; ], \u0026#34;env\u0026#34;: { \u0026#34;FASTMCP_LOG_LEVEL\u0026#34;: \u0026#34;ERROR\u0026#34; }, \u0026#34;disabled\u0026#34;: false, \u0026#34;autoApprove\u0026#34;: [] } } } Complete Installation and Start Chatting:\nThe MCP Server uses the specified profile to create a boto3 session for authenticating with AWS services. So our AWS IAM credentials remain local and are only used to access AWS services.\nStart the chat interface:\n\u0026gt;q chat \u0026gt;help me analyze last 2 months spending Output Example:\nSummary\nThe AAWS Cost Explorer MCP Server provides enterprises with an efficient and intelligent solution for cost analysis. Through the standardized MCP protocol, we can easily integrate cost analysis capabilities, enhancing our ability to manage cloud service costs effectively via AI powered AmazonQ and MCP server.\nThink of Amazon Q direct as having a helpful assistant who can look up your basic spending information and trends from Cost Explorer and explain them.\nThink of Amazon Q + MCP Server as giving that assistant access to a specialized financial analyst tool (the MCP server). This tool can perform much deeper dives, connect different data points (like pricing details or infrastructure definitions), generate formal reports, and provide concrete advice on how to save money.\nThe key difference lies in the depth of analysis, report generation, and actionable optimization suggestions provided by the dedicated AWS Cost Explorer MCP Server, which goes beyond the built-in capabilities of Amazon Q alone.\nMore AWS Labs MCP servers to be explored:\nAWS Bedrock Knowledge Base Retrieval MCP server: https://github.com/awslabs/mcp/tree/main/src/bedrock-kb-retrieval-mcp-server AWS Terraform MCP server: https://github.com/awslabs/mcp/tree/main/src/terraform-mcp-server ","permalink":"https://zackblog.work/posts/cost-optimization-with-amazon-q-cost-explorer-mcp-server/","summary":"\u003cp\u003eManaging cloud costs effectively, especially on AWS, is crucial. Wasted resources can easily inflate bills. This post introduces the \u003cstrong\u003eAWS Cost Explorer MCP Server\u003c/strong\u003e, a tool designed to simplify analyzing your AWS spending using the Model Context Protocol (MCP).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eGitHub Repository:\u003c/strong\u003e \u003ca href=\"https://github.com/awslabs/mcp/tree/main/src/cost-explorer-mcp-server\" target=\"_blank\" rel=\"noopener noreferrer\"\u003ehttps://github.com/awslabs/mcp/tree/main/src/cost-explorer-mcp-server\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"/images/mcp1.png\"\u003e\u003cimg alt=\"[Image Placeholder 01: Introduction Graphic]\" loading=\"lazy\" src=\"/images/mcp1.png\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eWhat this AWS Cost Explorer MCP Server Does\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis specific MCP Server, provided by AWS Labs, acts as a specialized tool that connects an AI assistant like Amazon Q directly to the detailed AWS cost and usage data. Think of it as giving us AI assistant the specific knowledge and tools needed to understand and analyze our cloud spending.\u003c/p\u003e","title":"Cost Optimization with Amazon Q \u0026 Cost Explorer MCP Server"},{"content":"In a previous post - MLOps - Build a Oscar Best Picture Winner Model, I was able to establish a baseline model using a RandomForestClassifier to predict the Oscar for Best Picture. This classic workflow involved data cleaning, training, and prediction, providing a solid starting point.\nHowever, a deeper look at the results revealed critical weaknesses that an experienced machine learning engineer would immediately flag:\nInadequate Model Choice: The initial model wasn\u0026rsquo;t powerful enough for the task. The classification report showed a recall of 0.00 for the \u0026ldquo;winner\u0026rdquo; class. This is a major red flag, indicating the model completely failed to identify any actual winners, likely due to the severe class imbalance. Misleading Evaluation Metrics: I think I relied too heavily on accuracy. On an imbalanced dataset, a model can achieve high accuracy simply by always predicting the majority class. Better to shift our focus to more robust metrics like the F1-score, ROC AUC, and Precision-Recall AUC. This analysis led to idea to enhance this Oscar prediction with a more sophisticated LightGBM (LGBM) classifier model, known for its high performance, speed, and efficiency on tabular data.\nStep 1: Installing LightGBM\nFirst, ensure LightGBM library is installed in our environment.\n# Install the LightGBM library !python3 -m pip install lightgbm Step 2: Data Preparation and Advanced Feature Engineering\nI can re-use the previous cleaned dataset and apply transformations for categorical features, and new interaction feature.\nimport pandas as pd import lightgbm as lgb from sklearn.metrics import classification_report, accuracy_score, roc_auc_score import seaborn as sns import matplotlib.pyplot as plt # Load the dataset df = pd.read_csv(\u0026#39;updated_with_changes.csv\u0026#39;) # --- Data Cleaning and Feature Engineering --- # Convert target variable \u0026#39;winner\u0026#39; to integer (1 for winner, 0 for nominee) df[\u0026#39;winner\u0026#39;] = df[\u0026#39;winner\u0026#39;].astype(int) # Convert \u0026#39;Tomatometer\u0026#39; from string (\u0026#39;97%\u0026#39;) to a float (0.97) df[\u0026#39;Tomatometer\u0026#39;] = df[\u0026#39;Tomatometer\u0026#39;].str.replace(\u0026#39;%\u0026#39;, \u0026#39;\u0026#39;, regex=False).astype(float) / 100.0 # Ordinal Encoding for precursor awards award_mapping = {\u0026#39;won\u0026#39;: 2, \u0026#39;nominated\u0026#39;: 1, \u0026#39;none\u0026#39;: 0} df[\u0026#39;GoldenGlobe\u0026#39;] = df[\u0026#39;GoldenGlobe\u0026#39;].map(award_mapping).fillna(0) df[\u0026#39;BAFTAs\u0026#39;] = df[\u0026#39;BAFTAs\u0026#39;].map(award_mapping).fillna(0) # Create a simple interaction feature df[\u0026#39;Critic_Score\u0026#39;] = df[\u0026#39;Metascore\u0026#39;] * df[\u0026#39;Tomatometer\u0026#39;] # Fill any remaining missing values with the column\u0026#39;s median for col in [\u0026#39;imdb_rating\u0026#39;, \u0026#39;Metascore\u0026#39;, \u0026#39;Tomatometer\u0026#39;, \u0026#39;Critic_Score\u0026#39;]: df[col] = df[col].fillna(df[col].median()) Step 3: Chronological Train/Test Split\nFor a time-based problem like Oscar predictions, a random split is inappropriate. I shall use a chronological split, training the model on older ceremonies to predict newer ones. We\u0026rsquo;ll train on ceremonies up to the 90th Academy Awards and validate on all subsequent ceremonies.\n# Define features (X) and target (y) features = [col for col in df.columns if col not in [\u0026#39;winner\u0026#39;, \u0026#39;category\u0026#39;, \u0026#39;film\u0026#39;]] X = df[features] y = df[\u0026#39;winner\u0026#39;] # Split data chronologically train_mask = df[\u0026#39;ceremony\u0026#39;] \u0026lt;= 90 test_mask = df[\u0026#39;ceremony\u0026#39;] \u0026gt; 90 X_train, y_train = X[train_mask], y[train_mask] X_test, y_test = X[test_mask], y[test_mask] print(f\u0026#34;Training data shape: {X_train.shape}\u0026#34;) print(f\u0026#34;Test data shape: {X_test.shape}\u0026#34;) Output\nNegative samples: 112 Positive samples: 19 Scale Pos Weight: 5.89 [LightGBM] [Info] Number of positive: 19, number of negative: 112 [LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.014733 seconds. You can set `force_col_wise=true` to remove the overhead. [LightGBM] [Info] Total Bins 138 [LightGBM] [Info] Number of data points in the train set: 131, number of used features: 7 [LightGBM] [Info] [binary:BoostFromScore]: pavg=0.145038 -\u0026gt; initscore=-1.774060 [LightGBM] [Info] Start training from score -1.774060 [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf [LightGBM] [Warning] No further splits with positive gain, best gain: -inf LightGBM model trained successfully! Analysis\nThis confirms our setup for handling the imbalanced data is working. The model is now aware that there are many more \u0026ldquo;losers\u0026rdquo; than \u0026ldquo;winners\u0026rdquo; and will give about 6 times more importance to learning the patterns of the winners.\nModel also indicated that it has learned as much as it can from a particular branch of a decision tree, I assume this is common with small datasets like this one (only 131 training samples). The model quickly finds the most important patterns and then stops itself from creating overly complex rules that wouldn\u0026rsquo;t apply to new data\nStep 4: Training an Imbalance-Aware LightGBM Model\nThis is the most critical step. To combat class imbalance, we will use the scale_pos_weight parameter in LightGBM. We calculate it as the ratio of negative samples (nominees) to positive samples (winners).\n# Calculate scale_pos_weight for handling class imbalance num_negatives = y_train.value_counts()[0] num_positives = y_train.value_counts()[1] scale_pos_weight_value = num_negatives / num_positives # Initialize and train the LightGBM model lgbm = lgb.LGBMClassifier( objective=\u0026#39;binary\u0026#39;, metric=\u0026#39;auc\u0026#39;, scale_pos_weight=scale_pos_weight_value, # Key parameter for imbalance random_state=42 ) # Train with early stopping to prevent overfitting lgbm.fit(X_train, y_train, eval_set=[(X_test, y_test)], eval_metric=\u0026#39;auc\u0026#39;, callbacks=[lgb.early_stopping(100, verbose=False)]) Output\n--- Classification Report --- precision recall f1-score support 0 0.94 0.92 0.93 49 1 0.43 0.50 0.46 6 accuracy 0.87 55 macro avg 0.68 0.71 0.69 55 weighted avg 0.88 0.87 0.88 55 Accuracy: 0.87 ROC AUC Score: 0.83 --- Feature Importances --- Analysis\nRecall (for class 1): 0.50 means very important number here. I think the model successfully identified 50% of the actual Best Picture winners in the test set. As my previous model likely had a recall of 0 for this class, meaning it never correctly picked a winner.\nStep 5: Evaluating Model Performance\nNow, let\u0026rsquo;s see how our new model performs on the test set, to see if significant improvement can be achieved in identifying the \u0026lsquo;winner\u0026rsquo; class.\n# Predict on the test set y_pred = lgbm.predict(X_test) y_pred_proba = lgbm.predict_proba(X_test)[:, 1] # --- Evaluation Metrics --- print(\u0026#34;--- Classification Report ---\u0026#34;) print(classification_report(y_test, y_pred)) print(f\u0026#34;ROC AUC Score: {roc_auc_score(y_test, y_pred_proba):.2f}\\n\u0026#34;) # --- Feature Importance --- feature_imp = pd.DataFrame(sorted(zip(lgbm.feature_importances_, X.columns)), columns=[\u0026#39;Value\u0026#39;,\u0026#39;Feature\u0026#39;]) plt.figure(figsize=(12, 8)) sns.barplot(x=\u0026#34;Value\u0026#34;, y=\u0026#34;Feature\u0026#34;, data=feature_imp.sort_values(by=\u0026#34;Value\u0026#34;, ascending=False)) plt.title(\u0026#39;LightGBM Feature Importances\u0026#39;) plt.show() The results show a massive improvement. The ROC AUC Score of 0.83 is excellent, indicating a strong ability to distinguish between winners and non-winners.\nThe feature importance plot reveals the model\u0026rsquo;s decision-making logic:\nGoldenGlobe and BAFTAs are at the top: The model learned that winning or being nominated for other major awards is a very strong predictor for the Oscars. This is exactly what we\u0026rsquo;d expect. Critic_Score / Metascore / Tomatometer are next: The model is using critic scores to help make its decision, which is also very logical. imdb_rating likely has some importance, but probably less than the major awards. This plot is crucial because it gives us confidence that the model isn\u0026rsquo;t just guessing. It has learned the real-world patterns that film experts use to make their own predictions\nStep 6: Predicting the 2025 Nominees\nNow for the exciting part: using our trained model to predict the win probabilities for a hypothetical list of 2025 nominees.\n# Example: New data for 2025 nominees nominees_2025 = pd.DataFrame({ \u0026#39;film\u0026#39;: [\u0026#39;Dune: Part Two\u0026#39;, \u0026#39;Conclave\u0026#39;, \u0026#39;The Brutalist\u0026#39;, \u0026#39;Anora\u0026#39;, \u0026#39;Wicked\u0026#39;], \u0026#39;ceremony\u0026#39;: [97, 97, 97, 97, 97], \u0026#39;imdb_rating\u0026#39;: [8.5, 7.4, 7.8, 7.7, 7.6], \u0026#39;Metascore\u0026#39;: [79, 79, 90, 91, 73], \u0026#39;Tomatometer\u0026#39;: [0.92, 0.93, 0.94, 0.94, 0.88], \u0026#39;GoldenGlobe\u0026#39;: [1, 1, 2, 1, 1], # 1=nominated, 2=won \u0026#39;BAFTAs\u0026#39;: [0, 2, 1, 1, 0] # 0=none, 1=nominated, 2=won }) # ... (Create interaction feature and select columns as before) # Predict probabilities and display results win_probabilities = lgbm.predict_proba(X_2025)[:, 1] results_df = pd.DataFrame({ \u0026#39;Film\u0026#39;: nominees_2025[\u0026#39;film\u0026#39;], \u0026#39;Win_Probability\u0026#39;: win_probabilities }).sort_values(by=\u0026#39;Win_Probability\u0026#39;, ascending=False) print(results_df) Prediction Results:\nFilm Win_Probability The Brutalist 58.96% Anora 36.98% Conclave 19.37% Dune: Part Two 11.07% Wicked 8.33% Comparing Predictions to the \u0026ldquo;Actual\u0026rdquo; Winner\nIn March Anora sweeps Oscar 2025\nHowever, I believe this model is still a resounding success. While the model didn\u0026rsquo;t place the winner at the very top, it identified \u0026ldquo;Anora\u0026rdquo; as the second most likely film to win with a strong probability. In the notoriously difficult world of Oscar predictions, this is a testament to the model\u0026rsquo;s effectiveness.\nConclusion\nThis project demonstrates a successful journey from a simple baseline to a robust, explainable machine learning model. The key takeaways are:\nChoose the Right Tool: Switching to a powerful model like LightGBM was crucial. Address Core Problems: Using scale_pos_weight to directly handle class imbalance was the single most important change. Trust, but Verify: Feature importance plots are essential for ensuring your model\u0026rsquo;s logic is sound and not just a black box. Notebooks and dataset are now avaliable at my GitHub repo\n","permalink":"https://zackblog.work/posts/mlops-enhancing-oscar-model-with-lightgbm/","summary":"\u003cp\u003eIn a previous post - \u003ca href=\"/posts/mlops-build-a-oscar-best-picture-winner-model/\"\u003eMLOps - Build a Oscar Best Picture Winner Model\u003c/a\u003e, I was able to establish a baseline model using a \u003ccode\u003eRandomForestClassifier\u003c/code\u003e to predict the Oscar for Best Picture. This classic workflow involved data cleaning, training, and prediction, providing a solid starting point.\u003c/p\u003e\n\u003cp\u003eHowever, a deeper look at the results revealed critical weaknesses that an experienced machine learning engineer would immediately flag:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eInadequate Model Choice:\u003c/strong\u003e The initial model wasn\u0026rsquo;t powerful enough for the task. The classification report showed a \u003cstrong\u003erecall of 0.00 for the \u0026ldquo;winner\u0026rdquo; class\u003c/strong\u003e. This is a major red flag, indicating the model completely failed to identify any actual winners, likely due to the severe class imbalance.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMisleading Evaluation Metrics:\u003c/strong\u003e I think I relied too heavily on accuracy. On an imbalanced dataset, a model can achieve high accuracy simply by always predicting the majority class. Better to shift our focus to more robust metrics like the \u003cstrong\u003eF1-score\u003c/strong\u003e, \u003cstrong\u003eROC AUC\u003c/strong\u003e, and \u003cstrong\u003ePrecision-Recall AUC\u003c/strong\u003e.\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003eThis analysis led to idea to enhance this Oscar prediction with a more sophisticated \u003cstrong\u003eLightGBM (LGBM) classifier\u003c/strong\u003e model, known for its high performance, speed, and efficiency on tabular data.\u003c/p\u003e","title":"MLOPS - Enhancing Oscar Model with LightGBM"},{"content":"In this post, we will continue to build a basic machine learning model to predict the Best Picture winner at the Academy Awards (Oscar).\nWe will use our previous processed dataset that includes information about the nominees and winners from the 72nd to the 96th Oscar ceremonies. The goal is to predict the winner based on various features like IMDb ratings, Metascore, Tomatometer percentage, Golden Globe and BAFTA wins/nominations.\nStep 1: Understand the features and model\nThe dataset includes several features:\nIMDb rating: The IMDb rating of the film. Metascore: The Metascore of the film. Tomatometer: The Tomatometer rating (percentage of positive reviews). Golden Globe/BAFTA nominations/wins: The number of nominations/wins for Golden Globe and BAFTA. Winner: Whether the film won the Best Picture award (True/False). The target variable is the winner column, where \u0026ldquo;True\u0026rdquo; means the film won Best Picture, and \u0026ldquo;False\u0026rdquo; means it did not.\nLogistic Regression for Binary Classification: Logistic regression is a well-known, interpretable model for binary outcomes (win vs. not win). It estimates the probability that a given input belongs to the positive class\nStep 2: Final Data Cleaning\nThe first step is to finally clean and preprocess the data. We will:\nConvert the \u0026ldquo;winner\u0026rdquo; column to a binary format (1 for winner, 0 for not winner). Remove the percentage sign from the \u0026ldquo;Tomatometer\u0026rdquo; column and convert it to a float. Encode the Golden Globe and BAFTA columns as numerical values (won = 2, nominated = 1, none = 0). import pandas as pd # Load data df = pd.read_csv(\u0026#34;updated_with_changes.csv\u0026#34;) # Clean \u0026#39;winner\u0026#39; column df[\u0026#39;winner\u0026#39;] = df[\u0026#39;winner\u0026#39;].astype(int) # Clean \u0026#39;Tomatometer\u0026#39; (remove % and convert to float) df[\u0026#39;Tomatometer\u0026#39;] = df[\u0026#39;Tomatometer\u0026#39;].str.replace(\u0026#39;%\u0026#39;, \u0026#39;\u0026#39;).astype(float) / 100 # Encode GoldenGlobe and BAFTAs award_mapping = {\u0026#39;won\u0026#39;: 2, \u0026#39;nominated\u0026#39;: 1, \u0026#39;none\u0026#39;: 0} df[\u0026#39;GoldenGlobe\u0026#39;] = df[\u0026#39;GoldenGlobe\u0026#39;].str.split().str[0].map(award_mapping).fillna(0) df[\u0026#39;BAFTAs\u0026#39;] = df[\u0026#39;BAFTAs\u0026#39;].map(award_mapping).fillna(0) Step 3: Feature Engineering\nNext, we will create new features to improve the prediction accuracy:\nTotal Awards Score: The sum of Golden Globe and BAFTA awards. # Create Total Awards Score df[\u0026#39;Total_Awards\u0026#39;] = df[\u0026#39;GoldenGlobe\u0026#39;] + df[\u0026#39;BAFTAs\u0026#39;] # Drop unnecessary columns df = df.drop([\u0026#39;ceremony\u0026#39;, \u0026#39;category\u0026#39;, \u0026#39;film\u0026#39;], axis=1) Step 4: Split Data into Training and Testing Sets\nWe will split the dataset into training and testing sets based on the ceremony year. For example, we can use the ceremonies from 1972 to 1990 for training and those from 1991 to 1996 for testing.\n# Assuming \u0026#39;ceremony\u0026#39; column exists (if not, reset index) train = df[df[\u0026#39;ceremony\u0026#39;] \u0026lt;= 90] test = df[df[\u0026#39;ceremony\u0026#39;] \u0026gt; 90] # Separate features (X) and target (y) X_train = train.drop(\u0026#39;winner\u0026#39;, axis=1) y_train = train[\u0026#39;winner\u0026#39;] X_test = test.drop(\u0026#39;winner\u0026#39;, axis=1) y_test = test[\u0026#39;winner\u0026#39;] Step 5: Train our Model\nNow that we have our training and testing data, let\u0026rsquo;s train a Logistic Regression model to predict the Best Picture winner. Logistic Regression is a simple, interpretable model that will help us understand which features matter most in predicting a winner.\nfrom sklearn.linear_model import LogisticRegression # Initialize and train the model model = LogisticRegression() model.fit(X_train, y_train) # Check accuracy on test data accuracy = model.score(X_test, y_test) print(f\u0026#34;Accuracy: {accuracy:.2f}\u0026#34;) Step 6: Evaluate the Model\nAfter training the model, we need to evaluate its performance. We will use metrics such as accuracy, precision, recall, and F1-score to assess the model. Since the dataset might be imbalanced (few winners each year), we also consider ROC-AUC.\nfrom sklearn.metrics import classification_report # Predict on test data y_pred = model.predict(X_test) # Generate evaluation report print(classification_report(y_test, y_pred)) Classification Report:The classification report provides precision, recall, and F1-score for each class. In the output, notice that for class 1 (likely representing the winning films) the model never predicts any positive cases (precision and recall are 0). This suggests that the model is favoring the majority class (class 0)—a common issue when dealing with imbalanced datasets\nStep 7: Predict This Year\u0026rsquo;s Winner\nOnce the model is trained and evaluated, we can use it to predict the Best Picture winner for the current year. The input data for this year\u0026rsquo;s nominees should be in the same format as the training data.\n# Example: New data for 2024 nominees new_data = pd.DataFrame({ \u0026#39;ceremony\u0026#39;: [97, 97, 97, 97, 97, 97, 97, 97, 97, 97], \u0026#39;imdb_rating\u0026#39;: [7.7, 7.8, 7.6, 7.4, 8.5, 5.5, 8.8, 7.2, 7.3, 7.6], \u0026#39;Metascore\u0026#39;: [91, 90, 70, 79, 79, 70, 48, 91, 78, 73], \u0026#39;Tomatometer\u0026#39;: [0.94, 0.94, 0.70, 0.79, 0.79, 0.70, 0.48, 0.91, 0.78, 0.73], \u0026#39;GoldenGlobe\u0026#39;: [1, 2, 1, 1, 1, 2, 1, 1, 1, 1], \u0026#39;BAFTAs\u0026#39;: [1, 1, 1, 2, 0, 1, 0, 0, 0, 0], \u0026#39;Total_Awards\u0026#39;: [2, 3, 2, 3, 1, 3, 1, 1, 1, 1] }) # Predict probabilities probabilities = model.predict_proba(new_data)[:, 1] print(\u0026#34;Win probabilities:\u0026#34;, probabilities) The win probabilities predicted by the model for the 2024 Best Picture nominees are as follows (in order of the films listed in the data):\n[0.016 (1.6%), 0.173 (17.3%), 0.0086 (0.86%), 0.0238 (2.38%), 0.0079 (0.79%), 0.0255 (2.55%), 0.0043 (0.43%), 0.0046 (0.46%), 0.0035 (0.35%), 0.0037 (0.37%)] Key Observations:\nThe Brutalist (17.3%) is the clear favorite according to the model, likely due to:\nGolden Globe win (encoded as 2 in the data, a strong predictor). High scores: Metascore 90, Tomatometer 94%. 3 Total Awards (tied for highest among nominees). Emilia Pérez (2.55%) and Conclave (2.38%) follow distantly, likely because:\nEmilia Pérez won a Golden Globe and has 3 Total Awards (despite a low IMDb rating of 5.5). Conclave won a BAFTA (encoded as 2) and has 3 Total Awards. Dune: Part Two (0.79%) underperforms despite its high IMDb rating (8.5) and box office success because:\nIt lacks major award wins (Golden Globe=1 = nominated, BAFTAs=0 = none). The model prioritizes awards over popularity/critical ratings. Low probabilities overall (summing to ~26.6%) suggest:\nThe model treats each nominee independently (binary classification), not as a competitive multi-class problem. No film strongly matches historical winner profiles, indicating a competitive year. Feature Insights:\nBy following these steps, This model pipeline—from data cleaning to prediction—illustrates a standard supervised learning workflow. Each step is chosen to make sure the input data is well-prepared for the model, the model is appropriately evaluated, and predictions are made with the correct feature set.\nAward wins matter most: Golden Globe/BAFTA wins (2 in the data) heavily influence predictions. Critical acclaim: High Metascore and Tomatometer values boost probabilities (e.g., The Brutalist). IMDb rating is less impactful: Dune’s high IMDb score doesn’t compensate for its lack of awards. Box office ignored: The model excludes box office data, explaining why Wicked (high revenue) has a low probability. Conclusion:\nThe model identifies The Brutalist as the most likely winner due to its award wins and critical acclaim. However, the low probabilities overall suggest uncertainty, possibly reflecting a lack of a dominant frontrunner in this year’s nominees based on historical patterns.\nImprovements (Optional)\nThere are several ways to improve the model:\nHandle class imbalance using techniques like class_weight='balanced'. Try advanced models like Random Forest or XGBoost. Include more data, such as budget, genre, or director popularity. With these steps, maybe we can predict the Best Picture winner and gain insights into what makes a film successful at the Oscars!\n","permalink":"https://zackblog.work/posts/mlops-build-a-oscar-best-picture-winner-model/","summary":"\u003cp\u003eIn this post, we will continue to build a basic machine learning model to predict the \u003cstrong\u003eBest Picture\u003c/strong\u003e winner at the Academy Awards (Oscar).\u003c/p\u003e\n\u003cp\u003eWe will use our previous processed dataset that includes information about the nominees and winners from the 72nd to the 96th Oscar ceremonies. The goal is to predict the winner based on various features like IMDb ratings, Metascore, Tomatometer percentage, Golden Globe and BAFTA wins/nominations.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStep 1: Understand the features and model\u003c/strong\u003e\u003c/p\u003e","title":"MLOps - Build a Oscar Best Picture Winner Model"},{"content":"After downloading a dataset from Kaggle about historical Oscar nominations and winners, it is time to move on to data cleansing and enrichment.\nThe original data contained seven headers, but only a few were useful: the film name, ceremony year, and winner status. Additionally, the dataset spanned from 1927 to 2024 and included all Oscar categories, making it quite complex—perhaps too overwhelming for me as a beginner. So, I decided to focus on Best Picture as it is always the most important award among all the other categories.\nTo enhance the dataset, I believe we need to add supplementary columns such as IMDb ratings, box office performance, Metascore, and Rotten Tomatoes scores. These features provide a diverse set of insights, capturing both audience ratings and trends from movie critics. I also handled missing values to ensure the dataset is clean and ready for the next phase of analysis.\nHere is the steps taken and the techniques used in the process.\nStep 1: Loading the Dataset\nFirst, let\u0026rsquo;s focus on movies after the year 2000, and only focus on:\nACTOR IN A LEADING ROLE ACTRESS IN A LEADING ROLE BEST PICTURE We need to load the original Oscar data from the Kaggle CSV file, filter by year and category, save it into a CSV file, and then define the new CSV in a DataFrame.\n# Filter the data for a specific year (e.g., 2000) subset_data = df[df[\u0026#39;year_ceremony\u0026#39;] \u0026gt;= 2000] # Output the filtered data to a new CSV file output_file_path = \u0026#39;/workspace/oscar/base-year-2000.csv\u0026#39; # Replace with your desired output path subset_data.to_csv(output_file_path, index=False) # index=False prevents writing row numbers # Importing the essential libraries import pandas as pd # Load the dataset file_path = \u0026#39;/workspace/oscar/the_oscar_award.csv\u0026#39; # Adjust the path as necessary df = pd.read_csv(file_path) # Step 1: Filter the data for year 2000 or after subset_data = df[df[\u0026#39;year_ceremony\u0026#39;] \u0026gt;= 2000] # Step 2: Filter to only keep rows where the category is ACTOR IN LEADING ROLE, ACTRESS IN LEADING ROLE, or BEST PICTURE categories_of_interest = [\u0026#39;ACTOR IN A LEADING ROLE\u0026#39;, \u0026#39;ACTRESS IN A LEADING ROLE\u0026#39;, \u0026#39;BEST PICTURE\u0026#39;] subset_data_filtered = subset_data[subset_data[\u0026#39;category\u0026#39;].isin(categories_of_interest)] # Output the filtered data to a new CSV file output_file_path = \u0026#39;/workspace/oscar/base-year-2000-filtered.csv\u0026#39; # Replace with your desired output path subset_data_filtered.to_csv(output_file_path, index=False) # index=False prevents writing row numbers # Display the filtered data import pandas as pd pd.set_option(\u0026#39;display.max_rows\u0026#39;, None) # This ensures that all rows are displayed display(subset_data_filtered) # Display the filtered data # Load the newly saved CSV into a DataFrame output_file_path = \u0026#39;/workspace/oscar/base-year-2000-filtered.csv\u0026#39; # Replace with the path to your output CSV year2000_df = pd.read_csv(output_file_path) # Display the first few rows of the new DataFrame display(year2000_df) # Or you can use print(new_df) if you prefer # Load the newly saved CSV into a DataFrame output_file_path = \u0026#39;/workspace/oscar/base-year-2000-filtered.csv\u0026#39; # Replace with the path to your output CSV year2000_df = pd.read_csv(output_file_path) # Display the first few rows of the new DataFrame display(year2000_df) # Or you can use print(new_df) if you prefer Step 2: Adding IMDb Ratings and Box Office Data\nNow we have a new csv ready, lets add new supplementary data into it.\nTo fetch IMDb ratings and box office data, we need IMDbPY library for IMDb ratings and the OMDb API for box office data. OMDb API can be obtained via OMDbAPI website, then use a loop to fetch the data for each movie, add a delay to avoid overwhelming the servers with requests.\n# install library IMDbPY and requests !pip install IMDbPY !pip install requests # import pandas as pd from imdb import Cinemagoer import time import requests # Create Cinemagoer object ia = Cinemagoer() # Define OMDb API key (you can get a free API key at https://www.omdbapi.com/) OMDB_API_KEY = \u0026#39;2xxxxxxe\u0026#39; # Read CSV file year2000_df = pd.read_csv(\u0026#39;/workspace/oscar/base-year-2000-filtered.csv\u0026#39;) # Replace with your file path # Create new columns for IMDb ratings and box office year2000_df[\u0026#39;imdb_rating\u0026#39;] = None year2000_df[\u0026#39;box_office\u0026#39;] = None # Set to track movies we\u0026#39;ve already processed processed_movies = set() # Function to fetch box office data from OMDb API def get_box_office(movie_name): try: # Request movie data from OMDb API response = requests.get(f\u0026#34;http://www.omdbapi.com/?t={movie_name}\u0026amp;apikey={OMDB_API_KEY}\u0026#34;) data = response.json() if data[\u0026#39;Response\u0026#39;] == \u0026#39;True\u0026#39;: return data.get(\u0026#39;BoxOffice\u0026#39;, None) # Return BoxOffice value if available else: print(f\u0026#34;Error fetching box office for {movie_name}: {data.get(\u0026#39;Error\u0026#39;, \u0026#39;Unknown error\u0026#39;)}\u0026#34;) return None except Exception as e: print(f\u0026#34;Error fetching box office for {movie_name}: {e}\u0026#34;) return None # Iterate through each film in year2000_df DataFrame for index, row in year2000_df.iterrows(): movie_name = row[\u0026#39;film\u0026#39;] # Check if movie_name is valid (not NaN or empty) if not isinstance(movie_name, str) or not movie_name.strip(): print(f\u0026#34;Skipping invalid movie name at index {index}: {movie_name}\u0026#34;) continue # Skip empty or invalid movie names # Skip duplicate movies (those already processed) if movie_name in processed_movies: print(f\u0026#34;Skipping already processed movie: {movie_name}\u0026#34;) continue # Skip already processed movies try: # Search for the movie movies = ia.search_movie(movie_name) if not movies: print(f\u0026#34;No results found for: {movie_name}\u0026#34;) continue # Get first result and fetch movie details movie = ia.get_movie(movies[0].movieID) # Extract IMDb rating if available if \u0026#39;rating\u0026#39; in movie.keys(): year2000_df.at[index, \u0026#39;imdb_rating\u0026#39;] = movie[\u0026#39;rating\u0026#39;] print(f\u0026#34;Found rating {movie[\u0026#39;rating\u0026#39;]} for {movie_name}\u0026#34;) else: print(f\u0026#34;No rating available for: {movie_name}\u0026#34;) # Fetch box office data from OMDb API box_office = get_box_office(movie_name) if box_office: year2000_df.at[index, \u0026#39;box_office\u0026#39;] = box_office print(f\u0026#34;Found box office {box_office} for {movie_name}\u0026#34;) else: print(f\u0026#34;No box office data available for: {movie_name}\u0026#34;) # Mark this movie as processed processed_movies.add(movie_name) except Exception as e: print(f\u0026#34;Error processing {movie_name}: {str(e)}\u0026#34;) # Add delay to prevent rate limiting time.sleep(1) # Be nice to IMDb\u0026#39;s servers # Save updated DataFrame to a new CSV year2000_df.to_csv(\u0026#39;/workspace/oscar/updated_movies_with_ratings_and_box_office.csv\u0026#39;, index=False) # Display the first few rows of the updated DataFrame year2000_df.head() # You can also use display() in Jupyter if needed Step 3: Dealing with skipped Values\nAfter adding new data, we need to address missing values. The previous cell skipped values if the films is duplicated, so in our case we need to fill them using the first available value within the group of the same film name.\nimport pandas as pd # Load the CSV file into a DataFrame (update the filename as needed) df = pd.read_csv(\u0026#39;/workspace/oscar/updated_movies_with_ratings_and_box_office.csv\u0026#39;) # Replace empty strings with NaN (if missing values are stored as empty strings) df[\u0026#39;imdb_rating\u0026#39;] = df[\u0026#39;imdb_rating\u0026#39;].replace(\u0026#39;\u0026#39;, pd.NA) df[\u0026#39;box_office\u0026#39;] = df[\u0026#39;box_office\u0026#39;].replace(\u0026#39;\u0026#39;, pd.NA) # Define a helper function to fill missing values within each group def fill_missing(series): non_missing = series.dropna() if not non_missing.empty: # Fill all missing values in the series with the first non-missing value return series.fillna(non_missing.iloc[0]) return series # Assuming the film name column is named \u0026#39;film\u0026#39;; change this if necessary. df[\u0026#39;imdb_rating\u0026#39;] = df.groupby(\u0026#39;film\u0026#39;)[\u0026#39;imdb_rating\u0026#39;].transform(fill_missing) df[\u0026#39;box_office\u0026#39;] = df.groupby(\u0026#39;film\u0026#39;)[\u0026#39;box_office\u0026#39;].transform(fill_missing) # Optionally, save the updated DataFrame to a new CSV file df.to_csv(\u0026#39;updated_movies_with_ratings_and_box_office1.csv\u0026#39;, index=False) print(\u0026#34;Missing values for \u0026#39;imdb_ranting\u0026#39; and \u0026#39;box_office\u0026#39; have been filled based on duplicate film entries.\u0026#34;) Step 4: Adding Supplementary Data from Metacritic and Rotten Tomatoes\nCritical \u0026amp; Audience Reception also play important role in Oscar winner, to further enrich the dataset, Let\u0026rsquo;s add additional movie ratings such as Metascore and Rotten Tomatoes scores by calling the OMDb API for each film. These ratings were fetched and added as new columns in the dataset.\nI also decide the drop the box offce as the Oscar academy tends to prioritize artistic and cinematic excellence over commercial success.\nimport pandas as pd import requests API_KEY = \u0026#34;2121a3ae\u0026#34; # Replace with your actual API key def get_metascore_from_omdb(title): url = f\u0026#34;http://www.omdbapi.com/?t={title}\u0026amp;apikey={API_KEY}\u0026#34; try: response = requests.get(url) if response.status_code == 200: data = response.json() return data.get(\u0026#34;Metascore\u0026#34;, \u0026#34;N/A\u0026#34;) except Exception as e: print(f\u0026#34;Error fetching data for {title}: {e}\u0026#34;) return \u0026#34;N/A\u0026#34; # Load your CSV (ensure it has a \u0026#39;film\u0026#39; column) df = pd.read_csv(\u0026#34;/workspace/oscar/updated_movies_with_ratings_and_box_office1.csv\u0026#34;) df[\u0026#34;Metascore\u0026#34;] = df[\u0026#34;film\u0026#34;].apply(get_metascore_from_omdb) df.to_csv(\u0026#34;movies_with_metascore.csv\u0026#34;, index=False) import pandas as pd import requests API_KEY = \u0026#34;2121a3ae\u0026#34; # Replace with your actual API key def get_rotten_tomatoes_rating(title): url = f\u0026#34;http://www.omdbapi.com/?t={title}\u0026amp;apikey={API_KEY}\u0026#34; try: response = requests.get(url) if response.status_code == 200: data = response.json() ratings = data.get(\u0026#34;Ratings\u0026#34;, []) for rating in ratings: if rating.get(\u0026#34;Source\u0026#34;) == \u0026#34;Rotten Tomatoes\u0026#34;: return rating.get(\u0026#34;Value\u0026#34;) except Exception as e: print(f\u0026#34;Error fetching data for {title}: {e}\u0026#34;) return \u0026#34;N/A\u0026#34; # Load your CSV (ensure it has a \u0026#39;film\u0026#39; column) df = pd.read_csv(\u0026#34;/workspace/oscar/movies_with_metascore.csv\u0026#34;) # Apply the function to get Rotten Tomatoes rating df[\u0026#34;Tomatometer\u0026#34;] = df[\u0026#34;film\u0026#34;].apply(get_rotten_tomatoes_rating) # Save the updated dataframe to a new CSV file df.to_csv(\u0026#34;movies_with_metascore_and_tomatometer.csv\u0026#34;, index=False) Step 5: Data Processing and Dropping\nIn this stages, I filtered the data for the \u0026ldquo;BEST PICTURE\u0026rdquo; category, dropped unnecessary columns, and performed additional check for counting missing values, Data types of each column and Summary statistics for numerical columns\nimport pandas as pd # Create a DataFrame from the data df = pd.read_csv(\u0026#34;/workspace/oscar/movies_with_metascore_and_tomatometer.csv\u0026#34;) # Filter rows where category is \u0026#34;BEST PICTURE\u0026#34; df_best_picture = df[df[\u0026#34;category\u0026#34;] == \u0026#34;BEST PICTURE\u0026#34;] # Check if columns exist before dropping columns_to_drop = [\u0026#34;year_film\u0026#34;, \u0026#34;year_ceremony\u0026#34;, \u0026#34;box_office\u0026#34;, \u0026#34;name\u0026#34;] columns_existing = [col for col in columns_to_drop if col in df.columns] # Drop the columns that exist df_best_picture_cleaned = df_best_picture.drop(columns=columns_existing) # Save the cleaned DataFrame to a CSV file output_file_path = \u0026#39;cleaned_best_picture_2000.csv\u0026#39; df_best_picture_cleaned.to_csv(output_file_path, index=False) # Display the path to the saved CSV print(f\u0026#34;The cleaned CSV has been saved to: {output_file_path}\u0026#34;) import pandas as pd # Load the cleaned data df = pd.read_csv(\u0026#34;/workspace/oscar/cleaned_best_picture_2000.csv\u0026#34;) # Check for missing values in each column missing_values = df.isnull().sum() # Display the missing values count for each column print(\u0026#34;Missing values count for each column:\u0026#34;) print(missing_values) # Check the data types of each column print(\u0026#34;\\nData types of each column:\u0026#34;) print(df.dtypes) # Get summary statistics for numerical columns print(\u0026#34;\\nSummary statistics for numerical columns:\u0026#34;) print(df.describe()) # Get counts for categorical columns print(\u0026#34;\\nCounts for categorical columns:\u0026#34;) print(df[\u0026#39;category\u0026#39;].value_counts()) Output: Missing values count for each column: ceremony 0 category 0 film 0 winner 0 imdb_rating 1 Metascore 2 Tomatometer 4 dtype: int64 Data types of each column: ceremony int64 category object film object winner bool imdb_rating float64 Metascore float64 Tomatometer object dtype: object Summary statistics for numerical columns: ceremony imdb_rating Metascore count 186.000000 185.000000 184.000000 mean 85.650538 7.683784 81.385870 std 6.908000 0.480100 10.725488 min 72.000000 5.000000 41.000000 25% 81.000000 7.400000 76.000000 50% 86.000000 7.700000 83.000000 75% 91.750000 8.000000 89.000000 max 96.000000 9.000000 100.000000 Counts for categorical columns: BEST PICTURE 186 Name: category, dtype: int64 Step 6: Data Filtering and Cleanup\nIn this stages, I performed additional updates for specific films like \u0026ldquo;Precious\u0026rdquo;, \u0026ldquo;Moulin Rouge\u0026rdquo; to add missing values. The cleaned dataset was then saved as a new CSV for further analysis.\nimport pandas as pd # Update the information for the film \u0026#34;Precious: Based on the Novel \u0026#39;Push\u0026#39; by Sapphire\u0026#34; df.loc[df[\u0026#39;film\u0026#39;] == \u0026#34;Precious: Based on the Novel \u0026#39;Push\u0026#39; by Sapphire\u0026#34;, \u0026#39;film\u0026#39;] = \u0026#34;Precious\u0026#34; df.loc[df[\u0026#39;film\u0026#39;] == \u0026#34;Precious\u0026#34;, \u0026#39;imdb_rating\u0026#39;] = 7.3 df.loc[df[\u0026#39;film\u0026#39;] == \u0026#34;Precious\u0026#34;, \u0026#39;Tomatometer\u0026#39;] = \u0026#34;97%\u0026#34; # Store as a string with the \u0026#39;%\u0026#39; symbol df.loc[df[\u0026#39;film\u0026#39;] == \u0026#34;Precious\u0026#34;, \u0026#39;Metascore\u0026#39;] = 78 # Save the updated DataFrame to a new CSV file df.to_csv(\u0026#34;/workspace/oscar/cleaned_best_picture_2000.csv\u0026#34;, index=False) # Check the updated information print(df[df[\u0026#39;film\u0026#39;] == \u0026#34;Precious\u0026#34;]) # Update information for multiple films # Moulin Rouge - Update Metascore df.loc[df[\u0026#39;film\u0026#39;] == \u0026#34;Moulin Rouge\u0026#34;, \u0026#39;Metascore\u0026#39;] = 66 # Don\u0026#39;t Look Up - Update Tomatometer to 56% df.loc[df[\u0026#39;film\u0026#39;] == \u0026#34;Don\u0026#39;t Look Up\u0026#34;, \u0026#39;Tomatometer\u0026#39;] = \u0026#34;56%\u0026#34; # Store as a string with the \u0026#39;%\u0026#39; symbol # Tár - Update Tomatometer to 91% df.loc[df[\u0026#39;film\u0026#39;] == \u0026#34;Tár\u0026#34;, \u0026#39;Tomatometer\u0026#39;] = \u0026#34;91%\u0026#34; # Store as a string with the \u0026#39;%\u0026#39; symbol # Maestro - Update Tomatometer to 78% df.loc[df[\u0026#39;film\u0026#39;] == \u0026#34;Maestro\u0026#34;, \u0026#39;Tomatometer\u0026#39;] = \u0026#34;78%\u0026#34; # Store as a string with the \u0026#39;%\u0026#39; symbol # Save the updated DataFrame to a new CSV file df.to_csv(\u0026#34;/workspace/oscar/updated_best_picture_2000.csv\u0026#34;, index=False) # Check the updated information print(df[df[\u0026#39;film\u0026#39;].isin([\u0026#34;Moulin Rouge\u0026#34;, \u0026#34;Don\u0026#39;t Look Up\u0026#34;, \u0026#34;Tár\u0026#34;, \u0026#34;Maestro\u0026#34;])]) Step 7: Update Golden Globe and BAFTA data\nIn the final stages, I think we better add Golden Globe and BAFTA against the Oscar nonimation and winners as they can be another strong predictor and often aligns with Oscar winners.\nThis Part I need help from DeepSeek R1, really impresive result 100% beat OpenAI o3 mini high.\nStep 8: Final thought\nSo far, I believe we have added enough data from the following sources:\nIMDb Metacritic Rotten Tomatoes Golden Globe BAFTA These factors help capture historical trends, industry patterns, and key influences on voting decisions that may impact Oscar winners.\nConclusion\nThis was not an easy task for someone without a data analytics background like me. It took me two days to complete the data processing, especially when I initially tried web scraping from Metacritic and Rotten Tomatoes before realizing that the data could be fetched via an API. However, overall, this was a great experience.\nThis process has significantly enhanced the original Oscar dataset by adding missing movie ratings, box office data, and filtering it to focus on \u0026ldquo;Best Picture\u0026rdquo; films. The dataset is now ready for more in-depth analysis, such as exploring correlations between ratings and box office performance or identifying trends in Oscar nominations and wins over time.\n","permalink":"https://zackblog.work/posts/mlops-data-processing-for-oscar-winner-model/","summary":"\u003cp\u003eAfter downloading a dataset from Kaggle about historical Oscar nominations and winners, it is time to move on to data cleansing and enrichment.\u003c/p\u003e\n\u003cp\u003eThe original data contained seven headers, but only a few were useful: the film name, ceremony year, and winner status. Additionally, the dataset spanned from 1927 to 2024 and included all Oscar categories, making it quite complex—perhaps too overwhelming for me as a beginner. So, I decided to focus on \u003cstrong\u003eBest Picture\u003c/strong\u003e as it is always the most important award among all the other categories.\u003c/p\u003e","title":"MLOps - Data Processing for Oscar Winner Model"},{"content":"The Idea\nThe 97th Academy Awards ceremony, presented by the Academy of Motion Picture Arts and Sciences (AMPAS), will take place on March 2, 2025, at the Dolby Theatre in Hollywood, Los Angeles. Last year 2024 I had some greate experience with some great movies like Dune Part2 and Wicked, not sure if some of my faviourate actors can grab a Oscar.\nso Why not go and use historical Oscar nominations dataset for the past 20 years, to create and train a model by feeding categories and results, to predict the winners each year, then input this years\u0026rsquo; nominations to get a prediction ??\nPlanning for the Oscar prediction model The final goal is building an ML model that predicts Oscar winners using historical data and additional features like box office numbers, IMDb scores, and other award nominations. Let\u0026rsquo;s get started with:\nProject Objective: Predict Oscar winners from historical data. Data Collection: Use Kaggle’s CSV and include extra data like box office, other awards, and IMDb scores. Data Processing: Clean and integrate data, standardize columns, feature engineering. Exploratory Data Analysis (EDA): Visualize data and analyze correlations for winning prediction. Model Design: Build and train models like logistic regression, random forest, or neural networks. Model Deployment: Deploy the model locally or on the cloud to serve API input. Prepare the Local development Environment: As I already have the Local GPU desktop with portable Python env and Dockerized Jupyter Notebook, easy.\nData Sources: I went to Kaggle to download a Historical Oscars CSV file containing nominations, categories, and win/loss outcomes from the 1st Oscar up to the 97th in 2024.\nHowever, this version only has 7 columns covering basic information such as the year and category. I think I will need a way to obtain supplementary data like box office figures, IMDb scores, and other major awards/nominations to achieve better accuracy.\nImport libraries and load base dataset: Let\u0026rsquo;s assess columns available in the CSV to gain a basic understanding of what we have now.\n# Importing the essential libraries import pandas as pd import numpy as np import matplotlib.pyplot as plt import seaborn as sns ## Load the dataset file_path = \u0026#39;/workspace/oscar/the_oscar_award.csv\u0026#39; # Adjust the path as necessary df = pd.read_csv(file_path) # Check the first few rows of the dataset df.head() Understand the base dataset and run some basic analysis: Let’s now gather some basic information about the dataset to understand its structure, including the column names, data types, and missing values\n# Get basic information about the dataset df.info() # Get descriptive statistics (for numerical columns) df.describe() # Check for missing values df.isnull().sum() Plot a pie chart of winners vs non-winners Group the data by the year of the ceremony and count the nominations\n# Group the data by the year of the ceremony and count the nominations yearly_counts = df.groupby(\u0026#39;year_ceremony\u0026#39;).size() yearly_counts.plot(kind=\u0026#39;line\u0026#39;, figsize=(10,6)) plt.title(\u0026#39;Number of Nominations Over the Years\u0026#39;) plt.xlabel(\u0026#39;Year\u0026#39;) plt.ylabel(\u0026#39;Number of Nominations\u0026#39;) plt.show() Next we display basic statistical summary of numerical columns and Check unique values in the \u0026lsquo;category\u0026rsquo; and \u0026lsquo;winner\u0026rsquo; columns\n# Display basic statistical summary of numerical columns print(df.describe()) # Check unique values in the \u0026#39;category\u0026#39; and \u0026#39;winner\u0026#39; columns print(df[\u0026#39;category\u0026#39;].value_counts()) print(df[\u0026#39;winner\u0026#39;].value_counts()) Summary and Next step: In this part, we have completed the design and planning of the idea, set up the local environment, and downloaded the Kaggle base dataset for a basic understanding of the data.\nNext step, I will need to run some data processing to:\nFind a way to scrape some supplementary datasets like box office revenue, other awards/nominations, and IMDb ratings. Inspect the CSV: Check for missing values, inconsistencies, and data types. Data Cleaning: Handle missing or erroneous entries and standardize movie titles and date formats. Merge the supplementary datasets with the base CSV using common identifiers. Initiate Exploratory Data Analysis (EDA) to identify how features like box office revenue or IMDb score correlate with winning. Visualize feature relationships (scatter plots, heatmaps for correlations) to determine which features might be most predictive. ","permalink":"https://zackblog.work/posts/mlops-how-about-predict-oscar-winner/","summary":"\u003cp\u003e\u003cstrong\u003eThe Idea\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe 97th Academy Awards ceremony, presented by the Academy of Motion Picture Arts and Sciences (AMPAS), will take place on March 2, 2025, at the Dolby Theatre in Hollywood, Los Angeles.\nLast year 2024 I had some greate experience with some great movies like Dune Part2 and Wicked, not sure if some of my faviourate actors can grab a Oscar.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"/images/mlops120.png\"\u003e\u003cimg alt=\"image tooltip here\" loading=\"lazy\" src=\"/images/mlops120.png\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eso Why not go and use historical Oscar nominations dataset for the past 20 years, to create and train a model by feeding categories and results, to predict the winners each year, then input this years\u0026rsquo; nominations to get a prediction ??\u003c/p\u003e","title":"MLOps - How About Predict Oscar Winner"},{"content":"Here\u0026rsquo;s a summary of the actions I took to delete an AWS account using an open source tool aws-nuke as it is at the end of free tier I donot need it anymore to aviod cost:\n1. Install aws-nuke:\napt install aws-nuke This command installs aws-nuke, a tool used to delete AWS resources in an account. It requires AWS credentials and the correct configuration file to specify what resources to remove.\n2. Create a nuke-config.yaml, which will be used during execution:\nroot@zackz:/mnt/f/aws-nuke# vim nuke-config.yaml regions: - ap-southeast-2 - global account-blocklist: - \u0026#34;85xxxxxxxx42\u0026#34; # production account ID accounts: \u0026#34;85xxxxxxxx15\u0026#34;: # Joe Account to be deleted filters: IAMUser: - \u0026#34;nuke\u0026#34; # the user excluded from deletion action IAMUserPolicyAttachment: - \u0026#34;nuke -\u0026gt; AdministratorAccess\u0026#34; IAMUserAccessKey: - \u0026#34;nuke -\u0026gt; AKIA4MTWMCWF7T6HLYXE\u0026#34; regions: Specifies which AWS regions the aws-nuke tool should scan. account-blocklist: Ensures that certain accounts (like the production account) are not accidentally nuked. accounts: Lists the accounts to be nuked and specifies any filters for exclusions (like the user \u0026ldquo;nuke\u0026rdquo; in this case). Filters prevent specific resources, like users or keys, from being deleted. I execluded a user nuke so it can perform the remove action.\n3. Ensure AWS Credentials:\nBefore running the command, make sure the AWS CLI is configured with credentials (using aws configure or a profile). I tested the credentials with:\nroot@zackz:/mnt/f/aws-nuke# ls -l ~/.aws/ total 8 drwxrwxrwx 1 root root 512 Oct 11 16:24 amazonq drwxrwxrwx 1 root root 512 Sep 5 14:11 cli -rwxrwxrwx 1 root root 6617 Feb 18 18:44 config -rwxrwxrwx 1 root root 142 Feb 18 14:49 credentials drwxrwxrwx 1 root root 512 Sep 5 14:11 sso aws iam list-account-aliases --profile joe This command checks if the alias for the joe account exists. If no alias exists, you must create one.\n4. Create an Account Alias:\naws iam create-account-alias --account-alias aws-joe-85xxxxxxxx15 --profile joe AWS requires an account alias for the aws-nuke process. Since my account did not have an alias, I just created one. The alias is important for aws-nuke because it checks the alias before proceeding with deletion to prevent accidentally deleting important accounts. If an alias is already taken, you need to choose a different one.\n5. Run aws-nuke with Dry-Run:\naws-nuke --profile joe --config /mnt/f/aws-nuke/nuke-config.yaml aws-nuke version unknown - unknown - unknown Do you really want to nuke the account with the ID 85xxxxxxxx15 and the alias \u0026#39;aws-joe-85xxxxxxxx15\u0026#39;? Do you want to continue? Enter account alias to continue. \u0026gt; aws-joe-85xxxxxxxx15 ap-southeast-2 - EC2Instance - i-05xxxxxxxxxx7efc - [Identifier: \u0026#34;i-05xxxxxxxxxxefc\u0026#34;, ImageIdentifier: \u0026#34;ami-09c8d5d747253fb7a\u0026#34;, InstanceState: \u0026#34;running\u0026#34;, InstanceType: \u0026#34;t2.micro\u0026#34;, LaunchTime: \u0026#34;2024-03-17T08:51:40Z\u0026#34;, tag:Application: \u0026#34;bb\u0026#34;, tag:Businessunit: \u0026#34;dd\u0026#34;, tag:Dataclassification: \u0026#34;aa\u0026#34;, tag:Environment: \u0026#34;cc\u0026#34;, tag:MSP-managed: \u0026#34;ee\u0026#34;, tag:Name: \u0026#34;joe-site\u0026#34;] - would remove global - IAMRolePolicyAttachment - AWSServiceRoleForTrustedAdvisor -\u0026gt; AWSTrustedAdvisorServiceRolePolicy - [PolicyArn: \u0026#34;arn:aws:iam::aws:policy/aws-service-role/AWSTrustedAdvisorServiceRolePolicy\u0026#34;, PolicyName: \u0026#34;AWSTrustedAdvisorServiceRolePolicy\u0026#34;, RoleName: \u0026#34;AWSServiceRoleForTrustedAdvisor\u0026#34;] - cannot detach from service roles global - IAMRolePolicyAttachment - k8s-master-role -\u0026gt; k8s-ec2-master-policy - [PolicyArn: \u0026#34;arn:aws:iam::85xxxxxxxx15:policy/k8s-ec2-master-policy\u0026#34;, PolicyName: \u0026#34;k8s-ec2-master-policy\u0026#34;, RoleName: \u0026#34;k8s-master-role\u0026#34;] - would remove global - IAMRolePolicyAttachment - k8s-worker-role -\u0026gt; k8s-worker-policy - [PolicyArn: \u0026#34;arn:aws:iam::85xxxxxxxx15:policy/k8s-worker-policy\u0026#34;, PolicyName: \u0026#34;k8s-worker-policy\u0026#34;, RoleName: \u0026#34;k8s-worker-role\u0026#34;] - would remove Scan complete: 126 total, 52 nukeable, 74 filtered. The above resources would be deleted with the supplied configuration. Provide --no-dry-run to actually destroy resources. Dry-run allows you to see which resources will be deleted without actually performing the deletion. This is crucial for verifying the configuration before executing the destructive operation.\n6. Confirm the Resources to be Nuked:\nDuring the dry run, aws-nuke listed resources like EC2 instances, IAM roles, policies, and security groups that would be removed. You can review the list to ensure that only the intended resources are being marked for deletion.\n7. Run aws-nuke with \u0026ndash;no-dry-run to Actually Delete:\nroot@zackz:/mnt/f/aws-nuke# aws-nuke --profile joe --config /mnt/f/aws-nuke/nuke-config.yaml --no-dry-run aws-nuke version unknown - unknown - unknown Do you really want to nuke the account with the ID 85xxxxxxxx15 and the alias \u0026#39;aws-joe-85xxxxxxxx15\u0026#39;? Do you want to continue? Enter account alias to continue. \u0026gt; aws-joe-85xxxxxxxx15 ERRO[0014] Listing CloudSearchDomain failed: NotAuthorized: New domain creation not supported on this account. Please reach out to AWS Support for assistance. status code: 401, request id: c46572cb-2f72-41d7-ad7c-64ebe2dfb41a ap-southeast-2 - EC2Instance - i-052b0511339457efc - [Identifier: \u0026#34;i-052b0511339457efc\u0026#34;, ImageIdentifier: \u0026#34;ami-09c8d5d747253fb7a\u0026#34;, InstanceState: \u0026#34;running\u0026#34;, InstanceType: \u0026#34;t2.micro\u0026#34;, LaunchTime: \u0026#34;2024-03-17T08:51:40Z\u0026#34;, tag:Application: \u0026#34;bb\u0026#34;, tag:Businessunit: \u0026#34;dd\u0026#34;, tag:Dataclassification: \u0026#34;aa\u0026#34;, tag:Environment: \u0026#34;cc\u0026#34;, tag:MSP-managed: \u0026#34;ee\u0026#34;, tag:Name: \u0026#34;joe-site\u0026#34;] - would remove ap-southeast-2 - CloudWatchEventsTarget - Rule: AutoScalingManagedRule Target ID: autoscaling - would remove ap-southeast-2 - EC2InternetGatewayAttachment - igw-0df8477f1aab5f7c2 -\u0026gt; vpc-0f5edc76a16636145 - [] - would remove ap-southeast-2 - EC2SecurityGroup - sg-087d7e1df8bf8197a - [Name: \u0026#34;launch-wizard-2\u0026#34;] - triggered remove ap-southeast-2 - EC2SecurityGroup - sg-089842a753c9309bb - [Name: \u0026#34;blog-sg\u0026#34;] - triggered remove ap-southeast-2 - EC2Subnet - subnet-04dc2cab029adbd46 - [DefaultForAz: \u0026#34;true\u0026#34;] - triggered remove ap-southeast-2 - EC2RouteTable - rtb-098b56bf20b6d2f97 - [] - failed ap-southeast-2 - EC2VPC - vpc-0f5edc76a16636145 - [ID: \u0026#34;vpc-0f5edc76a16636145\u0026#34;, IsDefault: \u0026#34;true\u0026#34;] - failed ap-southeast-2 - EC2InternetGatewayAttachment - igw-0df8477f1aab5f7c2 -\u0026gt; vpc-0f5edc76a16636145 - [] - triggered remove ap-southeast-2 - EC2DHCPOption - dopt-0f0b2ab768476bd8e - [] - failed ap-southeast-2 - EC2InternetGateway - igw-0df8477f1aab5f7c2 - [] - triggered remove global - IAMGroup - z101_admin_group - triggered remove global - IAMVirtualMFADevice - arn:aws:iam::85xxxxxxxx15:mfa/z101-joe - failed global - IAMPolicy - arn:aws:iam::85xxxxxxxx15:policy/k8s-worker-policy - [ARN: \u0026#34;arn:aws:iam::85xxxxxxxx15:policy/k8s-worker-policy\u0026#34;, Name: \u0026#34;k8s-worker-policy\u0026#34;, Path: \u0026#34;/\u0026#34;, PolicyID: \u0026#34;ANPA4MTWMCWFXWYIWRI7S\u0026#34;] - triggered remove global - IAMPolicy - arn:aws:iam::85xxxxxxxx15:policy/k8s-ec2-master-policy - [ARN: \u0026#34;arn:aws:iam::85xxxxxxxx15:policy/k8s-ec2-master-policy\u0026#34;, Name: \u0026#34;k8s-ec2-master-policy\u0026#34;, Path: \u0026#34;/\u0026#34;, PolicyID: \u0026#34;ANPA4MTWMCWFZY4I4LD2S\u0026#34;] - triggered remove global - IAMUser - BobJ - triggered remove global - IAMUser - infra_team_user - triggered remove global - IAMUser - joe - triggered remove global - IAMUser - MattS - triggered remove global - IAMUser - ZackZ - triggered remove global - IAMRole - k8s-master-role - [Name: \u0026#34;k8s-master-role\u0026#34;, Path: \u0026#34;/\u0026#34;] - triggered remove global - IAMRole - k8s-worker-role - [Name: \u0026#34;k8s-worker-role\u0026#34;, Path: \u0026#34;/\u0026#34;] - triggered remove Removal requested: 15 waiting, 2 failed, 78 skipped, 2 finished The --no-dry-run flag actually trigger and performs the deletion, removing the specified resources from the AWS account. You were prompted to confirm the deletion by entering the account alias (aws-joe-85xxxxxxxx15) twice to prevent accidental nuking.\n8. Encountered Errors and Failures:\nDuring the final run, I encountered one error:\nFailed Removal of one Resources: Resources such as IAMVirtualMFADevice failed after 3 times attempts due to issues like the device being in use, so maybe we need to remove all MFADevice configuration before we start nuke an account. 9. Once close account, you will not be able to console login anymore :\nKey Takeaways:\nWhy Create an Account Alias: AWS requires an account alias for safety purposes. This prevents accidental deletions by ensuring that only the intended account is nuked. Why Use Filters: Filters prevent critical resources (like specific IAM users or access keys) from being deleted. Dry Run Is Crucial: Always perform a dry run to verify what resources are marked for deletion, after nuke, there will be only the last user in the filter and the IAMVirtualMFADevice left. After aws-nuke: You can see from above, many default resources had been already deleted, which make the account unable to be reused, so this is a tool best for account resource final clean before close . ","permalink":"https://zackblog.work/posts/destruct-aws-account-using-aws-nuke/","summary":"\u003cp\u003eHere\u0026rsquo;s a summary of the actions I took to delete an AWS account using an open source tool \u003ccode\u003eaws-nuke\u003c/code\u003e as it is at the end of free tier I donot need it anymore to aviod cost:\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e1. Install \u003ca href=\"https://github.com/ekristen/aws-nuke/blob/main/docs/quick-start.md\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eaws-nuke\u003c/a\u003e:\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eapt install aws-nuke\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eThis command installs \u003ccode\u003eaws-nuke\u003c/code\u003e, a tool used to delete AWS resources in an account. It requires AWS credentials and the correct configuration file to specify what resources to remove.\u003c/p\u003e","title":"Destruct AWS account using AWS-Nuke"},{"content":"Ray is an open-source framework designed for scalable and distributed ML workloads, including training, tuning, and inference. It provides a simple API for scaling Python applications across multiple nodes.\nDistributed Training: Easily scale PyTorch, TensorFlow, and other ML jobs across multiple GPUs/instances. Hyperparameter Tuning: Integrates with Optuna and Ray Tune for distributed hyperparameter optimization. Parallel Inference: Supports inference pipelines that scale out dynamically based on demand. Fault Tolerance: If a node fails, Ray can reschedule tasks on other available nodes. Together with EKS, Karpenter, and Ray, a modern ML team can achieve dynamic Auto-Scaling and GPU resource allocation from local deployment to Running Distributed ML Jobs in Cloud Production environment:\nDynamic Auto-Scaling: Karpenter scales worker nodes based on Ray’s demand (CPU, GPU, memory). Ray autoscaler scales Ray worker pods dynamically. No need for pre-provisioned expensive GPU nodes. Multi-Tenant Resource Sharing and Seamless Transition from Local to Cloud: ML teams can submit workloads without managing Kubernetes pods directly. Ray manages job execution and ensures efficient resource utilization. ML Engineers can run jobs locally with Ray (ray.init()) and later scale seamlessly to AWS EKS by switching to ray.init(address=”ray://…”). Cost Optimization with Spot \u0026amp; On-Demand Nodes: Karpenter provisions spot instances for non-critical ML training. On-demand nodes handle critical, low-latency inference. Deployed a Ray cluster on Minikube with NVIDIA GPU support\nHere I will start a local Minikube Kubernetes Ray deployment to get started with the Ray cluster.\n# start minikube with GPU support root@zackz:~# minikube start --driver docker --container-runtime docker --gpus all --force --cpus=12 --memory=36g root@zackz:~# minikube addons enable nvidia-gpu-device-plugin # install ray and ray cluster helm chart on minikube root@zackz:~# helm repo add kuberay https://ray-project.github.io/kuberay-helm/ \u0026#34;kuberay\u0026#34; has been added to your repositories root@zackz:~# helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the \u0026#34;aws-ebs-csi-driver\u0026#34; chart repository ...Successfully got an update from the \u0026#34;kuberay\u0026#34; chart repository ...Successfully got an update from the \u0026#34;karpenter\u0026#34; chart repository ...Successfully got an update from the \u0026#34;eks-charts\u0026#34; chart repository ...Successfully got an update from the \u0026#34;grafana\u0026#34; chart repository ...Successfully got an update from the \u0026#34;external-secrets\u0026#34; chart repository ...Successfully got an update from the \u0026#34;prometheus-community\u0026#34; chart repository Update Complete. ⎈Happy Helming!⎈ root@zackz:~# helm install kuberay-operator kuberay/kuberay-operator NAME: kuberay-operator LAST DEPLOYED: Fri Feb 14 10:26:51 2025 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None root@zackz:~# helm install my-ray-cluster kuberay/ray-cluster NAME: my-ray-cluster LAST DEPLOYED: Fri Feb 14 10:27:25 2025 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None # check ray pods root@zackz:~# kubectl get pods NAMESPACE NAME READY STATUS RESTARTS AGE default kuberay-operator-975995b7d-xzjd4 1/1 Running 0 2m19s default my-ray-cluster-kuberay-head-q6gcz 1/1 Running 0 105s default my-ray-cluster-kuberay-workergroup-worker-mf5pr 1/1 Running 0 105s root@zackz:~# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kuberay-operator ClusterIP 10.98.253.157 \u0026lt;none\u0026gt; 8080/TCP 2m39s kubernetes ClusterIP 10.96.0.1 \u0026lt;none\u0026gt; 443/TCP 20d my-ray-cluster-kuberay-head-svc ClusterIP None \u0026lt;none\u0026gt; 10001/TCP,8265/TCP,8080/TCP,6379/TCP,8000/TCP 2m5s # forward ray dashboard port root@zackz:~# kubectl port-forward svc/my-ray-cluster-kuberay-head-svc 8265:8265 Forwarding from 127.0.0.1:8265 -\u0026gt; 8265 Forwarding from [::1]:8265 -\u0026gt; 8265 Handling connection for 8265 Access Ray dashboard via http://localhost:8265/ to verify Ray cluster head and worker nodes are running properly.\nResource Allocation and GPU training test\nHere I exec into Ray head pod to verify the resource and GPU support, and found that the Ray cluster is not configured to use GPU by default helm chart.\nroot@zackz:~# kubectl exec -it my-ray-cluster-kuberay-head-q6gcz -- bash (base) ray@my-ray-cluster-kuberay-head-q6gcz:~$ python -c \u0026#34;import ray; ray.init(); print(ray.cluster_resources())\u0026#34; 2025-02-13 15:31:21,699 INFO worker.py:1405 -- Using address 127.0.0.1:6379 set in the environment variable RAY_ADDRESS 2025-02-13 15:31:21,699 INFO worker.py:1540 -- Connecting to existing Ray cluster at address: 10.244.0.16:6379... 2025-02-13 15:31:21,706 INFO worker.py:1715 -- Connected to Ray cluster. View the dashboard at http://10.244.0.16:8265 {\u0026#39;node:__internal_head__\u0026#39;: 1.0, \u0026#39;node:10.244.0.16\u0026#39;: 1.0, \u0026#39;CPU\u0026#39;: 2.0, \u0026#39;memory\u0026#39;: 3000000000.0, \u0026#39;object_store_memory\u0026#39;: 540331621.0, \u0026#39;node:10.244.0.17\u0026#39;: 1.0} (base) ray@my-ray-cluster-kuberay-head-q6gcz:~$ nvidia-smi bash: nvidia-smi: command not found Hence I need to create a custom helm chart value file to add GPU resource request for Ray worker node and then update the Ray cluster.\nroot@zackz:~# helm list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION kuberay-operator default 1 2025-02-14 10:26:51.403488615 +1100 AEDT deployed kuberay-operator-1.2.2 my-ray-cluster default 1 2025-02-14 10:27:25.186940262 +1100 AEDT deployed ray-cluster-1.2.2 root@zackz:~# helm get values my-ray-cluster USER-SUPPLIED VALUES: null vim ray-values.yaml # Ray version rayVersion: \u0026#39;2.9.0\u0026#39; # Image configuration image: repository: rayproject/ray tag: \u0026#34;2.9.0\u0026#34; pullPolicy: IfNotPresent head: rayStartParams: dashboard-host: \u0026#34;0.0.0.0\u0026#34; num-cpus: \u0026#34;2\u0026#34; resources: limits: cpu: \u0026#34;2\u0026#34; memory: \u0026#34;4Gi\u0026#34; requests: cpu: \u0026#34;1\u0026#34; memory: \u0026#34;2Gi\u0026#34; volumeMounts: - mountPath: /dev/shm name: dshm volumes: - name: dshm emptyDir: medium: Memory worker: replicas: 1 rayStartParams: num-cpus: \u0026#34;2\u0026#34; resources: limits: cpu: \u0026#34;4\u0026#34; memory: \u0026#34;16Gi\u0026#34; nvidia.com/gpu: 1 requests: cpu: \u0026#34;2\u0026#34; memory: \u0026#34;8Gi\u0026#34; volumeMounts: - mountPath: /dev/shm name: dshm volumes: - name: dshm emptyDir: medium: Memory root@zackz:/mnt/f/ml-local/local-minikube/ray# helm upgrade my-ray-cluster kuberay/ray-cluster -f ray-values.yaml Release \u0026#34;my-ray-cluster\u0026#34; has been upgraded. Happy Helming! NAME: my-ray-cluster LAST DEPLOYED: Fri Feb 14 11:40:24 2025 NAMESPACE: default STATUS: deployed REVISION: 3 TEST SUITE: None Now I was able to run a Ray task-based workload with memory/custom resource constraints, Adjusted Ray\u0026rsquo;s memory thresholds to avoid OOM kills.\nroot@zackz:/mnt/f/ml-local/local-minikube/ray# kubectl exec -it my-ray-cluster-kuberay-workergroup-worker-6lcx5 -- bash (base) ray@my-ray-cluster-kuberay-workergroup-worker-6lcx5:~$ nvidia-smi Thu Feb 13 19:16:44 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 560.35.02 Driver Version: 560.94 CUDA Version: 12.6 | |-----------------------------------------+------------------------+----------------------| | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3070 Ti On | 00000000:01:00.0 On | N/A | | 47% 45C P8 19W / 232W | 2955MiB / 8192MiB | 28% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------| +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 27 G /Xwayland N/A | | 0 N/A N/A 37 G /Xwayland N/A | +-----------------------------------------------------------------------------------------+ (base) ray@my-ray-cluster-kuberay-workergroup-worker-6lcx5:~$ exit exit root@zackz:/mnt/f/ml-local/local-minikube/ray# \\ python -c \u0026#34; import ray from time import sleep # Initialize with memory threshold adjustment ray.init(runtime_env={ \u0026#39;env_vars\u0026#39;: { \u0026#39;RAY_memory_monitor_refresh_ms\u0026#39;: \u0026#39;0\u0026#39;, # Disable OOM killing \u0026#39;RAY_memory_usage_threshold\u0026#39;: \u0026#39;0.95\u0026#39; } }) # Specify resource requirements for the task @ray.remote( num_cpus=0.5, # Use less CPU to allow multiple tasks memory=500 * 1024 * 1024, # Request 500MB memory per task resources={\u0026#39;worker\u0026#39;: 1} # Ensure it runs on worker nodes ) def train_model_simulation(model_id): sleep(2) return f\u0026#39;Model {model_id} trained\u0026#39; # Run fewer parallel tasks initially futures = [train_model_simulation.remote(i) for i in range(2)] results = ray.get(futures) print(results) Check the job event and history in Ray dashboard.\nConclusion\nHere is what we achieved:\nRay cluster setup on Local Minikube with GPU support (KubeRay) Configuring Ray’s resource limits (memory, CPU allocation per task) Running remote tasks efficiently using Ray\u0026rsquo;s distributed execution Observing Ray cluster resource usage in real-time Next step: I will see how to run Ray cluster together with Karpenter on EKS once I have the GPU EC2 instance quota request approved by AWS.\n","permalink":"https://zackblog.work/posts/mlops-get-started-with-kuberay/","summary":"\u003cp\u003eRay is an open-source framework designed for scalable and distributed ML workloads, including training, tuning, and inference. It provides a simple API for scaling Python applications across multiple nodes.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ccode\u003eDistributed Training\u003c/code\u003e: Easily scale PyTorch, TensorFlow, and other ML jobs across multiple GPUs/instances.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eHyperparameter Tuning\u003c/code\u003e: Integrates with Optuna and Ray Tune for distributed hyperparameter optimization.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eParallel Inference\u003c/code\u003e: Supports inference pipelines that scale out dynamically based on demand.\u003c/li\u003e\n\u003cli\u003e\u003ccode\u003eFault Tolerance\u003c/code\u003e: If a node fails, Ray can reschedule tasks on other available nodes.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eTogether with EKS, Karpenter, and Ray, a modern ML team can achieve dynamic Auto-Scaling and GPU resource allocation from local deployment to Running Distributed ML Jobs in Cloud Production environment:\u003c/p\u003e","title":"MLOps - Get Started with KubeRay"},{"content":"Finally, machine learning workload into AWS EKS with Karpenter\nIn the previous post, I was able to complete both local Pytorch ML and AWS SageMaker practice, and containerize and deploy ML docker image locally.\nIn this post, I will explore the model deployment to EKS cluster with Karpenter to simulate a more scalable and real-life production-ready environment.\nChallenges when moving to Cloud deployment\nHere are key differences between local PyTorch model vs AWS SageMaker artifacts, and how they impact the Docker image size and Performance considerations for LLM images in EKS deployment.\nAspect Local PyTorch Model (.pth) AWS SageMaker Model (.tar.gz) Contents Full model state_dict + Python code Only model parameters + inference code Framework Raw PyTorch implementation Optimized MXNet framework Serialization torch.save() native format Framework-specific serialization Dependencies Requires full PyTorch installation Minimal runtime dependencies The latency between pod initialization and readiness to serve requests includes:\nContainer image pull time Model download from storage Framework initialization GPU context creation Model loading into memory Optimize Docker image size for AWS ECR and EKS deployment.\nOptimized Dockerfile (Target: ~450MB)\n# Use NVIDIA CUDA base image with Python FROM nvidia/cuda:11.2.2-base-ubuntu20.04 # Install system dependencies RUN apt-get update \u0026amp;\u0026amp; \\ DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \\ python3.8 \\ python3-pip \\ python3.8-venv \\ \u0026amp;\u0026amp; rm -rf /var/lib/apt/lists/* # Create and activate virtual environment RUN python3 -m venv /opt/venv ENV PATH=\u0026#34;/opt/venv/bin:$PATH\u0026#34; # Install Python dependencies COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # Copy inference code COPY app.py . # Environment variables ENV MODEL_S3_URI=\u0026#34;s3://sagemaker-bucket-85xxxxxxxx42/models/image_model/classifier-2025-01-26-02-58-03-005-dbf196d2/output/model.tar.gz\u0026#34; ENV AWS_REGION=ap-southeast-2 # Expose API port EXPOSE 8080 # Startup command CMD [\u0026#34;python3\u0026#34;, \u0026#34;app.py\u0026#34;] requirements.txt\nmxnet==1.9.1 flask==2.2.5 boto3==1.28.62 pillow==10.1.0 app.py (Optimized Inference Service)\nimport os import tarfile import boto3 from flask import Flask, request, jsonify import mxnet as mx import numpy as np from PIL import Image import io app = Flask(__name__) # Initialize model ctx = mx.gpu() if mx.context.num_gpus() \u0026gt; 0 else mx.cpu() model = None def download_and_extract_model(): global model s3 = boto3.client(\u0026#39;s3\u0026#39;, region_name=os.environ[\u0026#39;AWS_REGION\u0026#39;]) model_path = \u0026#39;/tmp/model.tar.gz\u0026#39; bucket, key = os.environ[\u0026#39;MODEL_S3_URI\u0026#39;].split(\u0026#39;//\u0026#39;)[1].split(\u0026#39;/\u0026#39;, 1) s3.download_file(bucket, key, model_path) with tarfile.open(model_path) as tar: tar.extractall(path=\u0026#39;/model\u0026#39;) sym, arg_params, aux_params = mx.model.load_checkpoint(\u0026#39;/model/model\u0026#39;, 0) mod = mx.mod.Module(symbol=sym, context=ctx) mod.bind(for_training=False, data_shapes=[(\u0026#39;data\u0026#39;, (1, 3, 224, 224))]) mod.set_params(arg_params, aux_params) model = mod @app.before_first_request def initialize(): download_and_extract_model() def transform_image(image_bytes): img = Image.open(io.BytesIO(image_bytes)).convert(\u0026#39;RGB\u0026#39;) img = img.resize((224, 224)) img = np.array(img).transpose(2, 0, 1).astype(np.float32) img = mx.nd.array((img - 128) / 128) # Match SageMaker preprocessing return img.reshape((1, 3, 224, 224)) @app.route(\u0026#39;/predict\u0026#39;, methods=[\u0026#39;POST\u0026#39;]) def predict(): if \u0026#39;image\u0026#39; not in request.files: return jsonify({\u0026#39;error\u0026#39;: \u0026#39;No image provided\u0026#39;}), 400 image = request.files[\u0026#39;image\u0026#39;].read() data = transform_image(image) batch = mx.io.DataBatch([data]) model.forward(batch, is_train=False) prob = model.get_outputs()[0].asnumpy().argmax() return jsonify({\u0026#39;prediction\u0026#39;: int(prob)}) if __name__ == \u0026#39;__main__\u0026#39;: app.run(host=\u0026#39;0.0.0.0\u0026#39;, port=8080) Size Comparison after optimization:\nComponent Local PyTorch Optimized MXNet Reduction Base Image 2.5GB 418MB -83% Framework 1.2GB 89MB -93% Model Storage Baked-in S3 Download -100% Total Image Size ~5GB 537MB -90% root@zackz:/mnt/f/ml-local/local-cv/eks# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE classifier-eks latest 92736798f67f 11 seconds ago 537MB pneumonia-frontend latest ac8f5e301dfd 3 days ago 47.1MB pneumonia-classifier-1 latest d0bf743556d8 3 days ago 4.73GB Provision EKS and Karpenter with Terraform\nHere I will use Terraform to provision an EKS cluster with Karpenter.\nroot@zackz:/mnt/f/1/spot-and-karpenter# kubectl get node NAME STATUS ROLES AGE VERSION ip-10-0-119-37.ap-southeast-2.compute.internal Ready \u0026lt;none\u0026gt; 21m v1.30.8-eks-aeac579 ip-10-0-65-17.ap-southeast-2.compute.internal Ready \u0026lt;none\u0026gt; 21m v1.30.8-eks-aeac579 root@zackz:/mnt/f/1/spot-and-karpenter# kubectl get po -A NAMESPACE NAME READY STATUS RESTARTS AGE karpenter karpenter-7c9f6776cc-5djcv 1/1 Running 0 23m karpenter karpenter-7c9f6776cc-ntgwj 1/1 Running 0 23m kube-system aws-node-8v5v5 2/2 Running 0 21m kube-system aws-node-pqlcg 2/2 Running 0 21m kube-system coredns-7dd48c8549-97dbr 1/1 Running 0 23m kube-system coredns-7dd48c8549-dv4n4 1/1 Running 0 23m kube-system ebs-csi-controller-56cb7b4bc-2gwgl 6/6 Running 0 23m kube-system ebs-csi-controller-56cb7b4bc-wbffq 6/6 Running 0 23m kube-system ebs-csi-node-lnvg8 3/3 Running 0 21m kube-system ebs-csi-node-nkg5c 3/3 Running 0 21m kube-system efs-csi-controller-75645855f5-jssl5 3/3 Running 0 4m22s kube-system efs-csi-controller-75645855f5-lfd8r 3/3 Running 0 4m22s kube-system efs-csi-node-jc6lm 3/3 Running 0 4m22s kube-system efs-csi-node-m6g7q 3/3 Running 0 4m23s kube-system kube-proxy-44px7 1/1 Running 0 21m kube-system kube-proxy-lrtfd 1/1 Running 0 21m root@zackz:/mnt/f/1/spot-and-karpenter# aws eks list-addons --cluster-name spot-and-karpenter --region ap-southeast-2 { \u0026#34;addons\u0026#34;: [ \u0026#34;aws-ebs-csi-driver\u0026#34;, \u0026#34;aws-efs-csi-driver\u0026#34;, \u0026#34;coredns\u0026#34;, \u0026#34;kube-proxy\u0026#34;, \u0026#34;vpc-cni\u0026#34; ] } Then tag and push the optimized image classifier-eks to ECR and deploy it to EKS using Karpenter.\nroot@zackz:~# aws ecr get-login-password --region ap-southeast-2 | docker login --username AWS --password-stdin 85xxxxxxxx42.dkr.ecr.ap-southeast-2.amazonaws.com Login Succeeded root@zackz:~# docker tag classifier-eks:latest 85xxxxxxxx42.dkr.ecr.ap-southeast-2.amazonaws.com/classifier-eks:latest root@zackz:~# docker push 85xxxxxxxx42.dkr.ecr.ap-southeast-2.amazonaws.com/classifier-eks:latest The push refers to repository [85xxxxxxxx42.dkr.ecr.ap-southeast-2.amazonaws.com/classifier-eks] 700cbfa4a29a: Pushed 1e026a0de221: Pushed e30ec0bde91f: Pushed 9994fd5f0914: Pushed 67796cf8ce29: Pushed 0474cd91a62d: Pushed 3c2c7e066741: Pushed 3d25fa2df354: Pushed 0f24c57a5268: Pushed 6c3e7df31590: Pushed latest: digest: sha256:ccb3a4f70dd01b59658bc365616346bb62c5389814857b78416e40499b6d35c2 size: 2416 Create Karpenter NodeClass and NodePool for GPU workloads.\n# nodepool-gpu.yaml apiVersion: karpenter.sh/v1beta1 kind: NodePool metadata: name: gpu-pool spec: template: metadata: labels: workload-type: custom-ml spec: nodeClassRef: apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass name: gpu-nodeclass requirements: - key: karpenter.sh/capacity-type operator: In values: [\u0026#34;on-demand\u0026#34;, \u0026#34;spot\u0026#34;] - key: node.kubernetes.io/instance-type operator: In values: [\u0026#34;g4dn.2xlarge\u0026#34;, \u0026#34;g5.xlarge\u0026#34;, \u0026#34;g5.2xlarge\u0026#34;] # Larger instance types - key: kubernetes.io/arch operator: In values: [\u0026#34;amd64\u0026#34;] taints: - key: \u0026#34;nvidia.com/gpu\u0026#34; value: \u0026#34;present\u0026#34; effect: NoSchedule disruption: consolidationPolicy: WhenUnderutilized expireAfter: 168h # gpu-nodeclass.yaml apiVersion: karpenter.k8s.aws/v1beta1 kind: EC2NodeClass metadata: name: gpu-nodeclass spec: role: \u0026#34;karpenter-spot-and-karpenter\u0026#34; subnetSelectorTerms: - tags: karpenter.sh/discovery: \u0026#34;spot-and-karpenter\u0026#34; securityGroupSelectorTerms: - tags: karpenter.sh/discovery: \u0026#34;spot-and-karpenter\u0026#34; amiFamily: Bottlerocket blockDeviceMappings: - deviceName: \u0026#34;/dev/xvda\u0026#34; ebs: volumeSize: 100Gi volumeType: gp3 userData: | [settings] [settings.kernel] lockdown = \u0026#34;integrity\u0026#34; [settings.kubernetes] node-labels = { \u0026#34;workload-type\u0026#34; = \u0026#34;custom-ml\u0026#34; } Create EKS deployment to request GPU nodes using Karpenter, here I will choose spot g4dn.2xlarge instance to deploy the classifier-eks:latest image from ECR.\n# custom-ml-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: custom-ml spec: replicas: 1 selector: matchLabels: app: custom-ml template: metadata: labels: app: custom-ml spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: workload-type operator: In values: [\u0026#34;custom-ml\u0026#34;] tolerations: - key: \u0026#34;nvidia.com/gpu\u0026#34; operator: \u0026#34;Exists\u0026#34; effect: \u0026#34;NoSchedule\u0026#34; - key: \u0026#34;karpenter.sh/interruption\u0026#34; operator: \u0026#34;Exists\u0026#34; effect: \u0026#34;NoSchedule\u0026#34; containers: - name: custom-ml image: 8xxxxxxxx42.dkr.ecr.ap-southeast-2.amazonaws.com/classifier-eks:latest resources: requests: nvidia.com/gpu: 1 cpu: 3000m # Reduced CPU request memory: 12Gi # Reduced memory request limits: nvidia.com/gpu: 1 cpu: 3000m memory: 12Gi command: [\u0026#34;python\u0026#34;, \u0026#34;app.py\u0026#34;] Check the EC2 GPU instance type and history price for spot instance, looks like the spot g4dn.2xlarge instance is very good choice with 8 vCPU and 32GB memory plus16GB GPU Memory with hourly cost $0.3121 could be a great choice to host some LLM models.\n# list available GPU instances and spot price root@zackz:/mnt/f/ml-local/local-cv# aws ec2 describe-spot-price-history \\ --instance-types $(aws ec2 describe-instance-types \\ --query \u0026#39;InstanceTypes[?GpuInfo.Gpus!=null].InstanceType\u0026#39; --output text --region ap-southeast-2) \\ --product-descriptions \u0026#34;Linux/UNIX\u0026#34; \\ --start-time \u0026#34;$(date -u +\u0026#39;%Y-%m-%dT%H:%M:%SZ\u0026#39;)\u0026#34; \\ --region ap-southeast-2 \\ --query \u0026#39;SpotPriceHistory[*].[InstanceType, SpotPrice]\u0026#39; \\ --output table --------------------------------- | DescribeSpotPriceHistory | +----------------+--------------+ | g5.4xlarge | 0.633900 | | g5.xlarge | 0.396100 | | g5.8xlarge | 0.996100 | | g6.12xlarge | 1.890000 | | g5.48xlarge | 6.971800 | | g5.12xlarge | 2.227300 | | g4dn.metal | 3.076200 | | g5.4xlarge | 0.644500 | | g5.2xlarge | 0.482900 | | g4dn.xlarge | 0.202500 | | gr6.4xlarge | 0.615000 | | gr6.4xlarge | 0.614200 | | g6.2xlarge | 0.418500 | | g4dn.12xlarge | 1.421100 | | g6.16xlarge | 1.397200 | | g5.8xlarge | 0.991000 | | g4dn.2xlarge | 0.290400 | | g4dn.16xlarge | 1.828200 | | g5.24xlarge | 3.025700 | | g6.4xlarge | 0.668600 | | g4dn.metal | 3.893000 | | g4dn.metal | 4.544200 | | g6.16xlarge | 1.395900 | | g4dn.8xlarge | 0.917800 | | g4dn.2xlarge | 0.312100 | | g5.12xlarge | 2.156600 | | g6.12xlarge | 1.775700 | | g5.xlarge | 0.401000 | | p3.2xlarge | 1.248900 | | g6.24xlarge | 2.484600 | | g6.8xlarge | 0.796000 | | g6.2xlarge | 0.459600 | | p3.16xlarge | 9.822300 | | gr6.8xlarge | 0.990000 | | gr6.8xlarge | 1.080300 | | g5.16xlarge | 1.683200 | | p3.8xlarge | 4.944200 | | p2.16xlarge | 11.141200 | | g5.48xlarge | 6.229500 | | g4dn.16xlarge | 1.870900 | | g6.48xlarge | 5.242900 | | g4dn.16xlarge | 1.655800 | | g6.4xlarge | 0.524600 | | g6.xlarge | 0.374900 | | g4dn.12xlarge | 1.478100 | | g6.8xlarge | 0.760100 | | g6.24xlarge | 2.538400 | | g4dn.xlarge | 0.213200 | | g4dn.xlarge | 0.205000 | | g6.xlarge | 0.355200 | | g5.16xlarge | 1.591500 | | g4dn.2xlarge | 0.295700 | | p2.xlarge | 0.530700 | | g4dn.8xlarge | 0.888700 | | p5.48xlarge | 38.966200 | | g4dn.4xlarge | 0.478200 | | g5.2xlarge | 0.490200 | | g4dn.8xlarge | 0.830500 | | g4dn.4xlarge | 0.442800 | | g4dn.12xlarge | 1.577900 | | g6.48xlarge | 5.181000 | | g4dn.4xlarge | 0.455200 | | g5.24xlarge | 3.213500 | | p2.xlarge | 1.542000 | | p2.8xlarge | 12.336000 | | p2.8xlarge | 12.336000 | | p2.16xlarge | 24.672000 | | p5.48xlarge | 127.816000 | | p5.48xlarge | 127.816000 | +----------------+--------------+ (END) ---------------------------------------------------------------------------------------------------- root@zackz:/mnt/f/ml-local/local-cv# aws ec2 describe-instance-types \\ --query \u0026#39;InstanceTypes[?GpuInfo.Gpus!=null].[InstanceType, GpuInfo.Gpus[0].Manufacturer, GpuInfo.Gpus[0].Name, GpuInfo.Gpus[0].Count, GpuInfo.TotalGpuMemoryInMiB]\u0026#39; \\ --region ap-southeast-2 --output table ----------------------------------------------------- | DescribeInstanceTypes | +----------------+---------+-------+-----+----------+ | g5.4xlarge | NVIDIA | A10G | 1 | 24576 | | g6.24xlarge | NVIDIA | L4 | 4 | 91552 | | g5.2xlarge | NVIDIA | A10G | 1 | 24576 | | g4dn.metal | NVIDIA | T4 | 8 | 131072 | | g6.8xlarge | NVIDIA | L4 | 1 | 22888 | | g6.48xlarge | NVIDIA | L4 | 8 | 183104 | | p2.8xlarge | NVIDIA | K80 | 8 | 98304 | | g4dn.12xlarge | NVIDIA | T4 | 4 | 65536 | | p3.2xlarge | NVIDIA | V100 | 1 | 16384 | | g5.24xlarge | NVIDIA | A10G | 4 | 98304 | | p5.48xlarge | NVIDIA | H100 | 8 | 655360 | | p4d.24xlarge | NVIDIA | A100 | 8 | 327680 | | g5.12xlarge | NVIDIA | A10G | 4 | 98304 | | g4dn.xlarge | NVIDIA | T4 | 1 | 16384 | | g6.2xlarge | NVIDIA | L4 | 1 | 22888 | | g5.xlarge | NVIDIA | A10G | 1 | 24576 | | g5.16xlarge | NVIDIA | A10G | 1 | 24576 | | g6.xlarge | NVIDIA | L4 | 1 | 22888 | | g6.16xlarge | NVIDIA | L4 | 1 | 22888 | | gr6.8xlarge | NVIDIA | L4 | 1 | 22888 | | g6.12xlarge | NVIDIA | L4 | 4 | 91552 | | g6.4xlarge | NVIDIA | L4 | 1 | 22888 | | g4dn.16xlarge | NVIDIA | T4 | 1 | 16384 | | gr6.4xlarge | NVIDIA | L4 | 1 | 22888 | | p2.xlarge | NVIDIA | K80 | 1 | 12288 | | p3.16xlarge | NVIDIA | V100 | 8 | 131072 | | g5.8xlarge | NVIDIA | A10G | 1 | 24576 | | g4dn.8xlarge | NVIDIA | T4 | 1 | 16384 | | g5.48xlarge | NVIDIA | A10G | 8 | 196608 | | p3.8xlarge | NVIDIA | V100 | 4 | 65536 | | p2.16xlarge | NVIDIA | K80 | 16 | 196608 | | g4dn.4xlarge | NVIDIA | T4 | 1 | 16384 | | g4dn.2xlarge | NVIDIA | T4 | 1 | 16384 | +----------------+---------+-------+-----+----------+ (END) Karpenter logs for troubleshooting:\nUnfortunately, the GPU instance for ML workload is not provisioned by Karpenter due to Max spot instance count exceeded and insufficient capacity with VcpuLimitExceeded, Here are the logs from the Karpenter controller:\nkubectl -n karpenter logs -l app.kubernetes.io/name=karpenter { \u0026#34;level\u0026#34;: \u0026#34;ERROR\u0026#34;, \u0026#34;time\u0026#34;: \u0026#34;2025-02-01T00:48:15.820Z\u0026#34;, \u0026#34;logger\u0026#34;: \u0026#34;controller.nodeclaim.lifecycle\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;creating instance, insufficient capacity, with fleet error(s), MaxSpotInstanceCountExceeded: Max spot instance count exceeded\u0026#34;, \u0026#34;commit\u0026#34;: \u0026#34;1072d3b\u0026#34;, \u0026#34;nodeclaim\u0026#34;: \u0026#34;gpu-pool-m7qwg\u0026#34;, \u0026#34;nodepool\u0026#34;: \u0026#34;gpu-pool\u0026#34; } { \u0026#34;level\u0026#34;: \u0026#34;ERROR\u0026#34;, \u0026#34;time\u0026#34;: \u0026#34;2025-02-01T01:01:49.074Z\u0026#34;, \u0026#34;logger\u0026#34;: \u0026#34;controller.nodeclaim.lifecycle\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;creating instance, insufficient capacity, with fleet error(s), VcpuLimitExceeded: You have requested more vCPU capacity than your current vCPU limit of 0 allows for the instance bucket that the specified instance type belongs to. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit.\u0026#34;, \u0026#34;commit\u0026#34;: \u0026#34;1072d3b\u0026#34;, \u0026#34;nodeclaim\u0026#34;: \u0026#34;gpu-pool-m6c45\u0026#34;, \u0026#34;nodepool\u0026#34;: \u0026#34;gpu-pool\u0026#34; } { \u0026#34;level\u0026#34;: \u0026#34;ERROR\u0026#34;, \u0026#34;time\u0026#34;: \u0026#34;2025-02-01T01:04:37.398Z\u0026#34;, \u0026#34;logger\u0026#34;: \u0026#34;controller.provisioner\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;Could not schedule pod, incompatible with nodepool \\\u0026#34;gpu-pool\\\u0026#34;, daemonset overhead={\\\u0026#34;cpu\\\u0026#34;:\\\u0026#34;210m\\\u0026#34;,\\\u0026#34;memory\\\u0026#34;:\\\u0026#34;240Mi\\\u0026#34;,\\\u0026#34;pods\\\u0026#34;:\\\u0026#34;5\\\u0026#34;}, no instance type satisfied resources {\\\u0026#34;cpu\\\u0026#34;:\\\u0026#34;1210m\\\u0026#34;,\\\u0026#34;memory\\\u0026#34;:\\\u0026#34;4336Mi\\\u0026#34;, \\\u0026#34;nvidia.com/gpu\\\u0026#34;:\\\u0026#34;1\\\u0026#34;,\\\u0026#34;pods\\\u0026#34;:\\\u0026#34;6\\\u0026#34;} and requirements karpenter.k8s.aws/instance-family In [g5 p3 p4], karpenter.sh/capacity-type In [on-demand spot], karpenter.sh/nodepool In [gpu-pool], kubernetes.io/arch In [amd64], workload-type In [custom-ml] (no instance type met all requirements)\u0026#34;, \u0026#34;commit\u0026#34;: \u0026#34;1072d3b\u0026#34;, \u0026#34;pod\u0026#34;: \u0026#34;default/custom-ml\u0026#34; } Need to engage with AWS support to request GPU instance limit increase.\nConclusion\nThis exploration of deploying an ML workload to EKS with Karpenter revealed several critical insights for production-grade MLOps. Key takeaways include:\nML docker container Optimization EKS with Karpenter ready for ML load deployment I will continue to explore the performance and cost practices after approved GPU quotas, by leveraging spot GPU instances and cold start and model cache to reduce the performance factors, particularly in terms of initialization time, model download, and container image pull times.\nThen I will deploy the frontend application and K8S services to expose the load balancer for image upload and prediction.\n","permalink":"https://zackblog.work/posts/mlops-deploy-ml-workload-into-eks-with-karpenter/","summary":"\u003cp\u003eFinally, machine learning workload into AWS EKS with Karpenter\u003c/p\u003e\n\u003cp\u003eIn the previous post, I was able to complete both local Pytorch ML and AWS SageMaker practice, and containerize and deploy ML docker image locally.\u003c/p\u003e\n\u003cp\u003eIn this post, I will explore the model deployment to EKS cluster with Karpenter to simulate a more scalable and real-life production-ready environment.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eChallenges when moving to Cloud deployment\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eHere are key differences between local PyTorch model vs AWS SageMaker artifacts, and how they impact the Docker image size and Performance considerations for LLM images in EKS deployment.\u003c/p\u003e","title":"MLOps - Deploy ML workload into EKS with Karpenter"},{"content":"Continue Pneumonia Classifier by moving to AWS Serverless\nGo AWS Serverless Deployment\nDeploying machine learning models in production requires additional considerations to address latency, scalability, cost-efficiency, and monitoring.\nA modern approach to hosting an ML application in AWS can be considered as a serverless architecture.\nThis allows users to upload images via a S3 static web page and send them to API Gateway. The API Gateway receives the HTTP POST request and forwards it to the Lambda function, which handles the image preprocessing, inference, and postprocessing logic. The Lambda function sends the image payload to the SageMaker endpoint, by calling the SageMaker endpoint using the SageMaker Runtime SDK (invoke_endpoint) which hosts my trained model, and then retrieves the prediction. The prediction result is sent back to the frontend for display.\nThis approach leverages S3, AWS Lambda, Amazon API Gateway, and Amazon SageMaker, to leverage with Amazon SageMaker Managed endpoints (Real-Time Inference) that handle auto-scaling, security, and monitoring out-of-the-box.\nAdvantage with AWS Serverless\nScalability: API Gateway scales automatically to handle high concurrency. Lambda scales horizontally (serverless) and is invoked only when needed. SageMaker Endpoint supports auto-scaling to handle varying inference loads. Low Latency: Real-time inference is achieved with the SageMaker Endpoint. Cost Optimized: Lambda is a pay-per-use service, so we\u0026rsquo;re not paying for idle compute resources. SageMaker Endpoint supports multi-model endpoints and elastic inference for cost savings. AWS Serverless and fully managed services: Fully managed services reduce operational overhead for model hosting and frontend and backend infrastructure. # Model flow S3 (Model Artifacts) → SageMaker Model → Real-Time Endpoint (GPU) → Auto-Scaling + Model Monitor # Image and Inference flow S3 Static web → Upload Image → API Gateway → Lambda (Image Preprocessing) → SageMaker Endpoint (Real-Time Inference) → API Gateway → S3 (Result) Design of the Pneumonia Classifier Application with AWS Serverless\nHere are the components of the serverless application:\nFrontend: Contains the static files for the website hosted on S3, allowing users to upload images and display predictions. Backend: Yaml file to deploy API Gateway to trigger the Lambda function. Lambda: Defines function to process the image and interact with the SageMaker endpoint. SageMaker: Configuration for the serverless endpoint that hosts the ML model. # Folder Structure root@zackz:/mnt/f/ml-local/local-cv/aws-deploy# tree . ├── backend │ └── api-gateway-config.yaml ├── frontend │ ├── assets │ │ ├── fonts │ │ └── images │ ├── index.html │ ├── script.js │ └── style.css └── sagemaker └── endpoint-config.yaml Frontend: S3 Static Website Hosting\nUpload files into the S3 bucket and enable Static Website Hosting, Specify index.html as the index document, Update the Bucket Policy to Allow Public Access, verify the URL for access.\n{ \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Sid\u0026#34;: \u0026#34;PublicReadGetObject\u0026#34;, \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Principal\u0026#34;: \u0026#34;*\u0026#34;, \u0026#34;Action\u0026#34;: \u0026#34;s3:GetObject\u0026#34;, \u0026#34;Resource\u0026#34;: \u0026#34;arn:aws:s3:::sagemaker-frontend--zz-imageclassification/*\u0026#34; } ] } Backend: API Gateway and Lambda Function\nThis folder will contain a CloudFormation YAML file to create API Gateway, Lambda function and Lambda Execution role. Outputs the API Gateway URL.\nAPI Gateway resource will create a REST API (ImageClassificationAPI) with a /predict resource. Defines a POST method that integrates with the Lambda function, then deploys to a prod stage that allows API Gateway to invoke the Lambda function.\nLambda defines the function (ImageClassificationLambda) that interacts with the SageMaker endpoint. Includes the Python code for handling the image upload and invoking the SageMaker endpoint.\n# api-gateway-config.yaml AWSTemplateFormatVersion: \u0026#39;2010-09-09\u0026#39; Description: CloudFormation template for API Gateway and Lambda integration Resources: # API Gateway ImageClassificationAPI: Type: AWS::ApiGateway::RestApi Properties: Name: ImageClassificationAPI Description: API for image classification # API Gateway Resource PredictResource: Type: AWS::ApiGateway::Resource Properties: RestApiId: !Ref ImageClassificationAPI ParentId: !GetAtt ImageClassificationAPI.RootResourceId PathPart: predict # API Gateway Method (POST) PredictMethod: Type: AWS::ApiGateway::Method Properties: RestApiId: !Ref ImageClassificationAPI ResourceId: !Ref PredictResource HttpMethod: POST AuthorizationType: NONE Integration: Type: AWS_PROXY IntegrationHttpMethod: POST Uri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ImageClassificationLambda.Arn}/invocations # Lambda Function ImageClassificationLambda: Type: AWS::Lambda::Function Properties: Handler: app.lambda_handler Runtime: python3.9 Role: !GetAtt LambdaExecutionRole.Arn Code: ZipFile: | import boto3 import json import base64 sagemaker = boto3.client(\u0026#39;sagemaker-runtime\u0026#39;) def lambda_handler(event, context): try: # Decode the image from the request body = json.loads(event[\u0026#39;body\u0026#39;]) image_bytes = base64.b64decode(body[\u0026#39;file\u0026#39;]) # Call SageMaker endpoint response = sagemaker.invoke_endpoint( EndpointName=\u0026#39;zack-aws-sagemaker-endpoint\u0026#39;, ContentType=\u0026#39;application/x-image\u0026#39;, Body=image_bytes ) # Parse the prediction prediction = json.loads(response[\u0026#39;Body\u0026#39;].read().decode()) return { \u0026#39;statusCode\u0026#39;: 200, \u0026#39;body\u0026#39;: json.dumps({\u0026#39;prediction\u0026#39;: prediction}) } except Exception as e: return { \u0026#39;statusCode\u0026#39;: 500, \u0026#39;body\u0026#39;: json.dumps({\u0026#39;error\u0026#39;: str(e)}) } # Lambda Execution Role LambdaExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: \u0026#39;2012-10-17\u0026#39; Statement: - Effect: Allow Principal: Service: lambda.amazonaws.com Action: sts:AssumeRole Policies: - PolicyName: LambdaSageMakerAccess PolicyDocument: Version: \u0026#39;2012-10-17\u0026#39; Statement: - Effect: Allow Action: - sagemaker:InvokeEndpoint Resource: \u0026#34;*\u0026#34; - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: \u0026#34;*\u0026#34; # API Gateway Deployment ApiGatewayDeployment: Type: AWS::ApiGateway::Deployment Properties: RestApiId: !Ref ImageClassificationAPI StageName: prod # API Gateway Permission to Invoke Lambda ApiGatewayPermission: Type: AWS::Lambda::Permission Properties: Action: lambda:InvokeFunction FunctionName: !GetAtt ImageClassificationLambda.Arn Principal: apigateway.amazonaws.com SourceArn: !Sub arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${ImageClassificationAPI}/*/POST/predict Outputs: ApiGatewayUrl: Description: URL of the API Gateway Value: !Sub https://${ImageClassificationAPI}.execute-api.${AWS::Region}.amazonaws.com/prod # deploy backend aws cloudformation create-stack \\ --stack-name image-classification-api \\ --template-body file://api-gateway-config.yaml \\ --capabilities CAPABILITY_NAMED_IAM \\ --region ap-southeast-2 # test the API curl -X POST -F \u0026#34;file=@data/chest_xray/val/val_normal0.jpeg\u0026#34; https://i4s4znf7bb.execute-api.ap-southeast-2.amazonaws.com/prod/ImageClassificationAPI/predict Output: b\u0026#39;[0.8592441082997322, 0.14075589179992676]\u0026#39; Sagemaker Endpoint\nThe CloudFormation template covers the SageMaker Execution Role, which grants the SageMaker service permissions to access S3 (for model artifacts) and CloudWatch (for logging), also includes the SageMaker Model, serverless Endpoint with a maximum concurrency of 5 and 2048 MB of memory, and outputs the SageMaker endpoint name.\nAWSTemplateFormatVersion: \u0026#39;2010-09-09\u0026#39; Description: CloudFormation template for SageMaker serverless endpoint Resources: # SageMaker Execution Role SageMakerExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: \u0026#39;2012-10-17\u0026#39; Statement: - Effect: Allow Principal: Service: sagemaker.amazonaws.com Action: sts:AssumeRole Policies: - PolicyName: SageMakerAccess PolicyDocument: Version: \u0026#39;2012-10-17\u0026#39; Statement: - Effect: Allow Action: - s3:GetObject - s3:PutObject Resource: arn:aws:s3:::sagemaker-bucket-851725491342/* - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: \u0026#34;*\u0026#34; # SageMaker Model ImageClassificationModel: Type: AWS::SageMaker::Model Properties: ModelName: zack-aws-sagemaker-endpoint PrimaryContainer: Image: algorithm_image ModelDataUrl: s3://sagemaker-bucket-851725491342/models/image_model/classifier-2025-01-26-02-58-03-001-a577816e/output/model.tar.gz ExecutionRoleArn: !GetAtt SageMakerExecutionRole.Arn # SageMaker Endpoint Configuration ImageClassificationEndpointConfig: Type: AWS::SageMaker::EndpointConfig Properties: ProductionVariants: - ModelName: !Ref ImageClassificationModel VariantName: AllTraffic ServerlessConfig: MaxConcurrency: 5 MemorySizeInMB: 2048 # SageMaker Endpoint ImageClassificationEndpoint: Type: AWS::SageMaker::Endpoint Properties: EndpointConfigName: !Ref ImageClassificationEndpointConfig EndpointName: ImageClassificationEndpoint Outputs: SageMakerEndpointName: Description: Name of the SageMaker endpoint Value: !Ref ImageClassificationEndpoint aws cloudformation create-stack \\ --stack-name sagemaker-endpoint \\ --template-body file://endpoint-config.yaml \\ --capabilities CAPABILITY_NAMED_IAM \\ --region ap-southeast-2 Test and Validate\nAccess the S3 static website URL, choose an image to verify the model prediction. Logs can be found via both CloudWatch SageMaker endpoint and Lambda logs.\nAPI and Model prediction verified from Postman\nKey Takeaways:\nMove Pneumonia image classification ML application from local Docker to AWS serverless deployment. Leverage AWS API Gateway and Lambda to preprocess images and call the ML model. Deployed the Pneumonia Classifier model with a SageMaker serverless endpoint for real-time inference. Automated AWS resource provisioning with CloudFormation. Tested end-to-end functionality, logging, and monitored with CloudWatch. ","permalink":"https://zackblog.work/posts/mlops-deploy-classifier-with-aws-serverless/","summary":"\u003cp\u003eContinue Pneumonia Classifier by moving to AWS Serverless\u003c/p\u003e\n\u003cp\u003eGo AWS Serverless Deployment\u003c/p\u003e\n\u003cp\u003eDeploying machine learning models in production requires additional considerations to address latency, scalability, cost-efficiency, and monitoring.\u003c/p\u003e\n\u003cp\u003eA modern approach to hosting an ML application in AWS can be considered as a \u003ccode\u003eserverless architecture\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eThis allows users to upload images via a S3 static web page and send them to API Gateway. The API Gateway receives the HTTP POST request and forwards it to the Lambda function, which handles the image preprocessing, inference, and postprocessing logic. The Lambda function sends the image payload to the SageMaker endpoint, by calling the SageMaker endpoint using the SageMaker Runtime SDK (\u003ccode\u003einvoke_endpoint\u003c/code\u003e) which hosts my trained model, and then retrieves the prediction. The prediction result is sent back to the frontend for display.\u003c/p\u003e","title":"MLOps - Deploy Classifier with AWS Serverless"},{"content":"Continue Pneumonia Classifier by moving to AWS SageMaker\nIn this post, we will continue Pneumonia Classifier practice by leveraging AWS managed AI service SageMaker. We will:\nProvision AWS Sagemaker AI domain and workspace. Run a JupiterLab to download, process, and upload chest X-ray dataset from Kaggle to S3 for SageMaker access. Use SageMaker Pre-built image classification algorithm, to define SageMaker Estimator to configure the SageMaker training job, including compute resources, training duration, and data input method. Count the number of training samples and set the hyperparameters, perform hyperparameter tuning to find the best configuration for the model. Launching the Hyperparameter Tuning Job. Using CloudWatch and SageMaker Train Job to monitor and troubleshoot. In the AWS SageMaker AI JupiterLab\nI will skip the dataset download as this task remains the same as what I did in the previous post.\nS3 will be the storage where SageMaker will access the dataset. Let\u0026rsquo;s start from data upload.\nNext step is to set up the SageMaker estimator to define the training job, including compute resources, training duration, and data input method. We will use the built-in algorithm for image classification.\n# Set up SageMaker framework, execution_role and S3 location import sagemaker from sagemaker import image_uris import boto3 from sagemaker import get_execution_role sess=sagemaker.Session() algorithm_image=image_uris.retrieve( region=boto3.Session().region_name, framework=\u0026#34;image-classification\u0026#34;, version=\u0026#34;latest\u0026#34; ) s3_output_location=f\u0026#34;s3://{bucket}/models/image_model\u0026#34; print(algorithm_image) role=get_execution_role() print(role) Set up the SageMaker estimator to define the training job. We will use the built-in algorithm for image classification with ml.g4dn.xlarge as spot GPU instance for job training.\n# Set up SageMaker estimator import sagemaker img_classifier_model=sagemaker.estimator.Estimator( algorithm_image, role=role, instance_count=1, instance_type=\u0026#34;ml.g4dn.xlarge\u0026#34;, use_spot_instances=True, # Enable spot instances max_run=432000, # 5 days (432,000 seconds) max_wait=432000, # Must be \u0026gt;= max_run volume_size=50, input_mode=\u0026#34;File\u0026#34;, output_path=s3_output_location, sagemaker_session=sess ) print(img_classifier_model) Setup the total number of labeled images to define epochs and batch size for training job.\n# Define epochs and batch size import glob count=0 for filepath in glob.glob(\u0026#39;./data/chest_xray/train/*.jpeg\u0026#39;): count+=1 print(count) count = 5216 # Example: Total training images img_classifier_model.set_hyperparameters( image_shape=\u0026#39;3,224,224\u0026#39;, num_classes=\u0026#39;2\u0026#39;, # As string use_pretrained_model=\u0026#39;1\u0026#39;, # As string num_training_samples=str(count), # As string augmentation_type=\u0026#39;crop_color_transform\u0026#39;, epochs=\u0026#39;15\u0026#39;, # As string early_stopping=\u0026#39;True\u0026#39;, # As string early_stopping_min_epochs=\u0026#39;8\u0026#39;, early_stopping_tolerance=\u0026#39;0.0\u0026#39;, early_stopping_patience=\u0026#39;5\u0026#39;, lr_scheduler_factor=\u0026#39;0.1\u0026#39;, lr_scheduler_step=\u0026#39;8,10,12\u0026#39; ) Perform hyperparameter tuning to find the best configuration for the model with metrics to evaluate model quality.\n# Hyperparameter tuning from sagemaker.tuner import CategoricalParameter,ContinuousParameter,HyperparameterTuner hyperparameter_ranges={ \u0026#34;learning_rate\u0026#34;:ContinuousParameter(0.01,0.1), \u0026#34;mini_batch_size\u0026#34;:CategoricalParameter([8,16,32]), \u0026#34;optimizer\u0026#34;:CategoricalParameter([\u0026#34;sgd\u0026#34;,\u0026#34;adam\u0026#34;]) } objective_metric_name=\u0026#34;validation:accuracy\u0026#34; objective_type=\u0026#34;Maximize\u0026#34; max_jobs=5 max_parallel_jobs=1 tuner=HyperparameterTuner(estimator=img_classifier_model, objective_metric_name=objective_metric_name, hyperparameter_ranges=hyperparameter_ranges, objective_type=objective_type, max_jobs=max_jobs, max_parallel_jobs=max_parallel_jobs ) from sagemaker.session import TrainingInput Configuring input data sources by specifying the S3 paths and content types for SageMaker training jobs.\nLaunching the Hyperparameter Tuning Job\n# Start the hyperparameter tuning job with the specified inputs and configurations import time job_name_prefix=\u0026#34;classifier\u0026#34; timestamp=time.strftime(\u0026#34;-%Y-%m-%d-%H-%M-%S\u0026#34;,time.gmtime()) job_name=job_name_prefix+timestamp tuner.fit(inputs=model_inputs,job_name=job_name,logs=True) Monitor the tuning job in the AWS SageMaker console to track progress.\nGo to CloudWatch SageMaker log group to see detailed logs of the training job.\nKey Factors Influencing Runtime\nTotal training runtime and performance influenced by Tesla T4 GPU instance ml.g4dn.xlarge and the total 15 epochs of 5216 training samples.\nTesla T4 typically takes ~0.5-1 second per batch for tasks of this complexity, so the total Hyperparameter Tuning Time: 5 jobs × 41 minutes per job = 205 minutes (3.4 hours).\nIncrease max_parallel_jobs to run multiple jobs concurrently (e.g., max_parallel_jobs=2 would cut the runtime in half). Use a more powerful instance (e.g., ml.p3.2xlarge with a V100 GPU for faster training).\nDeploy the trained model to validate prediction\nCreates a SageMaker model object using the trained model\u0026rsquo;s artifacts (model_data) and algorithm container (image_uri). Deploys the model as a SageMaker endpoint using the deploy() method. Using an instance type (ml.m4.xlarge) to offer endpoint for real-time inference.\nmodel = sagemaker.model.Model( image_uri=algorithm_image, model_data=\u0026#39;s3://sagemaker-bucket-851725491342/models/image_model/classifier-2025-01-26-02-58-03-001-a577816e/output/model.tar.gz\u0026#39;, role=role ) endpoint_name = \u0026#39;zack-super-cool-endpoint\u0026#39; deployment = model.deploy( initial_instance_count=1, instance_type=\u0026#39;ml.m4.xlarge\u0026#39;, endpoint_name=endpoint_name ) Setup and test the endpoint for real-time prediction. Send a testing payload in binary mode to the endpoint for prediction.\nfrom sagemaker.predictor import Predictor predictor = Predictor(\u0026#34;zack-super-cool-endpoint\u0026#34;) from sagemaker.serializers import IdentitySerializer import base64 file_name = \u0026#39;data/chest_xray/val/val_normal0.jpeg\u0026#39; predictor.serializer = IdentitySerializer(\u0026#34;image/jpeg\u0026#34;) with open(file_name, \u0026#34;rb\u0026#34;) as f: payload = f.read() inference = predictor.predict(data=payload) print(inference) Output: b\u0026#39;[0.8592441082997322, 0.14075589179992676]\u0026#39; print(inference[1]) Output: 48 Run a batch prediction and evaluate the matrix. Now let\u0026rsquo;s loop through all images in the validation dataset, send each image to the endpoint for prediction, collect predictions for all validation images to evaluate the model\u0026rsquo;s overall performance, and print the classification report.\nimport glob import json import numpy as np file_path = \u0026#39;data/chest_xray/val/*.jpeg\u0026#39; files = glob.glob(file_path) y_true = [] y_pred = [] def make_pred(): for file in files: if \u0026#34;normal\u0026#34; in file: with open(file, \u0026#34;rb\u0026#34;) as f: payload = f.read() inference = predictor.predict(data=payload).decode(\u0026#34;utf-8\u0026#34;) result = json.loads(inference) predicted_class = np.argmax(result) y_true.append(0) # Normal class y_pred.append(predicted_class) elif \u0026#34;pneumonia\u0026#34; in file: with open(file, \u0026#34;rb\u0026#34;) as f: payload = f.read() inference = predictor.predict(data=payload).decode(\u0026#34;utf-8\u0026#34;) result = json.loads(inference) predicted_class = np.argmax(result) y_true.append(1) # Pneumonia class y_pred.append(predicted_class) make_pred() print(y_true) print(y_pred) Output: [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0,] [0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0,] # Evaluate the metrics from sklearn.metrics import confusion_matrix confusion_matrix(y_true, y_pred) Output: array([[6, 2], [0, 8]]) # print classification report from sklearn.metrics import classification_report print(classification_report(y_true, y_pred)) Output: precision recall f1-score support 0 1.00 0.75 0.86 8 1 0.80 1.00 0.89 8 accuracy 0.88 16 macro avg 0.90 0.88 0.87 16 weighted avg 0.90 0.88 0.87 16 Result Analysis\nThe classification report shows the following metrics:\nClass Precision Recall F1-Score Support 0 1.00 0.75 0.86 8 1 0.80 1.00 0.89 8 Accuracy: 0.88 (88%)\nMacro Avg: Precision = 0.90, Recall = 0.88, F1-Score = 0.87\nWeighted Avg: Precision = 0.90, Recall = 0.88, F1-Score = 0.87\nThe confusion matrix is:\nPredicted 0 Predicted 1 Actual 0 6 2 Actual 1 0 8 True Positives (TP): 8 (correctly predicted pneumonia)\nTrue Negatives (TN): 6 (correctly predicted normal)\nFalse Positives (FP): 2 (normal misclassified as pneumonia)\nFalse Negatives (FN): 0 (pneumonia misclassified as normal)\nConclusion\nMove image classification model to cloud ML service using AWS SageMaker, integrated with AWS services (IAM, S3, SageMaker, CloudWatch) for data storage and model deployment and monitoring. Set up a cloud-based real-time image prediction endpoint for the trained model. Test the endpoint with an individual input image to confirm functionality. Run predictions on a validation dataset to evaluate model accuracy and robustness. Use confusion matrices and classification reports to assess the quality of predictions and identify areas for improvement. ","permalink":"https://zackblog.work/posts/mlops-move-image-classifier-to-aws-sagemaker/","summary":"\u003cp\u003eContinue Pneumonia Classifier by moving to AWS SageMaker\u003c/p\u003e\n\u003cp\u003eIn this post, we will continue Pneumonia Classifier practice by leveraging AWS managed AI service SageMaker. We will:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eProvision AWS Sagemaker AI domain and workspace.\u003c/li\u003e\n\u003cli\u003eRun a JupiterLab to download, process, and upload chest X-ray dataset from Kaggle to S3 for SageMaker access.\u003c/li\u003e\n\u003cli\u003eUse SageMaker Pre-built image classification algorithm, to define SageMaker Estimator to configure the SageMaker training job, including compute resources, training duration, and data input method.\u003c/li\u003e\n\u003cli\u003eCount the number of training samples and set the hyperparameters, perform hyperparameter tuning to find the best configuration for the model.\u003c/li\u003e\n\u003cli\u003eLaunching the Hyperparameter Tuning Job.\u003c/li\u003e\n\u003cli\u003eUsing CloudWatch and SageMaker Train Job to monitor and troubleshoot.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eIn the AWS SageMaker AI JupiterLab\u003c/strong\u003e\u003c/p\u003e","title":"MLOps - Move Image Classifier to AWS SageMaker"},{"content":" Last post we were able to build and train the pneumonia classifier model. In this post we are moving a step forward to: Containerizing the model and creating a frontend-backend application to allow users to upload images and get predictions. Overview of the Application Architecture\nA high-level diagram or description of the system architecture:\nBackend: A Dockerized Flask/FastAPI application serving the trained model. Frontend: A Dockerized React/Streamlit app for image upload and displaying predictions. Interaction: Frontend sends images to the backend, which processes them and returns predictions. Step 1: Containerizing the Backend\nObjective: Package the trained model into a Docker container. Steps: Save the Model: Ensure the trained model is saved (e.g., local_image_classifier_model.pth). Create a Flask/FastAPI App: Write a simple API endpoint to accept image uploads and return predictions. Example: /predict endpoint. Dockerize the Backend: Write a Dockerfile for the backend. Build and run the Docker container. Test the Backend: Use tools like curl or Postman to test the API. # build folder structure for frontend and backend applications root@zackz:/mnt/f/zack-gitops-project/image-class# tree . ├── StartingNotebook.ipynb ├── cv1.ipynb ├── cv1.py ├── docker-compose.yml ├── frontend │ ├── Dockerfile │ ├── index.html │ ├── script.js │ └── style.css └── model-docker ├── Dockerfile ├── app.py ├── local_image_classifier_model.pth └── requirements.txt ==========================================================\n# vim requirements.txt Flask==2.3.2 torch==2.0.1 torchvision==0.15.2 Pillow==10.0.0 numpy\u0026lt;2 flask-cors==4.0.0 # vim app.py from flask import Flask, request, jsonify from flask_cors import CORS # Import CORS import torch from torchvision import transforms from PIL import Image import torch.nn as nn import torchvision.models as models # Initialize Flask app app = Flask(__name__) CORS(app) # Enable CORS for all routes # Load the trained model model = models.resnet18(pretrained=False) model.fc = nn.Linear(model.fc.in_features, 1) model.load_state_dict(torch.load(\u0026#34;local_image_classifier_model.pth\u0026#34;, map_location=torch.device(\u0026#39;cpu\u0026#39;))) model.eval() # Define the same transformations used during training transform = transforms.Compose([ transforms.Resize((224, 224)), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]) ]) # Prediction function def predict_image(image_path): image = Image.open(image_path).convert(\u0026#39;RGB\u0026#39;) image = transform(image).unsqueeze(0) with torch.no_grad(): output = model(image) prediction = torch.sigmoid(output).item() return \u0026#34;Pneumonia (Positive)\u0026#34; if prediction \u0026gt; 0.5 else \u0026#34;Normal (Negative)\u0026#34; # Define the endpoint @app.route(\u0026#39;/predict\u0026#39;, methods=[\u0026#39;POST\u0026#39;]) def predict(): if \u0026#39;file\u0026#39; not in request.files: return jsonify({\u0026#34;error\u0026#34;: \u0026#34;No file provided\u0026#34;}), 400 file = request.files[\u0026#39;file\u0026#39;] if file.filename == \u0026#39;\u0026#39;: return jsonify({\u0026#34;error\u0026#34;: \u0026#34;No file selected\u0026#34;}), 400 # Save the uploaded file temporarily file_path = \u0026#34;temp_image.jpeg\u0026#34; file.save(file_path) # Make a prediction result = predict_image(file_path) return jsonify({\u0026#34;prediction\u0026#34;: result}) # Run the Flask app if __name__ == \u0026#39;__main__\u0026#39;: app.run(host=\u0026#39;0.0.0.0\u0026#39;, port=5000) ==========================================================\n# vim Dockerfile # Use an official Python runtime as a parent image FROM python:3.9-slim # Set the working directory in the container WORKDIR /app # Copy the requirements file into the container COPY requirements.txt . # Install any needed packages specified in requirements.txt RUN pip install --no-cache-dir -r requirements.txt # Copy the current directory contents into the container COPY . . # Copy the model file into the container COPY local_image_classifier_model.pth . # Expose port 5000 for the Flask app EXPOSE 5000 # Run the Flask app CMD [\u0026#34;python\u0026#34;, \u0026#34;app.py\u0026#34;] ==========================================================\n# build image docker build -t pneumonia-classifier-1 . Step 2: Building the Frontend\nObjective: Create a user-friendly interface for uploading images and displaying predictions. Steps: Create the Frontend App: React as framework, When a user selects an image, the FileReader API reads the file and displays it in the #previewImage element. When the form is submitted, the selected image is sent to the backend API (http://localhost:5000/predict) using a POST request. The image is sent as multipart/form-data. API Response Handling: If the API call is successful, the prediction result is displayed in the #result div. If there’s an error (e.g., no file selected or API failure), an error message is displayed. Dockerize the Frontend: Write a Dockerfile for the frontend. Build and run the Docker container. ========================================================== # vim script.js body { font-family: Arial, sans-serif; background-color: #f4f4f4; display: flex; justify-content: center; align-items: center; height: 100vh; margin: 0; } .container { background: white; padding: 20px; border-radius: 8px; box-shadow: 0 0 10px rgba(0, 0, 0, 0.1); text-align: center; max-width: 600px; width: 100%; } h1 { margin-bottom: 20px; } form { margin-bottom: 20px; } #imagePreview { margin-top: 20px; } #result { margin-top: 20px; font-size: 1.2em; color: #333; } ========================================================== # vim script.js document.getElementById(\u0026#39;imageUpload\u0026#39;).addEventListener(\u0026#39;change\u0026#39;, function (e) { const file = e.target.files[0]; if (file) { const reader = new FileReader(); reader.onload = function (e) { const previewImage = document.getElementById(\u0026#39;previewImage\u0026#39;); previewImage.src = e.target.result; previewImage.style.display = \u0026#39;block\u0026#39;; // Show the image }; reader.readAsDataURL(file); // Read the file as a data URL } }); document.getElementById(\u0026#39;uploadForm\u0026#39;).addEventListener(\u0026#39;submit\u0026#39;, async function (e) { e.preventDefault(); const fileInput = document.getElementById(\u0026#39;imageUpload\u0026#39;); const resultDiv = document.getElementById(\u0026#39;result\u0026#39;); if (fileInput.files.length === 0) { resultDiv.textContent = \u0026#39;Please select an image.\u0026#39;; return; } const file = fileInput.files[0]; const formData = new FormData(); formData.append(\u0026#39;file\u0026#39;, file); try { const response = await fetch(\u0026#39;http://localhost:5000/predict\u0026#39;, { method: \u0026#39;POST\u0026#39;, body: formData, }); if (!response.ok) { throw new Error(\u0026#39;Failed to get prediction\u0026#39;); } const data = await response.json(); resultDiv.textContent = `Prediction: ${data.prediction}`; } catch (error) { resultDiv.textContent = \u0026#39;Error: \u0026#39; + error.message; } }); ========================================================== # vim Dockerfile # Use an official Nginx image as the base image FROM nginx:alpine # Copy the frontend files to the Nginx HTML directory COPY . /usr/share/nginx/html # Expose port 80 for the web server EXPOSE 80 # Start Nginx when the container runs CMD [\u0026#34;nginx\u0026#34;, \u0026#34;-g\u0026#34;, \u0026#34;daemon off;\u0026#34;] ==========================================================\n# build the image docker build -t pneumonia-frontend . Step 3: Connecting Frontend and Backend\nObjective: Make the frontend and backend communicate seamlessly. Steps: Network Configuration: Use Docker Compose to manage both containers. Ensure the frontend can reach the backend API. End-to-End Testing: Upload an image via the frontend and verify the prediction is displayed correctly. ========================================================== # vim docker-compose.yml version: \u0026#39;3.8\u0026#39; services: backend: image: pneumonia-classifier-1 ports: - \u0026#34;5000:5000\u0026#34; networks: - pneumonia-net frontend: image: pneumonia-frontend ports: - \u0026#34;8080:80\u0026#34; depends_on: - backend networks: - pneumonia-net networks: pneumonia-net: driver: bridge Step 4: Deployment\nRun the Docker containers locally, with plans to deploy to a cloud platform later with Docker, Kubernetes, and CI/CD pipeline.\nroot@zackz:/mnt/f/ml-local/local-cv# docker-compose up WARN[0000] /mnt/f/ml-local/local-cv/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion [+] Running 2/2 ✔ Container local-cv-backend-1 Created 0.1s ✔ Container local-cv-frontend-1 Created 0.1s Attaching to backend-1, frontend-1 frontend-1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration frontend-1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ frontend-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh frontend-1 | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf frontend-1 | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf frontend-1 | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh frontend-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh frontend-1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh frontend-1 | /docker-entrypoint.sh: Configuration complete; ready for start up frontend-1 | 2025/01/25 13:30:22 [notice] 1#1: using the \u0026#34;epoll\u0026#34; event method frontend-1 | 2025/01/25 13:30:22 [notice] 1#1: nginx/1.27.3 frontend-1 | 2025/01/25 13:30:22 [notice] 1#1: built by gcc 13.2.1 20240309 (Alpine 13.2.1_git20240309) frontend-1 | 2025/01/25 13:30:22 [notice] 1#1: OS: Linux 5.15.153.1-microsoft-standard-WSL2 frontend-1 | 2025/01/25 13:30:22 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 frontend-1 | 2025/01/25 13:30:22 [notice] 1#1: start worker processes frontend-1 | 2025/01/25 13:30:22 [notice] 1#1: start worker process 30 backend-1 | /usr/local/lib/python3.9/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter \u0026#39;pretrained\u0026#39; is deprecated since 0.13 and may be removed in the future, please use \u0026#39;weights\u0026#39; instead. backend-1 | warnings.warn( backend-1 | /usr/local/lib/python3.9/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for \u0026#39;weights\u0026#39; are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=None`. backend-1 | warnings.warn(msg) backend-1 | * Serving Flask app \u0026#39;app\u0026#39; backend-1 | * Debug mode: off backend-1 | WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. backend-1 | * Running on all addresses (0.0.0.0) backend-1 | * Running on http://127.0.0.1:5000 backend-1 | * Running on http://172.18.0.2:5000 backend-1 | Press CTRL+C to quit ==========================================================\n# test endpoint root@zackz:/mnt/f/ml-local/local-cv/# curl -X POST -F \u0026#34;file=@data/chest_xray/zz/zz2.jpeg\u0026#34; http://localhost:5000/predict {\u0026#34;prediction\u0026#34;:\u0026#34;Pneumonia (Positive)\u0026#34;} root@zackz:/mnt/f/ml-local/local-cv/# curl -X POST -F \u0026#34;file=@data/chest_xray/zz/zz3.jpeg\u0026#34; http://localhost:5000/predict {\u0026#34;prediction\u0026#34;:\u0026#34;Normal (Negative)\u0026#34;} Frontend testing by providing a Google searched chest X-Ray image to the frontend and see the backend response for Pneumonia prediction.\nKey Takeaways\nSuccessfully built and containerized the pneumonia classifier model and created a functional frontend-backend application with portability and scalability. Successfully tested the application locally by uploading a chest X-ray image and receiving a prediction from the backend. Future work: Model Monitoring using tools like Prometheus, Grafana, or MLflow Fine-Tuning with Transfer Learning: Fine-tune a pre-trained model on a larger dataset to get even better accuracy Cloud Deployment with GPU Optimization: Optimize GPU usage in the public cloud for inference to reduce latency and cost. ","permalink":"https://zackblog.work/posts/mlops-containerize-classifier-application/","summary":"\u003cul\u003e\n\u003cli\u003eLast post we were able to build and train the pneumonia classifier model.\u003c/li\u003e\n\u003cli\u003eIn this post we are moving a step forward to: \u003cstrong\u003eContainerizing the model\u003c/strong\u003e and creating a \u003cstrong\u003efrontend-backend application\u003c/strong\u003e to allow users to upload images and get predictions.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eOverview of the Application Architecture\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eA high-level diagram or description of the system architecture:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eBackend\u003c/strong\u003e: A Dockerized Flask/FastAPI application serving the trained model.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFrontend\u003c/strong\u003e: A Dockerized React/Streamlit app for image upload and displaying predictions.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eInteraction\u003c/strong\u003e: Frontend sends images to the backend, which processes them and returns predictions.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eStep 1: Containerizing the Backend\u003c/strong\u003e\u003c/p\u003e","title":"MLOps - Containerize Classifier Application"},{"content":"Pneumonia causes over 2.5 million deaths annually worldwide. Traditional diagnosis through chest X-ray analysis is time-consuming and requires expert radiologists. This project aims to develop an automated deep learning system to classify pneumonia from chest X-rays with high accuracy, potentially assisting healthcare professionals in faster diagnosis.\nPurpose\nHere I will use a local environment with Pytorch and Jupyter Notebook to:\nDemonstrate end-to-end development of a medical image classifier Using Kaggle dataset for NIH Chest X-Ray image processing Create a baseline model for pneumonia detection (Normal vs Pneumonia) Local GPU acceleration for model training and prediction Typical ML steps to achieve image classifier:\nData Acquisition: Source NIH Chest X-Ray dataset from Kaggle 5,863 validated images (Train/Test/Val split) Preprocessing: Standardize image size (224x224px) Normalize pixel values Organize into class-specific directories Exploratory Analysis: Class distribution visualization Sample image inspection Model Development: Leverage pre-trained ResNet18 Custom head for binary classification GPU-accelerated training Evaluation: Accuracy metrics Model persistence Implementation Steps\nGet Kaggle Chest X-Ray dataset # prepare data set from kaggle !pip install -q kaggle !python -m pip install --upgrade pip !mkdir kaggle !touch kaggle/kaggle.json !chmod 600 kaggle/kaggle.json api_token = {\u0026#34;username\u0026#34;:\u0026#34;zhouzack\u0026#34;,\u0026#34;key\u0026#34;:\u0026#34;\u0026#34;} import json with open(\u0026#39;kaggle/kaggle.json\u0026#39;,\u0026#39;w\u0026#39;) as file: json.dump(api_token,file) !kaggle datasets download -d paultimothymooney/chest-xray-pneumonia --force Output: Dataset URL: https://www.kaggle.com/datasets/paultimothymooney/chest-xray-pneumonia License(s): other Downloading chest-xray-pneumonia.zip to /workspace 100%|██████████████████████████████████████| 2.29G/2.29G [03:32\u0026lt;00:00, 11.4MB/s] 100%|██████████████████████████████████████| 2.29G/2.29G [03:32\u0026lt;00:00, 11.6MB/s] # Extract the zip file to the \u0026#34;data\u0026#34; directory import zipfile import os # Create the \u0026#34;data\u0026#34; directory in current folder if it doesn\u0026#39;t exist os.makedirs(\u0026#39;./data\u0026#39;, exist_ok=True) # \u0026#34;./data\u0026#34; = \u0026#34;data\u0026#34; folder in your current directory # Corrected code (ZipFile instead of Zipfile) with zipfile.ZipFile(\u0026#39;chest-xray-pneumonia.zip\u0026#39;, \u0026#39;r\u0026#39;) as zip_ref: zip_ref.extractall(\u0026#39;./data\u0026#39;) # Extract to ./data (relative path) Test a random image from folder # Test a random image from folder import glob import random import matplotlib.pyplot as plt def get_random_image(dir,condition): placeholder=\u0026#39;\u0026#39; if condition == \u0026#39;n\u0026#39;: placeholder=\u0026#39;NORMAL\u0026#39; elif condition == \u0026#39;p\u0026#39;: placeholder=\u0026#39;PNEUMONIA\u0026#39; else: raise Exception(\u0026#34;Sorry, invalid condition\u0026#34;) folder=f\u0026#39;./data/chest_xray/{dir}/{placeholder}/*.jpeg\u0026#39; img_paths=glob.glob(folder) max_length=len(img_paths) randomNumber=random.randint(0,max_length) for index, item in enumerate(img_paths, start=1): if index == randomNumber: print(index,item) image = plt.imread(item) readyImage=plt.imshow(image) return readyImage get_random_image(\u0026#34;val\u0026#34;,\u0026#34;n\u0026#34;) Image processing, load image and Prints its format, converts image from RGBA to RGB, using Matplotlib for a cleaner figure size view. #loads and Prints the image format from PIL import Image # Replace \u0026#39;path/to/your/image.jpg\u0026#39; with your actual image file path image = Image.open(\u0026#39;./data/chest_xray/val/PNEUMONIA/person1947_bacteria_4876.jpeg\u0026#39;) print(image.format) print(image.size) print(image.mode) Output: JPEG (1152, 664) L # converts image from RGBA (Red, Green, Blue, Alpha) format to RGB (Red, Green, Blue) format. import PIL.Image rgba_image = PIL.Image.open(\u0026#39;./data/chest_xray/val/NORMAL/NORMAL2-IM-1436-0001.jpeg\u0026#39;) rgb_image = rgba_image.convert(\u0026#39;RGB\u0026#39;) # Reads an image using Matplotlib import matplotlib.pyplot as plt import matplotlib.image as mpimg # Provide the correct path to your image file # Replace with your actual image path img = mpimg.imread(\u0026#39;./data/chest_xray/val/NORMAL/NORMAL2-IM-1436-0001.jpeg\u0026#39;) # Display the image plt.figure(figsize=(10,8)) # Optional: set figure size imgplot = plt.imshow(img) plt.axis(\u0026#39;off\u0026#39;) # Optional: hide axes plt.show() Resizes and saves validation images into val_pneumonia and val_normal folders # Resizes and saves validation images into val_pneumonia and val_normal folders import glob import matplotlib.pyplot as plt from PIL import Image folder = f\u0026#39;./data/chest_xray/val/*/*.jpeg\u0026#39; counterPneu = 0 counterNormal = 0 img_paths = glob.glob(folder) for i in img_paths: if \u0026#34;person\u0026#34; in i: full_size_image = Image.open(i) im = full_size_image.resize((224,224)) plt.imsave(fname=\u0026#39;./data/chest_xray/val\u0026#39; + \u0026#39;/val_pneumonia\u0026#39; + str(counterPneu)+\u0026#39;.jpeg\u0026#39;, arr=im, format=\u0026#39;jpeg\u0026#39;, cmap=\u0026#39;gray\u0026#39;) counterPneu += 1 else: full_size_image = Image.open(i) im = full_size_image.resize((224,224)) plt.imsave(fname=\u0026#39;./data/chest_xray/val\u0026#39; + \u0026#39;/val_normal\u0026#39; + str(counterNormal)+\u0026#39;.jpeg\u0026#39;, arr=im, format=\u0026#39;jpeg\u0026#39;, cmap=\u0026#39;gray\u0026#39;) counterNormal += 1 Output: Processed 3875 Pneumonia images and 1341 Normal images. creates a DataFrame to organize the dataset by type (train, test, val) and condition (pneumonia, normal) # creates a DataFrame to organize the dataset by type (train, test, val) and condition (pneumonia, normal) import glob import pandas as pd folder = f\u0026#39;./data/chest_xray/*/*.jpeg\u0026#39; category = [] filenames = [] condition_of_lung = [] all_files = glob.glob(folder) for filename in all_files: if \u0026#34;train\u0026#34; in filename: if \u0026#34;pneumonia\u0026#34; in filename: category.append(\u0026#34;train\u0026#34;) filenames.append(filename) condition_of_lung.append(\u0026#34;pneumonia\u0026#34;) elif \u0026#34;normal\u0026#34; in filename: category.append(\u0026#34;train\u0026#34;) filenames.append(filename) condition_of_lung.append(\u0026#34;normal\u0026#34;) elif \u0026#34;test\u0026#34; in filename: if \u0026#34;pneumonia\u0026#34; in filename: category.append(\u0026#34;test\u0026#34;) filenames.append(filename) condition_of_lung.append(\u0026#34;pneumonia\u0026#34;) elif \u0026#34;normal\u0026#34; in filename: category.append(\u0026#34;test\u0026#34;) filenames.append(filename) condition_of_lung.append(\u0026#34;normal\u0026#34;) elif \u0026#34;val\u0026#34; in filename: if \u0026#34;pneumonia\u0026#34; in filename: category.append(\u0026#34;val\u0026#34;) filenames.append(filename) condition_of_lung.append(\u0026#34;pneumonia\u0026#34;) elif \u0026#34;normal\u0026#34; in filename: category.append(\u0026#34;val\u0026#34;) filenames.append(filename) condition_of_lung.append(\u0026#34;normal\u0026#34;) all_data_df = pd.DataFrame({\u0026#34;dataset type\u0026#34;: category, \u0026#34;x-ray result\u0026#34;: condition_of_lung, \u0026#34;filename\u0026#34;: filenames}) print(all_data_df.head()) Output: dataset type x-ray result filename 0 test normal ./data/chest_xray/test/test_normal0.jpeg 1 test normal ./data/chest_xray/test/test_normal1.jpeg 2 test normal ./data/chest_xray/test/test_normal10.jpeg 3 test normal ./data/chest_xray/test/test_normal100.jpeg 4 test normal ./data/chest_xray/test/test_normal101.jpeg visualizes the distribution of pneumonia and normal cases across the train, test, and validation datasets import seaborn as sns # Use `hue` with the same variable as `x` and set `legend=False` g = sns.catplot( x=\u0026#34;x-ray result\u0026#34;, # Variable for the x-axis col=\u0026#34;dataset type\u0026#34;, # Facet by dataset type (train, test, val) kind=\u0026#34;count\u0026#34;, # Plot counts palette=\u0026#34;ch:.55\u0026#34;, # Set color palette data=all_data_df, # Data source hue=\u0026#34;x-ray result\u0026#34;, # Assign `x` to `hue` to use `palette` legend=False # Avoid duplicate legend ) # Add annotations to the bars for i in range(0, 3): ax = g.facet_axis(0, i) for p in ax.patches: ax.text( p.get_x() + 0.3, # X position of the text p.get_height() * 1.05, # Y position of the text (slightly above the bar) \u0026#39;{0:.0f}\u0026#39;.format(p.get_height()), # Text to display (bar height) color=\u0026#39;black\u0026#39;, # Text color rotation=\u0026#39;horizontal\u0026#39;, # Text rotation size=\u0026#39;large\u0026#39; # Text size ) creates DataFrames for the training and testing datasets, labeling images as pneumonia (1) or normal (0) # Create DataFrames for the training and testing datasets import glob import pandas as pd import os train_folder = \u0026#39;./data/chest_xray/train/*.jpeg\u0026#39; train_df_lst = pd.DataFrame(columns=[\u0026#39;labels\u0026#39;, \u0026#39;filename\u0026#39;], dtype=object) train_imgs_path = glob.glob(train_folder) counter = 0 class_arg = \u0026#39;\u0026#39; for i in train_imgs_path: if \u0026#34;pneumonia\u0026#34; in i: class_arg = 1 else: class_arg = 0 train_df_lst.loc[counter] = [class_arg, os.path.basename(i)] counter += 1 print(train_df_lst.head()) save DataFrame with labels and filenames into a tab-separated .lst file # Save DataFrame with labels and filenames into a tab-separated .lst file def save_to_lst(df,prefix): return df[[\u0026#34;labels\u0026#34;,\u0026#34;filename\u0026#34;]].to_csv( f\u0026#34;{prefix}.lst\u0026#34;, sep=\u0026#39;\\t\u0026#39;,index=True,header=False ) save_to_lst(train_df_lst.copy(),\u0026#34;train\u0026#34;) save_to_lst(test_df_lst.copy(),\u0026#34;test\u0026#34;) install libraries for data and image and preprocessing !pip install torch torchvision pandas pillow Output: Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (2.3.0a0+6ddf5cf85e.nv24.4) Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (0.18.0a0) Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (1.5.3) Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (10.2.0) Setup model for training # Define the device device = torch.device(\u0026#34;cuda\u0026#34; if torch.cuda.is_available() else \u0026#34;cpu\u0026#34;) print(f\u0026#34;Using device: {device}\u0026#34;) # Load a pre-trained ResNet18 model model = models.resnet18(pretrained=True) # Modify the final fully connected layer for binary classification model.fc = nn.Linear(model.fc.in_features, 1) # Move the model to the appropriate device model = model.to(device) Train the model # Define the loss function (BCEWithLogitsLoss) criterion = nn.BCEWithLogitsLoss() # Define the optimizer optimizer = optim.Adam(model.parameters(), lr=0.0001) # Training loop num_epochs = 10 for epoch in range(num_epochs): model.train() running_loss = 0.0 for i, (images, labels) in enumerate(train_loader): images, labels = images.to(device), labels.to(device) # Zero the parameter gradients optimizer.zero_grad() # Forward pass outputs = model(images) loss = criterion(outputs, labels.float().view(-1, 1)) # Backward pass and optimize loss.backward() optimizer.step() running_loss += loss.item() if i % 10 == 9: # Print every 10 batches print(f\u0026#34;Epoch [{epoch+1}/{num_epochs}], Batch [{i+1}/{len(train_loader)}], Loss: {running_loss/10:.4f}\u0026#34;) running_loss = 0.0 print(f\u0026#34;Epoch [{epoch+1}/{num_epochs}], Loss: {running_loss/len(train_loader):.4f}\u0026#34;) Save the trained model locally # Save the model locally torch.save(model.state_dict(), \u0026#34;local_image_classifier_model.pth\u0026#34;) Key Takeaways:\nNow we have a functional pneumonia classifier with high accuracy, ready for deployment or further refinement.\nNext step I will containerize the model as a backend ML application and create a frontend app to interact with the classifier model\u0026rsquo;s endpoint. This will allow users to upload images via the frontend and receive predictions from the backend.\n","permalink":"https://zackblog.work/posts/mlops-build-a-image-classifier-model/","summary":"\u003cp\u003ePneumonia causes over 2.5 million deaths annually worldwide. Traditional diagnosis through chest X-ray analysis is time-consuming and requires expert radiologists. This project aims to develop an automated deep learning system to classify pneumonia from chest X-rays with high accuracy, potentially assisting healthcare professionals in faster diagnosis.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePurpose\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eHere I will use a local environment with Pytorch and Jupyter Notebook to:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDemonstrate end-to-end development of a medical image classifier\u003c/li\u003e\n\u003cli\u003eUsing Kaggle dataset for NIH Chest X-Ray image processing\u003c/li\u003e\n\u003cli\u003eCreate a baseline model for pneumonia detection (Normal vs Pneumonia)\u003c/li\u003e\n\u003cli\u003eLocal GPU acceleration for model training and prediction\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eTypical ML steps to achieve image classifier:\u003c/strong\u003e\u003c/p\u003e","title":"MLOps - Build a Image Classifier Model"},{"content":"Machine Learning workload deployed in K8S and minikube!!\nSo far the local ML practice is just the beginning, in a real world production environment, ML projects typically follow a structured lifecycle and often deployed in scalable, cloud-based environments. Cloud Providers like AWS with Managed Kubernetes Services (EKS) provide orchestration, scaling, and fault tolerance to handle ML workloads.\nPath for deploying MLOps workload in K8S\nTransition from local ML practice to a K8S-based deployment (This post) Start with MLOps tools like Kubeflow. Shift from local to cloud platforms (AWS) to deploy ML workload on EKS. Practice deploying models using REST APIs (local) and APT Gateway (AWS). Try local data engineering (ETL pipelines, data lakes, etc.), then move to AWS data services and solutions for ML workload. Setting up Minikube with GPU on WSL Ubuntu\nFirst, let\u0026rsquo;s create a Minikube cluster with GPU support on WSL Ubuntu. This will be the local K8S environment for testing and deploying ML workloads.\n# Install Minikube sudo apt-get update curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64 sudo install minikube-linux-amd64 /usr/local/bin/minikube # Start Minikube with the Docker driver and GPU support minikube start --driver docker --container-runtime docker --gpus all --force --cpus=8 --memory=16g --addons=nvidia-gpu-device-plugin # Verify Minikube addon with Nvidia GPU root@zackz:/mnt/f/ml-local/local-minikube/complex# minikube addons list | grep NVIDIA | nvidia-device-plugin | minikube | enabled ✅ | 3rd party (NVIDIA) | | nvidia-driver-installer | minikube | disabled | 3rd party (NVIDIA) | | nvidia-gpu-device-plugin | minikube | disabled | 3rd party (NVIDIA) | # Verify Minikube node root@zackz:/mnt/f/ml-local/local-minikube# kubectl get node NAME STATUS ROLES AGE VERSION minikube Ready control-plane 34m v1.32.0 # Verify Minikube node GPU capacity root@zackz:/mnt/f/ml-local/local-minikube# kubectl describe node $(kubectl get nodes -o name | cut -d\u0026#39;/\u0026#39; -f2) | grep -A 10 \u0026#34;Capacity\u0026#34; Capacity: cpu: 20 ephemeral-storage: 1055762868Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 49238360Ki nvidia.com/gpu: 1 pods: 110 Allocatable: cpu: 20 ephemeral-storage: 1055762868Ki Next step, once the K8S is ready, let\u0026rsquo;s run a GPU pod to test if a K8S pod can access GPU.\n# vim gpu-stes.yaml apiVersion: v1 kind: Pod metadata: name: gpu-pod spec: containers: - name: cuda-container image: nvidia/cuda:12.6.0-base-ubuntu22.04 resources: limits: nvidia.com/gpu: 1 # Request 1 GPU command: [\u0026#34;nvidia-smi\u0026#34;] root@zackz:/mnt/f/ml-local/local-minikube# kubectl apply -f gpu-test.yaml pod/gpu-pod created root@zackz:/mnt/f/ml-local/local-minikube# kubectl get po -A NAMESPACE NAME READY STATUS RESTARTS AGE default gpu-pod 0/1 Completed 1 (2s ago) 3s kube-system coredns-668d6bf9bc-2nwph 1/1 Running 0 3m1s kube-system etcd-minikube 1/1 Running 0 3m7s kube-system kube-apiserver-minikube 1/1 Running 0 3m7s kube-system kube-controller-manager-minikube 1/1 Running 0 3m6s kube-system kube-proxy-vblkm 1/1 Running 0 3m1s kube-system kube-scheduler-minikube 1/1 Running 0 3m6s kube-system nvidia-device-plugin-daemonset-72jwz 1/1 Running 0 3m1s kube-system storage-provisioner 1/1 Running 1 (2m39s ago) 3m5s root@zackz:/mnt/f/ml-local/local-minikube# kubectl logs gpu-pod Fri Jan 24 22:55:29 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 560.35.02 Driver Version: 560.94 CUDA Version: 12.6 | |-----------------------------------------+------------------------+----------------------| | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3070 Ti On | 00000000:01:00.0 On | N/A | | 0% 53C P8 17W / 186W | 1736MiB / 8192MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------| +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 27 G /Xwayland N/A | | 0 N/A N/A 37 G /Xwayland N/A | +-----------------------------------------------------------------------------------------+ Now let\u0026rsquo;s create a more complex GPU workload that trains a simple CNN on the MNIST dataset, with a ResourceQuota and LimitRange to allocate the GPU in minikube, we will use the tensorflow/tensorflow:2.14.0-gpu to better support CUDA and NVIDIA drivers to complete the training job.\n# MNIST-Training.yaml apiVersion: apps/v1 kind: Deployment metadata: name: mnist-gpu-training namespace: gpu-workloads spec: replicas: 1 selector: matchLabels: app: mnist-gpu-training template: metadata: labels: app: mnist-gpu-training spec: containers: - name: tensorflow image: tensorflow/tensorflow:2.14.0-gpu resources: limits: nvidia.com/gpu: 1 memory: \u0026#34;8Gi\u0026#34; requests: memory: \u0026#34;4Gi\u0026#34; command: [\u0026#34;python3\u0026#34;] args: - \u0026#34;-c\u0026#34; - | import tensorflow as tf import time # Load and preprocess MNIST data (x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data() x_train = x_train.reshape(-1, 28, 28, 1).astype(\u0026#39;float32\u0026#39;) / 255.0 x_test = x_test.reshape(-1, 28, 28, 1).astype(\u0026#39;float32\u0026#39;) / 255.0 # Build CNN model model = tf.keras.Sequential([ tf.keras.layers.Conv2D(32, 3, activation=\u0026#39;relu\u0026#39;, input_shape=(28, 28, 1)), tf.keras.layers.MaxPooling2D(), tf.keras.layers.Conv2D(64, 3, activation=\u0026#39;relu\u0026#39;), tf.keras.layers.MaxPooling2D(), tf.keras.layers.Flatten(), tf.keras.layers.Dense(128, activation=\u0026#39;relu\u0026#39;), tf.keras.layers.Dense(10, activation=\u0026#39;softmax\u0026#39;) ]) # Compile model model.compile( optimizer=\u0026#39;adam\u0026#39;, loss=\u0026#39;sparse_categorical_crossentropy\u0026#39;, metrics=[\u0026#39;accuracy\u0026#39;] ) print(\u0026#34;Starting training...\u0026#34;) start_time = time.time() # Train model history = model.fit( x_train, y_train, epochs=5, validation_data=(x_test, y_test), batch_size=128 ) end_time = time.time() print(f\u0026#34;\\nTraining completed in {end_time - start_time:.2f} seconds\u0026#34;) # Evaluate model test_loss, test_accuracy = model.evaluate(x_test, y_test) print(f\u0026#34;\\nTest accuracy: {test_accuracy:.4f}\u0026#34;) # gpu-resources.yaml apiVersion: v1 kind: Namespace metadata: name: gpu-workloads --- apiVersion: v1 kind: ResourceQuota metadata: name: gpu-quota namespace: gpu-workloads spec: hard: requests.nvidia.com/gpu: \u0026#34;1\u0026#34; limits.nvidia.com/gpu: \u0026#34;1\u0026#34; --- apiVersion: v1 kind: LimitRange metadata: name: gpu-limits namespace: gpu-workloads spec: limits: - type: Container defaultRequest: nvidia.com/gpu: \u0026#34;1\u0026#34; default: nvidia.com/gpu: \u0026#34;1\u0026#34; max: nvidia.com/gpu: \u0026#34;1\u0026#34; Deploy the MNIST Training and resource quota into minikube.\nroot@zackz:/mnt/f/ml-local/local-minikube/complex# kubectl apply -f gpu-quota.yaml namespace/gpu-workloads created resourcequota/gpu-quota created limitrange/gpu-limits created root@zackz:/mnt/f/ml-local/local-minikube/complex# kubectl apply -f MNIST-Training.yaml deployment.apps/mnist-gpu-training created root@zackz:/mnt/f/ml-local/local-minikube# kubectl get po -A -w NAMESPACE NAME READY STATUS RESTARTS AGE default tensorflow-gpu-test-d445455dc-slsg4 0/1 ContainerCreating 0 2m59s kube-system coredns-668d6bf9bc-2nwph 1/1 Running 0 25m kube-system etcd-minikube 1/1 Running 0 25m kube-system kube-apiserver-minikube 1/1 Running 0 25m kube-system kube-controller-manager-minikube 1/1 Running 0 25m kube-system kube-proxy-vblkm 1/1 Running 0 25m kube-system kube-scheduler-minikube 1/1 Running 0 25m kube-system nvidia-device-plugin-daemonset-72jwz 1/1 Running 0 25m kube-system storage-provisioner 1/1 Running 1 (25m ago) 25m root@zackz:/mnt/f/ml-local/local-minikube/complex# kubectl logs -n gpu-workloads mnist-gpu-training-778fb7bcf7-zsscr 2025-01-24 23:18:15.642781: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz 11490434/11490434 [==============================] - 2s 0us/step 2025-01-24 23:18:19.429788: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1886] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5558 MB memory: -\u0026gt; device: 0, name: NVIDIA GeForce RTX 3070 Ti, pci bus id: 0000:01:00.0, compute capability: 8.6 Starting training... Epoch 1/5 2025-01-24 23:18:20.175141: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:442] Loaded cuDNN version 8600 2025-01-24 23:18:20.399678: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f25deb91fb0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2025-01-24 23:18:20.399719: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA GeForce RTX 3070 Ti, Compute Capability 8.6 2025-01-24 23:18:20.402577: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable. 2025-01-24 23:18:20.461905: I ./tensorflow/compiler/jit/device_compiler.h:186] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process. 469/469 [==============================] - 3s 4ms/step - loss: 0.2068 - accuracy: 0.9409 - val_loss: 0.0664 - val_accuracy: 0.9778 Epoch 2/5 469/469 [==============================] - 2s 3ms/step - loss: 0.0555 - accuracy: 0.9828 - val_loss: 0.0476 - val_accuracy: 0.9849 Epoch 3/5 469/469 [==============================] - 2s 3ms/step - loss: 0.0407 - accuracy: 0.9875 - val_loss: 0.0329 - val_accuracy: 0.9900 Epoch 4/5 469/469 [==============================] - 2s 4ms/step - loss: 0.0304 - accuracy: 0.9905 - val_loss: 0.0385 - val_accuracy: 0.9883 Epoch 5/5 469/469 [==============================] - 2s 4ms/step - loss: 0.0229 - accuracy: 0.9929 - val_loss: 0.0296 - val_accuracy: 0.9899 Training completed in 9.94 seconds 313/313 [==============================] - 1s 2ms/step - loss: 0.0296 - accuracy: 0.9899 Test accuracy: 0.9899 What we\u0026rsquo;ve achieved\nSo here we are able to set up a Minikube with local GPU support, integrated NVIDIA GPU (RTX 3070 Ti), by validating CUDA with TensorFlow GPU integration, deploy a CNN (Convolutional Neural Network) training on MNIST with final test accuracy: 98.99%.\nIn the next post I will explore how to manage Kubeflow for more complex machine learning scenarios and enable Prometheus and Grafana for ML workload monitoring.\n","permalink":"https://zackblog.work/posts/mlops-deploy-ml-workload-to-k8s/","summary":"\u003cp\u003eMachine Learning workload deployed in K8S and minikube!!\u003c/p\u003e\n\u003cp\u003eSo far the local ML practice is just the beginning, in a real world production environment, ML projects typically follow a structured lifecycle and often deployed in scalable, cloud-based environments. Cloud Providers like AWS with Managed Kubernetes Services (EKS) provide orchestration, scaling, and fault tolerance to handle ML workloads.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePath for deploying MLOps workload in K8S\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eTransition from local ML practice to a K8S-based deployment (\u003cstrong\u003eThis post\u003c/strong\u003e)\u003c/li\u003e\n\u003cli\u003eStart with MLOps tools like \u003cstrong\u003eKubeflow\u003c/strong\u003e.\u003c/li\u003e\n\u003cli\u003eShift from local to cloud platforms (AWS) to deploy ML workload on EKS.\u003c/li\u003e\n\u003cli\u003ePractice deploying models using REST APIs (local) and APT Gateway (AWS).\u003c/li\u003e\n\u003cli\u003eTry local data engineering (ETL pipelines, data lakes, etc.), then move to AWS data services and solutions for ML workload.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eSetting up Minikube with GPU on WSL Ubuntu\u003c/strong\u003e\u003c/p\u003e","title":"MLOps - Deploy ML workload to K8S"},{"content":"\u0026lsquo;Deep Seek is really hot at the moment!\nIn this post, I want to build a local Knowledge Base using WSL, Docker, Ollama, Open WebUI and DeepSeek R1 7b.\nOllama is a desktop application designed to run and interact with large language models (LLMs) locally on a machine. It provides an easy interface for downloading, managing, and using various LLMs, ensuring privacy and local execution.\nOpen WebUI is a web-based user interface for interacting with AI models. It is often used in conjunction with locally hosted or remote LLMs, providing a customizable and user-friendly platform to input queries and manage model interactions.\nDeepSeek R1 7b is a large language model developed by DeepSeek, a Chinese AI company. It is designed to understand and generate human-like text based on input prompts. This model can be used for a variety of natural language processing tasks, including text generation, translation, and question answering.\nHere is what I am going to build:\nA local RAG (Retrieval Augmented Generation) system Using Ollama as the LLM server Open WebUI as the frontend interface Running everything through Docker/WSL Build Knowledge base from some aged tech docs and Zack blogs Query KB from the web interface with prompt to DeepSeek R1 Get Started\nSince I already have WSL, Nvidia Drivers, CUDA, Python, Docker Desktop ready by following the previous MLOPS posts, in order to have the docker container access GPU, I still need nvidia-container-toolkit to be installed.\n# a script to install nvidia-container-toolkit # Add NVIDIA GPG key and repository distribution=$(. /etc/os-release;echo $ID$VERSION_ID) curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \\ sed \u0026#39;s#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g\u0026#39; | \\ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list # Install the toolkit sudo apt-get update sudo apt-get install -y nvidia-container-toolkit sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker # verify docker container can access GPU docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu22.04 nvidia-smi Sun Jan 19 21:31:15 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 560.35.02 Driver Version: 560.94 CUDA Version: 12.6 | |-----------------------------------------+------------------------+----------------------| | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3070 Ti On | 00000000:01:00.0 On | N/A | | 0% 52C P8 17W / 186W | 1847MiB / 8192MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------| +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 27 G /Xwayland N/A | | 0 N/A N/A 37 G /Xwayland N/A | +-----------------------------------------------------------------------------------------+ Install Ollama and Open WebUI\nFollow Ollama website, run below script to install Ollama server.\n# Install and run Ollama root@zackz:/mnt/f/ml-local# curl -fsSL https://ollama.com/install.sh | sh \u0026gt;\u0026gt;\u0026gt; Installing ollama to /usr/local \u0026gt;\u0026gt;\u0026gt; Downloading Linux amd64 bundle ######################################################################## 100.0% \u0026gt;\u0026gt;\u0026gt; Creating ollama user... \u0026gt;\u0026gt;\u0026gt; Adding ollama user to render group... \u0026gt;\u0026gt;\u0026gt; Adding ollama user to video group... \u0026gt;\u0026gt;\u0026gt; Adding current user to ollama group... \u0026gt;\u0026gt;\u0026gt; Creating ollama systemd service... \u0026gt;\u0026gt;\u0026gt; Enabling and starting ollama service... Created symlink /etc/systemd/system/default.target.wants/ollama.service → /etc/systemd/system/ollama.service. \u0026gt;\u0026gt;\u0026gt; Nvidia GPU detected. \u0026gt;\u0026gt;\u0026gt; The Ollama API is now available at 127.0.0.1:11434. \u0026gt;\u0026gt;\u0026gt; Install complete. Run \u0026#34;ollama\u0026#34; from the command line. # pull and run deepseek-r1:8b models root@zackz:~# ollama list NAME ID SIZE MODIFIED deepseek-r1:14b ea35dfe18182 9.0 GB 6 days ago deepseek-r1:8b 28f8fd6cdc67 4.9 GB 11 days ago llama3:8b 365c0bd3c000 4.7 GB 12 days ago qwen:7b 2091ee8c8d8f 4.5 GB 12 days ago mistral:latest f974a74358d6 4.1 GB 12 days ago root@zackz:~# ollama run deepseek-r1:8b \u0026gt;\u0026gt;\u0026gt; who are you \u0026lt;think\u0026gt; \u0026lt;/think\u0026gt; Greetings! I\u0026#39;m DeepSeek-R1, an artificial intelligence assistant created by DeepSeek. I\u0026#39;m at your service and would be delighted to assist you with any inquiries or tasks you may have. \u0026gt;\u0026gt;\u0026gt; /bye Follow Open WebUI Github, install Open WebUI using docker, by configuring ollama API URL and docker persistent volume, Open WebUI will show the models running within Ollama, all the configuration and dialogue history data also can be persisted after server and application restart.\n# create persistent volume for webui-data, run docker root@zackz:/mnt/f/ml-local# mkdir -p webui-data root@zackz:~# docker run -d \\ -p 3000:8080 \\ --add-host=host.docker.internal:host-gateway \\ -e OLLAMA_API_BASE_URL=http://host.docker.internal:11434/api \\ --name open-webui \\ --restart always \\ ghcr.io/open-webui/open-webui:main # verify container status root@zackz:/mnt/f/ml-local# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5e7e2e63a4e1 ghcr.io/open-webui/open-webui:main \u0026#34;bash start.sh\u0026#34; 4 minutes ago Up 4 minutes (healthy) 0.0.0.0:3000-\u0026gt;8080/tcp open-webui # verify from open-webui log root@zackz:/mnt/f/ml-local# docker logs open-webui ___ __ __ _ _ _ ___ / _ \\ _ __ ___ _ __ \\ \\ / /__| |__ | | | |_ _| | | | | \u0026#39;_ \\ / _ \\ \u0026#39;_ \\ \\ \\ /\\ / / _ \\ \u0026#39;_ \\| | | || | | |_| | |_) | __/ | | | \\ V V / __/ |_) | |_| || | \\___/| .__/ \\___|_| |_| \\_/\\_/ \\___|_.__/ \\___/|___| |_| v0.5.4 - building the best open-source AI user interface. https://github.com/open-webui/open-webui Fetching 30 files: 100%|██████████| 30/30 [01:21\u0026lt;00:00, 2.73s/it] INFO: Started server process [1] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit) INFO: 172.17.0.1:59432 - \u0026#34;GET /static/splash.png HTTP/1.1\u0026#34; 200 OK INFO: 172.17.0.1:59432 - \u0026#34;GET /api/config HTTP/1.1\u0026#34; 200 OK INFO: 172.17.0.1:59430 - \u0026#34;GET /static/favicon.png HTTP/1.1\u0026#34; 200 OK INFO: (\u0026#39;172.17.0.1\u0026#39;, 59438) - \u0026#34;WebSocket /ws/socket.io/?EIO=4\u0026amp;transport=websocket\u0026#34; [accepted] INFO: connection open Open web browser and navigate to http://localhost:3000/ to access Open WebUI. First time access, you will be prompted to create a new admin account.\nCheck modules available from frontend webui:\nStart a dialogue to compare 3 models deepseek-r1:8b, llama3:8b and mistral, obviously llama3:8b is better than the other 2 models.\nBuild knowledge base collections\nBuilding knowledge base collections in Open WebUI typically involves organizing, indexing, and embedding relevant data so that it can be queried effectively by the LLM.\nPrepare Source Data\nI have some aged Linux, VMware and Windows admin docs in my computer with Microsoft Word format, I will use them to build some testing knowledge base collections, then I will upload my Zack blog posts as Markdown format into KB collections, then test from the dialogue to see how easily indexing and retrieval can be done.\nEnable Retrieval Features and Test Knowledge Base\nConfigure Open WebUI to use the knowledge base during interactions. This involves enabling a retrieval-augmented generation (RAG) feature or linking the embeddings to the model.\nUsing # in the beginning of the prompt to enable RAG feature, choose the knowledge collection from the collections we built earlier.\nStart a dialogue and choose deepseek-r1:8b model, use the knowledge base collections Zack Blog for MLops as a reference to provide an answer to summarize the posts I created in 2024:\nConclusion\nIn this post, I did some hands-on work to build a local DeepSeek R1-7B model. It is not accurate enough for real-world problems, considering my current GPU can only handle this 7B model—for fun, but still good to practise such local deployment. While it can retrieve some level of correct information based on the prompt and the given knowledge base collections, it is still limited. I assume the 32B model would perform better, but due to GPU memory constraints, the token output for DeepSeek 32B is too slow. Maybe later, I can try a cloud GPU instance or an AWS-managed AI service like Amazon Bedrock to test more powerful models in the future..\n","permalink":"https://zackblog.work/posts/mlops-build-a-knowledge-base-with-deepseek-r1/","summary":"\u003cp\u003e\u0026lsquo;Deep Seek is really hot at the moment!\u003c/p\u003e\n\u003cp\u003eIn this post, I want to build a local Knowledge Base using WSL, Docker, Ollama, Open WebUI and DeepSeek R1 7b.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eOllama\u003c/em\u003e is a desktop application designed to run and interact with large language models (LLMs) locally on a machine. It provides an easy interface for downloading, managing, and using various LLMs, ensuring privacy and local execution.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eOpen WebUI\u003c/em\u003e is a web-based user interface for interacting with AI models. It is often used in conjunction with locally hosted or remote LLMs, providing a customizable and user-friendly platform to input queries and manage model interactions.\u003c/p\u003e","title":"MLOps - Build a Knowledge Base with DeepSeek R1"},{"content":"Recently I got a task from the Company’s app team, to install an agent on EC2 instances across several AWS accounts and enable connection via AWS private link to a target AWS account where the management server is hosted. To achieve this task, here I will see how to use AWS Systems Manager for software distribution and installation for multiple AWS accounts and install and configure AWS private link using Terraform.\nAutomate agent installation via SSM Run Command\nHere I need a shell script to:\nIdentify Running Instances: Detect all running Linux and Windows EC2 instances in the specified AWS account. Tag Instances Based on SSM Availability: Check if the SSM agent is available for each instance and assign tags to reflect the SSM status (ssm-linux, no-ssm-linux, ssm-windows, no-ssm-windows). Create SSM Command Documents: For Linux: Install Agent by running a shell script. For Windows: Run a testing command to retrieve the Windows OS version. Execute Commands via SSM: Run the corresponding commands on Linux and Windows instances that have SSM agents available to install the agent to cover both Windows and Linux. #!/bin/bash # Define the tag key and new values TAG_KEY=\u0026#34;dtagent\u0026#34; REGION=\u0026#34;ap-southeast-2\u0026#34; # Step 1: Get a list of all running Linux and Windows instance IDs # Get Linux instance IDs LINUX_INSTANCE_IDS=$(aws ec2 describe-instances \\ --filters \u0026#34;Name=platform-details,Values=Linux/UNIX\u0026#34; \u0026#34;Name=instance-state-name,Values=running\u0026#34; \\ --query \u0026#34;Reservations[*].Instances[*].InstanceId\u0026#34; \\ --output text) # Get Windows instance IDs WINDOWS_INSTANCE_IDS=$(aws ec2 describe-instances \\ --filters \u0026#34;Name=platform-details,Values=Windows\u0026#34; \u0026#34;Name=instance-state-name,Values=running\u0026#34; \\ --query \u0026#34;Reservations[*].Instances[*].InstanceId\u0026#34; \\ --output text) # Step 2: Check for SSM agent availability and tag instances accordingly # Function to check SSM availability and assign tags tag_instance_based_on_ssm() { local INSTANCE_ID=$1 local PLATFORM=$2 # Check if the instance is managed by SSM SSM_STATUS=$(aws ssm describe-instance-information \\ --filters \u0026#34;Key=InstanceIds,Values=$INSTANCE_ID\u0026#34; \\ --query \u0026#34;InstanceInformationList[*].PingStatus\u0026#34; \\ --output text) # Determine tag value based on SSM status and platform if [ \u0026#34;$PLATFORM\u0026#34; == \u0026#34;Linux\u0026#34; ]; then if [ \u0026#34;$SSM_STATUS\u0026#34; == \u0026#34;Online\u0026#34; ]; then TAG_VALUE=\u0026#34;ssm-linux\u0026#34; else TAG_VALUE=\u0026#34;no-ssm-linux\u0026#34; fi elif [ \u0026#34;$PLATFORM\u0026#34; == \u0026#34;Windows\u0026#34; ]; then if [ \u0026#34;$SSM_STATUS\u0026#34; == \u0026#34;Online\u0026#34; ]; then TAG_VALUE=\u0026#34;ssm-windows\u0026#34; else TAG_VALUE=\u0026#34;no-ssm-windows\u0026#34; fi fi # Apply the determined tag to the instance aws ec2 create-tags --resources \u0026#34;$INSTANCE_ID\u0026#34; --tags Key=$TAG_KEY,Value=$TAG_VALUE } # Tag Linux instances based on SSM availability for INSTANCE_ID in $LINUX_INSTANCE_IDS; do tag_instance_based_on_ssm \u0026#34;$INSTANCE_ID\u0026#34; \u0026#34;Linux\u0026#34; done # Tag Windows instances based on SSM availability for INSTANCE_ID in $WINDOWS_INSTANCE_IDS; do tag_instance_based_on_ssm \u0026#34;$INSTANCE_ID\u0026#34; \u0026#34;Windows\u0026#34; done echo \u0026#34;Tagging complete. Instances have been tagged with SSM availability status.\u0026#34; sleep 10 # Step 3: Create JSON files for SSM command documents (Linux and Windows) cat EOF linux_dtcommand.json { \u0026#34;schemaVersion\u0026#34;: \u0026#34;2.2\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Dynatrace oneagent installation command for Linux via SSM\u0026#34;, \u0026#34;mainSteps\u0026#34;: [ { \u0026#34;action\u0026#34;: \u0026#34;aws:runShellScript\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;LinuxDTCommand\u0026#34;, \u0026#34;inputs\u0026#34;: { \u0026#34;runCommand\u0026#34;: [ \u0026#34;curl -o /tmp/dtssm.sh https://raw.githubusercontent.com/ZackZhouHB/zack-gitops-project/refs/heads/editing/Python_scripts/testssm.sh\u0026#34;, \u0026#34;chmod +x /tmp/dtssm.sh\u0026#34;, \u0026#34;/tmp/dtssm.sh\u0026#34; ] } } ] } EOF cat EOF windows_dtcommand.json { \u0026#34;schemaVersion\u0026#34;: \u0026#34;2.2\u0026#34;, \u0026#34;description\u0026#34;: \u0026#34;Fetch Windows OS version via SSM\u0026#34;, \u0026#34;mainSteps\u0026#34;: [ { \u0026#34;action\u0026#34;: \u0026#34;aws:runPowerShellScript\u0026#34;, \u0026#34;name\u0026#34;: \u0026#34;WindowsOSVersionCheck\u0026#34;, \u0026#34;inputs\u0026#34;: { \u0026#34;runCommand\u0026#34;: [ \u0026#34;(Get-ItemProperty -Path \u0026#39;HKLM:\\\\\\\\SOFTWARE\\\\\\\\Microsoft\\\\\\\\Windows NT\\\\\\\\CurrentVersion\u0026#39;).ProductName\u0026#34; ] } } ] } EOF # Step 4: Create the SSM documents aws ssm create-document \\ --name \u0026#34;LinuxDTCommand\u0026#34; \\ --document-type \u0026#34;Command\u0026#34; \\ --content file://linux_dtcommand.json aws ssm create-document \\ --name \u0026#34;WindowsDTCommand\u0026#34; \\ --document-type \u0026#34;Command\u0026#34; \\ --content file://windows_dtcommand.json # Step 5: Send the SSM command to Linux instances with \u0026#34;ssm-linux\u0026#34; tag aws ssm send-command \\ --document-name \u0026#34;LinuxDTCommand\u0026#34; \\ --targets \u0026#34;Key=tag:$TAG_KEY,Values=ssm-linux\u0026#34; \\ --comment \u0026#34;Execute Dynatrace oneagent installation on all Linux instances with SSM\u0026#34; \\ --max-concurrency \u0026#34;50\u0026#34; \\ --max-errors \u0026#34;0\u0026#34; \\ --region $REGION # Step 6: Send the SSM command to Windows instances with \u0026#34;ssm-windows\u0026#34; tag aws ssm send-command \\ --document-name \u0026#34;WindowsDTCommand\u0026#34; \\ --targets \u0026#34;Key=tag:$TAG_KEY,Values=ssm-windows\u0026#34; \\ --comment \u0026#34;Execute Dynatrace oneagent installation on all Windows instances with SSM\u0026#34; \\ --max-concurrency \u0026#34;50\u0026#34; \\ --max-errors \u0026#34;0\u0026#34; \\ --region $REGION echo \u0026#34;Commands sent to instances with SSM agent available.\u0026#34; The result can be verified via EC2 console for the tags, together in AWS SSM console for the commands execution status.\nCross account AWS private link setup\nHere I will use both of my own AWS accounts zack and joe, to set up and verify a PrivateLink connection:\nProvider Account (Zack): Hosting a service (Python HTTP server on port 9080) running on an EC2 instance behind a Network Load Balancer (NLB). Consumer Account (Joe): Accessing the service securely through a VPC Endpoint without exposing the service to the internet. Run a simple Python HTTP server on port 9080 in the provider account (Zack)\nIn Provider Account (Zack):\nSSH into Zackblog EC2 running a docker Python HTTP service on port 9080. root@ip-172-31-26-78:/var/snap/amazon-ssm-agent/9881# docker pull python:3 root@ip-172-31-26-78:/var/snap/amazon-ssm-agent/9881# docker run -d -p 9080:9080 --name test-container python:3 python root@ip-172-31-26-78:/var/snap/amazon-ssm-agent/9881# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6e0dabbd7d5b python:3 \u0026#34;python -m http.serv…\u0026#34; About an hour ago Up About an hour 0.0.0.0:9080-\u0026gt;9080/tcp, :::9080-\u0026gt;9080/tcp test-container root@ip-172-31-26-78:/var/snap/amazon-ssm-agent/9881# curl localhost:9080 Create an NLB with Security group forwards traffic to TCP 9080 port on the EC2 instance. Register the instance as a target group, create VPC Endpoint Service to expose the NLB to consumer accounts. Manual approve when receiving a connection request from the consumer account. In Consumer Account (Joe):\nCreate a VPC Interface Endpoint request and connect to the provider\u0026rsquo;s endpoint service. Create a testing EC2 with port 9080 allowed from the security group, run curl to the private DNS name with port 9080 to verify the private link. ubuntu@ip-172-31-54-206:~$ curl http://vpce-0961b67b61820a01e-6kvjc4rs.vpce-svc-056bd4182a8904985.ap-southeast-2.vpce.amazonaws.com:9080 Here it can be seen that the traffic from the consumer EC2 Ubuntu instance is routed through the endpoint to the NLB and then to the service in the provider account.\nUsing Terraform to create the above resources, so it can be easily destroyed for lab purposes and reused across more AWS accounts.\nFirst in the provider account, create NLB, Security group for NLB to allow 9080, Target group to register Zackblog EC2, Listener, VPC endpoint service.\n# main.tf for provider account # Provider Account Configuration provider \u0026#34;aws\u0026#34; { profile = \u0026#34;zack\u0026#34; # AWS account profile for provider region = \u0026#34;ap-southeast-2\u0026#34; } # Fetch Existing VPC and Subnet data \u0026#34;aws_vpc\u0026#34; \u0026#34;default\u0026#34; { default = true } data \u0026#34;aws_subnet\u0026#34; \u0026#34;subnet1\u0026#34; { id = \u0026#34;subnet-0ssssssssss69a\u0026#34; } data \u0026#34;aws_subnet\u0026#34; \u0026#34;subnet2\u0026#34; { id = \u0026#34;subnet-073ssssssssss46db\u0026#34; } data \u0026#34;aws_subnet\u0026#34; \u0026#34;subnet3\u0026#34; { id = \u0026#34;subnet-09sssssss129\u0026#34; # } # Fetch Existing Security Group for EC2 Instance data \u0026#34;aws_security_group\u0026#34; \u0026#34;ec2_sg\u0026#34; { id = \u0026#34;sg-01ssssssss7c\u0026#34; # Security Group ID of Zackblog EC2 instance } # Create Security Group for NLB resource \u0026#34;aws_security_group\u0026#34; \u0026#34;nlb_sg\u0026#34; { name = \u0026#34;nlb-private-link-sg\u0026#34; vpc_id = data.aws_vpc.default.id description = \u0026#34;Allow traffic to NLB from consumer accounts\u0026#34; egress { from_port = 0 to_port = 0 protocol = \u0026#34;-1\u0026#34; cidr_blocks = [\u0026#34;0.0.0.0/0\u0026#34;] } ingress { from_port = 9080 to_port = 9080 protocol = \u0026#34;tcp\u0026#34; cidr_blocks = [\u0026#34;172.31.48.0/20\u0026#34;] # Updated to match consumer\u0026#39;s new CIDR } } # Allow NLB to Access EC2 on Port 9080 resource \u0026#34;aws_security_group_rule\u0026#34; \u0026#34;ec2_inbound_from_nlb\u0026#34; { type = \u0026#34;ingress\u0026#34; security_group_id = data.aws_security_group.ec2_sg.id # Existing EC2 SG from_port = 9080 to_port = 9080 protocol = \u0026#34;tcp\u0026#34; source_security_group_id = aws_security_group.nlb_sg.id # Allow traffic from NLB SG } # Create NLB resource \u0026#34;aws_lb\u0026#34; \u0026#34;nlb\u0026#34; { name = \u0026#34;private-link-nlb\u0026#34; internal = true load_balancer_type = \u0026#34;network\u0026#34; subnets = [ data.aws_subnet.subnet1.id, data.aws_subnet.subnet2.id, data.aws_subnet.subnet3.id ] security_groups = [aws_security_group.nlb_sg.id] } # Create NLB Target Group resource \u0026#34;aws_lb_target_group\u0026#34; \u0026#34;tg\u0026#34; { name = \u0026#34;private-link-tg\u0026#34; port = 9080 protocol = \u0026#34;TCP\u0026#34; vpc_id = data.aws_vpc.default.id target_type = \u0026#34;instance\u0026#34; } # Attach EC2 Instance to Target Group resource \u0026#34;aws_lb_target_group_attachment\u0026#34; \u0026#34;tg_attachment\u0026#34; { target_group_arn = aws_lb_target_group.tg.arn target_id = \u0026#34;i-076sssssssscf2\u0026#34; # Zackblog instance ID port = 9080 } # Add Listener for NLB resource \u0026#34;aws_lb_listener\u0026#34; \u0026#34;nlb_listener\u0026#34; { load_balancer_arn = aws_lb.nlb.arn # Reference the NLB created in your Terraform configuration port = 9080 # Listener port protocol = \u0026#34;TCP\u0026#34; # Protocol for the listener default_action { type = \u0026#34;forward\u0026#34; target_group_arn = aws_lb_target_group.tg.arn # Reference the target group created in your Terraform configuration } } # Create Endpoint Service resource \u0026#34;aws_vpc_endpoint_service\u0026#34; \u0026#34;private_link_service\u0026#34; { acceptance_required = true network_load_balancer_arns = [ aws_lb.nlb.arn, ] allowed_principals = [ \u0026#34;arn:aws:iam::8ssssssssss5:root\u0026#34; # Replace with account joe\u0026#39;s ID ] private_dns_name = \u0026#34;zzservice.internal\u0026#34; # Provide a custom private DNS name } output \u0026#34;endpoint_service_name\u0026#34; { value = aws_vpc_endpoint_service.private_link_service.service_name description = \u0026#34;The name of the VPC Endpoint Service to share with the consumer account\u0026#34; } Then in the consumer account, create an interface endpoint, security group for the Interface Endpoint, and a testing Ubuntu EC2 in a new non-overlapping subnet.\n# main.tf for joe account as consumer # Consumer Account Configuration provider \u0026#34;aws\u0026#34; { profile = \u0026#34;joe\u0026#34; # AWS account profile for consumer region = \u0026#34;ap-southeast-2\u0026#34; } # Fetch Existing VPC and Subnet data \u0026#34;aws_vpc\u0026#34; \u0026#34;default\u0026#34; { default = true } # Create a new non-overlapping subnet in account Joe resource \u0026#34;aws_subnet\u0026#34; \u0026#34;new_consumer_subnet\u0026#34; { vpc_id = data.aws_vpc.default.id cidr_block = \u0026#34;172.31.48.0/20\u0026#34; # Updated CIDR to avoid overlap availability_zone = \u0026#34;ap-southeast-2c\u0026#34; # Same AZ as the testing EC2 instance map_public_ip_on_launch = false # Optional: Prevent public IP assignment tags = { Name = \u0026#34;Consumer-New-PrivateLink-Subnet\u0026#34; } } # Create Security Group for Interface Endpoint resource \u0026#34;aws_security_group\u0026#34; \u0026#34;endpoint_sg\u0026#34; { name = \u0026#34;private-link-endpoint-sg\u0026#34; vpc_id = data.aws_vpc.default.id description = \u0026#34;Allow traffic to the interface endpoint\u0026#34; ingress { from_port = 9080 to_port = 9080 protocol = \u0026#34;tcp\u0026#34; cidr_blocks = [\u0026#34;172.31.48.0/20\u0026#34;] # Updated to match new consumer CIDR } egress { from_port = 0 to_port = 0 protocol = \u0026#34;-1\u0026#34; cidr_blocks = [\u0026#34;0.0.0.0/0\u0026#34;] } } resource \u0026#34;aws_vpc_endpoint\u0026#34; \u0026#34;interface_endpoint\u0026#34; { vpc_id = data.aws_vpc.default.id service_name = \u0026#34;com.amazonaws.vpce.ap-southeast-2.vpce-svc-0ssssssssss5\u0026#34; subnet_ids = [aws_subnet.new_consumer_subnet.id] security_group_ids = [aws_security_group.endpoint_sg.id] private_dns_enabled = false # Temporarily disable for accept connection from provider account then change to true vpc_endpoint_type = \u0026#34;Interface\u0026#34; } # Replace the null resource with a manual acceptance process (preferred) output \u0026#34;accept_endpoint_instructions\u0026#34; { value = \u0026lt;\u0026lt;EOT To accept the VPC endpoint in the provider account, run: aws ec2 accept-vpc-endpoint-connections \\ --vpc-endpoint-service-id \u0026lt;SERVICE_ID_FROM_PROVIDER\u0026gt; \\ --vpc-endpoint-ids \u0026lt;ENDPOINT_ID\u0026gt; EOT } # Dynamically Fetch the Subnet data \u0026#34;aws_subnet\u0026#34; \u0026#34;new_consumer_subnet\u0026#34; { filter { name = \u0026#34;cidr-block\u0026#34; values = [\u0026#34;172.31.48.0/20\u0026#34;] # Replace with the CIDR block of the desired subnet } filter { name = \u0026#34;vpc-id\u0026#34; values = [data.aws_vpc.default.id] } } # Create Security Group for Testing resource \u0026#34;aws_security_group\u0026#34; \u0026#34;testing_sg\u0026#34; { name = \u0026#34;testing-private-link-sg\u0026#34; vpc_id = data.aws_vpc.default.id description = \u0026#34;Security group for testing EC2 instance\u0026#34; ingress { from_port = 22 to_port = 22 protocol = \u0026#34;tcp\u0026#34; cidr_blocks = [\u0026#34;0.0.0.0/0\u0026#34;] # Allow SSH from everywhere (change for production use) } ingress { from_port = 9080 to_port = 9080 protocol = \u0026#34;tcp\u0026#34; cidr_blocks = [\u0026#34;172.31.0.0/16\u0026#34;] # Allow traffic to PrivateLink endpoint within VPC } egress { from_port = 0 to_port = 0 protocol = \u0026#34;-1\u0026#34; cidr_blocks = [\u0026#34;0.0.0.0/0\u0026#34;] } } # Launch EC2 Instance in 172.31.48.0/20 Subnet resource \u0026#34;aws_instance\u0026#34; \u0026#34;testing_ec2\u0026#34; { ami = \u0026#34;ami-040e71e7b8391cae4\u0026#34; # Replace with a valid Amazon Linux 2 AMI for ap-southeast-2 instance_type = \u0026#34;t2.micro\u0026#34; subnet_id = data.aws_subnet.new_consumer_subnet.id vpc_security_group_ids = [aws_security_group.testing_sg.id] # Use the correct attribute for SGs with subnets associate_public_ip_address = true # Assign a public IP address key_name = \u0026#34;xxxxx1\u0026#34; tags = { Name = \u0026#34;PrivateLink-Testing-EC2\u0026#34; } } Finally, SSH into the new EC2 instance and run the following command to test the PrivateLink endpoint.\nConclusion\nTraffic Flow Consumer EC2 → PrivateLink Endpoint: Consumer EC2 uses the private DNS name to send requests to the endpoint. The VPC interface endpoint forwards traffic securely to the NLB in the provider account. NLB → EC2 Instance: The NLB routes traffic from the VPC endpoint to the EC2 instance based on the target group configuration. EC2 Instance → Service: The EC2 instance processes the request and responds back via the same path. What We Accomplished Used Terraform to automate resource creation for NLB, target group, VPC endpoint service, and consumer endpoint. Successfully accessed the Python HTTP server from the consumer EC2 instance using the private DNS name via PrivateLink. Verified connectivity from consumer EC2 to provider service. ","permalink":"https://zackblog.work/posts/aws-private-link-and-cross-account-package-deployment/","summary":"\u003cp\u003eRecently I got a task from the Company’s app team, to install an agent on EC2 instances across several AWS accounts and enable connection via AWS private link to a target AWS account where the management server is hosted. To achieve this task, here I will see how to use AWS Systems Manager for software distribution and installation for multiple AWS accounts and install and configure AWS private link using Terraform.\u003c/p\u003e","title":"AWS Private Link and Cross-account Package deployment"},{"content":"In the last post MLOPS - Lab Setup, I was able to set the local ML lab environment, and run validation in Jupyter Notebook to test the CODA device and performance on my local PC.\nAlthough Jupyter Notebooks can be user-friendly tools for ML practice, offering easy interaction and immediate feedback, which simplifies testing and debugging, it has limitations such as reproducibility issues, challenges in collaboration and version control, scalability concerns for larger projects, and a lack of automation for tasks like retraining.\nIn this post, I will try an ML project with tools like DVC, MLflow, Docker, Apache Airflow, and CI/CD frameworks to strengthen machine learning workflows. This way can ensure reproducibility by tracking data and code versions, while MLflow logs metrics for effective experiment tracking. Although their initial setup can be complex and resource-intensive, these tools automate processes, streamline workflows, and enhance collaboration and scalability, which could be excessive for smaller ML projects.\nML Tools explained\nData Versioning (DVC): DVC allows teams to manage and version datasets just like code. This ensures that data changes are tracked, making it easier to revert to previous versions if necessary. Experiment Tracking (MLflow): MLflow tracks experiments, capturing metrics, parameters, and model versions in one centralized location. This makes it easier to compare different runs and select the best-performing model. Containerization (Docker): Docker creates isolated environments, ensuring that code runs consistently across different platforms without dependency issues. This helps avoid the \u0026ldquo;it works on my machine\u0026rdquo; problem. Workflow Orchestration (Apache Airflow): Airflow schedules and manages complex workflows, allowing for the automation of tasks such as data retrieval, preprocessing, model training, and evaluation. CI/CD (Jenkins): I have a local Jenkins image to facilitate automatic testing and deployment of models and code changes. This ensures that new features or updates are quickly integrated without disrupting the existing workflow. Combining these tools to achieve a holistic pipeline enables reproducibility, scalability, and consistency in machine learning workflows.\nProject Structure\nCreate a new project directory with the following structure:\n(jupyter_env) root@zackz:/mnt/mlops-project# tree mlops-project/ ├── data/ # Data directory (for DVC) ├── models/ # Trained models ├── src/ # Source code for the ML model ├── notebooks/ # Jupyter notebooks for experimentation ├── Dockerfile # Docker config for packaging ├── dvc.yaml # DVC pipeline config ├── airflow_dags/ # Airflow DAG for automation └── mlflow/ # MLflow tracking directory Project Implementation\nStep 1: Data Versioning with DVC\nInitialize Git \u0026amp; DVC\npip install dvc git init dvc init -f Add the Iris Dataset:\nmkdir data curl -o data/iris.csv https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data dvc add data/iris.csv Step 2: Train the Model (Using MLflow)\nInstall MLflow\npip install mlflow Create a Training Script (src/train.py)\nvim src/train.py import mlflow import mlflow.sklearn import pandas as pd from sklearn.model_selection import train_test_split from sklearn.ensemble import RandomForestClassifier from sklearn.metrics import accuracy_score # Load the dataset data = pd.read_csv(\u0026#39;../data/iris.csv\u0026#39;, header=None) X = data.iloc[:, :-1] y = data.iloc[:, -1] # Split data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Track experiment with MLflow with mlflow.start_run(): # Train model model = RandomForestClassifier(n_estimators=100) model.fit(X_train, y_train) # Make predictions predictions = model.predict(X_test) accuracy = accuracy_score(y_test, predictions) # Log model and metrics to MLflow mlflow.log_metric(\u0026#34;accuracy\u0026#34;, accuracy) mlflow.sklearn.log_model(model, \u0026#34;model\u0026#34;) print(f\u0026#34;Model accuracy: {accuracy}\u0026#34;) Run the Training Script\npython src/train.py (jupyter_env) root@zackz:/mnt/f/1/mlops-project# python src/train.py 2024/10/05 13:33:39 WARNING mlflow.models.model: Model logged without a signature and input example. Please set `input_example` parameter when logging the model to auto infer the model signature. Model accuracy: 1.0 Launch the MLflow UI\nmlflow ui Navigate to http://127.0.0.1:5000 to view the experiment\nStep 3: Dockerize the Model for Deployment\nCreate Dockerfile:\nvim Dockerfile FROM python:3.8-slim WORKDIR /app # Install dependencies COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # Copy the source code COPY . . # Run the model training script CMD [\u0026#34;python\u0026#34;, \u0026#34;src/train.py\u0026#34;] Create a requirements.txt file and build the mlops-local-model Docker image:\nvim requirements.txt mlflow scikit-learn pandas dvc docker build -t mlops-local-model . docker run mlops-local-model (jupyter_env) root@zackz:~# docker run mlops-local-model 2024/10/05 02:50:58 WARNING mlflow.utils.git_utils: Failed to import Git (the Git executable is probably not on your PATH), so Git SHA is not available. Error: Failed to initialize: Bad git executable. The git executable must be specified in one of the following ways: - be included in your $PATH - be set via $GIT_PYTHON_GIT_EXECUTABLE - explicitly set via git.refresh(\u0026lt;full-path-to-git-executable\u0026gt;) All git commands will error until this is rectified. This initial message can be silenced or aggravated in the future by setting the $GIT_PYTHON_REFRESH environment variable. Use one of the following values: - quiet|q|silence|s|silent|none|n|0: for no message or exception - warn|w|warning|log|l|1: for a warning message (logging level CRITICAL, displayed by default) - error|e|exception|raise|r|2: for a raised exception Example: export GIT_PYTHON_REFRESH=quiet 2024/10/05 02:51:00 WARNING mlflow.models.model: Model logged without a signature and input example. Please set `input_example` parameter when logging the model to auto infer the model signature. Model accuracy: 1.0 Step 4: Automate with Apache Airflow\nInstall Apache Airflow:\npip install apache-airflow Create an Airflow DAG (airflow_dags/ml_pipeline.py)\nvim airflow_dags/ml_pipeline.py from airflow import DAG from airflow.operators.python_operator import PythonOperator from datetime import datetime import os # Define the DAG default_args = { \u0026#39;owner\u0026#39;: \u0026#39;airflow\u0026#39;, \u0026#39;start_date\u0026#39;: datetime(2023, 1, 1), \u0026#39;retries\u0026#39;: 1, } dag = DAG(\u0026#39;mlops_pipeline\u0026#39;, default_args=default_args, schedule_interval=\u0026#39;@daily\u0026#39;) # Define the task to retrain the model def retrain_model(): os.system(\u0026#39;python src/train.py\u0026#39;) retrain_task = PythonOperator( task_id=\u0026#39;retrain_model\u0026#39;, python_callable=retrain_model, dag=dag ) retrain_task Run Airflow:\nairflow db init airflow webserver --port 8080 airflow scheduler Create Airflow web UI Admin user\nairflow users create \\ --username admin \\ --firstname Admin \\ --lastname User \\ --role Admin \\ --email admin@xxx.com \\ --password the_password Navigate to http://127.0.0.1:8080 to view the Airflow\nStep 5: CICD with Jenkins\nCreate Jenkins pipeline for continuous model training with the following stages:\nConclusion\nBy integrating DVC, MLflow, Docker, Airflow, and CI/CD into a cohesive ML project environment, we can achieve enhanced efficiency, greater automation, and improved collaboration. This synergy not only streamlines the development process but also ensures that machine learning models are robust, reproducible, and ready for production deployment.\nIn summary, a production-level ML workflow integrates new data, automates model training and deployment, and continuously monitors model performance. By utilizing CI/CD pipelines, Docker for containerization, and tools for versioning and tracking, we can create a robust and efficient machine learning system that can adapt to changing data and business requirements.\nIn the next post, I will refactor the local tools into AWS ML services, to move the ML pipeline and deployment to the cloud.\n","permalink":"https://zackblog.work/posts/mlops-explore-ml-tools/","summary":"\u003cp\u003eIn the last post \u003ca href=\"/posts/mlops-setup-a-home-machine-learning-lab/\"\u003eMLOPS - Lab Setup\u003c/a\u003e, I was able to set the local ML lab environment, and run validation in Jupyter Notebook to test the CODA device and performance on my local PC.\u003c/p\u003e\n\u003cp\u003eAlthough \u003cem\u003eJupyter Notebooks\u003c/em\u003e can be user-friendly tools for ML practice, offering easy interaction and immediate feedback, which simplifies testing and debugging, it has limitations such as reproducibility issues, challenges in collaboration and version control, scalability concerns for larger projects, and a lack of automation for tasks like retraining.\u003c/p\u003e","title":"MLOps - Explore ML tools"},{"content":"Transitioning from DevOps to MLOps can be achieved by leveraging existing DevOps expertise by adding new layers specific to machine learning.\nKey Differences:\nModel Lifecycle Management: MLOps handles model training, deployment, and retraining.\nData Versioning: Tools like DVC ensure dataset version control.\nExperiment Tracking: MLflow and Weights \u0026amp; Biases track model training parameters and results.\nModel Serving: Deploy models with TensorFlow Serving or TorchServe.\nModel Drift: Monitor data changes over time to trigger retraining.\nCore MLOps Tools:\nModel Training \u0026amp; Experimentation: Tools like DVC, MLflow, and Kubeflow for managing data, tracking experiments, and distributed training.\nModel Deployment \u0026amp; Serving: Use CI/CD pipelines, Docker, Kubernetes, and frameworks like ONNX for deploying models at scale.\nMonitoring \u0026amp; Retraining: Use Prometheus, Grafana, and Seldon for monitoring performance and retraining pipelines.\nData Pipelines: Automate feature engineering with Apache Airflow, Dagster, or Kubeflow.\nLeverage DevOps Skills for MLOps:\nCI/CD Pipelines: Automate model training, testing, and deployment with Jenkins or cloud solutions.\nInfrastructure as Code: Use Terraform or Ansible for cloud-based ML infrastructure.\nContainerization \u0026amp; Orchestration: Deploy ML models with Docker and Kubernetes.\nMonitoring: Track both infrastructure and model-specific metrics like accuracy and drift.\nLocal Lab ML practice\nI will start the local lab by:\nSetting up a local ML environment Install ML-focused tools (Nvidia Cuda, Python3 and pip Virtual ENV, PyTorch, and Jupyter Notebook) Build and version simple ML models locally with tools like DVC, MLflow, and Docker. Next stages I will try:\nProvision AWS Sagemaker using terraform or Cloudformation. Implement CI pipelines for Model training and continuous packaging. CD pipelines to provision AWS ECS or EKS to deploy models. Prerequisites\nWindows 10 with Powershell and Windows Terminal installed CPU Virtualization enabled in BIOS WSL2 with Ubuntu LTS installed Docker Desktop Install WSL with Ubuntu\nFirst, we need to configure local WSL to install Ubuntu.\nC:\\Users\\zack\u0026gt;wsl --list --online Use \u0026#39;wsl.exe --install \u0026lt;Distro\u0026gt;\u0026#39; to install NAME FRIENDLY NAME Ubuntu Ubuntu Debian Debian GNU/Linux kali-linux Kali Linux Rolling Ubuntu-18.04 Ubuntu 18.04 LTS Ubuntu-20.04 Ubuntu 20.04 LTS Ubuntu-22.04 Ubuntu 22.04 LTS Ubuntu-24.04 Ubuntu 24.04 LTS OracleLinux_7_9 Oracle Linux 7.9 OracleLinux_8_7 Oracle Linux 8.7 OracleLinux_9_1 Oracle Linux 9.1 openSUSE-Leap-15.6 openSUSE Leap 15.6 SUSE-Linux-Enterprise-15-SP5 SUSE Linux Enterprise 15 SP5 SUSE-Linux-Enterprise-15-SP6 SUSE Linux Enterprise 15 SP6 openSUSE-Tumbleweed openSUSE Tumbleweed C:\\Users\\zack\u0026gt;wsl --install -d Ubuntu-24.04 Installing: Ubuntu 24.04 LTS Installed Ubuntu 24.04 LTS。 Launching Ubuntu 24.04 LTS... Installing, this may take a few minutes... Installation successful! ubuntu@zackz:~$ cat /etc/os-release PRETTY_NAME=\u0026#34;Ubuntu 24.04.1 LTS\u0026#34; NAME=\u0026#34;Ubuntu\u0026#34; VERSION_ID=\u0026#34;24.04\u0026#34; VERSION=\u0026#34;24.04.1 LTS (Noble Numbat)\u0026#34; VERSION_CODENAME=noble ID=ubuntu ID_LIKE=debian HOME_URL=\u0026#34;https://www.ubuntu.com/\u0026#34; SUPPORT_URL=\u0026#34;https://help.ubuntu.com/\u0026#34; BUG_REPORT_URL=\u0026#34;https://bugs.launchpad.net/ubuntu/\u0026#34; PRIVACY_POLICY_URL=\u0026#34;https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\u0026#34; UBUNTU_CODENAME=noble LOGO=ubuntu-logo Install Nvidia CUDA\nCUDA works with C. Thus, we need to install the gcc compiler first, then install CUDA from the official website of Nvidia, then configure the environment variable for post-installation The official CUDA installation guide from Nvidia.\nsudo apt install gcc --fix-missing wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600 wget https://developer.download.nvidia.com/compute/cuda/12.6.2/local_installers/cuda-repo-wsl-ubuntu-12-6-local_12.6.2-1_amd64.deb sudo dpkg -i cuda-repo-wsl-ubuntu-12-6-local_12.6.2-1_amd64.deb sudo cp /var/cuda-repo-wsl-ubuntu-12-6-local/cuda-*-keyring.gpg /usr/share/keyrings/ sudo apt-get update sudo apt-get -y install cuda-toolkit-12-6 vim .bashrc export PATH=/usr/local/cuda-12.6/bin${PATH:+:${PATH}} # To apply and validate the changes, source ~/.bashrc echo $PATH root@zackz:~# echo $PATH Install the Nvidia Cuda Toolkit, check the Driver and CUDA versions, validate Nvidia Cuda Compiler Driver has been installed.\nsudo apt install nvidia-cuda-toolkit root@zackz:~# nvidia-smi Wed Oct 9 10:53:26 2024 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.112 Driver Version: 537.42 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------| | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | | |=========================================+======================+======================| | 0 NVIDIA GeForce RTX 3070 Ti On | 00000000:01:00.0 On | N/A | | 0% 55C P0 80W / 148W | 1635MiB / 8192MiB | 1% Default | | | | | +-----------------------------------------+----------------------+----------------------+ +---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | 0 N/A N/A 27 G /Xwayland N/A | | 0 N/A N/A 30 G /Xwayland N/A | | 0 N/A N/A 37 G /Xwayland N/A | +---------------------------------------------------------------------------------------+ root@zackz:~# nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2024 NVIDIA Corporation Built on Thu_Sep_12_02:18:05_PDT_2024 Cuda compilation tools, release 12.6, V12.6.77 Build cuda_12.6.r12.6/compiler.34841621_0 Install Python3 and PIP Virtual ENV\nEnsure that python3 and PIP are installed, and create a virtual environment for PyTorch and Jupyter Notebook.\nroot@zackz:~# python3 --version Python 3.12.3 sudo apt-get install python3-pip apt install python3.12-venv python3 -m venv jupyter_env source jupyter_env/bin/activate (jupyter_env)root@zackz:~# Install PyTorch\nInstall PyTorch from the official website of PyTorch, and enable the Nvidia Developer Settings for using CUDA via WSL, then validate CUDA from Torch.\n(jupyter_env)root@zackz:~# pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu124 (jupyter_env)root@zackz:~# python3 Python 3.12.3 (main, Sep 11 2024, 14:17:37) [GCC 13.2.0] on linux Type \u0026#34;help\u0026#34;, \u0026#34;copyright\u0026#34;, \u0026#34;credits\u0026#34; or \u0026#34;license\u0026#34; for more information. \u0026gt;\u0026gt;\u0026gt; import torch ch.cuda.is_available() True \u0026gt;\u0026gt;\u0026gt; Install Jupyter Notebook\nInstalling Jupyter Notebook and running it in the virtual environment, create the first notebook to verify if it is using CPU or CUDA from GPU, then run a simple notebook to have a performance comparison between CPU and GPU.\n# install jupyter notebook pip install jupyter notebook # run jupyter notebook in the virtual env (jupyter_env) root@zackz:~# jupyter notebook --allow-root Verify Torch with CUDA device\nimport torch if torch.cuda.is_available(): device = torch.device(\u0026#34;cuda\u0026#34;) else: device = torch.device(\u0026#34;cpu\u0026#34;) print(\u0026#34;using\u0026#34;, device, \u0026#34;device\u0026#34;) Run performance comparison between my CPU and GPU (CUDA)\nimport time matrix_size = 32*512 x = torch.randn(matrix_size, matrix_size) y = torch.randn(matrix_size, matrix_size) print(\u0026#34;************* CPU SPEED *******************\u0026#34;) start = time.time() result = torch.matmul(x, y) print(time.time() - start) print(\u0026#34;verify device:\u0026#34;, result.device) x_gpu = x.to(device) y_gpu = y.to(device) torch.cuda.synchronize() for i in range(3): print(\u0026#34;************* GPU SPEED *******************\u0026#34;) start = time.time() result_gpu = torch.matmul(x_gpu, y_gpu) torch.cuda.synchronize() print(time.time() - start) print(\u0026#34;verify device:\u0026#34;, result_gpu.device) Here I run a pretrained model RedPajama, ask questions from prompt, the 2nd answer just killing me :)\nimport torch import transformers from transformers import AutoTokenizer, AutoModelForCausalLM MIN_TRANSFORMERS_VERSION = \u0026#39;4.25.1\u0026#39; # check transformers version assert transformers.__version__ \u0026gt;= MIN_TRANSFORMERS_VERSION, f\u0026#39;Please upgrade transformers to version {MIN_TRANSFORMERS_VERSION} or higher.\u0026#39; # init tokenizer = AutoTokenizer.from_pretrained(\u0026#34;togethercomputer/RedPajama-INCITE-Instruct-3B-v1\u0026#34;) model = AutoModelForCausalLM.from_pretrained(\u0026#34;togethercomputer/RedPajama-INCITE-Instruct-3B-v1\u0026#34;, torch_dtype=torch.float16) model = model.to(\u0026#39;cuda:0\u0026#39;) # infer prompt = \u0026#34;Q: who is the best soccer player?\\nA:\u0026#34; inputs = tokenizer(prompt, return_tensors=\u0026#39;pt\u0026#39;).to(model.device) input_length = inputs.input_ids.shape[1] outputs = model.generate( **inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.7, top_k=50, return_dict_in_generate=True ) token = outputs.sequences[0, input_length:] output_str = tokenizer.decode(token) print(output_str) Conclusion\nHere I have successfully set up a local machine learning lab environment, and installed ML tools on local Windows using WSL2. Next stage we will try to run a local ML module and containerize it into a Docker image.\n","permalink":"https://zackblog.work/posts/mlops-setup-a-home-machine-learning-lab/","summary":"\u003cp\u003eTransitioning from DevOps to MLOps can be achieved by leveraging existing DevOps expertise by adding new layers specific to machine learning.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eKey Differences:\u003c/strong\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ccode\u003eModel Lifecycle Management\u003c/code\u003e: MLOps handles model training, deployment, and retraining.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ccode\u003eData Versioning\u003c/code\u003e: Tools like DVC ensure dataset version control.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ccode\u003eExperiment Tracking\u003c/code\u003e: MLflow and Weights \u0026amp; Biases track model training parameters and results.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ccode\u003eModel Serving\u003c/code\u003e: Deploy models with TensorFlow Serving or TorchServe.\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003e\u003ccode\u003eModel Drift\u003c/code\u003e: Monitor data changes over time to trigger retraining.\u003c/p\u003e","title":"MLOps - Setup a Home Machine Learning Lab"},{"content":"AWS Managed Prometheus \u0026amp; Grafana is the \u0026ldquo;plug-and-play\u0026rdquo; choice for production workloads requiring minimal management, while on the other hand installing helm kube-prometheus-stack offers maximum control but requires more effort to maintain and scale effectively.\nHence for cost control and full customization, I decided to install kube-prometheus-stack on my local lab cluster.\nUnderstand Prometheus Pull-based Monitoring Flow\nExpose Metrics: Applications expose metrics in Prometheus format. Discover Targets: Kubernetes-native targets: Discovered via the Kubernetes API. Non-cloud-native targets: Defined statically or exposed through exporters. Scrape Metrics: Prometheus scrapes metrics periodically from /metrics endpoints. Store Metrics: Metrics are stored in Prometheus\u0026rsquo;s time-series database. Visualize Metrics: Grafana (in kube-prometheus-stack) is often used to query and visualize metrics. Prometheus metrics issue\nThen I found Prometheus was unable to scrape metrics from several Kubernetes components (etcd, kube-controller-manager, kube-scheduler, and kube-proxy). These targets were marked as DOWN in the Prometheus UI with errors such as: Connection refused.\nTroubleshooting Steps\nAs per Prometheus monitoring flow, let\u0026rsquo;s start with troubleshooting.\nStep 1: Check Prometheus service, pod and ServiceMonitor\nroot@asb:/home/ubuntu# kubectl get svc -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-dns ClusterIP 10.96.0.10 \u0026lt;none\u0026gt; 53/UDP,53/TCP,9153/TCP 63d metrics-server ClusterIP 10.110.18.85 \u0026lt;none\u0026gt; 443/TCP 61d prometheus-stack-kube-prom-coredns ClusterIP None \u0026lt;none\u0026gt; 9153/TCP 60d prometheus-stack-kube-prom-kube-controller-manager ClusterIP None \u0026lt;none\u0026gt; 10257/TCP 60d prometheus-stack-kube-prom-kube-etcd ClusterIP None \u0026lt;none\u0026gt; 2381/TCP 60d prometheus-stack-kube-prom-kube-proxy ClusterIP None \u0026lt;none\u0026gt; 10249/TCP 60d prometheus-stack-kube-prom-kube-scheduler ClusterIP None \u0026lt;none\u0026gt; 10259/TCP 60d prometheus-stack-kube-prom-kubelet ClusterIP None \u0026lt;none\u0026gt; 10250/TCP,10255/TCP,4194/TCP 60d root@asb:/home/ubuntu# kubectl get po -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-84b7b7fdbb-klzpf 1/1 Running 10 (42m ago) 39d calico-node-bzf6d 1/1 Running 20 (113m ago) 63d calico-node-ggw7r 1/1 Running 21 (112m ago) 63d calico-node-t8jfn 1/1 Running 20 (112m ago) 63d coredns-5d5dd8cb46-pwsvt 1/1 Running 1 (113m ago) 25h coredns-5d5dd8cb46-vthsr 1/1 Running 9 (112m ago) 39d etcd-asb-mst 1/1 Running 0 41m kube-apiserver-asb-mst 1/1 Running 19 (41m ago) 39d kube-controller-manager-asb-mst 1/1 Running 0 41m kube-proxy-956zq 1/1 Running 0 28m kube-proxy-9fcd4 1/1 Running 0 28m kube-proxy-tcnql 1/1 Running 0 28m kube-scheduler-asb-mst 1/1 Running 0 41m metrics-server-7766f59c77-xbxxr 1/1 Running 1 (112m ago) 25h root@asb:/home/ubuntu# kubectl get servicemonitors.monitoring.coreos.com -n monitoring NAME AGE prometheus-stack-grafana 60d prometheus-stack-kube-prom-alertmanager 60d prometheus-stack-kube-prom-apiserver 60d prometheus-stack-kube-prom-coredns 60d prometheus-stack-kube-prom-kube-controller-manager 60d prometheus-stack-kube-prom-kube-etcd 60d prometheus-stack-kube-prom-kube-proxy 60d prometheus-stack-kube-prom-kube-scheduler 60d prometheus-stack-kube-prom-kubelet 60d prometheus-stack-kube-prom-operator 60d prometheus-stack-kube-prom-prometheus 60d prometheus-stack-kube-state-metrics 60d prometheus-stack-prometheus-node-exporter 60d Step 2: Checked Prometheus ServiceMonitor configurations to ensure they matched the service labels, ports, and namespaces. Check Service Labels and Selectors vs Pod Labels vs ServiceMonitor Selector.\nStep 3: Verify Metrics Exposure\nkubectl port-forward -n kube-system svc/prometheus-stack-kube-prom-kube-etcd 2381:2381 curl http://localhost:2381/metrics See that API /metrics was accessible with a list of retrieves, then I need to create a debug pod to test the metrics endpoint\nkubectl run -it --rm debug-pod --image=busybox --restart=Never -- sh / # wget http://11.0.1.231:2381/metrics Connecting to 11.0.1.231:2381 (11.0.1.231:2381) wget: server returned error: HTTP/1.1 503 Service Unavailable The 503 Service Unavailable error indicates that the service is reachable, but it’s not properly routing requests to the etcd pod. This suggests a potential issue with the service configuration or the etcd pod itself.\nStep 4: Inspect ETCD configuration via /etc/kubernetes/manifests/etcd.yaml\nThe etcd pod must be configured to expose metrics on port 2381. Check the etcd deployment or static pod configuration file (often in /etc/kubernetes/manifests/ for static pods on control plane nodes).\nThe --listen-metrics-urls flag should include the :2381 endpoint: --listen-metrics-urls=http://127.0.0.1:2381\nStep 5: Implement Fixes\n# vim /etc/kubernetes/manifests/etcd.yaml --listen-metrics-urls=http://0.0.0.0:2381 # Updated ConfigMap and restart kube-proxy: kubectl edit cm kube-proxy -n kube-system kubectl rollout restart ds kube-proxy -n kube-system Confirmed that Prometheus targets were UP after the fixes.\nConclusion\nCommon Prometheus Metrics debug steps:\nCheck Prometheus pod, service, and servicemonitor status Check labels (servicemonitor label vs service label vs pod label) Check and test metrics API endpoint within Prometheus pod (e.g. /metrics) This troubleshooting experience highlights the importance of end-to-end configuration alignment in Prometheus Metrics and Targets setups, from endpoint exposure to scraping configurations. It can be a method to debug any other similar Metrics issue in a Prometheus monitoring for cloud-native and non-cloud-native applications.\nExport Redis Metrics\nHere I will run another practice to install Redis and Redis exporter, then use Prometheus to scrape its metrics and visualize them in Grafana.\n# Redis and Redis exporter deployment root@asb:~# cat k8s-redis-and-exporter-deployment.yaml --- apiVersion: v1 kind: Namespace metadata: name: redis --- apiVersion: apps/v1 kind: Deployment metadata: namespace: redis name: redis spec: replicas: 1 selector: matchLabels: app: redis template: metadata: annotations: prometheus.io/scrape: \u0026#34;true\u0026#34; prometheus.io/port: \u0026#34;9121\u0026#34; labels: app: redis spec: containers: - name: redis image: redis:4 resources: requests: cpu: 100m memory: 100Mi ports: - containerPort: 6379 - name: redis-exporter image: oliver006/redis_exporter:latest securityContext: runAsUser: 59000 runAsGroup: 59000 allowPrivilegeEscalation: false capabilities: drop: - ALL resources: requests: cpu: 100m memory: 100Mi ports: - containerPort: 9121 # redis service and servicemonitor root@asb:~# cat k8s-redis-and-exporter-svc-svcmonitor.yaml apiVersion: v1 kind: Service metadata: namespace: redis name: redis-metrics labels: app: redis spec: selector: app: redis ports: - name: http-metrics port: 9121 targetPort: 9121 --- apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: redis-monitor namespace: monitoring labels: app: redis release: prometheus-stack # important to match with kube-monitoring-stack spec: selector: matchLabels: app: redis namespaceSelector: matchNames: - redis endpoints: - port: http-metrics interval: 30s Head over to Prometheus Targets, we can see the metrics are being scraped from the Redis exporter.\nHead over to Grafana, then import the dashboard 763\n","permalink":"https://zackblog.work/posts/eks-debug-prometheus-metrics/","summary":"\u003cp\u003e\u003cem\u003eAWS Managed Prometheus \u0026amp; Grafana\u003c/em\u003e is the \u0026ldquo;plug-and-play\u0026rdquo; choice for production workloads requiring minimal management, while on the other hand installing helm \u003cem\u003ekube-prometheus-stack\u003c/em\u003e offers maximum control but requires more effort to maintain and scale effectively.\u003c/p\u003e\n\u003cp\u003eHence for cost control and full customization, I decided to install \u003cem\u003ekube-prometheus-stack\u003c/em\u003e on my local lab cluster.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUnderstand Prometheus Pull-based Monitoring Flow\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cem\u003eExpose Metrics:\u003c/em\u003e Applications expose metrics in Prometheus format.\u003c/li\u003e\n\u003cli\u003e\u003cem\u003eDiscover Targets:\u003c/em\u003e\n\u003cul\u003e\n\u003cli\u003eKubernetes-native targets: Discovered via the Kubernetes API.\u003c/li\u003e\n\u003cli\u003eNon-cloud-native targets: Defined statically or exposed through exporters.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003cem\u003eScrape Metrics:\u003c/em\u003e Prometheus scrapes metrics periodically from /metrics endpoints.\u003c/li\u003e\n\u003cli\u003e\u003cem\u003eStore Metrics:\u003c/em\u003e Metrics are stored in Prometheus\u0026rsquo;s time-series database.\u003c/li\u003e\n\u003cli\u003e\u003cem\u003eVisualize Metrics:\u003c/em\u003e Grafana (in kube-prometheus-stack) is often used to query and visualize metrics.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003ePrometheus metrics issue\u003c/strong\u003e\u003c/p\u003e","title":"EKS - Debug Prometheus Metrics"},{"content":"\u0026lsquo;Kubernetes Release vs EKS EOL\u0026rsquo;\nA Kubernetes version encompasses both the control plane and the data plane. While AWS manages and upgrades the control plane, we (cluster owner/customer) hold the responsibility for initiating upgrades for both cluster control plane as well as the data plane. When we initiate a cluster upgrade, AWS manages upgrading the control plane, and we are still responsible for initiating the upgrades of the data plane, which includes worker nodes provisioned via Self Managed node groups, Managed Node Groups, Fargate \u0026amp; other add-ons. If worker nodes are provisioned via Karpenter Controller, we can take advantage of Drift or Disruption Controller features (spec.expireAfter) for automatic node recycling and upgrade.\nUpgrade Strategy: in-place vs Blue-Green\nConsiderations when choosing an EKS upgrade strategy:\nDowntime tolerance: Consider the acceptable level of downtime for applications and services during the upgrade process. Upgrade complexity: Evaluate the complexity of application architecture, dependencies, and stateful components. Kubernetes version gap: Assess the gap between current Kubernetes version and the target version, as well as the compatibility of applications and add-ons. Resource constraints: Consider the available infrastructure resources and budget for maintaining multiple clusters during the upgrade process. A Canary strategy, similar to blue/green, except scale out the new cluster while scaling in the old cluster while ramping up workloads would minimize this. Team expertise: Evaluate team\u0026rsquo;s expertise and familiarity with managing multiple clusters and implementing traffic shifting strategies. EKS in-place Upgrade Workflow\nHere I will follow the below phases to run an in-place EKS cluster upgrade:\nPreparation Phase:\nVerify EKS Upgrade Insights and Checklist - Backup Cluster with Velero. Verify compatibility of workloads with Kubernetes 1.31. Execution Phases:\nUpgrade the Control Plane. root@zackz:~# export AWS_REGION=ap-southeast-2 root@zackz:~# export EKS_CLUSTER_NAME=ex-karpenter root@zackz:~# aws eks update-cluster-version --region ${AWS_REGION} --name $EKS_CLUSTER_NAME --kubernetes-version 1.31 { \u0026#34;update\u0026#34;: { \u0026#34;id\u0026#34;: \u0026#34;2c23dcc6-a8e0-337a-bf2a-c97de842a756\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;InProgress\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;VersionUpdate\u0026#34;, \u0026#34;params\u0026#34;: [ { \u0026#34;type\u0026#34;: \u0026#34;Version\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;1.31\u0026#34; }, { \u0026#34;type\u0026#34;: \u0026#34;PlatformVersion\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;eks.12\u0026#34; } ], \u0026#34;createdAt\u0026#34;: \u0026#34;2024-11-22T12:51:13.370000+11:00\u0026#34;, \u0026#34;errors\u0026#34;: [] } } root@zackz:~# aws eks describe-cluster --name $EKS_CLUSTER_NAME --query \u0026#34;cluster.{Name:name,Version:version}\u0026#34; --output table ----------------------------- | DescribeCluster | +---------------+-----------+ | Name | Version | +---------------+-----------+ | ex-karpenter | 1.31 | +---------------+-----------+ Upgrade EKS Addons root@zackz:~# eksctl get addon --cluster $EKS_CLUSTER_NAME 2024-11-22 13:06:07 [ℹ] Kubernetes version \u0026#34;1.31\u0026#34; in use by cluster \u0026#34;ex-karpenter\u0026#34; 2024-11-22 13:06:07 [ℹ] getting all addons 2024-11-22 13:06:09 [ℹ] to see issues for an addon run `eksctl get addon --name \u0026lt;addon-name\u0026gt; --cluster \u0026lt;cluster-name\u0026gt;` NAME VERSION STATUS ISSUES IAMROLE UPDATE AVAILABLE CONFIGURATION VALUES POD IDENTITY ASSOCIATION ROLES coredns v1.11.1-eksbuild.8 ACTIVE 0 v1.11.3-eksbuild.2,v1.11.3-eksbuild.1,v1.11.1-eksbuild.13,v1.11.1-eksbuild.11 eks-pod-identity-agent v1.3.4-eksbuild.1 ACTIVE 0 kube-proxy v1.30.6-eksbuild.3 ACTIVE 0 v1.31.2-eksbuild.3,v1.31.2-eksbuild.2,v1.31.1-eksbuild.2,v1.31.0-eksbuild.5,v1.31.0-eksbuild.2 vpc-cni v1.19.0-eksbuild.1 ACTIVE 0 root@zackz:~# aws eks update-addon \\ --cluster-name $EKS_CLUSTER_NAME \\ --addon-name coredns \\ --addon-version v1.11.3-eksbuild.2 { \u0026#34;update\u0026#34;: { \u0026#34;id\u0026#34;: \u0026#34;79ccd9e1-004e-3cc7-89bb-c7dc8b286281\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;InProgress\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;AddonUpdate\u0026#34;, \u0026#34;params\u0026#34;: [ { \u0026#34;type\u0026#34;: \u0026#34;AddonVersion\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;v1.11.3-eksbuild.2\u0026#34; } ], \u0026#34;createdAt\u0026#34;: \u0026#34;2024-11-22T13:08:18.368000+11:00\u0026#34;, \u0026#34;errors\u0026#34;: [] } } root@zackz:~# aws eks update-addon \\ --cluster-name $EKS_CLUSTER_NAME \\ --addon-name kube-proxy \\ --addon-version v1.31.2-eksbuild.3 { \u0026#34;update\u0026#34;: { \u0026#34;id\u0026#34;: \u0026#34;63288184-13f9-3d5b-8c63-9ecf5414bc82\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;InProgress\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;AddonUpdate\u0026#34;, \u0026#34;params\u0026#34;: [ { \u0026#34;type\u0026#34;: \u0026#34;AddonVersion\u0026#34;, \u0026#34;value\u0026#34;: \u0026#34;v1.31.2-eksbuild.3\u0026#34; } ], \u0026#34;createdAt\u0026#34;: \u0026#34;2024-11-22T13:08:29.617000+11:00\u0026#34;, \u0026#34;errors\u0026#34;: [] } } # after add-on upgrade root@zackz:~# eksctl get addon --cluster $EKS_CLUSTER_NAME 2024-11-22 13:10:15 [ℹ] Kubernetes version \u0026#34;1.31\u0026#34; in use by cluster \u0026#34;ex-karpenter\u0026#34; 2024-11-22 13:10:15 [ℹ] getting all addons 2024-11-22 13:10:16 [ℹ] to see issues for an addon run `eksctl get addon --name \u0026lt;addon-name\u0026gt; --cluster \u0026lt;cluster-name\u0026gt;` NAME VERSION STATUS ISSUES IAMROLE UPDATE AVAILABLE CONFIGURATION VALUES POD IDENTITY ASSOCIATION ROLES coredns v1.11.3-eksbuild.2 ACTIVE 0 eks-pod-identity-agent v1.3.4-eksbuild.1 ACTIVE 0 kube-proxy v1.31.2-eksbuild.3 ACTIVE 0 vpc-cni v1.19.0-eksbuild.1 ACTIVE 0 Upgrade Managed Node Groups or Self-Managed Nodes.\nWhen we initiate a managed node group update in EKS, the process automatically completes four phases:\nSetup Phase: Creates a new launch template version, updates the Auto Scaling group, and determines the max nodes to upgrade in parallel (default 1, up to 100). Scale Up Phase: Increases the Auto Scaling group size, ensures new nodes are ready, marks old nodes unschedulable, and excludes them from load balancers. Upgrade Phase: Randomly selects nodes to upgrade, drains pods, cordons nodes, terminates old nodes, and repeats until all nodes use the new configuration. Scale Down Phase: Reduces Auto Scaling group size back to its original values. root@zackz:~# aws eks describe-cluster --name $EKS_CLUSTER_NAME --query \u0026#34;cluster.version\u0026#34; --output text 1.31 root@zackz:~# kubectl get node NAME STATUS ROLES AGE VERSION ip-10-0-11-33.ap-southeast-2.compute.internal Ready \u0026lt;none\u0026gt; 62m v1.30.6-eks-94953ac ip-10-0-16-112.ap-southeast-2.compute.internal Ready \u0026lt;none\u0026gt; 61m v1.30.6-eks-94953ac root@zackz:~# aws eks list-nodegroups --cluster-name $EKS_CLUSTER_NAME { \u0026#34;nodegroups\u0026#34;: [ \u0026#34;karpenter-2024112200002335730000001f\u0026#34; ] } root@zackz:~# eksctl upgrade nodegroup --name=karpenter-2024112200002335730000001f --cluster=$EKS_CLUSTER_NAME --kubernetes-version=1.31 2024-11-22 13:16:35 [ℹ] upgrade of nodegroup \u0026#34;karpenter-2024112200002335730000001f\u0026#34; in progress 2024-11-22 13:16:35 [ℹ] waiting for upgrade of nodegroup \u0026#34;karpenter-2024112200002335730000001f\u0026#34; to complete 2024-11-22 13:25:40 [ℹ] nodegroup successfully upgraded root@zackz:~/zack-gitops-project/argocd-joesite# kubectl get node NAME STATUS ROLES AGE VERSION ip-10-0-2-232.ap-southeast-2.compute.internal Ready \u0026lt;none\u0026gt; 5m28s v1.31.2-eks-94953ac ip-10-0-24-55.ap-southeast-2.compute.internal Ready \u0026lt;none\u0026gt; 5m26s v1.31.2-eks-94953ac root@zackz:~/zack-gitops-project/argocd-joesite# kubectl get po -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system aws-node-58bcm 2/2 Running 0 8m11s kube-system aws-node-xq7lj 2/2 Running 0 8m13s kube-system coredns-864f654d7c-49ch8 1/1 Running 0 4m40s kube-system coredns-864f654d7c-trk6h 1/1 Running 0 7m44s kube-system eks-pod-identity-agent-5788x 1/1 Running 0 8m11s kube-system eks-pod-identity-agent-fbq9q 1/1 Running 0 8m13s kube-system karpenter-5f6bbf8cdc-gx6mn 1/1 Running 0 4m40s kube-system karpenter-5f6bbf8cdc-nh4fb 1/1 Running 0 7m43s kube-system kube-proxy-9798b 1/1 Running 0 8m13s kube-system kube-proxy-dnfgt 1/1 Running 0 8m11s Upgrade AWS Fargate Nodes.\nFor upgrading AWS Fargate nodes, we can re-start the K8s deployments so that the new pods will automatically get scheduled on the latest Kubernetes Version.\nConclusion\nBy following the above upgrade approach to move an EKS cluster from Kubernetes 1.30 to 1.31, what we have achieved:\nSeamless Control Plane Upgrade: Ensures the Kubernetes API server and control plane components are updated to 1.31 without disrupting workloads. Add-On Compatibility: Updates critical EKS add-ons (e.g., CoreDNS, kube-proxy, VPC CNI) to ensure compatibility and leverage new features in Kubernetes 1.31. Managed Node Group Updates: Automatically updates node groups to use the latest AMIs, applying new Kubernetes features, security patches, and optimized configurations while minimizing disruption. Workload Continuity: Ensures workloads remain available during the upgrade with controlled pod evictions, proper cordoning, and scaling mechanisms. ","permalink":"https://zackblog.work/posts/eks-cluster-upgrade/","summary":"\u003cp\u003e\u0026lsquo;Kubernetes Release vs EKS EOL\u0026rsquo;\u003c/p\u003e\n\u003cp\u003eA Kubernetes version encompasses both the control plane and the data plane. While AWS manages and upgrades the control plane, we (cluster owner/customer) hold the responsibility for initiating upgrades for both cluster control plane as well as the data plane. When we initiate a cluster upgrade, AWS manages upgrading the control plane, and we are still responsible for initiating the upgrades of the data plane, which includes worker nodes provisioned via Self Managed node groups, Managed Node Groups, Fargate \u0026amp; other add-ons. If worker nodes are provisioned via Karpenter Controller, we can take advantage of Drift or Disruption Controller features (spec.expireAfter) for automatic node recycling and upgrade.\u003c/p\u003e","title":"EKS - Cluster Upgrade"},{"content":"\u0026lsquo;Karpenter vs Cluster Autoscaler\u0026rsquo;\nKarpenter is more modern, flexible, and cost-efficient, making it a better choice for dynamic, complex, or large-scale workloads on EKS.\nCluster Autoscaler is simpler and integrates seamlessly with AWS Managed Node Groups, making it suitable for basic scaling needs.\nTransitioning to Karpenter from Cluster Autoscaler is a logical step when EKS cluster demands evolve toward more complex scaling with diverse workloads, cost optimization, fine-grained control over node provisioning.\nWhat Karpenter can do\nBasic and Advanced Node Management\nScaling Applications NodePools EC2 Node Class Cost Optimization\nSingle/Multi-Node Consolidation: Rebalance workloads to reduce node count On-Demand \u0026amp; Spot Split: Mix on-demand and spot instances to balance cost and reliability Scheduling Constraints\nNode and Pods Affinity \u0026amp; Taints Pod Disruption Budget Disruption Control Instance Type \u0026amp; AZ Get Started with Karpenter\nTo start with Karpenter, I will use below script to run a few steps:\nStep 1: Installation and Basic Setup Here I will set up a Kubernetes cluster using AWS EKS, configure IAM roles for service accounts to enable IRSA, install Karpenter using Helm, install eks monitoring tools eks-node-viewer to observe Karpenter scaling. - Step 2: Scaling and Resource Management Then I will define EC2 Node Class and Node pool, deploy a sample Application, change the replicas to observe Karpenter scaling behavior via eks-node-viewer. Karpenter will detect the pending pods to decide which instance type to launch to fit the workload. { \u0026#34;level\u0026#34;: \u0026#34;INFO\u0026#34;, \u0026#34;time\u0026#34;: \u0026#34;2024-11-18T12:39:19.332Z\u0026#34;, \u0026#34;logger\u0026#34;: \u0026#34;controller\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;disrupting nodeclaim(s) via delete, terminating 1 nodes (0 pods) ip-192-168-156-150.ap-southeast-2.compute.internal/c6a.large/on-demand\u0026#34;, \u0026#34;commit\u0026#34;: \u0026#34;a2875e3\u0026#34;, \u0026#34;controller\u0026#34;: \u0026#34;disruption\u0026#34;, \u0026#34;reconcileID\u0026#34;: \u0026#34;ec9527bb-ab80-4d55-b9fb-24b9083cf1e4\u0026#34;, \u0026#34;command-id\u0026#34;: \u0026#34;bf26fc67-52d3-410a-a6fd-96b00f229c5b\u0026#34;, \u0026#34;reason\u0026#34;: \u0026#34;empty\u0026#34; } { \u0026#34;level\u0026#34;: \u0026#34;INFO\u0026#34;, \u0026#34;time\u0026#34;: \u0026#34;2024-11-18T12:39:19.666Z\u0026#34;, \u0026#34;logger\u0026#34;: \u0026#34;controller\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;tainted node\u0026#34;, \u0026#34;commit\u0026#34;: \u0026#34;a2875e3\u0026#34;, \u0026#34;controller\u0026#34;: \u0026#34;node.termination\u0026#34;, \u0026#34;Node\u0026#34;: {\u0026#34;name\u0026#34;: \u0026#34;ip-192-168-156-150.ap-southeast-2.compute.internal\u0026#34;}, \u0026#34;reconcileID\u0026#34;: \u0026#34;95e576e4-b326-4870-b7e5-b64f8a013c9d\u0026#34;, \u0026#34;taint.Key\u0026#34;: \u0026#34;karpenter.sh/disrupted\u0026#34;, \u0026#34;taint.Value\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;taint.Effect\u0026#34;: \u0026#34;NoSchedule\u0026#34; } - Step 3: Clean up the resources Karpenter will detect the scale down from Application deployment, scheduled to terminate unnecessary nodes for the EKS cluster. Then I will remove all the resources created in this example.\n{ \u0026#34;level\u0026#34;: \u0026#34;INFO\u0026#34;, \u0026#34;time\u0026#34;: \u0026#34;2024-11-18T12:35:16.493Z\u0026#34;, \u0026#34;logger\u0026#34;: \u0026#34;controller\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;tainted node\u0026#34;, \u0026#34;commit\u0026#34;: \u0026#34;a2875e3\u0026#34;, \u0026#34;controller\u0026#34;: \u0026#34;node.termination\u0026#34;, \u0026#34;Node\u0026#34;: {\u0026#34;name\u0026#34;: \u0026#34;ip-192-168-71-167.ap-southeast-2.compute.internal\u0026#34;}, \u0026#34;reconcileID\u0026#34;: \u0026#34;cc1624d0-2014-4d44-92e9-c5ee1ddb316d\u0026#34;, \u0026#34;taint.Key\u0026#34;: \u0026#34;karpenter.sh/disrupted\u0026#34;, \u0026#34;taint.Value\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;taint.Effect\u0026#34;: \u0026#34;NoSchedule\u0026#34; } { \u0026#34;level\u0026#34;: \u0026#34;INFO\u0026#34;, \u0026#34;time\u0026#34;: \u0026#34;2024-11-18T12:35:49.254Z\u0026#34;, \u0026#34;logger\u0026#34;: \u0026#34;controller\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;deleted node\u0026#34;, \u0026#34;commit\u0026#34;: \u0026#34;a2875e3\u0026#34;, \u0026#34;controller\u0026#34;: \u0026#34;node.termination\u0026#34;, \u0026#34;Node\u0026#34;: {\u0026#34;name\u0026#34;: \u0026#34;ip-192-168-71-167.ap-southeast-2.compute.internal\u0026#34;}, \u0026#34;reconcileID\u0026#34;: \u0026#34;a3ef7364-5aaf-4e68-8280-c08d0b1012cc\u0026#34; } { \u0026#34;level\u0026#34;: \u0026#34;INFO\u0026#34;, \u0026#34;time\u0026#34;: \u0026#34;2024-11-18T12:35:49.497Z\u0026#34;, \u0026#34;logger\u0026#34;: \u0026#34;controller\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;deleted nodeclaim\u0026#34;, \u0026#34;commit\u0026#34;: \u0026#34;a2875e3\u0026#34;, \u0026#34;controller\u0026#34;: \u0026#34;nodeclaim.termination\u0026#34;, \u0026#34;NodeClaim\u0026#34;: {\u0026#34;name\u0026#34;: \u0026#34;default-drbmq\u0026#34;}, \u0026#34;reconcileID\u0026#34;: \u0026#34;0e20cb98-389d-4143-8177-5ec9c777c227\u0026#34;, \u0026#34;Node\u0026#34;: {\u0026#34;name\u0026#34;: \u0026#34;ip-192-168-71-167.ap-southeast-2.compute.internal\u0026#34;}, \u0026#34;provider-id\u0026#34;: \u0026#34;aws:///ap-southeast-2a/i-062db9708bac9d0dd\u0026#34; } #!/bin/bash # Setup environment mkdir eslf-karpenter \u0026amp;\u0026amp; cd eslf-karpenter export KARPENTER_NAMESPACE=\u0026#34;kube-system\u0026#34; export KARPENTER_VERSION=\u0026#34;1.0.8\u0026#34; export K8S_VERSION=\u0026#34;1.31\u0026#34; export AWS_PARTITION=\u0026#34;aws\u0026#34; export CLUSTER_NAME=\u0026#34;${USER}-karpenter-demo\u0026#34; export AWS_DEFAULT_REGION=\u0026#34;ap-southeast-2\u0026#34; # Fetch AWS Account and AMI information export AWS_ACCOUNT_ID=\u0026#34;$(aws sts get-caller-identity --query Account --output text)\u0026#34; export TEMPOUT=\u0026#34;$(mktemp)\u0026#34; export ARM_AMI_ID=\u0026#34;$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-arm64/recommended/image_id --query Parameter.Value --output text)\u0026#34; export AMD_AMI_ID=\u0026#34;$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2/recommended/image_id --query Parameter.Value --output text)\u0026#34; export GPU_AMI_ID=\u0026#34;$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-gpu/recommended/image_id --query Parameter.Value --output text)\u0026#34; # Deploy CloudFormation stack curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/v\u0026#34;${KARPENTER_VERSION}\u0026#34;/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml \u0026gt; \u0026#34;${TEMPOUT}\u0026#34; aws cloudformation deploy \\ --stack-name \u0026#34;Karpenter-${CLUSTER_NAME}\u0026#34; \\ --template-file \u0026#34;${TEMPOUT}\u0026#34; \\ --capabilities CAPABILITY_NAMED_IAM \\ --parameter-overrides \u0026#34;ClusterName=${CLUSTER_NAME}\u0026#34; # Create EKS Cluster with eksctl eksctl create cluster -f EOF apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: ${CLUSTER_NAME} region: ${AWS_DEFAULT_REGION} version: \u0026#34;${K8S_VERSION}\u0026#34; tags: karpenter.sh/discovery: ${CLUSTER_NAME} ... EOF # Fetch cluster endpoint and IAM role ARN export CLUSTER_ENDPOINT=\u0026#34;$(aws eks describe-cluster --name \u0026#34;${CLUSTER_NAME}\u0026#34; --query \u0026#34;cluster.endpoint\u0026#34; --output text)\u0026#34; export KARPENTER_IAM_ROLE_ARN=\u0026#34;arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter\u0026#34; # Install Karpenter with Helm helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \\ --version \u0026#34;${KARPENTER_VERSION}\u0026#34; \\ --namespace \u0026#34;${KARPENTER_NAMESPACE}\u0026#34; --create-namespace \\ --set \u0026#34;settings.clusterName=${CLUSTER_NAME}\u0026#34; \\ --set \u0026#34;settings.interruptionQueue=${CLUSTER_NAME}\u0026#34; \\ --set controller.resources.requests.cpu=1 \\ --set controller.resources.requests.memory=1Gi \\ --set controller.resources.limits.cpu=1 \\ --set controller.resources.limits.memory=1Gi \\ --wait # Install eks-node-viewer wget -O eks-node-viewer https://github.com/awslabs/eks-node-viewer/releases/download/v0.6.0/eks-node-viewer_Linux_x86_64 chmod +x eks-node-viewer sudo mv -v eks-node-viewer /usr/local/bin eks-node-viewer # Create NodePool and EC2NodeClass cat EOF | envsubst | kubectl apply -f - apiVersion: karpenter.sh/v1 kind: NodePool metadata: name: default spec: template: spec: requirements: - key: kubernetes.io/arch operator: In values: [\u0026#34;amd64\u0026#34;] ... EOF # Create a deployment cat EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: inflate spec: replicas: 0 selector: matchLabels: app: inflate template: metadata: labels: app: inflate spec: terminationGracePeriodSeconds: 0 securityContext: runAsUser: 1000 runAsGroup: 3000 fsGroup: 2000 containers: - name: inflate image: public.ecr.aws/eks-distro/kubernetes/pause:3.7 resources: requests: cpu: 1 securityContext: allowPrivilegeEscalation: false # Test scaling behavior kubectl scale deployment inflate --replicas 15 kubectl logs -f -n \u0026#34;${KARPENTER_NAMESPACE}\u0026#34; -l app.kubernetes.io/name=karpenter -c controller # Cleanup resources kubectl delete deployment inflate helm uninstall karpenter --namespace \u0026#34;${KARPENTER_NAMESPACE}\u0026#34; aws cloudformation delete-stack --stack-name \u0026#34;Karpenter-${CLUSTER_NAME}\u0026#34; eksctl delete cluster --name \u0026#34;${CLUSTER_NAME}\u0026#34; More Karpenter can do\nDisruption and Drift Management Disruption: Focuses on maintaining application availability during scaling or updates. Drift: Ensures nodes are reconciled with the desired state.\nCost Optimization Optimizes resource usage by consolidating workloads onto fewer nodes. Using Spot Instances to Leverage AWS spot instances for cost reduction, On-Demand \u0026amp; Spot Ratio Split\nScheduling Constraints Leverage with Node Affinity, Taints and Tolerations, Topology Spread, Pod Affinity to improve workload placement and resource utilization.\nI will see in next post to explore some Hands-On Steps for more Karpenter features:\nConfigure Pod Disruption Budgets (PDBs). Simulate disruption scenarios (e.g., delete a node). Observe Karpenter\u0026rsquo;s ability to recover workloads. Deploy a Provisioner with spot instance configuration. Create a workload that triggers the use of both on-demand and spot instances. Test consolidation by simulating reduced workloads. Configure nodeAffinity and taints in the workload YAML. Define topology spread constraints to ensure even distribution. Observe the impact of scheduling constraints on node provisioning. ","permalink":"https://zackblog.work/posts/eks-get-started-with-karpenter/","summary":"\u003cp\u003e\u0026lsquo;Karpenter vs Cluster Autoscaler\u0026rsquo;\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eKarpenter\u003c/em\u003e is more modern, flexible, and cost-efficient, making it a better choice for dynamic, complex, or large-scale workloads on EKS.\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eCluster Autoscaler\u003c/em\u003e is simpler and integrates seamlessly with AWS Managed Node Groups, making it suitable for basic scaling needs.\u003c/p\u003e\n\u003cp\u003eTransitioning to Karpenter from Cluster Autoscaler is a logical step when EKS cluster demands evolve toward more complex scaling with diverse workloads, cost optimization, fine-grained control over node provisioning.\u003c/p\u003e","title":"EKS - Get Started with Karpenter"},{"content":"\u0026lsquo;when EKS ConfigMap meet AWS Secret manager\nHere I will demo a mysql as database and a wordpress deployment as backend in EKS to reference ConfigMap and Secret via environment variables during run time by reading the environment-specific configuration (like environment names or feature flags) from a ConfigMap and sensitive information (like a database password) from a Secret.\nCreate demo resources # vim cf-demo.yaml to create demo namespace, configmap, secret and deployment root@asb:~/cf# cat secret.yaml apiVersion: v1 kind: Secret metadata: name: mysql-root-secret type: Opaque data: MYSQL_ROOT_PASSWORD: c2VjdXJlcGFzc3dvcmQ= # Base64 for \u0026#34;securepassword\u0026#34; root@asb:~/cf# cat mysql.yaml apiVersion: v1 kind: Service metadata: name: mysql-service spec: ports: - port: 3306 selector: app: mysql --- apiVersion: apps/v1 kind: Deployment metadata: name: mysql spec: selector: matchLabels: app: mysql template: metadata: labels: app: mysql spec: containers: - name: mysql image: mysql:5.7 env: - name: MYSQL_ROOT_PASSWORD valueFrom: secretKeyRef: name: mysql-root-secret key: MYSQL_ROOT_PASSWORD root@asb:~/cf# cat wordpress.yaml apiVersion: v1 kind: ConfigMap metadata: name: wordpress-config data: WORDPRESS_DB_HOST: \u0026#34;mysql-service:3306\u0026#34; WORDPRESS_DB_NAME: \u0026#34;wordpress\u0026#34; --- apiVersion: v1 kind: Secret metadata: name: wordpress-secret type: Opaque data: WORDPRESS_DB_USER: d29yZHByZXNz # Base64 for \u0026#34;wordpress\u0026#34; WORDPRESS_DB_PASSWORD: c2VjdXJlcGFzc3dvcmQ= # Base64 for \u0026#34;securepassword\u0026#34; --- apiVersion: apps/v1 kind: Deployment metadata: name: wordpress spec: replicas: 1 selector: matchLabels: app: wordpress template: metadata: labels: app: wordpress spec: containers: - name: wordpress image: wordpress:latest ports: - containerPort: 80 env: - name: WORDPRESS_DB_HOST valueFrom: configMapKeyRef: name: wordpress-config key: WORDPRESS_DB_HOST - name: WORDPRESS_DB_NAME valueFrom: configMapKeyRef: name: wordpress-config key: WORDPRESS_DB_NAME - name: WORDPRESS_DB_USER valueFrom: secretKeyRef: name: wordpress-secret key: WORDPRESS_DB_USER - name: WORDPRESS_DB_PASSWORD valueFrom: secretKeyRef: name: wordpress-secret key: WORDPRESS_DB_PASSWORD --- apiVersion: v1 kind: Service metadata: name: wordpress-service spec: selector: app: wordpress ports: - protocol: TCP port: 80 targetPort: 80 type: LoadBalancer # Or NodePort if LoadBalancer is not available Apply and verify the Configuration and Secrets in the Pod root@asb:~/cf# kubectl create ns cf-test root@asb:~/cf# kubectl apply -f . -n cf-test configmap/wordpress-config created secret/wordpress-secret created deployment.apps/wordpress created service/wordpress-service created secret/mysql-root-secret created service/mysql-service created deployment.apps/mysql created root@asb:~/cf# kubectl get all -n cf-test NAME READY STATUS RESTARTS AGE pod/mysql-fdff667f8-xzblb 1/1 Running 0 6s pod/wordpress-6dff4575b9-sgn8t 1/1 Running 0 12m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mysql-service ClusterIP 10.106.191.27 \u0026lt;none\u0026gt; 3306/TCP 6s service/wordpress-service LoadBalancer 10.97.169.213 pending 80:31208/TCP 12m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/mysql 1/1 1 1 6s deployment.apps/wordpress 1/1 1 1 12m NAME DESIRED CURRENT READY AGE replicaset.apps/mysql-fdff667f8 1 1 1 6s replicaset.apps/wordpress-6dff4575b9 1 1 1 12m # Kubectl exec into pod to verify the configmap and secret root@asb:~/cf# kubectl logs wordpress-6dff4575b9-sgn8t -n cf-test WordPress not found in /var/www/html - copying now... Complete! WordPress has been successfully copied to /var/www/html No \u0026#39;wp-config.php\u0026#39; found in /var/www/html, but \u0026#39;WORDPRESS_...\u0026#39; variables supplied; copying \u0026#39;wp-config-docker.php\u0026#39; (WORDPRESS_DB_HOST WORDPRESS_DB_NAME WORDPRESS_DB_PASSWORD WORDPRESS_DB_USER WORDPRESS_SERVICE_PORT WORDPRESS_SERVICE_PORT_80_TCP WORDPRESS_SERVICE_PORT_80_TCP_ADDR WORDPRESS_SERVICE_PORT_80_TCP_PORT WORDPRESS_SERVICE_PORT_80_TCP_PROTO WORDPRESS_SERVICE_SERVICE_HOST WORDPRESS_SERVICE_SERVICE_PORT) AH00558: apache2: Could not reliably determine the server\u0026#39;s fully qualified domain name, using 192.168.48.245. Set the \u0026#39;ServerName\u0026#39; directive globally to suppress this message AH00558: apache2: Could not reliably determine the server\u0026#39;s fully qualified domain name, using 192.168.48.245. Set the \u0026#39;ServerName\u0026#39; directive globally to suppress this message [Wed Nov 06 23:39:29.456836 2024] [mpm_prefork:notice] [pid 1:tid 1] AH00163: Apache/2.4.62 (Debian) PHP/8.2.25 configured -- resuming normal operations [Wed Nov 06 23:39:29.456923 2024] [core:notice] [pid 1:tid 1] AH00094: Command line: \u0026#39;apache2 -D FOREGROUND\u0026#39; root@asb:~/cf# kubectl exec -it wordpress-6dff4575b9-sgn8t -n cf-test -- env | grep WORDPRESS_DB WORDPRESS_DB_NAME=wordpress WORDPRESS_DB_USER=wordpress WORDPRESS_DB_PASSWORD=securepassword WORDPRESS_DB_HOST=mysql-service:3306 IRSA with AWS Secret Manager to ensure security best practices in EKS\nSome best practices for using secrets in AWS EKS include:\nAvoid Storing Secrets in Kubernetes: Instead of storing sensitive information in Kubernetes Secrets, use AWS Secrets Manager to store secrets securely. Use IAM Roles for Authorization: Rely on IAM roles and policies to control access to secrets in AWS Secrets Manager, ensuring that only authorized pods can retrieve specific secrets. Retrieve Secrets at Runtime: Configure your application to retrieve secrets directly from AWS Secrets Manager at runtime, using an SDK or API, rather than storing secrets as environment variables. Automate Secret Rotation: Set up AWS Secrets Manager to rotate secrets automatically, reducing the risk of stale or compromised credentials. Here I will run a Step-by-Step Guide to Using AWS Secrets Manager with Kubernetes in EKS so the application can retrieve secrets at Runtime from AWS Secret Manager with IRSA (Identity and Access Management Roles for Service Accounts).\nset up AWS Secrets Manager and create a secret, then create a IAM policy attach to IAM role, use EKS OIDC provider together with namespace and service account, so our deployment with an IAM role will have access to the secret in AWS. # Create a new AWS Secrets Manager secret aws secretsmanager create-secret \\ --name prod/myapp/db \\ --description \u0026#34;Secret for database credentials\u0026#34; \\ --secret-string \u0026#39;{\u0026#34;DB_USER\u0026#34;: \u0026#34;mydatabaseuser\u0026#34;, \u0026#34;DB_PASSWORD\u0026#34;: \u0026#34;mypassword\u0026#34;}\u0026#39; # create an IAM policy named ReadProdMyAppDBSecret with read-only access to the prod/myapp/db secret aws iam create-policy \\ --policy-name ReadProdMyAppDBSecret \\ --policy-document \u0026#39;{ \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: \u0026#34;secretsmanager:GetSecretValue\u0026#34;, \u0026#34;Resource\u0026#34;: \u0026#34;arn:aws:secretsmanager:$REGION:$ACCOUNT_ID:secret:prod/myapp/db*\u0026#34; } ] }\u0026#39; # get EKS OIDC provider aws eks describe-cluster --name zack-eks-cluster --query \u0026#34;cluster.identity.oidc.issuer\u0026#34; --output text # create the trust policy file, use EKS OIDC provider together with namespace and service account # vim trust-policy.json { \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Principal\u0026#34;: { \u0026#34;Federated\u0026#34;: \u0026#34;arn:aws:iam::$ACCOUNT_ID:oidc-provider/oidc.eks.$REGION.amazonaws.com/id/zack-eks-cluster\u0026#34; }, \u0026#34;Action\u0026#34;: \u0026#34;sts:AssumeRoleWithWebIdentity\u0026#34;, \u0026#34;Condition\u0026#34;: { \u0026#34;StringEquals\u0026#34;: { \u0026#34;arn:aws:iam::$ACCOUNT_ID:oidc-provider/oidc.eks.$REGION.amazonaws.com/id/zack-eks-cluster:sub\u0026#34;: \u0026#34;system:serviceaccount:ns-zack-irsa-demo:sa-zack-irsa-demo\u0026#34; } } } ] } # create the IAM role with the trust policy aws iam create-role \\ --role-name EKSSecretsAccessRole \\ --assume-role-policy-document file://trust-policy.json # Attach the Policy to the Role aws iam attach-role-policy \\ --role-name EKSSecretsAccessRole \\ --policy-arn arn:aws:iam:#ACCOUNT_ID:policy/ReadProdMyAppDBSecret Create namespace and service account in EKS and annotate it with above IAM role ARN # create service account \u0026#34;sa-zack-irsa-demo\u0026#34; in namespace \u0026#34;ns-zack-irsa-demo\u0026#34; with annotation of IAM role ARN: apiVersion: v1 kind: ServiceAccount metadata: name: sa-zack-irsa-demo namespace: ns-zack-irsa-demo annotations: eks.amazonaws.com/role-arn: arn:aws:iam::$ACCOUNT_ID:role/EKSSecretsAccessRole Create Zackblog Deployment to use the service account created above, so that pods launched by the Deployment will inherit permissions to access the secret apiVersion: apps/v1 kind: Deployment metadata: name: zackblog-app namespace: ns-zack-irsa-demo spec: replicas: 1 selector: matchLabels: app: zackblog-app template: metadata: labels: app: zackblog-app spec: serviceAccountName: sa-zack-irsa-demo # Use the custom service account containers: - name: zackblog-app image: zackz001/gitops-jekyll:latest Modify Application Code to Retrieve Secrets at Runtime, bellow python with boto3 can retrieve the DB\\_USER and DB\\_PASSWORD fields from the secret in AWS Secrets Manager import boto3 import json def get_secret(): secret_name = \u0026#34;prod/myapp/db\u0026#34; region_name = \u0026#34;ap-southeast-2\u0026#34; client = boto3.client(\u0026#34;secretsmanager\u0026#34;, region_name=region_name) response = client.get_secret_value(SecretId=secret_name) secret = json.loads(response[\u0026#34;SecretString\u0026#34;]) db_user = secret[\u0026#34;DB_USER\u0026#34;] db_password = secret[\u0026#34;DB_PASSWORD\u0026#34;] return db_user, db_password Conclusion\nThis setup with EKS and IAM role with Service Account (IRSA), which combines AWS Secrets Manager, IAM roles, and Kubernetes Service Accounts, provides a secure and scalable way to manage sensitive information in Kubernetes on EKS.\nMore to be considered to ensure EKS security best practices:\nEnable Encryption: AWS Secrets Manager secrets are encrypted by default Limit IAM Permissions: Follow the principle of least privilege by restricting IAM permissions to only the required secrets. Audit and Monitor Access: Use AWS CloudTrail to monitor access to secrets for security and auditing purposes. ","permalink":"https://zackblog.work/posts/eks-enable-iam-role-for-service-accounts-irsa/","summary":"\u003cp\u003e\u0026lsquo;when EKS ConfigMap meet AWS Secret manager\u003c/p\u003e\n\u003cp\u003eHere I will demo a mysql as database and a wordpress deployment as backend in EKS to reference ConfigMap and Secret via environment variables during run time by reading the environment-specific configuration (like environment names or feature flags) from a ConfigMap and sensitive information (like a database password) from a Secret.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eCreate demo resources\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# vim cf-demo.yaml to create demo namespace,  configmap, secret and deployment\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eroot@asb:~/cf# cat secret.yaml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eSecret\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emysql-root-secret\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eOpaque\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003edata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eMYSQL_ROOT_PASSWORD\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ec2VjdXJlcGFzc3dvcmQ= \u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"c\"\u003e# Base64 for \u0026#34;securepassword\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eroot@asb:~/cf# cat mysql.yaml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eService\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emysql-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003eport\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e3306\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eselector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emysql\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nn\"\u003e---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapps/v1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeployment\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emysql\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eselector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ematchLabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emysql\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003etemplate\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003elabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emysql\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003econtainers\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emysql\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eimage\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emysql:5.7\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eenv\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eMYSQL_ROOT_PASSWORD\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003evalueFrom\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"nt\"\u003esecretKeyRef\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003emysql-root-secret\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ekey\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eMYSQL_ROOT_PASSWORD\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eroot@asb:~/cf# cat wordpress.yaml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eConfigMap\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewordpress-config\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003edata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eWORDPRESS_DB_HOST\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;mysql-service:3306\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eWORDPRESS_DB_NAME\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;wordpress\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nn\"\u003e---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eSecret\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewordpress-secret\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eOpaque\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003edata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eWORDPRESS_DB_USER\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ed29yZHByZXNz \u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"c\"\u003e# Base64 for \u0026#34;wordpress\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eWORDPRESS_DB_PASSWORD\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ec2VjdXJlcGFzc3dvcmQ= \u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"c\"\u003e# Base64 for \u0026#34;securepassword\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nn\"\u003e---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapps/v1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeployment\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewordpress\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ereplicas\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eselector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ematchLabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewordpress\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003etemplate\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003elabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewordpress\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003econtainers\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewordpress\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eimage\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewordpress:latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003econtainerPort\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e80\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eenv\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eWORDPRESS_DB_HOST\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003evalueFrom\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"nt\"\u003econfigMapKeyRef\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewordpress-config\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ekey\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eWORDPRESS_DB_HOST\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eWORDPRESS_DB_NAME\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003evalueFrom\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"nt\"\u003econfigMapKeyRef\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewordpress-config\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ekey\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eWORDPRESS_DB_NAME\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eWORDPRESS_DB_USER\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003evalueFrom\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"nt\"\u003esecretKeyRef\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewordpress-secret\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ekey\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eWORDPRESS_DB_USER\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eWORDPRESS_DB_PASSWORD\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003evalueFrom\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"nt\"\u003esecretKeyRef\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewordpress-secret\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ekey\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eWORDPRESS_DB_PASSWORD\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nn\"\u003e---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eService\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewordpress-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eselector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewordpress\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003eprotocol\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eTCP\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eport\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e80\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003etargetPort\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e80\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eLoadBalancer \u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"c\"\u003e# Or NodePort if LoadBalancer is not available\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eApply and verify the Configuration and Secrets in the Pod\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eroot@asb:~/cf# kubectl create ns cf-test\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eroot@asb:~/cf# kubectl apply -f . -n cf-test\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003econfigmap/wordpress-config created\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003esecret/wordpress-secret created\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edeployment.apps/wordpress created\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eservice/wordpress-service created\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003esecret/mysql-root-secret created\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eservice/mysql-service created\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edeployment.apps/mysql created\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eroot@asb:~/cf# kubectl get all -n cf-test\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eNAME                             READY   STATUS    RESTARTS   AGE\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003epod/mysql-fdff667f8-xzblb        1/1     Running   \u003cspan class=\"m\"\u003e0\u003c/span\u003e          6s\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003epod/wordpress-6dff4575b9-sgn8t   1/1     Running   \u003cspan class=\"m\"\u003e0\u003c/span\u003e          12m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eNAME                        TYPE           CLUSTER-IP      EXTERNAL-IP   PORT\u003cspan class=\"o\"\u003e(\u003c/span\u003eS\u003cspan class=\"o\"\u003e)\u003c/span\u003e        AGE\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eservice/mysql-service       ClusterIP      10.106.191.27   \u0026lt;none\u0026gt;        3306/TCP       6s\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eservice/wordpress-service   LoadBalancer   10.97.169.213   pending       80:31208/TCP   12m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eNAME                        READY   UP-TO-DATE   AVAILABLE   AGE\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edeployment.apps/mysql       1/1     \u003cspan class=\"m\"\u003e1\u003c/span\u003e            \u003cspan class=\"m\"\u003e1\u003c/span\u003e           6s\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edeployment.apps/wordpress   1/1     \u003cspan class=\"m\"\u003e1\u003c/span\u003e            \u003cspan class=\"m\"\u003e1\u003c/span\u003e           12m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eNAME                                   DESIRED   CURRENT   READY   AGE\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ereplicaset.apps/mysql-fdff667f8        \u003cspan class=\"m\"\u003e1\u003c/span\u003e         \u003cspan class=\"m\"\u003e1\u003c/span\u003e         \u003cspan class=\"m\"\u003e1\u003c/span\u003e       6s\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ereplicaset.apps/wordpress-6dff4575b9   \u003cspan class=\"m\"\u003e1\u003c/span\u003e         \u003cspan class=\"m\"\u003e1\u003c/span\u003e         \u003cspan class=\"m\"\u003e1\u003c/span\u003e       12m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# Kubectl exec into pod to verify the configmap and secret\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eroot@asb:~/cf# kubectl logs wordpress-6dff4575b9-sgn8t -n cf-test\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eWordPress not found in /var/www/html - copying now...\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eComplete! WordPress has been successfully copied to /var/www/html\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eNo \u003cspan class=\"s1\"\u003e\u0026#39;wp-config.php\u0026#39;\u003c/span\u003e found in /var/www/html, but \u003cspan class=\"s1\"\u003e\u0026#39;WORDPRESS_...\u0026#39;\u003c/span\u003e variables supplied\u003cspan class=\"p\"\u003e;\u003c/span\u003e copying \u003cspan class=\"s1\"\u003e\u0026#39;wp-config-docker.php\u0026#39;\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003eWORDPRESS_DB_HOST WORDPRESS_DB_NAME WORDPRESS_DB_PASSWORD WORDPRESS_DB_USER WORDPRESS_SERVICE_PORT WORDPRESS_SERVICE_PORT_80_TCP WORDPRESS_SERVICE_PORT_80_TCP_ADDR WORDPRESS_SERVICE_PORT_80_TCP_PORT WORDPRESS_SERVICE_PORT_80_TCP_PROTO WORDPRESS_SERVICE_SERVICE_HOST WORDPRESS_SERVICE_SERVICE_PORT\u003cspan class=\"o\"\u003e)\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eAH00558: apache2: Could not reliably determine the server\u003cspan class=\"s1\"\u003e\u0026#39;s fully qualified domain name, using 192.168.48.245. Set the \u0026#39;\u003c/span\u003eServerName\u003cspan class=\"s1\"\u003e\u0026#39; directive globally to suppress this message\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"s1\"\u003eAH00558: apache2: Could not reliably determine the server\u0026#39;\u003c/span\u003es fully qualified domain name, using 192.168.48.245. Set the \u003cspan class=\"s1\"\u003e\u0026#39;ServerName\u0026#39;\u003c/span\u003e directive globally to suppress this message\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003eWed Nov \u003cspan class=\"m\"\u003e06\u003c/span\u003e 23:39:29.456836 2024\u003cspan class=\"o\"\u003e]\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003empm_prefork:notice\u003cspan class=\"o\"\u003e]\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003epid 1:tid 1\u003cspan class=\"o\"\u003e]\u003c/span\u003e AH00163: Apache/2.4.62 \u003cspan class=\"o\"\u003e(\u003c/span\u003eDebian\u003cspan class=\"o\"\u003e)\u003c/span\u003e PHP/8.2.25 configured -- resuming normal operations\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"o\"\u003e[\u003c/span\u003eWed Nov \u003cspan class=\"m\"\u003e06\u003c/span\u003e 23:39:29.456923 2024\u003cspan class=\"o\"\u003e]\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003ecore:notice\u003cspan class=\"o\"\u003e]\u003c/span\u003e \u003cspan class=\"o\"\u003e[\u003c/span\u003epid 1:tid 1\u003cspan class=\"o\"\u003e]\u003c/span\u003e AH00094: Command line: \u003cspan class=\"s1\"\u003e\u0026#39;apache2 -D FOREGROUND\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eroot@asb:~/cf# kubectl \u003cspan class=\"nb\"\u003eexec\u003c/span\u003e -it wordpress-6dff4575b9-sgn8t -n cf-test -- env \u003cspan class=\"p\"\u003e|\u003c/span\u003e grep WORDPRESS_DB\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eWORDPRESS_DB_NAME\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003ewordpress\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eWORDPRESS_DB_USER\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003ewordpress\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eWORDPRESS_DB_PASSWORD\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003esecurepassword\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eWORDPRESS_DB_HOST\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003emysql-service:3306\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eIRSA with AWS Secret Manager to ensure security best practices in EKS\u003c/strong\u003e\u003c/p\u003e","title":"EKS - Enable IAM role for Service Accounts (IRSA)"},{"content":"Valero is an open-source tool for storing, restoring, and migrating Kubernetes cluster resources and persistent volumes. Valero provides a way to hold the entire state of a Kubernetes cluster, all its objects and their consistent numbers, store backup files to Cloud storage like AWS S3, and then restore them to a previous state to ensure K8S data resilience, disaster recovery results and easy transport between clusters.\nVelero for EKS\nBackup EKS cluster using Velero, can be followed by the below path:\nInstall Velero in EKS cluster AWS CLI is configured with the correct credentials S3 bucket created and configured for Velero to communicate for backup and restore Prepare AWS S3 bucket and IAM user # create S3 bucket for Velero BUCKET=zz-asb-k8s-velero-backup-bucket REGION=ap-southeast-2 aws s3api create-bucket \\ --bucket $BUCKET \\ --region $REGION \\ --create-bucket-configuration LocationConstraint=$REGION # create IAM user for Velero aws iam create-user --user-name velero cat \u0026gt; velero-policy.json \u0026lt;\u0026lt;EOF { \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: [ \u0026#34;ec2:DescribeVolumes\u0026#34;, \u0026#34;ec2:DescribeSnapshots\u0026#34;, \u0026#34;ec2:CreateTags\u0026#34;, \u0026#34;ec2:CreateVolume\u0026#34;, \u0026#34;ec2:CreateSnapshot\u0026#34;, \u0026#34;ec2:DeleteSnapshot\u0026#34; ], \u0026#34;Resource\u0026#34;: \u0026#34;*\u0026#34; }, { \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: [ \u0026#34;s3:GetObject\u0026#34;, \u0026#34;s3:DeleteObject\u0026#34;, \u0026#34;s3:PutObject\u0026#34;, \u0026#34;s3:AbortMultipartUpload\u0026#34;, \u0026#34;s3:ListMultipartUploadParts\u0026#34; ], \u0026#34;Resource\u0026#34;: [ \u0026#34;arn:aws:s3:::${BUCKET}/*\u0026#34; ] }, { \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: [ \u0026#34;s3:ListBucket\u0026#34; ], \u0026#34;Resource\u0026#34;: [ \u0026#34;arn:aws:s3:::${BUCKET}\u0026#34; ] } ] } EOF aws iam put-user-policy \\ --user-name velero \\ --policy-name velero \\ --policy-document file://velero-policy.json aws iam create-access-key --user-name velero vim credentials-velero [default] aws_access_key_id=\u0026lt;AWS_ACCESS_KEY_ID\u0026gt; aws_secret_access_key=\u0026lt;AWS_SECRET_ACCESS_KEY\u0026gt; Install Velero # install Velero with AWS s3 and IAM user cred velero install \\ --provider aws \\ --bucket $BUCKET \\ --secret-file ./credentials-velero \\ --backup-location-config region=$REGION \\ --snapshot-location-config region=$REGION # check for Velero deployment status kubectl get all -n velero NAME READY STATUS RESTARTS AGE pod/velero-86c547688f-s6l4d 1/1 Running 3 (17m ago) 42h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/velero 1/1 1 1 25d NAME DESIRED CURRENT READY AGE replicaset.apps/velero-657bc85678 0 0 0 25d replicaset.apps/velero-86c547688f 1 1 1 42h # verify Velero version and S3 backup location velero version Client: Version: v1.11.0 Git commit: 0da2baa908c88ec3c45da15001f6a4b0bda64ae2 Server: Version: v1.11.0 velero backup-location get NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT default aws zz-asb-k8s-velero-backup-bucket available 2024-09-11 06:49:39 +0000 UTC ReadWrite true Initial namespace based backup and a full cluster backup, then verify backup status # mamually initiate a cluster full backup and a namespace backup velero backup logs full-cluster-backup velero backup create eks-backup --include-namespaces zackblog-dev velero get backup NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR eks-backup Completed 0 0 2024-10-11 07:17:58 +0000 UTC 29d default \u0026lt;none\u0026gt; full-cluster-backup Completed 2 0 2024-10-11 07:15:51 +0000 UTC 29d default \u0026lt;none\u0026gt; # check and verify backup details velero backup logs eks-backup velero backup describe eks-backup Name: eks-backup Namespace: velero Labels: velero.io/storage-location=default Annotations: velero.io/source-cluster-k8s-gitversion=v1.31.1 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=31 Phase: Completed Namespaces: Included: zackblog-dev Excluded: \u0026lt;none\u0026gt; Resources: Included: * Excluded: \u0026lt;none\u0026gt; Cluster-scoped: auto Restore from previous backup after deleting deployment under a namespace Now we are going to validate Velero restore from the previous backup, we will first delete the deployment under namespace zackblog-dev and then restore it from the previous backup eks-backup.\n# existing resource under namespace kubectl get all -n zackblog-dev NAME READY STATUS RESTARTS AGE pod/zackweb-674759b48f-b9frj 1/1 Running 1 (127m ago) 4h10m pod/zackweb-674759b48f-zl6gq 1/1 Running 1 (127m ago) 4h10m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/zackweb-service LoadBalancer 10.111.140.121 \u0026lt;none\u0026gt; 80:31132/TCP 7h56m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/zackweb 2/2 2 2 7h56m NAME DESIRED CURRENT READY AGE replicaset.apps/zackweb-55547554f6 0 0 0 7h56m replicaset.apps/zackweb-5bd544cfb4 0 0 0 7h39m replicaset.apps/zackweb-674759b48f 2 2 2 4h10m replicaset.apps/zackweb-f4f74b898 0 0 0 4h13m # delete deployment kubectl delete deployments.apps -n zackblog-dev zackweb deployment.apps \u0026#34;zackweb\u0026#34; deleted kubectl get all -n zackblog-dev NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/zackweb-service LoadBalancer 10.111.140.121 \u0026lt;none\u0026gt; 80:31132/TCP 7h57m # restore via velero velero restore create --from-backup eks-backup Restore request \u0026#34;eks-backup-20241011083747\u0026#34; submitted successfully. Run velero restore describe eks-backup-20241011083747 or velero restore logs eks-backup-20241011083747 for more details. # verify velero restore status velero restore get NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR eks-backup-20241011083747 eks-backup Completed 2024-10-11 08:37:48 +0000 UTC 2024-10-11 08:37:49 +0000 UTC 0 3 2024-10-11 08:37:48 +0000 UTC \u0026lt;none\u0026gt; kubectl get all -n zackblog-dev NAME READY STATUS RESTARTS AGE pod/zackweb-674759b48f-b9frj 1/1 Running 0 18s pod/zackweb-674759b48f-zl6gq 1/1 Running 0 18s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/zackweb-service LoadBalancer 10.111.140.121 \u0026lt;none\u0026gt; 80:31132/TCP 7h58m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/zackweb 2/2 2 2 17s NAME DESIRED CURRENT READY AGE replicaset.apps/zackweb-55547554f6 0 0 0 18s replicaset.apps/zackweb-5bd544cfb4 0 0 0 18s replicaset.apps/zackweb-674759b48f 2 2 2 18s replicaset.apps/zackweb-f4f74b898 0 0 0 17s Scheduled Backups Velero can be set to perform scheduled backups at regular intervals, here we will create a scheduled Backup hourly.\nvelero schedule create daily-backup --schedule \u0026#34;0 * * * *\u0026#34; # Cron schedules use the following format # ┌───────────── minute (0 - 59) # │ ┌───────────── hour (0 - 23) # │ │ ┌───────────── day of the month (1 - 31) # │ │ │ ┌───────────── month (1 - 12) # │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday; # │ │ │ │ │ 7 is also Sunday on some systems) # │ │ │ │ │ # │ │ │ │ │ # * * * * * # validate schedule velero schedule get NAME STATUS CREATED SCHEDULE BACKUP TTL LAST BACKUP SELECTOR PAUSED hourly-backup Enabled 2024-09-16 03:48:52 +0000 UTC 0 * * * * 72h0m0s 45m ago \u0026lt;none\u0026gt; false # check scheduled backup root@asb:~# velero backup get NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR eks-backup Completed 0 0 2024-10-11 07:17:58 +0000 UTC 29d default \u0026lt;none\u0026gt; full-cluster-backup Completed 2 0 2024-10-11 07:15:51 +0000 UTC 29d default \u0026lt;none\u0026gt; hourly-backup-20241011080039 Completed 2 0 2024-10-11 08:00:39 +0000 UTC 2d default \u0026lt;none\u0026gt; hourly-backup-20241011070039 Completed 2 0 2024-10-11 07:00:39 +0000 UTC 2d default \u0026lt;none\u0026gt; hourly-backup-20241011063839 Completed 2 0 2024-10-11 06:38:42 +0000 UTC 2d default \u0026lt;none\u0026gt; hourly-backup-20241011050003 Completed 2 0 2024-10-11 05:00:03 +0000 UTC 2d default \u0026lt;none\u0026gt; hourly-backup-20241011040003 Completed 2 0 2024-10-11 04:00:03 +0000 UTC 2d default \u0026lt;none\u0026gt; hourly-backup-20241011030003 Completed 2 0 2024-10-11 03:00:03 +0000 UTC 2d default \u0026lt;none\u0026gt; hourly-backup-20241011020003 Completed 2 0 2024-10-11 02:00:03 +0000 UTC 2d default \u0026lt;none\u0026gt; Cloud backup storage verification Verify backups in S3 bucket\nConclusion\nHere\u0026rsquo;s a summary of what we covered and achieved in the common Velero administrative tasks for backup and restore on EKS cluster:\nReliable Backup System: Set up a robust system to back up your Kubernetes cluster, including both application data and persistent volumes. Restore Flexibility: Gained the ability to restore the cluster to specific points in time, including partial or full cluster recovery. Disaster Recovery: Developed a strategy for recovering from a full cluster failure using Velero and S3 storage. Automated Backups: Automated backups with scheduling to ensure that regular backups are taken without manual effort. Storage Efficiency: Managed backup retention to save on storage costs and keep backups organized. Monitoring and Logs: Monitored the status of all backup and restore tasks through detailed logging and error reporting. ","permalink":"https://zackblog.work/posts/eks-cluster-backup-with-velero/","summary":"\u003cp\u003eValero is an open-source tool for storing, restoring, and migrating Kubernetes cluster resources and persistent volumes. Valero provides a way to hold the entire state of a Kubernetes cluster, all its objects and their consistent numbers, store backup files to Cloud storage like AWS S3, and then restore them to a previous state to ensure K8S data resilience, disaster recovery results and easy transport between clusters.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eVelero for EKS\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eBackup EKS cluster using Velero, can be followed by the below path:\u003c/p\u003e","title":"EKS - Cluster Backup with Velero"},{"content":"\u0026ldquo;After scalling, let\u0026rsquo;s go EKS Security !\u0026rdquo;\nIn the last post, I was able to implement EKS cluster autoscaler and Horizontal Pod Autoscaler (HPA), in this post I will continue with EKS security practice with Kube-Bench and OPA Gatekeeper.\nkube-bench: kube-bench is a tool that checks Kubernetes clusters against the CIS (Center for Internet Security) benchmarks, a set of best practices for securing Kubernetes. It is critical to ensure that a cluster complies with these security guidelines, helping identify potential vulnerabilities and misconfigurations. Key features include generating detailed audit reports, performing automated compliance checks, and easily integrating into existing CI/CD pipelines for continuous security assessments.\nOPA Gatekeeper: OPA (Open Policy Agent) is a policy engine that allows to define and enforce policies across various services, including Kubernetes. OPA Gatekeeper extends OPA\u0026rsquo;s functionality specifically for Kubernetes by providing admission control, enabling the enforcement of custom policies on resources before they are created or modified. The benefits of using OPA Gatekeeper include consistent policy enforcement across the cluster, fine-grained control over resource configurations, and reducing the risk of misconfigurations by ensuring compliance with defined rules.\nkube-bench in EKS\nWe will be based on official cis-kubernetes-benchmark-support webpage, to apply the kube-bench job yaml for EKS, when the job completed, the results are held in the pod\u0026rsquo;s logs.\n$ kubectl.exe get node NAME STATUS ROLES AGE VERSION ip-172-31-37-215.ap-southeast-2.compute.internal Ready \u0026lt;none\u0026gt; 2m29s v1.31.0-eks-a737599 # apply the job yaml for EKS kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/refs/heads/main/job-eks.yaml # check the job status $ kubectl get pods NAME READY STATUS RESTARTS AGE kube-bench-j76s9 0/1 ContainerCreating 0 3s # Wait for a few seconds for the job to complete $ kubectl get pods NAME READY STATUS RESTARTS AGE kube-bench-j76s9 0/1 Completed 0 11s # The results are held in the pod\u0026#39;s logs kubectl logs kube-bench-j76s9 [INFO] 3 Worker Node Security Configuration [INFO] 3.1 Worker Node Configuration Files [PASS] 3.1.1 Ensure that the kubeconfig file permissions are set to 644 or more restrictive (Manual) [PASS] 3.1.2 Ensure that the kubelet kubeconfig file ownership is set to root:root (Manual) [PASS] 3.1.3 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Manual) [PASS] 3.1.4 Ensure that the kubelet configuration file ownership is set to root:root (Manual) [INFO] 3.2 Kubelet [PASS] 3.2.1 Ensure that the Anonymous Auth is Not Enabled (Automated) [PASS] 3.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) [PASS] 3.2.3 Ensure that a Client CA File is Configured (Manual) [PASS] 3.2.4 Ensure that the --read-only-port is disabled (Manual) [PASS] 3.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automated) [PASS] 3.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) [PASS] 3.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) [WARN] 3.2.8 Ensure that the --hostname-override argument is not set (Manual) [WARN] 3.2.9 Ensure that the --eventRecordQPS argument is set to 0 or a level which ensures appropriate event capture (Automated) [PASS] 3.2.10 Ensure that the --rotate-certificates argument is not present or is set to true (Manual) [PASS] 3.2.11 Ensure that the RotateKubeletServerCertificate argument is set to true (Manual) [INFO] 3.3 Container Optimized OS [WARN] 3.3.1 Prefer using a container-optimized OS when possible (Manual) == Remediations node == 3.2.8 Edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and remove the --hostname-override argument from the KUBELET_SYSTEM_PODS_ARGS variable. Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service 3.2.9 If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service 3.3.1 audit test did not run: No tests defined == Summary node == 13 checks PASS 0 checks FAIL 3 checks WARN 0 checks INFO == Summary total == 13 checks PASS 0 checks FAIL 3 checks WARN 0 checks INFO Based on the scan result, the EKS managed worker node can be secure with 13 checks pass and 0 checks fail, by following the remediation suggestion we can set the EKS cluster compliant with the CIS benchmark.\nOPA Gatekeeper in EKS\nInstall the OPA Gatekeeper: We will first install Gatekeeper in EKS cluster by applying the Gatekeeper installation manifests from the official repository.\n$ kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.12.0/deploy/gatekeeper.yaml namespace/gatekeeper-system created resourcequota/gatekeeper-critical-pods created customresourcedefinition.apiextensions.k8s.io/assign.mutations.gatekeeper.sh created $ kubectl.exe get all -n gatekeeper-system NAME READY STATUS RESTARTS AGE pod/gatekeeper-audit-777f449c79-h7lzn 1/1 Running 1 (3m35s ago) 3m42s pod/gatekeeper-controller-manager-84bf857ff7-rzlb8 1/1 Running 0 3m42s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/gatekeeper-webhook-service ClusterIP 10.100.126.57 \u0026lt;none\u0026gt; 443/TCP 3m42s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/gatekeeper-audit 1/1 1 1 3m42s deployment.apps/gatekeeper-controller-manager 1/1 1 1 3m42s NAME DESIRED CURRENT READY AGE replicaset.apps/gatekeeper-audit-777f449c79 1 1 1 3m42s replicaset.apps/gatekeeper-controller-manager-84bf857ff7 1 1 1 3m42s OPA Gatekeeper works with two main components:\nConstraintTemplate: Defines a reusable logic for policies.\nConstraint: Applies specific policies based on the logic defined in the ConstraintTemplate.\nHere we will create a ConstraintTemplate and required-label-policy to enforce required labels during pod creation, and run a test pod with and without required labels to test the policy enforcement.\n# Create a ConstraintTemplate and policy to enforce require-labels for pod creation vim required-label-template.yaml apiVersion: templates.gatekeeper.sh/v1 kind: ConstraintTemplate metadata: name: k8srequiredlabels spec: crd: spec: names: kind: K8sRequiredLabels targets: - target: admission.k8s.gatekeeper.sh rego: | package k8srequiredlabels violation[{\u0026#34;msg\u0026#34;: msg}] { required_label := \u0026#34;required-label\u0026#34; not input.review.object.metadata.labels[required_label] msg := sprintf(\u0026#34;You must provide the \u0026#39;%s\u0026#39; label on every resource.\u0026#34;, [required_label]) } --- apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sRequiredLabels metadata: name: require-labels spec: match: kinds: - apiGroups: [\u0026#34;\u0026#34;] kinds: [\u0026#34;Pod\u0026#34;] # Apply the ConstraintTemplate and policy $ kubectl.exe apply -f required-label-template.yaml constrainttemplate.templates.gatekeeper.sh/k8srequiredlabels configured k8srequiredlabels.constraints.gatekeeper.sh/require-labels created # Validate constraint templates $ kubectl get constrainttemplates NAME AGE k8srequiredlabels 15s Then we need to create a test pod with and without required labels to test the policy enforcement\n$ vim test-pod.yaml apiVersion: v1 kind: Pod metadata: name: test-pod spec: containers: - name: test-container image: nginx $ kubectl apply -f test-pod.yaml Error from server (Forbidden): error when creating \u0026#34;test-pod.yaml\u0026#34;: admission webhook \u0026#34;validation.gatekeeper.sh\u0026#34; denied the request: [require-labels] You must provide the \u0026#39;required-label\u0026#39; label on every resource. $ vim test-pod-labeled.yaml apiVersion: v1 kind: Pod metadata: name: test-pod-labeled labels: required-label: \u0026#34;true\u0026#34; spec: containers: - name: test-container image: nginx $ kubectl apply -f test-pod-labeled.yaml pod/test-pod-labeled created zack@zackz MINGW64 /f/1/terraform-eks (master) $ kubectl.exe get po NAME READY STATUS RESTARTS AGE kube-bench-xrx8g 0/1 Completed 0 21m test-pod-labeled 0/1 ContainerCreating 0 5s $ kubectl.exe get po NAME READY STATUS RESTARTS AGE kube-bench-xrx8g 0/1 Completed 0 22m test-pod-labeled 1/1 Running 0 24s Based on the constraint we created before, the policy prevents the pod from being created with\nError from server (Forbidden): error when creating \u0026quot;test-pod.yaml\u0026quot;: admission webhook \u0026quot;validation.gatekeeper.sh\u0026quot; denied the request: [require-labels] You must provide the 'required-label' label on every resource\nthis can be expanded to more strict rules to enforce k8s cluster to achieve the desired control.\nConclusion\nIn this post, we dive into the process of enhancing the security of an AWS EKS cluster by implementing and validating two critical security tools: Kube-bench and OPA Gatekeeper to get some hands-on to achieve a secure EKS environment by ensuring compliance with best practices and enforcing security policies at runtime.\n","permalink":"https://zackblog.work/posts/eks-kubebench-and-opa-gatekeeper/","summary":"\u003cp\u003e\u0026ldquo;After scalling, let\u0026rsquo;s go EKS Security !\u0026rdquo;\u003c/p\u003e\n\u003cp\u003eIn the last post, I was able to implement \u003ca href=\"/posts/eks-cluster-autoscaler-and-horizontal-pod-autoscaler-hpa/\"\u003eEKS cluster autoscaler and Horizontal Pod Autoscaler (HPA)\u003c/a\u003e, in this post I will continue with EKS security practice with Kube-Bench and OPA Gatekeeper.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ekube-bench:\u003c/strong\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003ccode\u003ekube-bench\u003c/code\u003e is a tool that checks Kubernetes clusters against the CIS (Center for Internet Security) benchmarks, a set of best practices for securing Kubernetes. It is critical to ensure that a cluster complies with these security guidelines, helping identify potential vulnerabilities and misconfigurations. Key features include generating detailed audit reports, performing automated compliance checks, and easily integrating into existing CI/CD pipelines for continuous security assessments.\u003c/p\u003e","title":"EKS - KubeBench and OPA Gatekeeper"},{"content":"\u0026ldquo;Explore how pods and EC2 nodes can be scaled in EKS cluster \u0026quot;\nAutoscaler and Horizontal Pod Autoscaler (HPA) Practice on EKS\nIn last post, I was able to deploy EKS cluster via Jenkins and Terraform:\nJenkins - Multi-Destnation Continuse Deployment with Terraform This post walks through the process of setting up an Autoscaler and Horizontal Pod Autoscaler (HPA) in an Amazon EKS cluster. We will explore how to dynamically scale the number of nodes in EKS cluster and how to autoscale K8S pods based on CPU utilization using HPA.\nPrerequisites:\nUse the AWS CLI to update kubeconfig so that kubectl can communicate with the EKS cluster $ aws eks --region ap-southeast-2 update-kubeconfig --name module-eks-cluster Updated context arn:aws:eks:ap-southeast-2:851725491342:cluster/module-eks-cluster in C:\\Users\\zack\\.kube\\config Verify that the nodes are ready: $ kubectl.exe get node NAME STATUS ROLES AGE VERSION ip-172-31-37-57.ap-southeast-2.compute.internal Ready \u0026lt;none\u0026gt; 2m40s v1.31.0-eks-a737599 Deploy the Cluster Autoscaler from the official Kubernetes Autoscaler GitHub repository, The Cluster Autoscaler automatically adjusts the number of nodes in eks cluster based on the resource requirements of the workloads. $ kubectl.exe apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/refs/heads/master/cluster-autoscaler/cloudprovider/aws/examples/cluster-autoscaler-one-asg.yaml serviceaccount/cluster-autoscaler created clusterrole.rbac.authorization.k8s.io/cluster-autoscaler created role.rbac.authorization.k8s.io/cluster-autoscaler created clusterrolebinding.rbac.authorization.k8s.io/cluster-autoscaler created rolebinding.rbac.authorization.k8s.io/cluster-autoscaler created deployment.apps/cluster-autoscaler created Modify the Autoscaler Parameters to tuning Scale-Up and Down Behavior. Customized Autoscaler parameters to add arguments to control the scaling behavior, such as the minimum and maximum node counts, the time between scale-down events, stabilization window and cooldown period.\n$ kubectl -n kube-system edit deployment.apps/cluster-autoscaler deployment.apps/cluster-autoscaler edited labels: app: cluster-autoscaler spec: containers: - command: - ./cluster-autoscaler - --cluster-name=module-eks-cluster - --v=4 - --stderrthreshold=info - --cloud-provider=aws - --skip-nodes-with-local-storage=false - --balance-similar-node-groups - --skip-nodes-with-system-pods=false - --scale-down-unneeded-time=1m - --scale-down-delay-after-add=1m - --nodes=1:3:eks-module-eks-cluster-node-group-5ec93604-4bdc-a740-1fcc-707afc8431b HPA testing based on Pod CPU utilization metric\nNow, let\u0026rsquo;s deploy zackweb and set up an Horizontal Pod Autoscaler (HPA) to scale the number of pods based on CPU utilization. $ cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: zackblog spec: replicas: 1 # Start with 1 replica selector: matchLabels: app: zackblog template: metadata: labels: app: zackblog spec: containers: - name: zackblog image: zackz001/gitops-jekyll:latest resources: requests: cpu: \u0026#34;500m\u0026#34; # 0.5 vCPU memory: \u0026#34;512Mi\u0026#34; # 0.5 GiB limits: cpu: \u0026#34;1\u0026#34; # 1 vCPU memory: \u0026#34;1Gi\u0026#34; # 1 GiB --- apiVersion: v1 kind: Service metadata: name: zackblog spec: selector: app: zackblog # This must match the labels in the Deployment ports: - protocol: TCP port: 80 # Port that the service will expose targetPort: 80 # Port that the container listens on type: LoadBalancer kubectl.exe apply -f deployment.yaml deployment.apps/zackblog created service/zackblog created Set Resource Requests and Limits, and Configure Horizontal Pod Autoscaler (HPA) to automatically scale the deployment based on CPU usage $ kubectl set resources deployment zackblog --limits=cpu=200m,memory=200Mi --requests=cpu=100m,memory=100Mi deployment.apps/zackblog resource requirements updated kubectl autoscale deployment zackblog --cpu-percent=50 --min=1 --max=3 $ kubectl get hpa zackblog NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE zackblog Deployment/zackblog cpu: \u0026lt;unknown\u0026gt;/50% 1 3 1 18s Generate CPU Load to Test the HPA To test the HPA, we need to generate CPU load by running a busybox container to repeatedly request the zackblog service.\nkubectl run -i --tty load-generator --image=busybox /bin/sh # Inside the busybox shell, run this: while true; do wget -q -O- http://zackblog \u0026gt; /dev/null; sleep 0.5; done Monitor the HPA and Scaling Events, As CPU utilization increases, the HPA will automatically scale the number of pods: $ kubectl.exe get po NAME READY STATUS RESTARTS AGE load-generator 1/1 Running 0 2m6s zackblog-95f746486-wlmgx 1/1 Running 0 6m20s kubectl get hpa zackblog -w NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE zackblog Deployment/zackblog cpu: 0%/50% 1 10 1 12m zackblog Deployment/zackblog cpu: 92%/50% 1 10 2 5m zackblog Deployment/zackblog cpu: 52%/50% 1 10 3 2m kubectl get hpa zackblog -w zackblog Deployment/zackblog cpu: 37%/50% 1 10 3 27m zackblog Deployment/zackblog cpu: 38%/50% 1 10 3 27m zackblog Deployment/zackblog cpu: 6%/50% 1 10 3 27m zackblog Deployment/zackblog cpu: 0%/50% 1 10 3 27m zackblog Deployment/zackblog cpu: 0%/50% 1 10 1 28m It can be seen that pods got scaled up to 3 when CPU utilization reached 92% and scaled down to 1 when CPU utilization dropped below 50%.\nTesting EKS Cluster Autoscaler by changing deployment replicas:\nEKS Cluster Autoscaler comes with scale-in and scale-out policies, which define when nodes should be added or removed. The Cluster Autoscaler adds nodes when there aren\u0026rsquo;t enough resources to schedule pending pods and removes nodes when they are underutilized.\nChange the resource limitation for zackblog deployment for EKS node autoscaler testing, Set zackweb deployment with requests cpu 500m. As the EKS node group with a t3.small instance type (2c2g), which means one node can only handle one pod, so to make the Autoscaler testing easier to achieve. $ vim deployment.yaml resources: requests: cpu: \u0026#34;500m\u0026#34; # 0.5 vCPU memory: \u0026#34;512Mi\u0026#34; # 0.5 GiB limits: cpu: \u0026#34;1\u0026#34; # 1 vCPU memory: \u0026#34;1Gi\u0026#34; # 1 GiB Gracefully increase the number of deployment replicas to 2, to test EKS node scale up $ kubectl scale deployment zackblog --replicas=2 deployment.apps/zackblog scaled $ kubectl.exe get po NAME READY STATUS RESTARTS AGE load-generator 1/1 Running 0 7m39s zackblog-7f67584fbd-47jvt 1/1 Running 0 31s zackblog-7f67584fbd-9gtfn 0/1 Pending 0 6s $ kubectl.exe describe po zackblog-7f67584fbd-9gtfn Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 18s default-scheduler 0/1 nodes are available: 1 Insufficient memory. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod. Normal TriggeredScaleUp 9s cluster-autoscaler pod triggered scale-up: [{eks-module-eks-cluster-node-group-5ec93604-4bdc-a740-1fcc-707afc8431b3 1-\u0026gt;2 (max: 3)}] $ kubectl.exe get node NAME STATUS ROLES AGE VERSION ip-172-31-15-152.ap-southeast-2.compute.internal Ready \u0026lt;none\u0026gt; 31s v1.31.0-eks-a737599 ip-172-31-37-57.ap-southeast-2.compute.internal Ready \u0026lt;none\u0026gt; 37m v1.31.0-eks-a737599 $ kubectl.exe get po NAME READY STATUS RESTARTS AGE load-generator 1/1 Running 0 9m14s zackblog-7f67584fbd-47jvt 1/1 Running 0 2m6s zackblog-7f67584fbd-9gtfn 1/1 Running 0 101s It can be seen that pod zackblog-7f67584fbd-9gtfn was in pending state due to waiting for node to be scale-up, once we have 2 nodes in EKS cluster, that pod can be scheduled and run.\nIncrease the number of deployment replicas to 3, to trigger EKS node scale up again $ kubectl scale deployment zackblog --replicas=3 deployment.apps/zackblog scaled $ kubectl.exe get po NAME READY STATUS RESTARTS AGE load-generator 1/1 Running 0 9m44s zackblog-7f67584fbd-47jvt 1/1 Running 0 2m36s zackblog-7f67584fbd-9gtfn 1/1 Running 0 2m11s zackblog-7f67584fbd-f5s9d 1/1 Running 0 3s $ kubectl.exe get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES zackblog-7f67584fbd-47jvt 1/1 Running 0 5m58s 172.31.37.227 ip-172-31-37-57.ap-southeast-2.compute.internal \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; zackblog-7f67584fbd-9gtfn 1/1 Running 0 5m33s 172.31.5.139 ip-172-31-15-152.ap-southeast-2.compute.internal \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; zackblog-7f67584fbd-vzcx4 1/1 Running 0 2m3s 172.31.18.111 ip-172-31-20-24.ap-southeast-2.compute.internal \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; Now EKS scale up to 3 nodes to handle the replica increase again.\nChange the number of deployment replicas down to 1, to trigger EKS node scale down, and monitor by autoscaler log file to see the scale down behavior: $ kubectl scale deployment zackblog --replicas=1 $ kubectl -n kube-system logs -f deployment/cluster-autoscaler I1008 12:53:22.686377 1 static_autoscaler.go:598] Starting scale down I1008 12:53:22.686430 1 nodes.go:123] ip-172-31-15-152.ap-southeast-2.compute.internal was unneeded for 40.253885622s I1008 12:53:22.686452 1 nodes.go:123] ip-172-31-37-57.ap-southeast-2.compute.internal was unneeded for 1m0.362059191s I1008 12:53:22.686502 1 cluster.go:153] ip-172-31-37-57.ap-southeast-2.compute.internal for removal I1008 12:53:22.686644 1 hinting_simulator.go:77] Pod kube-system/cluster-autoscaler-5767f77d77-xgfjq can be moved to ip-172-31-20-24.ap-southeast-2.compute.internal I1008 12:53:22.686717 1 hinting_simulator.go:77] Pod kube-system/coredns-7575495454-9n6kd can be moved to ip-172-31-15-152.ap-southeast-2.compute.internal I1008 12:53:22.686832 1 hinting_simulator.go:77] Pod kube-system/coredns-7575495454-dxzdc can be moved to ip-172-31-15-152.ap-southeast-2.compute.internal I1008 12:53:22.686875 1 cluster.go:176] node ip-172-31-37-57.ap-southeast-2.compute.internal may be removed I1008 12:53:22.705270 1 delete.go:103] Successfully added ToBeDeletedTaint on node ip-172-31-37-57.ap-southeast-2.compute.internal I1008 12:53:22.705367 1 actuator.go:212] Scale-down: removing node ip-172-31-37-57.ap-southeast-2.compute.internal, utilization: {0.23316062176165803 0.49755477462428627 0 memory 0.49755477462428627}, pods to reschedule: cluster-autoscaler-5767f77d77-xgfjq,coredns-7575495454-9n6kd,coredns-7575495454-dxzdc $ kubectl.exe get node NAME STATUS ROLES AGE VERSION ip-172-31-15-152.ap-southeast-2.compute.internal NotReady,SchedulingDisabled \u0026lt;none\u0026gt; 9m10s v1.31.0-eks-a737599 ip-172-31-20-24.ap-southeast-2.compute.internal Ready \u0026lt;none\u0026gt; 6m59s v1.31.0-eks-a737599 ip-172-31-37-57.ap-southeast-2.compute.internal Ready,SchedulingDisabled \u0026lt;none\u0026gt; 45m v1.31.0-eks-a737599 $ kubectl.exe get node NAME STATUS ROLES AGE VERSION ip-172-31-20-24.ap-southeast-2.compute.internal Ready \u0026lt;none\u0026gt; 7m58s v1.31.0-eks-a737599 It can be seen that Cluster Autoscaler identified nodes that were underutilized or idle and marked them as \u0026ldquo;unneeded.\u0026rdquo; It simulated moving the existing pods to other nodes. Once it determined that the pods could be rescheduled, it marked the node for deletion (using a ToBeDeletedTaint), preventing new workloads from being scheduled. Finally, the node was removed from the cluster, and the pods were successfully rescheduled on other nodes.\nThis behavior ensures that the cluster\u0026rsquo;s resources are used efficiently, scaling down when there is no workload, thereby reducing costs.\nConclusion:\nBy following these steps, we can effectively manage the scaling of k8s applications and nodes in an EKS cluster using both the Cluster Autoscaler and Horizontal Pod Autoscaler (HPA). These tools ensure that the infrastructure adapts to varying workloads, optimizing resource utilization and costs.\n","permalink":"https://zackblog.work/posts/eks-cluster-autoscaler-and-horizontal-pod-autoscaler-hpa/","summary":"\u003cp\u003e\u0026ldquo;Explore how pods and EC2 nodes can be scaled in EKS cluster \u0026quot;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAutoscaler and Horizontal Pod Autoscaler (HPA) Practice on EKS\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIn last post, I was able to deploy EKS cluster via Jenkins and Terraform:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"/posts/jenkins-multi-destination-cd-pipeline-with-terraform/\"\u003eJenkins - Multi-Destnation Continuse Deployment with Terraform\u003c/a\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThis post walks through the process of setting up an Autoscaler and Horizontal Pod Autoscaler (HPA) in an Amazon EKS cluster. We will explore how to dynamically scale the number of nodes in EKS cluster and how to autoscale K8S pods based on CPU utilization using HPA.\u003c/p\u003e","title":"EKS - Cluster Autoscaler and Horizontal Pod Autoscaler (HPA)"},{"content":"\u0026ldquo;Design a Jenkins pipeline can handle multiple destination: EC2, EKS across different environments\u0026rdquo;\nJenkins CD Pipeline Design\nIn last post, I was able to create a Jenkins Universal CI Pipeline to create blog docker image and push to DockerHub:\nJenkins - Universal CI Pipeline with Ansible \u0026amp; Terraform\nNow it is time to design the continuous deployment pipeline with Ansible and Terraform for infrastructure provision and application configuration and deployment.\nContinuous Deployment Consideration\nContinuous deployment will be more Terraform focused. Starting Jenkins CD pipeline with single EC2 instance deployment for Blog website. The pipeline can be reusable for multiple destinations in later design (EC2, ECS, EKS). At the moment, this EC2 deployment can be achieved via below folder structure:\nJenkins CD pipeline with multi-stage Terraform to provision AWS EC2 Ansible to configure docker and deploy blog Validate Web Blog Access Delete Terraform Resources # Tree terraform-ec2# tree . ├── Jenkinsfile # the CD pipeline file ├── deploy-docker-playbook.yml # the Ansible playbook for ec2 webblog deployment ├── hosts # the Ansible inventory file ├── main.tf # the terraform file to provision AWS EC2 ├── test-playbook.yaml # the playbook for Ansible testing and validation └── variables.tf the terraform var file to provision AWS EC2 The CD pipeline design\nThis Jenkins CD (Continuous Deployment) pipeline covers the following task:\nJenkins Cred and Environment Setup Check Installed Package Versions (AWSCli, Ansible, Terraform) Validate Ansible and AWS credential Run Terraform Initialization and Apply Validate EC2 Readiness and then Deploy Docker Using Ansible Validate Web Blog Access by extracting EC2 public IP Delete Terraform Resources Jenkinsfile\n# Jenkinsfile pipeline { agent any environment { IMAGE_NAME = \u0026#34;zackz001/jenkins\u0026#34; IMAGE_TAG = \u0026#34;${env.BUILD_NUMBER}\u0026#34; LATEST_TAG = \u0026#34;latest\u0026#34; EMAIL_RECIPIENT = \u0026#34;zhbsoftboy1@gmail.com\u0026#34; GIT_REPO_URL = \u0026#39;https://github.com/ZackZhouHB/zack-gitops-project.git\u0026#39; // Git repository URL GIT_BRANCH = \u0026#39;jenkins-cd\u0026#39; // Git branch DOCKERHUB_CREDENTIALS_ID = \u0026#39;dockerhub\u0026#39; // Docker Hub credentials REGION = \u0026#39;ap-southeast-2\u0026#39; // AWS region } stages { stage(\u0026#39;Clean Workspace\u0026#39;) { steps { cleanWs() } } stage(\u0026#39;Checkout Code\u0026#39;) { steps { git branch: \u0026#34;${GIT_BRANCH}\u0026#34;, credentialsId: \u0026#39;gittoken\u0026#39;, url: \u0026#34;${GIT_REPO_URL}\u0026#34; } } stage(\u0026#39;Check Installed Package Versions\u0026#39;) { steps { script { try { // Check Docker version sh \u0026#39;\u0026#39;\u0026#39; if command -v docker \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then echo \u0026#34;Docker Version: $(docker --version)\u0026#34; else echo \u0026#34;Docker is not installed\u0026#34; exit 1 fi \u0026#39;\u0026#39;\u0026#39; } catch (Exception e) { echo \u0026#34;Error: Docker not found. ${e.message}\u0026#34; } try { // Check Terraform version sh \u0026#39;\u0026#39;\u0026#39; if command -v terraform \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then echo \u0026#34;Terraform Version: $(terraform -version)\u0026#34; else echo \u0026#34;Terraform is not installed\u0026#34; exit 1 fi \u0026#39;\u0026#39;\u0026#39; } catch (Exception e) { echo \u0026#34;Error: Terraform not found. ${e.message}\u0026#34; } try { // Check Kubectl version sh \u0026#39;\u0026#39;\u0026#39; if command -v kubectl \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then echo \u0026#34;Kubectl Version: $(kubectl version --client)\u0026#34; else echo \u0026#34;Kubectl is not installed\u0026#34; exit 1 fi \u0026#39;\u0026#39;\u0026#39; } catch (Exception e) { echo \u0026#34;Error: Kubectl not found. ${e.message}\u0026#34; } try { // Check Trivy version sh \u0026#39;\u0026#39;\u0026#39; if command -v trivy \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then echo \u0026#34;Trivy Version: $(trivy --version)\u0026#34; else echo \u0026#34;Trivy is not installed\u0026#34; exit 1 fi \u0026#39;\u0026#39;\u0026#39; } catch (Exception e) { echo \u0026#34;Error: Trivy not found. ${e.message}\u0026#34; } try { // Check Ansible version sh \u0026#39;\u0026#39;\u0026#39; if command -v ansible \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then echo \u0026#34;Ansible Version: $(ansible --version)\u0026#34; else echo \u0026#34;Ansible is not installed\u0026#34; exit 1 fi \u0026#39;\u0026#39;\u0026#39; } catch (Exception e) { echo \u0026#34;Error: Ansible not found. ${e.message}\u0026#34; } try { // Check AWS CLI version sh \u0026#39;\u0026#39;\u0026#39; if command -v aws \u0026gt;/dev/null 2\u0026gt;\u0026amp;1; then echo \u0026#34;AWS CLI Version: $(aws --version)\u0026#34; else echo \u0026#34;AWS CLI is not installed\u0026#34; exit 1 fi \u0026#39;\u0026#39;\u0026#39; } catch (Exception e) { echo \u0026#34;Error: AWS CLI not found. ${e.message}\u0026#34; } } } } stage(\u0026#39;Run a testing Ansible Playbook\u0026#39;) { steps { script { // Run the Ansible playbook using the hosts file from the repo sh \u0026#39;\u0026#39;\u0026#39; echo \u0026#34;Running Ansible playbook:\u0026#34; ansible-playbook -i \u0026#34;${WORKSPACE}/jenkins/terraform-ec2/hosts\u0026#34; \u0026#34;${WORKSPACE}/jenkins/terraform-ec2/test-playbook.yaml\u0026#34; \u0026#39;\u0026#39;\u0026#39; } } } stage(\u0026#39;Verify AWS credential\u0026#39;) { steps { withAWS(credentials: \u0026#39;aws\u0026#39;, region: \u0026#39;ap-southeast-2\u0026#39;) { // Replace with correct AWS credentials ID script { // List all existing S3 buckets and output the result to the Jenkins console sh \u0026#39;\u0026#39;\u0026#39; echo \u0026#34;Listing all S3 buckets:\u0026#34; aws s3 ls \u0026#39;\u0026#39;\u0026#39; } } } } stage(\u0026#39;Terraform Init and Apply\u0026#39;) { steps { withCredentials([[$class: \u0026#39;AmazonWebServicesCredentialsBinding\u0026#39;, credentialsId: \u0026#39;aws\u0026#39;]]) { sh \u0026#39;\u0026#39;\u0026#39; export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} cd jenkins/terraform-ec2 # Check if Terraform has been initialized if [ ! -d \u0026#34;.terraform\u0026#34; ]; then echo \u0026#34;Terraform not initialized. Running \u0026#39;terraform init\u0026#39;...\u0026#34; terraform init else echo \u0026#34;Terraform already initialized. Skipping \u0026#39;terraform init\u0026#39;.\u0026#34; fi terraform apply -auto-approve -var \u0026#34;aws_region=${REGION}\u0026#34; \u0026#39;\u0026#39;\u0026#39; } } } \u0026lt;!-- Stage to extract EC2 public IP --\u0026gt; stage(\u0026#39;Extract EC2 Public IP\u0026#39;) { steps { withCredentials([[$class: \u0026#39;AmazonWebServicesCredentialsBinding\u0026#39;, credentialsId: \u0026#39;aws\u0026#39;]]) { script { def ec2Ip = sh(script: \u0026#39;\u0026#39;\u0026#39; cd jenkins/terraform-ec2 terraform output -raw ec2_public_ip \u0026#39;\u0026#39;\u0026#39;, returnStdout: true).trim() echo \u0026#34;EC2 Public IP: ${ec2Ip}\u0026#34; // Set the environment variable for the next stages explicitly env.EC2_PUBLIC_IP = ec2Ip } } } } \u0026lt;!-- **Fix: Adding a small sleep to ensure env is populated** --\u0026gt; stage(\u0026#39;Validate EC2 Public IP\u0026#39;) { steps { script { sleep 2 // Ensure enough time for variable propagation if (env.EC2_PUBLIC_IP == null || env.EC2_PUBLIC_IP == \u0026#34;\u0026#34;) { error \u0026#34;EC2 Public IP is not available or failed to fetch.\u0026#34; } else { echo \u0026#34;EC2 Public IP is successfully fetched: ${env.EC2_PUBLIC_IP}\u0026#34; } } } } \u0026lt;!-- Wait for EC2 Readiness (SSH Validation) --\u0026gt; stage(\u0026#39;Wait for EC2 Readiness\u0026#39;) { steps { retry(20) { // Retry up to 4 times in case EC2 is not immediately ready sleep 2 // Wait for a bit before checking readiness withCredentials([sshUserPrivateKey(credentialsId: \u0026#39;sshkey\u0026#39;, keyFileVariable: \u0026#39;SSH_KEY\u0026#39;)]) { script { sh \u0026#34;ssh -o StrictHostKeyChecking=no -i ${SSH_KEY} ubuntu@${env.EC2_PUBLIC_IP} \u0026#39;echo EC2 is ready for deployment\u0026#39;\u0026#34; } } } } } \u0026lt;!-- Deploy Docker using Ansible --\u0026gt; stage(\u0026#39;Deploy Docker with Ansible\u0026#39;) { steps { withCredentials([sshUserPrivateKey(credentialsId: \u0026#39;sshkey\u0026#39;, keyFileVariable: \u0026#39;SSH_KEY\u0026#39;)]) { script { sh \u0026#39;\u0026#39;\u0026#39; echo \u0026#34;Running Ansible Playbook for Docker Deployment...\u0026#34; ansible-playbook -i \u0026#34;${EC2_PUBLIC_IP},\u0026#34; \u0026#34;${WORKSPACE}/jenkins/terraform-ec2/deploy-docker-playbook.yml\u0026#34; \\ --user ubuntu \\ --private-key ${SSH_KEY} \\ --extra-vars \u0026#34;ansible_ssh_private_key_file=${SSH_KEY} ec2_ip=${EC2_PUBLIC_IP}\u0026#34; \u0026#39;\u0026#39;\u0026#39; } } } } \u0026lt;!-- New stage: Validate web blog accessibility --\u0026gt; stage(\u0026#39;Validate Web Blog Access\u0026#39;) { steps { script { echo \u0026#34;Validating web blog access via http://${env.EC2_PUBLIC_IP}...\u0026#34; // Use curl to validate HTTP response from the web blog def response = sh(script: \u0026#34;curl -o /dev/null -s -w \u0026#39;%{http_code}\u0026#39; http://${env.EC2_PUBLIC_IP}\u0026#34;, returnStdout: true).trim() if (response == \u0026#39;200\u0026#39;) { echo \u0026#34;Web blog is accessible and returned HTTP status code 200.\u0026#34; } else { error \u0026#34;Web blog is not accessible. HTTP status code: ${response}\u0026#34; } } } } stage(\u0026#39;delete terraform resource\u0026#39;) { steps { withCredentials([[$class: \u0026#39;AmazonWebServicesCredentialsBinding\u0026#39;, credentialsId: \u0026#39;aws\u0026#39;]]) { sh \u0026#39;\u0026#39;\u0026#39; export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} export AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} cd jenkins/terraform-ec2 # Check if Terraform has been initialized if [ ! -d \u0026#34;.terraform\u0026#34; ]; then echo \u0026#34;Terraform not initialized. Running \u0026#39;terraform init\u0026#39;...\u0026#34; terraform init else echo \u0026#34;Terraform already initialized. Skipping \u0026#39;terraform init\u0026#39;.\u0026#34; fi terraform destroy -auto-approve -var \u0026#34;aws_region=${REGION}\u0026#34; \u0026#39;\u0026#39;\u0026#39; } } } } post { success { echo \u0026#34;Pipeline completed successfully.\u0026#34; } failure { echo \u0026#34;Pipeline failed.\u0026#34; } } } Terraform main.tf\n# Terraform main.tf provider \u0026#34;aws\u0026#34; { region = \u0026#34;ap-southeast-2\u0026#34; } terraform { backend \u0026#34;s3\u0026#34; { bucket = \u0026#34;zz-lambda-tag\u0026#34; key = \u0026#34;terraform/state/terraform.tfstate\u0026#34; region = \u0026#34;ap-southeast-2\u0026#34; encrypt = true } } # Security Group Data data \u0026#34;aws_security_group\u0026#34; \u0026#34;existing_sg\u0026#34; { filter { name = \u0026#34;group-name\u0026#34; values = [\u0026#34;launch-wizard-1\u0026#34;] } } # Key Pair Data data \u0026#34;aws_key_pair\u0026#34; \u0026#34;existing_key\u0026#34; { key_name = \u0026#34;zzzzzzzzzzzz\u0026#34; } # EC2 Instance Definition resource \u0026#34;aws_instance\u0026#34; \u0026#34;web\u0026#34; { ami = \u0026#34;ami-040e71e7b8391cae4\u0026#34; # Choose AMI instance_type = \u0026#34;t2.micro\u0026#34; key_name = data.aws_key_pair.existing_key.key_name security_groups = [ data.aws_security_group.existing_sg.name ] tags = { Name = \u0026#34;Jenkins-EC2\u0026#34; } } # Output EC2 Public IP output \u0026#34;ec2_public_ip\u0026#34; { value = aws_instance.web.public_ip } Ansible Playbook deploy-docker-playbook.yml\n# Ansible Playbook for EC2 web blog deployment --- - hosts: all become: yes tasks: - name: Check if Docker is already installed command: docker --version register: docker_installed ignore_errors: yes changed_when: false - name: Install required packages (if Docker is not installed) apt: name: - apt-transport-https - ca-certificates - curl - software-properties-common state: present update_cache: yes when: docker_installed.rc != 0 - name: Add Docker\u0026#39;s official GPG key (if Docker is not installed) apt_key: url: https://download.docker.com/linux/ubuntu/gpg state: present when: docker_installed.rc != 0 - name: Add Docker\u0026#39;s official APT repository (if Docker is not installed) apt_repository: repo: deb [arch=amd64] https://download.docker.com/linux/ubuntu {{ ansible_distribution_release }} stable state: present when: docker_installed.rc != 0 - name: Update APT cache (if Docker is not installed) apt: update_cache: yes when: docker_installed.rc != 0 - name: Install Docker CE (if Docker is not installed) apt: name: docker-ce state: present update_cache: yes when: docker_installed.rc != 0 - name: Start and enable Docker service systemd: name: docker enabled: yes state: started - name: Stop all running containers shell: docker stop $(docker ps -q) ignore_errors: true register: stopped_containers - name: Remove all stopped containers shell: docker rm $(docker ps -a -q) when: stopped_containers.rc == 0 ignore_errors: true - name: Pull Docker image docker_image: name: zackz001/gitops-jekyll tag: latest source: pull - name: Run Docker container docker_container: name: zackblog image: zackz001/gitops-jekyll:latest state: started restart_policy: unless-stopped published_ports: - \u0026#34;80:80\u0026#34; Pipeline debug and testing\nAfter thorough testing and validation, the CD pipeline also works like a charm.\nTerraform Modularization for Multi-Destination Deployment\nThe folder structure below is designed to organize Infrastructure as Code (IaC) using Terraform, breaking down the configuration into reusable modules for ECS, EKS, and EC2 deployments, along with different environments (production, stage, etc.). So The Jenkins reusable CD pipeline can manage multi-destination deployments based on this structure.\nSingle EC2 deployment Single ECS deployment Terraform ECS Module with multi-environment deployment (production and stage) Terraform single EKS deployment Terraform EKS Module with multi-environment deployment (production and stage) root@zackz:~/zack-gitops-project/jenkins# tree ├── terraform-ec2 │ ├── Jenkinsfile │ ├── deploy-docker-playbook.yml │ ├── hosts │ ├── main.tf │ ├── test-playbook.yaml │ └── variables.tf ├── module-ecs-cluster │ ├── Jenkinsfile │ ├── main.tf │ ├── modules │ │ ├── alb │ │ │ ├── main.tf │ │ │ ├── outputs.tf │ │ │ └── variables.tf │ │ ├── ecs_cluster │ │ │ ├── main.tf │ │ │ ├── outputs.tf │ │ │ └── variables.tf │ │ ├── ecs_service │ │ │ ├── main.tf │ │ │ ├── outputs.tf │ │ │ └── variables.tf │ │ ├── iam │ │ │ ├── main.tf │ │ │ ├── outputs.tf │ │ │ └── variables.tf │ │ ├── security_groups │ │ │ ├── main.tf │ │ │ ├── outputs.tf │ │ │ └── variables.tf │ │ └── task_definition │ │ ├── main.tf │ │ ├── outputs.tf │ │ └── variables.tf │ ├── outputs.tf │ ├── terraform.tfstate │ ├── terraform.tfstate.backup │ ├── terraform.tfvars │ └── variables.tf ├── module-ecs-env │ ├── environments │ │ ├── production │ │ │ ├── Jenkinsfile │ │ │ ├── backend.tf │ │ │ ├── main.tf │ │ │ ├── outputs.tf │ │ │ ├── terraform.tfvars │ │ │ └── variables.tf │ │ └── stage │ │ ├── Jenkinsfile │ │ ├── backend.tf │ │ ├── main.tf │ │ ├── outputs.tf │ │ ├── terraform.tfvars │ │ └── variables.tf │ └── modules │ ├── alb │ │ ├── main.tf │ │ ├── outputs.tf │ │ └── variables.tf │ ├── ecs_cluster │ │ ├── main.tf │ │ ├── outputs.tf │ │ └── variables.tf │ ├── ecs_service │ │ ├── main.tf │ │ ├── outputs.tf │ │ └── variables.tf │ ├── iam │ │ ├── main.tf │ │ ├── outputs.tf │ │ └── variables.tf │ ├── security_groups │ │ ├── main.tf │ │ ├── outputs.tf │ │ └── variables.tf │ └── task_definition │ ├── main.tf │ ├── outputs.tf │ └── variables.tf └── terraform-eks ├── Jenkinsfile ├── argo-setup.sh ├── backend.tf ├── deployment.yaml ├── iam.tf ├── main.tf ├── output.tf ├── terraform.tfvars └── variables.tf ├── module-eks-env │ ├── environments │ │ ├── prod │ │ │ ├── backend.tf │ │ │ ├── main.tf │ │ │ ├── output.tf │ │ │ ├── provider.tf │ │ │ └── variables.tf │ │ └── stage │ │ ├── backend.tf │ │ ├── main.tf │ │ ├── output.tf │ │ ├── provider.tf │ │ └── variables.tf │ └── modules │ └── eks │ ├── main.tf │ ├── output.tf │ └── variables.tf Conclusion\nUsing Terraform and Jenkins practices enables efficient management of complex, multi-environment, and multi-service deployments, which are crucial for cloud-native CI/CD processes to achieve:\nVersion Control: Allows tracking and managing infrastructure changes across different environments. Multi-Destination Deployment with Jenkins: Enables dynamic deployments to different environments (e.g., production, stage) by passing environment-specific parameters in the pipeline. Environment Separation: Each environment (production, stage) has its own Terraform configuration, ensuring proper isolation and customization. Terraform Modules Reusability: Reusable modules for infrastructure components (ECS, EKS, etc.) reduce code duplication and simplify updates. Multi-Environment and Multi-Component Deployment: Jenkins pipelines can deploy multiple services and environments concurrently by leveraging modular infrastructure and dynamic inputs. Jenkins Recap Summary\nOver this recap for Jenkins, I believe I had achieved:\nMulti-Stage, Multi-Environment Pipelines: These handle conditional execution using when blocks, try-catch for error handling, and post sections for notifications and cleanup. Integration with IaC Tools: Seamlessly provisions AWS resources using Terraform or CloudFormation within the pipeline. Security and Compliance: Integrates tools like Snyk, Trivy, and SonarQube to perform vulnerability scanning and code quality checks during the build. Secret Management: Securely manages sensitive data using AWS Secrets Manager, HashiCorp Vault, or Jenkins credentials plugin. Real-World Automation: Solves complex problems and improves efficiency, reducing build times and increasing deployment reliability. ","permalink":"https://zackblog.work/posts/jenkins-multi-destination-cd-pipeline-with-terraform/","summary":"\u003cp\u003e\u0026ldquo;Design a Jenkins pipeline can handle multiple destination: EC2, EKS across different environments\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eJenkins CD Pipeline Design\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIn last post, I was able to create a Jenkins Universal CI Pipeline to create blog docker image and push to DockerHub:\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"/posts/jenkins-universal-ci-pipeline-with-ansible-terraform/\"\u003eJenkins - Universal CI Pipeline with Ansible \u0026amp; Terraform\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003eNow it is time to design the continuous deployment pipeline with Ansible and Terraform for infrastructure provision and application configuration and deployment.\u003c/p\u003e","title":"Jenkins - Multi-Destination CD pipeline with Terraform"},{"content":"\u0026ldquo;Such a long time I nearly forgot how to use Jenkins..\u0026rdquo;\nJenkins recap\nIt has been some time since I adapted CI/CD pipelines from Jenkins to AWS CodePipeline and GitHub Actions workflows. Now, it\u0026rsquo;s time to recap and improve some of my previous Jenkins practices.\nUniversal Jenkins Docker image design This time, instead of installing Jenkins on a server, I prefer to containerize a universal Jenkins Docker image with the necessary packages installed, so it provides consistency and reproducibility, portability, and easy updates and rollbacks.\nDocker CLI Terraform Kubectl Trivy AWS CLI Ansible # Dockerfile FROM jenkins/jenkins:lts USER root # Install necessary packages, Docker CLI, Terraform, Kubectl, Trivy, AWS CLI, and Ansible RUN apt-get update \u0026amp;\u0026amp; \\ apt-get install -y \\ curl \\ wget \\ unzip \\ gnupg2 \\ apt-transport-https \\ lsb-release \\ ca-certificates \\ software-properties-common \\ python3 \\ python3-venv \\ python3-pip \u0026amp;\u0026amp; \\ # Install Docker CLI curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - \u0026amp;\u0026amp; \\ echo \u0026#34;deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable\u0026#34; \u0026gt; /etc/apt/sources.list.d/docker.list \u0026amp;\u0026amp; \\ apt-get update \u0026amp;\u0026amp; \\ apt-get install -y docker-ce-cli \u0026amp;\u0026amp; \\ # Install Terraform wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor \u0026gt; /usr/share/keyrings/hashicorp-archive-keyring.gpg \u0026amp;\u0026amp; \\ echo \u0026#34;deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main\u0026#34; \u0026gt; /etc/apt/sources.list.d/hashicorp.list \u0026amp;\u0026amp; \\ apt-get update \u0026amp;\u0026amp; \\ apt-get install -y terraform \u0026amp;\u0026amp; \\ # Install Kubectl curl -LO \u0026#34;https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl\u0026#34; \u0026amp;\u0026amp; \\ chmod +x kubectl \u0026amp;\u0026amp; \\ mv kubectl /usr/local/bin/ \u0026amp;\u0026amp; \\ # Install Trivy wget https://github.com/aquasecurity/trivy/releases/download/v0.56.0/trivy_0.56.0_Linux-64bit.deb \u0026amp;\u0026amp; \\ dpkg -i trivy_0.56.0_Linux-64bit.deb \u0026amp;\u0026amp; \\ rm trivy_0.56.0_Linux-64bit.deb \u0026amp;\u0026amp; \\ # Install AWS CLI curl \u0026#34;https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip\u0026#34; -o \u0026#34;awscliv2.zip\u0026#34; \u0026amp;\u0026amp; \\ unzip awscliv2.zip \u0026amp;\u0026amp; \\ ./aws/install \u0026amp;\u0026amp; \\ rm -rf awscliv2.zip aws/ \u0026amp;\u0026amp; \\ # Create a virtual environment for Python and Ansible python3 -m venv /opt/ansible_venv \u0026amp;\u0026amp; \\ /opt/ansible_venv/bin/pip install --upgrade pip \u0026amp;\u0026amp; \\ /opt/ansible_venv/bin/pip install ansible \u0026amp;\u0026amp; \\ # Create symlinks to make Ansible easily accessible ln -s /opt/ansible_venv/bin/ansible /usr/local/bin/ansible \u0026amp;\u0026amp; \\ ln -s /opt/ansible_venv/bin/ansible-playbook /usr/local/bin/ansible-playbook \u0026amp;\u0026amp; \\ # Clean up apt-get clean \u0026amp;\u0026amp; \\ rm -rf /var/lib/apt/lists/* USER jenkins Build and run this universal Jenkins image to mount Jenkins home directory and Docker socket from the host to the container, also add Docker group to the container so Jenkins can run Docker commands inside the container without needing root privileges.\ndocker build -t jenkins-all . docker run -d --name jenkins -p 8080:8080 -p 50000:50000 \\ -v /var/run/docker.sock:/var/run/docker.sock \\ -v /var/jenkins_home:/var/jenkins_home \\ --group-add $(getent group docker | cut -d: -f3) \\ jenkins-all Universal Jenkins CI pipeline design After installing a list of plugins and configuring all credentials and the GitHub webhook, the CI pipeline is designed below and can be triggered by a Git push event and will run automatically:\nEnable multi-language artifact build support. Integrate testing of Java versions. Enable security checks for static code analysis and Docker image scanning. Implement advanced Jenkins pipeline structuring using try-catch, if-else, timeouts, environment variables, post actions and error handling. // Define the detectJavaVersion function outside of the pipeline block def detectJavaVersion() { def javaVersionOutput = sh(script: \u0026#39;java -version 2\u0026gt;\u0026amp;1\u0026#39;, returnStatus: false, returnStdout: true).trim() def javaVersionMatch = javaVersionOutput =~ /openjdk version \u0026#34;(\\d+\\.\\d+)/ if (javaVersionMatch) { def javaVersion = javaVersionMatch[0][1] if (javaVersion.startsWith(\u0026#34;1.8\u0026#34;)) { return \u0026#39;8\u0026#39; } else if (javaVersion.startsWith(\u0026#34;11\u0026#34;)) { return \u0026#39;11\u0026#39; } else if (javaVersion.startsWith(\u0026#34;17\u0026#34;)) { return \u0026#39;17\u0026#39; } else { error(\u0026#34;Unsupported Java version detected: ${javaVersion}\u0026#34;) } } else { error(\u0026#34;Java version information not found in output.\u0026#34;) } } pipeline { agent any environment { REGISTRY_URL = \u0026#39;https://index.docker.io/v1/\u0026#39; IMAGE_NAME = \u0026#34;zackz001/jenkins\u0026#34; IMAGE_TAG = \u0026#34;${env.BUILD_NUMBER}\u0026#34; LATEST_TAG = \u0026#34;latest\u0026#34; TRIVY_OUTPUT = \u0026#34;trivy-report.txt\u0026#34; EMAIL_RECIPIENT = \u0026#34;zhbsoftboy1@gmail.com\u0026#34; GIT_REPO_URL = \u0026#39;https://github.com/ZackZhouHB/zack-gitops-project.git\u0026#39; // Git repository URL GIT_BRANCH = \u0026#39;jenkins\u0026#39; // Git branch DOCKERHUB_CREDENTIALS_ID = \u0026#39;dockerhub\u0026#39; // Docker Hub credentials SONAR_TOKEN = \u0026#39;sonar\u0026#39; // Fetch Sonar token securely SNYK_INSTALLATION = \u0026#39;snyk\u0026#39; // Replace with your Snyk installation SNYK_TOKEN = \u0026#39;snyktoken\u0026#39; // Fetch Snyk token securely } stages { stage(\u0026#39;Clean Workspace\u0026#39;) { steps { cleanWs() } } stage(\u0026#39;Checkout Code\u0026#39;) { steps { git branch: \u0026#34;${GIT_BRANCH}\u0026#34;, credentialsId: \u0026#39;gittoken\u0026#39;, url: \u0026#34;${GIT_REPO_URL}\u0026#34; } } stage(\u0026#39;Detect and Set Java\u0026#39;) { steps { script { try { def javaVersion = detectJavaVersion() // Detect the Java version, e.g., \u0026#34;17\u0026#34; def javaToolName = \u0026#34;Java_${javaVersion}\u0026#34; // Expected tool name // Try to set the Java version; fallback if the specific version isn\u0026#39;t found try { tool name: javaToolName, type: \u0026#39;jdk\u0026#39; echo \u0026#34;Using Java version ${javaVersion}.\u0026#34; } catch (Exception toolError) { echo \u0026#34;No JDK named ${javaToolName} found. Using default JDK.\u0026#34; } // Verify Java version, regardless of whether the specific version was found sh \u0026#39;java --version\u0026#39; } catch (Exception e) { echo \u0026#34;Error during Java version detection: ${e.message}\u0026#34; // Continue pipeline even if Java detection fails } } } } \u0026lt;!-- Code Security Analysis and Fixes --\u0026gt; stage(\u0026#39;snyk_analysis\u0026#39;) { steps { script { echo \u0026#39;Running Snyk security analysis...\u0026#39; timeout(time: 5, unit: \u0026#39;MINUTES\u0026#39;) { // Adjust the timeout value as necessary try { snykSecurity( snykInstallation: SNYK_INSTALLATION, snykTokenId: SNYK_TOKEN, failOnIssues: false, monitorProjectOnBuild: true, additionalArguments: \u0026#39;--severity-threshold=low\u0026#39; ) } catch (Exception e) { currentBuild.result = \u0026#39;FAILURE\u0026#39; error(\u0026#34;Error during snyk_analysis: ${e.message}\u0026#34;) } } } } } \u0026lt;!-- Language-specific build and test stages --\u0026gt; stage(\u0026#39;Frontend Build and Test\u0026#39;) { steps { script { try { if (fileExists(\u0026#39;package.json\u0026#39;)) { sh \u0026#39;npm install --force\u0026#39; sh \u0026#39;npm test\u0026#39; } else { echo \u0026#39;No package.json found, skipping Frontend build and test.\u0026#39; } } catch (Exception e) { currentBuild.result = \u0026#39;FAILURE\u0026#39; error(\u0026#34;Error during Frontend build and test: ${e.message}\u0026#34;) } } } } stage(\u0026#39;Java Spring Boot Build and Test\u0026#39;) { steps { script { try { if (fileExists(\u0026#39;pom.xml\u0026#39;)) { sh \u0026#39;mvn clean package\u0026#39; sh \u0026#39;mvn test\u0026#39; } else { echo \u0026#39;No pom.xml found, skipping Java Spring Boot build and test.\u0026#39; } } catch (Exception e) { currentBuild.result = \u0026#39;FAILURE\u0026#39; error(\u0026#34;Error during Java Spring Boot build and test: ${e.message}\u0026#34;) } } } } stage(\u0026#39;.NET Build and Test\u0026#39;) { steps { script { try { if (fileExists(\u0026#39;YourSolution.sln\u0026#39;)) { sh \u0026#39;dotnet build\u0026#39; sh \u0026#39;dotnet test\u0026#39; } else { echo \u0026#39;No YourSolution.sln found, skipping .NET build and test.\u0026#39; } } catch (Exception e) { currentBuild.result = \u0026#39;FAILURE\u0026#39; error(\u0026#34;Error during .NET build and test: ${e.message}\u0026#34;) } } } } stage(\u0026#39;PHP Build and Test\u0026#39;) { steps { script { try { if (fileExists(\u0026#39;composer.json\u0026#39;)) { sh \u0026#39;composer install\u0026#39; sh \u0026#39;phpunit\u0026#39; } else { echo \u0026#39;No composer.json found, skipping PHP build and test.\u0026#39; } } catch (Exception e) { currentBuild.result = \u0026#39;FAILURE\u0026#39; error(\u0026#34;Error during PHP build and test: ${e.message}\u0026#34;) } } } } stage(\u0026#39;iOS Build and Test\u0026#39;) { steps { script { try { if (fileExists(\u0026#39;YourProject.xcodeproj\u0026#39;)) { xcodebuild(buildDir: \u0026#39;build\u0026#39;, scheme: \u0026#39;YourScheme\u0026#39;) } else { echo \u0026#39;No YourProject.xcodeproj found, skipping iOS build and test.\u0026#39; } } catch (Exception e) { currentBuild.result = \u0026#39;FAILURE\u0026#39; error(\u0026#34;Error during iOS build and test: ${e.message}\u0026#34;) } } } } stage(\u0026#39;Android Build and Test\u0026#39;) { steps { script { try { if (fileExists(\u0026#39;build.gradle\u0026#39;)) { sh \u0026#39;./gradlew build\u0026#39; sh \u0026#39;./gradlew test\u0026#39; } else { echo \u0026#39;No build.gradle found, skipping Android build and test.\u0026#39; } } catch (Exception e) { currentBuild.result = \u0026#39;FAILURE\u0026#39; error(\u0026#34;Error during Android build and test: ${e.message}\u0026#34;) } } } } stage(\u0026#39;Ruby on Rails Build and Test\u0026#39;) { steps { script { try { if (fileExists(\u0026#39;Gemfile.lock\u0026#39;)) { sh \u0026#39;bundle install\u0026#39; sh \u0026#39;bundle exec rake db:migrate\u0026#39; sh \u0026#39;bundle exec rails test\u0026#39; } else { echo \u0026#39;No Gemfile.lock found, skipping Ruby on Rails build and test.\u0026#39; } } catch (Exception e) { currentBuild.result = \u0026#39;FAILURE\u0026#39; error(\u0026#34;Error during Ruby on Rails build and test: ${e.message}\u0026#34;) } } } } stage(\u0026#39;Flask Build and Test\u0026#39;) { steps { script { try { if (fileExists(\u0026#39;app.py\u0026#39;)) { sh \u0026#39;pip install -r requirements.txt\u0026#39; sh \u0026#39;python -m unittest discover\u0026#39; } else { echo \u0026#39;No app.py found, skipping Flask build and test.\u0026#39; } } catch (Exception e) { currentBuild.result = \u0026#39;FAILURE\u0026#39; error(\u0026#34;Error during Flask build and test: ${e.message}\u0026#34;) } } } } stage(\u0026#39;Django Build and Test\u0026#39;) { steps { script { try { if (fileExists(\u0026#39;manage.py\u0026#39;)) { sh \u0026#39;pip install -r requirements.txt\u0026#39; sh \u0026#39;python manage.py migrate\u0026#39; sh \u0026#39;python manage.py test\u0026#39; } else { echo \u0026#39;No manage.py found, skipping Django build and test.\u0026#39; } } catch (Exception e) { currentBuild.result = \u0026#39;FAILURE\u0026#39; error(\u0026#34;Error during Django build and test: ${e.message}\u0026#34;) } } } } stage(\u0026#39;Rust Build and Test\u0026#39;) { steps { script { try { if (fileExists(\u0026#39;Cargo.toml\u0026#39;)) { env.RUST_BACKTRACE = \u0026#39;full\u0026#39; sh \u0026#39;cargo build\u0026#39; sh \u0026#39;cargo test\u0026#39; } else { echo \u0026#34;No Cargo.toml file found. Skipping Rust build and test.\u0026#34; } } catch (Exception e) { currentBuild.result = \u0026#39;FAILURE\u0026#39; error(\u0026#34;Error during Rust build and test: ${e.message}\u0026#34;) } } } } stage(\u0026#39;Ruby Sinatra Build and Test\u0026#39;) { steps { script { try { if (fileExists(\u0026#39;app.rb\u0026#39;)) { sh \u0026#39;gem install bundler\u0026#39; sh \u0026#39;bundle install\u0026#39; sh \u0026#39;bundle exec rake test\u0026#39; } else { echo \u0026#34;No app.rb file found. Skipping Ruby Sinatra build and test.\u0026#34; } } catch (Exception e) { currentBuild.result = \u0026#39;FAILURE\u0026#39; error(\u0026#34;Error during Ruby Sinatra build and test: ${e.message}\u0026#34;) } } } } \u0026lt;!-- Build ZackBlog docker image --\u0026gt; stage(\u0026#39;Check and Build Docker Image\u0026#39;) { steps { script { try { // Check if Docker is available sh \u0026#39;docker --version\u0026#39; echo \u0026#34;Docker is installed. Proceeding to build the Docker image...\u0026#34; // Build the Docker image from the \u0026#39;zack_blog\u0026#39; folder dockerImage = docker.build(\u0026#34;${IMAGE_NAME}:${IMAGE_TAG}\u0026#34;, \u0026#34;zack_blog/\u0026#34;) } catch (Exception e) { // Handle the error if Docker is not available error(\u0026#34;Docker is not installed or accessible. Cannot proceed with the build.\u0026#34;) } } } } \u0026lt;!-- Scan docker image with Trivy --\u0026gt; stage(\u0026#39;Docker Image Scan\u0026#39;) { steps { // Use Trivy to scan the built Docker image sh \u0026#34;trivy image --severity HIGH,CRITICAL ${IMAGE_NAME}:${IMAGE_TAG} \u0026gt; ${TRIVY_OUTPUT}\u0026#34; } } \u0026lt;!-- Push to Dockerhub --\u0026gt; stage(\u0026#39;Push Docker Image to DockerHub\u0026#39;) { steps { script { docker.withRegistry(\u0026#34;${REGISTRY_URL}\u0026#34;, \u0026#34;${DOCKERHUB_CREDENTIALS_ID}\u0026#34;) { dockerImage.push(\u0026#34;${IMAGE_TAG}\u0026#34;) dockerImage.push(\u0026#34;${LATEST_TAG}\u0026#34;) // Push \u0026#39;latest\u0026#39; tag } } } } \u0026lt;!-- Output image scan result --\u0026gt; stage(\u0026#39;Display Trivy Scan Results\u0026#39;) { steps { script { // Display the contents of the Trivy report def scanReport = readFile(\u0026#34;${TRIVY_OUTPUT}\u0026#34;) echo \u0026#34;Trivy Scan Report:\\n${scanReport}\u0026#34; } } } \u0026lt;!-- Additional stages like Docker build, image scan, etc. --\u0026gt; } \u0026lt;!-- Post Build Emailing --\u0026gt; post { success { script { def scanReport = readFile(\u0026#34;${TRIVY_OUTPUT}\u0026#34;) emailext( to: \u0026#34;${EMAIL_RECIPIENT}\u0026#34;, subject: \u0026#34;CI Pipeline Success: Build ${IMAGE_TAG}\u0026#34;, body: \u0026#34;\u0026#34;\u0026#34; The pipeline has successfully completed. Docker image ${IMAGE_NAME}:${IMAGE_TAG} has been built and pushed to DockerHub. Trivy Scan Report: ${scanReport} \u0026#34;\u0026#34;\u0026#34; ) } } failure { emailext( to: \u0026#34;${EMAIL_RECIPIENT}\u0026#34;, subject: \u0026#34;CI Pipeline Failed: Build ${IMAGE_TAG}\u0026#34;, body: \u0026#34;\u0026#34;\u0026#34; The pipeline has failed at some stage. Please check the Jenkins console logs for more details. \u0026#34;\u0026#34;\u0026#34; ) } } } Pipeline test and debug\nAfter thorough testing and validation, the CI pipeline finally works like a charm.\nNext, I will create a CD pipeline to integrate with Ansible, AWS, and Terraform to deploy the blog onto AWS EC2, ECS, and EKS.\n","permalink":"https://zackblog.work/posts/jenkins-universal-ci-pipeline-with-ansible-terraform/","summary":"\u003cp\u003e\u0026ldquo;Such a long time I nearly forgot how to use Jenkins..\u0026rdquo;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eJenkins recap\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eIt has been some time since I adapted CI/CD pipelines from Jenkins to AWS CodePipeline and GitHub Actions workflows. Now, it\u0026rsquo;s time to recap and improve some of my previous Jenkins practices.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eUniversal Jenkins Docker image design\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThis time, instead of installing Jenkins on a server, I prefer to containerize a universal Jenkins Docker image with the necessary packages installed, so it provides consistency and reproducibility, portability, and easy updates and rollbacks.\u003c/p\u003e","title":"Jenkins - Universal CI Pipeline with Ansible \u0026 Terraform"},{"content":"When a company faces challenge to manage its Linux environments across local and public cloud, RedHat Identity management can be the solution to achieve:\nWith Local AD and Azure AD (AAD) Integration With AWS SSO Integration as external identity provider LDAP, Kerberos and NTP A web-based management front-end running on Apache A Typical AD User Authentication Flow End-to-End:\nUser Creation and Management:\nAzure AD / Local AD: Users are created in the Azure Active Directory or local Active Directory. Synchronization to RedHat IdM: The users are synchronized from AD to RedHat IdM using the two-way trust established between AD and IdM. Accessing EC2 Instances via SSH:\nUser Sync to RedHat IdM: Users synchronized to RedHat IdM are assigned roles and permissions, including SSH access to specific EC2 instances. Host-Based Access Control (HBAC): RedHat IdM enforces HBAC rules to control which users can access specific EC2 instances. SSH Access Control: When a user attempts to SSH into an EC2 instance, RedHat IdM verifies the user\u0026rsquo;s identity and permissions, allowing or denying access based on the defined HBAC rules. The design:\nFor Idm on AWS, configure the security groups to allow ports required by IdM. IdM desires below to be open:\nHTTP/HTTPS — 80, 443 — TCP LDAP/LDAPS — 389, 636 — TCP Kerberos — 88, 464 — Both TCP and UDP DNS — 53 — Both TCP and UDP NTP — 123 — UDP Here I am going to:\ninstall and configure a local freeIPA server enroll 2 Linux client machines (both CentOS and Ubuntu) Setup a local AD, build a 2 way trust between idm and AD Validate IDM and AD user to ssh into idm client machines. Prerequisites:\nWindows AD Domain ad.zack.world and Idm Domain ipa.zack.world Windows AD: 11.0.1.181 dc01.ad.zack.world (win server 2019) Windows client1: 11.0.1.182 win-client.ad.zack.world (win server 2019) idm Server: 11.0.1.180 server1.ipa.zack.world (CentOS 9) idm Client2: 11.0.1.184 ubt-client02.ipa.zack.world (Ubuntu 24.04) idm Client3: 11.0.1.185 idm-client3-centos7.ipa.zack.world (CentOS 9) FreeIPA Installation\nOn freeIPA Server server1.ipa.zack.world 11.0.1.180 (CentOS 9):\n# set hostname, IP and DNS hostnamectl set-hostname server1.ipa.zack.world # add 3 hosts to /etc/hosts echo 11.0.1.180 server1.ipa.zack.world ipa \u0026gt;\u0026gt; /etc/hosts echo 11.0.1.184 ubt-client02.ipa.zack.world ipa \u0026gt;\u0026gt; /etc/hosts echo 11.0.1.185 idm-client3-centos7.ipa.zack.world ipa \u0026gt;\u0026gt; /etc/hosts echo 11.0.1.181 dc01.ad.zack.world ipa \u0026gt;\u0026gt; /etc/hosts # install ipa-server dnf -y install freeipa-server freeipa-server-dns freeipa-client # Configure ipa-server and DNS, here set ipa console and domain admin password ipa-server-install --setup-dns # confirm or change NetBIOS domain name NetBIOS domain name [IPA]: IPA01 The ipa-server-install command was successful. # Configure firewall rules and services firewall-cmd --add-service={freeipa-ldap,freeipa-ldaps,dns,ntp} firewall-cmd --runtime-to-permanent firewall-cmd --reload # check ipastatus [root@freeipa ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa: INFO: The ipactl command was successful # Obtain a Kerberos ticket for the Kerberos admin user and Verify the ticket kinit admin klist Ticket cache: KEYRING:persistent:0:0 Default principal: admin@ZACKZ.OONLINE Valid starting Expires Service principal 07/13/24 22:17:29 07/14/24 22:02:43 HTTP/server1.ipa.zack.world@IPA.ZACK.WORLD # check content of /etc/resolv.conf cat /etc/resolv.conf search ipa.zack.world nameserver 127.0.0.1 # Configure default login shell to Bash and Create User tina ipa config-mod --defaultshell=/bin/bash ipa user-add tina --first=tina --last=qi --password Idm client Enrollment\nNow the idm web portal should be accessible, by adding \u0026ldquo;11.0.1.180 server1.ipa.zack.world\u0026rdquo; into local \u0026ldquo;c:/windows/system32/drivers/etc/hosts.\nThen enrol both CentOS and Ubuntu IDM client machines\nOn FreeIPA Server, add DNS entry for FreeIPA Client machines\n# ipa dnsrecord-add [domain name] [record name] [record type] [record] ipa dnsrecord-add ipa.zack.world idm-client3-centos7 --a-rec 11.0.1.185 ipa dnsrecord-add ipa.zack.world ubt-client02 --a-rec 11.0.1.184 - set IP, hostname, DNS on idm client # set idm server ID as client DNS nmcli connection modify ens33 ipv4.dns 11.0.1.180 nmcli connection up ens33 # Install FreeIPA Client packages. dnf -y install freeipa-client # enrol client to idm server with domain name ipa-client-install --server=server1.ipa.zack.world --domain ipa.zack.world Enrolled in IPA realm IPA.ZACK.WORLD Configuring ipa.zack.world as NIS domain. Client configuration complete. The ipa-client-install command was successful # set create home directory at initial login authselect enable-feature with-mkhomedir systemctl enable --now oddjobd # same as Ubuntu client echo 11.0.1.180 server1.ipa.zack.world server1 \u0026gt;\u0026gt; /etc/hosts echo 11.0.1.184 ubt-client02.ipa.zack.world ubt-client02 \u0026gt;\u0026gt; /etc/hosts echo 11.0.1.185 idm-client3-centos7.ipa.zack.world idm-client3-centos7 \u0026gt;\u0026gt; /etc/hosts # Edit host file and install client, then enrol into idm server domain apt update \u0026amp;\u0026amp; apt install freeipa-client oddjob-mkhomedir -y ipa-client-install --server=server1.ipa.zack.world --domain ipa.zack.world Client configuration complete. The ipa-client-install command was successful Setup idm and AD trust\nOn Windows DC, setup AD\ninstall ADDC role and feature create forest \u0026ldquo;ad.zack.world\u0026rdquo; promote to primary DC test AD to join Windows client machine to domain create AD user joez@ad.zack.world add idm domain to Windows AD zones # dnscmd 127.0.0.1 /ZoneAdd [FreeIPA domain name] /Secondary [FreeIPA IP address] C:\\Users\\Administrator\u0026gt;dnscmd 127.0.0.1 /ZoneAdd ipa.zack.world /Secondary 11.0.1.180 DNS Server 127.0.0.1 created zone ipa.zack.world: Command completed successfully. # Verify both AD and Idm DNS resolution, then setup trust dig SRV _ldap._tcp.ipa.zack.world dig SRV _ldap._tcp.ad.zack.world Install required packages then setup trust on FreeIPA Server\n# Install packages dnf -y install ipa-server-trust-ad # setup ad trust ipa-adtrust-install # FreeIPA admin password admin password: ============================================================================= Setup complete # add firewall service and ports for ad trust firewall-cmd --add-service=freeipa-trust firewall-cmd --permanent --add-port=135/tcp firewall-cmd --permanent --add-port=138/tcp firewall-cmd --permanent --add-port=139/tcp firewall-cmd --permanent --add-port=445/tcp firewall-cmd --permanent --add-port=1024-1300/tcp firewall-cmd --permanent --add-port=3268/tcp # Open UDP ports firewall-cmd --permanent --add-port=138/udp firewall-cmd --permanent --add-port=139/udp firewall-cmd --permanent --add-port=389/udp firewall-cmd --permanent --add-port=445/udp # Open TCP ports firewall-cmd --permanent --add-port=80/tcp firewall-cmd --permanent --add-port=443/tcp firewall-cmd --permanent --add-port=389/tcp firewall-cmd --permanent --add-port=636/tcp firewall-cmd --permanent --add-port=88/tcp sudo firewall-cmd --permanent --add-port=464/tcp sudo firewall-cmd --permanent --add-port=53/tcp # Open UDP ports firewall-cmd --permanent --add-port=88/udp firewall-cmd --permanent --add-port=464/udp firewall-cmd --permanent --add-port=53/udp firewall-cmd --permanent --add-port=123/udp firewall-cmd --reload # Configure DNS Setting on FreeIPA Server # ipa dnsforwardzone-add [AD domain name] --forwarder=[AD IP address] --forward-policy=only ipa dnsforwardzone-add ad.zack.world --forwarder=11.0.1.181 --forward-policy=only # ipa dnszone-mod [IPA domain name] --allow-transfer=[AD IP address] ipa dnszone-mod ipa.zack.world --allow-transfer=11.0.1.181 # ipa trust-add --type=ad [AD domain name] --admin Administrator --password ipa trust-add --two-way=true --type=ad ad.zack.world --admin Administrator --password Active Directory domain administrator\u0026#39;s password: ----------------------------------------------------- Added Active Directory trust for realm \u0026#34;ad.zack.world\u0026#34; ----------------------------------------------------- Realm name: ad.zack.world Domain NetBIOS name: AD01 Domain Security Identifier: S-1-5-21-726412840-3773945212-2352305327 Trust direction: Two-way trust Trust type: Active Directory domain Trust status: Established and verified # set home directory at initial login authselect enable-feature with-mkhomedir systemctl enable --now oddjobd Validation of both idm clients with idm and AD user\nValidate ssh into Ubuntu client with AD user \u0026ldquo;joez@ad.zack.world\u0026rdquo;\nValidate ssh into CentOS client with idm user \u0026ldquo;tina\u0026rdquo;\nlogin as: tina Keyboard-interactive authentication prompts from server: | Password: End of keyboard-interactive prompts from server Last login: Sun Jul 14 20:48:42 2024 from 11.0.1.1 [tina@idm-client3-centos7 ~]$ id uid=1240000004(tina) gid=1240000004(tina) groups=1240000004(tina) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 Conclusion\nNow we install Redhat IdM server and can enrol client hosts, set up AD trust, ssh and authenticate with both idm and AD users. IdM using Kerberos for authentication, together with user group, policy, HBAC and Sudo roles, provides a flexible and robust authentication framework that supports multiple authentication mechanisms, enabling organizations to authenticate users securely across their Linux and Unix environments.\nMore info can be found via Freeipa workshop, FreeIPA:FreeIPA trust AD, Red Hat product documentation, Redhat Idm on AWS with DNS forwarder, idmfreeipa DNS forwarder configurations on AWS, and Automating Red Hat Identity Management installation with Ansible.\n","permalink":"https://zackblog.work/posts/redhat-identity-management-idm-with-ad-integration/","summary":"\u003cp\u003eWhen a company faces challenge to manage its Linux environments across local and public cloud, RedHat Identity management can be the solution to achieve:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eWith Local AD and Azure AD (AAD) Integration\u003c/li\u003e\n\u003cli\u003eWith AWS SSO Integration as external identity provider\u003c/li\u003e\n\u003cli\u003eLDAP, Kerberos and NTP\u003c/li\u003e\n\u003cli\u003eA web-based management front-end running on Apache\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eA Typical AD User Authentication Flow End-to-End:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUser Creation and Management:\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAzure AD / Local AD: Users are created in the Azure Active Directory or local Active Directory.\u003c/li\u003e\n\u003cli\u003eSynchronization to RedHat IdM: The users are synchronized from AD to RedHat IdM using the two-way trust established between AD and IdM.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eAccessing EC2 Instances via SSH:\u003c/strong\u003e\u003c/p\u003e","title":"RedHat Identity Management (IdM) with AD Integration"},{"content":"\u0026lsquo;if people tend to move to serverless, how \u0026lsquo;infrastructure engineer\u0026rsquo; will end up\u0026rsquo; Why go serverless\nSome of the company\u0026rsquo;s applications recently moved from Rancher to Fargate, which is understandable as the cloud resource and traffic will be very intensive only during a certain period (HSC exam), hence AWS serverless with Fargate can be a better option for such business mode so the rest of the year without the exam we can save costs significantly.\nHosting our blog on Fargate? Why not!\nIn the past, I used to try different methods to host this blog:\nEC2 with docker K8s with ArgoCD S3 with static website Customize Helm Chart for Zack\u0026rsquo; Blog Here I will use AWS Fargate, together with AWS ECR, Docker, Terraform and Github Action workflow to move this blog to AWS serverless compute for containers.\nTerraform Provisioning\n# Provider Configuration \u0026#34;provider.tf\u0026#34; provider \u0026#34;aws\u0026#34; { region = \u0026#34;ap-southeast-2\u0026#34; } # Create an ECR Repository \u0026#34;ecr.tf\u0026#34; resource \u0026#34;aws_ecr_repository\u0026#34; \u0026#34;zackblog_repo\u0026#34; { name = \u0026#34;zackblog-repo\u0026#34; } # Fargate Task Definition \u0026#34;task_definition.tf\u0026#34; resource \u0026#34;aws_ecs_task_definition\u0026#34; \u0026#34;zackblog_task\u0026#34; { family = \u0026#34;zackblog-task\u0026#34; network_mode = \u0026#34;awsvpc\u0026#34; requires_compatibilities = [\u0026#34;FARGATE\u0026#34;] cpu = \u0026#34;256\u0026#34; memory = \u0026#34;512\u0026#34; container_definitions = jsonencode([ { name = \u0026#34;zackblog-container\u0026#34;, image = \u0026#34;${aws_ecr_repository.zackblog_repo.repository_url}:latest\u0026#34;, essential = true, portMappings = [ { containerPort = 80, hostPort = 80, protocol = \u0026#34;tcp\u0026#34; } ] } ]) } # Create an ECS Cluster \u0026#34;cluster.tf\u0026#34; resource \u0026#34;aws_ecs_cluster\u0026#34; \u0026#34;zackblog_cluster\u0026#34; { name = \u0026#34;zackblog-cluster\u0026#34; } # Configure Networking to Use Default VPC - save cost haha # use the data block to fetch existing resources data \u0026#34;aws_vpc\u0026#34; \u0026#34;default\u0026#34; { default = true } data \u0026#34;aws_subnet\u0026#34; \u0026#34;default\u0026#34; { filter { name = \u0026#34;vpc-id\u0026#34; values = [data.aws_vpc.default.id] } } resource \u0026#34;aws_security_group\u0026#34; \u0026#34;zackblog_sg\u0026#34; { name_prefix = \u0026#34;zackblog-sg\u0026#34; vpc_id = data.aws_vpc.default.id ingress { from_port = 80 to_port = 80 protocol = \u0026#34;tcp\u0026#34; cidr_blocks = [\u0026#34;0.0.0.0/0\u0026#34;] } egress { from_port = 0 to_port = 0 protocol = \u0026#34;-1\u0026#34; cidr_blocks = [\u0026#34;0.0.0.0/0\u0026#34;] } } # Define the ECS Service \u0026#34;service.tf\u0026#34; resource \u0026#34;aws_ecs_service\u0026#34; \u0026#34;zackblog_service\u0026#34; { name = \u0026#34;zackblog-service\u0026#34; cluster = aws_ecs_cluster.zackblog_cluster.id task_definition = aws_ecs_task_definition.zackblog_task.arn desired_count = 1 launch_type = \u0026#34;FARGATE\u0026#34; network_configuration { subnets = [for subnet in data.aws_subnet.default : subnet.id] security_groups = [aws_security_group.zackblog_sg.id] assign_public_ip = true } } # Configure Load Balancer and attach to Fargate service \u0026#34;load_balancer.tf\u0026#34; resource \u0026#34;aws_lb\u0026#34; \u0026#34;zackblog_lb\u0026#34; { name = \u0026#34;zackblog-lb\u0026#34; internal = false load_balancer_type = \u0026#34;application\u0026#34; security_groups = [aws_security_group.zackblog_sg.id] subnets = [for subnet in data.aws_subnet.default : subnet.id] } resource \u0026#34;aws_lb_target_group\u0026#34; \u0026#34;zackblog_tg\u0026#34; { name = \u0026#34;zackblog-tg\u0026#34; port = 80 protocol = \u0026#34;HTTP\u0026#34; vpc_id = data.aws_vpc.default.id } resource \u0026#34;aws_lb_listener\u0026#34; \u0026#34;zackblog_listener\u0026#34; { load_balancer_arn = aws_lb.zackblog_lb.arn port = 80 protocol = \u0026#34;HTTP\u0026#34; default_action { type = \u0026#34;forward\u0026#34; target_group_arn = aws_lb_target_group.zackblog_tg.arn } } resource \u0026#34;aws_lb_target_group_attachment\u0026#34; \u0026#34;zackblog_tg_attachment\u0026#34; { target_group_arn = aws_lb_target_group.zackblog_tg.arn target_id = aws_ecs_service.zackblog_service.id port = 80 } Github Action Workflow for CICD\nFirst we need to create Github Secret to contain dockerhub and aws credentials and some other vars: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_REGION # xxx.dkr.ecr.ap-southeast-2.amazonaws.com ECR_REGISTRY # zackblog-repo ECR_REPOSITORY # zackblog-cluster ECS_CLUSTER # zackblog-service ECS_SERVICE Then define the workflow to create /.github/workflows/zackblog-fargate.yaml, in this configure Github runner, it will: Log in to Amazon ECR\nBuild and push Docker Image to the ECR repository\nDeploy to ECS by updating the ECS service to use the new image by forcing a new deployment\nname: Deploy to AWS Fargate on: push: branches: - editing # not main branch jobs: deploy: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v3 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v2 - name: Log in to Amazon ECR env: AWS_REGION: ${{ secrets.AWS_REGION }} run: | aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin ${{ secrets.ECR_REGISTRY }} - name: Build and push Docker image env: IMAGE_TAG: ${{ github.sha }} ECR_REGISTRY: ${{ secrets.ECR_REGISTRY }} ECR_REPOSITORY: ${{ secrets.ECR_REPOSITORY }} run: | docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG . docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG - name: Deploy to ECS env: AWS_REGION: ${{ secrets.AWS_REGION }} ECS_CLUSTER: ${{ secrets.ECS_CLUSTER }} ECS_SERVICE: ${{ secrets.ECS_SERVICE }} ECR_REGISTRY: ${{ secrets.ECR_REGISTRY }} ECR_REPOSITORY: ${{ secrets.ECR_REPOSITORY }} IMAGE_TAG: ${{ github.sha }} run: | aws ecs update-service --cluster $ECS_CLUSTER --service $ECS_SERVICE --force-new-deployment --region $AWS_REGION Conclusion\nNow we have a seamless incurvature as a code together with CICD pipeline to ensure that the \u0026ldquo;Zack\u0026rsquo;s Blog\u0026rdquo; can be moved to AWS serverless container service Fargate. Every time I update the blog by committing changes to the \u0026ldquo;zack-gitops-project\u0026rdquo; editing branch, a new Docker image will be built, pushed to ECR, and the AWS Fargate service is automatically updated.\n","permalink":"https://zackblog.work/posts/serverless-with-aws-fargate/","summary":"\u003cp\u003e\u0026lsquo;if people tend to move to serverless, how \u0026lsquo;infrastructure engineer\u0026rsquo; will end up\u0026rsquo;\n\u003cstrong\u003eWhy go serverless\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eSome of the company\u0026rsquo;s applications recently moved from Rancher to Fargate, which is understandable as the cloud resource and traffic will be very intensive only during a certain period (HSC exam), hence AWS serverless with Fargate can be a better option for such business mode so the rest of the year without the exam we can save costs significantly.\u003c/p\u003e","title":"Serverless with AWS Fargate"},{"content":"Today I got a performance issue from our analytic team, saying they experienced a Production MySQL cluster running on RDS very slow since yesterday morning.\nI started to look into below areas for investigation:\nAWS CloudWatch Metrics for RDS AWS CloudWatch provides a wide range of metrics that can help diagnose resource usage for databases. So I started with:\nCloudWatch - Metrics - All metrics - Add query - RDS - Top 10 RDS instances by highest CPU utilization\nThis only queries the recent 3 hours metrics, but it is enough for me to identify the issue: CPU 100%\nTo further understand the high CPU, I go:\nCloudWatch - Metrics - All metrics - Browse - RDS - DBClusterIdentifier - CPUUtilization\nWhich gives me a long period of monitoring, so I can see it started to 100% CPU since yesterday morning.\nAWS console RDS Logs \u0026amp; events Now let\u0026rsquo;s find out from the RDS logs to see if any errors can indicate who could be the person. So I go:\nRDS - \u0026ldquo;the DB cluster\u0026rdquo; - \u0026ldquo;the DB instance\u0026rdquo; - \u0026ldquo;Logs \u0026amp; events\u0026rdquo; - \u0026ldquo;error/mysql-error-running.log.2024-06-18.02\u0026rdquo;\nI got:\n2024-06-18T00:04:58.750212Z 2831474 [Note] Aborted connection 2831474 to db: \u0026#39;xxxxxx\u0026#39; user: \u0026#39;xxxx\u0026#39; host: \u0026#39;10.xx.xx.xx\u0026#39; (Unknown error) 2024-06-18T00:12:00.798173Z 2831498 [Note] Aborted connection 2831498 to db: \u0026#39;xxxx\u0026#39; user: \u0026#39;xxxx\u0026#39; host: \u0026#39;10.xx.xx.xx\u0026#39; (Got an error writing communication packets) ----------------------- END OF LOG ---------------------- Up to here I generally have an idea of what is going on and can locate the person \u0026ldquo;xxxx\u0026rdquo; who was running something at the time CPU 100%.\nMySQL Client Tool to list, identify and terminate long-running queries It is time to log in to the RDS endpoint to see what is happening and which queries might cause the CPU usage. Here we need to log in via MySQL \u0026ldquo;root\u0026rdquo; to be able to see all other users\u0026rsquo; running processes. Then pay attention to the high \u0026ldquo;Time\u0026rdquo; and \u0026ldquo;State\u0026rdquo; values indicating all the stuck processes, then we kill them and restart the RDS instance.\nmysql -u root -p -h rds_endpoint SHOW PROCESSLIST; KILL \u0026lt;process_id\u0026gt;; Then I go AWS RDS console - Actions - Reboot the RDS instance.\nNow the CPU usage started to drop and back to normal after terminating the stuck processes and DB instance reboot.\nDone.\nConclusion\nEven though the issue had been fixed, I was still thinking about how to better monitor RDS resource usage. I think we need:\nA \u0026ldquo;CloudWatch Alarm\u0026rdquo; to set \u0026ldquo;CPUUtilization\u0026rdquo; metric threshold to 80%, then specify the period (e.g., 5 minutes) and the number of periods (e.g., 2 out of 3) that the metric must breach the threshold to trigger the alarm. Create an \u0026ldquo;SNS topic\u0026rdquo; with team Email for the alarm to send a notification. Enable \u0026ldquo;RDS Performance Insights\u0026rdquo;, this can monitor the load on the database, identify the source of bottlenecks, and understand how the DB is performing, especially during troubleshooting. Enable \u0026ldquo;Enhanced Monitoring\u0026rdquo; and select the monitoring interval (e.g., 1 minute), which provides real-time metrics for the operating system that the DB instance runs on, this helps for immediate investigation on OS level. Enable \u0026ldquo;Slow Query Log\u0026rdquo; for regularly analyzing slow query logs and performance insights to optimize RDS database queries, identify queries that take a long time to execute, use tools like EXPLAIN to understand query performance, add appropriate indexes, and then ultimately rewrite queries for better performance. ","permalink":"https://zackblog.work/posts/handling-a-rds-mysql-cluster-cpu-100/","summary":"\u003cp\u003eToday I got a performance issue from our analytic team, saying they experienced a Production MySQL cluster running on RDS very slow since yesterday morning.\u003c/p\u003e\n\u003cp\u003eI started to look into below areas for investigation:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAWS CloudWatch Metrics for RDS\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eAWS CloudWatch provides a wide range of metrics that can help diagnose resource usage for databases. So I started with:\u003c/p\u003e\n\u003cp\u003e\u003cem\u003eCloudWatch - Metrics - All metrics - Add query - RDS - Top 10 RDS instances by highest CPU utilization\u003c/em\u003e\u003c/p\u003e","title":"Handling a RDS MySQL cluster CPU 100%"},{"content":"Now It is time to change from docker-compose to deploy into Kubernetes.\nAs this is not new to me to deploy microservice into K8S, also I already have a running Kubernetes cluster in hand, so here I will just create docker images for the 3 services: API gateway, user and order, then push them into the docker hub repository, then create Kubernetes manifest for deployment and service.\n# Folder structure /07-with-k8s . ├── api_gateway.py ├── depolyment.yaml ├── Dockerfile_apigateway ├── Dockerfile_order ├── Dockerfile_user ├── order_service.py └── user_service.py # Build, tag and push the docker images docker login docker build -t zackz001/python-user:latest -f Dockerfile_user . docker build -t zackz001/python-order:latest -f Dockerfile_order . docker build -t zackz001/python-apigateway:latest -f Dockerfile_apigateway . docker push zackz001/python-apigateway:latest docker push zackz001/python-user:latest docker push zackz001/python-order:latest docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE zackz001/python-apigateway latest bc3db11f4be8 1 hours ago 138MB zackz001/python-user latest c93973bece33 1 hours ago 136MB zackz001/python-order latest e35d5de9254b 1 hours ago 136MB prom/prometheus latest 1bd2b9635267 8 days ago 271MB grafana/grafana latest c42c21cd0ebc 3 weeks ago 453MB consul 1.15.4 686495461132 4 months ago 155MB docker.elastic.co/elasticsearch/elasticsearch 7.13.2 11a830014f7c 3 years ago 1.02GB docker.elastic.co/logstash/logstash 7.13.2 8dc1af4dd662 3 years ago 965MB docker.elastic.co/kibana/kibana 7.13.2 6c4869a27be1 3 years ago 1.35GB # k8s deployment Manifests apiVersion: apps/v1 kind: Deployment metadata: name: user-service spec: replicas: 1 selector: matchLabels: app: user-service template: metadata: labels: app: user-service spec: containers: - name: user-service image: zackz001/python-user:latest ports: - containerPort: 5001 --- apiVersion: v1 kind: Service metadata: name: user-service spec: selector: app: user-service ports: - protocol: TCP port: 5001 targetPort: 5001 --- apiVersion: apps/v1 kind: Deployment metadata: name: order-service spec: replicas: 1 selector: matchLabels: app: order-service template: metadata: labels: app: order-service spec: containers: - name: order-service image: zackz001/python-order:latest ports: - containerPort: 5002 --- apiVersion: v1 kind: Service metadata: name: order-service spec: selector: app: order-service ports: - protocol: TCP port: 5002 targetPort: 5002 --- apiVersion: apps/v1 kind: Deployment metadata: name: api-gateway spec: replicas: 1 selector: matchLabels: app: api-gateway template: metadata: labels: app: api-gateway spec: containers: - name: api-gateway image: zackz001/python-apigateway:latest ports: - containerPort: 5000 --- apiVersion: v1 kind: Service metadata: name: api-gateway spec: type: NodePort selector: app: api-gateway ports: - protocol: TCP port: 5000 targetPort: 5000 Now Run kubectl apply -f to bring all deployments and services up and running. Should see all services in the Rancher console.\nkubectl create ns python kubectl apply -f depolyment.yaml -n python kubectl get all -n python NAME READY STATUS RESTARTS AGE pod/api-gateway-d664cf8c4-7l7q8 1/1 Running 2 (50m ago) 1h pod/order-service-856577f666-gk5gt 1/1 Running 2 (50m ago) 1h pod/user-service-5d8766d9cb-4rqnz 1/1 Running 2 (50m ago) 1h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/api-gateway NodePort 10.43.38.27 \u0026lt;none\u0026gt; 5000:32060/TCP 1h service/order-service ClusterIP 10.43.27.255 \u0026lt;none\u0026gt; 5002/TCP 1h service/user-service ClusterIP 10.43.160.232 \u0026lt;none\u0026gt; 5001/TCP 1h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/api-gateway 1/1 1 1 1h deployment.apps/order-service 1/1 1 1 1h deployment.apps/user-service 1/1 1 1 1h NAME DESIRED CURRENT READY AGE replicaset.apps/api-gateway-d664cf8c4 1 1 1 1h replicaset.apps/order-service-856577f666 1 1 1 1h replicaset.apps/user-service-5d8766d9cb 1 1 1 1h Verify API gateway, user and order services\nAccess API gateway via http://NodeIP:NodePort/users and http://NodeIP:NodePort/orders\nConclusion\nNow we complete all Python Flask sessions.\nI have done this End-to-End Python Microservice application solution development, which enhanced my DevOps practices of Python programming, microservices architecture design and deployment with docker-compose, API Gateway implementation, service registery with Consul, logging and monitoring, and finally Kubernetes deployment.\nsimple Python Flask app Microservice applications with user and order Create API gateway Flask app Service registery with Consul Logging with ELK Monitoring with Prometheus and Grafana K8S deployment ","permalink":"https://zackblog.work/posts/python-microservice-k8s-deployment/","summary":"\u003cp\u003eNow It is time to change from docker-compose to deploy into Kubernetes.\u003c/p\u003e\n\u003cp\u003eAs this is not new to me to deploy microservice into K8S, also I already have a running Kubernetes cluster in hand, so here I will just create docker images for the 3 services: API gateway, user and order, then push them into the docker hub repository, then create Kubernetes manifest for deployment and service.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Folder structure\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003e/07-with-k8s\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003e.\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003e├── api_gateway.py\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003e├── depolyment.yaml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003e├── Dockerfile_apigateway\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003e├── Dockerfile_order\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003e├── Dockerfile_user\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003e├── order_service.py\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003e└── user_service.py\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# Build, tag and push the docker images\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003edocker login\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003edocker build -t zackz001/python-user:latest -f Dockerfile_user .\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003edocker build -t zackz001/python-order:latest -f Dockerfile_order .\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003edocker build -t zackz001/python-apigateway:latest -f Dockerfile_apigateway .\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003edocker push zackz001/python-apigateway:latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003edocker push zackz001/python-user:latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003edocker push zackz001/python-order:latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003edocker image ls\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eREPOSITORY                                      TAG       IMAGE ID       CREATED        SIZE\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003ezackz001/python-apigateway                      latest    bc3db11f4be8   1 hours ago    138MB\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003ezackz001/python-user                            latest    c93973bece33   1 hours ago    136MB\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003ezackz001/python-order                           latest    e35d5de9254b   1 hours ago    136MB\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eprom/prometheus                                 latest    1bd2b9635267   8 days ago     271MB\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003egrafana/grafana                                 latest    c42c21cd0ebc   3 weeks ago    453MB\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003econsul                                          1.15.4    686495461132   4 months ago   155MB\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003edocker.elastic.co/elasticsearch/elasticsearch   7.13.2    11a830014f7c   3 years ago    1.02GB\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003edocker.elastic.co/logstash/logstash             7.13.2    8dc1af4dd662   3 years ago    965MB\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003edocker.elastic.co/kibana/kibana                 7.13.2    6c4869a27be1   3 years ago    1.35GB\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# k8s deployment Manifests\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapps/v1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeployment\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003euser-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003ereplicas\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003eselector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ematchLabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e   \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003euser-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003etemplate\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e   \u003c/span\u003e\u003cspan class=\"nt\"\u003elabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003euser-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e   \u003c/span\u003e\u003cspan class=\"nt\"\u003econtainers\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003euser-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eimage\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ezackz001/python-user:latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e       \u003c/span\u003e- \u003cspan class=\"nt\"\u003econtainerPort\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5001\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nn\"\u003e---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eService\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003euser-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003eselector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003euser-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eprotocol\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eTCP\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eport\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5001\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003etargetPort\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5001\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nn\"\u003e---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapps/v1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeployment\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eorder-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003ereplicas\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003eselector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ematchLabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e   \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eorder-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003etemplate\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e   \u003c/span\u003e\u003cspan class=\"nt\"\u003elabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eorder-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e   \u003c/span\u003e\u003cspan class=\"nt\"\u003econtainers\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eorder-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eimage\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ezackz001/python-order:latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e       \u003c/span\u003e- \u003cspan class=\"nt\"\u003econtainerPort\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5002\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nn\"\u003e---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eService\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eorder-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003eselector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eorder-service\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eprotocol\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eTCP\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eport\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5002\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003etargetPort\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5002\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nn\"\u003e---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapps/v1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eDeployment\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapi-gateway\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003ereplicas\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003eselector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ematchLabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e   \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapi-gateway\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003etemplate\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e   \u003c/span\u003e\u003cspan class=\"nt\"\u003elabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapi-gateway\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e   \u003c/span\u003e\u003cspan class=\"nt\"\u003econtainers\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapi-gateway\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eimage\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ezackz001/python-apigateway:latest\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e       \u003c/span\u003e- \u003cspan class=\"nt\"\u003econtainerPort\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5000\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nn\"\u003e---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eService\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapi-gateway\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003etype\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eNodePort\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003eselector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapi-gateway\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eprotocol\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eTCP\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eport\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5000\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003etargetPort\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5000\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eNow Run kubectl apply -f to bring all deployments and services up and running. Should see all services in the Rancher console.\u003c/p\u003e","title":"Python Microservice: K8S deployment"},{"content":"Both Prometheus and Grafana are compatible with microservice applications, integrating Prometheus with Flask is straightforward to provide performance and monitoring metrics.\nHere I will update the docker-compose to add Prometheus and Grafana as services, then add Prometheus metrics in both order and user application code by importing PrometheusMetrics from the prometheus_flask_exporter module, which is used to expose Prometheus metrics for the Flask application. Then initialize Prometheus metrics with metrics = PrometheusMetrics(app). Use @metrics.counter('get_orders_count', 'Count of calls to the get_orders endpoint') to define a Prometheus counter metric that increments each time the get_orders endpoint is called.\nImport os to enable environment variable for Logstash host, logstash_host = os.getenv('LOGSTASH_HOST', 'localhost') to fetch the Logstash host from environment variables, defaulting to localhost.\n# Folder structure /06-with-monitoringstack ├── api_gateway │ ├── api_gateway.py │ └── Dockerfile ├── docker-compose.yml ├── logstash.conf ├── order_service │ ├── Dockerfile │ ├── order_service.py ├── prometheus.yaml └── user_service ├── Dockerfile └── user_service.py # add Prometheus and Grafana in docker-compose.yaml version: \u0026#39;3\u0026#39; services: prometheus: image: prom/prometheus volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml ports: - \u0026#34;9090:9090\u0026#34; grafana: image: grafana/grafana ports: - \u0026#34;3000:3000\u0026#34; # create prometheus.yml for Prometheus configration # vim prometheus.yml global: scrape_interval: 15s scrape_configs: - job_name: \u0026#39;flask\u0026#39; static_configs: - targets: [\u0026#39;user-service:5001\u0026#39;, \u0026#39;order-service:5002\u0026#39;] # add Monitoring logic into python code # order_service.py import logging import requests from flask import Flask, jsonify from pygelf import GelfUdpHandler from prometheus_flask_exporter import PrometheusMetrics import os app = Flask(__name__) metrics = PrometheusMetrics(app) @app.route(\u0026#39;/orders\u0026#39;) def get_orders(): orders = [ {\u0026#39;id\u0026#39;: 1, \u0026#39;item\u0026#39;: \u0026#39;Laptop\u0026#39;, \u0026#39;price\u0026#39;: 1200}, {\u0026#39;id\u0026#39;: 2, \u0026#39;item\u0026#39;: \u0026#39;Phone\u0026#39;, \u0026#39;price\u0026#39;: 800} ] app.logger.info(\u0026#34;Fetched order data\u0026#34;) return jsonify(orders) def register_service(): payload = { \u0026#34;ID\u0026#34;: \u0026#34;order-service\u0026#34;, \u0026#34;Name\u0026#34;: \u0026#34;order-service\u0026#34;, \u0026#34;Address\u0026#34;: \u0026#34;order-service\u0026#34;, \u0026#34;Port\u0026#34;: 5002 } response = requests.put(\u0026#39;http://consul:8500/v1/agent/service/register\u0026#39;, json=payload) if response.status_code == 200: app.logger.info(\u0026#34;Order service registered successfully\u0026#34;) else: app.logger.error(\u0026#34;Failed to register order service\u0026#34;) if __name__ == \u0026#39;__main__\u0026#39;: # Configure logging logstash_host = os.getenv(\u0026#39;LOGSTASH_HOST\u0026#39;, \u0026#39;localhost\u0026#39;) handler = GelfUdpHandler(host=logstash_host, port=12201) app.logger.addHandler(handler) app.logger.setLevel(logging.INFO) register_service() app.run(host=\u0026#39;0.0.0.0\u0026#39;, port=5002) # user_service.py import logging import requests from flask import Flask, jsonify from pygelf import GelfUdpHandler from prometheus_flask_exporter import PrometheusMetrics import os app = Flask(__name__) metrics = PrometheusMetrics(app) @app.route(\u0026#39;/users\u0026#39;) def get_users(): users = [ {\u0026#39;id\u0026#39;: 1, \u0026#39;name\u0026#39;: \u0026#39;Alice\u0026#39;}, {\u0026#39;id\u0026#39;: 2, \u0026#39;name\u0026#39;: \u0026#39;Bob\u0026#39;} ] app.logger.info(\u0026#34;Fetched user data\u0026#34;) return jsonify(users) def register_service(): payload = { \u0026#34;ID\u0026#34;: \u0026#34;user-service\u0026#34;, \u0026#34;Name\u0026#34;: \u0026#34;user-service\u0026#34;, \u0026#34;Address\u0026#34;: \u0026#34;user-service\u0026#34;, \u0026#34;Port\u0026#34;: 5001 } response = requests.put(\u0026#39;http://consul:8500/v1/agent/service/register\u0026#39;, json=payload) if response.status_code == 200: app.logger.info(\u0026#34;User service registered successfully\u0026#34;) else: app.logger.error(\u0026#34;Failed to register user service\u0026#34;) if __name__ == \u0026#39;__main__\u0026#39;: # Configure logging logstash_host = os.getenv(\u0026#39;LOGSTASH_HOST\u0026#39;, \u0026#39;localhost\u0026#39;) handler = GelfUdpHandler(host=logstash_host, port=12201) app.logger.addHandler(handler) app.logger.setLevel(logging.INFO) register_service() app.run(host=\u0026#39;0.0.0.0\u0026#39;, port=5001) Now Run docker-compose to bring all containers up and running. Should see all services with Prometheus and Grafana.\ndocker-compose up --build Creating 06-with-monitoringstack_api-gateway_1 ... done Creating 06-with-monitoringstack_order-service_1 ... done Creating 06-with-monitoringstack_user-service_1 ... done Creating 06-with-monitoringstack_kibana_1 ... done Creating 06-with-monitoringstack_grafana_1 ... done Creating 06-with-monitoringstack_logstash_1 ... done Creating 06-with-monitoringstack_prometheus_1 ... done Creating 06-with-monitoringstack_elasticsearch_1 ... done Creating 06-with-monitoringstack_consul_1 ... done Verify Prometheus and Grafana\nAccess Grafana via http://localhost:3000, log in with admin/admin.\nAdd Prometheus as a Data Source in Grafana by setting the URL to http://localhost:9090 and save.\nCreate Dashboards in Grafana to use the Prometheus data source to visualize metrics from user-service and order-service.\nConclusion\nNow we can enable monitoring with Prometheus and Grafana.\nIn the next post, I will see how to deploy our Python Flask microservice into Kubernetes.\n","permalink":"https://zackblog.work/posts/python-microservice-monitoring-stack/","summary":"\u003cp\u003eBoth Prometheus and Grafana are compatible with microservice applications, integrating Prometheus with Flask is straightforward to provide performance and monitoring metrics.\u003c/p\u003e\n\u003cp\u003eHere I will update the docker-compose to add Prometheus and Grafana as services, then add Prometheus metrics in both order and user application code by importing \u003ccode\u003ePrometheusMetrics\u003c/code\u003e from the \u003ccode\u003eprometheus_flask_exporter\u003c/code\u003e module, which is used to expose Prometheus metrics for the Flask application. Then initialize Prometheus metrics with \u003ccode\u003emetrics = PrometheusMetrics(app)\u003c/code\u003e. Use \u003ccode\u003e@metrics.counter('get_orders_count', 'Count of calls to the get_orders endpoint')\u003c/code\u003e to define a Prometheus counter metric that increments each time the get_orders endpoint is called.\u003c/p\u003e","title":"Python Microservice: Monitoring Stack"},{"content":"ELK (Elasticsearch, Logstash, Kibana) is a popular log management solution. We will use the ELK Stack to collect and analyze logs.\nHere I need to extend the current configuration by adding services in the docker-compose file for Elasticsearch, Logstash, and Kibana, and configure the microservices to send logs to Logstash.\nAlso, I will need to configure the logging in both user and order Python application code to send logs to Logstash. By importing the built-in logging module and GelfUdpHandler from the pygelf module, to provide a flexible framework for emitting log messages from Python programs to send log messages in the GELF (Graylog Extended Log Format) to a remote Graylog server, which is typically part of the ELK stack.\nBy adding log messages using app.logger.info and app.logger.error, together with the defined logging level, I can set the logging level to INFO, which means all log messages at this level or higher will be emitted.\n# folder structure 05-with-ELK/ ├── api_gateway/ │ └── Dockerfile ├── order_service/ │ └── Dockerfile ├── user_service/ │ └── Dockerfile ├── logstash.conf └── docker-compose.yml # create Logstash Configuration # vim logstash.conf input { gelf { port =\u0026gt; 12201 } } output { elasticsearch { hosts =\u0026gt; [\u0026#34;elasticsearch:9200\u0026#34;] index =\u0026gt; \u0026#34;%{[@metadata][beat]}-%{+YYYY.MM.dd}\u0026#34; } } # vim user_service.py import logging import requests from flask import Flask, jsonify from pygelf import GelfUdpHandler app = Flask(__name__) @app.route(\u0026#39;/users\u0026#39;) def get_users(): users = [ {\u0026#39;id\u0026#39;: 1, \u0026#39;name\u0026#39;: \u0026#39;Alice\u0026#39;}, {\u0026#39;id\u0026#39;: 2, \u0026#39;name\u0026#39;: \u0026#39;Bob\u0026#39;} ] app.logger.info(\u0026#34;Fetched user data\u0026#34;) return jsonify(users) def register_service(): payload = { \u0026#34;ID\u0026#34;: \u0026#34;user-service\u0026#34;, \u0026#34;Name\u0026#34;: \u0026#34;user-service\u0026#34;, \u0026#34;Address\u0026#34;: \u0026#34;user-service\u0026#34;, \u0026#34;Port\u0026#34;: 5001 } response = requests.put(\u0026#39;http://consul:8500/v1/agent/service/register\u0026#39;, json=payload) if response.status_code == 200: app.logger.info(\u0026#34;User service registered successfully\u0026#34;) else: app.logger.error(\u0026#34;Failed to register user service\u0026#34;) if __name__ == \u0026#39;__main__\u0026#39;: # Configure logging handler = GelfUdpHandler(host=\u0026#39;logstash\u0026#39;, port=12201) app.logger.addHandler(handler) app.logger.setLevel(logging.INFO) register_service() app.run(host=\u0026#39;0.0.0.0\u0026#39;, port=5001) # vim order_service.py import logging import requests from flask import Flask, jsonify from pygelf import GelfUdpHandler app = Flask(__name__) @app.route(\u0026#39;/orders\u0026#39;) def get_orders(): orders = [ {\u0026#39;id\u0026#39;: 1, \u0026#39;item\u0026#39;: \u0026#39;Laptop\u0026#39;, \u0026#39;price\u0026#39;: 1200}, {\u0026#39;id\u0026#39;: 2, \u0026#39;item\u0026#39;: \u0026#39;Phone\u0026#39;, \u0026#39;price\u0026#39;: 800} ] app.logger.info(\u0026#34;Fetched order data\u0026#34;) return jsonify(orders) def register_service(): payload = { \u0026#34;ID\u0026#34;: \u0026#34;order-service\u0026#34;, \u0026#34;Name\u0026#34;: \u0026#34;order-service\u0026#34;, \u0026#34;Address\u0026#34;: \u0026#34;order-service\u0026#34;, \u0026#34;Port\u0026#34;: 5002 } response = requests.put(\u0026#39;http://consul:8500/v1/agent/service/register\u0026#39;, json=payload) if response.status_code == 200: app.logger.info(\u0026#34;Order service registered successfully\u0026#34;) else: app.logger.error(\u0026#34;Failed to register order service\u0026#34;) if __name__ == \u0026#39;__main__\u0026#39;: # Configure logging handler = GelfUdpHandler(host=\u0026#39;logstash\u0026#39;, port=12201) app.logger.addHandler(handler) app.logger.setLevel(logging.INFO) register_service() app.run(host=\u0026#39;0.0.0.0\u0026#39;, port=5002) # create user_service/requirements.txt for each service (user, order) flask requests pygelf # modify each Dockerfile: Dockerfile-user # Use an official Python runtime as a parent image FROM python:3.9-slim # Set the working directory in the container WORKDIR /app # Copy the current directory contents into the container at /app COPY . /app # Install any needed packages specified in requirements.txt RUN pip install --no-cache-dir -r requirements.txt # Make port 5001 available to the world outside this container EXPOSE 5001 # Define environment variable ENV FLASK_APP=user_service.py # Run user_service.py when the container launches CMD [\u0026#34;flask\u0026#34;, \u0026#34;run\u0026#34;, \u0026#34;--host=0.0.0.0\u0026#34;, \u0026#34;--port=5001\u0026#34;] # vim Dockerfile-order # Use an official Python runtime as a parent image FROM python:3.9-slim # Set the working directory in the container WORKDIR /app # Copy the current directory contents into the container at /app COPY . /app # Install any needed packages specified in requirements.txt RUN pip install --no-cache-dir -r requirements.txt # Make port 5002 available to the world outside this container EXPOSE 5002 # Define environment variable ENV FLASK_APP=order_service.py # Run order_service.py when the container launches CMD [\u0026#34;flask\u0026#34;, \u0026#34;run\u0026#34;, \u0026#34;--host=0.0.0.0\u0026#34;, \u0026#34;--port=5002\u0026#34;] # modify docker-compose.yaml version: \u0026#39;3\u0026#39; services: consul: image: consul:1.15.4 ports: - \u0026#34;8500:8500\u0026#34; elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:7.13.2 environment: - discovery.type=single-node ports: - \u0026#34;9200:9200\u0026#34; - \u0026#34;9300:9300\u0026#34; volumes: - esdata:/usr/share/elasticsearch/data logstash: image: docker.elastic.co/logstash/logstash:7.13.2 volumes: - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf ports: - \u0026#34;12201:12201/udp\u0026#34; - \u0026#34;5044:5044\u0026#34; kibana: image: docker.elastic.co/kibana/kibana:7.13.2 ports: - \u0026#34;5601:5601\u0026#34; depends_on: - elasticsearch user-service: build: context: ./user_service depends_on: - consul - logstash environment: - CONSUL_HTTP_ADDR=consul:8500 ports: - \u0026#34;5001:5001\u0026#34; logging: driver: gelf options: gelf-address: udp://logstash:12201 order-service: build: context: ./order_service depends_on: - consul - logstash environment: - CONSUL_HTTP_ADDR=consul:8500 ports: - \u0026#34;5002:5002\u0026#34; logging: driver: gelf options: gelf-address: udp://logstash:12201 api-gateway: build: context: ./api_gateway depends_on: - consul - user-service - order-service - logstash environment: - CONSUL_HTTP_ADDR=consul:8500 ports: - \u0026#34;5000:5000\u0026#34; logging: driver: gelf options: gelf-address: udp://logstash:12201 volumes: esdata: Now Run docker-compose to bring all containers up and running. Should see all services with ES, Logstash and Kibana populating logs on the screen.\ndocker-compose up --build Creating 05-with-elk_logstash_1 ... done Creating 05-with-elk_consul_1 ... done Creating 05-with-elk_elasticsearch_1 ... done Creating 05-with-elk_kibana_1 ... done Creating 05-with-elk_user-service_1 ... done Creating 05-with-elk_order-service_1 ... done Creating 05-with-elk_api-gateway_1 ... done logstash_1 | [2024-05-18T14:55:13,140][INFO ][logstash.inputs.udp ][main][a30d8db137f99f1de18acbd53c081374cd720430a4dd0e752ff4a99c3005f9d0] Starting UDP listener {:address=\u0026gt;\u0026#34;0.0.0.0:12201\u0026#34;} logstash_1 | [2024-05-18T14:55:13,187][INFO ][logstash.inputs.udp ][main][a30d8db137f99f1de18acbd53c081374cd720430a4dd0e752ff4a99c3005f9d0] UDP listener started {:address=\u0026gt;\u0026#34;0.0.0.0:12201\u0026#34;, :receive_buffer_bytes=\u0026gt;\u0026#34;106496\u0026#34;, :queue_size=\u0026gt;\u0026#34;2000\u0026#34;} consul_1 | 2024-05-18T14:55:46.686Z [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth consul_1 | 2024-05-18T14:55:46.688Z [DEBUG] agent: Node info in sync logstash_1 | https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html elasticsearch_1 | {\u0026#34;type\u0026#34;: \u0026#34;deprecation.elasticsearch\u0026#34;, \u0026#34;timestamp\u0026#34;: \u0026#34;2024-05-18T14:55:09,016Z\u0026#34;, \u0026#34;level\u0026#34;: \u0026#34;DEPRECATION\u0026#34;, \u0026#34;component\u0026#34;: \u0026#34;o.e.d.r.RestController\u0026#34;, \u0026#34;cluster.name\u0026#34;: \u0026#34;docker-cluster\u0026#34;, \u0026#34;node.name\u0026#34;: \u0026#34;430bff78a529\u0026#34;, \u0026#34;message\u0026#34;: \u0026#34;Legacy index templates are deprecated in favor of composable templates.\u0026#34;, \u0026#34;cluster.uuid\u0026#34;: \u0026#34;B9QKhgEGTA6Ot5auY9skQQ\u0026#34;, \u0026#34;node.id\u0026#34;: \u0026#34;QzKPL7DYSB2_CeWJpUaxXg\u0026#34; } kibana_1 | {\u0026#34;type\u0026#34;:\u0026#34;log\u0026#34;,\u0026#34;@timestamp\u0026#34;:\u0026#34;2024-05-18T14:55:09+00:00\u0026#34;,\u0026#34;tags\u0026#34;:[\u0026#34;info\u0026#34;,\u0026#34;plugins\u0026#34;,\u0026#34;monitoring\u0026#34;,\u0026#34;monitoring\u0026#34;,\u0026#34;kibana-monitoring\u0026#34;],\u0026#34;pid\u0026#34;:952,\u0026#34;message\u0026#34;:\u0026#34;Starting monitoring stats collection\u0026#34;} Verify ElasticSearch and Kibana\nValidate ElasticSearch status via localhost:9200\nVisit localhost:5601 to access the Kibana dashboard, add Index Pattern \u0026ldquo;logs-*\u0026rdquo; to see data populated in the Discover tab\nConclusion\nNow we can enable logging with the ELK stack, and use Logstash, ElasticSearch, and Kibana.\nIn the next post, I will see how to enable monitoring with Prometheus and Grafana stack.\n","permalink":"https://zackblog.work/posts/python-microservice-logging-with-elk/","summary":"\u003cp\u003eELK (Elasticsearch, Logstash, Kibana) is a popular log management solution. We will use the ELK Stack to collect and analyze logs.\u003c/p\u003e\n\u003cp\u003eHere I need to extend the current configuration by adding services in the docker-compose file for \u003ccode\u003eElasticsearch\u003c/code\u003e, \u003ccode\u003eLogstash\u003c/code\u003e, and \u003ccode\u003eKibana\u003c/code\u003e, and configure the microservices to send logs to Logstash.\u003c/p\u003e\n\u003cp\u003eAlso, I will need to configure the logging in both \u003ccode\u003euser\u003c/code\u003e and \u003ccode\u003eorder\u003c/code\u003e Python application code to send logs to Logstash. By importing the built-in \u003ccode\u003elogging\u003c/code\u003e module and \u003ccode\u003eGelfUdpHandler\u003c/code\u003e from the \u003ccode\u003epygelf\u003c/code\u003e module, to provide a flexible framework for emitting log messages from Python programs to send log messages in the \u003ccode\u003eGELF\u003c/code\u003e (Graylog Extended Log Format) to a remote Graylog server, which is typically part of the ELK stack.\u003c/p\u003e","title":"Python Microservice: Logging with ELK"},{"content":"API Gateway acts as a single entry point for all clients and handles the request routing, composition, and protocol translation in a microservices architecture, here I will create an API Gateway using Python Flask and the requests library, to route both \u0026ldquo;user\u0026rdquo; and \u0026ldquo;order\u0026rdquo; services.\nHere I will create an API Gateway to handle the 2 services (user and order). By importing the requests module, which allows us to send HTTP requests in Python. It\u0026rsquo;s used for making API calls to other services.\nThen with route decorator to specify that the get_users function should handle requests to the /users URL endpoint.\nThen define response to send GET request to the user-service and order-service running on port 5001/5002 at each endpoint.\n# Folder Structure 03-with-api-gatway/ ├── user_service/ │ ├── Dockerfile │ └── user_service.py ├── order_service/ │ ├── Dockerfile │ └── order_service.py ├── api_gateway/ │ ├── Dockerfile │ └── api_gateway.py ├── docker-compose.yml # api_gateway.py from flask import Flask, jsonify import requests app = Flask(__name__) @app.route(\u0026#39;/users\u0026#39;) def get_users(): response = requests.get(\u0026#39;http://user-service:5001/users\u0026#39;) return jsonify(response.json()) @app.route(\u0026#39;/orders\u0026#39;) def get_orders(): response = requests.get(\u0026#39;http://order-service:5002/orders\u0026#39;) return jsonify(response.json()) if __name__ == \u0026#39;__main__\u0026#39;: app.run(host=\u0026#39;0.0.0.0\u0026#39;, port=5000) # Dockerfile_apigateway # Use an official Python runtime as a parent image FROM python:3.9-slim # Set the working directory in the container WORKDIR /app # Copy the current directory contents into the container at /app COPY . /app # Install flask requests RUN pip install --no-cache-dir flask requests # Make port 5000 available to the world outside this container EXPOSE 5000 # Define environment variable ENV FLASK_APP=api_gateway.py # Run api_gateway.py when the container launches CMD [\u0026#34;flask\u0026#34;, \u0026#34;run\u0026#34;, \u0026#34;--host=0.0.0.0\u0026#34;, \u0026#34;--port=5000\u0026#34;] # update docker-compose.yaml ..... api-gateway: build: context: . dockerfile: Dockerfile_apigateway ports: - \u0026#34;5000:5000\u0026#34; # run docker-compose up --build docker-compose up --build Verify the 2 services can be accessed via API Gateway address and port plus /users and /orders by defining request functions.\nAbout Consul\nConsul is a popular open-source tool for service discovery and service registration, here I will update the py files to register both services with Consul\nIn both user_service.py and order_service.py, add service registration logic.\nTo import time module, also define function named register_service that will handle the service registration logic with Consul, use dictionary defines the payload for the service registration. It includes the service ID, name, address, and port.\nI will use while True to start an infinite loop, which will keep trying to register the service with Consul until it succeeds, add try, else and if to handle exceptions during the registration process.\n# vim order_service.py import requests from flask import Flask, jsonify import time app = Flask(__name__) @app.route(\u0026#39;/orders\u0026#39;) def get_orders(): orders = [ {\u0026#39;id\u0026#39;: 1, \u0026#39;item\u0026#39;: \u0026#39;Laptop\u0026#39;, \u0026#39;price\u0026#39;: 1200}, {\u0026#39;id\u0026#39;: 2, \u0026#39;item\u0026#39;: \u0026#39;Phone\u0026#39;, \u0026#39;price\u0026#39;: 800} ] return jsonify(orders) def register_service(): payload = { \u0026#34;ID\u0026#34;: \u0026#34;order-service\u0026#34;, \u0026#34;Name\u0026#34;: \u0026#34;order-service\u0026#34;, \u0026#34;Address\u0026#34;: \u0026#34;order-service\u0026#34;, \u0026#34;Port\u0026#34;: 5002 } while True: try: response = requests.put(\u0026#39;http://consul:8500/v1/agent/service/register\u0026#39;, json=payload) if response.status_code == 200: print(\u0026#34;Successfully registered order-service with Consul\u0026#34;) break else: print(f\u0026#34;Failed to register order-service with Consul, status code: {response.status_code}\u0026#34;) except requests.exceptions.RequestException as e: print(f\u0026#34;Error registering order-service with Consul: {e}\u0026#34;) time.sleep(5) if __name__ == \u0026#39;__main__\u0026#39;: print(\u0026#34;Registering order-service with Consul\u0026#34;) register_service() app.run(host=\u0026#39;0.0.0.0\u0026#39;, port=5002) # vim user_service.py import requests from flask import Flask, jsonify import time app = Flask(__name__) @app.route(\u0026#39;/users\u0026#39;) def get_users(): users = [ {\u0026#39;id\u0026#39;: 1, \u0026#39;name\u0026#39;: \u0026#39;Alice\u0026#39;}, {\u0026#39;id\u0026#39;: 2, \u0026#39;name\u0026#39;: \u0026#39;Bob\u0026#39;} ] return jsonify(users) def register_service(): payload = { \u0026#34;ID\u0026#34;: \u0026#34;user-service\u0026#34;, \u0026#34;Name\u0026#34;: \u0026#34;user-service\u0026#34;, \u0026#34;Address\u0026#34;: \u0026#34;user-service\u0026#34;, \u0026#34;Port\u0026#34;: 5001 } while True: try: response = requests.put(\u0026#39;http://consul:8500/v1/agent/service/register\u0026#39;, json=payload) if response.status_code == 200: print(\u0026#34;Successfully registered user-service with Consul\u0026#34;) break else: print(f\u0026#34;Failed to register user-service with Consul, status code: {response.status_code}\u0026#34;) except requests.exceptions.RequestException as e: print(f\u0026#34;Error registering user-service with Consul: {e}\u0026#34;) time.sleep(5) if __name__ == \u0026#39;__main__\u0026#39;: print(\u0026#34;Registering user-service with Consul\u0026#34;) register_service() app.run(host=\u0026#39;0.0.0.0\u0026#39;, port=5001) # update docker-compose.yaml version: \u0026#39;3\u0026#39; services: consul: image: consul:1.15.4 ports: - \u0026#34;8500:8500\u0026#34; user-service: build: context: ./user_service depends_on: - consul environment: - CONSUL_HTTP_ADDR=consul:8500 ports: - \u0026#34;5001:5001\u0026#34; order-service: build: context: ./order_service depends_on: - consul environment: - CONSUL_HTTP_ADDR=consul:8500 ports: - \u0026#34;5002:5002\u0026#34; api-gateway: build: context: ./api_gateway depends_on: - consul - user-service - order-service environment: - CONSUL_HTTP_ADDR=consul:8500 ports: - \u0026#34;5000:5000\u0026#34; Now run the docker-compose and verify in Consul via localhost:\ndocker-compose up --build Creating 04-with-consul_consul_1 ... done Creating 04-with-consul_user-service_1 ... done Creating 04-with-consul_order-service_1 ... done Creating 04-with-consul_api-gateway_1 ... done consul_1 | 2024-05-18T14:28:45.465Z [DEBUG] agent: Node info in sync consul_1 | 2024-05-18T14:28:45.465Z [DEBUG] agent: Service in sync: service=order-service consul_1 | 2024-05-18T14:28:45.465Z [DEBUG] agent: Service in sync: service=user-service Conclusion\nNow we can use API Gateway and Consul to manage routing and service discovery.\nIn the next post, I will see how to enable logging with the ELK stack and monitoring with the Prometheus and Grafana stack.\n==================== ","permalink":"https://zackblog.work/posts/python-microservice-api-gateway-consul/","summary":"\u003cp\u003eAPI Gateway acts as a single entry point for all clients and handles the request routing, composition, and protocol translation in a microservices architecture, here I will create an API Gateway using Python Flask and the requests library, to route both \u0026ldquo;user\u0026rdquo; and \u0026ldquo;order\u0026rdquo; services.\u003c/p\u003e\n\u003cp\u003eHere I will create an API Gateway to handle the 2 services (user and order). By importing the \u003ccode\u003erequests\u003c/code\u003e module, which allows us to send HTTP requests in Python. It\u0026rsquo;s used for making API calls to other services.\u003c/p\u003e","title":"Python Microservice: API Gateway \u0026 Consul"},{"content":"Flask stands out as one of Python’s most popular web frameworks. Designed for versatility and ease of use, Flask offers a robust starting point for crafting web apps.\nBy the following posts, I will use Flask to:\nCreate a series of Python web applications from simple app (Hello Zack) Develop microservices applications (order and user), deploy using Docker compose Create Python API gateway application Integrate with Consul for service discovery and register Enable logging with ELK, monitoring with Prometheus \u0026amp; Grafana Lastly, I will create Kubernetes manifest for K8S deployment I will skip ArgoCD and Github Action as I had done similar posts before.\nGet started with Flask env and simple app\nI will skip the installation for Docker and python3 as those can be found via the official Docker and Python website.\nIn the application Python code, I will import the Flask class from the flask module. Flask is a micro web framework written in Python. The Flask class is used to create a Flask application instance, which will be used to handle incoming web requests and route them to the appropriate functions in our code.\n# local create a new virtualenv virtualenv flask # pip3 install virtualenv cd flask # activate the virtualenv source bin/activate # install flask pip install flask # create a simple Flask app vim app.py from flask import Flask # create an instance of the Flask class and store it in the app variable, The __name__ variable is passed as an argument to the Flask constructor. This helps Flask determine the root path for the application, which is useful for locating resources, templates, and static files app = Flask(__name__) # This is a decorator that Flask provides. A decorator is a way to modify the behavior of a function or method. In this case, @app.route(\u0026#39;/\u0026#39;) tells Flask to execute the following function (in this case, hello_zack) when the root URL (\u0026#39;/\u0026#39;) of the web application is accessed @app.route(\u0026#39;/\u0026#39;) # This line defines a function named hello_zack. This function will be called when a request to the root URL (\u0026#39;/\u0026#39;) is made. def hello_zack(): # This line returns the string \u0026#39;Hello, Zack!\u0026#39; as the response to the web request. When someone accesses the root URL, they will see this message in their web browser. return \u0026#39;Hello, Zack!\u0026#39; # This is a common Python idiom that checks whether the script is being run directly or being imported as a module in another script if __name__ == \u0026#39;__main__\u0026#39;: # This line starts the Flask development server. The host=\u0026#39;0.0.0.0\u0026#39; argument makes the server accessible from any network interface, with the default port for Flask is 5000 app.run(host=\u0026#39;0.0.0.0\u0026#39;, port=5000) # run it (flask) root@ubt-server:~/zack-gitops-project/Python-flask/01-single-app# python3 app.py * Serving Flask app \u0026#39;app\u0026#39; * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:5000 * Running on http://11.0.1.199:5000 Press CTRL+C to quit 11.0.1.1 - - [18/Jul/2024 13:50:41] \u0026#34;GET / HTTP/1.1\u0026#34; 200 - Create \u0026ldquo;user\u0026rdquo; and \u0026ldquo;order\u0026rdquo; flask microservice apps\nNow I will move to create 2 microservice applications with Flask.\nIn design, I will import the Flask class and jsonify function to create a Flask application instance, define a route (/orders) and a corresponding function (get_orders) to handle requests to that route, then return a JSON response with order details. Same design for user.py as well.\n# order_service.py from flask import Flask, jsonify app = Flask(__name__) @app.route(\u0026#39;/orders\u0026#39;) def get_orders(): orders = [ {\u0026#39;id\u0026#39;: 1, \u0026#39;item\u0026#39;: \u0026#39;Laptop\u0026#39;, \u0026#39;price\u0026#39;: 1200}, {\u0026#39;id\u0026#39;: 2, \u0026#39;item\u0026#39;: \u0026#39;Phone\u0026#39;, \u0026#39;price\u0026#39;: 800} ] return jsonify(orders) if __name__ == \u0026#39;__main__\u0026#39;: app.run(host=\u0026#39;0.0.0.0\u0026#39;, port=5002) # user_service.py from flask import Flask, jsonify app = Flask(__name__) @app.route(\u0026#39;/users\u0026#39;) def get_users(): users = [ {\u0026#39;id\u0026#39;: 1, \u0026#39;name\u0026#39;: \u0026#39;Alice\u0026#39;}, {\u0026#39;id\u0026#39;: 2, \u0026#39;name\u0026#39;: \u0026#39;Bob\u0026#39;} ] return jsonify(users) if __name__ == \u0026#39;__main__\u0026#39;: app.run(host=\u0026#39;0.0.0.0\u0026#39;, port=5001) Create Dockerfile for containerization\nBelow is the Dockerfile for creating the container for each service:\n# Dockerfile-order # Use an official Python runtime as a parent image FROM python:3.9-slim # Set the working directory in the container WORKDIR /app # Copy the current directory contents into the container at /app COPY order_service.py /app # Install any needed packages specified in requirements.txt RUN pip install --no-cache-dir flask # Make port 5002 available to the world outside this container EXPOSE 5002 # Define environment variable ENV FLASK_APP=order_service.py # Run order_service.py when the container launches CMD [\u0026#34;flask\u0026#34;, \u0026#34;run\u0026#34;, \u0026#34;--host=0.0.0.0\u0026#34;, \u0026#34;--port=5002\u0026#34;] # Dockerfile-user # Use an official Python runtime as a parent image FROM python:3.9-slim # Set the working directory in the container WORKDIR /app # Copy the current directory contents into the container at /app COPY user_service.py /app # Install any needed packages specified in requirements.txt RUN pip install --no-cache-dir flask # Make port 5001 available to the world outside this container EXPOSE 5001 # Define environment variable ENV FLASK_APP=user_service.py # Run order_service.py when the container launches CMD [\u0026#34;flask\u0026#34;, \u0026#34;run\u0026#34;, \u0026#34;--host=0.0.0.0\u0026#34;, \u0026#34;--port=5001\u0026#34;] Build, tag and run it using Docker Compose\n# docker-compose.yml version: \u0026#39;3\u0026#39; services: user-service: build: context: . dockerfile: Dockerfile_user ports: - \u0026#34;5001:5001\u0026#34; order-service: build: context: . dockerfile: Dockerfile_order ports: - \u0026#34;5002:5002\u0026#34; docker-compose up --build Creating 02-microservice-docker-compose_user-service_1 ... done Creating 02-microservice-docker-compose_order-service_1 ... done Attaching to 02-microservice-docker-compose_order-service_1, 02-microservice-docker-compose_user-service_1 order-service_1 | * Serving Flask app \u0026#39;order_service.py\u0026#39; order-service_1 | * Debug mode: off order-service_1 | WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. order-service_1 | * Running on all addresses (0.0.0.0) order-service_1 | * Running on http://127.0.0.1:5002 order-service_1 | * Running on http://192.168.16.3:5002 order-service_1 | Press CTRL+C to quit user-service_1 | * Serving Flask app \u0026#39;user_service.py\u0026#39; user-service_1 | * Debug mode: off user-service_1 | WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. user-service_1 | * Running on all addresses (0.0.0.0) user-service_1 | * Running on http://127.0.0.1:5001 user-service_1 | * Running on http://192.168.16.2:5001 user-service_1 | Press CTRL+C to quit Conclusion\nNow we can use Python Flask to create a simple web application and containerize it using Dockerfile. In the next post, I will see how to use API Gateway and Consul for more features to this Python application.\n==================== ","permalink":"https://zackblog.work/posts/python-microservice-containerization/","summary":"\u003cp\u003eFlask stands out as one of Python’s most popular web frameworks. Designed for versatility and ease of use, Flask offers a robust starting point for crafting web apps.\u003c/p\u003e\n\u003cp\u003eBy the following posts, I will use Flask to:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eCreate a series of Python web applications from simple app (Hello Zack)\u003c/li\u003e\n\u003cli\u003eDevelop microservices applications (order and user), deploy using Docker compose\u003c/li\u003e\n\u003cli\u003eCreate Python API gateway application\u003c/li\u003e\n\u003cli\u003eIntegrate with Consul for service discovery and register\u003c/li\u003e\n\u003cli\u003eEnable logging with ELK, monitoring with Prometheus \u0026amp; Grafana\u003c/li\u003e\n\u003cli\u003eLastly, I will create Kubernetes manifest for K8S deployment\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eI will skip ArgoCD and Github Action as I had done similar posts before.\u003c/p\u003e","title":"Python Microservice: Containerization"},{"content":"In the previous post I developed shell script + awscli to apply aws EC2 tags, since the last post we discovered Python Boto3 scripts for AWS resource automation and management, I think it is time to improve the EC2 tagging task with Python and boto3, together with file handling to achieve:\nList and export EC2 information to a CSV file (instanceID, default instance name, Existing tags) Define 4 mandatory tags in CSV header (Env, BizOwner, Technology, Project) Validate exported tags against the 4 mandatory new tags, if any of the new mandatory tags exists, then keep the tag and value, if any of the new mandatory tags do not exist, add the key and leave the value blank Get CSV file filled with mandatory tags input from Biz team (manual work) Open the updated CSV file, apply the mandatory tags based on the input value Create and trigger Lambda function with AWS config rules to enforce 4 mandatory tags whenever a new instance is launched List and export EC2 information to a CSV\nHere we need Python libraries for \u0026ldquo;boto3\u0026rdquo; and \u0026ldquo;csv\u0026rdquo;, to call boto3 sessions to retrieve EC2 information, then use Python \u0026ldquo;with open\u0026rdquo; and \u0026ldquo;for\u0026rdquo; loops to write each EC2 info to a CSV file, also add mandatory tags write in the header fields \u0026ldquo;Env\u0026rdquo;, \u0026ldquo;BizOwner\u0026rdquo;, \u0026ldquo;Technology\u0026rdquo;, \u0026ldquo;Project\u0026rdquo;:\nroot@ubt-server:~/pythonwork/new# vim export1.py # Import libraries import boto3 import csv # Define the mandatory tags MANDATORY_TAGS = [\u0026#34;Env\u0026#34;, \u0026#34;BizOwner\u0026#34;, \u0026#34;Technology\u0026#34;, \u0026#34;Project\u0026#34;] # Initialize boto3 clients ec2 = boto3.client(\u0026#39;ec2\u0026#39;) def list_ec2_instances(): instances = [] response = ec2.describe_instances() for reservation in response[\u0026#39;Reservations\u0026#39;]: for instance in reservation[\u0026#39;Instances\u0026#39;]: instance_id = instance[\u0026#39;InstanceId\u0026#39;] default_name = next((tag[\u0026#39;Value\u0026#39;] for tag in instance.get(\u0026#39;Tags\u0026#39;, []) if tag[\u0026#39;Key\u0026#39;] == \u0026#39;Name\u0026#39;), \u0026#39;No Name\u0026#39;) tags = {tag[\u0026#39;Key\u0026#39;]: tag[\u0026#39;Value\u0026#39;] for tag in instance.get(\u0026#39;Tags\u0026#39;, [])} instance_info = { \u0026#39;InstanceId\u0026#39;: instance_id, \u0026#39;DefaultName\u0026#39;: default_name, **tags } # Ensure mandatory tags are included with empty values if not present for mandatory_tag in MANDATORY_TAGS: if mandatory_tag not in instance_info: instance_info[mandatory_tag] = \u0026#39;\u0026#39; instances.append(instance_info) return instances # Define export to CSV def export_to_csv(instances, filename=\u0026#39;ec2_instances.csv\u0026#39;): # Collect all possible tag keys all_tags = set() for instance in instances: all_tags.update(instance.keys()) # Ensure mandatory tags are included in the header all_tags.update(MANDATORY_TAGS) fieldnames = [\u0026#39;InstanceId\u0026#39;, \u0026#39;DefaultName\u0026#39;] + sorted(all_tags - {\u0026#39;InstanceId\u0026#39;, \u0026#39;DefaultName\u0026#39;}) with open(filename, \u0026#39;w\u0026#39;, newline=\u0026#39;\u0026#39;) as csvfile: writer = csv.DictWriter(csvfile, fieldnames=fieldnames) writer.writeheader() for instance in instances: writer.writerow(instance) def main(): instances = list_ec2_instances() export_to_csv(instances) print(\u0026#34;CSV export complete. Please update the mandatory tags in \u0026#39;ec2_instances.csv\u0026#39;.\u0026#34;) if __name__ == \u0026#39;__main__\u0026#39;: main() root@ubt-server:~/pythonwork/new# python3 export1.py CSV export complete. Please update the mandatory tags in \u0026#39;ec2_instances.csv\u0026#39;. Next, download and update \u0026rsquo;ec2_instances.csv\u0026rsquo; with all required tags, then rename and upload as \u0026rsquo;ec2_instances_updated.csv\u0026rsquo;, create second script \u0026ldquo;update1.py\u0026rdquo; to apply new tags:\nroot@ubt-server:~/pythonwork/new# vim update1.py import boto3 import csv # Define the mandatory tags MANDATORY_TAGS = [\u0026#34;Env\u0026#34;, \u0026#34;BizOwner\u0026#34;, \u0026#34;Technology\u0026#34;, \u0026#34;Project\u0026#34;] def update_tags_from_csv(filename=\u0026#39;ec2_instances_updated.csv\u0026#39;): ec2 = boto3.client(\u0026#39;ec2\u0026#39;) with open(filename, newline=\u0026#39;\u0026#39;) as csvfile: reader = csv.DictReader(csvfile) for row in reader: instance_id = row[\u0026#39;InstanceId\u0026#39;] tags = [{\u0026#39;Key\u0026#39;: tag, \u0026#39;Value\u0026#39;: row[tag]} for tag in MANDATORY_TAGS if row[tag]] if tags: ec2.create_tags(Resources=[instance_id], Tags=tags) def main(): update_tags_from_csv() print(\u0026#34;Tags updated successfully from \u0026#39;ec2_instances_updated.csv\u0026#39;.\u0026#34;) if __name__ == \u0026#39;__main__\u0026#39;: main() root@ubt-server:~/pythonwork/new# python3 update1.py Tags updated successfully from \u0026#39;ec2_instances_updated.csv\u0026#39;. How about managing tags for multiple AWS accounts\nConsidering we have 20+ AWS accounts across the company and with more than 200 EC2 instances that need to apply tagging strategy, here I will:\nUse AWS CLI profile to configure each AWS account creds, here I will use my own 2 AWS accounts (ZackBlog and JoeSite) to create AWS CLI profiles to validate the Python scripts # Add account creds into ~/.aws/credentials vim ~/.aws/credentials [aws_account_zackblog] aws_access_key_id = xxxx aws_secret_access_key = yyyy [aws_account_joesite] aws_access_key_id = zzzz aws_secret_access_key = yyyy # add profiles into ~/.aws/config vim ~/.aws/config [profile aws_account_zackblog] region = ap-southeast-2 [profile aws_account_joesite] region = ap-southeast-2 Now update Python scripts to call each account profile to apply all 20+ AWS accounts in sequence.\nroot@ubt-server:~/pythonwork# mkdir mutiple-aws root@ubt-server:~/pythonwork# cd mutiple-aws/ root@ubt-server:~/pythonwork/mutiple-aws# vim export2.py import boto3 import csv from botocore.exceptions import ProfileNotFound # Define the mandatory tags MANDATORY_TAGS = [\u0026#34;Env\u0026#34;, \u0026#34;BizOwner\u0026#34;, \u0026#34;Technology\u0026#34;, \u0026#34;Project\u0026#34;] # List of AWS account profiles AWS_PROFILES = [\u0026#34;aws_account_zackblog\u0026#34;, \u0026#34;aws_account_joesite\u0026#34;] # Add more profiles as needed def list_ec2_instances(profile_name): session = boto3.Session(profile_name=profile_name) ec2 = session.client(\u0026#39;ec2\u0026#39;) instances = [] response = ec2.describe_instances() for reservation in response[\u0026#39;Reservations\u0026#39;]: for instance in reservation[\u0026#39;Instances\u0026#39;]: instance_id = instance[\u0026#39;InstanceId\u0026#39;] default_name = next((tag[\u0026#39;Value\u0026#39;] for tag in instance.get(\u0026#39;Tags\u0026#39;, []) if tag[\u0026#39;Key\u0026#39;] == \u0026#39;Name\u0026#39;), \u0026#39;No Name\u0026#39;) tags = {tag[\u0026#39;Key\u0026#39;]: tag[\u0026#39;Value\u0026#39;] for tag in instance.get(\u0026#39;Tags\u0026#39;, [])} instance_info = { \u0026#39;InstanceId\u0026#39;: instance_id, \u0026#39;DefaultName\u0026#39;: default_name, **tags } # Ensure mandatory tags are included with empty values if not present for mandatory_tag in MANDATORY_TAGS: if mandatory_tag not in instance_info: instance_info[mandatory_tag] = \u0026#39;\u0026#39; instances.append(instance_info) return instances def export_to_csv(instances, profile_name): filename = f\u0026#34;ec2_instances_{profile_name}.csv\u0026#34; # Collect all possible tag keys all_tags = set() for instance in instances: all_tags.update(instance.keys()) # Ensure mandatory tags are included in the header all_tags.update(MANDATORY_TAGS) fieldnames = [\u0026#39;InstanceId\u0026#39;, \u0026#39;DefaultName\u0026#39;] + sorted(all_tags - {\u0026#39;InstanceId\u0026#39;, \u0026#39;DefaultName\u0026#39;}) with open(filename, \u0026#39;w\u0026#39;, newline=\u0026#39;\u0026#39;) as csvfile: writer = csv.DictWriter(csvfile, fieldnames=fieldnames) writer.writeheader() for instance in instances: writer.writerow(instance) def process_all_profiles(): for profile in AWS_PROFILES: try: print(f\u0026#34;Processing profile: {profile}\u0026#34;) instances = list_ec2_instances(profile) export_to_csv(instances, profile) print(f\u0026#34;CSV export complete for profile {profile}. Please update the mandatory tags in \u0026#39;ec2_instances_{profile}.csv\u0026#39;.\u0026#34;) except ProfileNotFound: print(f\u0026#34;Profile {profile} not found. Skipping.\u0026#34;) if __name__ == \u0026#39;__main__\u0026#39;: process_all_profiles() Export 2 CSV files for each AWS account based on given profile, update mandatory tags in the 2 CSV files, then upload and rename as _updated_:\n# export 2 csv files root@ubt-server:~/pythonwork/mutiple-aws# python3 export2.py Processing profile: aws_account_zackblog CSV export complete for profile aws_account_zackblog. Please update the mandatory tags in \u0026#39;ec2_instances_aws_account_zackblog.csv\u0026#39;. Processing profile: aws_account_joesite CSV export complete for profile aws_account_joesite. Please update the mandatory tags in \u0026#39;ec2_instances_aws_account_joesite.csv\u0026#39;. # update all mandatory tags in the files root@ubt-server:~/pythonwork/mutiple-aws# cat ec2_instances_updated_aws_account_zackblog.csv InstanceId,DefaultName,BizOwner,Env,Name,Project,Technology,Tuned,zz1,zz2 i-076226daa5aaf7cf2,zack-blog,Zack,Prod,zack-blog,zack-web,Jekyll,,aa1,aa2 i-0b5c0fec84073a6d9,Py_test_zackweb,Zack,Testing,Py_test_zackweb,python-test,None,Yes,, root@ubt-server:~/pythonwork/mutiple-aws# cat ec2_instances_updated_aws_account_joesite.csv InstanceId,DefaultName,BizOwner,Env,Location,Name,Project,Technology,TimeLaunched i-012fb886802435ff2,joe-account-py-test,Joe,Prod,SYD,joe-account-py-test,joesite,Ruby-Jekyll, i-052b0511339457efc,joe-site,Joe,Testing,,joe-site,Python-test,None,20240301 Now create python script \u0026ldquo;update_tags_2.py\u0026rdquo; to apply new tags for 2 AWS accounts by given profile:\nimport boto3 import csv import re # Define the mandatory tags MANDATORY_TAGS = [\u0026#34;Tag1\u0026#34;, \u0026#34;Tag2\u0026#34;, \u0026#34;Tag3\u0026#34;, \u0026#34;Tag4\u0026#34;, \u0026#34;Tag5\u0026#34;] # Initialize boto3 session for a given profile def get_boto3_session(profile_name): return boto3.Session(profile_name=profile_name) # Fetch account ID using sts client def get_account_id(session): sts_client = session.client(\u0026#39;sts\u0026#39;) return sts_client.get_caller_identity()[\u0026#39;Account\u0026#39;] # Modified function to fetch and compare tags without applying changes def check_tags_from_csv(session, filename, account_id, region): ec2 = session.client(\u0026#39;ec2\u0026#39;) rds = session.client(\u0026#39;rds\u0026#39;) s3 = session.client(\u0026#39;s3\u0026#39;) lambda_client = session.client(\u0026#39;lambda\u0026#39;) elbv2 = session.client(\u0026#39;elbv2\u0026#39;) efs = session.client(\u0026#39;efs\u0026#39;) ecs = session.client(\u0026#39;ecs\u0026#39;) with open(filename, newline=\u0026#39;\u0026#39;) as csvfile: reader = csv.DictReader(csvfile) for row in reader: identifier = row.get(\u0026#39;Identifier\u0026#39;, \u0026#39;\u0026#39;).strip() service = row.get(\u0026#39;Service\u0026#39;, \u0026#39;\u0026#39;).strip() if not identifier or not service: print(f\u0026#34;Skipping row due to missing \u0026#39;Identifier\u0026#39; or \u0026#39;Service\u0026#39;: {row}\u0026#34;) continue # Fetch current tags and identify missing mandatory tags current_tags = fetch_existing_tags(service, identifier, session, account_id, region) missing_tags = [tag for tag in MANDATORY_TAGS if tag not in [t[\u0026#39;Key\u0026#39;] for t in current_tags]] # Output resources missing mandatory tags if missing_tags: print(f\u0026#34;Resource: {identifier}, Service: {service}\u0026#34;) print(f\u0026#34;Current Tags: {current_tags}\u0026#34;) print(f\u0026#34;Missing Mandatory Tags: {missing_tags}\\n\u0026#34;) # Function to fetch existing tags for a given resource def fetch_existing_tags(service, identifier, session, account_id, region): try: if service == \u0026#39;EC2\u0026#39;: ec2 = session.client(\u0026#39;ec2\u0026#39;) response = ec2.describe_tags(Filters=[{\u0026#39;Name\u0026#39;: \u0026#39;resource-id\u0026#39;, \u0026#39;Values\u0026#39;: [identifier]}]) return [{\u0026#39;Key\u0026#39;: tag[\u0026#39;Key\u0026#39;], \u0026#39;Value\u0026#39;: tag[\u0026#39;Value\u0026#39;]} for tag in response[\u0026#39;Tags\u0026#39;]] elif service == \u0026#39;RDS\u0026#39;: rds = session.client(\u0026#39;rds\u0026#39;) arn = f\u0026#39;arn:aws:rds:{region}:{account_id}:db:{identifier}\u0026#39; response = rds.list_tags_for_resource(ResourceName=arn) return response.get(\u0026#39;TagList\u0026#39;, []) elif service == \u0026#39;RDScluster\u0026#39;: rds = session.client(\u0026#39;rds\u0026#39;) arn = f\u0026#39;arn:aws:rds:{region}:{account_id}:cluster:{identifier}\u0026#39; response = rds.list_tags_for_resource(ResourceName=arn) return response.get(\u0026#39;TagList\u0026#39;, []) elif service == \u0026#39;S3\u0026#39;: s3 = session.client(\u0026#39;s3\u0026#39;) response = s3.get_bucket_tagging(Bucket=identifier) return response.get(\u0026#39;TagSet\u0026#39;, []) elif service == \u0026#39;Lambda\u0026#39;: lambda_client = session.client(\u0026#39;lambda\u0026#39;) arn = f\u0026#39;arn:aws:lambda:{region}:{account_id}:function:{identifier}\u0026#39; response = lambda_client.list_tags(Resource=arn) return [{\u0026#39;Key\u0026#39;: k, \u0026#39;Value\u0026#39;: v} for k, v in response.get(\u0026#39;Tags\u0026#39;, {}).items()] elif service == \u0026#39;ElasticLoadBalancingV2\u0026#39;: elbv2 = session.client(\u0026#39;elbv2\u0026#39;) arn = get_elbv2_arn_by_arn(elbv2, identifier) response = elbv2.describe_tags(ResourceArns=[arn]) return response[\u0026#39;TagDescriptions\u0026#39;][0][\u0026#39;Tags\u0026#39;] if response[\u0026#39;TagDescriptions\u0026#39;] else [] elif service == \u0026#39;EFS\u0026#39;: efs = session.client(\u0026#39;efs\u0026#39;) response = efs.describe_tags(FileSystemId=identifier) return response.get(\u0026#39;Tags\u0026#39;, []) elif service == \u0026#39;ECS\u0026#39;: ecs = session.client(\u0026#39;ecs\u0026#39;) arn = f\u0026#39;arn:aws:ecs:{region}:{account_id}:service/{identifier}\u0026#39; response = ecs.list_tags_for_resource(resourceArn=arn) return response.get(\u0026#39;tags\u0026#39;, []) else: print(f\u0026#34;Unsupported service type {service}, cannot fetch tags.\u0026#34;) return [] except Exception as e: print(f\u0026#34;Error fetching tags for {service} {identifier}: {e}\u0026#34;) return [] def get_elbv2_arn_by_arn(elbv2_client, load_balancer_partial_name): \u0026#34;\u0026#34;\u0026#34;Retrieve the ARN for a given ALB by partial name.\u0026#34;\u0026#34;\u0026#34; try: response = elbv2_client.describe_load_balancers() for lb in response[\u0026#39;LoadBalancers\u0026#39;]: if load_balancer_partial_name in lb[\u0026#39;LoadBalancerArn\u0026#39;] or load_balancer_partial_name in lb[\u0026#39;LoadBalancerName\u0026#39;]: return lb[\u0026#39;LoadBalancerArn\u0026#39;] print(f\u0026#34;No matching ALB found for {load_balancer_partial_name}\u0026#34;) return None except Exception as e: print(f\u0026#34;Error retrieving ARN for ALB {load_balancer_partial_name}: {e}\u0026#34;) return None if __name__ == \u0026#39;__main__\u0026#39;: # Define the CSV files and AWS profiles csv_files = { \u0026#39;account1\u0026#39;: \u0026#39;account1.csv\u0026#39;, # AWS profile and CSV for account 1 resources \u0026#39;account2\u0026#39;: \u0026#39;account2.csv\u0026#39;, # AWS profile and CSV for account 2 resources #\u0026#39;mst\u0026#39;: \u0026#39;mst.csv\u0026#39;, # AWS profile and CSV for mst account resources #\u0026#39;itbs\u0026#39;: \u0026#39;itbs.csv\u0026#39;, # AWS profile and CSV for itbs account resources } # Process each profile and its corresponding CSV for profile, csv_file in csv_files.items(): print(f\u0026#34;Processing tags for AWS profile: {profile}\u0026#34;) # Get the session for the current profile session = get_boto3_session(profile) # Fetch the account ID and region for the session account_id = get_account_id(session) region = session.region_name or \u0026#39;us-east-1\u0026#39; # Set default region if not found print(f\u0026#34;Using account ID: {account_id}, Region: {region}\u0026#34;) # Check tags from the CSV without applying changes check_tags_from_csv(session, csv_file, account_id, region) print(\u0026#34;Tag comparison operation completed for all profiles.\u0026#34;) Run the script to apply tags:\nroot@ubt-server:~/pythonwork/mutiple-aws# python3 update_tags_2.py Processing profile: aws_account_zackblog Tags updated successfully from \u0026#39;ec2_instances_updated_aws_account_zackblog.csv\u0026#39; for profile aws_account_zackblog. Processing profile: aws_account_joesite Tags updated successfully from \u0026#39;ec2_instances_updated_aws_account_joesite.csv\u0026#39; for profile aws_account_joesite. Now double-check the tags for both accounts:\nConclusion\nNow we can use Python Boto3 and file handling to achieve multiple-aws account EC2 tagging. With Python \u0026ldquo;csv\u0026rdquo; library, functions like \u0026ldquo;csv.DictReader\u0026rdquo;, \u0026ldquo;with open\u0026rdquo; and \u0026ldquo;csv.DictWriter\u0026rdquo; to open, update and export CSV file, Python also supports handling data in JSON format with dictionary.\nIn the next post I will see how to use Python Flask to redesign Zack\u0026rsquo;s blog for Web application development.\n==================== ","permalink":"https://zackblog.work/posts/python-file-handling-for-aws-tagging/","summary":"\u003cp\u003eIn the previous post I developed shell script + awscli to apply aws EC2 tags, since the last post we discovered \u003ca href=\"/posts/python-boto3-for-aws/\"\u003ePython Boto3\u003c/a\u003e scripts for AWS resource automation and management, I think it is time to improve the EC2 tagging task with Python and boto3, together with file handling to achieve:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003eList and export EC2 information to a CSV file (instanceID, default instance name, Existing tags)\u003c/li\u003e\n\u003cli\u003eDefine 4 mandatory tags in CSV header (Env, BizOwner, Technology, Project)\u003c/li\u003e\n\u003cli\u003eValidate exported tags against the 4 mandatory new tags, if any of the new mandatory tags exists, then keep the tag and value, if any of the new mandatory tags do not exist, add the key and leave the value blank\u003c/li\u003e\n\u003cli\u003eGet CSV file filled with mandatory tags input from Biz team (manual work)\u003c/li\u003e\n\u003cli\u003eOpen the updated CSV file, apply the mandatory tags based on the input value\u003c/li\u003e\n\u003cli\u003eCreate and trigger Lambda function with AWS config rules to enforce 4 mandatory tags whenever a new instance is launched\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003e\u003cstrong\u003eList and export EC2 information to a CSV\u003c/strong\u003e\u003c/p\u003e","title":"Python: File Handling for AWS tagging"},{"content":"Boto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python. It enables developers to build software that uses Amazon services like EC2, S3, RDS, etc.\nI will build a portable Python3.9 + Boto3 Docker environment to test some AWS automation tasks.\nBuild and run a docker with Python3.9 + Boto3\nAs I do not want to install Python, Boto3, and AWScli on my local PC, creating a Docker image with all software ready as a portable env is the best way to start.\nroot@ubt-server:~# vim Dockerfile # Build from python:3.9.19-alpine3.19 From python:3.9.19-alpine3.19 # install boto3 and awscli RUN pip install --upgrade pip \u0026amp;\u0026amp; \\ pip install --upgrade awscli \u0026amp;\u0026amp; \\ pip install --upgrade boto3 # set work dir WORKDIR /work # run Python CMD \u0026#34;python\u0026#34; # build a docker image from the above Dockerfile root@ubt-server:~# docker image build -t zack_aws_boto3 . # ls docker images root@ubt-server:~# docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE zack_aws_boto3 v1 07a13f7801ed 1 days ago 998MB zackpy latest 287ba6873741 4 days ago 48.2MB zackz001/gitops-jekyll latest d92894f7be6d 6 days ago 70.9MB postgres 15.0 027eba2e8939 19 months ago 377MB # run docker and mount local python work dir root@ubt-server:~/pythonwork# docker run -ti -v ${PWD}:/work zack_aws_boto3:v1 bash root@c04670a43564:/# root@c04670a43564:/# cd work \u0026amp;\u0026amp; ls # configure aws in the container root@c04670a43564:/work# aws configure AWS Access Key ID [****************GFNW]: AWS Secret Access Key [****************Db7O]: Default region name [ap-southeast-2]: Default output format [None]: # validate aws cred by listing ec2 instance id root@c04670a43564:/work# aws ec2 describe-instances --query \u0026#34;Reservations[*].Instances[*].InstanceId\u0026#34; --output json [ [ \u0026#34;i-076226daa5aaf7cf2\u0026#34; ] ] Manage AWS resource with Python Boto3 script\nHere we have Python and boto3 env ready; I will list some AWS tasks that I want to be achieved by Python scripts.\nList EC2 instance name, instanceID, and state root@ubt-server:~/pythonwork# vim app.py # import boto3 library import boto3 def list_ec2_instances(): # Create a session using default AWS profile session = boto3.Session() # Create an EC2 client ec2_client = session.client(\u0026#39;ec2\u0026#39;) # Describe EC2 instances response = ec2_client.describe_instances() # Iterate over the instances for reservation in response[\u0026#39;Reservations\u0026#39;]: for instance in reservation[\u0026#39;Instances\u0026#39;]: # Get the instance ID instance_id = instance[\u0026#39;InstanceId\u0026#39;] # Get the instance state instance_state = instance[\u0026#39;State\u0026#39;][\u0026#39;Name\u0026#39;] # Get the instance Name tag if exists instance_name = \u0026#39;No Name\u0026#39; if \u0026#39;Tags\u0026#39; in instance: for tag in instance[\u0026#39;Tags\u0026#39;]: if tag[\u0026#39;Key\u0026#39;] == \u0026#39;Name\u0026#39;: instance_name = tag[\u0026#39;Value\u0026#39;] break # Print instance ID, Name, and State print(f\u0026#34;Instance ID: {instance_id}, Name: {instance_name}, State: {instance_state}\u0026#34;) if __name__ == \u0026#34;__main__\u0026#34;: list_ec2_instances() root@c04670a43564:/work# python app.py Instance ID: i-076226daa5aaf7cf2, Name: zack-blog, State: stopped Filter EC2 instance without tag \u0026ldquo;owner\u0026rdquo; # create app-untagged.py root@ubt-server:~/pythonwork# vim app-untagged.py import boto3 def get_untagged_ec2_instances(): ec2_client = boto3.client(\u0026#39;ec2\u0026#39;) response = ec2_client.describe_instances() untagged_instances = [] for reservation in response[\u0026#39;Reservations\u0026#39;]: for instance in reservation[\u0026#39;Instances\u0026#39;]: has_owner_tag = False if \u0026#39;Tags\u0026#39; in instance: for tag in instance[\u0026#39;Tags\u0026#39;]: if tag[\u0026#39;Key\u0026#39;].lower() == \u0026#39;owner\u0026#39;: has_owner_tag = True break if not has_owner_tag: instance_id = instance[\u0026#39;InstanceId\u0026#39;] instance_state = instance[\u0026#39;State\u0026#39;][\u0026#39;Name\u0026#39;] untagged_instances.append({\u0026#39;InstanceId\u0026#39;: instance_id, \u0026#39;State\u0026#39;: instance_state}) return untagged_instances untagged_instances = get_untagged_ec2_instances() print(\u0026#34;Untagged Instances:\u0026#34;, untagged_instances) # run script to filter untagged \u0026#34;owner\u0026#34; ec2 root@c04670a43564:/work# python app-untagged.py Untagged Instances: [{\u0026#39;InstanceId\u0026#39;: \u0026#39;i-076226daa5aaf7cf2\u0026#39;, \u0026#39;State\u0026#39;: \u0026#39;stopped\u0026#39;}] Create lambda function to list EBS volume snapshots older than 30 days and delete them To achieve this we need:\nCreate lambda IAM role for lambda to manage EBS volume snapshot Create below Python lambda function Zip and upload zip function Create CloudWatch Event to Trigger and run it every 30 days # create lambda function to delete snapshots older than 30 days root@ubt-server:~/pythonwork# vim app-snapshot-older-30days.py import boto3 from datetime import datetime, timezone, timedelta def lambda_handler(event, context): ec2_client = boto3.client(\u0026#39;ec2\u0026#39;) # Get the current time now = datetime.now(timezone.utc) # Define the time threshold time_threshold = now - timedelta(days=30) # Describe snapshots snapshots = ec2_client.describe_snapshots(OwnerIds=[\u0026#39;self\u0026#39;])[\u0026#39;Snapshots\u0026#39;] # Filter snapshots older than 30 days old_snapshots = [snap for snap in snapshots if snap[\u0026#39;StartTime\u0026#39;] \u0026lt; time_threshold] # Delete old snapshots for snapshot in old_snapshots: snapshot_id = snapshot[\u0026#39;SnapshotId\u0026#39;] ec2_client.delete_snapshot(SnapshotId=snapshot_id) print(f\u0026#34;Deleted snapshot: {snapshot_id}\u0026#34;) return { \u0026#39;statusCode\u0026#39;: 200, \u0026#39;body\u0026#39;: f\u0026#34;Deleted {len(old_snapshots)} snapshots.\u0026#34; } # zip Package for the Lambda Function root@ubt-server:~/pythonwork# zip function.zip app-snapshot-older-30days.py Email me when a security group allows inbound SSH (port 22) from everywhere (0.0.0.0/0) To achieve this, we need:\nAWS CloudTrail enable Create CloudWatch Event Rule to capture AWS CloudTrail logs for security group changes Create below Lambda Function if inbound allows port 22 from everywhere are met Allow CloudWatch Events to Invoke the Lambda Function Add the Lambda Function as a Target for the CloudWatch Event Rule Conclusion\nThere are many ways to automate AWS tasks using Python Boto3 script. Together with Lambda and trigger, many resource tasks can be scheduled and managed in a scripted way.\n","permalink":"https://zackblog.work/posts/python-boto3-for-aws/","summary":"\u003cp\u003eBoto3 is the Amazon Web Services (AWS) Software Development Kit (SDK) for Python. It enables developers to build software that uses Amazon services like EC2, S3, RDS, etc.\u003c/p\u003e\n\u003cp\u003eI will build a portable Python3.9 + Boto3 Docker environment to test some AWS automation tasks.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBuild and run a docker with Python3.9 + Boto3\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAs I do not want to install Python, Boto3, and AWScli on my local PC, creating a Docker image with all software ready as a portable env is the best way to start.\u003c/p\u003e","title":"Python: Boto3 for AWS"},{"content":"In the company, we use AWS SSO for user authentication; when a new user is created from Azure AD, it will automatically synced by AWS IAM Identity Center and Azure AD integration, and then our team will need to handle the SSO user assignment to put them in the required AWS accounts with requested permission sets, it became a pain when such requests coming more frequently and every time an individual user or a whole team with different accounts and permission requirements need to be fulfilled, so how to handle this efficiently become my recent topic.\nSo far, I have tried shell script and Python script to read the user name, AWS account ID, and permission sets ARN from a CSV file, then complete the task with AWScli. However it is not smart enough when the request or scenario changes. I have to adjust the script every time.\nManaging individual request via Terraform\nLet\u0026rsquo;s start with handling user assignment individually with terraform first. Here I have 2 requests:\nA user \u0026ldquo;user1@company.com\u0026rdquo;, under a group called \u0026ldquo;AD-RDS-READ-ONLY\u0026rdquo; in AWS IAM Identity Center, I need to create a permission set in AWS account \u0026ldquo;123456789\u0026rdquo;, and assign to this user. The second request is from the security team, we have 3 security team members (security1@company.com, security2@company.com, security3@company.com), under a group called \u0026ldquo;AD-ACM-FULL-ACCESS\u0026rdquo; in AWS IAM Identity Center, they all need full access for AWS certificate manager access, for all of our 3 AWS accounts (12345678901, 12345678902, and 12345678903). vim main.tf # For Request 1 for RDS read-only access for 1 user in 1 AWS account # for AWS in ap-southeast-2 provider \u0026#34;aws\u0026#34; { region = \u0026#34;ap-southeast-2\u0026#34; } # Define the AWS SSO Instance ARN data \u0026#34;aws_ssoadmin_instances\u0026#34; \u0026#34;main\u0026#34; {} resource \u0026#34;aws_ssoadmin_permission_set\u0026#34; \u0026#34;request1\u0026#34; { instance_arn = data.aws_ssoadmin_instances.main.arns[0] name = \u0026#34;RDS-ReadOnly\u0026#34; description = \u0026#34;Read-only access to RDS resources\u0026#34; session_duration = \u0026#34;PT1H\u0026#34; # Add the policies you need for this permission set managed_policies = [ \u0026#34;arn:aws:iam::aws:policy/AmazonRDSReadOnlyAccess\u0026#34;, ] } resource \u0026#34;aws_ssoadmin_account_assignment\u0026#34; \u0026#34;request1\u0026#34; { instance_arn = data.aws_ssoadmin_instances.main.arns[0] permission_set_arn = aws_ssoadmin_permission_set.request1.arn principal_id = \u0026#34;user1@company.com\u0026#34; principal_type = \u0026#34;USER\u0026#34; target_id = \u0026#34;123456789\u0026#34; # Replace with your AWS Account ID target_type = \u0026#34;AWS_ACCOUNT\u0026#34; } # Ensure the user is part of the required group data \u0026#34;aws_identitystore_group\u0026#34; \u0026#34;request1\u0026#34; { identity_store_id = data.aws_ssoadmin_instances.main.identity_store_id display_name = \u0026#34;AD-RDS-READ-ONLY\u0026#34; } resource \u0026#34;aws_ssoadmin_group_membership\u0026#34; \u0026#34;request1\u0026#34; { identity_store_id = data.aws_ssoadmin_instances.main.identity_store_id group_id = data.aws_identitystore_group.request1.group_id user_ids = [\u0026#34;user1@company.com\u0026#34;] } # handle request 2 for ACM full access for whole security team in all 3 AWS accounts vim main.tf # use existing main.tf file # provider \u0026#34;aws\u0026#34; { # region = \u0026#34;ap-southeast-2\u0026#34; # } # data \u0026#34;aws_ssoadmin_instances\u0026#34; \u0026#34;main\u0026#34; {} # Create the Permission Set for ACM Full Access resource \u0026#34;aws_ssoadmin_permission_set\u0026#34; \u0026#34;acm_full_access\u0026#34; { instance_arn = data.aws_ssoadmin_instances.main.arns[0] name = \u0026#34;ACM-FullAccess\u0026#34; description = \u0026#34;Full access to AWS Certificate Manager\u0026#34; session_duration = \u0026#34;PT1H\u0026#34; managed_policies = [ \u0026#34;arn:aws:iam::aws:policy/AWSCertificateManagerFullAccess\u0026#34;, ] } # List of AWS account IDs # here we use terraform \u0026#34;locals\u0026#34;, \u0026#34;dynamic\u0026#34; and \u0026#34;for_each\u0026#34; to loop SSO user assignment for security team within all AWS accounts locals { aws_account_ids = [\u0026#34;12345678901\u0026#34;, \u0026#34;12345678902\u0026#34;, \u0026#34;12345678903\u0026#34;] } # Security team members locals { security_team_members = [\u0026#34;security1@company.com\u0026#34;, \u0026#34;security2@company.com\u0026#34;, \u0026#34;security3@company.com\u0026#34;] } # Ensure the users are part of the required group data \u0026#34;aws_identitystore_group\u0026#34; \u0026#34;acm_full_access_group\u0026#34; { identity_store_id = data.aws_ssoadmin_instances.main.identity_store_id display_name = \u0026#34;AD-ACM-FULL-ACCESS\u0026#34; } resource \u0026#34;aws_ssoadmin_group_membership\u0026#34; \u0026#34;acm_full_access_membership\u0026#34; { identity_store_id = data.aws_ssoadmin_instances.main.identity_store_id group_id = data.aws_identitystore_group.acm_full_access_group.group_id user_ids = local.security_team_members } # Assign the Permission Set to Each User for Each Account resource \u0026#34;aws_ssoadmin_account_assignment\u0026#34; \u0026#34;acm_full_access_assignments\u0026#34; { for_each = { for acc_id in local.aws_account_ids : acc_id =\u0026gt; acc_id } instance_arn = data.aws_ssoadmin_instances.main.arns[0] permission_set_arn = aws_ssoadmin_permission_set.acm_full_access.arn principal_type = \u0026#34;USER\u0026#34; target_type = \u0026#34;AWS_ACCOUNT\u0026#34; dynamic \u0026#34;assignment\u0026#34; { for_each = local.security_team_members content { principal_id = assignment.value target_id = each.key } } } Terraform Modularity\nHow about the Terraform module, as I will get different user assignment requests with different permission sets and AWS accounts? I guess a Terraform module for SSO user assignment is the best way to make the Terraform code more clean and reusable. There are many benefits to infrastructure as code with modularity. It can reduce code duplication, is easy to update, and has a clear code structure, which fits my AWS SSO user assignment task and challenge perfectly.\nTo achieve this, I will need to create a folder called \u0026ldquo;sso_user_assignment_module\u0026rdquo;, inside the folder it will contain:\nA \u0026ldquo;main.tf\u0026rdquo; file to define the resources for creating permission sets and assigning them to users\n# modules/sso_account_assignment/main.tf provider \u0026#34;aws\u0026#34; { region = var.aws_region } data \u0026#34;aws_ssoadmin_instances\u0026#34; \u0026#34;main\u0026#34; {} resource \u0026#34;aws_ssoadmin_permission_set\u0026#34; \u0026#34;this\u0026#34; { for_each = var.permission_sets instance_arn = data.aws_ssoadmin_instances.main.arns[0] name = each.key description = each.value.description session_duration = each.value.session_duration managed_policies = each.value.managed_policies } resource \u0026#34;aws_ssoadmin_account_assignment\u0026#34; \u0026#34;this\u0026#34; { for_each = { for ps_key, ps_value in var.permission_sets : ps_key =\u0026gt; ps_value.accounts } instance_arn = data.aws_ssoadmin_instances.main.arns[0] permission_set_arn = aws_ssoadmin_permission_set.this[each.key].arn principal_type = \u0026#34;USER\u0026#34; target_type = \u0026#34;AWS_ACCOUNT\u0026#34; dynamic \u0026#34;assignment\u0026#34; { for_each = each.value.users content { principal_id = assignment.value target_id = each.value.account_id } } } data \u0026#34;aws_identitystore_group\u0026#34; \u0026#34;this\u0026#34; { for_each = var.groups identity_store_id = data.aws_ssoadmin_instances.main.identity_store_id display_name = each.key } resource \u0026#34;aws_ssoadmin_group_membership\u0026#34; \u0026#34;this\u0026#34; { for_each = var.groups identity_store_id = data.aws_ssoadmin_instances.main.identity_store_id group_id = data.aws_identitystore_group.this[each.key].group_id user_ids = each.value } A \u0026ldquo;variables.tf\u0026rdquo; to define the input variables for the module\n# variables.tf vim variables.tf provider \u0026#34;aws\u0026#34; { region = var.region } variable \u0026#34;sso_instance_arn\u0026#34; { description = \u0026#34;The ARN of the AWS SSO instance\u0026#34; type = string } variable \u0026#34;assignments\u0026#34; { description = \u0026#34;Map of account IDs to users and their permission sets\u0026#34; type = map(list(object({ principal_id = string permission_set_arn = string }))) } variable \u0026#34;region\u0026#34; { description = \u0026#34;AWS region\u0026#34; type = string default = \u0026#34;ap-southeast-2\u0026#34; } module \u0026#34;sso_account_assignments\u0026#34; { source = \u0026#34;./modules/sso_account_assignment\u0026#34; for_each = var.assignments sso_instance_arn = var.sso_instance_arn account_id = each.key users = each.value } A \u0026ldquo;outputs.tf\u0026rdquo; file to define the outputs of the module.\nvim outputs.tf # define outputs of permission_set_arns and group_ids output \u0026#34;permission_set_arns\u0026#34; { value = { for k, v in aws_ssoadmin_permission_set.this : k =\u0026gt; v.arn } } output \u0026#34;group_ids\u0026#34; { value = { for k, v in data.aws_identitystore_group.this : k =\u0026gt; v.group_id } } Now we need to create a Terraform configuration that uses this module and set the environment variables accordingly. Go back to the root folder, create a root main.tf file to call the module and pass the necessary variables.\ncd .. vim main.tf # the root main.tf file module \u0026#34;sso_permission_sets\u0026#34; { source = \u0026#34;./modules/aws_sso_permission_sets\u0026#34; aws_region = var.aws_region permission_sets = var.permission_sets groups = var.groups } # Optionally output the values output \u0026#34;permission_set_arns\u0026#34; { value = module.sso_permission_sets.permission_set_arns } output \u0026#34;group_ids\u0026#34; { value = module.sso_permission_sets.group_ids } root \u0026ldquo;variables.tf\u0026rdquo; file to define the input variables for the root configuration.\n# the root variables.tf vim variables.tf variable \u0026#34;aws_region\u0026#34; { description = \u0026#34;The AWS region to use.\u0026#34; type = string default = \u0026#34;ap-southeast-2\u0026#34; } variable \u0026#34;permission_sets\u0026#34; { description = \u0026#34;A map of permission sets with their configurations.\u0026#34; type = map(object({ description = string session_duration = string managed_policies = list(string) accounts = map(object({ account_id = string users = list(string) })) })) } variable \u0026#34;groups\u0026#34; { description = \u0026#34;A map of groups with their associated user emails.\u0026#34; type = map(list(string)) } now is the place we can reuse the module to create the root \u0026ldquo;terraform.tfvars\u0026rdquo; which provides the actual values for the variables to define each assignment request. In future we only set each request here as environment variables, and then apply the terraform module.\naws_region = \u0026#34;ap-southeast-2\u0026#34; permission_sets = { # The 1st request RDS read-only permission sets and user assignment redefine in the module using variables \u0026#34;RDS-ReadOnly\u0026#34; = { description = \u0026#34;Read-only access to RDS resources\u0026#34; session_duration = \u0026#34;PT1H\u0026#34; managed_policies = [ \u0026#34;arn:aws:iam::aws:policy/AmazonRDSReadOnlyAccess\u0026#34;, ] accounts = { \u0026#34;123456789\u0026#34; = { account_id = \u0026#34;123456789\u0026#34; users = [\u0026#34;user1@company.com\u0026#34;] } } } # the 2nd request security full access for ACM redefine in the module using variables \u0026#34;ACM-FullAccess\u0026#34; = { description = \u0026#34;Full access to AWS Certificate Manager\u0026#34; session_duration = \u0026#34;PT1H\u0026#34; managed_policies = [ \u0026#34;arn:aws:iam::aws:policy/AWSCertificateManagerFullAccess\u0026#34;, ] accounts = { \u0026#34;12345678901\u0026#34; = { account_id = \u0026#34;12345678901\u0026#34; users = [\u0026#34;security1@company.com\u0026#34;, \u0026#34;security2@company.com\u0026#34;, \u0026#34;security3@company.com\u0026#34;] }, \u0026#34;12345678902\u0026#34; = { account_id = \u0026#34;12345678902\u0026#34; users = [\u0026#34;security1@company.com\u0026#34;, \u0026#34;security2@company.com\u0026#34;, \u0026#34;security3@company.com\u0026#34;] }, \u0026#34;12345678903\u0026#34; = { account_id = \u0026#34;12345678903\u0026#34; users = [\u0026#34;security1@company.com\u0026#34;, \u0026#34;security2@company.com\u0026#34;, \u0026#34;security3@company.com\u0026#34;] } } } # Add 3rd request a developer needs S3 full access for 2 AWS accounts redefine in the module using variables \u0026#34;S3-ModifyAccess\u0026#34; = { description = \u0026#34;Modify access to S3 buckets\u0026#34; session_duration = \u0026#34;PT1H\u0026#34; managed_policies = [ \u0026#34;arn:aws:iam::aws:policy/AmazonS3FullAccess\u0026#34;, ] accounts = { \u0026#34;12345678902\u0026#34; = { account_id = \u0026#34;12345678902\u0026#34; users = [\u0026#34;developer1@company.com\u0026#34;] }, \u0026#34;12345678903\u0026#34; = { account_id = \u0026#34;12345678903\u0026#34; users = [\u0026#34;developer1@company.com\u0026#34;] } } } } groups = { \u0026#34;AD-RDS-EAD-ONLY\u0026#34; = [\u0026#34;user1@company.com\u0026#34;] \u0026#34;AD-ACM-FULL-ACCESS\u0026#34; = [\u0026#34;security1@company.com\u0026#34;, \u0026#34;security2@company.com\u0026#34;, \u0026#34;security3@company.com\u0026#34;] # Optionally add a group for the developer, if needed: # \u0026#34;AD-S3-Modify-Access\u0026#34; = [\u0026#34;developer1@company.com\u0026#34;] } Conclusion\nNow, we can achieve the task individually via terraform code and a Terraform module to handle the creation of AWS SSO users. This setup combines all three requests into a single Terraform configuration, leveraging the reusable module for creating permission sets and managing user assignments, it is more efficient, dynamic, and reusable. In future, we only define permission sets and maintain new users and assignments in the environment variables .tf file, then run Terraform apply to get the job done. The change also can be tracked when leveraging Git as version control.\nStreamlining AWS SSO in Complex Multi-Account Environments\n","permalink":"https://zackblog.work/posts/modularizing-aws-sso-user-assignment-with-terraform/","summary":"\u003cp\u003eIn the company, we use AWS SSO for user authentication; when a new user is created from Azure AD, it will automatically synced by AWS IAM Identity Center and Azure AD integration, and then our team will need to handle the SSO user assignment to put them in the required AWS accounts with requested permission sets, it became a pain when such requests coming more frequently and every time an individual user or a whole team with different accounts and permission requirements need to be fulfilled, so how to handle this efficiently become my recent topic.\u003c/p\u003e","title":"Modularizing AWS SSO User Assignment with Terraform"},{"content":"Kustomize is a configuration management tool for Kubernetes that allows users to customize application manifests without modifying the original YAML files, in this post I will explore Kustomize with overlays, bases, and transformers, then create simple kustomization.yaml files for different environments using Zackblog, then to practice using Kustomize’s built-in resources like configMapGenerator and secretGenerator.\nLet\u0026rsquo;s take Zackblog k8s deployment manifest as an example and convert it into a Kustomize setup. We will start by organizing the files and gradually exploring key Kustomize features.\nSet Up Kustomize Folder Structure\nThe base directory contains the common configuration. The overlays directories are for environment-specific customizations (e.g., dev and prod).\nmkdir -p zackblog/base mkdir -p zackblog/overlays/dev mkdir -p zackblog/overlays/prod Create the Base Kustomization\nMove the original zackblog.yaml manifest to the base directory and split it into separate files for the deployment.yaml and service.yaml, then create a kustomization.yaml file in the base directory to manage these resources:\n# zackblog/base/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: zackweb labels: app: zackweb spec: replicas: 1 selector: matchLabels: app: zackweb template: metadata: labels: app: zackweb spec: containers: - name: zackweb image: zackz001/gitops-jekyll:latest ports: - containerPort: 80 # zackblog/base/service.yaml apiVersion: v1 kind: Service metadata: name: zackweb-service spec: selector: app: zackweb ports: - protocol: TCP port: 80 targetPort: 80 type: LoadBalancer # zackblog/base/kustomization.yaml resources: - deployment.yaml - service.yaml Create Environment Overlays\nIn Kustomize, we create environment-specific overlays by overriding or patching the base configuration.\nDev Overlay: In the dev directory, create a kustomization.yaml file to customize the base configuration for development, then create a patch file patch.yaml to modify the replicas for the dev environment.\nmkdir -p zackblog/overlays/dev/ mkdir -p zackblog/overlays/prod # zackblog/overlays/dev/kustomization.yaml resources: - ../../base patchesStrategicMerge: - patch.yaml commonLabels: environment: dev # zackblog/overlays/dev/patch.yaml apiVersion: apps/v1 kind: Deployment metadata: name: zackweb spec: replicas: 2 # Increase replica count in dev Similarly, for production, create another kustomization.yaml and patch.yaml in the prod directory and set replicas to 4:\n# zackblog/overlays/prod/kustomization.yaml resources: - ../../base patchesStrategicMerge: - patch.yaml commonLabels: environment: prod # zackblog/overlays/prod/patch.yaml apiVersion: apps/v1 kind: Deployment metadata: name: zackweb spec: replicas: 5 # Scale to 5 replicas in prod The final Kustomize Folder tree\nThe final Kustomize Folder structure looks like this, then we go and apply both dev and prod deployments into 2 namespaces using Kustomize environment overlays.\n/zackblog# tree . ├── base │ ├── deployment.yaml │ ├── kustomization.yaml │ └── service.yaml └── overlays ├── dev │ ├── kustomization.yaml │ └── patch.yaml └── prod ├── kustomization.yaml └── patch.yaml kubectl create namespace zackblog-dev kubectl create namespace zackblog-prod namespace/zackblog-dev created namespace/zackblog-prod created kubectl apply -k ./zackblog/overlays/dev service/zackweb-service created deployment.apps/zackweb created kubectl apply -k ./zackblog/overlays/prod service/zackweb-service created deployment.apps/zackweb created root@asb:~/kustomizahelm/kustomize-z# kubectl get deployment zackweb -n zackblog-dev --show-labels NAME READY UP-TO-DATE AVAILABLE AGE LABELS zackweb 2/2 2 2 3m20s app=zackweb,environment=dev root@asb:~/kustomizahelm/kustomize-z# kubectl get deployment zackweb -n zackblog-prod --show-labels NAME READY UP-TO-DATE AVAILABLE AGE LABELS zackweb 4/4 4 4 2m47s app=zackweb,environment=prod Generate ConfigMaps and Secrets dynamically without manually creating YAML files\nHere we will add a ConfigMap by updating the Base kustomization.yaml, then update deployment.yaml to merge and use the ConfigMap:\n# zackblog/base/kustomization.yaml resources: - deployment.yaml - service.yaml configMapGenerator: - name: zackweb-config literals: - LOG_LEVEL=debug # zackblog/base/deployment.yaml spec: containers: - name: zackweb image: zackz001/gitops-jekyll:latest ports: - containerPort: 80 envFrom: # add configmap - configMapRef: name: zackweb-config # zackblog/overlays/dev/kustomization.yaml configMapGenerator: - name: zackweb-config behavior: merge # Merge with the base ConfigMap literals: - LOG_LEVEL=debug # Dev-specific log level # zackblog/overlays/prod/kustomization.yaml configMapGenerator: - name: zackweb-config behavior: merge # Merge with the base ConfigMap literals: - LOG_LEVEL=info # Prod-specific log level root@asb:~/kustomizahelm/kustomize-z# kubectl apply -k zackblog/base configmap/zackweb-config-47668c6k28 created service/zackweb-service created deployment.apps/zackweb created root@asb:~/kustomizahelm/kustomize-z# kubectl apply -k zackblog/overlays/dev configmap/zackweb-config-hf678c7m2b created service/zackweb-service unchanged deployment.apps/zackweb configured root@asb:~/kustomizahelm/kustomize-z# kubectl apply -k zackblog/overlays/prod configmap/zackweb-config-hf678c7m2b created service/zackweb-service unchanged deployment.apps/zackweb configured Using Transformers for Advanced Customizations\nNext, we will create a Transformer File to add Resource Limits for deployment in dev\n# zackblog/overlays/dev/resource-limits.yaml apiVersion: builtin kind: PatchTransformer metadata: name: add-resource-limits patch: | - op: add path: /spec/template/spec/containers/0/resources value: limits: cpu: \u0026#34;500m\u0026#34; memory: \u0026#34;256Mi\u0026#34; requests: cpu: \u0026#34;250m\u0026#34; memory: \u0026#34;128Mi\u0026#34; target: kind: Deployment name: zackweb # zackblog/overlays/dev/kustomization.yaml namespace: zackblog-dev transformers: - resource-limits.yaml root@asb:~/kustomizahelm/kustomize-z# kubectl apply -k zackblog/overlays/dev configmap/zackweb-config-hf678c7m2b unchanged service/zackweb-service unchanged deployment.apps/zackweb configured root@asb:~/kustomizahelm/kustomize-z# kubectl describe deployments.apps -n zackblog-dev Pod Template: Labels: app=zackweb environment=dev Containers: zackweb: Image: zackz001/gitops-jekyll:latest Port: 80/TCP Host Port: 0/TCP Limits: cpu: 500m memory: 256Mi Requests: cpu: 250m memory: 128Mi Environment Variables from: zackweb-config-hf678c7m2b ConfigMap Optional: false Override container image tags for different environments.\nLastly, we can update image tags in Dev to use zackz001/gitops-jekyll:v222 by updating overlay kustomization.yaml\n# zackblog/overlays/dev/kustomization.yaml namespace: zackblog-dev images: - name: zackz001/gitops-jekyll newTag: dev root@asb:~/kustomizahelm/kustomize-z# kubectl describe deployments.apps -n zackblog-dev | grep Image Image: zackz001/gitops-jekyll:v222 Summary about Kustomize\nWe had done the following practice using Kustomize:\nNamespace Isolation: Use separate namespaces (zackblog-dev and zackblog-prod) to isolate environments. Environment-Specific Customizations: Utilize overlays to manage environment-specific configurations, such as replica counts, resource limits, and image tags. Maintainable Structure: Keep a clear base and overlay structure to manage configurations efficiently. Leverage Kustomize Features: Explore generators, transformers, and image overrides to maximize Kustomize\u0026rsquo;s capabilities. Version Control: Keep Kustomize configurations under version control to track changes and collaborate effectively. Next stage I want to explore:\nStrategic Merge vs. JSON Patches: Understand the differences and use cases for each patch type. Custom Transformers: Create custom transformers for complex modifications. Integration with GitOps Tools: Integrate Kustomize with ArgoCD or Flux for automated, Git-driven deployments. Managing Secrets Securely: Use tools like sealed-secrets or SOPS with Kustomize to manage sensitive information. About Helm\nHelm is another popular package manager for Kubernetes application deployment, not new to me as I had tried many charts previously with Kafka, and Redis helm charts installation, today I am going to explore how to build my helm chart for this zack blog, and deep dive into the chart development for advanced templating, together with helm release management and version control, finally integrate my chart with CI/CD for GitOps.\nCommon Helm command\nhelm list -A # list releases across all namespaces helm pull bitnami/postgresql-ha \u0026ndash;untar # untar the chart after pull online chart helm repo add bitnami https://charts.bitnami.com/bitnami # add a repo helm create zackblog-helm # create a new chart helm install zackblog-helm ~/zackblog-helm -n NAMESPACE -f dev-values.yaml # define ns and override with a new value file helm upgrade zackblog-helm ~/zackblog-helm \u0026ndash;set image.repository= \u0026ndash;set image.tag= # \u0026ndash;set to upgrade chart with override a new value helm lint ~/zackblog-helm # lint syntax helm rollback zackblog-helm 2 # rollback to revision 2 of a release helm uninstall zackblog-helm -n Production # uninstall a chart from a ns Start with own chart\ncreate a new helm chart [root@freeipa-server ~]# helm create zackblog-helm Creating zackblog-helm # modify values.yaml [root@freeipa-server zackblog]# vim values.yaml replicaCount: 3 image: repository: zackz001/gitops-jekyll pullPolicy: IfNotPresent # Overrides the image tag. tag: \u0026#34;latest\u0026#34; service: type: NodePort port: 80 Lint chart syntax before install # lint syntax [root@freeipa-server ~]# helm lint zackblog-helm/ ==\u0026gt; Linting zackblog-helm/ [INFO] Chart.yaml: icon is recommended 1 chart(s) linted, 0 chart(s) failed # install own chart [root@freeipa-server ~]# helm install zackblog-helm zackblog-helm NAME: zackblog-helm LAST DEPLOYED: Mon May 13 21:27:14 2024 NAMESPACE: default STATUS: deployed REVISION: 1 NOTES: 1. Get the application URL by running these commands: export NODE_PORT=$(kubectl get --namespace default -o jsonpath=\u0026#34;{.spec.ports[0].nodePort}\u0026#34; services zackblog-helm) export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath=\u0026#34;{.items[0].status.addresses[0].address}\u0026#34;) echo http://$NODE_IP:$NODE_PORT Customize value.yaml by changing replica and image tag # modify value.yaml to scale down and change image to v138 [root@freeipa-server ~]# vim zackblog-helm/values.yaml replicaCount: 1 image: repository: zackz001/gitops-jekyll pullPolicy: IfNotPresent # Overrides the image tag. tag: \u0026#34;v139\u0026#34; Override values.yaml by -f and deploy the same chart to different environments # create a dev ns then deploy and override with dev-values.yaml [root@freeipa-server ~]# vim zackblog-helm/dev-values.yaml image: repository: zackz001/gitops-jekyll tag: v140 replicaCount: 2 service: type: NodePort port: 80 [root@freeipa-server ~]# kubectl create ns dev namespace/dev created [root@freeipa-server ~]# helm install dev-zackblog-helm zackblog-helm -f zackblog-helm/dev-values.yaml -n dev NAME: dev-zackblog-helm LAST DEPLOYED: Mon May 13 21:36:39 2024 NAMESPACE: dev STATUS: deployed REVISION: 1 Advanced templating to add PVC into chart\n# add templates/pvc.yaml [root@freeipa-server ~]# vim zackblog-helm/templates/pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: longhron-pvc spec: accessModes: - ReadWriteOnce storageClassName: longhorn resources: requests: storage: 1Gi # add pvc in values.yaml [root@freeipa-server ~]# vim zackblog-helm/values.yaml pvc: enabled: true templateFiles: - pvc.yaml # add persistentVolumeClaim in templates/deployment.yaml [root@freeipa-server ~]# vim zackblog-helm/templates/deployment.yaml ... volumes: - name: data persistentVolumeClaim: claimName: longhron-pvc Integrate chart deployment with ArgoCD # argoCD application manifest project: default source: repoURL: \u0026#39;https://github.com/ZackZhouHB/zack-gitops-project.git\u0026#39; path: argo-helm-zackblog targetRevision: editing helm: valueFiles: - values.yaml destination: server: \u0026#39;https://kubernetes.default.svc\u0026#39; namespace: helm syncPolicy: automated: {} syncOptions: - CreateNamespace=true Conclusion\nFinally, I had a chance to go over helm, it makes package management easier and more convenient, through charts, k8s deployment can be more flexible with values and templates that can be deployed and reusable into different environments, it provides versioning and rollbacks, also allow customization of the template. However using online charts can also be risky in a production environment with quality, dependency, and security risks.\nNext stage for Helm advanced templating:\nHelmfile for Multi-Release Management Use Helm’s built-in test hooks to create automated tests for your deployments Master dependencies and subcharts for modular Helm chart creation Helm Chart private Repositories and Version Control Helm + Kustomize Hybrid Approaches ","permalink":"https://zackblog.work/posts/helm-kustomize-for-zackblog/","summary":"\u003cp\u003eKustomize is a configuration management tool for Kubernetes that allows users to customize application manifests without modifying the original YAML files, in this post I will explore Kustomize with overlays, bases, and transformers, then create simple kustomization.yaml files for different environments using Zackblog, then to practice using Kustomize’s built-in resources like configMapGenerator and secretGenerator.\u003c/p\u003e\n\u003cp\u003eLet\u0026rsquo;s take Zackblog k8s deployment manifest as an example and convert it into a Kustomize setup. We will start by organizing the files and gradually exploring key Kustomize features.\u003c/p\u003e","title":"Helm + Kustomize for ZackBlog"},{"content":"The application team managing Rancher clusters in AWS EC2 faced a compliance challenge with their Rancher node template golden AMI. To meet security and compliance requirements, they needed to ensure that this AMI is patched regularly with the latest updates. This process had to be automated to guarantee that a newly patched AMI is available every month.\nOverview of workflow with Lambda and Cloudformation\nI developed this CloudFormation template to set up an automated process for patching Amazon Machine Images (AMIs) on a monthly schedule using AWS services such as Lambda, EventBridge, SNS, and Parameter Store. Here\u0026rsquo;s a workflow by design:\nThe Lambda function is triggered on the 1st of every month to automate the creation of a patched AMI for Rancher. It interacts with EC2 to create the AMI, update the SSM Parameter Store with the new AMI ID, and terminate any temporary EC2 instances used for the patching process. The AMI ID of the latest patched image is stored in the AWS Systems Manager (SSM) Parameter Store (/ami/latest), ensuring that the latest AMI can be referenced easily in other systems. An SNS Topic is used to send email notifications, informing stakeholders about the status of the AMI patching process. The template sets up IAM roles and policies with least privilege for EC2 instances and the Lambda function, ensuring that the required actions can be performed securely within AWS. The Cloudformation Template\nBellow resources will be created by this CloudFormation template:\nSSM Parameter Store: A Parameter (/ami/latest) is created to store the ID of the latest patched AMI. The initial AMI ID is set to Ubuntu 20.04 TLS (ami-03xxxxxxxxxa6). SNS Topic: A SNS Topic is created for sending notifications about the AMI patching process. It is configured to send notifications via email (zhbsoftboy1@gmail). EC2 Instance Role and Profile: An IAM role (EC2InstanceRole) is created with permissions for various EC2 and SSM actions, including updating instance information, sending commands, and listing associations. An instance profile (EC2InstanceProfile-for-AMI-Patching) is associated with this role, which allows EC2 instances to assume the role for patching purposes. Lambda Execution Role: A Lambda execution role (LambdaExecutionRole) is created with permissions to manage EC2 instances (create, run, terminate), interact with SSM and SNS, and log activities to CloudWatch Logs. It also allows the Lambda function to pass the necessary roles (iam:PassRole). Lambda Function: A Lambda function (Rancher-AMI-Patching-Function) is defined to handle the actual AMI patching process. The function code is stored in an S3 bucket as a zip file (lambda_function.zip). The Lambda function uses Python 3.9, has a 15-minute timeout (max allowed), and is allocated 256 MB of memory. It reads the SNS topic ARN and AMI parameter name from environment variables. EventBridge Rule: An EventBridge rule is created to trigger the Lambda function on the 1st of every month at midnight (UTC) using a cron expression (cron(0 0 1 * ? *)). Lambda Invoke Permission: A Lambda permission is added to allow the EventBridge rule to invoke the Lambda function. # cnf-ami-lab.yaml AWSTemplateFormatVersion: \u0026#39;2010-09-09\u0026#39; Description: \u0026gt; This template deploys a Lambda function for automating AMI patching, along with a Parameter Store to track the AMI IDs, and a monthly EventBridge rule to trigger the Lambda. Resources: # Parameter Store to store AMI ID AMIIDParameter: Type: AWS::SSM::Parameter Properties: Name: /ami/latest Description: \u0026#39;Stores the ID of the latest patched AMI\u0026#39; Type: String Value: ami-xxxxxxxxxxxxx # initial AMI ID # SNS Topic for notifications SNSTopic: Type: AWS::SNS::Topic Properties: DisplayName: Rancher AMI Patching Notifications Subscription: - Protocol: email Endpoint: zhbsoftboy1@gmail # test email address The lambda function:\nThis AWS Lambda function automates the process of patching an Amazon Machine Image (AMI) used for a Rancher cluster or similar workloads. It is designed to run on a schedule (e.g., triggered monthly by an EventBridge rule) and performs the following key tasks:\nRetrieve the Latest AMI ID: The Lambda function starts by fetching the latest AMI ID from the AWS Systems Manager (SSM) Parameter Store. This ID is used to launch an EC2 instance for patching. Launch an EC2 Instance: An EC2 instance is launched using the retrieved AMI. The instance type is set to t2.medium, and it is associated with an IAM instance profile that grants necessary permissions for patching and AMI creation. Wait for Instance Readiness: The function waits for the EC2 instance to be fully initialized and ready to receive commands using SSM (AWS Systems Manager). Apply Patches via SSM: The function sends a command to the EC2 instance via SSM to run system updates and apply patches. Specifically, it runs sudo apt-get update and sudo apt-get upgrade -y on the instance. Create a New Patched AMI: After the patching process is complete, the function creates a new AMI from the patched instance. The new AMI is given a name that includes the current date and time for identification. Update the AMI ID in Parameter Store: Once the new AMI is created, its ID is stored back into the SSM Parameter Store, replacing the previous AMI ID. This ensures that the latest AMI can be tracked and used for future patching or deployments. Send Notifications: A notification is sent via SNS (Simple Notification Service) to inform the relevant team members about the successful creation of the new AMI. The notification includes the new AMI ID and a message advising the team to test the AMI before rolling it out to production. Terminate the EC2 Instance: After the AMI is created, the EC2 instance used for patching is terminated to avoid unnecessary costs. Error Handling: If any error occurs during the process, it is logged, and the EC2 instance is terminated regardless of success or failure, ensuring proper cleanup. import boto3 import time import os import logging from datetime import datetime # Set up logging logger = logging.getLogger() logger.setLevel(logging.INFO) ec2 = boto3.client(\u0026#39;ec2\u0026#39;) ssm = boto3.client(\u0026#39;ssm\u0026#39;) sns = boto3.client(\u0026#39;sns\u0026#39;) parameter_store = boto3.client(\u0026#39;ssm\u0026#39;) def lambda_handler(event, context): logger.info(\u0026#34;Lambda function started\u0026#34;) # Retrieve last AMI ID from Parameter Store parameter_name = os.environ[\u0026#39;AMI_PARAMETER_NAME\u0026#39;] response = parameter_store.get_parameter(Name=parameter_name) old_ami_id = response[\u0026#39;Parameter\u0026#39;][\u0026#39;Value\u0026#39;] logger.info(f\u0026#34;Using AMI ID: {old_ami_id} to launch the instance\u0026#34;) instance = ec2.run_instances( ImageId=old_ami_id, InstanceType=\u0026#39;t2.medium\u0026#39;, MinCount=1, MaxCount=1, IamInstanceProfile={\u0026#39;Name\u0026#39;: \u0026#39;EC2InstanceProfile-for-AMI-Patching\u0026#39;} ) Conclusion\nThis setup ensures that the team\u0026rsquo;s AMIs are always up to date with the latest patches, improving security and reliability for the applications or environments that use them.\nAutomated AMI Patching: The function automates the entire process of launching an EC2 instance, applying patches, creating a new AMI, and updating the parameter store. Cost Optimization: By terminating the EC2 instance after the patching process, it ensures resources are only used when necessary. Ease of Management: The function updates the SSM Parameter Store with the latest AMI ID, which simplifies the tracking of the most recent patched AMI. Team Notification: Through SNS, it keeps the team informed about the newly patched AMI, streamlining communication for testing and production rollouts. ","permalink":"https://zackblog.work/posts/automate-ami-patching-with-lambda-and-cloudformation/","summary":"\u003cp\u003eThe application team managing Rancher clusters in AWS EC2 faced a compliance challenge with their Rancher node template golden AMI. To meet security and compliance requirements, they needed to ensure that this AMI is patched regularly with the latest updates. This process had to be automated to guarantee that a newly patched AMI is available every month.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eOverview of workflow with Lambda and Cloudformation\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eI developed this CloudFormation template to set up an automated process for patching Amazon Machine Images (AMIs) on a monthly schedule using AWS services such as Lambda, EventBridge, SNS, and Parameter Store. Here\u0026rsquo;s a workflow by design:\u003c/p\u003e","title":"Automate AMI patching with Lambda and Cloudformation"},{"content":"Despite all the challenges, in the last 2 years, clever people still managed ways to deploy production-grade database within a Kubernetes cluster by using Kubernetes as a platform to develop custom resource definition (CRDs) like helm charts like bitnami/postgresql-ha, and PostgreSQL Operator like CrunchyData/postgres-operator or zalando/postgres-operator.\nLast post I was able to deploy a single PostgreSQL in local k8s, but I had to manually create Kubernetes namespaces, define database creds, configuration and environment variables into k8s secret and configmap, also to define the statefulset yaml with volume claim template.\nStill I was not able to configure HA and failover as I found it is so limited and a headache within K8S if only relying on statefulset. Luckily there are engineers out there to develop helm and operator to get the job done.\nCrunchyData Postgres-Operator\nIn this session, I will follow bellow steps to:\nDeploy PostgreSQL Operator # Clone the CrunchyData Postgres Operator [root@freeipa-server ~]# git clone https://github.com/CrunchyData/postgres-operator-examples.git # create namespace and deploy GPO Postgres Operatorusing kustomize [root@freeipa-server postgres-operator-examples]# kubectl apply -k kustomize/install/namespace namespace/postgres-operator created [root@freeipa-server postgres-operator-examples]# kubectl apply --server-side -k kustomize/install/default customresourcedefinition.apiextensions.k8s.io/pgadmins.postgres-operator.crunchydata.com serverside-applied customresourcedefinition.apiextensions.k8s.io/pgupgrades.postgres-operator.crunchydata.com serverside-applied customresourcedefinition.apiextensions.k8s.io/postgresclusters.postgres-operator.crunchydata.com serverside-applied serviceaccount/pgo serverside-applied clusterrole.rbac.authorization.k8s.io/postgres-operator serverside-applied clusterrolebinding.rbac.authorization.k8s.io/postgres-operator serverside-applied deployment.apps/pgo serverside-applied # validate deploy status [root@freeipa-server postgres-operator-examples]# kubectl get all -n postgres-operator NAME READY STATUS RESTARTS AGE pod/pgo-77d6b49b8-wrdjp 1/1 Running 0 2m47s Deploy HA PostgreSQL Cluster\n# Create a Postgres Cluster named \u0026#34;hippo\u0026#34; in \u0026#34;postgres-operator\u0026#34; ns [root@freeipa-server postgres-operator-examples]# kubectl apply -k kustomize/postgres postgrescluster.postgres-operator.crunchydata.com/hippo created [root@freeipa-server postgres-operator-examples]# kubectl get all -n postgres-operator NAME READY STATUS RESTARTS AGE pod/hippo-backup-dvks-m4z5m 1/1 Running 0 56s pod/hippo-instance1-582s-0 4/4 Running 0 2m14s pod/hippo-repo-host-0 2/2 Running 0 2m14s pod/pgo-77d6b49b8-wrdjp 1/1 Running 0 6m38s Connect an application to PostgreSQL cluster\nHere we use Keycloak, a popular open-source identity management tool that is backed by a PostgreSQL database. Using the hippo cluster we created, we can deploy the following manifest file:\n# create deployment keycloak to connect PostgreSQL database [root@freeipa-server postgres-operator-examples]# vim kustomize/keycloak/keycloak.yaml apiVersion: apps/v1 kind: Deployment metadata: name: keycloak namespace: postgres-operator labels: app.kubernetes.io/name: keycloak spec: selector: matchLabels: app.kubernetes.io/name: keycloak template: metadata: labels: app.kubernetes.io/name: keycloak spec: containers: - image: quay.io/keycloak/keycloak:latest args: [\u0026#34;start-dev\u0026#34;] name: keycloak env: - name: DB_VENDOR value: \u0026#34;postgres\u0026#34; - name: DB_ADDR valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: host } } - name: DB_PORT valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: port } } - name: DB_DATABASE valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: dbname } } - name: DB_USER valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: user } } - name: DB_PASSWORD valueFrom: { secretKeyRef: { name: hippo-pguser-hippo, key: password } } - name: KEYCLOAK_ADMIN value: \u0026#34;admin\u0026#34; - name: KEYCLOAK_ADMIN_PASSWORD value: \u0026#34;admin\u0026#34; - name: KC_PROXY value: \u0026#34;edge\u0026#34; ports: - name: http containerPort: 8080 - name: https containerPort: 8443 readinessProbe: httpGet: path: /realms/master port: 8080 restartPolicy: Always [root@freeipa-server postgres-operator-examples]# kubectl apply -f kustomize/keycloak/keycloak.yaml deployment.apps/keycloak created Scale Up / Down\nEdit manifest to add 2 more replicas\n[root@freeipa-server kustomize]# kubectl apply -k postgres -n postgres-operator postgrescluster.postgres-operator.crunchydata.com/hippo configured # watch change [root@freeipa-server postgres-operator-examples]# watch kubectl get pod -L postgres-operator.crunchydata.com/role -l postgres-operator.crunchydata.com/instance -n postgres-operator Failover testing:\nNow I am going to delete the primary instance, one of the standby pod will take over and become primary automatically\n# delete the primary pod hippo-instance1-nhbc-0, then previous replica pod hippo-instance1-q8kk-0 promoted as master # pod hippo-instance1-nhbc-0 will up again as a replica [root@freeipa-server kustomize]# kubectl delete po hippo-instance1-nhbc-0 -n postgres-operator pod \u0026#34;hippo-instance1-nhbc-0\u0026#34; deleted Perform Minor version rolling upgrade\nHere I changed the database version to 16.1, the cluster will start a rolling update by\nApplying new version to one of the standby pod first Then update another replica pod Promote the first upgraded replica as master Lastly the previous master pod will be updated and become a replica # validate DB version before miner upgrade [root@freeipa-server kustomize]# kubectl exec -it hippo-instance1-q8kk-0 -n postgres-operator -- psql --version Defaulted container \u0026#34;database\u0026#34; out of: database, replication-cert-copy, pgbackrest, pgbackrest-config, postgres-startup (init), nss-wrapper-init (init) psql (PostgreSQL) 16.2 # validate DB version after miner version change [root@freeipa-server kustomize]# kubectl exec -it hippo-instance1-q8kk-0 -n postgres-operator -- psql --version Defaulted container \u0026#34;database\u0026#34; out of: database, replication-cert-copy, pgbackrest, pgbackrest-config, postgres-startup (init), nss-wrapper-init (init) psql (PostgreSQL) 16.1 Backup\nAdd backup Cron job into manifest to add weekly full backup and daily incremental\n[root@freeipa-server ~]# kubectl get cronjobs -n postgres-operator NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE hippo-repo1-full 0 1 * * 0 False 0 \u0026lt;none\u0026gt; 5m21s hippo-repo1-incr 0 1 * * 1-6 False 0 \u0026lt;none\u0026gt; 5m21s Deploy Monitoring (Prom + Grafaba)\nFinally, let\u0026rsquo;s set up the monitoring stack for PostgreSQL by using Prometheus and Grafana.\n# deploy monitoring stack [root@freeipa-server kustomize]# kubectl apply -k monitoring serviceaccount/alertmanager created serviceaccount/grafana created serviceaccount/prometheus created clusterrole.rbac.authorization.k8s.io/prometheus created clusterrolebinding.rbac.authorization.k8s.io/prometheus created configmap/alert-rules-config created configmap/alertmanager-config created configmap/crunchy-prometheus created configmap/grafana-dashboards created configmap/grafana-datasources created secret/grafana-admin created service/crunchy-alertmanager created service/crunchy-grafana created service/crunchy-prometheus created persistentvolumeclaim/alertmanagerdata created persistentvolumeclaim/grafanadata created persistentvolumeclaim/prometheusdata created deployment.apps/crunchy-alertmanager created deployment.apps/crunchy-grafana created deployment.apps/crunchy-prometheus created # Edit Grafana service to NodePort [root@freeipa-server postgres-operator-examples]# kubectl edit svc crunchy-grafana -n postgres-operator service/crunchy-grafana edited Execute into master database container, using pgbench to generate tables\n[root@freeipa-server postgres-operator-examples]# kubectl exec -it hippo-instance1-nhbc-0 -c database -n postgres-operator -- bash bash-4.4$ pgbench -i -s 100 -U postgres -d postgres dropping old tables... NOTICE: table \u0026#34;pgbench_accounts\u0026#34; does not exist, skipping NOTICE: table \u0026#34;pgbench_branches\u0026#34; does not exist, skipping NOTICE: table \u0026#34;pgbench_history\u0026#34; does not exist, skipping NOTICE: table \u0026#34;pgbench_tellers\u0026#34; does not exist, skipping creating tables... generating data (client-side)... 10000000 of 10000000 tuples (100%) done (elapsed 45.61 s, remaining 0.00 s) Some Grafana predefined PostgreSQL dashboard, unfortunately I do not have much data in it to show more monitoring status.\nConclusion\nThis is the final session of this PostgreSQL series, together I have explored PostgreSQL from very basic docker deployment with replica, to production-grade deployment in Kubernetes using operator, practice from backup, monitoring, rolling update, to HA, failover, and scale up. HAHA!\n","permalink":"https://zackblog.work/posts/postgresql-prod-grade-with-k8s-operator/","summary":"\u003cp\u003eDespite all the challenges, in the last 2 years, clever people still managed ways to deploy production-grade database within a Kubernetes cluster by using Kubernetes as a platform to develop custom resource definition (CRDs) like helm charts like bitnami/postgresql-ha, and PostgreSQL Operator like CrunchyData/postgres-operator or zalando/postgres-operator.\u003c/p\u003e\n\u003cp\u003eLast post I was able to deploy a single PostgreSQL in local k8s, but I had to manually create Kubernetes namespaces, define database creds, configuration and environment variables into k8s secret and configmap, also to define the statefulset yaml with volume claim template.\u003c/p\u003e","title":"PostgreSQL: Prod-Grade with k8s Operator"},{"content":"PostgreSQL by default does not build for Kubernetes, and a database with StatefulSet workload in K8S can be brutal to manage. In my lab K8S cluster, here we create namespace, secret, configmap, PVC, and StatefulSet to run a single PostgreSQL.\n# create namespace \u0026#34;postgresql\u0026#34; [root@freeipa-server ~]# kubectl create ns postgresql namespace/postgresql created # create secret to store database creds [root@freeipa-server ~]# kubectl -n postgresql create secret generic postgresql --from-literal POSTGRES_USER=\u0026#34;postgresadmin\u0026#34; --from-literal POSTGRES_PASSWORD=\u0026#39;admin123\u0026#39; --from-literal POSTGRES_DB=\u0026#34;postgresdb\u0026#34; --from-literal REPLICATION_USER=\u0026#34;replicationuser\u0026#34; --from-literal REPLICATION_PASSWORD=\u0026#39;replicationPassword\u0026#39; secret/postgresql created # create configmap, pvc, statefulset with init container to run postgresql [root@freeipa-server ~]# vim stateful.yaml apiVersion: v1 kind: ConfigMap metadata: name: postgres data: pg_hba.conf: |+ # TYPE DATABASE USER ADDRESS METHOD host replication replicationuser 0.0.0.0/0 md5 # \u0026#34;local\u0026#34; is for Unix domain socket connections only local all all trust # IPv4 local connections: host all all 127.0.0.1/32 trust # IPv6 local connections: host all all ::1/128 trust # Allow replication connections from localhost, by a user with the # replication privilege. local replication all trust host replication all 127.0.0.1/32 trust host replication all ::1/128 trust host all all all scram-sha-256 postgresql.conf: |+ data_directory = \u0026#39;/data/pgdata\u0026#39; hba_file = \u0026#39;/config/pg_hba.conf\u0026#39; ident_file = \u0026#39;/config/pg_ident.conf\u0026#39; port = 5432 listen_addresses = \u0026#39;*\u0026#39; max_connections = 100 shared_buffers = 128MB dynamic_shared_memory_type = posix max_wal_size = 1GB min_wal_size = 80MB log_timezone = \u0026#39;Etc/UTC\u0026#39; datestyle = \u0026#39;iso, mdy\u0026#39; timezone = \u0026#39;Etc/UTC\u0026#39; #locale settings lc_messages = \u0026#39;en_US.utf8\u0026#39;\t# locale for system error message lc_monetary = \u0026#39;en_US.utf8\u0026#39;\t# locale for monetary formatting lc_numeric = \u0026#39;en_US.utf8\u0026#39;\t# locale for number formatting lc_time = \u0026#39;en_US.utf8\u0026#39;\t# locale for time formatting default_text_search_config = \u0026#39;pg_catalog.english\u0026#39; #replication wal_level = replica archive_mode = on archive_command = \u0026#39;test ! -f /data/archive/%f \u0026amp;\u0026amp; cp %p /data/archive/%f\u0026#39; max_wal_senders = 3 --- apiVersion: apps/v1 kind: StatefulSet metadata: name: postgres spec: selector: matchLabels: app: postgres serviceName: \u0026#34;postgres\u0026#34; replicas: 1 template: metadata: labels: app: postgres spec: terminationGracePeriodSeconds: 30 initContainers: - name: init image: postgres:15.0 command: [ \u0026#34;bash\u0026#34;, \u0026#34;-c\u0026#34; ] args: - | #create archive directory mkdir -p /data/archive \u0026amp;\u0026amp; chown -R 999:999 /data/archive volumeMounts: - name: data mountPath: /data readOnly: false containers: - name: postgres image: postgres:15.0 args: [\u0026#34;-c\u0026#34;, \u0026#34;config_file=/config/postgresql.conf\u0026#34;] ports: - containerPort: 5432 name: database env: - name: PGDATA value: \u0026#34;/data/pgdata\u0026#34; - name: POSTGRES_USER valueFrom: secretKeyRef: name: postgresql key: POSTGRES_USER optional: false - name: POSTGRES_PASSWORD valueFrom: secretKeyRef: name: postgresql key: POSTGRES_PASSWORD optional: false - name: POSTGRES_DB valueFrom: secretKeyRef: name: postgresql key: POSTGRES_DB optional: false volumeMounts: - name: config mountPath: /config readOnly: false - name: data mountPath: /data readOnly: false volumes: - name: config configMap: name: postgres defaultMode: 0755 volumeClaimTemplates: - metadata: name: data spec: accessModes: [ \u0026#34;ReadWriteOnce\u0026#34; ] storageClassName: \u0026#34;standard\u0026#34; resources: requests: storage: 100Mi --- apiVersion: v1 kind: Service metadata: name: postgres labels: app: postgres spec: ports: - port: 5432 targetPort: 5432 name: postgres clusterIP: None selector: app: postgres [root@freeipa-server ~]# kubectl create -f stateful.yaml -n postgresql configmap/postgres created statefulset.apps/postgres created service/postgres created # validate for pvc, pods [root@freeipa-server ~]# kubectl get pvc -n postgresql NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-postgres-0 Bound pvc-dd89fc0a-915f-40eb-b61f-917234074a61 100Mi RWO longhorn 19m [root@freeipa-server ~]# kubectl get all -n postgresql NAME READY STATUS RESTARTS AGE pod/postgres-0 1/1 Running 0 6m29s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/postgres ClusterIP None \u0026lt;none\u0026gt; 5432/TCP 6m29s NAME READY AGE statefulset.apps/postgres 1/1 6m29s # check container logs for database connection status [root@freeipa-server ~]# kubectl logs -n postgresql postgres-0 Defaulted container \u0026#34;postgres\u0026#34; out of: postgres, init (init) The files belonging to this database system will be owned by user \u0026#34;postgres\u0026#34;. This user must also own the server process. The database cluster will be initialized with locale \u0026#34;en_US.utf8\u0026#34;. The default database encoding has accordingly been set to \u0026#34;UTF8\u0026#34;. The default text search configuration will be set to \u0026#34;english\u0026#34;. Data page checksums are disabled. fixing permissions on existing directory /data/pgdata ... ok creating subdirectories ... ok selecting dynamic shared memory implementation ... posix selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting default time zone ... Etc/UTC creating configuration files ... ok running bootstrap script ... ok performing post-bootstrap initialization ... ok initdb: warning: enabling \u0026#34;trust\u0026#34; authentication for local connections initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. syncing data to disk ... ok Success. You can now start the database server using: pg_ctl -D /data/pgdata -l logfile start waiting for server to start....2024-01-12 00:52:56.718 UTC [49] LOG: starting PostgreSQL 15.0 (Debian 15.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit 2024-01-12 00:52:56.719 UTC [49] LOG: listening on Unix socket \u0026#34;/var/run/postgresql/.s.PGSQL.5432\u0026#34; 2024-01-12 00:52:56.730 UTC [49] LOG: could not open usermap file \u0026#34;/config/pg_ident.conf\u0026#34;: No such file or directory 2024-01-12 00:52:56.733 UTC [52] LOG: database system was shut down at 2024-01-12 00:52:55 UTC 2024-01-12 00:52:56.744 UTC [49] LOG: database system is ready to accept connections done server started CREATE DATABASE /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/* 2024-01-12 00:52:56.957 UTC [49] LOG: received fast shutdown request waiting for server to shut down....2024-01-12 00:52:56.961 UTC [49] LOG: aborting any active transactions 2024-01-12 00:52:56.962 UTC [49] LOG: background worker \u0026#34;logical replication launcher\u0026#34; (PID 56) exited with exit code 1 2024-01-12 00:52:56.963 UTC [49] LOG: shutting down 2024-01-12 00:52:57.042 UTC [49] LOG: checkpoint starting: shutdown immediate ..2024-01-12 00:52:59.314 UTC [49] LOG: checkpoint complete: wrote 918 buffers (5.6%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.434 s, sync=0.014 s, total=2.279 s; sync files=250, longest=0.007 s, average=0.001 s; distance=11271 kB, estimate=11271 kB 2024-01-12 00:52:59.318 UTC [49] LOG: database system is shut down done server stopped PostgreSQL init process complete; ready for start up. 2024-01-12 00:52:59.385 UTC [1] LOG: starting PostgreSQL 15.0 (Debian 15.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit 2024-01-12 00:52:59.385 UTC [1] LOG: listening on IPv4 address \u0026#34;0.0.0.0\u0026#34;, port 5432 2024-01-12 00:52:59.385 UTC [1] LOG: listening on IPv6 address \u0026#34;::\u0026#34;, port 5432 2024-01-12 00:52:59.389 UTC [1] LOG: listening on Unix socket \u0026#34;/var/run/postgresql/.s.PGSQL.5432\u0026#34; 2024-01-12 00:52:59.398 UTC [1] LOG: could not open usermap file \u0026#34;/config/pg_ident.conf\u0026#34;: No such file or directory 2024-01-12 00:52:59.404 UTC [67] LOG: database system was shut down at 2024-01-12 00:52:59 UTC 2024-01-12 00:52:59.415 UTC [1] LOG: database system is ready to accept connections Conclusion\nNow we are able to deploy a PostgreSQL in a local K8S cluster, with defined environment variables in Kubernetes secret and configmap, together with an init container to create a data archive volume in persistent storage class. In the next blog, I will discover how to run PostgreSQL HA with persistent volume on Kubernetes with both Helm and Operator, then validate scale-up and scale-down, backup using cronjob, etc.\n","permalink":"https://zackblog.work/posts/postgresql-deploy-into-k8s/","summary":"\u003cp\u003ePostgreSQL by default does not build for Kubernetes, and a database with StatefulSet workload in K8S can be brutal to manage. In my lab K8S cluster, here we create namespace, secret, configmap, PVC, and StatefulSet to run a single PostgreSQL.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-yaml\" data-lang=\"yaml\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# create namespace \u0026#34;postgresql\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003eroot@freeipa-server ~]# kubectl create ns postgresql\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003enamespace/postgresql created\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# create secret to store database creds\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003eroot@freeipa-server ~]# kubectl -n postgresql create secret generic postgresql --from-literal POSTGRES_USER=\u0026#34;postgresadmin\u0026#34; --from-literal POSTGRES_PASSWORD=\u0026#39;admin123\u0026#39; --from-literal POSTGRES_DB=\u0026#34;postgresdb\u0026#34; --from-literal REPLICATION_USER=\u0026#34;replicationuser\u0026#34; --from-literal REPLICATION_PASSWORD=\u0026#39;replicationPassword\u0026#39;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003esecret/postgresql created\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# create configmap, pvc, statefulset with init container to run postgresql\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003eroot@freeipa-server ~]# vim stateful.yaml\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eConfigMap\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgres\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003edata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epg_hba.conf\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|+\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    # TYPE  DATABASE        USER            ADDRESS                 METHOD\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    host     replication     replicationuser         0.0.0.0/0        md5\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    # \u0026#34;local\u0026#34; is for Unix domain socket connections only\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    local   all             all                                     trust\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    # IPv4 local connections:\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    host    all             all             127.0.0.1/32            trust\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    # IPv6 local connections:\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    host    all             all             ::1/128                 trust\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    # Allow replication connections from localhost, by a user with the\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    # replication privilege.\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    local   replication     all                                     trust\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    host    replication     all             127.0.0.1/32            trust\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    host    replication     all             ::1/128                 trust\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    host all all all scram-sha-256\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003epostgresql.conf\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e|+\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    data_directory = \u0026#39;/data/pgdata\u0026#39;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    hba_file = \u0026#39;/config/pg_hba.conf\u0026#39;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    ident_file = \u0026#39;/config/pg_ident.conf\u0026#39;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    port = 5432\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    listen_addresses = \u0026#39;*\u0026#39;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    max_connections = 100\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    shared_buffers = 128MB\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    dynamic_shared_memory_type = posix\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    max_wal_size = 1GB\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    min_wal_size = 80MB\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    log_timezone = \u0026#39;Etc/UTC\u0026#39;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    datestyle = \u0026#39;iso, mdy\u0026#39;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    timezone = \u0026#39;Etc/UTC\u0026#39;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    #locale settings\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    lc_messages = \u0026#39;en_US.utf8\u0026#39;\t\t\t# locale for system error message\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    lc_monetary = \u0026#39;en_US.utf8\u0026#39;\t\t\t# locale for monetary formatting\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    lc_numeric = \u0026#39;en_US.utf8\u0026#39;\t\t\t# locale for number formatting\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    lc_time = \u0026#39;en_US.utf8\u0026#39;\t\t\t\t# locale for time formatting\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    default_text_search_config = \u0026#39;pg_catalog.english\u0026#39;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    #replication\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    wal_level = replica\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    archive_mode = on\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    archive_command = \u0026#39;test ! -f /data/archive/%f \u0026amp;\u0026amp; cp %p /data/archive/%f\u0026#39;\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e    max_wal_senders = 3\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nn\"\u003e---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eapps/v1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eStatefulSet\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgres\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eselector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ematchLabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgres\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eserviceName\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;postgres\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ereplicas\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003etemplate\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003elabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgres\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eterminationGracePeriodSeconds\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e30\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003einitContainers\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003einit\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eimage\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgres:15.0\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003ecommand\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;bash\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;-c\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eargs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"p\"\u003e|\u003c/span\u003e\u003cspan class=\"sd\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          #create archive directory\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"sd\"\u003e          mkdir -p /data/archive \u0026amp;\u0026amp; chown -R 999:999 /data/archive\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003evolumeMounts\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edata\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003emountPath\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e/data\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003ereadOnly\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003econtainers\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgres\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eimage\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgres:15.0\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eargs\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;-c\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e,\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;config_file=/config/postgresql.conf\u0026#34;\u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003econtainerPort\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5432\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edatabase\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003eenv\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePGDATA\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003evalue\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;/data/pgdata\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePOSTGRES_USER\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003evalueFrom\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"nt\"\u003esecretKeyRef\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgresql\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ekey\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePOSTGRES_USER\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003eoptional\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePOSTGRES_PASSWORD\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003evalueFrom\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"nt\"\u003esecretKeyRef\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgresql\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ekey\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePOSTGRES_PASSWORD\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003eoptional\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePOSTGRES_DB\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003evalueFrom\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e            \u003c/span\u003e\u003cspan class=\"nt\"\u003esecretKeyRef\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgresql\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003ekey\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ePOSTGRES_DB\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e              \u003c/span\u003e\u003cspan class=\"nt\"\u003eoptional\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003evolumeMounts\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003econfig\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003emountPath\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e/config\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003ereadOnly\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edata\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003emountPath\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e/data\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003ereadOnly\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"kc\"\u003efalse\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003evolumes\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e- \u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003econfig\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003econfigMap\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgres\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003edefaultMode\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e0755\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003evolumeClaimTemplates\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edata\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eaccessModes\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;ReadWriteOnce\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"p\"\u003e]\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003estorageClassName\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;standard\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e      \u003c/span\u003e\u003cspan class=\"nt\"\u003eresources\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e        \u003c/span\u003e\u003cspan class=\"nt\"\u003erequests\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e          \u003c/span\u003e\u003cspan class=\"nt\"\u003estorage\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003e100Mi\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nn\"\u003e---\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eapiVersion\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ev1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ekind\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eService\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003emetadata\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgres\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003elabels\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgres\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003espec\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eports\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e- \u003cspan class=\"nt\"\u003eport\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5432\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003etargetPort\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"m\"\u003e5432\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003ename\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgres\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eclusterIP\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eNone\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003eselector\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"nt\"\u003eapp\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003epostgres\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003eroot@freeipa-server ~]# kubectl create -f stateful.yaml -n postgresql\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003econfigmap/postgres created\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003estatefulset.apps/postgres created\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eservice/postgres created\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# validate for pvc, pods\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003eroot@freeipa-server ~]# kubectl get pvc -n postgresql\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eNAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003edata-postgres-0   Bound    pvc-dd89fc0a-915f-40eb-b61f-917234074a61   100Mi      RWO            longhorn       19m\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003eroot@freeipa-server ~]# kubectl get all -n postgresql\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eNAME             READY   STATUS    RESTARTS   AGE\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003epod/postgres-0   1/1     Running   0          6m29s\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eNAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eservice/postgres   ClusterIP   None         \u0026lt;none\u0026gt;        5432/TCP   6m29s\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eNAME                        READY   AGE\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003estatefulset.apps/postgres   1/1     6m29s\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c\"\u003e# check container logs for database connection status\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"p\"\u003e[\u003c/span\u003e\u003cspan class=\"l\"\u003eroot@freeipa-server ~]# kubectl logs -n postgresql postgres-0\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eDefaulted container \u0026#34;postgres\u0026#34; out of: postgres, init (init)\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eThe files belonging to this database system will be owned by user \u0026#34;postgres\u0026#34;.\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eThis user must also own the server process.\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eThe database cluster will be initialized with locale \u0026#34;en_US.utf8\u0026#34;.\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eThe default database encoding has accordingly been set to \u0026#34;UTF8\u0026#34;.\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eThe default text search configuration will be set to \u0026#34;english\u0026#34;.\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eData page checksums are disabled.\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003efixing permissions on existing directory /data/pgdata ... ok\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003ecreating subdirectories ... ok\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eselecting dynamic shared memory implementation ... posix\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eselecting default max_connections ... 100\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eselecting default shared_buffers ... 128MB\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eselecting default time zone ... Etc/UTC\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003ecreating configuration files ... ok\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003erunning bootstrap script ... ok\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eperforming post-bootstrap initialization ... ok\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003einitdb\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003ewarning\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eenabling \u0026#34;trust\u0026#34; authentication for local connections\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003einitdb\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"nt\"\u003ehint\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eYou can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003esyncing data to disk ... ok\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003eSuccess. You can now start the database server using\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e    \u003c/span\u003e\u003cspan class=\"l\"\u003epg_ctl -D /data/pgdata -l logfile start\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ewaiting for server to start....2024-01-12 00:52:56.718 UTC [49] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003estarting PostgreSQL 15.0 (Debian 15.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:56.719 UTC [49] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003elistening on Unix socket \u0026#34;/var/run/postgresql/.s.PGSQL.5432\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:56.730 UTC [49] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003ecould not open usermap file \u0026#34;/config/pg_ident.conf\u0026#34;: No such file or directory\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:56.733 UTC [52] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003edatabase system was shut down at 2024-01-12 00:52:55 UTC\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:56.744 UTC [49] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003edatabase system is ready to accept connections\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edone\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eserver started\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eCREATE DATABASE\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e/usr/local/bin/docker-entrypoint.sh\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eignoring /docker-entrypoint-initdb.d/*\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:56.957 UTC [49] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003ereceived fast shutdown request\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003ewaiting for server to shut down....2024-01-12 00:52:56.961 UTC [49] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003eaborting any active transactions\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:56.962 UTC [49] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003ebackground worker \u0026#34;logical replication launcher\u0026#34; (PID 56) exited with exit code 1\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:56.963 UTC [49] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003eshutting down\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:57.042 UTC [49] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003echeckpoint starting\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003eshutdown immediate\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e..2024-01-12 00:52:59.314 UTC [49] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"nt\"\u003echeckpoint complete\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003ewrote 918 buffers (5.6%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.434 s, sync=0.014 s, total=2.279 s; sync files=250, longest=0.007 s, average=0.001 s; distance=11271 kB, estimate=11271 kB\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:59.318 UTC [49] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003edatabase system is shut down\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e \u003c/span\u003e\u003cspan class=\"l\"\u003edone\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003eserver stopped\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"l\"\u003ePostgreSQL init process complete; ready for start up.\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:59.385 UTC [1] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003estarting PostgreSQL 15.0 (Debian 15.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:59.385 UTC [1] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003elistening on IPv4 address \u0026#34;0.0.0.0\u0026#34;, port 5432\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:59.385 UTC [1] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003elistening on IPv6 address \u0026#34;::\u0026#34;, port 5432\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:59.389 UTC [1] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003elistening on Unix socket \u0026#34;/var/run/postgresql/.s.PGSQL.5432\u0026#34;\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:59.398 UTC [1] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003ecould not open usermap file \u0026#34;/config/pg_ident.conf\u0026#34;: No such file or directory\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:59.404 UTC [67] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003edatabase system was shut down at 2024-01-12 00:52:59 UTC\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nt\"\u003e2024-01-12 00:52:59.415 UTC [1] LOG\u003c/span\u003e\u003cspan class=\"p\"\u003e:\u003c/span\u003e\u003cspan class=\"w\"\u003e  \u003c/span\u003e\u003cspan class=\"l\"\u003edatabase system is ready to accept connections\u003c/span\u003e\u003cspan class=\"w\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eConclusion\u003c/strong\u003e\u003c/p\u003e","title":"PostgreSQL: Deploy into K8S"},{"content":"Last post I started a hands-on session with PostgreSQL installation on both Docker and Docker Compose, explored important PostgreSQL configuration, and was able to mount persistent volumes and config files to customize PostgreSQL.\nIn this post, I will set up a second PostgreSQL instance to establish primary and standby replication for PostgreSQL HA, using some pg tools. Lastly, we will test failover by shutting down the primary and promoting the standby instance.\nTo achieve this, follow these steps:\nSetup Docker network and Create Replication User in Primary instance To establish PostgreSQL replication, it is necessary to set unique data volumes for data between instances and unique config files for each instance.\n# create both primary and standby folders root@ubt-server:~# mkdir postgres-1 root@ubt-server:~# mkdir postgres-2 # move previous post config file to postgres-1 and postgres-2 root@ubt-server:~# cp -r config/* postgres-1/config/ root@ubt-server:~# mv config/* postgres-2/config/ # create docker network so PostgreSQL containers on the same network root@ubt-server:~/postgres-1/config# docker network create postgres 9891c6d9cd3bdbeea2fdfc2b287c868a0f67a3cec7f2939e1299cfb0ae293021 # run primary root@ubt-server:~/postgres-1# docker run -it --rm --name postgres-1 \\ --net postgres \\ -e POSTGRES_USER=postgresadmin \\ -e POSTGRES_PASSWORD=admin123 \\ -e POSTGRES_DB=postgresdb \\ -e PGDATA=\u0026#34;/data\u0026#34; \\ -v ${PWD}/postgres-1/pgdata:/data \\ -v ${PWD}/postgres-1/config:/config \\ -v ${PWD}/postgres-1/archive:/mnt/server/archive \\ -p 5000:5432 postgres:15.0 \\ -c \u0026#39;config_file=/config/postgresql.conf\u0026#39; 2024-01-11 13:15:07.762 UTC [1] LOG: starting PostgreSQL 15.0 (Debian 15.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit 2024-01-11 13:15:07.763 UTC [1] LOG: listening on IPv4 address \u0026#34;0.0.0.0\u0026#34;, port 5432 2024-01-11 13:15:07.763 UTC [1] LOG: listening on IPv6 address \u0026#34;::\u0026#34;, port 5432 2024-01-11 13:15:07.764 UTC [1] LOG: listening on Unix socket \u0026#34;/var/run/postgresql/.s.PGSQL.5432\u0026#34; 2024-01-11 13:15:07.766 UTC [63] LOG: database system was shut down at 2024-01-11 13:15:07 UTC 2024-01-11 13:15:07.768 UTC [1] LOG: database system is ready to accept connections # create Replication User, chown postgres to access archive folder root@ubt-server:~# docker exec -it postgres-1 bash root@2bf6be3e4fa8:/# createuser -U postgresadmin -P -c 5 --replication replicationUser Enter password for new role: Enter it again: root@2bf6be3e4fa8:/# chown postgres:postgres /mnt/server/archive # add replication into configuration file root@ubt-server:~/postgres-1/config# vim pg_hba.conf # TYPE DATABASE USER ADDRESS METHOD # add replication user host replication replicationUser 0.0.0.0/0 md5 Enable Write-Ahead Log, archive, and Replication Write-Ahead Log (WAL) is a PostgreSQL data integrity mechanism of writing transaction logs to a file. PostgreSQL does not accept the transaction until it has been written to the transaction log and flushed to disk. This ensures that if there is a crash in the system, the database can be recovered from the transaction log.\nSo, we need to add the following lines into postgresql.conf to enable Write-Ahead Log and set up the replication and archive.\nroot@ubt-server:~/postgres-1/config# vim postgresql.conf # replication wal_level = replica archive_mode = on archive_command = \u0026#39;test ! -f /mnt/server/archive/%f \u0026amp;\u0026amp; cp %p /mnt/server/archive/%f\u0026#39; max_wal_senders = 3 Set up standby instance and validate replication Here, we need to use the tool pg_basebackup to create a standby instance by taking a primary instance base backup, entering the \u0026ldquo;replicationUser\u0026rdquo; password, and then backing up the postgres-1 database into the postgres-2 pgdata folder.\nroot@ubt-server:~# docker run -it --rm --net postgres -v ${PWD}/postgres-2/pgdata:/data --entrypoint /bin/bash postgres:15.0 root@56e38636a87b:/# pg_basebackup -h postgres-1 -p 5432 -U replicationUser -D /data/ -Fp -Xs -R Password: Now, we start the standby instance. See the log below. Postgres-2 is entering standby mode, ready to accept read-only connections, and starting streaming WAL from the primary.\nroot@ubt-server:~# docker run -it --rm --name postgres-2 --net postgres -e POSTGRES_USER=postgresadmin -e POSTGRES_PASSWORD=admin123 -e POSTGRES_DB=postgresdb -e PGDATA=\u0026#34;/data\u0026#34; -v ${PWD}/postgres-2/pgdata:/data -v ${PWD}/postgres-2/config:/config -v ${PWD}/postgres-2/archive:/mnt/server/archive -p 5001:5432 postgres:15.0 -c \u0026#39;config_file=/config/postgresql.conf\u0026#39; PostgreSQL Database directory appears to contain a database; Skipping initialization 2024-01-11 14:25:21.008 UTC [1] LOG: starting PostgreSQL 15.0 (Debian 15.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit 2024-01-11 14:25:21.008 UTC [1] LOG: listening on IPv4 address \u0026#34;0.0.0.0\u0026#34;, port 5432 2024-01-11 14:25:21.008 UTC [1] LOG: listening on IPv6 address \u0026#34;::\u0026#34;, port 5432 2024-01-11 14:25:21.010 UTC [1] LOG: listening on Unix socket \u0026#34;/var/run/postgresql/.s.PGSQL.5432\u0026#34; 2024-01-11 14:25:21.012 UTC [29] LOG: database system was interrupted; last known up at 2024-01-11 14:21:16 UTC 2024-01-11 14:25:21.017 UTC [29] LOG: entering standby mode 2024-01-11 14:25:21.026 UTC [29] LOG: redo starts at 0/5000028 2024-01-11 14:25:21.026 UTC [29] LOG: consistent recovery state reached at 0/5000100 2024-01-11 14:25:21.026 UTC [1] LOG: database system is ready to accept read-only connections 2024-01-11 14:25:21.034 UTC [30] LOG: started streaming WAL from primary at 0/6000000 on timeline 1 Test replication and failover First, let\u0026rsquo;s test the replication by logging into postgres-1, creating a zack_customers table, and validating from postgres-2.\n# bash into postgres-1, create zack_customers table root@ubt-server:~# docker exec -it postgres-1 bash root@06dd98085df7:/# psql --username=postgresadmin postgresdb psql (15.0 (Debian 15.0-1.pgdg110+1)) Type \u0026#34;help\u0026#34; for help. postgresdb=# CREATE TABLE zack_customers (zackname text, z_customer_id serial, date_created timestamp); CREATE TABLE postgresdb=# \\dt List of relations Schema | Name | Type | Owner --------+----------------+-------+--------------- public | zack_customers | table | postgresadmin (1 row) postgresdb=# \\q root@06dd98085df7:/# exit exit # bash into postgres-2, validate zack_customers table root@ubt-server:~/postgres-2/pgdata# docker exec -it postgres-2 bash root@b333ff290624:/# psql --username=postgresadmin postgresdb psql (15.0 (Debian 15.0-1.pgdg110+1)) Type \u0026#34;help\u0026#34; for help. postgresdb=# \\dt List of relations Schema | Name | Type | Owner --------+----------------+-------+--------------- public | zack_customers | table | postgresadmin (1 row) postgresdb=# \\q root@b333ff290624:/# exit exit Now, we simulate failover by using the load balancer tool pgctl, shutting down the primary instance, and then promoting the standby read-only instance into a read-write instance.\n# shut down the primary instance root@ubt-server:~# docker rm -f postgres-1 postgres-1 # exec standby try to create a table zack_customers_2, get error as it\u0026#39;s read-only root@ubt-server:~# docker exec -it postgres-2 bash root@b333ff290624:/# psql --username=postgresadmin postgresdb psql (15.0 (Debian 15.0-1.pgdg110+1)) Type \u0026#34;help\u0026#34; for help. postgresdb=# CREATE TABLE zack_customers_2 (zackname text, z_customer_id serial, date_created timestamp); ERROR: cannot execute CREATE TABLE in a read-only transaction postgresdb-# \\q # promote postgres-2 from standby to primary root@b333ff290624:/# runuser -u postgres -- pg_ctl promote waiting for server to promote.... done server promoted # exec to create table zack_customers_2, this time works root@b333ff290624:/# psql --username=postgresadmin postgresdb psql (15.0 (Debian 15.0-1.pgdg110+1)) Type \u0026#34;help\u0026#34; for help. postgresdb=# CREATE TABLE zack_customers_2 (zackname text, z_customer_id serial, date_created timestamp); CREATE TABLE postgresdb=# \\dt List of relations Schema | Name | Type | Owner --------+-------------------+-------+--------------- public | zack_customers | table | postgresadmin public | zack_customers_2 | table | postgresadmin (2 rows) postgresdb=# \\q root@b333ff290624:/# exit exit root@ubt-server:~# Conclusion\nNow we are able to run PostgreSQL primary and standby instances to test replication and failover. In the next blog, I will explore how to deploy a single PostgreSQL on Kubernetes.\n","permalink":"https://zackblog.work/posts/postgresql-replication-failover/","summary":"\u003cp\u003eLast post I started a hands-on session with PostgreSQL installation on both Docker and Docker Compose, explored important PostgreSQL configuration, and was able to mount persistent volumes and config files to customize PostgreSQL.\u003c/p\u003e\n\u003cp\u003eIn this post, I will set up a second PostgreSQL instance to establish primary and standby replication for PostgreSQL HA, using some pg tools. Lastly, we will test failover by shutting down the primary and promoting the standby instance.\u003c/p\u003e","title":"PostgreSQL: Replication \u0026 Failover"},{"content":"PostgreSQL is a very popular open-source relational database management system (RDBMS), known for its extensibility and feature-rich capabilities, making it suitable for mission-critical applications. Not to mention its active PostgreSQL community.\nIn the upcoming posts, I will start a series of PostgreSQL studies to:\nExplore PostgreSQL main features, installation, and basic administration tasks. Deploy a PostgreSQL cluster onto K8S with PostgreSQL Operator, validate backup and rolling upgrades. Create a simple Flash microservice application to connect to the PostgreSQL cluster and validate failover. Integrate the whole deployment into a CICD pipeline for automation. Create AWS RDS PostgreSQL, with S3 Block storage as a replica. By the end of the series, we should be able to have a comprehensive understanding of PostgreSQL from a DevOps perspective.\nPostgreSQL Basic\nTo begin, we will:\nInstall PostgreSQL as a docker container on a local Ubuntu machine to get it up and running. # ubuntu install docker root@ubt-server:~# curl -fsSL https://get.docker.com -o get-docker.sh root@ubt-server:~# sh get-docker.sh root@ubt-server:~# docker --version root@ubt-server:~# systemctl enable docker # install PostgreSQL 15.0 root@ubt-server:~# docker run --name zack-postgres -e POSTGRES_PASSWORD=password -d postgres:15.0 root@ubt-server:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b4fc638dfde3 postgres:15.0 \u0026#34;docker-entrypoint.s…\u0026#34; 7 seconds ago Up 6 seconds 5432/tcp zack-postgres Run a simple PostgreSQL database with Docker Compose # create docker-compose.yaml and run postgres and adminer from docker-compose root@ubt-server:~# vim docker-compose.yaml version: \u0026#39;3.1\u0026#39; services: db: image: postgres:15.0 restart: always environment: POSTGRES_PASSWORD: password ports: - 5000:5432 adminer: image: adminer restart: always ports: - 8080:8080 # run docker compose root@ubt-server:~# docker compose up Validate from the Adminer web console localhost:8080 with the password set in the environment variables Persist data to mount the PostgreSQL container volume, validate data table after start/stop container. PostgreSQL stores its data by default under /var/lib/postgresql/data. Here, we create a /pgdata folder on the local machine to mount PostgreSQL\u0026rsquo;s default volume. # create local Persist data directory /pgdata root@ubt-server:~# mkdir pgdata # run PostgreSQL to mount local Persist data and Bind a different port root@ubt-server:~# docker run -d -it --rm --name zack-postgres2 -e POSTGRES_PASSWORD=password -v ${PWD}/pgdata:/var/lib/postgresql/data -p 5000:5432 postgres:15.0 PostgreSQL Database directory appears to contain a database; Skipping initialization 2024-01-08 00:58:47.540 UTC [1] LOG: starting PostgreSQL 15.0 (Debian 15.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit 2024-01-08 00:58:47.541 UTC [1] LOG: listening on IPv4 address \u0026#34;0.0.0.0\u0026#34;, port 5432 2024-01-08 00:58:47.541 UTC [1] LOG: listening on IPv6 address \u0026#34;::\u0026#34;, port 5432 2024-01-08 00:58:47.542 UTC [1] LOG: listening on Unix socket \u0026#34;/var/run/postgresql/.s.PGSQL.5432\u0026#34; 2024-01-08 00:58:47.545 UTC [28] LOG: database system was shut down at 2024-01-08 00:57:45 UTC 2024-01-08 00:58:47.547 UTC [1] LOG: database system is ready to accept connections Connect to DB container and validate # enter the container root@ubt-server:~# docker exec -it zack-postgres2 bash # login to postgres root@d7386c566872:/# psql -h localhost -U postgres psql (15.0 (Debian 15.0-1.pgdg110+1)) Type \u0026#34;help\u0026#34; for help. # create a table postgres=# CREATE TABLE customers (firstname text,lastname text, customer_id serial); CREATE TABLE # add record postgres=# INSERT INTO customers (firstname, lastname) VALUES ( \u0026#39;Bob\u0026#39;, \u0026#39;Smith\u0026#39;); INSERT 0 1 # show table postgres=# \\dt List of relations Schema | Name | Type | Owner --------+-----------+-------+---------- public | customers | table | postgres (1 row) # get records postgres=# SELECT * FROM customers; firstname | lastname | customer_id -----------+----------+------------- Bob | Smith | 1 (1 row) # quit postgres=# \\q # exit db container root@d7386c566872:/# exit exit Add persist data in Docker Compose and run PostgreSQL from compose # add persist data folder in compose yaml root@ubt-server:~# vim docker-compose.yaml version: \u0026#39;3.1\u0026#39; services: db: image: postgres:15.0 restart: always environment: POSTGRES_PASSWORD: admin123 ports: - 5000:5432 volumes: - ./pgdata:/var/lib/postgresql/data adminer: image: adminer restart: always ports: - 8080:8080 root@ubt-server:~# docker compose up Validate the previous table and record from the Adminer console. Table and record still there because of the persistent data mount. PostgreSQL Configuration\nBefore jumping into replication, it is more important to explore the PostgreSQL configuration files to have a better understanding of its important config, take the default conf files out of a running database and learn them. Then, mount these conf files into the container, telling PostgreSQL to use our own configuration files to perform our preferred way.\nTo achieve this, we need the db user \u0026ldquo;postgres\u0026rdquo; to have an ID of 999 with access to custom conf files.\nroot@ubt-server:~/pgdata# chown 999:999 config/postgresql.conf root@ubt-server:~/pgdata# chown 999:999 config/pg_hba.conf root@ubt-server:~/pgdata# chown 999:999 config/pg_ident.conf root@ubt-server:~/pgdata# ll *.conf -rw------- 1 lxd docker 4821 May 8 00:30 pg_hba.conf -rw------- 1 lxd docker 1636 May 8 00:30 pg_ident.conf -rw------- 1 lxd docker 88 May 8 00:30 postgresql.auto.conf -rw------- 1 lxd docker 29525 May 8 00:30 postgresql.conf root@ubt-server:~# mkdir config root@ubt-server:~# cd config/ root@ubt-server:~# cp *.conf /config root@ubt-server:~/pgdata# chown 999:999 config/postgresql.conf root@ubt-server:~/pgdata# chown 999:999 config/pg_hba.conf root@ubt-server:~/pgdata# chown 999:999 config/pg_ident.conf The official PostgreSQL documentation explains those configuration files as below:\npg_hba.conf: This file stands for \u0026ldquo;PostgreSQL Host-Based Authentication.\u0026rdquo; It controls client authentication based on the host and user information. It specifies which hosts are allowed to connect to the PostgreSQL server, which databases and users they can access, and what authentication methods they must use. It\u0026rsquo;s a crucial security measure for controlling access to the PostgreSQL server.\npg_ident.conf: This file, \u0026ldquo;PostgreSQL Identification Mapping,\u0026rdquo; allows defining mappings between external (e.g., operating system) and internal (PostgreSQL) user names.\npostgresql.conf: Main configuration file for PostgreSQL which contains global settings to tailor its behavior to specific requirements and environment. Create custom config file\nNow we can adjust the command by adding environment variables to run PostgreSQL from Docker and Docker Compose using our custom conf files.\nroot@ubt-server:~# vim docker-compose.yaml version: \u0026#39;3.1\u0026#39; services: db: container_name: postgres image: postgres:15.0 # important: passing argument to postgres container to tell where conf file is located, to match the custom conf file we created before when DB initiated command: \u0026#34;postgres -c config_file=/config/postgresql.conf\u0026#34; environment: POSTGRES_USER: \u0026#34;postgresadmin\u0026#34; POSTGRES_PASSWORD: \u0026#34;admin123\u0026#34; POSTGRES_DB: \u0026#34;postgresdb\u0026#34; PGDATA: \u0026#34;/data\u0026#34; volumes: - ./pgdata:/data - ./config:/config/ ports: - 5000:5432 adminer: image: adminer restart: always ports: - 8080:8080 root@ubt-server:~# docker run -it --rm --name postgres -e POSTGRES_USER=postgresadmin -e POSTGRES_PASSWORD=admin123 -e POSTGRES_DB=postgresdb -e PGDATA=\u0026#34;/data\u0026#34; -v ${PWD}/pgdata:/data -v ${PWD}/config:/config -p 5000:5432 postgres:15.0 -c \u0026#39;config_file=/config/postgresql.conf\u0026#39; PostgreSQL Database directory appears to contain a database; Skipping initialization 2024-01-10 10:40:44.685 UTC [1] LOG: starting PostgreSQL 15.0 (Debian 15.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit 2024-01-10 10:40:44.685 UTC [1] LOG: listening on IPv4 address \u0026#34;0.0.0.0\u0026#34;, port 5432 2024-01-10 10:40:44.685 UTC [1] LOG: listening on IPv6 address \u0026#34;::\u0026#34;, port 5432 2024-01-10 10:40:44.685 UTC [1] LOG: listening on Unix socket \u0026#34;/var/run/postgresql/.s.PGSQL.5432\u0026#34; 2024-01-10 10:40:44.688 UTC [28] LOG: database system was shut down at 2024-01-10 10:30:42 UTC 2024-01-10 10:40:44.690 UTC [1] LOG: database system is ready to accept connections root@ubt-server:~# docker compose up -d WARN[0000] /root/docker-compose.yaml: `version` is obsolete [+] Running 2/2 ✔ Container postgres Started 0.4s ✔ Container root-adminer-1 Started Conclusion\nNow we can run a PostgreSQL container from Docker and Docker Compose with persistent data and custom configuration mounted into the container. In the next blog, we will discover primary and standby replication, WAL (write-ahead log) options.\n","permalink":"https://zackblog.work/posts/postgresql-get-started/","summary":"\u003cp\u003ePostgreSQL is a very popular open-source relational database management system (RDBMS), known for its extensibility and feature-rich capabilities, making it suitable for mission-critical applications. Not to mention its active PostgreSQL community.\u003c/p\u003e\n\u003cp\u003eIn the upcoming posts, I will start a series of PostgreSQL studies to:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eExplore PostgreSQL main features, installation, and basic administration tasks.\u003c/li\u003e\n\u003cli\u003eDeploy a PostgreSQL cluster onto K8S with PostgreSQL Operator, validate backup and rolling upgrades.\u003c/li\u003e\n\u003cli\u003eCreate a simple Flash microservice application to connect to the PostgreSQL cluster and validate failover.\u003c/li\u003e\n\u003cli\u003eIntegrate the whole deployment into a CICD pipeline for automation.\u003c/li\u003e\n\u003cli\u003eCreate AWS RDS PostgreSQL, with S3 Block storage as a replica.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eBy the end of the series, we should be able to have a comprehensive understanding of PostgreSQL from a DevOps perspective.\u003c/p\u003e","title":"PostgreSQL: Get Started"},{"content":"Last post we were able to deploy Istio and manage traffic for a book review microservice application. This session we will dive deeper into Istio for its add-on Jaeger for Microservice tracing.\nJaeger is an open-source end-to-end distributed tracing tool to monitor and troubleshoot the performance of microservices-based distributed systems by providing insights into the latency and other performance metrics.\nTrace A trace represents the entire journey of a request or transaction as it propagates through various services and components of a distributed system. It captures the path the request takes, including all the microservices it interacts with, from start to finish. A trace is composed of multiple spans.\nSpan A span is a single unit of work within a trace. It represents an individual operation within a microservice, such as a function call, database query, or external API request. Each span contains metadata such as:\nPreparation for Hands on\nHere we will use Fleetman GPS simulator microservice application as an example to explore Jaeger and its capabilities.\nEnable Istio sidecar injection for existing deployment # label the namespace to allow istio sidecar container injection [root@freeipa-server ~]# kubectl label namespace default istio-injection=enabled --overwrite # Redeploy fleetman application [root@freeipa-server ~]# kubectl rollout restart deployment -n default Validate pod for Istio sidecar injection, also check service status in Kiali [root@freeipa-server ~]# kubectl get po NAME READY STATUS RESTARTS AGE api-gateway-58f978dfc6-phdgp 2/2 Running 4 (30m ago) 17h position-simulator-6f5df9b447-57d75 2/2 Running 4 (30m ago) 17h position-tracker-6698577777-fz52v 2/2 Running 4 (30m ago) 17h staff-service-59987757dc-mfm2t 2/2 Running 4 (30m ago) 17h vehicle-telemetry-56c7f8d859-jvtpj 2/2 Running 4 (30m ago) 17h webapp-59bc7757fb-trnnv 2/2 Running 6 (30m ago) 17h How Jaeger Works\nWhen a request enters a microservice (e.g., a user making a request to a frontend service), the tracing library creates a span and assigns it a trace ID. As the request propagates through other services, additional spans are created and linked to the same trace ID. Each span is recorded with its respective start and end timestamps, operation name, and other metadata.\nThe Jaeger UI provides a way to visualize traces. Users can search for traces based on various criteria (e.g., service name, operation name, duration) and view the detailed structure of individual traces, like durations of time spent between microservices.\nAs the request flows through different services, each service creates additional or child spans. (e.g., The frontend service might call an authentication service. Then the authentication service calls a user service, thus Jaeger will create 2 child spans)\nLatency and Performance Analysis\nBy examining the durations of each span, if a particular span has a long duration, that service might be a bottleneck. If spans have significant gaps between them, network latency or queuing delays might be an issue. So we can identify which part of the request is taking the most time and investigate further to optimize performance.\nManage routing in each service from Kiali\nManaging routing in Istio can be done either through the Kiali console or by defining VirtualServices and DestinationRules using Kubernetes YAML manifests. Here from the Kiali console, we have the visualization of each service\u0026rsquo;s traffic flow, metrics, and dependencies between services in real-time.\nBy creating weighted routing or suspending traffic, Kiali will create its own VirtualServices and DestinationRules to manage the traffic.\nAdd timeout in Istio virtual service YAML\nTo add a timeout into Istio virtual service YAML and ensure it works with Jaeger for better visibility and efficiency in the microservice architecture.\nBy adding this timeout to 3s for below \u0026ldquo;api-gateway\u0026rdquo; virtual service, Jaeger trace will avoid long response times when a request calls the api-gateway. Any response longer than 3s will return an HTTP timeout, which adds visibility to the Jaeger UI to determine if the request was successful or not.\napiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: api-gateway spec: hosts: - api-gateway http: - route: - destination: host: api-gateway port: number: 80 timeout: 3s # 3 seconds timeout added Conclusion\nIn this session, we deep-dived into Istio\u0026rsquo;s add-on Jaeger for distributed tracing, which Jaeger facilitates, involving tracking requests as they flow through various services and components of an application. This helps identify bottlenecks, understand service dependencies, and improve overall performance.\nIn the next post, I will see how to use Istio and Kiali to run some Canary Releases, Blue-Green deployment, Rolling Updates, and A/B Testing.\n","permalink":"https://zackblog.work/posts/istio-distributed-tracing-with-jaeger/","summary":"\u003cp\u003eLast post we were able to deploy \u003ca href=\"/posts/istio-traffic-routing/\"\u003eIstio\u003c/a\u003e and manage traffic for a book review microservice application. This session we will dive deeper into Istio for its add-on Jaeger for Microservice tracing.\u003c/p\u003e\n\u003cp\u003eJaeger is an open-source end-to-end distributed tracing tool to monitor and troubleshoot the performance of microservices-based distributed systems by providing insights into the latency and other performance metrics.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eTrace\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eA trace represents the entire journey of a request or transaction as it propagates through various services and components of a distributed system. It captures the path the request takes, including all the microservices it interacts with, from start to finish. A trace is composed of multiple spans.\u003c/p\u003e","title":"Istio: Distributed Tracing with Jaeger"},{"content":"Here we use helm to install istio (istio-base, istiod, istio gateway), then deploy a sample online book store microservice \u0026ldquo;bookinfo\u0026rdquo;. Practise istio tasks include Traffic Management, Observability, Security.\nBookinfo Topology:\nHelm install istio (istiod, istio-ingress) kubectl create namespace istio-system helm pull istio/base helm install istio-base . -n istio-system --set defaultRevision=default helm pull istio/istiod helm install istiod . -n istio-system kubectl create namespace istio-ingress helm pull istio/gateway helm install istio-ingress . -n istio-ingress helm ls -n istio-system NAME NAMESPACE REVISION\tUPDATED STATUS CHART APP VERSION istio-base\tistio-system\t1 2023-12-17 08:14:06.943276388 +0800 CST\tdeployed\tbase-1.20.1 1.20.1 istiod istio-system\t1 2023-12-17 08:15:40.370551503 +0800 CST\tdeployed\tistiod-1.20.1\t1.20.1 helm ls -n istio-ingress NAME NAMESPACE REVISION\tUPDATED STATUS CHART APP VERSION istio-ingress\tistio-ingress\t1 2023-12-17 08:25:07.111999373 +0800 CST\tdeployed\tgateway-1.20.1\t1.20.1 Deploy bookinfo microservice and istio ingressgateway and virtualservice kubectl label namespace istio-system istio-injection=enabled kubectl apply -f https://github.com/istio/istio/blob/master/samples/bookinfo/platform/kube/bookinfo.yaml -oyaml \u0026gt; bookinfo.yaml kubectl apply -f bookinfo.yaml kubectl get po NAME READY STATUS RESTARTS AGE details-v1-698d88b-wmfcb 2/2 Running 0 21m ratings-v1-6484c4d9bb-cb6gx 2/2 Running 0 21m reviews-v1-5b5d6494f4-jrsvc 2/2 Running 0 21m reviews-v2-5b667bcbf8-jgfzj 2/2 Running 0 21m reviews-v3-5b9bd44f4-tmmfz 2/2 Running 0 21m kubectl apply -f https://github.com/istio/istio/blob/master/samples/bookinfo/networking/bookinfo-gateway.yaml -oyaml \u0026gt; bookinfo-gateway.yaml kubectl apply -f bookinfo-gateway.yaml Deploy Kiali, jaeger, grafana, prometheus wget https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/prometheus.yaml wget https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/jaeger.yaml wget https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/grafana.yaml kubectl create -f prometheus.yaml -f jaeger.yaml -f grafana.yaml Visit http://book.istio:31000/productpage, with review (v1, v2, v3) Define destination rules and virtual service for reviews wget https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/networking/destination-rule-all.yaml wget https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/networking/virtual-service-reviews-90-10.yaml kubectl create -f destination-rule-all.yaml -f virtual-service-reviews-90-10.yaml # route v1 10% and v3 90% kubectl scale deployment reviews-v2 -n istio-system --replicas=0 # scale down v2 to 0 kubectl get dr -A NAMESPACE NAME HOST AGE istio-system details details 6m56s istio-system productpage productpage 6m56s istio-system ratings ratings 6m56s istio-system reviews reviews 6m56s kubectl get vs -A NAMESPACE NAME GATEWAYS HOSTS AGE istio-system bookinfo [\u0026#34;bookinfo-gateway\u0026#34;] [\u0026#34;book.istio\u0026#34;] 19h istio-system reviews [\u0026#34;reviews\u0026#34;] 5m26s Spec: Hosts: reviews Http: Route: Destination: Host: reviews Subset: v1 Weight: 10 Destination: Host: reviews Subset: v3 Weight: 90 Refresh the bookinfo webpage, test Traffic route weight as below:\n90% traffic for reviews v3 vs 10% traffic for review v1\n","permalink":"https://zackblog.work/posts/istio-traffic-routing/","summary":"\u003cp\u003eHere we use helm to install istio (istio-base, istiod, istio gateway), then deploy a sample online book store microservice \u0026ldquo;bookinfo\u0026rdquo;. Practise istio tasks include Traffic Management, Observability, Security.\u003c/p\u003e\n\u003cp\u003eBookinfo Topology:\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"/images/bookinfo.png\"\u003e\u003cimg alt=\"image tooltip here\" loading=\"lazy\" src=\"/images/bookinfo.png\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eHelm install istio (istiod, istio-ingress)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl create namespace istio-system\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm pull istio/base\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm install istio-base . -n istio-system --set \u003cspan class=\"nv\"\u003edefaultRevision\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003edefault\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm pull istio/istiod\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm install istiod . -n istio-system\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl create namespace istio-ingress\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm pull istio/gateway\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm install istio-ingress . -n istio-ingress\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm ls -n istio-system\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eNAME      \tNAMESPACE   \tREVISION\tUPDATED                                \tSTATUS  \tCHART        \tAPP VERSION\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eistio-base\tistio-system\t\u003cspan class=\"m\"\u003e1\u003c/span\u003e       \t2023-12-17 08:14:06.943276388 +0800 CST\tdeployed\tbase-1.20.1  \t1.20.1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eistiod    \tistio-system\t\u003cspan class=\"m\"\u003e1\u003c/span\u003e       \t2023-12-17 08:15:40.370551503 +0800 CST\tdeployed\tistiod-1.20.1\t1.20.1\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm ls -n istio-ingress\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eNAME         \tNAMESPACE    \tREVISION\tUPDATED                                \tSTATUS  \tCHART         \tAPP VERSION\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eistio-ingress\tistio-ingress\t\u003cspan class=\"m\"\u003e1\u003c/span\u003e       \t2023-12-17 08:25:07.111999373 +0800 CST\tdeployed\tgateway-1.20.1\t1.20.1\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eDeploy bookinfo microservice and istio ingressgateway and virtualservice\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl label namespace istio-system istio-injection\u003cspan class=\"o\"\u003e=\u003c/span\u003eenabled\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl apply -f https://github.com/istio/istio/blob/master/samples/bookinfo/platform/kube/bookinfo.yaml -oyaml \u0026gt; bookinfo.yaml\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl apply -f bookinfo.yaml\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get po\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eNAME                                                     READY   STATUS    RESTARTS       AGE\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003edetails-v1-698d88b-wmfcb                                 2/2     Running   \u003cspan class=\"m\"\u003e0\u003c/span\u003e              21m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eratings-v1-6484c4d9bb-cb6gx                              2/2     Running   \u003cspan class=\"m\"\u003e0\u003c/span\u003e              21m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ereviews-v1-5b5d6494f4-jrsvc                              2/2     Running   \u003cspan class=\"m\"\u003e0\u003c/span\u003e              21m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ereviews-v2-5b667bcbf8-jgfzj                              2/2     Running   \u003cspan class=\"m\"\u003e0\u003c/span\u003e              21m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ereviews-v3-5b9bd44f4-tmmfz                               2/2     Running   \u003cspan class=\"m\"\u003e0\u003c/span\u003e              21m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl apply -f https://github.com/istio/istio/blob/master/samples/bookinfo/networking/bookinfo-gateway.yaml -oyaml \u0026gt; bookinfo-gateway.yaml\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl apply -f bookinfo-gateway.yaml\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eDeploy Kiali, jaeger, grafana, prometheus\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ewget https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/prometheus.yaml\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ewget https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/jaeger.yaml\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ewget https://raw.githubusercontent.com/istio/istio/release-1.20/samples/addons/grafana.yaml\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl create -f prometheus.yaml -f jaeger.yaml -f grafana.yaml\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eVisit \u003ca href=\"http://book.istio:31000/productpage\" target=\"_blank\" rel=\"noopener noreferrer\"\u003ehttp://book.istio:31000/productpage\u003c/a\u003e, with review (v1, v2, v3)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003ca href=\"/images/kiali.png\"\u003e\u003cimg alt=\"image tooltip here\" loading=\"lazy\" src=\"/images/kiali.png\"\u003e\u003c/a\u003e\u003c/p\u003e","title":"Istio: Traffic Routing"},{"content":"My manage said we have many ubuntu 16.04, can you believe?\nEvery single Ubuntu LTS comes with 5 years of standard support. During those five years, bug fixes and security patches will be provided. Ubuntu 18.04 ‘Bionic Beaver’ is reaching End of Standard Support this May. So today we are going to run in-place upgrade for Ubuntu 18.04 LTS to 22.04 LTS.\nPre-upgrade checklist\nValidate current OS version and running service (nginx) # current OS version root@ubuntu-test:~# cat /etc/os-release NAME=\u0026#34;Ubuntu\u0026#34; VERSION=\u0026#34;18.04.6 LTS (Bionic Beaver)\u0026#34; ID=ubuntu ID_LIKE=debian PRETTY_NAME=\u0026#34;Ubuntu 18.04.6 LTS\u0026#34; VERSION_ID=\u0026#34;18.04\u0026#34; HOME_URL=\u0026#34;https://www.ubuntu.com/\u0026#34; SUPPORT_URL=\u0026#34;https://help.ubuntu.com/\u0026#34; BUG_REPORT_URL=\u0026#34;https://bugs.launchpad.net/ubuntu/\u0026#34; PRIVACY_POLICY_URL=\u0026#34;https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\u0026#34; VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic # nginx service status root@ubuntu-test:~# echo \u0026#34;ubuntu-inplace-upgrade zack-testing-nginx-service!!\u0026#34; \u0026gt;\u0026gt; /var/www/html/index.html root@ubuntu-test:~# systemctl restart nginx root@ubuntu-test:~# curl localhost ubuntu-inplace-upgrade zack-testing-nginx-service!! Fully update the system # update system root@ubuntu-test:~# sudo apt update Hit:1 http://au.archive.ubuntu.com/ubuntu bionic InRelease Hit:2 http://au.archive.ubuntu.com/ubuntu bionic-updates InRelease Hit:3 http://au.archive.ubuntu.com/ubuntu bionic-backports InRelease Hit:4 http://au.archive.ubuntu.com/ubuntu bionic-security InRelease Reading package lists... Done Building dependency tree Reading state information... Done All packages are up to date. root@ubuntu-test:~# sudo apt upgrade -y Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. # reboot system before upgrade root@ubuntu-test:~# sudo do-release-upgrade Checking for a new Ubuntu release You have not rebooted after updating a package which requires a reboot. Please reboot before upgrading. root@ubuntu-test:~# reboot Connection closing...Socket close. Take full system backup Here I took a VM snapshot before upgrade\n18.04 to 22.04 upgrade\nThere is no direct upgrade path from 18.04 LTS to Ubuntu 22.04 LTS, so we go Ubuntu 20.04 LTS first and then to Ubuntu 22.04 LTS.\nFirst upgrade to 20.04 # run upgrade root@ubuntu-test:~# sudo do-release-upgrade This session appears to be running under ssh. It is not recommended to perform a upgrade over ssh currently because in case of failure it is harder to recover. If you continue, an additional ssh daemon will be started at port \u0026#39;1022\u0026#39;. Do you want to continue? Continue [yN] y Starting additional sshd Calculating the changes MarkInstall libfwupdplugin1:amd64 \u0026lt; none -\u0026gt; 1.5.11-0ubuntu1~20.04.2 @un uN Ib \u0026gt; FU=1 Installing libxmlb1 as Depends of libfwupdplugin1 MarkInstall libxmlb1:amd64 \u0026lt; none -\u0026gt; 0.1.15-2ubuntu1~20.04.1 @un uN \u0026gt; FU=0 Do you want to start the upgrade? Continue [yN] Details [d]y Allow service restart during upgrade Reboot after upgrade The installation and removing of packages may take some time, then reboot is required after upgrade completion.\nPurging configuration files for ebtables (2.0.11-3build1) ... Purging configuration files for python3.6-minimal (3.6.9-1~18.04ubuntu1.12) ... Purging configuration files for mlocate (0.26-3ubuntu3) ... Processing triggers for dbus (1.12.16-2ubuntu2.3) ... Processing triggers for systemd (245.4-4ubuntu3.23) ... System upgrade is complete. Restart required To finish the upgrade, a restart is required. If you select \u0026#39;y\u0026#39; the system will be restarted. Continue [yN] y Validate OS and service # validate nginx service root@ubuntu-test:~# curl localhost ubuntu-inplace-upgrade zack-testing-nginx-service!! # validate OS version root@ubuntu-test:~# cat /etc/os-release NAME=\u0026#34;Ubuntu\u0026#34; VERSION=\u0026#34;20.04.6 LTS (Focal Fossa)\u0026#34; ID=ubuntu ID_LIKE=debian PRETTY_NAME=\u0026#34;Ubuntu 20.04.6 LTS\u0026#34; VERSION_ID=\u0026#34;20.04\u0026#34; HOME_URL=\u0026#34;https://www.ubuntu.com/\u0026#34; SUPPORT_URL=\u0026#34;https://help.ubuntu.com/\u0026#34; BUG_REPORT_URL=\u0026#34;https://bugs.launchpad.net/ubuntu/\u0026#34; PRIVACY_POLICY_URL=\u0026#34;https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\u0026#34; VERSION_CODENAME=focal UBUNTU_CODENAME=focal Then upgrade to 22.04 root@ubuntu-test:~# sudo apt update Hit:1 http://au.archive.ubuntu.com/ubuntu focal InRelease Hit:2 http://au.archive.ubuntu.com/ubuntu focal-updates InRelease Hit:3 http://au.archive.ubuntu.com/ubuntu focal-backports InRelease Hit:4 http://au.archive.ubuntu.com/ubuntu focal-security InRelease Reading package lists... Done Building dependency tree Reading state information... Done All packages are up to date. root@ubuntu-test:~# sudo apt upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. root@ubuntu-test:~# sudo do-release-upgrade # validate after upgrade root@ubuntu-test:~# curl localhost ubuntu-inplace-upgrade zack-testing-nginx-service!! root@ubuntu-test:~# lsb_release -a No LSB modules are available. Distributor ID:\tUbuntu Description:\tUbuntu 22.04.4 LTS Release:\t22.04 Codename:\tjammy Conclusion\nNow we complete the in-place Ubuntu OS release upgrade from 18.04 to 22.04. The whole upgrade took about 1 hour to finish, with several confirmations required during the upgrade process. The service nginx was running after each upgrade.\n","permalink":"https://zackblog.work/posts/ubuntu-18-04-to-22-04-in-place-upgrade/","summary":"\u003cp\u003eMy manage said we have many ubuntu 16.04, can you believe?\u003c/p\u003e\n\u003cp\u003eEvery single Ubuntu LTS comes with 5 years of standard support. During those five years, bug fixes and security patches will be provided. Ubuntu 18.04 ‘Bionic Beaver’ is reaching End of Standard Support this May. So today we are going to run in-place upgrade for Ubuntu 18.04 LTS to 22.04 LTS.\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"/images/ubt-upg1.png\"\u003e\u003cimg alt=\"image tooltip here\" loading=\"lazy\" src=\"/images/ubt-upg1.png\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003ePre-upgrade checklist\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eValidate current OS version and running service (nginx)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# current OS version\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eroot@ubuntu-test:~# cat /etc/os-release\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eNAME\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Ubuntu\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eVERSION\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;18.04.6 LTS (Bionic Beaver)\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eID\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003eubuntu\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eID_LIKE\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003edebian\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003ePRETTY_NAME\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;Ubuntu 18.04.6 LTS\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eVERSION_ID\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;18.04\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eHOME_URL\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;https://www.ubuntu.com/\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eSUPPORT_URL\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;https://help.ubuntu.com/\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eBUG_REPORT_URL\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;https://bugs.launchpad.net/ubuntu/\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003ePRIVACY_POLICY_URL\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;https://www.ubuntu.com/legal/terms-and-policies/privacy-policy\u0026#34;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eVERSION_CODENAME\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003ebionic\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nv\"\u003eUBUNTU_CODENAME\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003ebionic\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# nginx service status\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eroot@ubuntu-test:~# \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e \u003cspan class=\"s2\"\u003e\u0026#34;ubuntu-inplace-upgrade zack-testing-nginx-service!!\u0026#34;\u003c/span\u003e  \u0026gt;\u0026gt; /var/www/html/index.html\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eroot@ubuntu-test:~# systemctl restart nginx\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eroot@ubuntu-test:~# curl localhost\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eubuntu-inplace-upgrade zack-testing-nginx-service!!\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eFully update the system\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# update system\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eroot@ubuntu-test:~# sudo apt update\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eHit:1 http://au.archive.ubuntu.com/ubuntu bionic InRelease\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eHit:2 http://au.archive.ubuntu.com/ubuntu bionic-updates InRelease\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eHit:3 http://au.archive.ubuntu.com/ubuntu bionic-backports InRelease\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eHit:4 http://au.archive.ubuntu.com/ubuntu bionic-security InRelease\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eReading package lists... Done\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eBuilding dependency tree\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eReading state information... Done\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eAll packages are up to date.\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eroot@ubuntu-test:~# sudo apt upgrade -y\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eReading package lists... Done\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eBuilding dependency tree\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eReading state information... Done\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eCalculating upgrade... Done\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"m\"\u003e0\u003c/span\u003e upgraded, \u003cspan class=\"m\"\u003e0\u003c/span\u003e newly installed, \u003cspan class=\"m\"\u003e0\u003c/span\u003e to remove and \u003cspan class=\"m\"\u003e0\u003c/span\u003e not upgraded.\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# reboot system before upgrade\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eroot@ubuntu-test:~# sudo \u003cspan class=\"k\"\u003edo\u003c/span\u003e-release-upgrade\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eChecking \u003cspan class=\"k\"\u003efor\u003c/span\u003e a new Ubuntu release\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eYou have not rebooted after updating a package which requires a reboot. Please reboot before upgrading.\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eroot@ubuntu-test:~# reboot\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eConnection closing...Socket close.\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eTake full system backup\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eHere I took a VM snapshot before upgrade\u003c/p\u003e","title":"Ubuntu 18.04 to 22.04 in-place upgrade"},{"content":"My current employer using Rancher, due to the vender prefers this??\nSo far many different ways I have used to deploy k8s cluster, each with its own pros and cons.\nhome lab build k8s components (etcd, keepalived, apiserver, scheduler, coreDNS, calico) home lab k8s cluster with kubeadm k8s on AWS using kops and eksctl AWS self-managed k8s cluster directly on ec2 by ansible Rancher Support matrix\nLocal docker Installation\nTo enable Rancher on homelab env, we need a Linux box to run Rancher as docker.\n# Create Persisting rancher data directory to map within the Rancher Docker container ubuntu@ubt-server:/$ mkdir -p /path/to/rancher-data ubuntu@ubt-server:/$ sudo docker run -d --restart=unless-stopped \\ -p 80:80 -p 443:443 \\ -v /path/to/rancher-data:/var/lib/rancher \\ --privileged rancher/rancher:latest d26e32094657b598f61233d0d86e448ab4bfd980763928ca6f298ae0d3774a56 ubuntu@ubt-server:/$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d26e32094657 rancher/rancher:latest \u0026#34;entrypoint.sh\u0026#34; 7 seconds ago Up 6 seconds 0.0.0.0:80-\u0026gt;80/tcp, :::80-\u0026gt;80/tcp, 0.0.0.0:443-\u0026gt;443/tcp, :::443-\u0026gt;443/tcp flamboyant_bassi ubuntu@ubt-server:/$ sudo docker logs d26e32094657 2\u0026gt;\u0026amp;1 | grep \u0026#34;Bootstrap Password:\u0026#34; 2023/06/29 04:07:34 [INFO] Bootstrap Password: zn7nd25rfmkm7kztkfmnk8m84gtlw76gd96sgxz8j2rdm6pnkpqgt9 Rancher Web Portal login\nvia https://localhost/dashboard/home\nImport existing k8s cluster vs create new k8s from rancher console\nUnder \u0026ldquo;cluster management\u0026rdquo;, it supports importing k8s from cloud providers to local k8s, unfortunately my previous k8s cluster is v1.28 which is too high to be imported and managed by this rancher. Hence I will use Rancher to create a new one here. First prepare 3 local Linux VM boxes, come back to Rancher console under cluster management, give name to the new cluster, then run the command to initiate control plane. ubuntu@rancher-master01:~$ curl --insecure -fL https://11.0.1.220/system-agent-install.sh | sudo sh -s - --server https://11.0.1.220 --label \u0026#39;cattle.io/os=linux\u0026#39; --token kx92bf7gxdfx2nfnl6rvw4hlmcwdxcb2rt442vgsvgb7tz29rmd4c6 --ca-checksum 31478d0c1db90313258de7fa258cc60de1a3e67dfb2b285cb682463644474780 --etcd --controlplane % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 30845 0 30845 0 0 2037k 0 --:--:-- --:--:-- --:--:-- 2151k [INFO] Label: cattle.io/os=linux [INFO] Role requested: etcd [INFO] Role requested: controlplane [INFO] Using default agent configuration directory /etc/rancher/agent [INFO] Using default agent var directory /var/lib/rancher/agent [INFO] Determined CA is necessary to connect to Rancher [INFO] Successfully downloaded CA certificate [INFO] Value from https://11.0.1.220/cacerts is an x509 certificate [INFO] Successfully tested Rancher connection [INFO] Downloading rancher-system-agent binary from https://11.0.1.220/assets/rancher-system-agent-amd64 [INFO] Successfully downloaded the rancher-system-agent binary. [INFO] Downloading rancher-system-agent-uninstall.sh script from https://11.0.1.220/assets/system-agent-uninstall.sh [INFO] Successfully downloaded the rancher-system-agent-uninstall.sh script. [INFO] Generating Cattle ID [INFO] Successfully downloaded Rancher connection information [INFO] systemd: Creating service file [INFO] Creating environment file /etc/systemd/system/rancher-system-agent.env [INFO] Enabling rancher-system-agent.service Created symlink /etc/systemd/system/multi-user.target.wants/rancher-system-agent.service → /etc/systemd/system/rancher-system-agent.service. [INFO] Starting/restarting rancher-system-agent.service Updating new machine as a K8S rancher node as control plane. Then join the 2 worker nodes ubuntu@racher-worker01:~$ curl --insecure -fL https://11.0.1.220/system-agent-install.sh | sudo sh -s - --server https://11.0.1.220 --label \u0026#39;cattle.io/os=linux\u0026#39; --token hdsvptc74zvzz62hw9gtt6p7m6nl5k4fs6vk92zqm4f6tvj4tf8m54 --ca-checksum 31478d0c1db90313258de7fa258cc60de1a3e67dfb2b285cb682463644474780 --worker % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 30845 0 30845 0 0 5455k 0 --:--:-- --:--:-- --:--:-- 6024k [INFO] Label: cattle.io/os=linux [INFO] Role requested: worker [INFO] Using default agent configuration directory /etc/rancher/agent [INFO] Using default agent var directory /var/lib/rancher/agent [INFO] Determined CA is necessary to connect to Rancher [INFO] Successfully downloaded CA certificate [INFO] Value from https://11.0.1.220/cacerts is an x509 certificate [INFO] Successfully tested Rancher connection [INFO] Downloading rancher-system-agent binary from https://11.0.1.220/assets/rancher-system-agent-amd64 [INFO] Successfully downloaded the rancher-system-agent binary. [INFO] Downloading rancher-system-agent-uninstall.sh script from https://11.0.1.220/assets/system-agent-uninstall.sh [INFO] Successfully downloaded the rancher-system-agent-uninstall.sh script. [INFO] Generating Cattle ID [INFO] Successfully downloaded Rancher connection information [INFO] systemd: Creating service file [INFO] Creating environment file /etc/systemd/system/rancher-system-agent.env [INFO] Enabling rancher-system-agent.service Created symlink /etc/systemd/system/multi-user.target.wants/rancher-system-agent.service → /etc/systemd/system/rancher-system-agent.service. [INFO] Starting/restarting rancher-system-agent.service Create zackweb and joesite as deployment from Rancher console Conclusion\nNow we can use Rancher to deploy a local k8s cluster based on 3 Linux machines without any trouble just a few commands. Then we will be able to create deployment and service in Rancher console instead of using “kubectl” all the time. It also provides an app market for most popular helm charts ready to be installed with just one click, like Istio and Prometheus. The only downside is, Rancher itself requires resources to run, which may impact the performance and resources on each node. It also brings complexity in upgrade for both Rancher and k8s. Overall, I love the concept and tools that Rancher provides to manage k8s clusters. I will explore more in the next blog.\n","permalink":"https://zackblog.work/posts/move-to-rancher/","summary":"\u003cp\u003eMy current employer using Rancher, due to the vender prefers this??\u003c/p\u003e\n\u003cp\u003eSo far many different ways I have used to deploy k8s cluster, each with its own pros and cons.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ehome lab build k8s components (etcd, keepalived, apiserver, scheduler, coreDNS, calico)\u003c/li\u003e\n\u003cli\u003ehome lab k8s cluster with kubeadm\u003c/li\u003e\n\u003cli\u003ek8s on AWS using kops and eksctl\u003c/li\u003e\n\u003cli\u003eAWS self-managed k8s cluster directly on ec2 by ansible\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eRancher Support matrix\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"/images/rancher1.png\"\u003e\u003cimg alt=\"image tooltip here\" loading=\"lazy\" src=\"/images/rancher1.png\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLocal docker Installation\u003c/strong\u003e\u003c/p\u003e","title":"Move To Rancher"},{"content":"Kubespary is a powerful and highly configurable tool that automates and supports deployment on various cloud providers (AWS, GCE, Azure, etc.) and on-premises infrastructure. In this post I will run its build-in Terraform and ansible code to provision a k8s cluster based on AWS EC2.\nBenefit of Kubespary\nWe can define specific settings in cluster-config.yaml for networking, authentication, and other Kubernetes features, it also supports for multiple networking plugins (Calico, Flannel, Weave Net, etc.), By managing hosts file and rerun the playbook to make add or remove nodes easily, Integrated CI/CD Pipelines and Infrastructure as Code (IaC) with Terraform and Ansible to achieve automation.\nKubespary on Local Lab\nIn local machine, ensure\nGit, Ansible, Python, Terraform installed 5 VM ready with ssh configured clone the Kubespray git repo in a python virtual environment to install Kubespray required packages based on its requirements.txt file create a local folder under inventory homelab-k8s, create a hosts.yaml to define the local nodes, create a cluster-config.yaml to define specific cofiguration of the k8s cluster run the playbook to have a k8s cluster ready to use # ensure 5 vms ready with ssh ssh-keygen -t rsa -b 2048 ssh-copy-id ubuntu@11.0.1.121 # Repeat for other VMs # clone kubespray repo git clone https://github.com/kubernetes-sigs/kubespray.git # create a python virtual environment python3 -m venv kubespray-venv source kubespray-venv/bin/activate # install kubespray required packages cd kubespray pip install -U -r requirements.txt # create a local folder under inventory, manage hosts and cluster configure mkdir -p inventory/homelab-k8s vim inventory/homelab-k8s/hosts.yaml all: hosts: node1: ansible_host: 11.0.1.121 ip: 11.0.1.121 node2: ansible_host: 11.0.1.122 ip: 11.0.1.122 node3: ansible_host: 11.0.1.123 ip: 11.0.1.123 node4: ansible_host: 11.0.1.124 ip: 11.0.1.124 node5: ansible_host: 11.0.1.125 ip: 11.0.1.125 children: kube_control_plane: hosts: node1: kube_node: hosts: node2: node3: node4: node5: etcd: hosts: node1: node2: node3: k8s_cluster: children: kube_control_plane: kube_node: calico_rr: hosts: {} vim inventory/homelab-k8s/cluster-config.yaml cluster_name: kubespray-k8s kube_version: v1.30.4 # run the playbook to create the cluster ansible-playbook -i inventory/homelab-k8s/hosts.yaml -e inventory/homelab-k8s/cluster-config.yaml \\ --user=ubuntu \\ --become \\ --become-user=root \\ cluster.yml Ansible will execute for 15-20mins, then a cluster is ready\nubuntu@node1:~$ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node1 Ready control-plane 28m v1.31.1 11.0.1.121 \u0026lt;none\u0026gt; Ubuntu 22.04.4 LTS 5.15.0-97-generic containerd://1.7.22 node2 Ready \u0026lt;none\u0026gt; 28m v1.31.1 11.0.1.122 \u0026lt;none\u0026gt; Ubuntu 22.04.4 LTS 5.15.0-97-generic containerd://1.7.22 node3 Ready \u0026lt;none\u0026gt; 28m v1.31.1 11.0.1.123 \u0026lt;none\u0026gt; Ubuntu 22.04.4 LTS 5.15.0-97-generic containerd://1.7.22 node4 Ready \u0026lt;none\u0026gt; 28m v1.31.1 11.0.1.124 \u0026lt;none\u0026gt; Ubuntu 22.04.4 LTS 5.15.0-97-generic containerd://1.7.22 node5 Ready \u0026lt;none\u0026gt; 28m v1.31.1 11.0.1.125 \u0026lt;none\u0026gt; Ubuntu 22.04.4 LTS 5.15.0-97-generic containerd://1.7.22 Kubespray on AWS EC2 with Terraform\nnow let\u0026rsquo;s move to cloud practice on AWS with Kubespray,\nMove to terraform aws folder, fill out credentials.tfvars with our AWS credentials Fill desired cluster config like instance type, count and AMI in terraform.tfvars Create 1 ec2 master node and 1 ec2 worker node using terraform. cd kubespray/contrib/terraform/aws/ #AWS Access Key AWS_ACCESS_KEY_ID = \u0026#34;zzzz\u0026#34; #AWS Secret Key AWS_SECRET_ACCESS_KEY = \u0026#34;zzzz\u0026#34; #EC2 SSH Key Name AWS_SSH_KEY_NAME = \u0026#34;zzzzzzzzzz\u0026#34; #AWS Region AWS_DEFAULT_REGION = \u0026#34;ap-southeast-2\u0026#34; vim terraform.tfvars #Global Vars aws_cluster_name = \u0026#34;zack-ec2-k8s-cluster-via-kubespray\u0026#34; #VPC Vars aws_vpc_cidr_block = \u0026#34;10.250.192.0/18\u0026#34; aws_cidr_subnets_private = [\u0026#34;10.250.192.0/20\u0026#34;, \u0026#34;10.250.208.0/20\u0026#34;] aws_cidr_subnets_public = [\u0026#34;10.250.224.0/20\u0026#34;, \u0026#34;10.250.240.0/20\u0026#34;] #Bastion Host aws_bastion_size = \u0026#34;t2.micro\u0026#34; #Kubernetes Cluster aws_kube_master_num = 1 aws_kube_master_size = \u0026#34;t3.small\u0026#34; aws_etcd_num = 3 aws_etcd_size = \u0026#34;t2.medium\u0026#34; aws_kube_worker_num = 1 aws_kube_worker_size = \u0026#34;t3.small\u0026#34; #Settings AWS ELB aws_elb_api_port = 6443 k8s_secure_api_port = 6443 kube_insecure_apiserver_address = \u0026#34;0.0.0.0\u0026#34; default_tags = { # Env = \u0026#34;devtest\u0026#34; # Product = \u0026#34;kubernetes\u0026#34; } inventory_file = \u0026#34;../../../inventory/hosts\u0026#34; # create ec2 by terraform terraform init terraform plan -var-file=credentials.tfvars terraform apply -var-file=credentials.tfvars # Terraform output aws_nlb_api_fqdn = \u0026#34;kubernetes-nlb-devtest-a9cae6ee92bc1b6e.elb.ap-southeast-2.amazonaws.com:6443\u0026#34; bastion_ip = \u0026#34;54.206.92.197\u0026#34; default_tags = tomap({}) etcd = \u0026#34;10.250.202.194\u0026#34; inventory = \u0026#34; [all] ip-10-250-202-194.ap-southeast-2.compute.internal ansible_host=10.250.202.194 ip-10-250-196-14.ap-southeast-2.compute.internal ansible_host=10.250.196.14 bastion ansible_host=54.206.92.197 [bastion] bastion ansible_host=54.206.92.197 [kube_control_plane] ip-10-250-202-194.ap-southeast-2.compute.internal [kube_node] ip-10-250-196-14.ap-southeast-2.compute.internal [etcd] ip-10-250-202-194.ap-southeast-2.compute.internal [calico_rr] [k8s_cluster:children] kube_node kube_control_plane calico_rr [k8s_cluster:vars] apiserver_loadbalancer_domain_name=\u0026#34;kubernetes-nlb-devtest-a9cae6ee92bc1b6e.elb.ap-southeast-2.amazonaws.com\u0026#34; \u0026#34; masters = \u0026#34;10.250.202.194\u0026#34; workers = \u0026#34;10.250.196.14\u0026#34; Ansible will pass the ec2 ip into kubespray hosts file, lets verify the hosts file is correct, configure ssh key agent so the playbook can communicate with AWS ec2, then run the playbook to install kubernetes cluster.\n# verify hosts cd ~/kubespray cat inventory/hosts # enable ssh key agent cat “” \u0026gt; ~/.ssh/zzz.pem eval $(ssh-agent) ssh-add -D ssh-add ~/.ssh/zzz.pem # execute playbook to deploy k8s cluster ansible-playbook -i ./inventory/hosts ./cluster.yml -e ansible_user=ubuntu -b --become-user=root # configure kubeconfig to manage cluster ubuntu@ip-10-250-202-194:~$ mkdir -p /home/ubuntu/.kube ubuntu@ip-10-250-202-194:~$ sudo cp /etc/kubernetes/admin.conf /home/ubuntu/.kube/config ubuntu@ip-10-250-202-194:~$ sudo chown ubuntu:ubuntu /home/ubuntu/.kube/config ubuntu@ip-10-250-202-194:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION ip-10-250-196-14.ap-southeast-2.compute.internal Ready \u0026lt;none\u0026gt; 22m v1.31.1 ip-10-250-202-194.ap-southeast-2.compute.internal Ready control-plane 22m v1.31.1 Conclusion\nNow we are able to create k8s cluster using Kubespray locally and with cloud providers like AWS, The combination of automated deployment, cloud infrastructure management, offers me hands-on experience and valuable skills for building and maintaining scalable, secure applications in diverse environments. This is a great starting point for anyone looking to explore Kubernetes and cloud computing. I hope this tutorial has been helpful in anyone\u0026rsquo;s journey to learn Kubernetes and cloud computing.\n","permalink":"https://zackblog.work/posts/kubespary-with-terraform-on-aws/","summary":"\u003cp\u003eKubespary is a powerful and highly configurable tool that automates and supports deployment on various cloud providers (AWS, GCE, Azure, etc.) and on-premises infrastructure. In this post I will run its build-in Terraform and ansible code to provision a k8s cluster based on AWS EC2.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eBenefit of Kubespary\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe can define specific settings in \u003ccode\u003ecluster-config.yaml\u003c/code\u003e for networking, authentication, and other Kubernetes features, it also supports for multiple networking plugins (Calico, Flannel, Weave Net, etc.), By managing hosts file and rerun the playbook to make add or remove nodes easily, Integrated CI/CD Pipelines and Infrastructure as Code (IaC) with Terraform and Ansible to achieve automation.\u003c/p\u003e","title":"Kubespary with Terraform on AWS"},{"content":"In this article, I will see how to host \u0026ldquo;zackweb\u0026rdquo; as a static web application using the following AWS serverless options:\nS3 static webhosting AWS CDK + CloudFront Prerequisite\nAdd one more step in the existing Github Action workflow to copy the static web content to the newly created S3 bucket # edit github action workflow aws s3 cp ~/zack-gitops-project/zack_blog/_site/* s3://zackweb-serverless/ --recursive # validate content in s3 bucket ubuntu@ip-172-31-26-78:~$ aws s3 ls s3://zackweb-serverless --summarize PRE aboutme/ PRE assets/ PRE certificate/ PRE gitrepo/ PRE jekyll/ PRE pro/ PRE skillroadmap/ 2023-07-30 14:55:05 4455 404.html 2023-07-30 14:55:05 504 Dockerfile 2023-07-30 14:55:06 80555 feed.xml 2023-07-30 14:55:06 7760 index.html 2023-07-30 14:55:06 0 nginx.conf Total Objects: 5 Total Size: 93274 Option 1: S3 static webhosting\nGo to the AWS console, under the S3 bucket \u0026ldquo;zackweb-serverless\u0026rdquo; properties, enable static website hosting, and update the bucket website endpoint address to the GoDaddy DNS record.\nOption 2: using AWS CDK + CDN\nWith AWS CDK and CDN, the \u0026ldquo;zackweb\u0026rdquo; can be straightforwardly distributed from an S3 bucket accessible to the public by using CloudFront.\nThe steps will be:\nEnable AWS CDK on the EC2 bastion host. S3 bucket ready and copy static web content into it (done above with modification of the existing GitHub action workflow). Establish a CloudFront distribution to host a static To-Do web application. Deploy the AWS CDK solution to host the To-do application. Install AWS CDK on the bastion EC2 host # AWS CDK requires nodejs newer version ubuntu@ip-172-31-26-78:~$ sudo apt-get install nodejs -y ubuntu@ip-172-31-26-78:~$ sudo npm cache clean -f ubuntu@ip-172-31-26-78:~$ sudo npm install -g n ubuntu@ip-172-31-26-78:~$ sudo n stable ubuntu@ip-172-31-26-78:~$ nodejs --version v12.22.9 # install aws-cdk cli ubuntu@ip-172-31-26-78:~$ npm install -g aws-cdk ubuntu@ip-172-31-26-78:~$ cdk --version 2.139.1 (build b88f959) # check aws credential and bootstrap CDK ubuntu@ip-172-31-26-78:~$ aws sts get-caller-identity { \u0026#34;UserId\u0026#34;: \u0026#34;AIDxxxxxxxxx7ZV\u0026#34;, \u0026#34;Account\u0026#34;: \u0026#34;8xxxxxx342\u0026#34;, \u0026#34;Arn\u0026#34;: \u0026#34;arn:aws:iam::8xxxxx342:user/zackcdk\u0026#34; } # bootstrap CDK ubuntu@ip-172-31-26-78:~$ sudo cdk bootstrap aws://8xxxxxxx2/ap-southeast-2 # init app ubuntu@ip-172-31-26-78:~$ mkdir cdk ubuntu@ip-172-31-26-78:~$ cd cdk ubuntu@ip-172-31-26-78:~/cdk# cdk init app --language=typescript Initializing a new git repository... Executing npm install... ✅ All done! # create CDK code ubuntu@ip-172-31-26-78:~/cdk/lib# vim cdk-stack.ts import * as cdk from \u0026#39;@aws-cdk/core\u0026#39;; import * as cloudfront from \u0026#39;@aws-cdk/aws-cloudfront\u0026#39;; import * as origins from \u0026#39;@aws-cdk/aws-cloudfront-origins\u0026#39;; export class ZackWebStack extends cdk.Stack { constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) { super(scope, id, props); // existing S3 bucket const existingBucketName = \u0026#39;zackweb-serverless\u0026#39;; // Create a CloudFront distribution const distribution = new cloudfront.Distribution(this, \u0026#39;MyDistribution\u0026#39;, { defaultBehavior: { origin: new origins.S3OriginFromBucketName(existingBucketName) }, defaultRootObject: \u0026#39;index.html\u0026#39; // default root object }); // Output the CloudFront distribution domain name new cdk.CfnOutput(this, \u0026#39;CloudFrontDomainName\u0026#39;, { value: distribution.distributionDomainName }); } } # install required module ubuntu@ip-172-31-26-78:~/cdk/lib# npm install @aws-cdk/core ubuntu@ip-172-31-26-78:~/cdk/lib# npm install @aws-cdk/aws-cloudfront ubuntu@ip-172-31-26-78:~/cdk/lib# npm install @aws-cdk/aws-cloudfront-origins # Deploy stack ubuntu@ip-172-31-26-78:~/cdk/lib# cd .. ubuntu@ip-172-31-26-78:~/cdk/# cdk deploy The \u0026ldquo;zackweb\u0026rdquo; is now hosted on AWS with serverless deployment! Conclusion\nNow we move the blog onto AWS with serverless website hosting, using both S3 static webhosting and AWS CDK plus Cloudfront.\n","permalink":"https://zackblog.work/posts/zackblog-aws-serverless-webhosting/","summary":"\u003cp\u003eIn this article, I will see how to host \u0026ldquo;zackweb\u0026rdquo; as a static web application using the following AWS serverless options:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eS3 static webhosting\u003c/li\u003e\n\u003cli\u003eAWS CDK + CloudFront\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003ePrerequisite\u003c/strong\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eAdd one more step in the existing Github Action workflow to copy the static web content to the newly created S3 bucket\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# edit github action workflow\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eaws s3 cp ~/zack-gitops-project/zack_blog/_site/* s3://zackweb-serverless/ --recursive\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# validate content in s3 bucket\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eubuntu@ip-172-31-26-78:~$ aws s3 ls s3://zackweb-serverless --summarize\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                     PRE aboutme/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                     PRE assets/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                     PRE certificate/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                     PRE gitrepo/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                     PRE jekyll/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                     PRE pro/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e                     PRE skillroadmap/\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e2023-07-30 14:55:05       \u003cspan class=\"m\"\u003e4455\u003c/span\u003e 404.html\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e2023-07-30 14:55:05        \u003cspan class=\"m\"\u003e504\u003c/span\u003e Dockerfile\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e2023-07-30 14:55:06      \u003cspan class=\"m\"\u003e80555\u003c/span\u003e feed.xml\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e2023-07-30 14:55:06       \u003cspan class=\"m\"\u003e7760\u003c/span\u003e index.html\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e2023-07-30 14:55:06          \u003cspan class=\"m\"\u003e0\u003c/span\u003e nginx.conf\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eTotal Objects: \u003cspan class=\"m\"\u003e5\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e   Total Size: \u003cspan class=\"m\"\u003e93274\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eOption 1: S3 static webhosting\u003c/strong\u003e\u003c/p\u003e","title":"ZackBlog AWS Serverless webhosting"},{"content":"An interviewer told me this blog is insecure!!\nGenerate free SSL certificate from https://zerossl.com/\nValidate the Certificate with Private Key via https://www.sslshopper.com/certificate-key-matcher.html\nUpload \u0026lsquo;certificate.crt\u0026rsquo; and \u0026lsquo;private.key\u0026rsquo; to web server /etc/nginx/ssl/\nSetting up NGINX HTTPS Server by including the ssl parameter to the listen directive in the server block under \u0026lsquo;http\u0026rsquo; in \u0026rsquo;nginx.conf\u0026rsquo;:\nhttp { server { listen 443 ssl; server_name zackdevops.online; ssl_certificate /etc/nginx/ssl/certificate.crt; ssl_certificate_key /etc/nginx/ssl/private.key; } ... } Fix 2 errors: 2023/02/07 10:29:33 [emerg] 73175#73175: \u0026#34;server\u0026#34; directive is not allowed here in /etc/nginx/nginx.conf:11 2023/02/07 10:30:25 [error] 73207#73207: *1 directory index of \u0026#34;/usr/share/nginx/html/\u0026#34; is forbidden,client: 163.53.144.82, server: zackdevops.online, request: \u0026#34;GET / HTTP/1.1\u0026#34;, host: \u0026#34;zackdevops.online\u0026#34; 2023/02/07 10:30:36 [error] 73207#73207: *1 directory index of \u0026#34;/usr/share/nginx/html/\u0026#34; is forbidden, client: 163.53.144.82, server: zackdevops.online, request: \u0026#34;GET / HTTP/1.1\u0026#34;, host: \u0026#34;zackdevops.online\u0026#34; Bingo! https://zackdevops.online connection is secure!\n","permalink":"https://zackblog.work/posts/enable-free-ssl-certificate-for-blog/","summary":"\u003cp\u003eAn interviewer told me this blog is insecure!!\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003eGenerate free SSL certificate from \u003ca href=\"https://zerossl.com/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003ehttps://zerossl.com/\u003c/a\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eValidate the Certificate with Private Key via \u003ca href=\"https://www.sslshopper.com/certificate-key-matcher.html\" target=\"_blank\" rel=\"noopener noreferrer\"\u003ehttps://www.sslshopper.com/certificate-key-matcher.html\u003c/a\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eUpload \u0026lsquo;certificate.crt\u0026rsquo; and \u0026lsquo;private.key\u0026rsquo; to web server \u003ccode\u003e/etc/nginx/ssl/\u003c/code\u003e\u003c/p\u003e\n\u003c/li\u003e\n\u003cli\u003e\n\u003cp\u003eSetting up NGINX HTTPS Server by including the \u003ccode\u003essl\u003c/code\u003e parameter to the listen directive in the server block under \u0026lsquo;http\u0026rsquo; in \u0026rsquo;nginx.conf\u0026rsquo;:\u003c/p\u003e\n\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehttp {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    server {\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        listen 443 ssl;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        server_name zackdevops.online;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        ssl_certificate /etc/nginx/ssl/certificate.crt;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e        ssl_certificate_key /etc/nginx/ssl/private.key;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    }\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e    ...\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e}\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eFix 2 errors:\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-text\" data-lang=\"text\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e2023/02/07 10:29:33 [emerg] 73175#73175: \u0026#34;server\u0026#34; directive is not allowed here in /etc/nginx/nginx.conf:11\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e2023/02/07 10:30:25 [error] 73207#73207: *1 directory index of \u0026#34;/usr/share/nginx/html/\u0026#34; is forbidden,client: 163.53.144.82, server: zackdevops.online, request: \u0026#34;GET / HTTP/1.1\u0026#34;, host: \u0026#34;zackdevops.online\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e2023/02/07 10:30:36 [error] 73207#73207: *1 directory index of \u0026#34;/usr/share/nginx/html/\u0026#34; is forbidden, client: 163.53.144.82, server: zackdevops.online, request: \u0026#34;GET / HTTP/1.1\u0026#34;, host: \u0026#34;zackdevops.online\u0026#34;\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eBingo! \u003ca href=\"https://zackdevops.online\" target=\"_blank\" rel=\"noopener noreferrer\"\u003ehttps://zackdevops.online\u003c/a\u003e connection is secure!\u003c/p\u003e","title":"Enable Free SSL certificate for blog"},{"content":"When using Ansible with AWS, maintaining the inventory file will be a hectic task as AWS has frequently changed IPs, autoscaling instances, and much more.\nHere we will install and apply ansible plugin for AWS dynamic inventory which makes an API call to AWS to get the instance information in the run time. It gives the EC2 instance details dynamically to manage the AWS infrastructure.\nIt supports most of the public and private cloud platforms not limited to just AWS.\nThe Dynamic Inventory Topology: Setup Ansible AWS Dynamic Inventory\n# Ensure python3 \u0026amp; pip3 installed in Ansible server python3 --version sudo apt-get install python3 -y sudo apt-get install python3-pip -y # Install the boto3 library for ansible boot core to make API calls to AWS to retrieve ec2 instance details sudo pip3 install boto3 fix error ERROR! The ec2 dynamic inventory plugin requires boto3 and botocore. # Create an inventory directory under /opt and cd into the directory sudo mkdir -p /opt/ansible/inventory cd /opt/ansible/inventory sudo vi aws_ec2.yaml --- plugin: aws_ec2 aws_access_key: \u0026lt;xxx-AWS-ACCESS-KEY-HERE\u0026gt; aws_secret_key: \u0026lt;xx-AWS-SECRET-KEY-HERE\u0026gt; regions: - us-west-2 keyed_groups: - key: tags prefix: tag - prefix: instance_type key: instance_type - key: placement.region prefix: aws_region # edit ansible config file to enable AWS plugin and set inventory as above yaml sudo vi /etc/ansible/ansible.cfg [inventory] enable_plugins = aws_ec2 inventory = /opt/ansible/inventory/aws_ec2.yaml Test if Ansible is able to ping all the machines returned by the dynamic inventory\nansible-inventory -i /opt/ansible/inventory/aws_ec2.yaml --list ansible all -m ping Execute Ansible Commands With ec2 Dynamic Inventory\nansible-inventory --graph List all instances grouped under tags, zones, and regions with dynamic group names like:\naws_region_ap_southeast_2 instance_type_t2_micro tag_Name ","permalink":"https://zackblog.work/posts/ansible-for-aws-dynamic-inventory/","summary":"\u003cp\u003eWhen using Ansible with AWS, maintaining the inventory file will be a hectic task as AWS has frequently changed IPs, autoscaling instances, and much more.\u003c/p\u003e\n\u003cp\u003eHere we will install and apply ansible plugin for AWS dynamic inventory which makes an API call to AWS to get the instance information in the run time. It gives the EC2 instance details dynamically to manage the AWS infrastructure.\u003c/p\u003e\n\u003cp\u003eIt supports most of the public and private cloud platforms not limited to just AWS.\u003c/p\u003e","title":"Ansible for AWS Dynamic Inventory"},{"content":"Today we are playing around with Redis, Here we use helm to install Redis cluster, then validate statefulset storage and cluster availability.\nTypical Redis cluster (3 master + 3 slave for slots) Topology:\nHelm install bitnami/redis-cluster\nhelm repo add bitnami https://charts.bitnami.com/bitnami helm pull bitnami/redis-cluster kubectl create ns redis helm install zz-redis bitnami/redis-cluster -n redis Redis-cluster status\nkubectl get po | grep zz-redis zz-redis-redis-cluster-0 1/1 Running 3 (33m ago) 36m zz-redis-redis-cluster-1 1/1 Running 1 (32m ago) 36m zz-redis-redis-cluster-2 1/1 Running 1 (33m ago) 36m zz-redis-redis-cluster-3 1/1 Running 1 (33m ago) 36m zz-redis-redis-cluster-4 1/1 Running 1 (33m ago) 36m zz-redis-redis-cluster-5 1/1 Running 1 (33m ago) 36m kubectl get svc zz-redis-redis-cluster ClusterIP 10.96.68.34 \u0026lt;none\u0026gt; 6379/TCP 37m zz-redis-redis-cluster-headless ClusterIP None \u0026lt;none\u0026gt; 6379/TCP,16379/TCP 37m kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE redis-data-zz-redis-redis-cluster-0 Bound pvc-81f69c15-2f59-4704-9fec-c3ab217ebca5 1Gi RWO rook-ceph-block 37m redis-data-zz-redis-redis-cluster-1 Bound pvc-871dbb68-a78b-48a8-8feb-8726eb8a795e 1Gi RWO rook-ceph-block 37m redis-data-zz-redis-redis-cluster-2 Bound pvc-afd8c82e-c314-426a-a3cd-3c8d10b42bb1 1Gi RWO rook-ceph-block 37m redis-data-zz-redis-redis-cluster-3 Bound pvc-e8fd5d47-dd60-4358-8cf5-a17d6574bbe2 1Gi RWO rook-ceph-block 37m redis-data-zz-redis-redis-cluster-4 Bound pvc-04d4f148-b7c1-407e-b9a8-b2fa911405f0 1Gi RWO rook-ceph-block 37m redis-data-zz-redis-redis-cluster-5 Bound pvc-99e8b2cc-c5d5-4636-a19b-042922eca3cc 1Gi RWO rook-ceph-block 37m Validate cluster by set key\nkubectl exec -it zz-redis-redis-cluster-0 -- sh redis-cli info replication cluster info cluster nodes 054ff137e9530c0e4d8afd1d00162d01952580de 172.16.122.179:6379@16379 slave 2ebcfdf7e74aa8755250384f7efa466f4d18e9d4 0 1702703474000 3 connected 7cfeb1d88dc3838e600a32c1e233b1c5f05006f1 172.16.58.197:6379@16379 master - 0 1702703473000 2 connected 5461-10922 1e56bad62197062fad465bbcc6b625bea8364db2 172.16.58.224:6379@16379 slave 7cfeb1d88dc3838e600a32c1e233b1c5f05006f1 0 1702703474954 2 connected 7e539b2209581c8375f7fb0aa9eedf5b98754b05 172.16.85.252:6379@16379 slave 889515187dbbb525fd73dc840d5bcad78305645d 0 1702703473947 1 connected 889515187dbbb525fd73dc840d5bcad78305645d 172.16.195.10:6379@16379 myself,master - 0 1702703469000 1 connected 0-5460 2ebcfdf7e74aa8755250384f7efa466f4d18e9d4 172.16.85.229:6379@16379 master - 0 1702703473000 3 connected 10923-16383 127.0.0.1:6379\u0026gt; set dad zack OK 127.0.0.1:6379\u0026gt; get dad \u0026#34;zack\u0026#34; kubectl exec -it zz-redis-redis-cluster-4 -- sh redis-cli 127.0.0.1:6379\u0026gt; KEYS * 1) \u0026#34;dad\u0026#34; ","permalink":"https://zackblog.work/posts/redis-cluster-with-helm/","summary":"\u003cp\u003eToday we are playing around with Redis, Here we use helm to install Redis cluster, then validate statefulset storage and cluster availability.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTypical Redis cluster (3 master + 3 slave for slots) Topology:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"/images/redis-slot.png\"\u003e\u003cimg alt=\"image tooltip here\" loading=\"lazy\" src=\"/images/redis-slot.png\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eHelm install bitnami/redis-cluster\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm repo add bitnami https://charts.bitnami.com/bitnami\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm pull bitnami/redis-cluster\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl create ns redis\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm install zz-redis bitnami/redis-cluster -n redis\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eRedis-cluster status\u003c/strong\u003e\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get po \u003cspan class=\"p\"\u003e|\u003c/span\u003e grep zz-redis\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ezz-redis-redis-cluster-0                                 1/1     Running   \u003cspan class=\"m\"\u003e3\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003e33m ago\u003cspan class=\"o\"\u003e)\u003c/span\u003e     36m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ezz-redis-redis-cluster-1                                 1/1     Running   \u003cspan class=\"m\"\u003e1\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003e32m ago\u003cspan class=\"o\"\u003e)\u003c/span\u003e     36m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ezz-redis-redis-cluster-2                                 1/1     Running   \u003cspan class=\"m\"\u003e1\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003e33m ago\u003cspan class=\"o\"\u003e)\u003c/span\u003e     36m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ezz-redis-redis-cluster-3                                 1/1     Running   \u003cspan class=\"m\"\u003e1\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003e33m ago\u003cspan class=\"o\"\u003e)\u003c/span\u003e     36m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ezz-redis-redis-cluster-4                                 1/1     Running   \u003cspan class=\"m\"\u003e1\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003e33m ago\u003cspan class=\"o\"\u003e)\u003c/span\u003e     36m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ezz-redis-redis-cluster-5                                 1/1     Running   \u003cspan class=\"m\"\u003e1\u003c/span\u003e \u003cspan class=\"o\"\u003e(\u003c/span\u003e33m ago\u003cspan class=\"o\"\u003e)\u003c/span\u003e     36m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get svc\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ezz-redis-redis-cluster                    ClusterIP   10.96.68.34     \u0026lt;none\u0026gt;        6379/TCP                        37m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ezz-redis-redis-cluster-headless           ClusterIP   None            \u0026lt;none\u0026gt;        6379/TCP,16379/TCP              37m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get pvc\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eNAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eredis-data-zz-redis-redis-cluster-0   Bound    pvc-81f69c15-2f59-4704-9fec-c3ab217ebca5   1Gi        RWO            rook-ceph-block   37m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eredis-data-zz-redis-redis-cluster-1   Bound    pvc-871dbb68-a78b-48a8-8feb-8726eb8a795e   1Gi        RWO            rook-ceph-block   37m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eredis-data-zz-redis-redis-cluster-2   Bound    pvc-afd8c82e-c314-426a-a3cd-3c8d10b42bb1   1Gi        RWO            rook-ceph-block   37m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eredis-data-zz-redis-redis-cluster-3   Bound    pvc-e8fd5d47-dd60-4358-8cf5-a17d6574bbe2   1Gi        RWO            rook-ceph-block   37m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eredis-data-zz-redis-redis-cluster-4   Bound    pvc-04d4f148-b7c1-407e-b9a8-b2fa911405f0   1Gi        RWO            rook-ceph-block   37m\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eredis-data-zz-redis-redis-cluster-5   Bound    pvc-99e8b2cc-c5d5-4636-a19b-042922eca3cc   1Gi        RWO            rook-ceph-block   37m\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eValidate cluster by set key\u003c/strong\u003e\u003c/p\u003e","title":"Redis cluster with Helm"},{"content":"Apache Kafka has proven to be an extremely popular event streaming platform, as its scalable distributed architecture, high performance, and use cases, some key terms and concepts as below:\nKafka clusters and Kafka brokers Kafka clients and servers Producers, and Consumers, and Consumer groups Kafka topics \u0026amp; Kafka partitions, offsets Kafka topic replication, leaders, and followers ZooKeeper or not Typical Kafka Topology:\nHelm install bitnami/kafka Here we use helm to install Kafka, then validate statefulset storage and cluster availability by creating a topic, producer, and consumer.\nhelm repo add bitnami https://charts.bitnami.com/bitnami helm pull bitnami/kafka kubectl create ns kafka helm install zz-kafka . -n kafka Kafka-cluster status kubectl get all | grep zz-kaf pod/zz-kafka-client 1/1 Running 0 6m27s pod/zz-kafka-controller-0 1/1 Running 0 8m52s pod/zz-kafka-controller-1 1/1 Running 0 8m52s pod/zz-kafka-controller-2 1/1 Running 0 8m52s service/zz-kafka ClusterIP 10.96.58.62 \u0026lt;none\u0026gt; 9092/TCP 8m52s service/zz-kafka-controller-headless ClusterIP None \u0026lt;none\u0026gt; 9094/TCP,9092/TCP,9093/TCP 8m52s statefulset.apps/zz-kafka-controller 3/3 8m52s kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE data-zz-kafka-controller-0 Bound pvc-f3831a5c-c9cf-46bb-a47d-58ea80f82e28 8Gi RWO rook-ceph-block 9m19s data-zz-kafka-controller-1 Bound pvc-a45fd063-7979-4c0d-8ae4-93b4b4b0bf7f 8Gi RWO rook-ceph-block 9m19s data-zz-kafka-controller-2 Bound pvc-ed48b71d-81be-4067-b05f-9e8dc64fab04 8Gi RWO rook-ceph-block 9m19s Validate cluster by setting key Create client.properties with SASL authentication details and copy to client.\nkubectl run zz-kafka-client --restart=\u0026#39;Never\u0026#39; --image docker.io/bitnami/kafka:3.6.1-debian-11-r0 --namespace default --command -- sleep infinity kubectl cp --namespace default client.properties zz-kafka-client:/tmp/client.properties Open two bash windows to access the Kafka client.\nkubectl exec --tty -i zz-kafka-client --namespace default -- bash Window 1 (Producer):\nkafka-console-producer.sh \\ --producer.config /tmp/client.properties \\ --broker-list zz-kafka-controller-0.zz-kafka-controller-headless.default.svc.cluster.local:9092,zz-kafka-controller-1.zz-kafka-controller-headless.default.svc.cluster.local:9092,zz-kafka-controller-2.zz-kafka-controller-headless.default.svc.cluster.local:9092 \\ --topic test Window 2 (Consumer):\nkafka-console-consumer.sh \\ --consumer.config /tmp/client.properties \\ --bootstrap-server zz-kafka.default.svc.cluster.local:9092 \\ --topic test \\ --from-beginning Test topic and PRODUCER with CONSUMER.\n","permalink":"https://zackblog.work/posts/apache-kafka-cluster-with-testing-topics/","summary":"\u003cp\u003eApache Kafka has proven to be an extremely popular event streaming platform, as its scalable distributed architecture, high performance, and use cases, some key terms and concepts as below:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eKafka clusters and Kafka brokers\u003c/li\u003e\n\u003cli\u003eKafka clients and servers\u003c/li\u003e\n\u003cli\u003eProducers, and Consumers, and Consumer groups\u003c/li\u003e\n\u003cli\u003eKafka topics \u0026amp; Kafka partitions, offsets\u003c/li\u003e\n\u003cli\u003eKafka topic replication, leaders, and followers\u003c/li\u003e\n\u003cli\u003eZooKeeper or not\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eTypical Kafka Topology:\u003c/p\u003e\n\u003cp\u003e\u003ca href=\"/images/kafka.png\"\u003e\u003cimg alt=\"image tooltip here\" loading=\"lazy\" src=\"/images/kafka.png\"\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eHelm install bitnami/kafka\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eHere we use helm to install Kafka, then validate statefulset storage and cluster availability by creating a topic, producer, and consumer.\u003c/p\u003e","title":"Apache Kafka cluster with testing topics"},{"content":"In this post I will run the steps required to set up MetalLB for LoadBalancer IP allocation, the NGINX Ingress Controller for routing traffic based on subdomains, and how to configure local DNS to enable subdomain-based routing for Kubernetes services.\nIngress-nginx install\nWe will use Helm to install our ingress controller ingress-nginx, which will be used to route traffic based on subdomains.\n# install ingress controller ingress-nginx helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx helm repo update helm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace # check for ingressclasses kubectl get ingressclasses.networking.k8s.io NAME CONTROLLER PARAMETERS AGE nginx k8s.io/ingress-nginx \u0026lt;none\u0026gt; 149m MetalLB for loadbalancing\nThen we will set up MetalLB in local K8s cluster, to enable LoadBalancer service.\nMetalLB works in two modes: Layer 2 (simpler for home labs) or BGP (more complex, used in production environments), here we will deploy MetalLB manifest and create MetalLB ConfigMap For a Layer 2 configuration, assign a range of IP addresses on local network that MetalLB will use for load balancers.\n# install metallb manifest kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml root@asb:~# kubectl get all -n metallb-system NAME READY STATUS RESTARTS AGE pod/controller-fbf54885d-skzkl 1/1 Running 0 174m pod/speaker-49m7r 1/1 Running 0 174m pod/speaker-5zvfk 1/1 Running 0 174m pod/speaker-gdm26 1/1 Running 0 174m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/webhook-service ClusterIP 10.101.12.65 \u0026lt;none\u0026gt; 443/TCP 174m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/speaker 3 3 3 3 3 kubernetes.io/os=linux 174m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/controller 1/1 1 1 174m NAME DESIRED CURRENT READY AGE replicaset.apps/controller-fbf54885d 1 1 1 174m # Create a IP range for loadbalancer IP pool vim metalab-pool.yaml apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: my-ip-pool namespace: metallb-system spec: addresses: - 11.0.1.240-11.0.1.252 # verify IP pool kubectl describe ipaddresspool my-ip-pool -n metallb-system kubectl get ipaddresspool -n metallb-system NAME AUTO ASSIGN AVOID BUGGY IPS ADDRESSES my-ip-pool true false [\u0026#34;11.0.1.240-11.0.1.252\u0026#34;] # Loadbalancer auto assigned for Ingress Controller and joesite service with LoadBalancer kubectl get svc -A | grep Load ingress-nginx ingress-nginx-controller LoadBalancer 10.106.123.55 11.0.1.242 80:32279/TCP,443:31291/TCP 157m joesite-argo joesite-service LoadBalancer 10.111.85.29 11.0.1.240 80:30500/TCP 23d Local DNS setup\nI want to replace all the NodePort services with a subdomain for simplicity, so I need to edit both local PC and kubeconfig VM\u0026rsquo;s host file to point all subdomains to the Ingress Controller LoadBalancer IP (11.0.1.242)\nvim /etc/hosts 11.0.1.242 pm.tina.place am.tina.place gf.tina.place lh.tina.place ag.tina.place Ingress rules for traffic control\nNow we can create a list of Ingress rules to route the K8S services to subdomains. As those services come with different namespaces, we need to split each Ingress rule for services in different namespaces, also need to pay attention to TLS setting for https requests to avoid too many redirects.\nvim monitoring-ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: monitoring-ingress namespace: monitoring annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx rules: - host: gf.tina.place http: paths: - path: / pathType: Prefix backend: service: name: prometheus-stack-grafana port: number: 80 - host: am.tina.place http: paths: - path: / pathType: Prefix backend: service: name: prometheus-stack-kube-prom-alertmanager port: number: 9093 - host: pm.tina.place http: paths: - path: / pathType: Prefix backend: service: name: prometheus-stack-kube-prom-prometheus port: number: 9090 kubectl apply -f monitoring-ingress.yaml kubectl apply -f other-ingress.yaml root@asb:~/path-based# kubectl get ingress -A NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE argocd argocd-ingress nginx ag.tina.place 11.0.1.242 80, 443 ","permalink":"https://zackblog.work/posts/ingress-controller-for-subdomain-based-routing/","summary":"\u003cp\u003eIn this post I will run the steps required to set up MetalLB for LoadBalancer IP allocation, the NGINX Ingress Controller for routing traffic based on subdomains, and how to configure local DNS to enable subdomain-based routing for Kubernetes services.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eIngress-nginx install\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe will use Helm to install our ingress controller \u003ccode\u003eingress-nginx\u003c/code\u003e, which will be used to route traffic based on subdomains.\u003c/p\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# install  ingress controller ingress-nginx\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm repo update\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ehelm -n ingress-nginx install ingress-nginx ingress-nginx/ingress-nginx --create-namespace\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"c1\"\u003e# check for ingressclasses\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get ingressclasses.networking.k8s.io\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eNAME    CONTROLLER             PARAMETERS   AGE\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003enginx   k8s.io/ingress-nginx   \u0026lt;none\u0026gt;      149m\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003e\u003cstrong\u003eMetalLB for loadbalancing\u003c/strong\u003e\u003c/p\u003e","title":"Ingress controller for Subdomain based Routing"},{"content":"Today, I will explore how to install and configure Persistent Volume for all the deployment in K8S to consume.\nHelm install Rook-Ceph for persistent storage\nConfigure local VM block storage to add 50Gb sdb to all k8s master and worker nodes Install rook-ceph cluster git clone --single-branch --branch master https://github.com/rook/rook.git cd rook/deploy/examples kubectl create -f crds.yaml -f common.yaml -f operator.yaml kubectl create -f cluster.yaml Ceph toolbox to check cluster status kubectl create -f toolbox.yaml kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- bash ceph status ceph osd status ceph df rados df Ceph Dashboard service for HTTPS login kubectl create -f dashboard-external-https.yaml kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath=\u0026#34;{[\u0026#39;data\u0026#39;][\u0026#39;password\u0026#39;]}\u0026#34; | base64 --decode \u0026amp;\u0026amp; echo Create storage pool and storage class kubectl create -f pool.yaml cd csi/rbd kubectl create -f storageclass.yaml Set \u0026ldquo;rook-ceph-block\u0026rdquo; as the default storage class kubectl patch storageclass rook-ceph-block -p \u0026#39;{\u0026#34;metadata\u0026#34;: {\u0026#34;annotations\u0026#34;:{\u0026#34;storageclass.kubernetes.io/is-default-class\u0026#34;:\u0026#34;true\u0026#34;}}}\u0026#39; kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE rook-ceph-block (default) rook-ceph.rbd.csi.ceph.com Delete Immediate true 8d Check pool and OSDs in Ceph web UI\n","permalink":"https://zackblog.work/posts/rook-ceph-for-dynamic-persistent-volume/","summary":"\u003cp\u003eToday, I will explore how to install and configure Persistent Volume for all the deployment in K8S to consume.\u003c/p\u003e\n\u003cp\u003eHelm install Rook-Ceph for persistent storage\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eConfigure local VM block storage to add 50Gb sdb to all k8s master and worker nodes\u003c/li\u003e\n\u003cli\u003eInstall rook-ceph cluster\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003egit clone --single-branch --branch master https://github.com/rook/rook.git\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003ecd\u003c/span\u003e rook/deploy/examples\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl create -f crds.yaml -f common.yaml -f operator.yaml\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl create -f cluster.yaml\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eCeph toolbox to check cluster status\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl create -f toolbox.yaml\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl -n rook-ceph \u003cspan class=\"nb\"\u003eexec\u003c/span\u003e -it deploy/rook-ceph-tools -- bash\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eceph status\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eceph osd status\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eceph df\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003erados df\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eCeph Dashboard service for HTTPS login\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl create -f dashboard-external-https.yaml\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl -n rook-ceph get secret rook-ceph-dashboard-password -o \u003cspan class=\"nv\"\u003ejsonpath\u003c/span\u003e\u003cspan class=\"o\"\u003e=\u003c/span\u003e\u003cspan class=\"s2\"\u003e\u0026#34;{[\u0026#39;data\u0026#39;][\u0026#39;password\u0026#39;]}\u0026#34;\u003c/span\u003e \u003cspan class=\"p\"\u003e|\u003c/span\u003e base64 --decode \u003cspan class=\"o\"\u003e\u0026amp;\u0026amp;\u003c/span\u003e \u003cspan class=\"nb\"\u003eecho\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eCreate storage pool and storage class\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl create -f pool.yaml\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\u003cspan class=\"nb\"\u003ecd\u003c/span\u003e csi/rbd\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl create -f storageclass.yaml\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eSet \u0026ldquo;rook-ceph-block\u0026rdquo; as the default storage class\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl patch storageclass rook-ceph-block -p \u003cspan class=\"s1\"\u003e\u0026#39;{\u0026#34;metadata\u0026#34;: {\u0026#34;annotations\u0026#34;:{\u0026#34;storageclass.kubernetes.io/is-default-class\u0026#34;:\u0026#34;true\u0026#34;}}}\u0026#39;\u003c/span\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003e\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl get sc\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003eNAME                        PROVISIONER                  RECLAIMPOLICY  VOLUMEBINDINGMODE  ALLOWVOLUMEEXPANSION   AGE\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003erook-ceph-block \u003cspan class=\"o\"\u003e(\u003c/span\u003edefault\u003cspan class=\"o\"\u003e)\u003c/span\u003e   rook-ceph.rbd.csi.ceph.com   Delete          Immediate          \u003cspan class=\"nb\"\u003etrue\u003c/span\u003e                  8d\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cp\u003eCheck pool and OSDs in Ceph web UI\u003c/p\u003e","title":"Rook-Ceph for dynamic Persistent Volume"},{"content":"Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes applications. This is a post to show how to enable Argo CD on local k8s and AWS EKS, deploy applications and sync with GitHub manifests.\nArgo CD install ingress controller ingress-nginx kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml Download and install Argo CD CLI curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64 sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd rm argocd-linux-amd64 Configure Argo CD API Server and deploy zackweb Change the argocd-server service type to NodePort, initialize admin password, and deploy \u0026ldquo;zackweb\u0026rdquo; via Argo CD application manifests (argo-zackweb-application.yaml).\nkubectl patch svc argocd-server -n argocd -p \u0026#39;{\u0026#34;spec\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;NodePort\u0026#34;}}\u0026#39; argocd admin initial-password -n argocd kubectl create -f argo-zackweb-application.yaml argo-zackweb-application.yaml\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: zackweb namespace: argocd spec: destination: namespace: \u0026#39;zackweb\u0026#39; server: \u0026#39;https://kubernetes.default.svc\u0026#39; source: path: eks-deploy repoURL: \u0026#39;https://github.com/ZackZhouHB/zack-gitops-project\u0026#39; targetRevision: editing project: default syncPolicy: automated: prune: true selfHeal: true syncOptions: - CreateNamespace=true Access the Argo CD UI to check for sync ","permalink":"https://zackblog.work/posts/gitops-with-argo-cd/","summary":"\u003cp\u003eArgo CD is a declarative, GitOps continuous delivery tool for Kubernetes applications. This is a post to show how to enable Argo CD on local k8s and AWS EKS, deploy applications and sync with GitHub manifests.\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eArgo CD install ingress controller ingress-nginx\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl create namespace argocd\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ekubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eDownload and install Argo CD CLI\u003c/li\u003e\n\u003c/ul\u003e\n\u003cdiv class=\"highlight\"\u003e\u003cpre tabindex=\"0\" class=\"chroma\"\u003e\u003ccode class=\"language-bash\" data-lang=\"bash\"\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003ecurl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003esudo install -m \u003cspan class=\"m\"\u003e555\u003c/span\u003e argocd-linux-amd64 /usr/local/bin/argocd\n\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"line\"\u003e\u003cspan class=\"cl\"\u003erm argocd-linux-amd64\n\u003c/span\u003e\u003c/span\u003e\u003c/code\u003e\u003c/pre\u003e\u003c/div\u003e\u003cul\u003e\n\u003cli\u003eConfigure Argo CD API Server and deploy zackweb\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eChange the argocd-server service type to NodePort, initialize admin password, and deploy \u0026ldquo;zackweb\u0026rdquo; via Argo CD application manifests (argo-zackweb-application.yaml).\u003c/p\u003e","title":"GitOps with Argo CD"},{"content":"Project Introduction\nThis is my first web blog using Jekyll, as a practical way by following Cloud Resume Challenge to build my cloud and devops concept and technical skillsets. ~~\nThe Design\nBy design, I will create:\na web blog Zack\u0026rsquo;s Blog: with content and details to introduce myself a github repo zack-gitops-project: to source control all code that I build and run locally by \u0026ldquo;jekyll serve\u0026rdquo;, validate site and pages, then push the source code to github. a Dockerfile to build jekyll code into a docker image running by nginx/alpine 3 folders with manifests for staging and prod webapp deploy: /terraform for creating AWS VPC and EKS to host the website as the production environment; /k8s-local-deploy for website image deployment to local K8S as testing; /eks-deploy for prod deployment with ArgoCD application manifest a EC2 instance: as staging environment for AWS with Godaddy domain hosting test a EKS cluster: as PROD environment to validate ArgoCD sync for web deployment The Architecture\nThis is the design of the CICD pipeline in GitHub Action workflow to auto build docker images for this website every time I make a code change and commit to my git repo zack-gitops-project [Branch: editing]\nZack\u0026rsquo;s Blog\ndef hello_world(): print(\u0026#34;Hello, world!\u0026#34;) hello_world() ","permalink":"https://zackblog.work/posts/about-zack-web-gitops-project/","summary":"\u003cp\u003e\u003cstrong\u003eProject Introduction\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis is my first web blog using Jekyll, as a practical way by following \u003ca href=\"https://cloudresumechallenge.dev/docs/the-challenge/aws/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eCloud Resume Challenge\u003c/a\u003e to build my cloud and devops concept and technical skillsets. ~~\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eThe Design\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eBy design, I will create:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003ea web blog \u003ca href=\"https://zackblog.work/\" target=\"_blank\" rel=\"noopener noreferrer\"\u003eZack\u0026rsquo;s Blog\u003c/a\u003e: with content and details to introduce myself\u003c/li\u003e\n\u003cli\u003ea github repo \u003ca href=\"https://github.com/ZackZhouHB/zack-gitops-project\" target=\"_blank\" rel=\"noopener noreferrer\"\u003ezack-gitops-project\u003c/a\u003e: to source control all code that I build and run locally by \u0026ldquo;jekyll serve\u0026rdquo;, validate site and pages, then push the source code to github.\u003c/li\u003e\n\u003cli\u003ea Dockerfile to build jekyll code into a docker image running by nginx/alpine\u003c/li\u003e\n\u003cli\u003e3 folders with manifests for staging and prod webapp deploy:\n\u003cul\u003e\n\u003cli\u003e\u003cins\u003e/terraform\u003c/ins\u003e for creating AWS VPC and EKS to host the website as the production environment;\u003c/li\u003e\n\u003cli\u003e\u003cins\u003e/k8s-local-deploy\u003c/ins\u003e for website image deployment to local K8S as testing;\u003c/li\u003e\n\u003cli\u003e\u003cins\u003e/eks-deploy\u003c/ins\u003e for prod deployment with ArgoCD application manifest\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003ea EC2 instance: as staging environment for AWS with Godaddy domain hosting test\u003c/li\u003e\n\u003cli\u003ea EKS cluster: as PROD environment to validate ArgoCD sync for web deployment\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eThe Architecture\u003c/strong\u003e\u003c/p\u003e","title":"About Zack-web Gitops project!"},{"content":"Professional Summary I am a seasoned IT professional with over a decade of experience, characterized by continuous evolution and mastery across diverse technological domains. My career began at a Fortune Global 500 PC manufacturer company, where I developed a strong foundation in traditional IT, gaining comprehensive expertise in hardware, databases, operating systems, and SAP applications. This experience provided me with a robust understanding of enterprise-level IT ecosystems.\nUpon relocating to Australia in 2017, I broadened my ICT experience within the financial and entertainment industries. This experience led to my transition into cloud computing, with a focus on plubic cloud platforms and DevOps practices. In subsequent roles, I implemented crucial cloud cost optimization strategies and spearheaded digital transformations, significantly improving organizational agility and cloud presence. Now, as a Senior Cloud Engineer in the public sector, I specialize in AWS automation and Kubernetes (EKS) platform engineering. I have also advanced into Machine Learning Operations (MLOps), focused on automating ML model provisioning and deployment on AWS EKS.\nWith extensive AWS and DevOps experience, I possess proven expertise in Kubernetes orchestration, CI/CD pipeline development, and serverless architecture in Cloud environment. My strong capabilities in automation through Infrastructure as Code (IaC) using tools like CloudFormation, Terraform, and Ansible have enabled me to streamline infrastructure provisioning and dynamic EKS cluster management. I hold several industry-recognized certifications, including AWS Certified Solutions Architect, AWS Certified DevOps Engineer, CNCF Certified Kubernetes Administrator (CKA), and CNCF Certified Kubernetes Security Specialist (CKS).\nI am passionate about leveraging my technical acumen and strategic mindset to contribute to cloud-native architectures and automation practices that drive continuous improvement in DevOps operations.\nEducation \u0026amp; Certificate Master of Information Communication Technology\nUniversity of Wollongong – 2008-2010 Wollongong, NSW\nBachelor of Computer Science\nSouth-Central University – 2004-2008 China\nAWS Certified Machine Learning - Associate (MLA)\nAWS Certified Solutions Architect - Professional (SAP)\nAWS Certified DevOps Engineer - Professional (DOP)\nCNCF Certified Kubernetes Administrator (CKA)\nCNCF Certified Kubernetes Security Specialist (CKS)\nRed Hat Certified System Administrator (RHCSA)\nRed Hat Certified Engineer (RHCE)\nVMware Certified Professional - Data Center Virtulization 2022 (VCP)\n","permalink":"https://zackblog.work/about/","summary":"\u003ch3 id=\"professional-summary\"\u003eProfessional Summary\u003c/h3\u003e\n\u003cp\u003eI am a seasoned IT professional with over a decade of experience, characterized by continuous evolution and mastery across diverse technological domains. My career began at a Fortune Global 500 PC manufacturer company, where I developed a strong foundation in traditional IT, gaining comprehensive expertise in hardware, databases, operating systems, and SAP applications. This experience provided me with a robust understanding of enterprise-level IT ecosystems.\u003c/p\u003e\n\u003cp\u003eUpon relocating to Australia in 2017, I broadened my ICT experience within the financial and entertainment industries. This experience led to my transition into cloud computing, with a focus on plubic cloud platforms and DevOps practices. In subsequent roles, I implemented crucial cloud cost optimization strategies and spearheaded digital transformations, significantly improving organizational agility and cloud presence. Now, as a \u003cstrong\u003eSenior Cloud Engineer\u003c/strong\u003e in the public sector, I specialize in \u003cstrong\u003eAWS automation\u003c/strong\u003e and \u003cstrong\u003eKubernetes (EKS) platform engineering\u003c/strong\u003e. I have also advanced into \u003cstrong\u003eMachine Learning Operations (MLOps)\u003c/strong\u003e, focused on automating ML model provisioning and deployment on AWS EKS.\u003c/p\u003e","title":"About Me"}]