Machine Learning

Last month, I did something that, looking back, was probably the most educational practice of my year. I tried using Claude and GPT to build a complete multi-source RAG system from scratch. This wasn’t a “toy demo” for social media; it was a real internal tool for my lab designed to handle messy document data, query rewriting, hybrid search, and deployment for my team. I set a strict rule for myself: Let the AI write almost all the code, while I handle the requirements and decision-making. ...

I constantly rely on AWS Billing and Cost Management and Trusted Advisor to monitor costs and security. However, manual billing reviews are time-consuming, and raw Cost Explorer data doesn’t provide the full picture. What if an intelligent program could proactively call the Billing and Trusted Advisor APIs, retrieve the raw data, and pass it to an LLM for in-depth analysis of account health—then deliver actionable insights and AI-powered recommendations? That would save a significant amount of time and effort. ...

I constantly rely on AWS Billing and Cost Management and Trusted Advisor to monitor costs and security. However, manual billing reviews are time-consuming, and raw Cost Explorer data doesn’t provide the full picture. What if an intelligent program could proactively call the Billing and Trusted Advisor APIs, retrieve the raw data, and pass it to an LLM for in-depth analysis of account health—then deliver actionable insights and AI-powered recommendations? That would save a significant amount of time and effort. ...

After building a custom RAG system on Amazon EKS, my son asked why the application could only handle one question at a time, while ChatGPT allows users to continue conversations through follow-up questions with contextual memory. That made me realize there was an opportunity to leverage the power of the LangChain framework for better abstractions, conversational memory, and access to the broader LangChain ecosystem. This post documents the journey of migrating to a LangChain-powered solution — the challenges faced, and the new capabilities and benefits gained. ...

Recently Data team reached out, trying to build an EC2 running Open WebUI, connecting to AWS Bedrock, to offer team member AI application. This guide provides step-by-step instructions for deploying and integrating with AWS Bedrock models. . The architecture consists of two Docker containers operating on a single EC2 instance: Open WebUI serves as the user-facing chat interface, while a Bedrock Access Gateway acts as middleware. This gateway securely forwards requests from Open WebUI to the AWS Bedrock API using the EC2 instance’s attached IAM role. The containers communicate over a private Docker network, isolating traffic between them. ...

In the past few days, I got an idea from my recent electricity and gas bills—thinking about how to run data analysis to examine usage patterns and trends using the Python approach: employing Pandas for reading data, using Numpy for normalisation, and Matplotlib to create a few visualisations. However, this time I wanted to try talking to Gemini CLI via the terminal to see if it could generate a web-based analytical platform for me, as someone with no developer background. ...

This was a linguistic analysis project where the primary goal was not just to count words, but to evaluate the language, themes, and emotional tone of the children’s show “Peppa Pig” (specifically, the first four seasons) to determine its suitability for a pre-kindergarten audience. In this study, I will try to answer a broader question: “Beyond a simple word list, what can a multi-faceted data analysis tell us about the show’s true educational and emotional value?” ...

In a previous post - MLOps - Build a Oscar Best Picture Winner Model, I was able to establish a baseline model using a RandomForestClassifier to predict the Oscar for Best Picture. This classic workflow involved data cleaning, training, and prediction, providing a solid starting point. However, a deeper look at the results revealed critical weaknesses that an experienced machine learning engineer would immediately flag: Inadequate Model Choice: The initial model wasn’t powerful enough for the task. The classification report showed a recall of 0.00 for the “winner” class. This is a major red flag, indicating the model completely failed to identify any actual winners, likely due to the severe class imbalance. Misleading Evaluation Metrics: I think I relied too heavily on accuracy. On an imbalanced dataset, a model can achieve high accuracy simply by always predicting the majority class. Better to shift our focus to more robust metrics like the F1-score, ROC AUC, and Precision-Recall AUC. This analysis led to idea to enhance this Oscar prediction with a more sophisticated LightGBM (LGBM) classifier model, known for its high performance, speed, and efficiency on tabular data. ...

In this post, we will continue to build a basic machine learning model to predict the Best Picture winner at the Academy Awards (Oscar). We will use our previous processed dataset that includes information about the nominees and winners from the 72nd to the 96th Oscar ceremonies. The goal is to predict the winner based on various features like IMDb ratings, Metascore, Tomatometer percentage, Golden Globe and BAFTA wins/nominations. Step 1: Understand the features and model ...

After downloading a dataset from Kaggle about historical Oscar nominations and winners, it is time to move on to data cleansing and enrichment. The original data contained seven headers, but only a few were useful: the film name, ceremony year, and winner status. Additionally, the dataset spanned from 1927 to 2024 and included all Oscar categories, making it quite complex—perhaps too overwhelming for me as a beginner. So, I decided to focus on Best Picture as it is always the most important award among all the other categories. ...