The Team

Irina Ponomarenko

Working as a pastry chef for 10 years I discovered that…

what I am really good at is creating algorithms for myself and others to follow and gathering data for informed decisions, hence the choice of career in tech. I am a swing and jazz dance enthusiast. Also, I’m cat mom of 2 beautiful Ragdoll cats.

Ciaran Kyle

After working in a treasury function I decided to build on…

my previous background from my degree to transition into the data World. Like Irina I also have 2 cats. They will both miss attending the afternoon seminars every day!

Tes Ryu

Creative Junior Data Engineer with seven years of…

experience in the fashion design industry and a passion for recipe development. I love turning complex data into clear, actionable insights that help drive smart decisions. With a unique mix of analytical, creative, and technical skills, I’m eager to make a real impact and drive meaningful results.

William Tait

With years of experience in banking and finance, followed…

by time spent in academia at Cambridge University, completing an intensive data engineering bootcamp feels like the natural next step into the tech world. My goal? To not just crunch numbers, but to find meaning in data, streamline processes, and drive smarter, more informed decisions. I’m also a proud dad to a little girl who loves pretending to be the scrum master during our stand-up calls.

Bradley Clayton

Coming from a background as a science teacher, I’m…

passionate about learning how things work and sharing this knowledge with others. I’m now applying this curiosity to the world of tech and data. It’s been challenging, but loved overcoming many hurdles and working as part of an awesome team. Thank you Northcoders for this fantastic experience!

Tech Stack

We used: Python, PostgreSQL, AWS S3, AWS Lambda, AWS Secrets Manager, AWS CloudWatch, AWS SNS, Terraform, Python modules: Pandas, pg8000, boto3, patch, mock, logging … We chose these technologies for our ETL pipeline as they were the core technologies we covered throughout the Data Engineering course. Python, with libraries like Pandas and boto3, allows us to manipulate data and integrate with AWS services. PostgreSQL offers querying capabilities for structured data. AWS Lambda enables serverless processing, and Secrets Manager ensures secure handling of credentials. We use CloudWatch for monitoring and SNS for notifications. Terraform automates our infrastructure so we can quickly and easily apply changes to the pipeline architecture. Python modules like patch and mock help us test and debug effectively, ensuring the reliability of our pipeline.

Challenges Faced

The project provided ample challenges every step of the way! We knew that we had what it takes to complete the pipeline, but there were some notable hurdles that proved particularly challenging: One of the first was the the creation of the dependency layers for the Lambda functions using Terraform. It took our best brains a lot of blood sweat and tears but they eventually cracked it. Another was deciding how to trigger each of the lambda functions in the pipeline; our initial focus was on how to successfully complete the initial extraction and to fill our ingestion bucket with data. It only become apparent after accomplishing this that we would have to go back and edit our extract lambda in order to provide the utility to provide data in batches for our transform lambda. We then had similar issues with having to refactor our transform lambda when it became apparent that our load lambda was throwing errors related with ‘duplicate keys’ whenever we restarted our pipeline. So there was a fair bit of backtracking as at the beginning of the project we couldn’t quite see the forest for the trees. Of course we’ve learnt a lot from this process.