Made by Team Funland
Give us your data, we will give you a warehouse.
This repository contains the final team project of the North Coders Data Engineering Bootcamp, showcasing a full-stack ETL (Extract, Transform, Load) pipeline designed for real-world data engineering practice. – Data ingestion from PostgreSQL into AWS S3 data lakes. – Transformation into star-schema format using pandas and awswrangler. – Deployment managed with Infrastructure-as-Code (Terraform). – Automated testing and deployment with CI/CD pipelines via GitHub Actions and Makefile. – Monitoring, logging, and alerts integrated via AWS CloudWatch and SNS. – Business dashboards and insights delivered through Tableau. – Ensure good security and engineering practises.
The Team
Shayan Shaterpoori
Physics graduate looking to enter the field of data…
engineering and solve business problems with technological and analytical skills.
Hari Sapkota
As someone from medical research, a brief exposure to…
programming languages such as Python and R sparked a great enthusiasm in me to use emerging technologies to utilise data and generate valuable insights for businesses. Such curiosity has guided my interest to become a data engineer and explore avenues for data extraction and analysis to enhance data science and machine learning.
Sarah Larkin
My professional experience is based in supply chain,…
logistics, and customer service. My most recent role was in the food industry as a customer relationship manager. Over the years I have developed an interest in how technology can be used to improve processes and assist business decisions. This has led me towards learning technical skills through the Northcoders Data Engineering bootcamp and onto a career in tech.
Elisa Quku
My name is Elisa Quku. I am a former Maths Teacher now…
trained to become a Data Engineer. I joined Northcoders to learn how to work with data and build real-world tech projects. I am very excited to start a new career where I can keep learning and apply my skills in a new way.
Adel
A creative person with an artistic flair, enjoys coding and…
exploring new technologies at coding meetups, blending art with technology.
Tech Stack
We used Python, Terraform, Makefile, Github workflows, AWS. Allows to deploy Serverless python functions, deployed with infrastructure as code.
Challenges Faced
Planning the project at the beginning (what combination of step functions and lambdas we should use), and finding python logic to extract newly updated data. However, as time went on, and we explored these problems, we overcame them.