Made by Team Culinary
Team Get it DONE
This group project was carried out during the final phase of the 13-week Northcoders Data Engineering Bootcamp. The aim was to design and build a reliable data platform that extracts data from a PostgreSQL operational database (Totesys), transforms it into a denormalised form star schema, and loads it into a data warehouse to support analysis and reporting. The system is designed to run on a schedule, monitor and log all activity, and maintain a full history of changes to fact data. As a final step, we visualised the data to demonstrate its value for business insights. This project showcases our ability to collaboratively deliver a full-stack data engineering solution using best practices in automation, testing, and data architecture.
The Team
Tech Stack
We used Python Terraform Lambda S3 IAM PostgreSQL Pandas pg8000 Pytest CloudWatch SNS Boto3 Moto SQLAlchemy PyArrow Makefile Github Actions. Most efficient tools for the given task
Challenges Faced
We very much enjoyed the overall process of making this repo and how we supported and learned from each other. We all benefited from the pair-programming process, but it was a challenge to assign tickets evenly, given the odd number of people in the group. In the first week, we set up the very ambitious target of an automated process of updating the warehouse with all three star schemas. We realised that this would not have been achievable within the timeframe of the project, so we decided to focus on delivering the MVP. Another crucial insight from our project is that planning and communication are key (e.g., discussing interfaces to functions/more detailed planning of the three stages before starting to work separately on the different parts of the repo).
Culinary
Made by Team Culinary Team Get it DONE This group project was carried out during the final phase of the 13-week Northcoders Data Engineering Bootcamp. The aim was to design and build a reliable data platform that extracts data from a PostgreSQL operational database (Totesys), transforms it into a denormalised form star schema, and loads…
Atkinson
Made by Team Atkinson We put the terra in terraform We worked on a etl data pipeline that extracts data from a database and loads it into a data warehouse. The Team AL-Amin Al-Amin, 27, previously worked as a data technician… training to be a data engineer Myles Myles, Accounting turned data engineer Deep Deep…
Ainsdale Beach
Made by Team Ainsdale Beach High Tides! High Tech! The Ainsdale Beach ETL project is an Extract Transform Load pipeline, using Python, hosted in AWS. The primary function of the pipeline is to ingest and transform data from an OLTP optimised database into an OLAP optimised data warehouse. The main objectives of this project are…
Student Projects – Co-Write
A Project made by Team Indecisive –
Turning creative writing into a collaborative experience
Student Projects – kinnected
A project made by Team undivfined – Where Your Connections Become Kin
Student Projects – Pedal Out
A project created by Team Syntactic Sugar Rush – Meet, Pedal, Repeat
Student Projects – Posable
A project made by The Query Cowboys – A social media app for creating shared event albums
Student Projects – RainCheck
A project created by Team Git Commitment Issues – Not Feelin’ It? Plans change, friends don’t.
Student Projects – Stack Dew Valley
A project made by Team Hack Street Boys/Girls – Train Code Fight
Student Projects – starSteps
A project created by Team StarsStep – Motivate, Reward, Repeat
Student Projects – Study BuddAI
A project made by Team Study BuddAI – An on the go learner App
Student Projects – Text to Speak
A project created by Team Slick’s Splendiferous Solo Services – Speak, without speaking.