Made by Fenor
We wrangle data so you don’t have to.
This project is an ETL (Extract, Transform, Load) pipeline that successfully automates data movement from a live PostgreSQL relational database (ToteSys) into a data warehouse optimised for analytics. The pipeline is designed for scalability, reliability and automation using AWS services and Infrastructure as Code. This project has been an opportunity for our team to showcase our knowledge of Python, SQL, database modelling, AWS, good DevOps practices and Agile working, amongst other technical and non-technical skills learnt over our time on the Northcoders Data Engineering Bootcamp. The project uses AWS cloud services for storage (S3), computing (lambda), and monitoring (Cloudwatch), allowing for security and scalability. Terraform is used to store and deploy the project as Infrastructure as Code, allowing for reliable and repeatable deployment. A GitHub Actions CI/CD pipeline is used alongside rigourus automated testing and compliance checks to ensure the quality and consistency of our codebase. We adhered to the principles of TDD (Test driven development) throughout our time working on the project using Pytest, and practiced pair programming to ensure that each piece of work met the highest possible standards. Our team had a wide range of expertise and knowledge, and we ensured that each team member worked on various elements of the project to encourage knowledge sharing and collaboration. Our working practices included daily morning standup meetings, task-ticketing through a Trello Kanban board, and regular code reviews and group reflections. Our team members are: – Brendan Corbett | https://github.com/BrendanC8450 – Connor Creed | https://github.com/creedy293 | https://www.linkedin.com/in/c-creed-942132231/ – Fabio Greenwood | https://github.com/FabioGreenwood | https://www.linkedin.com/in/fabio-greenwood-b0832676/ – Tor Satherley | https://github.com/TorSatherley | https://www.linkedin.com/in/tor-satherley-95a789265/ – Vincent Toor-Azorin | https://github.com/Armatoor147 | https://www.linkedin.com/in/vincent-toor-azorin/
The Team
Connor Creed
worked as a smart energy expert at British Gas for the last…
9 years. Looking for something new to try and really enjoying the experience of coding.
Tor Satherley
After a successful career in youth work, Tor is excited to…
be transitioning into the field of Data Engineering. He combines his experience with managing people and teams with his passion for problem-solving and finding actionable insights. He thrives in roles that require analytical thinking, communication and continuous learning.
Fabio Greenwood
Preview: Team member image Enthusiastic Analyst with 7.5…
years’ previous experience in custom analytical solutions within operations research and mech eng. Recently completed a MSc in Data Science and a 13-week Data Engineering bootcamp to realise his wish of relining his ongoing career into Data Engineering or ML/Data Sci.
Vincent Toor-Azorin
Vincent is a recent Maths and Physics Master’s graduate…
from the University of Warwick who decided to transition into data engineering by joining the Northcoders Data Engineering Bootcamp.
Brendan Corbett
Brendan decided to transition from mental health work after…
Covid to explore other careers. After spotting Northcoders and recognising the current and forthcoming need for technical skills, he quickly adapted to make steps towards a future in tech.
Tech Stack
We used: Python, AWS Lambda, SQL, cloud watch, Terraform, github actions. Effective tools for creating a cloud ETL solution
Challenges Faced
Learning to work cooperatively, Learning new technologies on the job and communicating complex concepts during meetings effectively.