Made by Baltica 2
Collaborative and Creative
This project will create and deploy an AWS pipeline for transferring data from a database into a data warehouse. The system takes data from an initial database, converts to json in a lambda function, and stores in a s3 bucket. It then uses another lambda function to reformat into a schema matching that of our target data warehouse and store in a second s3. Finally, the data is uploaded from the bucket to a data warehouse with a lambda function.
The Team
Ana Terra Camilo Silveira
With a strong foundation in Mathematics and extensive…
experience in data analysis during my Master’s and PhD in Science, I have developed a deep interest in data structures and analytical problem-solving. My academic background has provided me with the ability to work with large datasets, draw meaningful insights, and apply data-driven approaches to complex issues. After relocating to a new country, I decided to pivot my career towards the tech industry, completing a Data Engineering Bootcamp to build on my passion for data.
Laura Messenger
Laura has developed a range of both analytic and…
communication skills during her degree in Natural Sciences at the University of Cambridge and in the course of becoming a qualified Physics teacher. She is eager to embark on a career in data, and has completed a 13 week Northcoders Data Engineering Bootcamp to learn the cutting edge industry tools.
Oscar Ogilvie
Oscar Ogilvie is an aspiring data engineer with a passion…
for technology. Recently completing the NorthCoders data engineering bootcamp, He has developed strong skills in data pipeline design, cloud platforms, and database management. With a solid understanding of tools like Python, SQL, and AWS Infrastructure, Oscar is eager to apply his knowledge and love of problem solving to real-world challenges.
Wesley Shaw
After years of working within purchasing and warehousing,…
and getting a working insight in to how useful clean data can be to an organisation, I felt that I was ready for a career change. With my technical training from the bootcamp, passion for problem solving and eagerness to learn, I feel well equipped to transition in to the field of Data Engineering.
Zishaan Asif
An enthusiastic and driven individual with a strong…
mathematical background, finance expertise, and a passion for leveraging technology. Skilled in data engineering and equipped with advanced technical and analytical capabilities, I am eager to contribute to the development of robust data solutions and deliver meaningful business insights.
Tech Stack
We used: AWS, Terraform, Python, PostgreSQL, GitHub Actions, Tableau We used Terraform as it can be updated, redeployed, and reused. The AWS platform allowed us to work flexibly in the cloud. We used Python as we have advanced knowledge and could use useful packages such as pg8000 to integrate it with PostgreSQL. GitHub Actions allowed us to work collaboratively and build a CI/CD pipeline to test and deploy changes. We used Tableau for efficient data visualisation using our final product.
Challenges Faced
One challenge was ensuring that all data would be captured, including that which came in during the runtime of the ingestion code, but without duplicating data. To do this we ensured it ran on a schedule at set times, allowing us to program capture windows with no gaps or overlap.