Made by TotesOps
Streamline, Automate, Innovate: Revolutionising TotesOps
Our ETL Data Engineering Project at TotesOps showcases an adept utilisation of AWS services, exemplifying our Python proficiency. The data processing pipeline relies on three AWS Lambda functions, seamlessly handling extraction, transformation, and loading tasks. Infrastructure as Code (IaC) with Terraform ensures automated deployment, scalability, and consistent resource management. Continuous Integration/Continuous Deployment (CI/CD) using Github Actions enhances workflow efficiency. In terms of data storage, AWS S3 buckets provide scalability and durability for ingested and processed data. The inclusion of CloudWatch for monitoring and alerting adds a layer of proactive oversight, ensuring optimal performance. Throughout the project, we have diligently embraced Agile methodologies, employing an iterative and adaptive approach to development. This commitment underscores our dedication to innovation and excellence in the field of data engineering. The project has challenged us in unexpected ways, and we are proud of the outcome of the final product.
The Team
Tom Roberts
Hey, I’m Tom—a finance guy turned data enthusiast! Recently…
made the switch from a career in finance and now buzzing with excitement to bring my finance skills into the data engineering scene. Ready for this new adventure—excited to explore the world of data engineering!
Minnie Taylor Manson
Hi, I’m Minnie, an Astrophysics graduate with a solid…
foundation in data cleansing and manipulation. Eager to push my coding abilities further, I enrolled at Northcoders. My goal is to harness the skills acquired in this bootcamp to tackle practical data issues within the realms of Data Science and Data Engineering.
Kirsten Brindle
Hi, I’m Kirsten. After 5 years as a German teacher and…
pastoral lead, I was ready for a new challenge. Learning to code and manipulate data has employed my linguistic skills, as I have enjoyed learning new programming languages, syntax and structures, and communicating with others via paired programming. I am looking forward to consolidating what I have learnt during the bootcamp in the world of Data Engineering.
Leah Morden-Tew
Hi, I’m Leah. I decided to hang up my scene suit and…
channel my problem solving skills from being a CSI into the world of technology and data engineering. I’m looking forward to utilising my new found skills in the next chapter of my career
Cinthya Sánchez
Hi, I´m Cinthya and I hold a degree in Economics. I have…
valuable experience in finance and I am currently enthusiastic about pivoting towards a new career in the tech sector, particularly in data engineering. I am eager to apply the skills I have acquired from my recent Bootcamp experience.
Elliott Mullins
I’m Elliott Mullins, I have a degree in Artist…
Blacksmithing and after working in the social sector for a while I decided I wanted something that involved flexing my problem solving muscles a bit more so I am making the change to Data Engineering and after this bootcamp I am rearing to go.
Tech Stack
We used AWS S3, Lambda, Cloudwatch, IAM, Secrets Manager, Python, SQL (Protgres), Terraform, Github actions, Pandas, pytest, moto, unittest and Trello. The combination of AWS services, GitHub Actions, Terraform and boto3 allowed for seamless deployment and automation of our project. Pandas and Postgresql assisted in the data-wrangling aspect of our project. For high-quality testing, we utilized Pytest, Moto and Unittest (MagicMock, patch, Mock) ensuring code integrity.
Challenges Faced
In the project’s concluding phases, code refactoring was necessary because of unanticipated structural issues in the data warehouse schema. To address this challenge, we had to reassess our code’s organisation and make necessary adjustments to align with the final schema.