Made by Team Ainsdale Beach
High Tides! High Tech!
The Ainsdale Beach ETL project is an Extract Transform Load pipeline, using Python, hosted in AWS. The primary function of the pipeline is to ingest and transform data from an OLTP optimised database into an OLAP optimised data warehouse. The main objectives of this project are to be able to handle regular data updates and execute performant TDD code (Test Driven Development), providing an effective, usable star schema output with robust versioning. An additional feature of this project is to have an S3 bucket containing processed data in Parquet, that mirrors our processed data to permit further analysis better suited to that format. Our aim with this project is to enable analysis of important business data without compromising performance for day-to-day transactional operations.
The Team
Dilesh Parmar
Dilesh Parmar 36 -Career changer from Restaurateur to Data…
Engineer. Excited for a new career in the tech industry.
Elliot Weaver
Elliot Weaver lover of tech and all around nerd, when not…
working as a Tech wizard Elliot often spends their time Live action roleplaying as a magical wizard and leading people in efforts to save the cosmos (running a surprising amount of meetings to do so
Josh Gilling
Josh Gilling, 22 , from the north west region, passionate…
about tech and its constant innovation. Part of the Ainsdale-beach data engineering team and project at Northcoders
Seb Allen
A true adventurer… I’ve lived abroad, learnt a new…
language, dived professionally, explored caves, made things from wood, and delivered your post. My latest challenge has been to learn everything I can about Data Engineering!
Kyungmin Yang
Kyungmin Yang (23), living in London, the data engineering…
became an interest of mine during my time at university. During my 3rd year project at university, I developed a simple code to model and handle to data of the tyre-road friction which sparked my passion for data engineering
Tech Stack
Programming: Python, Test Driven Development (TDD), Pytest DevOps: Terraform, CI/CD (GitHub Actions) Cloud: AWS, including EC2, Lambda, RDS, CloudWatch, IAM, AWS API usage via command line, SDK, & Step Functions Data: Advanced SQL, Postgres, database modelling, normalisation, warehouse design Design patterns: Star Schema warehouse. : Agile/Scrum, Trello management & Orchestration. Our team had a great depth of knowledge of the technologies which enabled ourselves to successfully build a stable, streamlined and efficient ETL data pipeline, as well using our initial knowledge we learnt and successfully deployed new technologies successfully to provide a fully working and operational solution.
Challenges Faced
AWS lambda Layer limit caused issues but with learning of new technologies enabled us to overcome this issue effectively by refactoring to use of Docker image.