Made by Team Rannoch
We know what you sold last Summer.
Create a data platform that extracts data from an operational database (and potentially other sources), archives it in a data lake, and makes it available in a remodelled OLAP data warehouse, using an ETL format.
The Team
Luna Birtles
Recent electronics graduate retraining towards data…
engineering to apply an interest and background with statistics.
Ali Anvari
Studied computing at Imperial College and Cornell…
University; interned at JP Morgan and Reuters; experience includes BI at Sophus3 and ETL at Mattereum
Stephen Molano-James
Junior Data Engineer with experience in Full-Stack…
development, mostly using JavaScript and Python.
James Ault
Structural biologist pursuing a career pivot to scientific…
data engineering/scientist.
Yaroslav Davydchuk
I’m originally from Ukraine and bring a rich background in…
staff training and managing educational projects. I have hands-on experience in automating business processes and crafting chatbots to streamline interactions. Now, I’m redirecting my talents towards a more concentrated focus on data and analysis :smiling_imp:️. Outside the world of numbers and trends, you can find me enjoying a good cup of coffee, hitting the pavement for a run, or getting competitive over a board game. Looking forward to diving into the DATA!
Tech Stack
We used: AWS (Eventbridge, Lambda, S3, SNS, Cloudwatch), Terraform, GitHub Actions, Python (pandas, pytest, boto3, moto), PostgreSQL They were the most appropriate technologies available for the project.
Challenges Faced
We faced challenges figuring out approprate properties for our lambda functions (e.g. layers, memory and timeout), ensuring NaN representation from pandas was handled in the PostgreSQL database, ensuring that updates to the lambda code were deployed properly with Terraform, and sequencing our functions correctly to overcome issues with foreigns keys when loading data in to the OLAP database. All of which we overcame.