The Team

Beth Suffield

I have a background in Digital Marketing and SEO, and…

particularly enjoyed the technical side of these roles. I am especially motivated by the exciting advancements currently being made in Machine Learning and how this impacts the data field.

Mathieu Rees

Junior Software Engineer | Backend, Data, Cloud | Python,…

SQL, AWS

Heila Lok

Motivated by hands on development, learning and engaging…

myself in practical experiences in the world of technology with the use of languages; Python and SQL.

Matthew Burns

Maths and Physics graduate looking to get into tech…

industry. Interested in creating data pipelines and learning about Machine Learning models.

Rajni Bhagat

A computer science graduate, with prior IT industry…

experience. I have a strong interest in the dynamic and innovative field of data engineering.

Tech Stack

We used Python, Terraform, SQL, Pandas, AWS, Boto3, Moto. Python is an intuitive and powerful language with good library support. For example, boto3 allowed us to write our lambda handler functions, which we tested using moto; and Pandas and SQL allowed us to reformat our databases. Terraform enabled us to automate AWS infrastructure.

Challenges Faced

We were trying to upload the code and dependencies for the transformation lambda handler into the aws lambda function. We were reaching a size limit for the upload, so we tried using a Docker container to install the dependencies. We did test-driven coding throughout the project, but found it difficult to effectively test the process of uploading data to the warehouse in the final step, because we couldn’t create a mock of the data warehouse. We connected to a local Postgres database and used it for testing.