Elastic Block Storage (EBS) allows the persistence of data after an EC2 machine is terminated. Let’s go over a few key points about storage as it pertains to EC2 instances. Root device volumes contain the image used to boot the instance (this is the volume mounted to the / directory). …

This article is a continuation of my efforts to learn Go by solving easy Leetcode problems. After working through my last article, I am now pretty comfortable with the basics of Go. Let’s jump right into the next problem — 9. Palindrome Number Palindrome Number - LeetCode Given an integer x, return true if x is palindrome integer. An integer is a palindrome when it reads the same backward…leetcode.com

I want to learn Go, however, I don’t have the energy or time for anything super structured (i.e. courses, books, etc.). I thought it’d be fun to learn Go by jumping into the easiest Leetcode problem I could find. Here’s my plan… Come up with a logical plan for the…

Replicating objects between S3 buckets is a very common ask in the world of data engineering. Let’s take a look at how we could copy data across buckets. Prerequisites 🌏 Knowledge of Terraform. ☁️ An active AWS account. ☕️ Coffee. Setting the Stage Say we have a colleague who needs access to data that…

Introduction and motivation. My goal is to write a few articles, each focusing on a popular data source found at any company, and then tie them all together for a big picture view of a fully-fledged data platform. This article would be the first in such a series. …

👋🏼 Hello all! This article is about gaining familiarity with a few of data engineering’s cornerstone tools. We’ll fetch (manually) an IMBD review dataset from Kaggle, upload it to a Hadoop cluster, aggregate the data using PySpark, and load the results into a Hive table. This exercise is just as…

Introduction Data modeling is an art. It's the ultimate intersection between analytics and engineering, the point at which you are simultaneously thinking about performance, extensibility, and scalability, along with the coverage of possible business queries. I won’t get into the weeds of choosing the ‘right’ data model for your firm, however…

The ability to build and serve an API is an invaluable skill for anyone who works with data. For some companies, the expectations for what you can do with their data may exceed dashboards and reporting. Your data warehouse may contain specific insights and metrics that should be shared with…

As someone who is frequently building data engineering projects, I find myself spending a good deal of time sifting through public api’s, searching for the “right” kind of data that will fit my needs. I may need updated time-series data to build something using an orchestrator such as Airflow. I…