What is Terraform and why should you use it — Have you ever used the AWS console? If so, you’ve probably noticed how tedious it can be to manage services. There must be a better way… Well, there is. After the launch of the major cloud providers in the middle 2000’s, there was a need to manage vast data infrastructures…

The default file format for any data science workflow — Have you ever used pd.read_csv() in pandas? Well, that command could have run ~50x faster if you had used parquet instead of CSV. In this post we will discuss apache parquet, an extremely efficient and well-supported file format. The post is geared towards data practitioners (ML, DE, DS) so we’ll…

Exactly what you need, and no more — There are lots of overly-complex posts about data skew, a deceptively simple topic. In this post, we will cover the necessary basics in 5minutes. The primary source for this post was Spark: The Definitive Guide and here’s the code. Let’s dive in… What is Data Skew? In spark, data are split into chunk of…

A quick guide for moving from SQL to PySpark. — If you know SQL but need to work in PySpark, this post is for you! Spark is rapidly becoming one of the most widely adopted frameworks for big data processing. But why work in native PySpark instead of SQL? Well, you don’t have to. PySpark allows you to create a…

The math and intuition behind ordinary least squares (OLS) — Do you know how linear regression measures effects while “holding everything else constant”? Or how it minimizes the sum of squared error? In this post we’ll discuss how, and much more. We will leverage both a matrix algebra and python to understand what’s going on. Where possible, we will go…

Python is an object oriented programming language but can behave strangely. If you come from other OOP languages, this post may benefit you — In chapter 8 of Fluent Python, Luciano Ramalho discusses how python objects under the hood. Here will define the fundamental concept behind variable storage in python and explore some relevant notes. Without further ado, let’s dive in. 1 — Python Variables are not Boxes

How to use multiple linear regression to determine ATE and statistical significance — Have you ever wanted to speed up an A/B test? Well, here’s arguably the highest ROI solution for variance reduction in an A/B testing setting. Frequentist experimentation is commonly leveraged for experimentation. However, as compared to Bayesian or sequential regimes, frequentist A/B tests often require large sample sizes, which slows…

Do you want to write concise, readable, and efficient code? Well, python decorators may help you on your journey. — In chapter 7 of Fluent Python, Luciano Ramalho discusses decorators and closures. They are not super common in basic DS work, however as you start building production models writing async code, they become an invaluable tool. Without further ado, let’s dive in. 1 — What’s a decorator?

A possible implementation of IBM’s FreaAI — Any time you simplify data using a summary statistic, you lose information. Model accuracy is no different. When simplifying your model’s fit to a summary statistic, you lose the ability to determine where your performance is lowest/highest and why. In this post we discuss the code behind IBM’s FreaAI, an…