Apache Airflow + Docker [+ Kubernetes] for relatively painless data pipelines

Gordon Inggs

LinuxConf [ZA] 2019

So you have a data science environment, and you want to do a series of ETL operations? Sounds like you need a data pipeline.

In this talk, Gordon will first motivate why you might want a data pipeline. He will then describe how to use Apache Airflow and Docker to build MVP data pipelines with minimal effort and a moderate degree of fuss. Finally, possibly overstretching, he will show you how to complicate this idea by introducing Kubernetes into the mix.

pipelines