Speaker Detail

Dealing Workflow using Airflow

Apache airflow is a platform to programmatically author, schedule and monitor workflows. It originally created by Airbnb and recently has become Apache Top-level Project. Many companies use Airflow to manage their workflows. One of them is HappyFresh. HappyFresh utilize Airflow to manage and monitor its ETL process. Another use cases that suitable for Airflow are: machine learning pipelines, data warehousing, orchestrating automated test, etc.

Airflow is written using Python, and it extensible. You just need to define your entire workflow using python code, or you can extend Airflow using your own python modules.

This talk will discuss about Airflow, define DAG for workflow, and how to scale Airflow for huge number of workflow. This is suitable for all audience levels, especially who wants to manage their workflow.