Speaker Detail

Best Practice for Managing Data Lineage using Python

# Data Engineer Pain Points

Being a data engineer that maintaining data transformation pipeline is having several pain points. Since we care most about data quality, we want to make sure our data is 100% correct, valid, completed, and unique per key. With that mandatory quality, we also want to deploy our daily job in the most efficient way. Moreover, if the data points are huge and the relations between tables are complicated, it's hard to know the parent-child relationship and its data transformation. So that we need data lineage for that particular reason.

# Data Build Tool Python

Nowadays, many analytics consultant experts find a way to create better data transformation workflow. They chose Python as the programming language and created a tool called Data Build Tool (DBT). Since python is easy to read for the data engineer to the data analyst, it gives a boost to the tools until it becomes used by many companies across the world.

DBT is a command-line tool python-based that using SQL to the data engineer. With DBT, data engineer takes ownership of the entire data transformation workflow, from writing data transformation code to deployment, data quality checking, and data lineage documentation in a single repository.

# Data Transformation in Gojek

We always do the *poc* with the latest technology and concept to finding the best way to maintain our data transformation job. And we think DBT Python is the new way to develop our data pipeline with better quality and documentation.