Working Paper
Authors
- Michael Statt Modelyst LLC ,
- Kristopher Brown Stanford University ,
- Santosh Suram Toyota Research Institute ,
- Linda Hung Toyota Research Institute ,
- Daniel Schweigert Toyota Research Institute ,
- John Gregoire California Institute of Technology ,
- Brian Rohr
Modelyst LLC
Abstract
In this work, we present DBgen, a Python library that provides a framework for defining extract-transform-load (ETL) pipelines to create and populate SQL databases. DBgen is most useful when the underlying data has complex relationships, requires multi-step analysis, is large-scale, and the type of data being collected changes frequently. Scientific data often fits this description. With current tooling, defining ETL pipelines for this particularly difficult- to-manage data is so onerous that a great deal of it does not end up being stored in a database and is opaque. DBgen is designed to fill the gap in the current tooling and reduce the barrier to defining ETL pipelines such data.
Content

Supplementary material

Supplemental Information for: DBgen: A Python Library for Defining Scalable, Maintainable, Accessible, Reconfigurable, Transparent (SMART) Data Pipelines
Supplemental Information corresponding to the main text.