Medium
10 months agoData science
Psyberg: Automated end to end catch up
The stateful sequential load ETL pipeline in Psyberg monitors the customer lifecycle using fact tables for signup, plan, and cancel data.
The ETL pipeline has features like catchup threshold, data load type, and metadata recording.
The system automatically catches up with late-arriving data by identifying and appending the new data to the corresponding partitions in the fact tables. [ more ]