Netflix has migrated all its data processing to Maestro, the latest generation of home-grown workflow orchestration platform catering to various use cases, including ETL pipelines, ML model training, batch jobs, etc.
The IPS solution implemented a lightweight approach to create an additional Iceberg table (called ICDC table) that only stores references from the original table without copying any data files, which is efficient and cost-effective.
[
add
]
[
|
|
...
]