fromTheregister
4 days agoDuckDB uses RDBMS to tackle lakehouse 'small changes' issue
You make a small change to your table, adding a single row, and it affects data lake performance because, due to the way they work, a new file has to be written that contains one row, and then a bunch of metadata has to be written. This is very inefficient, because formats like Parquet really don't want to store a single row, they want to store a million rows.
Data science











