#delta-lake

[ follow ]
Medium
2 months ago
Scala

Building an IoT Monitoring System with Spark Structured Streaming, Kafka and Scala

Developed IoT Smart Farm Monitoring system using Scala, Apache Spark Streaming, and Kafka for real-time data processing. [ more ]
Medium
7 months ago
Scala

Data Engineering: Getting Started with Delta Lake

Delta Lake is gaining popularity in the realm of Data Lakes compared to Apache Hudi and Apache Iceberg.
This article provides a simple introduction to Delta Lake using Apache Spark + Scala programming language on Spark Shell. [ more ]
thenewstack.io
10 months ago
Data science

Integrating a Data Warehouse and a Data Lake

The Delta Lake open source project integrates data lakes and data warehouses to provide more performance and features in the data lake.
Delta Lake breaks down the barriers between data warehouses and data lakes by providing ACID transactions, RPI ingestions, and metadata scalability.
Databricks is working on providing ways for anyone to contribute their connectors through Delta Lake, allowing compatibility with different databases. [ more ]
[ Load more ]