#data-pipelines
#data-pipelines

[ follow ]

Decathlon Switches to Polars to Optimize Data Pipelines and Infrastructure Costs

Migrating small-to-medium data workloads from Apache Spark to Polars yields major performance and cost improvements by enabling single-node execution and faster in-memory processing.

Software development

fromInfoWorld

1 month ago

Why data contracts need Apache Kafka and Apache Flink

Data contracts prevent pipeline failures by requiring producers and consumers to agree on schemas, types, and quality constraints enforced with tools like Kafka and Flink.

Artificial intelligence

fromInfoQ

1 month ago

Nexla Launches Express: A Conversational Platform for AI Data Engineering

Express enables natural-language creation of production-ready data pipelines by automating data discovery, connection, and transformation without manual coding.

Social media marketing

fromTechzine Global

4 months ago

CrowdStrike acquires Onum to enrich security data in motion

Real-time, encrypted data-in-motion and preprocessed pipelines are essential for SOCs to stop breaches and reduce noise.

Data science

fromMedium

5 months ago

Building Resilient Data Systems: Key Lessons from Veronika Durgin

Neglected data engineering tasks are crucial for stable and agile data pipelines.