#data-pipelines

[ follow ]
#data-engineering #data-contracts #schema-management #python #cicd #django #terminal-tools #security-testing #sast
Django
fromPycoders
4 hours ago

PyCoder's Weekly | Issue #734

Python and related tooling updates cover CI performance, Django concurrent task APIs, terminal-based code assistance, and guidance for debugging data pipelines and terminal UIs.
#data-engineering
DevOps
fromMedium
4 hours ago

Building Secure Data Pipelines: Where SAST Fits in the Development Lifecycle

Data pipelines move sensitive data across systems and are vulnerable; SAST scans pipeline code, configs, and infrastructure to catch flaws before production.
DevOps
fromMedium
4 hours ago

Building Secure Data Pipelines: Where SAST Fits in the Development Lifecycle

Data pipelines move sensitive data across systems and are vulnerable; SAST scans pipeline code, configs, and infrastructure to catch flaws before production.
more#data-engineering
DevOps
fromInfoQ
1 week ago

From Batch to Micro-Batch Streaming: Lessons Learned the Hard Way in a Delta Index Pipeline

Batch pipelines often face more delays from scheduling than processing costs; continuous micro-batching can reduce latency effectively.
Startup companies
fromInfoQ
3 months ago

Etleap Launches Iceberg Pipeline Platform to Simplify Enterprise Adoption of Apache Iceberg

Managed Iceberg pipeline platform unifies ingestion, transformation, orchestration, and table operations inside customers' VPCs, enabling enterprise Iceberg adoption without building custom stacks.
#data-contracts
more#data-contracts
Data science
fromInfoQ
4 months ago

Decathlon Switches to Polars to Optimize Data Pipelines and Infrastructure Costs

Migrating small-to-medium data workloads from Apache Spark to Polars yields major performance and cost improvements by enabling single-node execution and faster in-memory processing.
Artificial intelligence
fromInfoQ
5 months ago

Nexla Launches Express: A Conversational Platform for AI Data Engineering

Express enables natural-language creation of production-ready data pipelines by automating data discovery, connection, and transformation without manual coding.
[ Load more ]