data-engineering

Are All Monoliths Bad?

Monolith vs. Microservices: Complexity and team size influence architecture choice. [ more ]

Medium

More Speakers and Sessions Announced for the 2024 Data Engineering Summit

The importance of leveraging big data tools in making business decisions

Strategies and technologies for avoiding monolithic data infrastructure [ more ]

Medium

Announcing the First Speakers for the 2024 Data Engineering Summit

Data-centric AI enables improving models without code changes

Apache Arrow and Parquet allow fast analytic operations and interoperability. [ more ]

Medium

11 Open-Source Data Engineering Tools Every Pro Should Use

Apache Spark is a leading framework for large-scale data processing, offering versatile functionalities like batch processing and stream processing.

Apache Kafka is an open-source streaming platform that is ideal for handling real-time data and high-throughput data feeds.

Snowflake, Amazon Redshift, and Google BigQuery are popular cloud data warehouses, each with unique features that data engineers should understand in order to choose the best fit for their projects. [ more ]

Medium

What Does a Data Engineering Job Involve in 2024?

Data engineers play a crucial role in collecting, storing, and processing data for analysis and decision-making.

Data integration is a key responsibility of data engineers, involving combining data from multiple sources into a single, usable format. [ more ]

Medium

How to Shift from Data Science to Data Engineering

There is a high demand for skilled data engineers.

Data scientists can transition into data engineering due to transferable skills. [ more ]

Medium

Artificial intelligence

Unlocking the Power of Gen AI with Data Engineering

Data engineering is crucial for unlocking the potential of Gen AI applications.

Gen AI and data engineering have a symbiotic relationship, enhancing innovation and efficiency. [ more ]

Medium

Artificial intelligence

Must-Have Prompt Engineering Skills, Preventing Data Poisoning, and How AI Will Impact Various...

Baidu's chatbot Ernie Bot has gained over 100 million users

ODSC has various events and resources related to AI, including a podcast and conference [ more ]

Open Data Science - Your News Source for AI, Machine Learning & more

2 months ago

Artificial intelligence

Where are AI Investments Going in 2024?

The conference will cover data science and AI trends, tools, and techniques.

Partnerships in organizing events can enhance the quality and participation. [ more ]

Medium

EU data protection

The Future of Data Engineering Goes Through Data Contracts

Data engineering grows exponentially with company expansion and mergers.

Data Mesh is a disruptive solution with significant organizational level changes. [ more ]

Talkpython

Python

Data Pipelines with Dagster

Dagster is a powerful tool for creating data pipelines using Python.

Pedram Navid, Head of Data Engineering at Dagster Labs, discusses data pipelines on Talk Python. [ more ]

Medium

2 months ago

Scala

Spark Starter Guide 4.13: Importing Data from a Relational Database (MySQL)

Relational databases are vital for operational data but can also hold valuable analytics data.

Spark simplifies accessing databases to populate Spark DataFrames for analysis. [ more ]

Medium

6 months ago

Scala

Spark Tutorial: Master the Essential Skills for Data Engineering and Data Science

Spark has become the defacto tool for big data processing.

Understanding and mastering Spark is crucial for data engineering and data science job interviews. [ more ]

channelpro

4 months ago

Accenture looks to bolster AI capabilities with Redkite acquisition

Accenture acquires data consultancy Redkite to enhance data and AI capabilities [ more ]

Berlin Startup Jobs

5 months ago

Job Vacancy: (Senior) Big Data Engineer * // GameDuell | IT / Software Development Jobs | Berlin Startup Jobs

GameDuell is a leading casual gaming company in Germany with over 130 million registered players worldwide.

They are looking for a (Senior) Big Data Engineer to build and maintain their Big Data infrastructure. [ more ]

Theregister

5 months ago

Microsoft and Databricks try to sew up the data platform

Microsoft Fabric is being hailed as one of Microsoft's biggest data product launches since SQL Server.

Fabric offers data engineering, data lakes, data warehousing, machine learning, and AI on a single platform.

Competitors like Snowflake and Google are catching up quickly in the race to offer similar features and capabilities. [ more ]

DATAVERSITY

6 months ago

Managing Missing Data in Analytics - DATAVERSITY

Today, corporate boards and executives understand the importance of data and analytics for improved business performance.

Summit.ai

2 months ago