Serving Data in the Data Engineering Lifecycle: A Comprehensive GuideData engineering culminates in serving data for analytics, ML, and operations.Data quality and trust are critical in serving data effectively.
InfoQ AI, ML, and Data Engineering Trends in 2024The podcast discusses current AI and ML trends with expert insights, showcasing innovations and the impact of community contributions in these technologies.
These AI & Data Engineering Sessions Are a Must-Attend at ODSC East 2025Organizations are focusing on efficiently and securely integrating advanced AI models at scale.Practical strategies and real-world insights are essential for navigating AI and data engineering challenges.
The State of Data Science 2024: 6 Key Data Science Trends | The PyCharm BlogPython usage in data analysis and machine learning is declining, indicating changing trends in data science.
Serving Data in the Data Engineering Lifecycle: A Comprehensive GuideData engineering culminates in serving data for analytics, ML, and operations.Data quality and trust are critical in serving data effectively.
InfoQ AI, ML, and Data Engineering Trends in 2024The podcast discusses current AI and ML trends with expert insights, showcasing innovations and the impact of community contributions in these technologies.
These AI & Data Engineering Sessions Are a Must-Attend at ODSC East 2025Organizations are focusing on efficiently and securely integrating advanced AI models at scale.Practical strategies and real-world insights are essential for navigating AI and data engineering challenges.
The State of Data Science 2024: 6 Key Data Science Trends | The PyCharm BlogPython usage in data analysis and machine learning is declining, indicating changing trends in data science.
Scala Vs. Python-What Data Engineers Need To KnowScala improves upon Java while remaining JVM-compatible, making it attractive for organizations.
100 Days of Data Engineering on Databricks Day 44: PySpark vs. Scala:The choice between PySpark and Scala significantly affects performance and maintainability in Spark development.
Scala Applications in Data Engineering: A Comprehensive OverviewScala is an ideal choice for data engineering, particularly with big data frameworks like Apache Spark.
Scala Vs. Python-What Data Engineers Need To KnowScala improves upon Java while remaining JVM-compatible, making it attractive for organizations.
100 Days of Data Engineering on Databricks Day 44: PySpark vs. Scala:The choice between PySpark and Scala significantly affects performance and maintainability in Spark development.
Scala Applications in Data Engineering: A Comprehensive OverviewScala is an ideal choice for data engineering, particularly with big data frameworks like Apache Spark.
The Future of Data Engineering: Security, Privacy, and the Path AheadSecurity and privacy are essential to data engineering, integral to ethics and resilience amid evolving challenges.
Building a Data Engineering Center of ExcellenceData engineering is crucial for managing complex data needs and enabling accurate, data-driven business decisions.
The Two Types of Data Engineers You Meet at Work | HackerNoonData engineers are categorized into two archetypes: business-oriented and tech-oriented, each with distinct roles and responsibilities.
Skills required for data engineering success | Computer WeeklyData engineering is critical for managing complex data sources and unlocking data value in organizations.
Understanding Data Generation in Source Systems: How It Works and Real-Time ApplicationsUnderstanding data generation is essential for effective data engineering and creating scalable data pipelines.
The Future of Data Engineering: Security, Privacy, and the Path AheadSecurity and privacy are essential to data engineering, integral to ethics and resilience amid evolving challenges.
Building a Data Engineering Center of ExcellenceData engineering is crucial for managing complex data needs and enabling accurate, data-driven business decisions.
The Two Types of Data Engineers You Meet at Work | HackerNoonData engineers are categorized into two archetypes: business-oriented and tech-oriented, each with distinct roles and responsibilities.
Skills required for data engineering success | Computer WeeklyData engineering is critical for managing complex data sources and unlocking data value in organizations.
Understanding Data Generation in Source Systems: How It Works and Real-Time ApplicationsUnderstanding data generation is essential for effective data engineering and creating scalable data pipelines.
Can Your Data Architecture Handle Tomorrow? Building for Flexibility and Lasting ImpactGood data architecture is essential for effective data engineering and organizational competitiveness.
Can Your Data Architecture Handle Tomorrow? Building for Flexibility and Lasting ImpactGood data architecture is vital for effective data engineering and organizational competitiveness.
Web3 Data Engineering Crash Course | HackerNoonWeb3 data architecture is transforming how enterprise and scientific data are approached, emphasizing cross-organizational data exchange over internal data.
Can Your Data Architecture Handle Tomorrow? Building for Flexibility and Lasting ImpactGood data architecture is essential for effective data engineering and organizational competitiveness.
Can Your Data Architecture Handle Tomorrow? Building for Flexibility and Lasting ImpactGood data architecture is vital for effective data engineering and organizational competitiveness.
Web3 Data Engineering Crash Course | HackerNoonWeb3 data architecture is transforming how enterprise and scientific data are approached, emphasizing cross-organizational data exchange over internal data.
Understanding Data Generation in Source Systems: How It Works and Real-Time ApplicationsData generation is crucial in data engineering lifecycle for reliable processing and transformation.
Serving Data in the Data Engineering Lifecycle: A Comprehensive GuideData serving is the culmination of data engineering, delivering value to users through analytics and applications.
Understanding Data Generation in Source Systems: How It Works and Real-Time ApplicationsData generation is crucial in data engineering lifecycle for reliable processing and transformation.
Serving Data in the Data Engineering Lifecycle: A Comprehensive GuideData serving is the culmination of data engineering, delivering value to users through analytics and applications.
Networking, Hackathons, Meetups, and Other Extra Events Coming to ODSC West 2024The conference provides hands-on AI learning and immersive networking opportunities.Participants can engage in various thematic events including hackathons and summits.ODSC West fosters connections among AI professionals and enthusiasts.
Snowflake brings Cortex Agents to data engineering: what can you do with it?Data engineering is crucial in AI but faces challenges like staff shortages and high expectations.Snowflake's Cortex Agents aim to streamline data tasks, enhancing productivity without human intervention.
A path to better data engineering | Computer WeeklyOrganizations face challenges processing diverse data formats and overcoming data silos.Traditional data engineering methods struggle with the variability of real-world data.Understanding the required skills for data sciences is critical for modern data challenges.
Networking, Hackathons, Meetups, and Other Extra Events Coming to ODSC West 2024The conference provides hands-on AI learning and immersive networking opportunities.Participants can engage in various thematic events including hackathons and summits.ODSC West fosters connections among AI professionals and enthusiasts.
Snowflake brings Cortex Agents to data engineering: what can you do with it?Data engineering is crucial in AI but faces challenges like staff shortages and high expectations.Snowflake's Cortex Agents aim to streamline data tasks, enhancing productivity without human intervention.
A path to better data engineering | Computer WeeklyOrganizations face challenges processing diverse data formats and overcoming data silos.Traditional data engineering methods struggle with the variability of real-world data.Understanding the required skills for data sciences is critical for modern data challenges.
The Future of Data Engineering: Security, Privacy, and the Path AheadSecurity and privacy are essential components of data engineering, not optional add-ons.
Why I Chose Google Cloud Platform (GCP) for Data Engineering: Real-World BenefitsGCP is preferred for data engineering due to its scalability, integrated analytics, and cost-effectiveness.
Oh, you wanted data? Confessions of a Test Management EngineerA focus on data engineering from the outset can enhance test case management tools and improve software development outcomes.
Why I Chose Google Cloud Platform (GCP) for Data Engineering: Real-World BenefitsGCP is preferred for data engineering due to its scalability, integrated analytics, and cost-effectiveness.
Oh, you wanted data? Confessions of a Test Management EngineerA focus on data engineering from the outset can enhance test case management tools and improve software development outcomes.
Shaping an Impactful Data Product StrategyData teams need a collaborative strategy to align and deliver long-term value rather than reacting to immediate demands.
A Platform-Agnostic Approach in Cloud Security for Data Engineers | HackerNoonData is becoming crucial for businesses, and effective data engineering is essential for leveraging cloud technology while addressing associated security risks.
Are All Monoliths Bad?Monolith vs. Microservices: Complexity and team size influence architecture choice.
Council Post: 15 Reasons To Choose Node.js For Product Development In 2025Node.js is a crucial framework for modern web development due to its speed, scalability, and efficiency in handling complex applications.
Are All Monoliths Bad?Monolith vs. Microservices: Complexity and team size influence architecture choice.
Council Post: 15 Reasons To Choose Node.js For Product Development In 2025Node.js is a crucial framework for modern web development due to its speed, scalability, and efficiency in handling complex applications.
A guide to transitioning from a data engineer to product manager role - LogRocket BlogTransitioning from data engineering to product management expands influence, fosters cross-functional leadership, and enhances opportunities to impact business outcomes.
Leveraging AI Platforms to Improve Developer Experience - From Personal Hackathon to AI at ScaleLekan Elesin transitioned from software engineering to data engineering, driven by an early interest in Big Data and self-directed learning.
A guide to transitioning from a data engineer to product manager role - LogRocket BlogTransitioning from data engineering to product management expands influence, fosters cross-functional leadership, and enhances opportunities to impact business outcomes.
Leveraging AI Platforms to Improve Developer Experience - From Personal Hackathon to AI at ScaleLekan Elesin transitioned from software engineering to data engineering, driven by an early interest in Big Data and self-directed learning.
Deep dive on Spark Aggregation APIsComplex aggregation problems require advanced solutions beyond straightforward SQL functions.User Defined Aggregate Functions (UDAFs) are essential for calculating median values in Spark.Performance and implementation ease are critical factors in selecting aggregation techniques.
Computer Weekly Buyer's Guide features list 2025 | Computer WeeklyComputer Weekly's Buyer's Guides educate and guide readers through the IT buying cycle to ensure informed purchasing decisions.
What's the Deal With Data Engineers Anyway? | HackerNoonData engineers build essential data infrastructure, enabling analysts to access structured and timely data for informed decision-making.
10 Important Topics Featured at the 2024 Data Engineering Summit - Summit.aiGenerative AI involves collaboration between data engineers and software engineers.Data infrastructure challenges include data wrangling, scaling systems, and data security.
What's the Deal With Data Engineers Anyway? | HackerNoonData engineers build essential data infrastructure, enabling analysts to access structured and timely data for informed decision-making.
10 Important Topics Featured at the 2024 Data Engineering Summit - Summit.aiGenerative AI involves collaboration between data engineers and software engineers.Data infrastructure challenges include data wrangling, scaling systems, and data security.
Job Vacancy: Data Engineer - Climate Tech - Python // Climatiq | IT / Software Development Jobs | Berlin Startup JobsClimatiq is a climate tech startup focused on driving action through a carbon calculation engine used by organizations globally.
Job Vacancy: Lead Frontend Engineer // GlassFlow | IT / Software Development Jobs | Berlin Startup JobsGlassFlow is developing a hands-free data streaming platform to simplify real-time data management for engineers.
Job Vacancy: Lead Frontend Engineer // GlassFlow | IT / Software Development Jobs | Berlin Startup JobsGlassFlow is developing a user-friendly data streaming platform that simplifies real-time data access for engineers.The role offers a unique chance to shape the future of an innovative data infrastructure startup.The company focuses on creating an inclusive workplace with competitive benefits for its employees.
Job Vacancy: Lead Frontend Engineer // GlassFlow | IT / Software Development Jobs | Berlin Startup JobsGlassFlow is developing a hands-free data streaming platform to simplify real-time data management for engineers.
Job Vacancy: Lead Frontend Engineer // GlassFlow | IT / Software Development Jobs | Berlin Startup JobsGlassFlow is developing a user-friendly data streaming platform that simplifies real-time data access for engineers.The role offers a unique chance to shape the future of an innovative data infrastructure startup.The company focuses on creating an inclusive workplace with competitive benefits for its employees.
Optimizing Uber's Search Infrastructure: Upgrading to Apache Lucene 9.5Uber upgraded its search infrastructure from Apache Lucene 8.0 to 9.5, improving search capabilities and overall performance.
Why Many Data Science Jobs Are Actually Data Engineering | HackerNoonMany data scientist roles primarily involve data preparation and cleaning, not advanced data analysis or machine learning as expected.
Unlocking the Power of Gen AI with Data EngineeringData engineering is crucial for unlocking the potential of Gen AI applications.Gen AI and data engineering have a symbiotic relationship, enhancing innovation and efficiency.
The Future of the Data EngineerMaxime Beauchemin paved the way for data engineering with projects like Apache Airflow and Apache Superset, highlighting the importance of specialized engineers in scaling data science.
Why Many Data Science Jobs Are Actually Data Engineering | HackerNoonMany data scientist roles primarily involve data preparation and cleaning, not advanced data analysis or machine learning as expected.
Unlocking the Power of Gen AI with Data EngineeringData engineering is crucial for unlocking the potential of Gen AI applications.Gen AI and data engineering have a symbiotic relationship, enhancing innovation and efficiency.
The Future of the Data EngineerMaxime Beauchemin paved the way for data engineering with projects like Apache Airflow and Apache Superset, highlighting the importance of specialized engineers in scaling data science.
Choosing Your First Language in Data Engineering: A Beginner's GuideChoosing the right programming language is crucial for your data engineering career.Python is favored for its simplicity, rich libraries, and big data integration.
I failed Meta's technical interview. Here's what it was like and what I wish I'd done differently.Preparation for technical interviews is crucial, but adapting one's approach may be equally important for success.
Mastering Data in the Modern Age with Vishwanadham Mandala | HackerNoonVishwanadham Mandala's career in data engineering exemplifies dedication, leadership, and a commitment to mentoring future technology professionals.
Choosing Your First Language in Data Engineering: A Beginner's GuideChoosing the right programming language is crucial for your data engineering career.Python is favored for its simplicity, rich libraries, and big data integration.
I failed Meta's technical interview. Here's what it was like and what I wish I'd done differently.Preparation for technical interviews is crucial, but adapting one's approach may be equally important for success.
Mastering Data in the Modern Age with Vishwanadham Mandala | HackerNoonVishwanadham Mandala's career in data engineering exemplifies dedication, leadership, and a commitment to mentoring future technology professionals.
Data Observability: Multicloud, GenAI Make Challenges HarderAcceldata's focus on data observability capitalizes on the exponential growth of data and the increasing complexity of managing it across multicloud systems.
Edo Liberty on Vector Databases for Successful Adoption of Generative AI and LLM based ApplicationsVector databases play a critical role in the generative AI or GenAI space.
Data Observability: Multicloud, GenAI Make Challenges HarderAcceldata's focus on data observability capitalizes on the exponential growth of data and the increasing complexity of managing it across multicloud systems.
Edo Liberty on Vector Databases for Successful Adoption of Generative AI and LLM based ApplicationsVector databases play a critical role in the generative AI or GenAI space.
Why to avoid multiple chaining of withColumn() function in Spark job.Chaining multiple withColumn() calls in Spark may lead to performance issues and inefficient resource usage.
Understanding Spark Re-PartitionSpark's repartition() function is crucial for managing data skewness, optimizing performance, memory utilization, and downstream query efficiency.
Why to avoid multiple chaining of withColumn() function in Spark job.Chaining multiple withColumn() in Spark can slow down execution and increase memory usage.
Why to avoid multiple chaining of withColumn() function in Spark job.Chaining multiple withColumn() calls in Spark may lead to performance issues and inefficient resource usage.
Understanding Spark Re-PartitionSpark's repartition() function is crucial for managing data skewness, optimizing performance, memory utilization, and downstream query efficiency.
Why to avoid multiple chaining of withColumn() function in Spark job.Chaining multiple withColumn() in Spark can slow down execution and increase memory usage.
With Databricks Apps, business users get more out of dataDatabricks Apps enhance data accessibility for business users, enabling quicker insights without extensive engineering work.
Databricks launches LakeFlow to help its customers build their data pipelines | TechCrunchDatabricks introduced LakeFlow as its internal data engineering solution to handle data ingestion, transformation, and orchestration, reducing the reliance on third-party tools.
Snowflake & Databricks need Data FinOps - Something Chaos Genius excels at - AmazicChaos Genius offers a solution to optimize data engineering for cost and performance.
Are the table format wars entering the final chapter?Databricks' acquisition of Tabular for $1 billion underscores the rising importance of the Apache Iceberg table format in data engineering.
Databricks launches LakeFlow to help its customers build their data pipelines | TechCrunchDatabricks introduced LakeFlow as its internal data engineering solution to handle data ingestion, transformation, and orchestration, reducing the reliance on third-party tools.
Snowflake & Databricks need Data FinOps - Something Chaos Genius excels at - AmazicChaos Genius offers a solution to optimize data engineering for cost and performance.
Are the table format wars entering the final chapter?Databricks' acquisition of Tabular for $1 billion underscores the rising importance of the Apache Iceberg table format in data engineering.
The Importance of Data Structures and Algorithms in the Life of a Data EngineerMastering Data Structures and Algorithms is crucial for optimizing data engineering tasks.
Breaking Down the Worker Task Execution in Apache DolphinScheduler | HackerNoonApache DolphinScheduler is an enterprise-level visual workflow scheduling system that offers flexibility, scalability, and robust fault tolerance.
LLMs: An Assessment From a Data Engineer | HackerNoonAI like GenAI and ChatGPT can enhance data engineering productivity with precise requirements.AI is not likely to fully replace human expertise in data engineering; areas like basic data querying, troubleshooting pipeline failures, and anomaly detection still require human intervention.
Job Vacancy: Senior Data Engineer // Latana | IT / Software Development Jobs | Berlin Startup JobsLatana provides brand insights for better marketing decisions and works with top B2C brands like Headspace and Unilever to optimize brand performance.
Using OpenTelemetry to monitor Apache AirflowMonitoring Airflow is vital for optimizing performance and reliability of data pipelines.
Podcast: HPCC-Open-Source Platform High-Performance Computing on Large-Scale DataDiscover HPCC, a high-performance computer cluster for data analytics, through the insights of Bob Foreman, in an episode of Ai X Podcast.