#etl

[ follow ]
#aws-glue
Medium
1 month ago
Scala

Scala Jobs on AWS Glue: A Practical Guide to Development, Local Testing and Deployment

AWS Glue is highly scalable, cost-effective, and integrates well with other AWS services for orchestrating complex pipelines.
Performance issues exist in AWS Glue when dealing with large Python-based Pyspark jobs due to expensive data shuffling between JVM and Python processes. [ more ]
InfoQ
2 months ago
Business intelligence

Amazon Q Data Integration in AWS Glue Simplifies Data Transformation on AWS

AWS Glue introduces natural language processing for ETL jobs.
Amazon Q provides AI-powered assistance for data integration tasks. [ more ]
moreaws-glue
Medium
3 months ago
UX design

Streamlining chaos: Redesign of a complex Workflow canvas

Understanding the concept of a workflow and its role in organizing tasks and achieving goals.
The ETL workflow focuses on extracting, transforming, and loading data for efficient analysis and use. [ more ]
Medium
4 months ago
Scala

Unlocking Spark's Potential with Scala: split Data Frame by left join. Part 1.

In my practice, there are a lot of ETLs that require to split some data frame into two data frames by condition.A common use case of splitting of data frame is splitting by left join.
[ Load more ]