This project illustrates integrating Apache Spark and Airflow, submitting Python and Scala jobs for scalable architectures, focusing on simple WordCount tasks as templates for complex workflows.
Ideal for data engineers, it initiates Spark and Airflow integration in containerized environments, offering basic job functionalities extendable to intricate workflows.
Configuration note: Specify Spark master as spark://spark-master in Airflow UI for seamless connection setup.
Collection
[
|
...
]