Databricks originally was a Notebook interface to run Spark, without having to worry about the distributed compute infrastructure.
Scala really shines in big data processing for many reasons, the type safety, immutability and functional paradigms alone allow you to write incredibly concise and much less error prone code.
Collection
[
|
...
]