"The procedure put my knowledge of Hadoop and Spark technical details as well as my Scala coding abilities to the test. Here is a thorough, round-by-round account of my experience. Phase 1: Online Assessment Both theoretical and coding questions were covered in the first round of the online exam. MCQs on Java, NoSQL, Hadoop, Spark, and Scala. To assess practical coding abilities, there are two Scala programming questions. This test primarily assessed my proficiency in Scala fundamentals and problem-solving."
"2. What is YARN? Hadoop's resource management layer is called YARN (Yet Another Resource Negotiator). It controls how CPU and memory resources are distributed throughout the cluster. YARN increases the system's scalability and efficiency by separating resource management from job execution. Key components: Resource Manager (RM): Allocates cluster-wide resources. Node Manager (NM): Manages resources on each node. Application Master (AM): Manages execution of a single job."
A mid-level data engineer interview consisted of an online assessment followed by a technical interview. The online assessment combined MCQs on Java, NoSQL, Hadoop, Spark, and Scala with two Scala programming questions emphasizing Scala fundamentals and problem-solving. The technical interview included a Scala coding problem to check Fibonacci membership and conceptual questions on Hadoop YARN and Spark dynamic allocation. YARN was described as Hadoop's resource management layer with Resource Manager, Node Manager, and Application Master components. Spark dynamic allocation was explained as automatic scaling of executors with parameters like spark.dynamicAllocation.enabled and min/max executors. Dynamic allocation saves resources by removing idle executors and improves performance by adding executors when tasks are pending.
Read at Medium
Unable to calculate read time
Collection
[
|
...
]