"I work for codecentric, a small consultancy in Germany. Before I joined codecentric, I made my professional education in an insurance company. When I finished this education, I came back to the company and they said, congratulations, and yes, we want you to stay with us. We have a new job for you. Back then I was in the architecture team, whatever that meant. The job is going to be in the data warehouse team."
"The analytics side still doesn't select order id and quantity, and wonders, we have more orders than we had in stock because they didn't know there was an unfulfilled field that they had to track and that has to be selected as well. They're eventually coming up to select this as well, but they want to measure now the unfulfilled items and then they change the data type from a Boolean to an integer."
A professional transitioned from software architecture into the data space and began working with Apache Spark about ten years ago, bringing a software engineering mindset to data problems. Persistent challenges arise when application and analytics teams treat data differently, such as omitting fields or changing types. An example shows analytics missing an unfulfilled field and later encountering a type change from Boolean to integer, producing silent conversions or runtime failures. The scenario highlights common schema evolution issues, the importance of coordinated field selection, explicit data versioning, and engineering practices to maintain compatibility and reliable analytics.
Read at InfoQ
Unable to calculate read time
Collection
[
|
...
]