Why A Separation Of Church And State Created Broken Data Pipelines

"Data relationships are complicated. Your airline flight membership login doesn't show your last flight or the fact that you got miles for a broken in-seat screen, your bank doesn't register the fact that it offered you the same investment last month (as it peddles it again this week), or your TV movies login suddenly forgets who you are - and that's just at the consumer level."

"Part of the answer comes down to how data is ingested, absorbed and ultimately given a place to live as it is stored and inwardly managed inside an enterprise data stack. Traditionally, there has been a separation of church and state in this area of information management i.e. structured data (customer records or financial data as accurately identifiable and quantifiable business transactions, some of the more standardized sensor data from the IoT etc.) is stored in databases;"

"What's a file system in this instance if a database itself is a collection of files? A database is agreed to be a formalised repository of data that uses a tabular and column-based structure to enable queryable relationships that exist between data records. A file system (which might be a data lake, a data warehouse or a combo-hybrid data lakehouse) lacks the structure, security of a database and... essentially, lacks the concurrent queryability of a database, because it's a ship built for storage, not steerage."

Data relationships often break across systems, producing missing transactions and inconsistent customer records. Poor data management and weak integration cause lost commercial opportunities, reduced operational efficiency, failed or inadequate compliance actions, and increased business risk. A major root cause is the historical separation of systems: structured, tabular data lives in databases while unstructured content is stored in file systems or data lakes. Databases provide formalized, secure, queryable relationships; file systems and lakes prioritize storage over structure, security and concurrent queryability. That architectural divide undermines accurate, timely, and integrated enterprise data views.

Read at Techzine Global