100% the human necessity is going to get disrupted, but the severity based on the complexity is high. Even the SQL practitioners won't feel much difference here, and self-serve analytics will catch fire in this area which is something phenomenal! Again, to quote positively, data engineers won't need to guide the stakeholders nearly as much!
LLMs coupled with agents will slash on-call load massively. LLMs processing stack traces/quality failures can recommend the built-in remedies and nudge the ball down the field to address the problem, presumably even over Slack. And that is a fantastic percentage of data engineering hours saved! To be clear, some hard failures require manual troubleshooting.
Regardless of the structure, format, source, complexity, timeliness, or even natural play of data, large language models can identify anomalies. LLMs are well-suited to anomaly identification because they can work across a variety of data sources, identify subtle patterns, and distinguish when data isn't behaving as expected. This will help data engineers to identify anomalies.
Collection
[
|
...
]