Last year witnessed the explosive rise of large models, generating global enthusiasm and making AI seem like a solution to all problems. This year, as the hype subsides, large models have entered a deeper phase, aiming to reshape the foundational logic of various industries.
Large models, capable of autonomously learning rules and discovering patterns from vast datasets, are undeniably impressive. However, my answer is clear: ETL will not disappear. Large models still fail to address several core data challenges: Efficiency Issues.
Despite their outstanding performance in specific tasks, large models incur enormous computational costs. Training a large-scale Transformer model may take weeks and consume vast amounts of energy and financial resources. By contrast, ETL, which relies on predefined rules and logic, is efficient, resource-light.
For everyday enterprise data tasks, many operations remain rule-driven, such as Data Cleaning: Removing anomalies using clear rules or regular expressions and Format Conversion: Standardizing formats to facilitate data transmission and integration across systems.
Collection
[
|
...
]