Google fuses SQL, Python, and Spark in Colab Enterprise push

"Yasmeen Ahmad, managing director, Google Data Cloud, said that the greatest barrier to productivity in data science was switching between database/data warehouse environments to get data with SQL code, only to export and load it into a Python notebook to do machine learning, while configuring a separate Spark cluster. They might then switch to a BI tool just to visualize results, she said."

"Google is therefore previewing a number of enhancements to its Colab Enterprise notebooks in its BigQuery data warehouse and the ML platform Vertex AI, which it says will bring these ideas into reality. Within Colab Enterprise notebooks, Google is previewing native SQL cells that let users employ SQL for data exploration and see the results in a BigQuery DataFrame, a Pythonic DataFrame and machine learning (ML) API powered by the BigQuery engine, where they can build models in Python."

"The Chocolate Factory is also previewing interactive visualization cells, which generate editable charts in the same environment, breaking the barrier between SQL, Python, and visualization, the vendor claimed. Also in Colab Enterprise notebooks, Google offers Data Science Agent, which it claims to have enhanced to incorporate tool usage within its detailed plans, including the use of BigQuery ML for training and inferencing, BigQuery DataFrames for analysis using Python, or large-scale Spark transformations (currently in preview)."

Google is previewing enhancements to Colab Enterprise notebooks that integrate BigQuery and Vertex AI to combine SQL, Python, visualization, and Spark in one environment. Native SQL cells will let users run SQL for data exploration and view results as BigQuery DataFrames or Pythonic DataFrames powered by the BigQuery engine. Interactive visualization cells will produce editable charts inside the notebook. Data Science Agent gains tool-aware plans that include BigQuery ML for training and inferencing, BigQuery DataFrames for Python analysis, and large-scale Spark transformations (currently preview). The aim is to remove data movement and context switching between warehouses, notebooks, Spark clusters, and BI tools.

Read at Theregister