"Today on Talk Python: What really happens when your data work outgrows your laptop. Matthew Rocklin, creator of Dask and cofounder of Coiled, and Nat Tabris a staff software engineer at Coiled join me to unpack the messy truth of cloud-scale Python. During the episode we actually spin up a 1,000 core cluster from a notebook, twice! We also discuss picking between pandas and Polars, when GPUs help, and how to avoid surprise bills. Real lessons, real tradeoffs, shared by people who have built this stuff. Stick around."
"Spinning up thousands of cores from a notebook is now a few lines of Python From a local Jupyter or VS Code session, you can create a Coiled cluster with parameters like number of workers, architecture, region, and Spot policy, then attach Dask or other engines to it. In the episode we kick off a 2,000-core cluster from a notebook, twice, to show how quickly you can go from idea to massive parallelism. The key is to keep the developer experience simple while hiding the cloud plumbing."
Large Python data workflows can move from a laptop to thousands of cloud cores by creating managed clusters and attaching engines such as Dask. Developers can launch clusters from local Jupyter or VS Code sessions with parameters for worker count, architecture, region, and Spot policy. Tool selection depends on workload shape: pandas for single-machine use, Polars for high-performance single-node analytics, Dask for distributed processing, and DuckDB for certain analytical patterns. GPUs provide benefits for compatible workloads but introduce cost and compatibility tradeoffs. Operational practices, monitoring, and simple developer-facing APIs help control costs and prevent surprise cloud bills.
Read at Talkpython
Unable to calculate read time
Collection
[
|
...
]