Pandas emerged 17 years ago and became a core tool for interacting with tabular data in Python. Newer dataframe frameworks addressed pandas limitations and inspired a return to improved syntax. A new column-assignment syntax arriving in pandas 3.0 supports chainable operations without lambdas. Traditional assignments either modify dataframes in-place or use lambda functions for chaining. Lambda functions suffer from unpredictable scoping and poor introspection, making behavior hard to reason about and validating user-provided functions impractical. The new pd.col-style syntax aims to enable safer, more predictable, and more readable column transformations suitable for method chaining.
17 years ago, pandas came onto the scene and took the Python data science scene by storm. It provided data scientists with an efficient way to interact with tabular data and solve real problems. Over time, other frameworks have emerged, taking inspiration from pandas whilst addressing some of its many limitations. Recently, we've come full-circle, and pandas has introduced a new syntax inspired by the newer wave of dataframe libraries. Let's learn about why, and how you can use it!
Suppose you've got a dataframe of city names and temperatures: Let's look at how we can make a new column 'temp_f' which converts 'temp_c' to Fahrenheit. The first option modifies the original object df in-place, and isn't suitable for method-chaining. The second option allows for method chaining, but requires using a lambda function. The third option uses the new syntax coming in pandas 3.0. But why is it an improvement over the second option, what's so bad about lambda functions? There are a few reasons:
Scoping rules make their behaviour hard to predict (example below!). They are opaque and non-introspectable. Try printing one out on the console, and you'll see something incomprehensible like <function <lambda> at 0x76b583037560>. If you receive a lambda function from user input, you have no way to validate what's inside (unless you enjoy reverse-engineering byte-code, and even then, you won't be able to do it in general).
Collection
[
|
...
]