Testing the compiler optimizations your code relies on

"But there's a problem with this sort of trick: how do you know the compiler will keep doing it? What happens when the compiler's next release comes out? How can you catch performance regressions? One solution is benchmarking: you measure your code's speed, and if it gets a lot slower, something has gone wrong. This is useful and important if you care about speed. But it's also less localized, so it won't necessarily immediately pinpoint where the regression happened."

"By running it with different input sizes and comparing the run time. The problem with run time is that it's noisy: To get around this problem of noise, we can run the function many times and average the run time: Now that we have a slightly more reliable measure of speed, we can see how range_sum()'s speed changes for different input sizes."

A loop that appears to be O(n) can be optimized by a compiler into a constant-time sequence of instructions, offering substantial speedups. The optimization behavior can vary across compiler releases, creating a risk of silent regressions. Benchmarking by measuring runtime across input sizes can reveal whether an operation is effectively constant-time or truly linear, but raw timings are noisy. Running the function many times and averaging reduces noise and produces a more reliable speed measure. Comparing optimized and non-optimized variants, such as range_sum() versus range_sum_of_logs(), shows constant-time behavior versus O(n) scaling. Targeted tests that require the optimization can complement benchmarking.

Read at PythonSpeed