Our investigation revealed that the high number of concurrent image pulls during updates led to severe disk utilization and prolonged cold start times for pods.
The Docker pulls, combined with the sheer volume of pods needing to be started at once, resulted in a bottleneck that caused the system to become entirely stuck.
Utilizing spot instances for cost-efficiency added an additional layer of complexity to our deployment process, especially during simultaneous upgrades.
We found that changing our deployment strategy could significantly reduce the cold start times by staggering pod launches and managing the image pull load more effectively.
Collection
[
|
...
]