The recent updates to Goku have focused on optimizing storage and resource usage, resulting in a remarkable 37% reduction in stored time series data. By introducing features like metrics namespace and top write-heavy metrics identification, Pinterest has streamlined data management, ensuring efficient use of resources without compromising service quality.
Through the introduction of improved indexing for metric names and dictionary encoding, Pinterest has successfully decreased memory usage from 12 GB to just 3 GB per host. This overhaul, alongside optimizations in memory allocation, has not only enhanced performance but also enabled the use of less expensive hardware, thus significantly cutting infrastructure costs.
The designed metrics namespace dynamically organizes metric configurations and dramatically improves data management. This development ensures that the Goku ecosystem can efficiently adjust to changes in configurations, ultimately leading to lower data storage needs and better performance.
Addressing internal fragmentation and over-allocated memory has also been a focal point for the Goku team. With specific changes made to the folly::IOBuf structure, Pinterest has been able to conserve significant memory resources, yielding savings of 8-11 GB per host, further enhancing the system's efficiency.
Collection
[
|
...
]