To investigate, we took a heap dump using a free trial of this amazing tool called Jxray to analyze memory usage. Jxray revealed that we had underutilized buffers managed by the Netty framework, which was causing excessive memory usage.
We found that our Play WS client relies on Netty through an older version of AsyncHttpClient. This version of Netty was allocating large 16MB buffers by default, while newer versions use 4MB buffers.
Using Datadog's memory leak feature, we noticed that 'Heap After Full GC' kept growing, meaning some objects in the heap's Old generation weren't being cleared.
We traced this to a JSON library, Jsoniter, which uses a default 32KB buffer for serializing data. When handling larger data blobs, Jsoniter's buffer had to expand (in powers of 2) to hold the data, creating a lot of temporary garbage.
Collection
[
|
...
]