Synapse media repository workers seem unbounded in memory usage, causing high GC times

[michaelkaye]

Currently we seem to be unbounded in memory usage on the media repository. This doesn't appear to be cache related (i'm not even sure what we would cache). Having added 3 media_repo workers we're now tripling this overhead.

For what they do, I don't believe they should be using this much (seen up to 24Gb * 3) memory, and they have a very clean sawtooth profile for memory usage based on restarts

This memory is making its way into gen2 and so GC times are starting to take up to 50s when we near the full memory size.

Could be a leak in some thumbnailing code or something?

[anoadragon453]

This looks to have been a problem for multiple months on matrix.org now, and is not a recent regression afaik.

[michaelkaye]

Is this something we're investigating now?

[anoadragon453]

Not actively. I've added it to the board so we don't forget about it.

[erikjohnston]

Next steps:

1. Use pympler in a manhole to see if there are any obvious objects that are leaking (after having left it to run for a while).
2. Check if GC gen 2 is taking as long as expected. If not try disabling Jaeger/opentracing for the media repos to see if that helps (we might be leaking in native land rather than Python land)

[turt2live]

There also seems to be a matching file descriptor leak on this - maybe it's holding files in memory or something?

[erikjohnston]

We seem to be leaking sockets in CLOSE_WAIT:

$ lsof -n -p 54367 | grep -Eo "\(.*\)$" | sort | uniq -c
   1229 (CLOSE_WAIT)
    190 (ESTABLISHED)
      3 (LISTEN)

[richvdh]

This seems much improved now (presumably thanks to #9421):

We still have a sawtooth on the "open FDs": raising that as #9488