parallel_bulk implementation causes potential memory leak

[ekneg54]

as mentioned here: opensearch-project/opensearch-py#915
this has good potential to resolve our memory leak problems.

[mhoff]

Nice! Looking forward to the upstream changes :)

I have already tried to reproduce the opensearch-related memory leakage locally but couldn't reproduce the behavior up to now. The reason for this might be that the spawned subprocesses are not properly instrumented by memray, though. I will investigate further.

I found another memory leakage in the configuration refresh which stems from the Metrics objects which are constructed for each refresh but not completely deallocated afterwards.

Also the setup of some components, which is also carried out for the processors in the refresh, might lead to further leakages as these in parts register callbacks on global objects (refreshable getter, scheduler) which circumvent garbage collection.

[ekneg54]

If you spend issue tickets, I could spend time.