More memory allocated for GC than arenas+rawmalloc+nursery (bug?)
So, I’ve been testing moving some of my projects to using PyPy as an interpreter from CPython, but PyPy 6.0.0/ Python 3.5.3 seems to memory bloat and trigger OOM protections in about 6 hours on a server with 2GB of RAM, while CPython 3.6.2 ran for 24 days with a more or less constant memory footprint of ~670MB (+/- 40MB) on my discord bot, Birb. Output of gc.get_stats() about 2-3 hours before the crash are as follows: Total memory consumed: GC used: 641.5MB (peak: 775.5MB) in arenas: 536.5MB rawmalloced: 102.9MB nursery: 2.0MB raw assembler used: 5.5MB —————————————— Total: 647.0MB Total memory allocated: GC allocated: 1235.9MB (peak: 1303.8MB) in arenas: 602.8MB rawmalloced: 170.8MB nursery: 2.0MB raw assembler allocated: 6.0MB —————————————— Total: 1241.9MB This behaviour can be successfully replicated on brew’s macOS build of PyPy3 6.0.0 and the official PyPy3 6.0.0 release for 64-bit Ubuntu. To replicate this behaviour, I can use the following code, after installing the rewrite branch on discord.py from git+https://github.com/Rapptz/discord.py@rewrite#egg=discord.py; ``` import gc from discord.ext import commands bot = commands.AutoShardedBot(“>>”) # Variable passed is prefix, changing will not affect result @bot.command() async def get_gc_state(ctx): await ctx.send(gc.get_stats()) bot.run(TOKEN) # Replace TOKEN with authentication token, number of guilds/users visible will affect result ``` Unfortunately, I can only reproduce this issue on my production instance, with ~8.6k guilds, ~143.7k channels, and ~305k visible user accounts. On my test instance with 1 guild, 7 channels and 12 users, there are no measurable memory issues. Of note is that under total memory allocated, the sum of the parts of the GC allocation (602.8+170.8+2.0=775.6) is just a little over 62.75% of the total memory reported as being allocated for the GC. Is this a PyPy bug? Thanks for reading, Oxylibrium.
participants (1)
-
Oxylibrium Person