Something really strange is happening to me. I've stumbled upon a memory leak that happens only when answering a POST request using Twisted and Pypy. The problem is described here: http://stackoverflow.com/questions/21063842/using-twisted-cyclone-pypy-to-ha... It could either be a bug in the Python code that handles the request and leaks, a but in Twisted, or in Pypy. I've been investing the problem for weeks now, with the following conclusion: * CPython does not leak with this code * Tornado with Pypy does not leak with this code * Answering GET requests instead of POST requests does not leak with this code * I tried switching GCs, collecting using gc.collect and setting PYPY_GC_MAX, those attempts did not help * This happens both on Linux and Mac Insights as to how to debug this are very welcome. Thanks, -- Ron Reiter [image: Facebook] <http://www.facebook.com/ron.reiter> [image: Twitter]<http://twitter.com/#!/ronreiter> [image: LinkedIn]<http://www.linkedin.com/profile/view?id=4436607&locale=en_US&trk=tab_pro> Contact me: [image: Skype] ron.reiter
Hi Ron, Exactly same problem I have discovered yesterday in one of my production servers running PyPy 2.2.1, Flask==0.10.1, gunicorn==18.0 and tornado==3.2 on ArchLinux updated on daily basis. I do not have more details on this, but I can say only that POST requests takes at least 10x slower then on CPython. However, I measured time spent inside method and its 0.01s, and actual sending of response takes 30s. This is very strange for reason. How I can help on this and send you more details? Regards, Marko Tasic On Wed, Jan 22, 2014 at 2:08 PM, Ron Reiter <ron.reiter@gmail.com> wrote:
I've discovered the cause of the leak - Cyclone uses BytesIO instead of StringIO. Turns out the following code leaks on Pypy only! from io import BytesIO while True: a = BytesIO() I'm filing a bug right now. On Thu, Jan 23, 2014 at 6:27 PM, Maciej Fijalkowski <fijall@gmail.com>wrote:
-- Ron Reiter [image: Facebook] <http://www.facebook.com/ron.reiter> [image: Twitter]<http://twitter.com/#!/ronreiter> [image: LinkedIn]<http://www.linkedin.com/profile/view?id=4436607&locale=en_US&trk=tab_pro> Contact me: [image: Skype] ron.reiter
On Wed, Jan 22, 2014 at 2:08 PM, Ron Reiter <ron.reiter@gmail.com> wrote:
Hi. I could not quite get it to leak at all (the memory was not growing), but I can try again. Can we start with updating the example so it's runnable (in the comments JP calderone mentions something that's not obvious to me what it should do). Can you also explain for how long you need to run it? What parameters are there etc? All those things makes it really hard for me to debug.
Hi Ron, Exactly same problem I have discovered yesterday in one of my production servers running PyPy 2.2.1, Flask==0.10.1, gunicorn==18.0 and tornado==3.2 on ArchLinux updated on daily basis. I do not have more details on this, but I can say only that POST requests takes at least 10x slower then on CPython. However, I measured time spent inside method and its 0.01s, and actual sending of response takes 30s. This is very strange for reason. How I can help on this and send you more details? Regards, Marko Tasic On Wed, Jan 22, 2014 at 2:08 PM, Ron Reiter <ron.reiter@gmail.com> wrote:
I've discovered the cause of the leak - Cyclone uses BytesIO instead of StringIO. Turns out the following code leaks on Pypy only! from io import BytesIO while True: a = BytesIO() I'm filing a bug right now. On Thu, Jan 23, 2014 at 6:27 PM, Maciej Fijalkowski <fijall@gmail.com>wrote:
-- Ron Reiter [image: Facebook] <http://www.facebook.com/ron.reiter> [image: Twitter]<http://twitter.com/#!/ronreiter> [image: LinkedIn]<http://www.linkedin.com/profile/view?id=4436607&locale=en_US&trk=tab_pro> Contact me: [image: Skype] ron.reiter
On Wed, Jan 22, 2014 at 2:08 PM, Ron Reiter <ron.reiter@gmail.com> wrote:
Hi. I could not quite get it to leak at all (the memory was not growing), but I can try again. Can we start with updating the example so it's runnable (in the comments JP calderone mentions something that's not obvious to me what it should do). Can you also explain for how long you need to run it? What parameters are there etc? All those things makes it really hard for me to debug.
participants (3)
-
Maciej Fijalkowski -
Marko Tasic -
Ron Reiter