[New-bugs-announce] [issue23655] Memory corruption using pickle over pipe to subprocess

John Nagle report at bugs.python.org
Fri Mar 13 07:47:15 CET 2015

New submission from John Nagle:

I'm porting a large, working system from Python 2 to Python 3, using "six", so the same code works with both. One part of the system works a lot like the multiprocessing module, but predates it. It launches child processes with "Popen" and talks to them using "pickle" over stdin/stdout as pipes.  Works fine under Python 2, and has been working in production for years.

Under Python 3, I'm getting errors that indicate memory corruption:

Fatal Python error: GC object already tracked

Current thread 0x00001a14 (most recent call first):
  File "C:\python34\lib\site-packages\pymysql\connections.py", line 411
in description
  File "C:\python34\lib\site-packages\pymysql\connections.py", line 1248
in _get_descriptions
  File "C:\python34\lib\site-packages\pymysql\connections.py", line 1182
in _read_result_packet
  File "C:\python34\lib\site-packages\pymysql\connections.py", line 1132
in read
  File "C:\python34\lib\site-packages\pymysql\connections.py", line 929
in _read_query_result
  File "C:\python34\lib\site-packages\pymysql\connections.py", line 768
in query
  File "C:\python34\lib\site-packages\pymysql\cursors.py", line 282 in
  File "C:\python34\lib\site-packages\pymysql\cursors.py", line 134 in
  File "C:\projects\sitetruth\domaincacheitem.py", line 128 in select
  File "C:\projects\sitetruth\domaincache.py", line 30 in search
  File "C:\projects\sitetruth\ratesite.py", line 31 in ratedomain
  File "C:\projects\sitetruth\RatingProcess.py", line 68 in call
  File "C:\projects\sitetruth\subprocesscall.py", line 140 in docall
  File "C:\projects\sitetruth\subprocesscall.py", line 158 in run
  File "C:\projects\sitetruth\RatingProcess.py", line 89 in main
  File "C:\projects\sitetruth\RatingProcess.py", line 95 in <module>

That's clear memory corruption.


  File "C:\projects\sitetruth\InfoSiteRating.py", line 200, in scansite
    if len(self.badbusinessinfo) > 0 :                  # if bad stuff
NameError: name 'len' is not defined

There are others, but those two should be impossible to cause from Python source. 

I've done the obvious stuff - deleted all .pyc files and Python cache directories.  All my code is in Python. Every library module came in via "pip", into a clean Python 3.4.3 (32 bit) installation on Win7/x86-64.

Currently installed packages (via "pip list")

beautifulsoup4 (4.3.2)
dnspython3 (1.12.0)
html5lib (0.999)
pip (6.0.8)
PyMySQL (0.6.6)
pyparsing (2.0.3)
setuptools (12.0.5)
six (1.9.0)

Nothing exotic there.  The project has zero local C code; any C code came 
from the Python installation or the above packages, most of which are pure Python.

It all works fine with Python 2.7.9.  Everything else in the program seems
to be working fine under both 2.7.9 and 3.4.3, until subprocesses are involved.

What's being pickled is very simple; no custom objects, although Exception types are sometimes pickled if the subprocess raises an exception.  

Pickler and Unpickler instances are being reused here.  A message is pickled, piped to the subprocess, unpickled, work is done, and a response comes back later via the return pipe.  A send looks like:

    self.writer.dump(args)      # send data
    self.dataout.flush()        # finish output
    self.writer.clear_memo()    # no memory from cycle to cycle

and a receive looks like:

    result = self.reader.load() # read and return from child
    self.reader.memo = {}       # no memory from cycle to cycle

Those were the recommended way to reset "pickle" for new traffic years ago.
(You have to clear the receive side as well as the send side, or the dictionary
of saved objects grows forever.) My guess is that there's something about reusing "pickle" instances that botches memory uses in CPython 3's C code 
for "cpickle".  That should work, though; the "multiprocessing" module works
by sending pickled data over pipes.

The only code difference between Python 2 and 3 is that under Python 3 I have to use "sys.stdin.buffer" and "sys.stdout.buffer" as arguments to Pickler and Unpickler. Otherwise they complain that they're getting type "str".

Unfortunately, I don't have an easy way to reproduce this bug yet. 

Is there some way to force the use of the pure Python pickle module under Python 3? That would help isolate the problem.

				John Nagle

components: Library (Lib)
messages: 238009
nosy: nagle
priority: normal
severity: normal
status: open
title: Memory corruption using pickle over pipe to subprocess
versions: Python 3.4

Python tracker <report at bugs.python.org>

More information about the New-bugs-announce mailing list