Python3 "pickle" vs. stdin/stdout - unable to get clean byte streams in Python 3

[John Nagle]

On 3/12/2015 2:56 PM, Cameron Simpson wrote:
> On 12Mar2015 12:55, John Nagle <nagle at animats.com> wrote:
>> I have working code from Python 2 which uses "pickle" to talk to a
>> subprocess via stdin/stdio.  I'm trying to make that work in Python
>> 3. First, the subprocess Python is invoked with the "-d' option, so
>> stdin and stdio are supposed to be unbuffered binary streams.
> 
> You shouldn't need to use unbuffered streams specifically. It should
> be enough to .flush() the output stream (at whichever end) after you
> have written the pickle data.

    Doing that.

    It's a repeat-transaction thing.  Main process sends pickeled
item to subprocess, subprocess reads item, subprocess does work,
subprocess writes picked item to parent.  This repeats.

    I call writer.clear_memo() and set reader.memo = {} at the
end of each cycle, to clear Pickle's cache.  That all worked
fine in Python 2.  Are there any known problems with reusing
Python 3 "pickle"s streams?

    The identical code works with Python 2.7.9; it's converted to Python
3 using "six" so I can run on both Python versions and look for
differences.  I'm using Pickle format 2, for compatibility.
(Tried 0, the ASCII format; it didn't help.)

> I'm skipping some of your discussion; I can see nothing wrong. I
> don't use pickle itself so aside from saying that your use seems to
> conform to the python 3 docs I can't comment more deeply. That said,
> I do use subprocess a fair bit.

     I'll have to put in more logging and see exactly what's going
over the pipes.

				John Nagle