[Python-Dev] PEP 574 -- Pickle protocol 5 with out-of-band data

[Nathaniel Smith]

On Thu, Mar 29, 2018, 02:02 Chris Angelico <rosuav at gmail.com> wrote:

> On Thu, Mar 29, 2018 at 7:56 PM, Paul Moore <p.f.moore at gmail.com> wrote:
> > On 29 March 2018 at 09:49, Chris Angelico <rosuav at gmail.com> wrote:
> >> On Thu, Mar 29, 2018 at 7:18 PM, Nathaniel Smith <njs at pobox.com> wrote:
> >>> Another example is the multiprocessing module: it's very safe to
> >>> assume that the parent and the child are using the same interpreter
> >>> :-). There's no fundamental reason you shouldn't be able to send
> >>> bytecode between them.
> >>
> >> You put a smiley on it, but is this actually guaranteed on all
> >> platforms? On Unix-like systems, presumably it's using fork() and thus
> >> will actually use the exact same binary, but what about on Windows,
> >> where a new process has to be spawned? Can you say "spawn me another
> >> of this exact binary blob", or do you have to identify it by a file
> >> name?
> >>
> >> It wouldn't be a problem for the nonportable mode to toss out an
> >> exception in weird cases like this, but it _would_ be a problem if
> >> that causes a segfault or something.
> >
> > If you're embedding, you need multiprocessing.set_executable()
> > (
> https://docs.python.org/3.6/library/multiprocessing.html#multiprocessing.set_executable
> ),
> > so in that case you definitely *won't* have the same binary...
>
> Ah, and that also showed me that forking isn't mandatory on Unix
> either. So yeah, there's no assuming that they use the same binary.
>

Normally it spawns children using `sys.executable`, which I think on
Windows in particular is guaranteed to be the same binary that started the
main process, because the OS locks the file while it's executing. But yeah,
I didn't think about the embedding case, and apparently there's also a
little-known set of features for using multiprocessing between arbitrary
python processes:
https://docs.python.org/3/library/multiprocessing.html#multiprocessing-listeners-clients


> I doubt it'll be a problem to pickle though as it'll use some form of
> versioning even in NONPORTABLE mode right?
>

I guess the (merged, but undocumented?) changes in
https://bugs.python.org/issue28053 should make it possible to set the
pickle version, and yeah, if we did add a NONPORTABLE mode then presumably
it would have some kind of header saying which version of python it was
created with, so version mismatches could give a sensible error message.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180329/bde9b4c3/attachment.html>