[Python-ideas] solving multi-core Python

[Stephen J. Turnbull]

Barry Warsaw writes:

 > A crazy offshoot idea would be something like Emacs' unexec, where
 > during the build process you could preload a bunch of always-used
 > immutable modules,

XEmacs doesn't do this any more if it can avoid it, we now have a
portable dumper that we use on almost all platforms.  And everybody at
GNU Emacs who works with the unexec code wants to get rid of it.
XEmacs's legacy unexec requires defeating address space randomization
as well as certain optimizations that combine segments.  I believe
Emacs's does too.  From a security standpoint, Emacsen are a child's
garden of diseases and it will take decades, maybe centuries, to fix
that, so those aren't huge problems for us.  But I suppose Python
needs to be able to work and play nicely with high-security
environments, and would like to take advantage of security-oriented OS
facilities like base address randomization.  That kind of thing hugely
complicates unexec -- last I heard it wasn't just "way too much work
to be worth it", the wonks who created the portable dumper didn't know
how to do it and weren't sure it could be done.

XEmacs's default "portable dumper" is a poor man's relocating loader.
I don't know exactly how it works, can't give details.  Unlike the
unexecs of some Lisps, however, this is a "do it once per build
process" design.  There's no explicit provision for keeping multiple
dumpfiles around, although I believe it can be done "by hand" by
someone with a little bit of knowledge.  The reason for this is that
the dumpfile is actually added to the executable.

Regarding performance, the dumper itself is fast enough to be
imperceptible to humans at load time, and doesn't take very long to
build the dump file containing the "frozen" objects when building.  I
suspect Python has applications where it would be like to be faster
than that, but I don't have benchmarks so don't know if this approach
would be fast enough.

This approach has the feature (disadvantage?) that some objects can't
be dumped including editing buffers, network connections, and
processes.  I suppose those restrictions are very similar to the
restrictions imposed by pickle.

If somebody wants to know more about the portable dumper, I can
probably connect them with the authors of that feature.