2.6, 3.0, and truly independent intepreters

[Rhamphoryncus]

On Oct 22, 10:32 am, Andy <and... at gmail.com> wrote:
> Dear Python dev community,
>
> I'm CTO at a small software company that makes music visualization
> software (you can check us out atwww.soundspectrum.com).  About two
> years ago we went with decision to use embedded python in a couple of
> our new products, given all the great things about python.  We were
> close to using lua but for various reasons we decided to go with
> python.  However, over the last two years, there's been one area of
> grief that sometimes makes me think twice about our decision to go
> with python...
>
> Some background first...   Our software is used for entertainment and
> centers around real time, high-performance graphics, so python's
> performance, embedded flexibility, and stability are the most
> important issues for us.  Our software targets a large cross section
> of hardware and we currently ship products for Win32, OS X, and the
> iPhone and since our customers are end users, our products have to be
> robust, have a tidy install footprint, and be foolproof.  Basically,
> we use embedded python and use it to wrap our high performance C++
> class set which wraps OpenGL, DirectX and our own software renderer.
> In addition to wrapping our C++ frameworks, we use python to perform
> various "worker" tasks on worker thread (e.g. image loading andprocessing).  However, we require *true* thread/interpreter
> independence so python 2 has been frustrating at time, to say the
> least.  Please don't start with "but really, python supports multiple
> interpreters" because I've been there many many times with people.
> And, yes, I'm aware of the multiprocessing module added in 2.6, but
> that stuff isn't lightweight and isn't suitable at all for many
> environments (including ours).  The bottom line is that if you want to
> perform independentprocessing (in python) on different threads, using
> the machine's multiple cores to the fullest, then you're out of luck
> under python 2.
>
> Sadly, the only way we could get truly independent interpreters was to
> put python in a dynamic library, have our installer make a *duplicate*
> copy of it during the installationprocess(e.g. python.dll/.bundle ->
> python2.dll/.bundle) and load each one explicitly in our app, so we
> can get truly independent interpreters.  In other words, we load a
> fresh dynamic lib for each thread-independent interpreter (you can't
> reuse the same dynamic library because the OS will just reference the
> already-loaded one).
>
> From what I gather from the python community, the basis for not
> offering "real" muti-threaded support is that it'd add to much
> internal overhead--and I couldn't agree more.  As a high performance C
> and C++ guy, I fully agree that thread safety should be at the high
> level, not at the low level.  BUT, the lack of truly independent
> interpreters is what ultimately prevents using python in cool,
> powerful ways.  This shortcoming alone has caused game developers--
> both large and small--to choose other embedded interpreters over
> python (e.g. Blizzard chose lua over python).  For example, Apple's
> QuickTime API is powerful in that high-level instance objects can
> leverage performance gains associated with multi-threadedprocessing.
> Meanwhile, the QuickTime API simply lists the responsibilities of the
> caller regarding thread safety and that's all its needs to do.  In
> other words, CPython doesn't need to step in an provide a threadsafe
> environment; it just needs to establish the rules and make sure that
> its own implementation supports those rules.
>
> More than once, I had actually considered expending company resources
> to develop a high performance, truly independent interpreter
> implementation of the python core language and modules but in the end
> estimated that the size of that project would just be too much, given
> our company's current resources.  Should such an implementation ever
> be developed, it would be very attractive for companies to support,
> fund, and/or license.  The truth is, we just love python as a
> language, but it's lack of true interpreter independence (in a
> interpreter as well as in a thread sense) remains a *huge* liability.
>
> So, my question becomes: is python 3 ready for true multithreaded
> support??  Can we finally abandon our Frankenstein approach of loading
> multiple identical dynamic libs to achieve truly independent
> interpreters?? I've reviewed all the new python 3 C API module stuff,
> and all I have to say is: whew--better late then never!!  So, although
> that solves modules offering truly independent interpreter support,
> the following questions remain:
>
> - In python 3, the C module API now supports true interpreter
> independence, but have all the modules in the python codebase been
> converted over?  Are they all now truly compliant?  It will only take
> a single static/global state variable in a module to potentially cause
> no end of pain in a multiple interpreter environment!  Yikes!
>
> - How close is python 3 really to true multithreaded use?  The
> assumption here is that caller ensures safety (e.g. ensuring that
> neither interpreter is in use when serializing data from one to
> another).
>
> I believe that true python independent thread/interpreter support is
> paramount and should become the top priority because this is the key
> consideration used by developers when they're deciding which
> interpreter to embed in their app. Until there's a hello world that
> demonstrates running independent python interpreters on multiple app
> threads, lua will remain the clear choice over python.  Python 3 needs
> true interpreter independence and multi-threaded support!

What you describe, truly independent interpreters, is not threading at
all: it is processes, emulated at the application level, with all the
memory cost and none of the OS protections.  True threading would
involve sharing most objects.

Your solution depends on what you need:
* Killable "threads" -> OS processes
* multicore usage (GIL removal) -> OS processes or alternative Python
implementations (PyPy/Jython/IronPython)
* Sane shared objects -> safethread