2.6, 3.0, and truly independent intepreters

[Rhamphoryncus]

On Oct 22, 7:04 pm, Andy <and... at gmail.com> wrote:
> > What you describe, truly independent interpreters, is not threading at
> > all: it is processes, emulated at the application level, with all the
> > memory cost and none of the OS protections.  True threading would
> > involve sharing most objects.
>
> > Your solution depends on what you need:
> > * Killable "threads" -> OS processes
> > * multicore usage (GIL removal) -> OS processes or alternative Python
> > implementations (PyPy/Jython/IronPython)
> > * Sane shared objects -> safethread
>
> I realize what you're saying, but it's better said there's two issues
> at hand:
>
> 1) Independent interpreters (this is the easier one--and solved, in
> principle anyway, by PEP 3121, by Martin v. Löwis, but is FAR from
> being carried through in modules as he pointed out).  As you point
> out, this doesn't directly relate to multi-threading BUT it is
> intimately tied to the issue because if, in principle, every module
> used instance data (rather than static data), then python would be
> WELL on its way to "free threading" (as Jesse Noller calls it), or as
> I was calling it "true multi-threading".

If you want processes, use *real* processes.  Your arguments fail to
get transaction because you don't provide a good, justified reason why
they don't and can't work.

Although isolated interpreters would be convenient to you, it's a
specialized use case, and bad language design.  There's far more use
cases that aren't isolated (actual threading), so why exclude them?


> 2) Barriers to "free threading".  As Jesse describes, this is simply
> just the GIL being in place, but of course it's there for a reason.
> It's there because (1) doesn't hold and there was never any specs/
> guidance put forward about what should and shouldn't be done in multi-
> threaded apps (see my QuickTime API example).  Perhaps if we could go
> back in time, we would not put the GIL in place, strict guidelines
> regarding multithreaded use would have been established, and PEP 3121
> would have been mandatory for C modules.  Then again--screw that, if I
> could go back in time, I'd just go for the lottery tickets!! :^)

You seem confused.  PEP 3121 is for isolated interpreters (ie emulated
processes), not threading.

Getting threading right would have been a massive investment even back
then, and we probably wouldn't have as mature of a python we do
today.  Make no mistake, the GIL has substantial benefits.  It may be
old and tired, surrounded by young bucks, but it's still winning most
of the races.


> Anyway, I've been at this issue for quite a while now (we're
> approaching our 3rd release cycle), so I'm pretty comfortable with the
> principles at hand.  I'd say the theme of your comments share the
> theme of others here, so perhaps consider where end-user software
> houses (like us) are coming from.  Specifically, developing commercial
> software for end users imposes some restrictions that open source
> development communities aren't often as sensitive to, namely:
>
> - Performance -- emulation is a no-go (e.g. Jython)

Got some real benchmarks to back that up?  How about testing it on a
16 core (or more) box and seeing how it scales?


> - Maturity and Licensing -- experimental/academic projects are no-go
> (PyPy)
> - Cross platform support -- love it or hate it, Win32 and OS X are all
> that matter when you're talking about selling (and supporting)
> software to the masses.  I'm just the messenger here (ie. this is NOT
> flamebait).  We publish for OS X, so IronPython is therefore out.

You might be able to use Java on one, IronPython on another, and PyPy
in between.  Regardless, my point is that CPython will *never* remove
the GIL.  It cannot be done in an effective, highly scalable fashion
without a total rewrite.


> Basically, our company is at a crossroads where we really need light,
> clean "free threading" as Jesse calls it (e.g. on the iPhone, using
> our python drawing wrapper to do primary drawing while running python
> jobs on another thread doing image decoding and processing).  In our
> current iPhone app, we achieve this by using two python bundles
> (dynamic libs) in the way I described in my initial post.  Sure, thus
> solves our problem, but it's pretty messy, sucks up resources, and has
> been a pain to maintain.

Is the iPhone multicore, or is it an issue of fairness (ie a soft
realtime app)?


> Moving forward, please understand my posts here are also intended to
> give the CPython dev community a glimpse of the issues that may not be
> as visible to you guys (as they are for dev houses like us).  For
> example, it'd be pretty cool if Blizzard went with python instead of
> lua, wouldn't you think?  But some of the issues I've raised here no
> doubt factor in to why end-user dev houses ultimately may have to pass
> up python in favor of another interpreted language.
>
> Bottom line: why give prospective devs any reason to turn down python--
> there's just so many great things about python!

I'd like to see python used more, but fixing these things properly is
not as easy as believed.  Those in the user community see only their
immediate problem (threads don't use multicore).  People like me see
much bigger problems.  We need consensus on the problems, and how to
solve it, and a commitment to invest what's required.