2.6, 3.0, and truly independent intepreters

[Patrick Stinson]

We are in the same position as Andy here.

I think that something that would help people like us produce
something in code form is a collection of information outlining the
problem and suggested solutions, appropriate parts of the CPython's
current threading API, and pros and cons of the many various proposed
solutions to the different levels of the problem. The most valuable
information I've found is contained in the many (lengthy!) discussions
like this one, a few related PEP's, and the CPython docs, but has
anyone condensed the state of the problem into a wiki or something
similar? Maybe we should start one?

For example, Guido's post here
http://www.artima.com/weblogs/viewpost.jsp?thread=214235describes some
possible solutions to the problem, like interpreter-specific locks, or
fine-grained object locks, and he also mentions the primary
requirement of not harming from the performance of single-threaded
apps. As I understand it, that requirement does not rule out new build
configurations that provide some level of concurrency, as long as you
can still compile python so as to perform as well on single-threaded
apps.

To add to the heap of use cases, the most important thing to us is to
simple have the python language and the sip/PyQt modules available to
us. All we wanted to do was embed the interpreter and language core as
a local scripting engine, so had we patched python to provide
concurrent execution, we wouldn't have cared about all of the other
unsuppported extension modules since our scripts are quite
application-specific.

It seems to me that the very simplest move would be to remove global
static data so the app could provide all thread-related data, which
Andy suggests through references to the QuickTime API. This would
suggest compiling python without thread support so as to leave it up
to the application.

Anyway, I'm having fun reading all of these papers and news postings,
but it's true that code talks, and it could be a little easier if the
state of the problems was condensed. This could be an intense and fun
project, but frankly it's a little tough to keep it all in my head. Is
there a wiki or something out there or should we start one, or do I
just need to read more code?

On Fri, Oct 24, 2008 at 6:40 AM, Andy O'Meara <andy55 at gmail.com> wrote:
> On Oct 24, 2:12 am, greg <g... at cosc.canterbury.ac.nz> wrote:
>> Andy wrote:
>> > 1) Independent interpreters (this is the easier one--and solved, in
>> > principle anyway, by PEP 3121, by Martin v. Löwis
>>
>> Something like that is necessary for independent interpreters,
>> but not sufficient. There are also all the built-in constants
>> and type objects to consider. Most of these are statically
>> allocated at the moment.
>>
>
> Agreed--I  was just trying to speak generally.  Or, put another way,
> there's no hope for independent interpreters without the likes of PEP
> 3121.  Also, as Martin pointed out, there's the issue of module
> cleanup some guys here may underestimate (and I'm glad Martin pointed
> out the importance of it).  Without the module cleanup, every time a
> dynamic library using python loads and unloads you've got leaks.  This
> issue is a real problem for us since our software is loaded and
> unloaded many many times in a host app (iTunes, WMP, etc).  I hadn't
> raised it here yet (and I don't want to turn the discussion to this),
> but lack of multiple load and unload support has been another painful
> issue that we didn't expect to encounter when we went with python.
>
>
>> > 2) Barriers to "free threading".  As Jesse describes, this is simply
>> > just the GIL being in place, but of course it's there for a reason.
>> > It's there because (1) doesn't hold and there was never any specs/
>> > guidance put forward about what should and shouldn't be done in multi-
>> > threaded apps
>>
>> No, it's there because it's necessary for acceptable performance
>> when multiple threads are running in one interpreter. Independent
>> interpreters wouldn't mean the absence of a GIL; it would only
>> mean each interpreter having its own GIL.
>>
>
> I see what you're saying, but let's note that what you're talking
> about at this point is an interpreter containing protection from the
> client level violating (supposed) direction put forth in python
> multithreaded guidelines.  Glenn Linderman's post really gets at
> what's at hand here.  It's really important to consider that it's not
> a given that python (or any framework) has to be designed against
> hazardous use.  Again, I refer you to the diagrams and guidelines in
> the QuickTime API:
>
> http://developer.apple.com/technotes/tn/tn2125.html
>
> They tell you point-blank what you can and can't do, and it's that's
> simple.  Their engineers can then simply create the implementation
> around those specs and not weigh any of the implementation down with
> sync mechanisms.  I'm in the camp that simplicity and convention wins
> the day when it comes to an API.  It's safe to say that software
> engineers expect and assume that a thread that doesn't have contact
> with other threads (except for explicit, controlled message/object
> passing) will run unhindered and safely, so I raise an eyebrow at the
> GIL (or any internal "helper" sync stuff) holding up an thread's
> performance when the app is designed to not need lower-level global
> locks.
>
> Anyway, let's talk about solutions.  My company looking to support
> python dev community endeavor that allows the following:
>
> - an app makes N worker threads (using the OS)
>
> - each worker thread makes its own interpreter, pops scripts off a
> work queue, and manages exporting (and then importing) result data to
> other parts of the app.  Generally, we're talking about CPU-bound work
> here.
>
> - each interpreter has the essentials (e.g. math support, string
> support, re support, and so on -- I realize this is open-ended, but
> work with me here).
>
> Let's guesstimate about what kind of work we're talking about here and
> if this is even in the realm of possibility.  If we find that it *is*
> possible, let's figure out what level of work we're talking about.
> >From there, I can get serious about writing up a PEP/spec, paid
> support, and so on.
>
> Regards,
> Andy
>
>
>
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>