2.6, 3.0, and truly independent intepreters

Andy O'Meara andy55 at gmail.com
Thu Oct 30 12:05:43 EDT 2008


On Oct 28, 6:11 pm, "Martin v. Löwis" <mar... at v.loewis.de> wrote:
> > Because then we're back into the GIL not permitting threads efficient
> > core use on CPU bound scripts running on other threads (when they
> > otherwise could).
>
> Why do you think so? For C code that is carefully written, the GIL
> allows *very well* to write CPU bound scripts running on other threads.
> (please do get back to Jesse's original remark in case you have lost
> the thread :-)
>

I don't follow you there.  If you're referring to multiprocessing, our
concerns are:

- Maturity (am I willing to tell my partners and employees that I'm
betting our future on a brand-new module that imposes significant
restrictions as to how our app operates?)
- Liability (am I ready to invest our resources into lots of new
python module-specific code to find out that a platform that we want
to target isn't supported or has problems?).  Like it not, we're a
company and we have to show sensitivity about new or fringe packages
that make our codebase less agile -- C/C++ continues to win the day in
that department.
- Shared memory -- for the reasons listed in my other posts, IPC or a
shared/mapped memory region doesn't work for our situation (and I
venture to say, for many real world situations otherwise you'd see end-
user/common apps use forking more often than threading).


>
> > It's turns out that this isn't an exotic case
> > at all: there's a *ton* of utility gained by making calls back into
> > the interpreter. The best example is that since code more easily
> > maintained in python than in C, a lot of the module "utility" code is
> > likely to be in python.
>
> You should really reconsider writing performance-critical code in
> Python.

I don't follow you there...  Performance-critical code in Python??
Suppose you're doing pixel-level filters on images or video, or
Patrick needs to apply a DSP to some audio...  Our app's performance
would *tank*, in a MAJOR way (that, and/or background tasks would take
100x+ longer to do their work).

> Regardless of the issue under discussion, a lot of performance
> can be gained by using "flattened" data structures, less pointer,
> less reference counting, less objects, and so on - in the inner loops
> of the computation. You didn't reveal what *specific* computation you
> perform, so it's difficult to give specific advise.

I tried to list some abbreviated examples in other posts, but here's
some elaboration:

- Pixel-level effects and filters, where some filters may use C procs
while others may call back into the interpreter to execute logic --
while some do both, multiple times.
- Image and video analysis/recognition where there's TONS of intricate
data structures and logic.  Those data structures and logic are
easiest to develop and maintain in python, but you'll often want to
call back to C procs which will, in turn, want to access Python (as
well as C-level) data structures.

The common pattern here is where there's a serious mix of C and python
code and data structures, BUT it can all be done with a free-thread
mentality since the finish point is unambiguous and distinct -- where
all the "results" are handed back to the "main" app in a black and
white handoff.  It's *really* important for an app to freely make
calls into its interpreter (or the interpreter's data structures)
without having to perform lock/unlocking because that affords an app a
*lot* of options and design paths.  It's just not practical to be
locking and locking the GIL when you want to operate on python data
structures or call back into python.

You seem to have placed the burden of proof on my shoulders for an app
to deserve the ability to free-thread when using 3rd party packages,
so how about we just agree it's not an unreasonable desire for a
package (such as python) to support it and move on with the
discussion.

>
> Again, if you do heavy-lifting in Python, you should consider to rewrite
> the performance-critical parts in C. You may find that the need for
> multiple CPUs goes even away.

Well, the entire premise we're operating under here is that we're
dealing with "embarrassingly easy" parallelization scenarios, so when
you suggest that the need for multiple CPUs may go away, I'm worried
that you're not keeping the big picture in mind.

>
> > I appreciate your arguments these a PyC concept is a lot of work with
> > some careful design work, but let's not kill the discussion just
> > because of that.
>
> Any discussion in this newsgroup is futile, except when it either
> a) leads to a solution that is already possible, and the OP didn't
> envision, or
> b) is followed up by code contributions from one of the participants.
>
> If neither is likely to result, killing the discussion is the most
> productive thing we can do.
>

Well, most others here seem to have a lot different definition of what
qualifies as a "futile" discussion, so how about you allow the rest of
us continue to discuss these issues and possible solutions.  And, for
the record, I've said multiple times I'm ready to contribute
monetarily, professionally, and personally, so if that doesn't qualify
as the precursor to "code contributions from one of the participants"
then I don't know WHAT does.


Andy




More information about the Python-list mailing list