2.6, 3.0, and truly independent intepreters

[Andy]

Jesse, Terry, Martin -

First off, thanks again for your time and interest in this matter.
It's definitely encouraging to know that time and real effort is being
put into the matter and I hope my posts on this subject are hopefully
an informative data point for everyone here.

Thanks for that link to Adam Olsen's work, Jesse--I'll definitely look
more closely at it.  As I mentioned in my previous post, end-user devs
like me are programmed to get nervous around new mods but at first
glance there definitely seems to be interesting.  My initial reaction,
as interesting as the project is, goes back to by previous post about
putting all the object safety responsibility on the shoulders of the
API client.  That way, one gets the best of both worlds: free
threading and no unnecessary object locking/blocking (ie. the API
client will manage moving the synchronization req'd to move objects
from one interpreter to another).  I could have it wrong, but it seems
like safethread inserts some thread-safety features but they come at
the cost of performance. I know I keep mentioning it, but I think the
QuickTime API (and its documentation) is a great model for how any API
should approach threading.  Check out their docs to see how they
address it; conceptually speaking, there's not a single line of thread
safety in QuickTime:

http://developer.apple.com/technotes/tn/tn2125.html

In short: multiple thread is tricky; it's the responsibility of the
API client to not do hazardous things.

And for the record: the module multiprocessing is totally great answer
for python-level MP stuff--very nice work, Jesse!

I'd like to post and discuss more, but I'll pick it up tomorrow...
All this stuff is fun and interesting to talk about, but I have to get
to some other things and it unfortunately comes down to cost
analysis.  Sadly, I look at it as I can allocate 2-3 man months (~
$40k) to build our own basic python interpreter implementation that
solves our need for free threading and increased performance (we've
built various internal interpreters over the years so we have good
experience in house, our tools are high performance, and we only use a
pretty small subset of python).  Or, there's the more attractive
approach to work with the python dev community and put that dev
expenditure into a form everyone can benefit from.


Regards,
Andy




On Oct 22, 5:21 pm, "Jesse Noller" <jnol... at gmail.com> wrote:
> On Wed, Oct 22, 2008 at 12:32 PM, Andy <and... at gmail.com> wrote:
> > And, yes, I'm aware of the multiprocessing module added in 2.6, but
> > that stuff isn't lightweight and isn't suitable at all for many
> > environments (including ours).  The bottom line is that if you want to
> > perform independent processing (in python) on different threads, using
> > the machine's multiple cores to the fullest, then you're out of luck
> > under python 2.
>
> So, as the guy-on-the-hook for multiprocessing, I'd like to know what
> you might suggest for it to make it more apt for your - and other
> environments.
>
> Additionally, have you looked at:https://launchpad.net/python-safethreadhttp://code.google.com/p/python-safethread/w/list
> (By Adam olsen)
>
> -jesse