[C++-sig] RAII for the GIL in Boost.Python?

[Niall Douglas]

On 25 Dec 2012 at 12:14, John Zwinck wrote:

> The code I published is agnostic about iterators and iteration--the user 
> must explicitly decide when the GIL is to be released and reacquired. 
> I'm sure there could be fancier ways of doing it, e.g. with tags given 
> to BPL's class_::def() for long-running methods.  But I didn't worry 
> about that level of complexity--I think the simple solution offers 
> benefits today, and can be expanded upon later.
> 
> Having multiple Python interpreters in one process seems fraught anyway, 
> so I haven't really considered whether my code can (or should) support 
> that.  I hope you'll agree it is not the common case.

Solving multiple interpreters is absolutely the same problem as 
solving GIL management correctly. You get multiple interpreters "for 
free".

I also didn't mention the problem of multiple copies of BPL, which is 
a related issue. Right now, if python loads two BPL based modules, 
two separate copies of the BPL runtime are loaded. The problem is 
that there are now two BPL registries, so you start encountering 
"Unknown type" exceptions when the wrong registry gets looked up.

The traditional solution is to hack the flags passed to dlopen() by 
python to use RTLD_GLOBAL. Now you get one single type registry, and 
things start to work again. But now you have the problem of symbol 
collision due to the ODR rule, so if extension A defines type Foo and 
extension B defines a different type Foo, they collide. Basically, 
you get a runtime warning when you try to load the second extension, 
and it segfaults or memory corrupts during usage.

What BPL needs is to do execution stack aware type registry lookups, 
so you need to examine from which extension the lookup is coming from 
and search the type registry appropriately. Thankfully, 
metaprogramming makes this uber easy (just slip in a pointer to any 
local function, and have a O(log N) routine map it to its containing 
module, then cache that so lookup henceforth is instant).

As I mentioned earlier, BPL basically needs to be refactored and 
rewritten pretty much from scratch. The wider solution I'm currently 
prototyping - if it gets greenlit - solves all these problems, and a 
ton load more I haven't mentioned here (e.g. what happens if the two 
BPLs are different versions, and are slightly binary incompatible?). 
Chances are though it will be rejected sadly. And even if greenlit, 
we're talking 2014 at the earliest.

So, for now, you may be right that submitting a toy solution now in 
the hope its limitations spur people to get it fixed properly may be 
the better approach. Certainly our collective hesitation so far 
hasn't achieved much. So sure, press on.

Niall

-- 
Any opinions or advice expressed here do NOT reflect those
of my employer Research In Motion Inc.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/



-------------- next part --------------
A non-text attachment was scrubbed...
Name: SMime.p7s
Type: application/x-pkcs7-signature
Size: 6061 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/cplusplus-sig/attachments/20121225/59404bee/attachment.bin>