[Cython] Acquisition counted cdef classes
Dag Sverre Seljebotn
d.s.seljebotn at astro.uio.no
Tue Oct 25 21:01:00 CEST 2011
On 10/25/2011 06:58 PM, mark florisson wrote:
> On 25 October 2011 12:22, Stefan Behnel<stefan_ml at behnel.de> wrote:
>> The problem is not so much the INCREF (which is just an indirect add), it's
>> the DECREF, which contains a conditional jump based on an unknown external
>> value, that may trigger external code. That can kill several C compiler
>> optimisations for the surrounding code. (And that would only get worse by
>> using a dedicated locking mechanism.)
What you could do is a form of psuedo-garbage-collection where, when the
Cython refcount/acquisition count reaches 0, you enqueue a Python DECREF
until you're holding the GIL anyway. If sticking it into the queue is
unlikely(), and it is transparent to the compiler that it doesn't
dispatch into unknown code.
(And regarding Stefan's comment about Erlang: It's all about available
libraries. A language for concurrent computing running on CPython and
able to use all the libraries available for CPython would be awesome. It
doesn't need to be named Cython -- show me an Erlang port to the CPython
platform and I'd perhaps jump ship.)
> Anyway, sorry for the long mail. I agree this is likely not feasible
> to implement, although I would like the functionality to be there.
> Perhaps I'm trying to solve problems which don't really need to be
> solved. Maybe we should just use multiprocessing, or MPI and numpy
> with global arrays and pickling. Maybe memoryviews could help out with
> that as well.
Nice conclusion. I think prange was a very nice 80%-there-solution
(which is also the way we framed it when starting), but the GIL just
creates to many barriers. Real garbage collection is needed, and CPython
just isn't there.
What I'd like to see personally is:
- A convenient utility to allocate an array in shared memory, so that
when you pickle a view of it and send it to another Python process with
multiprocessing and it unpickles, it gets a slice into to the same
shared memory. People already do this but it's just a lot of jumping
through hoops. A good place would probably be in NumPy.
- Decent message passing using ZeroMQ in Cython code without any
Python overhead, for fine-grained communication in Cython code in Python
processes spawned using multiprocessing. I think this requires some
syntax candy in Cython to feel natural enough, but perhaps it can be put
on a form so that it is not ZeroMQ-specific.
More information about the cython-devel