[Cython] Acquisition counted cdef classes

Dag Sverre Seljebotn d.s.seljebotn at astro.uio.no
Tue Oct 25 21:02:19 CEST 2011


On 10/25/2011 09:01 PM, Dag Sverre Seljebotn wrote:
> On 10/25/2011 06:58 PM, mark florisson wrote:
>> On 25 October 2011 12:22, Stefan Behnel<stefan_ml at behnel.de> wrote:
>>> The problem is not so much the INCREF (which is just an indirect
>>> add), it's
>>> the DECREF, which contains a conditional jump based on an unknown
>>> external
>>> value, that may trigger external code. That can kill several C compiler
>>> optimisations for the surrounding code. (And that would only get
>>> worse by
>>> using a dedicated locking mechanism.)
>
> What you could do is a form of psuedo-garbage-collection where, when the
> Cython refcount/acquisition count reaches 0, you enqueue a Python DECREF
> until you're holding the GIL anyway. If sticking it into the queue is
> unlikely(), and it is transparent to the compiler that it doesn't
> dispatch into unknown code.

...then the C compiler optimizations should presumably not be killed.

DS

>
> (And regarding Stefan's comment about Erlang: It's all about available
> libraries. A language for concurrent computing running on CPython and
> able to use all the libraries available for CPython would be awesome. It
> doesn't need to be named Cython -- show me an Erlang port to the CPython
> platform and I'd perhaps jump ship.)
>
>
>> Anyway, sorry for the long mail. I agree this is likely not feasible
>> to implement, although I would like the functionality to be there.
>> Perhaps I'm trying to solve problems which don't really need to be
>> solved. Maybe we should just use multiprocessing, or MPI and numpy
>> with global arrays and pickling. Maybe memoryviews could help out with
>> that as well.
>
> Nice conclusion. I think prange was a very nice 80%-there-solution
> (which is also the way we framed it when starting), but the GIL just
> creates to many barriers. Real garbage collection is needed, and CPython
> just isn't there.
>
> What I'd like to see personally is:
>
> - A convenient utility to allocate an array in shared memory, so that
> when you pickle a view of it and send it to another Python process with
> multiprocessing and it unpickles, it gets a slice into to the same
> shared memory. People already do this but it's just a lot of jumping
> through hoops. A good place would probably be in NumPy.
>
> - Decent message passing using ZeroMQ in Cython code without any Python
> overhead, for fine-grained communication in Cython code in Python
> processes spawned using multiprocessing. I think this requires some
> syntax candy in Cython to feel natural enough, but perhaps it can be put
> on a form so that it is not ZeroMQ-specific.
>
> Dag Sverre



More information about the cython-devel mailing list