[Cython] test crash in Py3.2 (NumPy/memoryview/refcounting related?)

mark florisson markflorisson88 at gmail.com
Sun Apr 22 16:41:07 CEST 2012

On 22 April 2012 15:31, mark florisson <markflorisson88 at gmail.com> wrote:
> On 22 April 2012 13:34, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> mark florisson, 22.04.2012 13:54:
>>> On 22 April 2012 11:57, Stefan Behnel wrote:
>>>> I keep seeing test crashes in Py3.2 debug builds on Jenkins with the latest
>>>> master, referring to ref-counting problems. Here, for example:
>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND=c,PYVERSION=py32-ext/328/console
>>>> and likewise in the series of builds starting at #312.
>>>> I'm not sure what changed in that corner, it looks like one of those
>>>> problems that was there for a while and suddenly starts showing when you
>>>> touch that innocent looking brick at the other end of the house that was
>>>> holding the balance, until now. In this case, that brick was this commit:
>>>> https://github.com/cython/cython/commit/c8f61a668d0ce8af1020f520253e1b5f623cf349
>>>> I reverted it here and that fixed the problem at least temporarily:
>>>> https://github.com/cython/cython/commit/5aac8caf1ba21933cc85a51af3319c78fc08d675
>>>> but it seems to be back now (after my refnanny optimisations). Before
>>>> reverting my changes, I was able to reproduce it somewhat reliably on
>>>> sage.math by running these three tests together (non-forking):
>>>>        memslice numpy_bufacc numpy_memoryview
>>>> None of the tests shows the problem when run by itself. I can't tell if
>>>> it's also in the latest py3k because I don't have a NumPy lying around that
>>>> works there. So 3.2 is basically the latest Python version this can be
>>>> tested with, and it doesn't occur in the 3.1 tests. The tests use NumPy
>>>> 1.6.1, i.e. the latest official release.
>>>> Mark, Dag, could any of you take a look to see if it appears in any way
>>>> more obvious to you than to me?
>>> Hm, I think you can try to disable the numpy_memoryview test and just
>>> continue development as normal. numpy_memoryview has a testcase
>>> function which inserts the test into __test__, so you could just
>>> comment out that line.
>> ... as long as we remember to put it back in ;)
>>> The test seems to fail before it runs though? Is it possible to obtain
>>> a backtrace?
>> When I reproduced it in the Jenking workspace, it crashed while trying to
>> clean up objects to free memory, specifically in the deallocation visitor
>> function of one of the memory views of numpy_memoryview (IIRC). That didn't
>> really tell me where that object was created or what happened to it to get
>> it to crash. Needs some more investigation.
>> Stefan
>> _______________________________________________
>> cython-devel mailing list
>> cython-devel at python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
> Hm, I can't reproduce the issue, but that assertion triggers only for
> update_refs and subtract_refs (when the gc is trying to determine
> which object may be potentially unreachable). I think the problem here
> is that a single memoryview object is traversed multiple times through
> different traverse functions, and that the refcount doesn't match the
> number of traverses. Indeed, the refcount is only one, as the actual
> count is the acquisition count. So we shouldn't traverse the
> memoryview objects in memoryview slices, i.e. not
> _memoryviewslice.from_slice.memview. I'll come up with a commit
> shortly, would you be willing to test it?

A fix is here: https://github.com/markflorisson88/cython/commit/cd32184f3f782b6d7275cf430694b59801ce642a

Lets see what jenkins has to say :)

BTW, tp_clear calls Py_CLEAR on Py_buffer.obj, shouldn't it call
releasebuffer instead?

More information about the cython-devel mailing list