Mailman 3 April 2013 - cython-devel

regarding ticket #810 (segfault in Sage)
by Stefan Behnel April 26, 2013

April 26, 2013

Hi, looking at this ticket: http://trac.cython.org/cython_trac/ticket/810 and at the underlying Sage ticket: http://trac.sagemath.org/sage_trac/ticket/14452 ISTM that this crash is not actually related to cdef kwargs but simply to the fact that the C code is now calling directly through the C layer and no longer through Python. The major difference being that Python puts arguments into a tuple that is kept alive for the whole timespan of the call, whereas C just passes in the reference and forgets about it. Since the method in question replaces references internally, it's quite possible that it somehow deletes the last reference to the object that was passed in. I wouldn't really know why, as we specifically guard against these things since somewhere around 0.17 (and comment #7 in the Sage ticket shows an example of that), but maybe this is just a corner case that we failed to handle so far, or maybe the Sage code is doing something 'unexpected' here. Stefan

1 1

Cython code producing different, incorrect results under Python 2.7 (not 3.x) under Cython 0.19
by Josh Warner April 25, 2013

April 25, 2013

Hi Cython devs, Over in scikit-image we have traced an odd problem with a particular Cython file to the 0.19 update of Cython. From at least Cython 0.15.1 (probably earlier) through 0.18, `_mcp.pyx` in `skimage.graph` compiled and executed correctly, passing all package tests on both Python 2.7 and Python 3. After 0.19 was released and the Travis builds began using it, we began getting 100% repeatable errors from the previously clean master branch (example of an otherwise clean Python 2.7<https://travis-ci.org/scikit-image/scikit-image/jobs/6545505>Travis build; the Python 3 build passed<https://travis-ci.org/scikit-image/scikit-image/builds/6545504>all tests). All of these errors/failures trace back to this Cython file. Oddly, the errors only happen on Python 2.7; our Python 3 Travis build passes. We are discussing this issue in scikit-image Github Issue #534<https://github.com/scikit-image/scikit-image/issues/534>; feel free to join the discussion there. The .pyx Cython file is located here<https://github.com/scikit-image/scikit-image/blob/master/skimage/graph/_mcp…>and it has an associated .pxd file here<https://github.com/scikit-image/scikit-image/blob/master/skimage/graph/_mcp…>. It should be noted the file compiles and executes without errors, but its output is now incorrect in Python 2.x. In case the compiled results might be relevant, for your diffing pleasure here is the compiled .c file from Cython 0.18<https://gist.github.com/JDWarner/af4f8ea85dce356ce95c>which passes all tests on both Python 2.7 and Python 3.x, while here is the compiled .c file from Cython 0.19<https://gist.github.com/JDWarner/56d15b7a7527b8d4314e>which produces different, incorrect results in Python 2.7. In the short term we are temporarily forcing Travis to use the 0.18 release of Cython, but that isn't a viable long term solution. It's possible the error is on our end, but seeing as it worked with prior Cython releases we'd appreciate you taking a look. Thanks!

3 6

Unit testing Cython extensions
by Torsten Landschoff April 23, 2013

April 23, 2013

Hi again, I am just wondering how to do unit testing on extensions built using Cython with py.test. My problem: The extension module I am working on is installed in our global python environment (inside the build slaves as well as on local machines) already. Before installing a new version of the extension module, I'd like to run the unit tests. However, that way the original module is tested. I am trying to illustrate. Doing TDD I create a new module which will fail the test: torsten@sharokan:~/foo$ cat foo.pyx def it_works(): return False torsten@sharokan:~/foo$ cat tests/test_foo.py import pprint, foo def test_foo(): pprint.pprint(foo.__file__) assert foo.it_works() torsten@sharokan:~/foo$ py.test -v tests ============================================================ test session starts ============================================================ platform linux2 -- Python 2.7.3 -- pytest-2.3.4 -- /opt/dynasdk/loco2-precise/bin/python plugins: cov, capturelog collected 0 items / 1 errors ================================================================== ERRORS =================================================================== ____________________________________________________ ERROR collecting tests/test_foo.py _____________________________________________________ tests/test_foo.py:1: in <module> > import sys, pprint, foo E ImportError: No module named foo ========================================================== 1 error in 0.01 seconds ========================================================== Sure, module foo does not exists. I could use pyximport, but I want to check that the extension itself is correctly built. Easily done: torsten@sharokan:~/foo$ python setup.py build_ext -i running build_ext cythoning foo.pyx to foo.c building 'foo' extension creating build creating build/temp.linux-x86_64-2.7 gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/dynasdk/loco2-precise/include/python2.7 -c foo.c -o build/temp.linux-x86_64-2.7/foo.o gcc -pthread -shared build/temp.linux-x86_64-2.7/foo.o -L/opt/dynasdk/loco2-precise/lib -lpython2.7 -o /home/torsten/foo/foo.so But alas, py.test still will not find foo.so, which is now installed into the current directory (due to using the "-i" flag to build_ext). Workaround: Overwrite PYTHONPATH: torsten@sharokan:~/foo$ PYTHONPATH=`pwd` py.test ============================================================ test session starts ============================================================ platform linux2 -- Python 2.7.3 -- pytest-2.3.4 plugins: cov, capturelog collected 1 items tests/test_foo.py F ================================================================= FAILURES ================================================================== _________________________________________________________________ test_foo __________________________________________________________________ def test_foo(): pprint.pprint(foo.__file__) > assert foo.it_works() E assert <built-in function it_works>() E + where <built-in function it_works> = foo.it_works tests/test_foo.py:5: AssertionError -------------------------------------------------------------- Captured stdout -------------------------------------------------------------- '/home/torsten/foo/foo.so' ========================================================= 1 failed in 0.02 seconds ========================================================== Now it really loads our extension module and the failure is actually genuine. Let's assume we have the old version installed in our current Python environment: torsten@sharokan:~/foo$ python setup.py install [...] creating /usr/opt/dynasdk/loco2-precise/lib/python2.7/site-packages/foo-0.0-py2.7-linux-x86_64.egg Extracting foo-0.0-py2.7-linux-x86_64.egg to /usr/opt/dynasdk/loco2-precise/lib/python2.7/site-packages Adding foo 0.0 to easy-install.pth file Installed /usr/opt/dynasdk/loco2-precise/lib/python2.7/site-packages/foo-0.0-py2.7-linux-x86_64.egg [...] Great. Now let's implement the feature that our test checks: torsten@sharokan:~/foo$ vim foo.pyx torsten@sharokan:~/foo$ cat foo.pyx def it_works(): return True The unit tests should pass now, after we rebuilt the extension: torsten@sharokan:~/foo$ python setup.py build_ext -i running build_ext cythoning foo.pyx to foo.c building 'foo' extension gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/dynasdk/loco2-precise/include/python2.7 -c foo.c -o build/temp.linux-x86_64-2.7/foo.o gcc -pthread -shared build/temp.linux-x86_64-2.7/foo.o -L/opt/dynasdk/loco2-precise/lib -lpython2.7 -o /home/torsten/foo/foo.so torsten@sharokan:~/foo$ PYTHONPATH=`pwd` py.test ============================================================ test session starts ============================================================ platform linux2 -- Python 2.7.3 -- pytest-2.3.4 plugins: cov, capturelog collected 1 items tests/test_foo.py F ================================================================= FAILURES ================================================================== _________________________________________________________________ test_foo __________________________________________________________________ def test_foo(): pprint.pprint(foo.__file__) > assert foo.it_works() E assert <built-in function it_works>() E + where <built-in function it_works> = foo.it_works tests/test_foo.py:5: AssertionError -------------------------------------------------------------- Captured stdout -------------------------------------------------------------- '/opt/dynasdk/loco2-precise/lib/python2.7/site-packages/foo-0.0-py2.7-linux-x86_64.egg/foo.so' ========================================================= 1 failed in 0.02 seconds ========================================================== Unfortunately, it doesn't. The unit tests actually uses the installed version of the library, which we only want to replace after our tests pass. Let's try it another way: By creating a virtualenv just for our tests, we should be fine: torsten@sharokan:~/foo$ virtualenv --system-site-packages fooenv New python executable in fooenv/bin/python Please make sure you remove any previous custom paths from your /home/torsten/.pydistutils.cfg file. Installing setuptools............done. Installing pip...............done. torsten@sharokan:~/foo$ . fooenv/bin/activate (fooenv)torsten@sharokan:~/foo$ pip install --upgrade . Unpacking /home/torsten/foo Running setup.py egg_info for package from file:///home/torsten/foo Downloading/unpacking Cython from http://pypi.python.org/packages/source/C/Cython/Cython-0.19.tar.gz#md5=7698… (from foo==0.0) Downloading Cython-0.19.tar.gz (1.4MB): 270kB downloaded ... Good start, but I don't want to recreate the whole thing inside the virtualenv fooenv. This would pull Cython, numpy, scipy, paramiko and more. Any hint how to ensure that we are testing the local version of that extension instead of the installed one? Thanks, Torsten -- DYNAmore Gesellschaft fuer Ingenieurdienstleistungen mbH Torsten Landschoff Office Dresden Tel: +49-(0)351-4519587 Fax: +49-(0)351-4519561 mailto:torsten.landschoff@dynamore.de http://www.dynamore.de DYNAmore Gesellschaft für FEM Ingenieurdienstleistungen mbH Registration court: Stuttgart, HRB 733694 Managing director: Prof. Dr. Karl Schweizerhof, Dipl.-Math. Ulrich Franz

2 1

libc.string msvc incompatibility
by Dave Hirschfeld April 23, 2013

April 23, 2013

When trying to use strcasecmp from libc.string I get an error compiling with msvc: error C3861: 'strcasecmp': identifier not found It seems MS have decided to call it by another name - _stricmp ...amongst others: http://botsikas.blogspot.co.uk/2011/12/strcasecmp-identifier-not-found- when.html I'm wondering if it's possible for cython to include the mentioned ifdef defines by default or otherwise solve the problem so that code is portable between compilers on Windows? I understand if it's not a high priority issue. The workaround is for me to make sure all our code compiles with mingw which was on the radar anyway. Thanks, Dave

2 1

gsoc-kurt-regraft branch and ticket #469
by Stefan Behnel April 19, 2013

April 19, 2013

Hi, since the current code base can explicitly express dependencies between utility code sections, do we still need this branch? Or can we close it together with the ticket? ISTM that explicit dependencies are a more general feature than just honouring type hierarchies. Stefan

1 0

Cython 0.19 released
by Stefan Behnel April 19, 2013

April 19, 2013

Hi all, Cython 0.19 has cleared the dockyard. This is a feature release that adds some major usability features especially for code that needs to run in both Py2 and Py3, as well as better Python compatibility and optimisations. You can get it from here: http://cython.org/release/Cython-0.19.tar.gz http://cython.org/release/Cython-0.19.zip Release notes: https://github.com/cython/cython/blob/29bf3493fdc80cb3261a9dfb9f73e3ccd46fe… Documentation: http://docs.cython.org/ The major new features include: * New directives ``c_string_type`` and ``c_string_encoding`` to automatically convert between C strings and the different Python string types. * Keyword arguments are supported for cdef functions, including external C/C++ functions. * New freelist decorator for extension types. * Fast extension type instantiation using the ``Type.__new__(Type)`` idiom has gained support for passing arguments. * Slicing was optimised for several builtin types and otherwise conforms better with Python semantics for user defined types. * The mapping of dict view/item methods in Python 3 was improved. Have fun, Stefan

1 0

Cython 0.19 release candidate
by Stefan Behnel April 17, 2013

April 17, 2013

Hi, I uploaded a release candidate for Cython 0.19. Note that there were changes to the buffer validation code for memory views since the last beta, so please give this release another try with your code, especially if you use memory views in it. http://cython.org/release/Cython-0.19rc1.tar.gz http://cython.org/release/Cython-0.19rc1.zip Cython 0.19 features many enhancements and changes regarding performance, usability and Python compatibility, including some very long standing bugs and issues. The changelog is on github: https://github.com/cython/cython/blob/0ee15b1829125b6844ac13f437c2d9bd61a4c… The latest documentation is here: https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/ Have fun, Stefan

1 0

[PATCH] Refcount error when transposing memoryview attribute of an extension class
by Matěj Laitl April 17, 2013

April 17, 2013

Hi cython-devel and Mark, I was getting > Fatal Python error: Acquisition count is 0 (line XYZ) when I was doing > cdef class MemViewContainer: > cdef double[:, :] A > > cdef a_method(self): > self.A = np.eye(2) > some_function(self.A.T) > some_function(self.A.T) I have found out that it is caused by the self.A.T expression - Cython emits __PYX_XDEC_MEMVIEW() after the call, but no __PYX_INC_MEMVIEW() before the call. This doesn't happen if the memoryview is a function-local variable. Proper test case is in the pull request [1] along with an ad-hoc patch that fixes the problem here, but needs review whether it is actually correct. I'd be very grateful if a fix for this problem could get it into Cython 0.19. Cython version: 0.19b1 3a6b9856187d7e490e08 [1] https://github.com/cython/cython/pull/201 Regards, Matěj

4 6

Re: [Cython] [cython-users] Possible enumerate automatic conversion
by Stefan Behnel April 17, 2013

April 17, 2013

Daniele Nicolodi, 16.04.2013 15:09: > would it be desirable to have an enumerate() automatic conversion > similar to the range() conversion when the enumerated iterable is a C > array or a numpy array? > > http://docs.cython.org/src/userguide/pyrex_differences.html#automatic-range… enumerate() is optimised since something like 0.13 or so, maybe longer. It certainly works for char* and maybe also other C arrays. > I imagine that detecting the opportunity for the optimization may be a > bit tricky, but it should not be much different than a combination of > what is done for iteration over a C array and for the range() function. Yes, the tricky bit is to get the iterable through the type analysis without coercion errors. This couldn't easily be done at the time but became possible with my recent analyse_types() phase changes. Essentially, for-loop optimisations can now be moved into the type analysis phase and take the type of the iterable into account without running into accidental coercion errors. I'm not sure yet if this is the right way to do it - maybe we still want an explicit transformation instead. In any case, the right place to discuss this is the cython-devel mailing list, not the users mailing list. Stefan

1 0

Cython 0.19 beta 2 released
by Stefan Behnel April 17, 2013

April 17, 2013

Hi, I uploaded a second (and hopefully last) beta release for Cython 0.19. Please give it a try with your code. http://cython.org/release/Cython-0.19b2.tar.gz http://cython.org/release/Cython-0.19b2.zip Cython 0.19 features many enhancements and changes regarding performance, usability and Python compatibility, including some very long standing bugs and issues. The changelog is on github: https://github.com/cython/cython/blob/58131b68dc033fc7ca269d875a2aab2b4e964… The latest documentation is here: https://sage.math.washington.edu:8091/hudson/job/cython-docs/doclinks/1/ Have fun, Stefan

2 1