Mailman 3 February 2012 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

0.16 release
by mark florisson March 31, 2012

March 31, 2012

Hey, It's been almost three months since we talked about a 0.16 release, I think it's quite ready. It would already be a big release, it would be good to see how people like it, and to catch any issues etc before we pile on more features. Mark

7 46

PEP 380 approved: syntax for generator delegation
by Stefan Behnel March 6, 2012

March 6, 2012

Hi, PEP 380 ("yield from") recently went through the approval gate. http://www.python.org/dev/peps/pep-0380/ The implementation in CPython isn't rounded up yet and lots of tests are missing from the test suite. But it's agreed that it'll make it into 3.3, and we should try to support it as well. It's a really nice feature. Stefan

2 3

Cython 0.16 Beta 0
by mark florisson March 5, 2012

March 5, 2012

The Cython team is pleased to announce the first beta release for the upcoming release of Cython 0.16. A tarball can be grabbed from here: http://cython.org/release/Cython-0.16.beta0.tar.gz. This release comes with several great new features such as better function introspection, super without arguments, fused types, memoryviews and more, as well as many bug fixes. For a complete overview see http://wiki.cython.org/ReleaseNotes-0.16. This beta release means a feature freeze for 0.16. Have a try everyone and remember, bonus points to anyone who can break things! Mark

2 2

Cython 0.16 and ndarray fields deprecation
by Dag Sverre Seljebotn March 2, 2012

March 2, 2012

I'm wondering what the best course of action for deprecating the shape field in numpy.pxd is. The thing is, currently "shape" really gets in the way. In most situations it is OK with slow access to shape through the Python layer, and "arr.shape[0]" is often just fine, but currently one is in a situation where one must either write "(<object>arr).shape[0])" or "np.PyArray_DIMS(arr)[0]", or be faced with code that isn't forward-compatible with NumPy. It would really be good to do the transition as fast as possible, so that all Cython code eventually becomes ready for upcoming NumPy releases. The simplest change to make would be to simply remove all the ndarray fields from numpy.pxd, and inform about the alternatives in the release notes. That could be done in time for 0.16. The alternatives of sounding sane deprecation warnings in just the right places takes more work. I can't work on that myself until PyCon sprints... perhaps put out 0.16.1 with just this change after PyCon sprints though... Dag

5 11

Re: [Cython] [cython-users] cleaning up in a module
by Stefan Behnel Feb. 29, 2012

Feb. 29, 2012

Stefan Behnel, 29.02.2012 14:42: > Stefan Behnel, 29.02.2012 14:34: >> Henry Gomersall, 29.02.2012 14:30: >>> What's the preferred way to clean up a C library when a module is >>> deleted? >> >> PEP 3121: >> >> http://www.python.org/dev/peps/pep-3121/ >> >> However, given that CPython doesn't currently support unloading extension >> modules, your question is rather hypothetical. > > Oh, what you *can* do, however, is register an "atexit" function that does > the cleanup when the interpreter terminates. Speaking of which, what about allowing users to implement a function cdef void __dealloc__() at the module level, which would then be called by the module cleanup function (if generated) before running the rest of the cleanup code? That would allow for an easier transition of user code towards a better support of PEP 3121, specifically m_clear(). http://trac.cython.org/cython_trac/ticket/218 Stefan

1 0

Re: [Cython] [cython-users] What's up with PyEval_InitThreads() in python 2.7?
by Stefan Behnel Feb. 28, 2012

Feb. 28, 2012

I'm going to reimplement this, but not for 0.16 anymore, I'd say. -------- Original-Message -------- Betreff: Re: [cython-users] What's up with PyEval_InitThreads() in python 2.7? Mike Cui, 28.02.2012 10:18: >> Thanks for the test code, you hadn't mentioned that you use a "with gil" >> block. Could you try the latest github version of Cython? >> > > Ahh, much better! > > #if CYTHON_REFNANNY > #ifdef WITH_THREAD > __pyx_gilstate_save = PyGILState_Ensure(); > #endif > #endif /* CYTHON_REFNANNY */ > __Pyx_RefNannySetupContext("callback"); > #if CYTHON_REFNANNY > #ifdef WITH_THREAD > PyGILState_Release(__pyx_gilstate_save); > #endif > #endif /* CYTHON_REFNANNY */ Hmm, thanks for posting this - it can be further improved. There's no reason for code bloat here, it should all just go into the __Pyx_RefNannySetupContext() macro. Stefan

2 20

__cinit__ swallowing exceptions
by Lisandro Dalcin Feb. 28, 2012

Feb. 28, 2012

This is something I really have no idea about how to fix, so I'll ask any of you to do it. How to reproduce. The quick example below should fail in the second to last line in test_cinit.py, but it succeeds: $ cat cinit.pyx cdef class A: def __cinit__(self, A a=None): pass $ cat test_cinit.py import pyximport; pyximport.install() from cinit import A a = A(123) print (a) $ python test_cinit.py /home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13: warning: ‘__pyx_clear_code_object_cache’ defined but not used [-Wunused-function] <cinit.A object at 0x7fd28f1b7920> I think the issue is bad code generation. Please try to follow the flow below, when ArgTypeTest fails, then "goto __pyx_L1_error" but __pyx_r is never initialized to -1, so the function (accidentally) returns 0 indicating success. BTW, GCC warns about __pyx_r used before initialization is some other code of mine (more complex, involving inheritance, and the C compiler inlining code), but not for this simple example. static int __pyx_pw_5cinit_1A_1__cinit__(PyObject *__pyx_v_self, PyObject *__pyx_args, PyObject *__pyx_kwds) { ... int __pyx_r; ... { .... <arg unpacking code>.... } goto __pyx_L4_argument_unpacking_done; ... __pyx_L4_argument_unpacking_done:; if (unlikely(!__Pyx_ArgTypeTest(((PyObject *)__pyx_v_a), __pyx_ptype_5cinit_A, 1, "a", 0))) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 2; __pyx_clineno = __LINE__; goto __pyx_L1_error;} ... __pyx_L1_error:; __pyx_L0:; __Pyx_RefNannyFinishContext(); return __pyx_r; } BTW, valgrind is able to catch the issue: $ touch cinit.pyx $ CFLAGS=-O0 valgrind python test_cinit.py ==6735== Memcheck, a memory error detector ==6735== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al. ==6735== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info ==6735== Command: python test_cinit.py ==6735== /home/dalcinl/.pyxbld/temp.linux-x86_64-2.7/pyrex/cinit.c:1429:13: warning: ‘__pyx_clear_code_object_cache’ defined but not used [-Wunused-function] ==6735== Conditional jump or move depends on uninitialised value(s) ==6735== at 0x12DA4635: __pyx_tp_new_5cinit_A (cinit.c:548) ==6735== by 0x3A87C9DCF2: ??? (in /usr/lib64/libpython2.7.so.1.0) ==6735== by 0x3A87C49192: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0) ==6735== by 0x3A87CDE794: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==6735== by 0x3A87CE15A4: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==6735== by 0x3A87CE16D1: PyEval_EvalCode (in /usr/lib64/libpython2.7.so.1.0) ==6735== by 0x3A87CFB9EB: ??? (in /usr/lib64/libpython2.7.so.1.0) ==6735== by 0x3A87CFC7EF: PyRun_FileExFlags (in /usr/lib64/libpython2.7.so.1.0) ==6735== by 0x3A87CFD26E: PyRun_SimpleFileExFlags (in /usr/lib64/libpython2.7.so.1.0) ==6735== by 0x3A87D0E744: Py_Main (in /usr/lib64/libpython2.7.so.1.0) ==6735== by 0x56F969C: (below main) (in /lib64/libc-2.14.90.so) ==6735== <cinit.A object at 0xec47430> ==6735== ==6735== HEAP SUMMARY: ==6735== in use at exit: 8,870,441 bytes in 53,569 blocks ==6735== total heap usage: 391,334 allocs, 337,765 frees, 94,515,287 bytes allocated ==6735== ==6735== LEAK SUMMARY: ==6735== definitely lost: 0 bytes in 0 blocks ==6735== indirectly lost: 0 bytes in 0 blocks ==6735== possibly lost: 2,319,018 bytes in 15,424 blocks ==6735== still reachable: 6,551,423 bytes in 38,145 blocks ==6735== suppressed: 0 bytes in 0 blocks ==6735== Rerun with --leak-check=full to see details of leaked memory ==6735== ==6735== For counts of detected and suppressed errors, rerun with: -v ==6735== Use --track-origins=yes to see where uninitialised values come from ==6735== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 2 from 2) -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

3 5

memoryview test warnings
by Stefan Behnel Feb. 25, 2012

Feb. 25, 2012

Hi, I get these warnings in the tests: """ compiling (cpp) and running memoryview ... memoryview.cpp: In function ‘PyObject* __pyx_memoryview_setitem_slice_assign_scalar(__pyx_memoryview_obj*, PyObject*, PyObject*)’: memoryview.cpp:19703: warning: comparison between signed and unsigned integer expressions memoryview.cpp: At global scope: memoryview.cpp:10210: warning: ‘__Pyx_memviewslice __pyx_f_10memoryview_func()’ defined but not used compiling (cpp) and running memslice ... memslice.cpp: In function ‘PyObject* __pyx_memoryview_setitem_slice_assign_scalar(__pyx_memoryview_obj*, PyObject*, PyObject*)’: memslice.cpp:39266: warning: comparison between signed and unsigned integer expressions """ Would be nice if they could be fixed for the release. Also, the memslice test writes output to the test log: """ acquired default acquired Global_A """ Normally, all test output should get collected by the doctest runner. Stefan

2 1

cython tests and py3k hash randomization
by Vitja Makarov Feb. 25, 2012

Feb. 25, 2012

Recent py3k version has new feature "hash randomization" it solves some security issues. But has some drawbacks, for instance, dict.items() order is now unknown. So it breaks randomly some doctests that rely on exact order of dict items. vitja@mchome:~/python$ ./py3k/bin/python -c "print({'a':1, 'b':2})" {'b': 2, 'a': 1} vitja@mchome:~/python$ ./py3k/bin/python -c "print({'a':1, 'b':2})" {'a': 1, 'b': 2} As a workaround we can set PYTHONHASHSEED environment variable to zero for all cyhon-*-tests targets This also affects generated code it internally depends on order of items returned by dict.items() -- vitja.

3 5