Mailman 3 January 2012 - cython-devel

0.16 release
by mark florisson March 31, 2012

March 31, 2012

Hey, It's been almost three months since we talked about a 0.16 release, I think it's quite ready. It would already be a big release, it would be good to see how people like it, and to catch any issues etc before we pile on more features. Mark

7 46

PEP 380 approved: syntax for generator delegation
by Stefan Behnel March 6, 2012

March 6, 2012

Hi, PEP 380 ("yield from") recently went through the approval gate. http://www.python.org/dev/peps/pep-0380/ The implementation in CPython isn't rounded up yet and lots of tests are missing from the test suite. But it's agreed that it'll make it into 3.3, and we should try to support it as well. It's a really nice feature. Stefan

2 3

Bug in Cython producing incorrect C code
by Konrad Hinsen Feb. 12, 2012

Feb. 12, 2012

Compiling the attached Cython file produced the attached C file which has errors in lines 532-534: __pyx_v_self->xx = None; __pyx_v_self->yy = None; __pyx_v_self->zz = None; There is no C symbol "None", so this doesn't compile. I first noticed the bug in Cython 0.15, but it's still in the latest revision from Github. Konrad.

6 24

Re: [Cython] [cython-users] Re: How to find out where an AttributeError is ignored
by mark florisson Feb. 2, 2012

Feb. 2, 2012

On 27 January 2012 16:22, mark florisson <markflorisson88(a)gmail.com> wrote: > On 27 January 2012 15:47, Simon King <simon.king(a)uni-jena.de> wrote: >> Hi all, >> >> I am still *very* frustrated about the fact that Cython does not tell >> where the error occurs. Since about one week, I am adding lots and >> lots of lines into Sage that write a log into some file, so that I get >> at least some idea where the error occurs. But still: Even these >> extensive logs do not provide a hint on what exactly is happening. >> >> How can I patch Cython such that some more information on the location >> of the error is printed? I unpacked Sage's Cython spkg, and did "grep - >> R ignored .", but the code lines containing the word "ignored" did not >> seem to be the lines that are responsible for printing the warning >> message >> Exception AttributeError: 'PolynomialRing_field_with_category' >> object has no attribute '_modulus' in ignored >> >> Can you point me to the file in Sage's Cython spkg which is >> responsible for printing the warning? >> >> Best regards, >> Simon > > These messages are written by PyErr_WriteUnraisable, which is a > CPython C API function that writes unraisable exceptions. There are > typically two reasons for unraisable exceptions: > > 1) as Robert mentioned, a function that does not allow propagation > of exceptions, e.g. > > cdef int func(): > raise Exception > > Here there is no way to propagate the raised exception, so > instead one should write something like > > cdef int func() except -1: ... > > Alternatively one may use 'except *' in case there is no error > indicator and Cython should always check, or "except ? -1" which means > "-1 may or may not indicate an error". > > 2) in deallocators or finalizers (e.g. __dealloc__ or __del__) > > For functions the right thing is to add an except clause, for > finalizers and destructors one could use the traceback module, e.g. > > try: > ... > except: > traceback.print_exc() > > If this all still doesn't help, try setting a (deferred) breakpoint on > __Pyx_WriteUnraisable or PyErr_WriteUnraisable. Actually, I don't see why the default is to write unraisable exceptions. Instead Cython could detect that exceptions may propagate and have callers do the check (i.e. make it implicitly "except *"). Was this not implemented because Cython only knows whether functions may propagate exceptions at code generation time by looking at the presence of an error label? Maybe it could keep code insertion points around for every call to such a potential function and if the function uses the error label have the caller perform the check? Although I do forsee problems for external such functions... maybe Cython could have it's own threadstate regardless of the GIL which would indicate whether an error has occurred? e.g. CyErr_Occurred()?

4 11

Upcoming issues with NumPy deprecated APIs and Cython's sizeof checks
by Lisandro Dalcin Jan. 31, 2012

Jan. 31, 2012

I'm testing my code with numpy-dev. They are trying to discourage use of deprecated APIs, this includes direct access to the ndarray struct. In order to update your code, you have to pass -DNPY_NO_DEPRECATED_API to the C compiler (or #define it before including NumPy headers). However, they have implemented this feature by exposing the ndarray type with just the Python object header: https://github.com/numpy/numpy/blob/master/numpy/core/include/numpy/ndarray… Obviously, this interact bad with Cython's sizeof check, I'm getting this runtime warning: build/lib.linux-x86_64-2.7/petsc4py/lib/__init__.py:64: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility I think there is nothing Cython can do about this (other than special-casing NumPy to disable this VERY useful warning). I've tried the patch below with success, but I'm not convinced... Does any of you have a suggestion for NumPy folks about how to improve this? diff --git a/numpy/core/include/numpy/ndarraytypes.h b/numpy/core/include/numpy/ndarraytypes.h index 0288272..1fcbf52 100644 --- a/numpy/core/include/numpy/ndarraytypes.h +++ b/numpy/core/include/numpy/ndarraytypes.h @@ -695,6 +695,7 @@ typedef struct tagPyArrayObject_fields { #ifdef NPY_NO_DEPRECATED_API typedef struct tagPyArrayObject { PyObject_HEAD + char _npy_array_fields[sizeof(PyArrayObject_fields)-sizeof(PyObject)]; } PyArrayObject; #else /* -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

3 6

Slow traceback reconstruction in IPython between 0.15.1 and master
by Wes McKinney Jan. 29, 2012

Jan. 29, 2012

Just wanted to bring this issue to your guys' attention in case you knew what was responsible for this: https://github.com/ipython/ipython/issues/1317#issuecomment-3652550 I traced down the problem (with git bisect) to a seemingly innocuous commit referenced in that GitHub thread. The issue seemed to only present itself in IPython, so likely there was some problem with inspecting the Cython frames for giving context around the full traceback. best, Wes

4 10

Bug in multiplied tuple optimization
by Vitja Makarov Jan. 29, 2012

Jan. 29, 2012

Investigating sage-tests segfaults I found that this code causes sigsegv: def foo(): return (0,) * len('abc') foo() -- vitja.

1 0

AddTraceback() slows down generators
by Stefan Behnel Jan. 29, 2012

Jan. 29, 2012

Hi, I did some callgrind profiling on Cython's generators and was surprised to find that AddTraceback() represents a serious performance penalty for short running generators. I profiled a compiled Python implementation of itertools.groupby(), which yields (key, group) tuples where the group is an iterator again. I ran this code in Python for benchmarking: """ L = sorted(range(1000)*5) all(list(g) for k,g in groupby(L)) """ Groups tend to be rather short in real code, often just one or a couple of items, so unpacking the group iterator into a list will usually be a quick loop and then the generator raises StopIteration on termination and builds a traceback for it. According to callgrind (which, I should note, tends to overestimate the amount of time spent in memory allocation), the iteration during the group unpacking takes about 30% of the overall runtime of the all() loop, and the AddTraceback() call at the end of each group traversal takes up to 25% (!) on my side. That means that more than 80% of the group unpacking time goes into raising StopIteration from the generators. I attached the call graph with the relative timings. About half of the exception raising time is eaten by PyString_FromFormat() that builds the function-name + line-position string (which, I may note, is basically a convenience feature). This string is a constant for a generator's StopIteration exception, at least for each final return point in a generator, but here it is being recreated over and over again, for each exception that gets raised. Even if we keep creating a new frame instance each time (which should be ok because CPython has a frame instance cache already and we'd only create one during the generator lifetime), the whole code object could actually be cached after the first creation, preferably bound to the lifetime of the generator creator function/method. Or, more generally, one code object per generator termination point, which will be a single point in the majority of cases. For the specific code above, that should shave off almost 20% of the overall runtime of the all() loop. I think that's totally worth doing. Stefan

5 24

inline defnode calls
by mark florisson Jan. 25, 2012

Jan. 25, 2012

I just noticed the inline defnode call code. When I try to compile with 'cython -Xoptimize.inline_defnode_calls=True test.pyx' with the following code: def foo(x): print foo foo(10) I get Error compiling Cython file: ------------------------------------------------------------ ... def foo(x): print x foo(10) ^ ------------------------------------------------------------ test.pyx:4:3: Compiler crash in InlineDefNodeCalls ModuleNode.body = StatListNode(test.pyx:1:0) StatListNode.stats[2] = ExprStatNode(test.pyx:4:3) ExprStatNode.expr = SimpleCallNode(test.pyx:4:3, result_is_used = True, use_managed_ref = True) Compiler crash traceback from this point on: File "/Users/mark/cy/Cython/Compiler/Visitor.py", line 176, in _visitchild result = handler_method(child) File "/Users/mark/cy/Cython/Compiler/Optimize.py", line 1656, in visit_SimpleCallNode if not function_name.cf_state.is_single: AttributeError: 'NoneType' object has no attribute 'is_single'

3 6

Speedup module-level lookup
by Vitja Makarov Jan. 21, 2012

Jan. 21, 2012

I tried to optimize module lookups (__pyx_m) by caching internal PyDict state. In this example bar() is 1.6 time faster (500us against 842us): C = 123 def foo(a): return C * adef bar(): for i in range(10000): foo(i) Here is proof of concept:https://github.com/vitek/cython/commit/1d134fe54a74e6fc6d39d09973db… So the question is: does it worth it? -- vitja.

5 16