Mailman 3 January 2014 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

Compiler crash in RemoveUnreachableCode
by Andriy Kornatskyy Sept. 7, 2014

Sept. 7, 2014

The cython compiler crash report below. Steps to reproduce: 1. virtualenv env 2. env/bin/easy_install cython 3. env/bin/easy_install wheezy.http The wheezy.http has dependency on wheezy.core. If I install those two packages separately there is no error, only if through dependencies. Thanks. Andriy Kornatskyy File "Visitor.py", line 178, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4437) File "Visitor.py", line 137, in Cython.Compiler.Visitor.TreeVisitor._raise_compiler_error (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:3655) Cython.Compiler.Errors.CompilerCrash: Error compiling Cython file: ------------------------------------------------------------ ... """ """ __version__ = '0.1.129' ^ ------------------------------------------------------------ src/wheezy/core/__init__.py:5:14: Compiler crash in RemoveUnreachableCode ModuleNode.body = StatListNode(__init__.py:5:14) Compiler crash traceback from this point on: File "Visitor.py", line 170, in Cython.Compiler.Visitor.TreeVisitor._visit (/var/folders/g8/2kym1h8n7qbgrwg4qkfqw1gw0000gn/T/easy_install-JVXbGE/Cython-0.20/Cython/Compiler/Visitor.c:4275) File “/.../env/lib/python2.7/site-packages/Cython-0.20-py2.7-macosx-10.9-x86_64.egg/Cython/Compiler/ParseTreeTransforms.py", line 2135, in visit_StatListNode if not self.current_directives['remove_unreachable']: TypeError: 'NoneType' object has no attribute '__getitem__'

2 7

Broken list multiplication
by R. Andrew Ohana Feb. 4, 2014

Feb. 4, 2014

It seems like eaca39b00b1451fe6c846a0a670e4c8b39935525 broke some (very simple) instances of list multiplication when there are at least 3 terms in the product. e.g. on my system I'm getting these results: L = [None] * 2 * 3 print(len(L)) 3 L = 2 * [None] * 3 print(len(L)) 3 L = [None] * 2 * 3 * 5 print(len(L)) 5 -- Andrew

3 7

breaking news in Py3.4: inspect.isfunction(cyfunction) == True
by Stefan Behnel Feb. 2, 2014

Feb. 2, 2014

Hi, Yury Selivanov just committed a revised version of a patch I wrote for CPython's inspect module last year. It now accepts Cython's function type as Python function, based on the function-like attributes that it exports. http://hg.python.org/cpython/rev/32a660a52aae That means that things like inspect.signature(cyfunction) will also work now, and finally have a reason to be further improved, including parameter introspection, annotations, etc. :) Stefan

1 3

memoryview refcount error
by Syam Gadde Jan. 31, 2014

Jan. 31, 2014

Hi again, I'm exploring the typed memoryview feature with numpy and encountered a refcount error. It seems to happen when I create a slice of a memoryview, and transpose it. If I don't do both, I don't get the refcount error. It happens in the error cleanup code. I suspect that any error that causes the function to jump to the error cleanup code will do the same thing as the Exception below -- but that seemed the easiest way to demonstrate the problem. However, if there truly is an error in my code, please let me know! Thanks, -syam # BEGIN CODE import numpy cimport numpy class MyException(Exception): def __init__(self): Exception.__init__(self) def testfunc(): a = numpy.arange(12).reshape([3,4]) cdef numpy.int_t[:,:] a_view = a ## here is a slice followed by a transpose cdef numpy.int_t[:,:] b = a_view[:,:].T ## same thing happens with a more realistic slicing: #cdef numpy.int_t[:,:] b = a_view[1:2,:].T ## however, there is no error if I do this instead: #cdef numpy.int_t[:,:] b = a_view.T ## also no error if I do this instead: #cdef numpy.int_t[:,:] b = a_view[:,:] # The exception below is just to force the function to abort # and run the extraneous __PYX_XDEC_MEMVIEW in the error # cleanup code: # Fatal Python error: Acquisition count is 0 (line XXXX) # Comment out this exception and we don't get the error raise MyException() try: testfunc() except MyException: pass # END CODE

1 1

segfault due to using DECREF instead of XDECREF
by Syam Gadde Jan. 31, 2014

Jan. 31, 2014

(apologies to the moderator for multiple postings -- I'm trying to figure out which of my email addresses is actually being exported to the public!) Hi, It seems that cython's static code analysis is failing on the following code. It thinks that a particular variable (myobject2) must not be NULL, but in fact it is, so the __Pyx_DECREF_SET segfaults. This came from third-party code, so I could fix it with an explicit initialization to None, but I'd love to have Cython deal with it directly. Thanks for your help! #<<<BEGIN tmpnone.py>>> # When compiled with Cython 0.19.2 or 0.20dev, the following code # segfaults on import import random def myfunc(): ## uncommenting the following fixes things #myobject2 = None myfalse = random.random() > 2 if myfalse: myobject = None myobject2 = None if not myfalse: # here Cython uses __Pyx_XDECREF_SET myobject = None print "Made it past myobject assignment" if not myfalse or myobject2 is None: # here Cython uses Pyx_DECREF_SET because it has determined # that __pyx_v_myobject2 can't be null, but it really, really can! # (if no one assigned myobject2 yet) myobject2 = None print "Made it past myobject2 assignment" myfunc() #<<<END tmpnone.py>>> #<<BEGIN setup.py>>> #<from distutils.core import setup from Cython.Build import cythonize setup( ext_modules = cythonize("tmpnone.py") ) #<<END setup.py>>> [prompt]$ python setup.py build_ext --inplace [prompt]$ ls tmpnone.* tmpnone.c tmpnone.py tmpnone.so [prompt]$ python Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37) [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import tmpnone Made it past myobject assignment Segmentation fault

3 8

Sage build problem
by Stefan Behnel Jan. 31, 2014

Jan. 31, 2014

Hi, There's some kind of C++ enum issue in the Sage build. Generated code in sage/libs/ppl.cpp: """ static PyObject *__pyx_pf_4sage_4libs_3ppl_11MIP_Problem_30solve( struct __pyx_obj_4sage_4libs_3ppl_MIP_Problem *__pyx_v_self) { enum Parma_Polyhedra_Library::PPL_MIP_Problem_Status __pyx_v_tmp; """ gcc error: """ sage/libs/ppl.cpp: In function 'PyObject* __pyx_pf_4sage_4libs_3ppl_11MIP_Problem_30solve(__pyx_obj_4sage_4libs_3ppl_MIP_Problem*)': sage/libs/ppl.cpp:5514:33: error: 'PPL_MIP_Problem_Status' in namespace 'Parma_Polyhedra_Library' does not name a type sage/libs/ppl.cpp:5514:67: error: invalid type in declaration before ';' token """ The type of "tmp" seems to be inferred. That might be a hint that it could be a problem in Cython. The Cython code in question is also a bit funny: sage/libs/ppl.pyx:1076 """ sig_on() try: tmp = self.thisptr.solve() # <<<<<<<<<<<<<< finally: sig_off() """ The try-finally is useless here, since the call to solve() cannot raise Python exceptions. The new try-finally code generation shows this clearly, it does not generate an exception handling block. The idea might have been that it could raise C++ exceptions, but they don't seem to be declared, so Cython won't do anything about them. Might be a hint that Cython could do more to make C++ exception handling safer by default. Stefan

1 0

"embedsignature" and "autotestdict" features in Py3.4
by Stefan Behnel Jan. 30, 2014

Jan. 30, 2014

Hi, recent changes in Py3.4 affect the functionality of the two directives "embedsignature" and "autotestdict". The so-called "argument clinic" extracts any embedded signatures from docstrings and moves them into a new property "__text_signature__". This is done at runtime, i.e. when either "__doc__" or "__text_signature__" are requested. http://hg.python.org/cpython/file/6a3e09cd96f3/Objects/methodobject.c#l182 I personally consider this a feature (and it works nicely with the signatures that Cython embeds), but you may or may not agree. It broke some of our own doctests, at least, because the "__doc__" value that we tested for was no longer the same. Regarding the "autotestdict", CPython also got smarter and now finds doctests embedded in C implemented functions all by itself. This is clearly an improvement compared to the previous behaviour. However, this also means that it now executes both the doctest of the function itself and its copy in the generated "__test__" dict (stored in the module under that name), meaning that it executes all function doctests twice. This also broke some of Cython's own doctests for us, which modified global state and thus failed the second time. I'm yet unsure if we should try to do something about this. Currently, the explicit doctest dict is generated by default, so all code that uses doctests in function docstrings suffers from this. We might be able to make Cython a bit smarter so that these docstrings are left out of the test dict when (C-)compiling in CPython 3.4. Deciding which ones are still needed might be a bit tricky, though... Stefan

2 2

0.20 tests fail with python3.4b2
by Julian Taylor Jan. 25, 2014

Jan. 25, 2014

hi, with python3.4b2 numpy_memoryview.acquire_release_cycle fails looking at the source the test should be disabled, but its still run. This might be a py3.4 change to doctest, but I know nothing about doctest, so I wanted to check first if you know something about this. Interestingly py3.4 runs 64 tests, while python3.3 only runs 32. the __test__ variable contains the same functions in both python versions (acquire_release is not in it) ====================================================================== FAIL: acquire_release_cycle (numpy_memoryview) Doctest: numpy_memoryview.acquire_release_cycle ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python3.4/doctest.py", line 2187, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for numpy_memoryview.acquire_release_cycle File "/tmp/buildd/cython-0.20/build/work-dir/memoryview/c/numpy_memoryview/numpy_memoryview.cpython-34m.so", line unknown line number, in acquire_release_cycle ---------------------------------------------------------------------- File "/tmp/buildd/cython-0.20/build/work-dir/memoryview/c/numpy_memoryview/numpy_memoryview.cpython-34m.so", line ?, in numpy_memoryview.acquire_release_cycle Failed example: acquire_release_cycle(a) Expected: deallocated! Got nothing

2 1

C code churn and benchmark diffs
by Stefan Behnel Jan. 23, 2014

Jan. 23, 2014

Hi, to get a better idea of what new optimisations (or C code changes in general) actually bring, I've added a diffing step to the benchmark runner jobs. The bigger ones now generate an additional output file "cfiles.diff.gz" that contains a diff of the C files generated in the current run against those of the last (successful) run. E.g. here: https://sage.math.washington.edu:8091/hudson/view/bench/job/cython-devel-py… (That one should start looking a lot smaller tomorrow :) While doing that, I noticed that there are a couple of unnecessary differences in the C files that are due to dict iteration, which is randomised in Py3.3+ (for security reasons). I fixed (most of?) them, so repeated Cython runs should now generate pretty much identical output files, if you strip the file header comment (diff can do that). The rest is generated deterministically in source code order already. It should be possible to do the same thing for the Sage build, I guess. That would also be interesting to look at from time to time, especially when things start failing for some reason. Stefan

1 0