Mailman 3 April 2012 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

CEP1000: Native dispatch through callables
by Dag Sverre Seljebotn May 5, 2012

May 5, 2012

Travis Oliphant recently raised the issue on the NumPy list of what mechanisms to use to box native functions produced by his Numba so that SciPy functions can call it, e.g. (I'm making the numba part up): @numba # Compiles function using LLVM def f(x): return 3 * x print scipy.integrate.quad(f, 1, 2) # do many callbacks natively! Obviously, we want something standard, so that Cython functions can also be called in a fast way. This is very similar to CEP 523 (http://wiki.cython.org/enhancements/nativecall), but rather than Cython-to-Cython, we want something that both SciPy, NumPy, numba, Cython, f2py, fwrap can implement. Here's my proposal; Travis seems happy to implement something like it for numba and parts of SciPy: http://wiki.cython.org/enhancements/nativecall Obviously this is (in a modified form) PEP-material, but I think it is much better to just get it working with a nice range of tools first (makes the PEP application stronger as well). Feedback most welcome! Dag

7 124

Re: [Cython] [cython-users] cimport numpy fails with Python 3 semantics
by Stefan Behnel May 2, 2012

May 2, 2012

mark florisson, 28.04.2012 21:57: > On 28 April 2012 19:50, Stefan Behnel wrote: >> mark florisson, 28.04.2012 20:33: >>> I think each module should have its own language level, so I think >>> that's a bug. I think the rules should be: >>> >>> - if passed as command line argument, use that for all cimported >>> modules, unless they define their only language level through the >>> directive >>> - if set as a directive, the language level will apply only to that module >> >> That's how it works. We don't run the tests with language level 3 in >> Jenkins because the majority of the tests is not meant to be run with Py3 >> semantics. Maybe it's time to add a numpy_cy3 test. >> >> If there are more problems than just this (which was a bug in numpy.pxd), >> we may consider setting language level 2 explicitly in numpy.pxd. > > Ah, great. Do we have any documentation for that? We do now. ;) However, I'm not sure cimported .pxd files should always inherit the language_level setting. It's somewhat of a grey area because user provided .pxd files would benefit from it since they likely all use the same language level as the main module, whereas the Cython shipped (and otherwise globally installed) .pxd files wouldn't gain anything and could potentially break. I think we may want to keep the current behaviour and set the language level explicitly in the few shipped .pxd files that are not language level agnostic (i.e. those that actually contain code). Stefan

2 2

Re: [Cython] [cython-users] Conditional import in pure Python mode
by mark florisson May 1, 2012

May 1, 2012

On 30 April 2012 13:14, Ian Bell <ian.h.bell(a)gmail.com> wrote: > > On Sun, Apr 29, 2012 at 10:58 PM, mark florisson <markflorisson88(a)gmail.com> > wrote: >> >> On 29 April 2012 01:33, Ian Bell <ian.h.bell(a)gmail.com> wrote: >> > Hello Cython users, >> > >> > I haven't the foggiest idea how easy this would be to implement, or how >> > to >> > do it in the first place, but my idea is to be able to have some sort of >> > idiom like >> > >> > if cython.compiled: >> > cython.import('from libc.math cimport sin') >> > else: >> > from math import sin >> > >> > that would allow for a pure Python file to still have 100% CPython code >> > in >> > it, but when it goes through the .py+.pxd-->.pyd compilation, use the >> > math.h >> > version instead. Is there any way that this could work? I can (and >> > will) >> > hack together a really nasty preprocessor that can edit the .py file at >> > Cython build time, but it is a nasty hack, and very un-Pythonic. For >> > me, >> > the computational efficiency using the math.h trig functions justify a >> > bit >> > of nastiness. >> > >> > Regards, >> > Ian >> >> I think a cython.cimport would be nice. If you want to have a crack at >> it, you may want to look in Cython/Compiler/ParseTreeTransforms.py at >> the TransformBuiltinMethods transform, and also at the >> Cython/Shadow.py module. > > > Mark, > > Any chance you can give me a wee bit more help? I can see how you need to > add a cimport function in Shadow.py. Supposing I pass it an import string > that it will parse for the import, whereto from there? There's quite a lot > of code to dig through. > > Ian > Certainly, it's great to see some enthusiasm. So first, it's important to determine how you want it. You can either simulate importing the rightmost module from the package, or you can simulate importing everything from the package. Personally I think if you say cython.cimport_module("libc.stdio") you want conceptually the stdio module to be returned. Maybe a 'cimport_star' function could cimport everything from the scope, but I'm not sure how great an idea that is in pure mode (likely not a good one). So lets assume we want to use the following syntax: stdio = cython.cimport_module("libc.stdio"). In the TransformBuiltinMethods you add another case to the visit_SimpleCallNode, which will match for the function name "cimport_module" (which is already known to be used as an attribute of the Cython module). TransformBuiltinMethods is a subclass of Visitor.EnvTransform, which keeps track of the scope (e.g. if the node is in a function, the local function scope, etc). This class is a subclass of CythonTransform, which has a Main.Context set up in Pipeline.py, accessible through the 'self.context' attribute. Main.Context has methods to find and process pxd files, i.e. through the find_pxd_file and the process_pxd methods (or maybe the find_module method will work here). So you want to validate that the string that gets passed to the 'cimport_module' function call is a string literal, and that this is in fact happening from the module scope. You can then create an Entry with an as_module attribute and put it in the current scope. The as_module attribute holds the scope of the cimported pxd file, which means its a "cimported module". You can do this through the 'declare_module' method of the current scope (self.current_env().declare_module(...)) (see Symtab.py for that method). I hope that helps you get set up up somewhat, don't hesitate to bug the cython-devel mailing list with any further questions. BTW, we probably want to formally discuss the exact syntax and semantics before implementing it, so I think it will be a good idea to summarize what you want and how you want it on the cython-dev mailing list. Good luck :)

2 3

Wacky idea: proper macros
by Nathaniel Smith May 1, 2012

May 1, 2012

Was chatting with Wes today about the usual problem many of us have encountered with needing to use some sort of templating system to generate code handling multiple types, operations, etc., and a wacky idea occurred to me. So I thought I'd through it out here. What if we added a simple macro facility to Cython, that worked at the AST level? (I.e. I'm talking lisp-style macros, *not* C-style macros.) Basically some way to write arbitrary Python code into a .pyx file that gets executed at compile time and can transform the AST, plus some nice convenience APIs for simple transformations. E.g., if we steal the illegal token sequence @@ as our marker, we could have something like: @@ # alone on a line, starts a block of Python code from Cython.MacroUtil import replace_ctype def expand_types(placeholder, typelist): def my_decorator(function_name, ast): functions = {} for typename in typelist: new_name = "%s_%s" % (function_name, typename) functions[name] = replace_ctype(ast, placeholder, typename) return functions return function_decorator @@ # this token sequence cannot occur in Python, so it's a safe end-marker # Compile-time function decorator # Results in two cdef functions named sum_double and sum_int @@expand_types("T", ["double", "int"]) cdef T sum(np.ndarray[T] arr): cdef T start = 0; for i in range(arr.size): start += arr[i] return start I don't know if this is a good idea, but it seems like it'd be very easy to do on the Cython side, fairly clean, and be dramatically less horrible than all the ad-hoc templating stuff people do now. Presumably there'd be strict limits on how much backwards compatibility we'd be willing to guarantee for code that went poking around in the AST by hand, but a small handful of functions like my notional "replace_ctype" would go a long way, and wouldn't impose much of a compatibility burden. -- Nathaniel

6 19

dynamically compile and import pure python modules
by mark florisson April 29, 2012

April 29, 2012

Hey, For the fused type runtime dispatch I found it very convenient to use the with statement, but that is not supported in Python 2.4. However, the compiler could dynamically compile compiler code with the compiler itself and import it (pyximport), if it is not needed to compile Cython itself. I gave it a try and it seems to work like a charm (but probably needs more testing :): https://github.com/markflorisson88/cython/commit/0c2983056919f7f4d30a809724… It seems like a bit of a hack, so I thought I'd bring it up here and ask for opinions, i.e. ask if anyone thinks it is a terrible idea. Mark

3 12

Re: [Cython] Cython 0.16 issue Windows with Mingw32
by Nathan Dunfield April 24, 2012

April 24, 2012

On Apr 23, 2012, at 1:28 PM, Stefan Behnel wrote: > Hmm, that line basically just says "PyCFunction_Call", which is a function > exported by CPython. I wonder why gcc would consider this "not a constant". > > Could you check if the preprocessor (gcc -E, with all the above includes) > also sees that on your side? It turns out I was running a version of MinGW that, while only a few years old, had a quite elderly version of gcc, namely 3.4 (yes, that's 3.4 and not 4.3). Once I updated to the most recent MinGW, which has gcc 4.6, and worked around a known bug in distutils ( http://bugs.python.org/issue12641 ), the issue with PyCFunction_Call went away and the entire 76k line Cython generated C file compiled without a hitch. So probably this is only a problem on very old compilers, and so perhaps not worth investigating further. Best, Nathan

2 1

Cython 0.16 issue Windows with Mingw32
by Nathan Dunfield April 24, 2012

April 24, 2012

I've encountered the following issue with Cython 0.16 on Windows with using the Mingw32 compiler (I'm using Python 3.2 here, but I don't think that's the issue): $ python3 setup.py build -c mingw32 running build running build_py running build_ext skipping 'SnapPy.c' Cython extension (up-to-date) building 'snappy.SnapPy' extension c:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -Iheaders -Iunix_kit -Iaddl_code -I. -Ipari/pari-2.3.4/include/ -Ipari/pari-2.3.4/include/pari -Ic:\Python32\include -Ic:\Python32\PC -c SnapPy.c -o build\temp.win32-3.2\Release\snappy.o SnapPy.c: In function `__pyx_f_6snappy_6SnapPy_13Triangulation_build_rep_into_Sn': SnapPy.c:25187: warning: implicit declaration of function `fg_get_num_orig_gens' SnapPy.c:25423: warning: implicit declaration of function `candidateSn_is_valid' SnapPy.c:25435: warning: implicit declaration of function `candidateSn_is_transitive' SnapPy.c: At top level: SnapPy.c:76434: error: initializer element is not constant SnapPy.c:76434: error: (near initialization for `__pyx_CyFunctionType_type.tp_call') error: command 'gcc' failed with exit status 1 The problem seems to be in the code that's pulled in from CythonFunction.c. I apologize for not providing a more minimal example (the above code is available at "hg clone static-http://math.uic.edu/t3m/hg/SnapPy") but the small module I tried didn't pull in the CythonFunction.c code. Thanks, Nathan

3 3

Julialang
by Dag Sverre Seljebotn April 24, 2012

April 24, 2012

Just heard about the Julia language and wanted to make sure it's on everybody's radar: http://julialang.org It's the first really decent language designed for scientists. Seems impressive to me, there's a few Cython features: - Dynamic typing with optional static types - Call C directly And then comes: - JIT - Templates - "Green" threading/coroutiens - Multiple dispatch (yay!) - Lisp-like macros and other metaprogramming facilities - Designed for parallelism and distributed computation Dag

7 13

test crash in Py3.2 (NumPy/memoryview/refcounting related?)
by Stefan Behnel April 23, 2012

April 23, 2012

Hi, I keep seeing test crashes in Py3.2 debug builds on Jenkins with the latest master, referring to ref-counting problems. Here, for example: https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND… and likewise in the series of builds starting at #312. I'm not sure what changed in that corner, it looks like one of those problems that was there for a while and suddenly starts showing when you touch that innocent looking brick at the other end of the house that was holding the balance, until now. In this case, that brick was this commit: https://github.com/cython/cython/commit/c8f61a668d0ce8af1020f520253e1b5f623… I reverted it here and that fixed the problem at least temporarily: https://github.com/cython/cython/commit/5aac8caf1ba21933cc85a51af3319c78fc0… but it seems to be back now (after my refnanny optimisations). Before reverting my changes, I was able to reproduce it somewhat reliably on sage.math by running these three tests together (non-forking): memslice numpy_bufacc numpy_memoryview None of the tests shows the problem when run by itself. I can't tell if it's also in the latest py3k because I don't have a NumPy lying around that works there. So 3.2 is basically the latest Python version this can be tested with, and it doesn't occur in the 3.1 tests. The tests use NumPy 1.6.1, i.e. the latest official release. Mark, Dag, could any of you take a look to see if it appears in any way more obvious to you than to me? Stefan

2 9