Mailman 3 May 2012 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

SEP 201 draft: Native callable objects
by Dag Sverre Seljebotn June 1, 2012

June 1, 2012

[Discussion on numfocus(a)googlegroups.com please] I've uploaded a draft-state SEP 201 (previously CEP 1000): https://github.com/numfocus/sep/blob/master/sep201.rst """ Many callable objects are simply wrappers around native code. This holds for any Cython function, f2py functions, manually written CPython extensions, Numba, etc. Obviously, when native code calls other native code, it would be nice to skip the significant cost of boxing and unboxing all the arguments. """ The thread about this on the Cython list is almost endless: http://thread.gmane.org/gmane.comp.python.cython.devel/13416/focus=13443 There was a long discussion on the key-comparison vs. interned-string approach. I've written both up in SEP 201 since it was the major point of contention. There was some benchmarks starting here: http://thread.gmane.org/gmane.comp.python.cython.devel/13416/focus=13443 And why provide a table and not a get_function_pointer starting here: http://thread.gmane.org/gmane.comp.python.cython.devel/13416/focus=13443 For those who followed that and don't want to read the entire spec, the aspect of flags is new. How do we avoid to duplicate entries/check against two signatures for cases like a GIL-holding caller wanting to call a nogil function? My take: For key-comparison you can compare under a mask, for interned-string we should have additional flags field. The situation is a bit awkward: The Cython list consensus (well, me and Robert Bradshaw) decided on what is "Approach 1" (key-comparison) in SEP 201. I pushed for that. Still, now that a month has passed, I just think key-comparison is too ugly, and that the interning mechanism shouldn't be *that* hard to code up, probably 500 lines of C code if one just requires the GIL in a first iteration, and that keeping the spec simpler is more important. So I'm tentatively proposing Approach 2. Dag

2 6

Fwd: Re: SEP 201 draft: Native callable objects
by Dag Sverre Seljebotn May 31, 2012

May 31, 2012

Forgot to CC this list... -------- Original Message -------- Subject: Re: [Cython] SEP 201 draft: Native callable objects Date: Thu, 31 May 2012 22:06:02 +0200 From: Dag Sverre Seljebotn <d.s.seljebotn(a)astro.uio.no> Reply-To: numfocus(a)googlegroups.com To: numfocus(a)googlegroups.com On 05/31/2012 09:29 PM, Dag Sverre Seljebotn wrote: > On 05/31/2012 08:50 PM, Robert Bradshaw wrote: >> On Thu, May 31, 2012 at 7:04 AM, Dag Sverre Seljebotn >> <d.s.seljebotn(a)astro.uio.no> wrote: >>> [Discussion on numfocus(a)googlegroups.com please] >>> >>> I've uploaded a draft-state SEP 201 (previously CEP 1000): >>> >>> https://github.com/numfocus/sep/blob/master/sep201.rst >>> >>> """ >>> Many callable objects are simply wrappers around native code. This >>> holds for >>> any Cython function, f2py functions, manually written CPython >>> extensions, >>> Numba, etc. >>> >>> Obviously, when native code calls other native code, it would be nice to >>> skip the significant cost of boxing and unboxing all the arguments. >>> """ >>> >>> >>> The thread about this on the Cython list is almost endless: >>> >>> http://thread.gmane.org/gmane.comp.python.cython.devel/13416/focus=13443 >>> >>> There was a long discussion on the key-comparison vs. interned-string >>> approach. I've written both up in SEP 201 since it was the major >>> point of >>> contention. There was some benchmarks starting here: >>> >>> http://thread.gmane.org/gmane.comp.python.cython.devel/13416/focus=13443 >>> >>> And why provide a table and not a get_function_pointer starting here: >>> >>> http://thread.gmane.org/gmane.comp.python.cython.devel/13416/focus=13443 >>> >>> For those who followed that and don't want to read the entire spec, the >>> aspect of flags is new. How do we avoid to duplicate entries/check >>> against >>> two signatures for cases like a GIL-holding caller wanting to call a >>> nogil >>> function? My take: For key-comparison you can compare under a mask, for >>> interned-string we should have additional flags field. >>> >>> The situation is a bit awkward: The Cython list consensus (well, me and >>> Robert Bradshaw) decided on what is "Approach 1" (key-comparison) in SEP >>> 201. I pushed for that. >>> >>> Still, now that a month has passed, I just think key-comparison is >>> too ugly, >>> and that the interning mechanism shouldn't be *that* hard to code up, >>> probably 500 lines of C code if one just requires the GIL in a first >>> iteration, and that keeping the spec simpler is more important. >>> >>> So I'm tentatively proposing Approach 2. >> >> I'm still not convinced that a hybrid approach, where signatures below >> some cutoff are compiled down to keys, is not a worthwhile approach. >> This gets around variable-length keys (both the complexity and >> possible runtime costs for long keys) and allows simple libraries to >> produce and consume fast callables without participating in the >> interning mechanism. > > I still think this gives us the "worst of both worlds", all the > disadvantages and none of the advantages. Wait -- the complexity of the key approach is in the compilation, but avoiding any encoding/decoding would remove a major source of spec complexity. So this could just work: typedef struct { union { char *interned_sigptr; char short_sig[8]; } uintptr_t flags; void *funcptr; }; And then flags contains whether the signature is short. I think compiling down a signature that doesn't end with 0x000 is rather complicated even if there's no Huffman. The point is to be able to hand off a char* to a signature parsing routine easily, with no decompilation. Using a flag avoids that but requires a couple more instructions. Pro: Get somewhere without actually implementing interning (that's like a 3-hour job if you require the GIL, a little more to make sure it's forward-compatible) Cons: Is more complicated. Couple of extra assembly instructions but they probably don't matter. Dag > > How many simple libraries are there really? Cython on one end, the > magnificently complicated NumPy ufuncs on the other? Thinking big, > perhaps PyPy and Julia? Cython, PyPy, Julia would all have to deal with > long signatures anyway. And NumPy ufuncs are already complicated so even > more low-level stuff wouldn't hurt. > >> It's unclear how to rendezvous on a common interning interface without >> the GIL/Python, so perhaps requiring the GIL to use it not to onerous. >> An alternative is to acquire the GIL in the first/reference >> implementation (which could allow the interning function pointers to >> be cached by an external GIL-oblivions JIT for example). Presumably >> some other locking mechanism would be required if the GIL is not used, >> so the overhead would likely not be that great. > > Yes. I guess a goal could be to make sure there's no ABI breakage > if/when the GIL requirement is lifted. > > Since modules can already have a reference to the interner by the time > the first module interfacing with a GIL-less world is imported, this is > non-trivial, but "every problem can be solved with another level of > indirection", and particularly this one. > > Good idea on separating out interning as a separate spec; that's > definitely useful for interfaces etc. as well down the line. I can get > to work on a string interning spec and implementation as SEP 202 in > spare minutes over the next month or so, but I won't bother unless SEP > 201 will uses interning. My role in that depends on Travis' timeline as > well, as my ETA is so unpredictable. > > Dag

1 0

Re: [Cython] [Python-Dev] C-level duck typing
by Dag Sverre Seljebotn May 31, 2012

May 31, 2012

On 05/18/2012 12:57 AM, Nick Coghlan wrote: > I think the main things we'd be looking for would be: > - a clear explanation of why a new metaclass is considered too complex a > solution > - what the implications are for classes that have nothing to do with the > SciPy/NumPy ecosystem > - how subclassing would behave (both at the class and metaclass level) > > Yes, defining a new metaclass for fast signature exchange has its > challenges - but it means that *our* concerns about maintaining > consistent behaviour in the default object model and avoiding adverse > effects on code that doesn't need the new behaviour are addressed > automatically. > > Also, I'd consider a functioning reference implementation using a custom > metaclass a requirement before we considered modifying type anyway, so I > think that's the best thing to pursue next rather than a PEP. It also > has the virtue of letting you choose which Python versions to target and > iterating at a faster rate than CPython. This seems right on target. I could make a utility code C header for such a metaclass, and then the different libraries can all include it and handshake on which implementation becomes the real one through sys.modules during module initialization. That way an eventual PEP will only be a natural incremental step to make things more polished, whether that happens by making such a metaclass part of the standard library or by extending PyTypeObject. Thanks, Dag

4 13

vcvarsall.bat error on win32 with mingw when using cython_inline
by Dave Hirschfeld May 31, 2012

May 31, 2012

Hi cython devs, I got the "unable to find vcvarsall.bat" error when using cython_inline when trying to compile with mingw. Using cython normally (creating a setup.py file) worked fine however. I was able to fix the problem for me by changing inline.py to parse the config files before building the extension. I've created a pull request with my changes: https://github.com/cython/cython/pull/129 Apologies if I haven't used the appropriate workflow - I'm fairly new to git and this is my first pull request! HTH, Dave

2 1

Re: [Cython] [cython-users] Re: How to use Cython to wrap a C++ template class for use in Python?
by Robert Bradshaw May 30, 2012

May 30, 2012

On Tue, May 29, 2012 at 5:42 PM, Paul Leopardi <paul.leopardi(a)iinet.net.au> wrote: > > On Wednesday, 30 May 2012 02:40:47 UTC+10, Chris Barker wrote: >> >> >> Well, the third option is to use your own home-made templatesto >> auto-generate cython code for each type. That's actually not as >> painful as it sounds -- there are lot's of templating systems for >> python -- Cheetah, for instance is designed for "any" use, not just >> html (though I'm never used it for Cython). >> >> http://www.cheetahtemplate.org/ >> >> You might want to look at the "bottleneck" project -- they did >> something like this -- not for calling a C++ templates, but the >> principle is the same. >> >> http://berkeleyanalytics.com/bottleneck/ >> > > Chris, thanks for your suggestions. I will take a look at Cheetah, but I'm > afraid it would add extra complication to an already very complicated build > process for PyClical. What I would like is a way for Cython to make > instantiations of my C++ template classes visible to Python. Cython supports > C++ templates, and Cython now has fused types, but as far as I can tell, > these two template concepts within Cython do not work with each other in any > meaningful, documented way, let alone support what I am proposing. Maybe I > am just too late to propose another Google Summer of Code project, or > perhaps the whole idea is all too hard? Surely there must be a wider > use-case for this idea than just me and my one library? Should I repost to > the core developers list? I think there are several issues with why Cython does not (yet?) have these capabilities, primarily: (1) Metaprogramming done Right can be very nice, but done wrong is disastrous, (2) The AST of Cython is really not that nice to work with, and (3) If it's just about specifying types, template preprocessing and, eventually, using JITs may be sufficient, more flexible, and certainly better than a half-baked solution. The thread you linked to also has some good discussion. We actually had a lot of discussion about this issue at the Cython Days workshop last year, and never hit upon a metaprogramming framework that seemed Right. Consensus was that until a clean proposal was put forward, we would focus on making it easy to use your templating engine of choice and implement fused types which would cover 85% or more of the need for metaproramming (especially tight loops over numeric types, as opposed to more generic cases where Python objects and vtables are often good enough). The fact that fused types don't work with C++ specializations is certainly a bug, though this still limits us to a fixed number of instantiations (and separate Python classes) in Python space due to the nature of C++ templates. - Robert

2 1

Re: [Cython] gsoc: array expressions
by mark florisson May 30, 2012

May 30, 2012

On 25 May 2012 21:53, Frédéric Bastien <nouiz(a)nouiz.org> wrote: > Hi, > > > Sorry for the delay, I had some schedule change. > > thanks for adding me. Should I subscribe to cython-dev? How much email > daily there is? I didn't found this on the archives. Fell free to add > me in CC again when you think it is appropriate. There is usually not so much traffic on cython-dev, unless something comes up that is debated to the death :) > I'll reply here to all email at the same time. Do you prefer that I > reply to each email individually if this happen again? I'll try to > reply faster next time. No worries, either way works fine, don't worry too much about protocol (the only thing to note is that we do bottom posting). > - About pickling theano, we currently can't pick Theano function. It > could be made to work in some cases, but not for all cases as there is > hardware dependent optimization in the Theano function. Currently it > is mostly CPU vs GPU operation. So if we stay on the CPU, we could do > some pickling, but we should make sure that the compiled c code into > python module are still there when we unpickle or recompile them. > > - I think it make sense to make a theano graph from cython ast, > optimize and redo a cython ast from the optimized graph. This would > allow using Theano optimizations. Ok, the important thing is that the graph can be pickled, it should be pretty straightforward to generate code to build the function again from the loaded graph. > - It also make sense to do the code generation in Theano and reuse it > in Cython. But this would make the Theano dependency much stronger. > I'm not sure you want this. > > > - Another point not raised, theano need to know at compile time is the > dtype, number of dimensions and witch dimensions are broadcastable for > each variable. I think that the last one could cause problem, but if > you use specialization for the dtype, the same can be done for the > broadcsatability of a dimensions. Hm, that would lead to kind of an explosion of combinations. I think we could specialize only on no broadcasting at all (except for operands with lesser dimensionality). > - The compyte(gpu nd array) project do collapsing of dimensions. This > is an important optimization on the GPU as doing the index computation > in parallel is costlier. I think on the CPU we could probably do > collapsing just of the inner dimensions to make it faster. > > - Theano don't generate intrinsect or assembly, but we suppose that > g++ will generate vectorized operation for simple loop. Recent version > of gcc/g++ do this. Right, the aim is definitely to specialize for contiguous arrays, where you collapse everything. Specializing statically for anything more would be unfeasible, and better handled by a runtime compiler I think. For the C backend, I'll start by generating simple C loops and see if the compilers vectorize that already. > - Our generated code for element-wise operation take care a little > about the memory access pattern. We swap dimensions to iterate on the > dimensions with the smallest strides. But we don't go further. > > - What do you mean by CSE? Constant optimization? Yes, common subexpression elimination and also hoisting of unchanging expressions outside the loop. > Fred I started a new project, https://github.com/markflorisson88/minivect , which currently features a simple C code generator. The specializer and astbuilder do most of the work of creating the right AST, so the code generator only has to implement code generation functions for simple expressions. Depending on how it progresses I will look at incorporating Theano's optimizations into it and having Theano use it as a C backend for compatible expressions.

3 9

NumFOCUS and continuous integration
by Dag Sverre Seljebotn May 28, 2012

May 28, 2012

There's NumFOCUS money for CI -- also for the work used to configure and running it etc. It would not be only for Cython, but for open source scientific Python (Cython, NumPy, SciPy etc.). So if anybody would like to work up to 50% time on that (or perhaps just want to work for some weeks with setting up a CI system that can be shared by said projects) then please get in touch with Travis and NumFOCUS (see thread below). Some context: There's been heavy discussion on the NumFOCUS list on identifying a "core subset" of packages (Cython+NumPy+SciPy+Pandas+...) that could be tested and promoted together [1]. I started a discussion [2] to try to divert the question of CI-ing that core subset in the usual way ("release managing a software distribution") to instead be about giving those resources to the open source projects involved, so that there's not duplicate work in doing the CI-ing. Of course, Cython is in better standing than most already on the CI front thanks to Stefan and the Sage project, but the fact that the work is paid for may perhaps make it more appealing. Full threads: [1] https://groups.google.com/forum/?fromgroups#!topic/numfocus/MnRzBhmqXqk [2] https://groups.google.com/forum/?fromgroups#!topic/numfocus/I_kmL4FUGaY Dag -------- Original Message -------- Subject: Re: Continuous integration Date: Mon, 28 May 2012 13:47:52 -0500 From: Travis Oliphant <teoliphant(a)gmail.com> Reply-To: numfocus(a)googlegroups.com To: numfocus(a)googlegroups.com On May 28, 2012, at 1:27 PM, Dag Sverre Seljebotn wrote: > On 05/28/2012 05:55 PM, David wrote: >> >> >> On Sunday, May 27, 2012 3:05:06 AM UTC+9, Dag Sverre Seljebotn wrote: >> >> That Other Thread contained some references to CI. So I'm mainly >> wondering what the current NumFOCUS plans for supporting CI efforts >> are, >> if any? Was there a mention on money being available for somebody to >> work on that? >> >> I think of Cython as one vertex in a CI graph: >> >> a) Upstream, Cython depends on various versions of NumPy and Python as >> part of its CI >> >> b) Downstream, we rely on building and testing Sage as an extended >> regression test suite. It's not just that our test suite doesn't have >> 100% coverage, but also sometimes that we intentionally break backwards >> compatibility, and fixing up Sage gives a nice indication of the >> consequences of a change. (I imagine NumPy is in a very similar >> situations with lots of libraries depending on it and potential for >> very >> subtle breakage of backwards compatibility.) >> >> Ideally, we'd have LOTS of libraries using Cython in our CI -- mpi4py, >> Pandas, scikits-learn... BUT, we're running into problems as it is with >> the CI server hardware keeping up (if there was infinite CI there'd >> probably be a lot more tests set up). >> >> Since most scientific-python libraries have both dependencies and >> dependees, it seems like there should be some benefit to having the >> same >> CI system test all of them. That could conserve both hardware use and >> administration overhead. And, which I believe is very important, >> make it >> easier for small projects like scikits-sparse to participate in >> automatic CI by simply participating in an environment maintained by >> the >> larger projects like Cython and NumPy. >> >> I think there's two ways of spinning this: >> >> 1) Build "Sciome" in a CI and approach it by "release managing" Sciome >> >> 2) Focus on providing "infrastructure for library developers", where >> there's a relatively big CI graph where each project has a node. I.e. >> something like "ShiningPanda for scientific Python", but with a >> critical >> difference being that each project can use the build artifacts of >> others; Cython would flip a switch and have all projects depending on >> Cython being rebuilt to try the new Cython version. This seems >> complicated, but certainly Jenkins for one seems to support such setups >> already so its mainly about hardware and administration... >> >> I know I get a lot more excited by 2) than 1), even if it's perhaps >> mainly the spin put on it rather than a technical difference. >> >> >> I know I prefer 2 as well. I know of at least one attempt of doing >> something like this (linux-only, though): the build service from open >> suse (http://www.open-build-service.org/). 4 years ago, it could already >> do useful bits that are not trivial in any CI I am aware of: >> >> - dependencies between packages is known: if you update the build >> service package numpy, and the build is successful, it will >> automatically rebuild all the dependent packages >> - it handles completely isolated build environments through vms >> - it produces rpm, debian, etc.. that can be easily installed in the >> supported distributions. >> >> My ideal setup would be something like this, but working on windows and >> mac (and not developed in perl), and with easier set up ala travis CI. I >> don't know yet the best way to go there: is is based on existing >> infrastructure (jenkins, travis, something else), building our own, a >> hybrid between the two ? > > What I was imagined was really something very simple (and I put it too convoluted). Just have NumFOCUS cash out :-) for one or more servers, then give everybody who wants shell access to a shared CI instance (running Jenkins or whatever). This sounds like a great plan. NumFOCUS can provide for this. We will just need a budget and board approval (but I suspect the board will be happy to approve this kind of plan). So, who wants to spec out the machine? There is some money. The problem is actually a people problem. Who is going to be available to do the work. So, far, I have not been able to find anyone experienced who is willing to work on this stuff at 50% time. If you are willing, let me know in a private email or an email to admin(a)numfocus.org Best, -Travis

1 0

gsoc: array expressions
by Frédéric Bastien May 25, 2012

May 25, 2012

I just resended this email as it was rejected by the mailing list. So I subscribed to it. Hi, Sorry for the delay, I had some schedule change. thanks for adding me. Should I subscribe to cython-dev? How much email daily there is? I didn't found this on the archives. Fell free to add me in CC again when you think it is appropriate. I'll reply here to all email at the same time. Do you prefer that I reply to each email individually if this happen again? I'll try to reply faster next time. - About pickling theano, we currently can't pick Theano function. It could be made to work in some cases, but not for all cases as there is hardware dependent optimization in the Theano function. Currently it is mostly CPU vs GPU operation. So if we stay on the CPU, we could do some pickling, but we should make sure that the compiled c code into python module are still there when we unpickle or recompile them. - I think it make sense to make a theano graph from cython ast, optimize and redo a cython ast from the optimized graph. This would allow using Theano optimizations. - It also make sense to do the code generation in Theano and reuse it in Cython. But this would make the Theano dependency much stronger. I'm not sure you want this. - Another point not raised, theano need to know at compile time is the dtype, number of dimensions and witch dimensions are broadcastable for each variable. I think that the last one could cause problem, but if you use specialization for the dtype, the same can be done for the broadcsatability of a dimensions. - The compyte(gpu nd array) project do collapsing of dimensions. This is an important optimization on the GPU as doing the index computation in parallel is costlier. I think on the CPU we could probably do collapsing just of the inner dimensions to make it faster. - Theano don't generate intrinsect or assembly, but we suppose that g++ will generate vectorized operation for simple loop. Recent version of gcc/g++ do this. - Our generated code for element-wise operation take care a little about the memory access pattern. We swap dimensions to iterate on the dimensions with the smallest strides. But we don't go further. - What do you mean by CSE? Constant optimization? Fred

1 0

0.17
by mark florisson May 25, 2012

May 25, 2012

Hey, I think we already have quite a bit of functionality (nearly) ready, after merging some pending pull requests maybe it will be a good time for a 0.17 release? I think it would be good to also document to what extent pypy support works, what works and what doesn't. Stefan, since you added a large majority of the features, would you want to be the release manager? In summary, the following pull requests should likely go in - array.array support (unless further discussion prevents that) - fused types runtime buffer dispatch - newaxis - more? The memoryview documentation should also be reworked a bit. Matthew, are you still willing to have a go at that? Otherwise I can clean up the mess first, some things are no longer true and simply outdated, and then have a second opinion. Mark

7 23