Mailman 3 August 2011 - cython-devel

Cython, Go and static duck typing
by Dag Sverre Seljebotn Aug. 14, 2011

Aug. 14, 2011

I read up on Go's interface model last week and am very excited about it. Perhaps this was old news to a lot of you, but it was new to me. This post is mostly just to make sure everybody is aware of prior art here in case it is a good fit for Cython down the road. I hope I'm not starting a long wasteful thread, let's keep it FYI, and if it's interesting I can try to learn a bit more of the technical details and write up a CEP prior to next summer's GSoC or something... The idea of polymorphism in Go might be phrased in the Cython language as: cdef interface Walker: int walk(float) cdef class Foo: # note: Does not inherit from Walker! cdef int walk(self, float length): .... def f(Walker obj): obj.walk(2.3) f(Foo()) This is duck typing, in the sense that Foo has no knowledge of the Walker interface (at module compilation time) -- it just satisfies it by implementing the same interface. The point of casting/conversion has all the information needed to build the vtable (somehow...). This of course uses a little more memory, but it is still bounded by the number of code lines you could write... This has a lot more flexibility (libraries don't have to agree on which one should have the "ultimate superclass", they can just agree on method names), and it just fits my brain and workflow much better as well: I'd use profiling to pinpoint the spots where I really need a vtable instead of dict lookups, and then it is more natural to introduce the interface in the caller, rather than to go back into the callee to introduce a type hierarchy (which might be more geared towards optimizing calls than towards what type relation is "natural"). Implementing this in Cython is likely going to be tougher than in Go if we want to support <Walker>obj, where obj has type "object", and not only <Walker><Foo>obj. But I'd think it is not impossible, just somewhat more costly on assignment...but that is often amortized over multiple calls. Dag Sverre

3 3

Vitja's CyFunction branch
by Stefan Behnel Aug. 14, 2011

Aug. 14, 2011

Hi, I wonder what we should do with Vitja's CyFunction branch. He mentioned issues with it in the past (I remember that there was one specific changeset that he considered questionable), and it seems that we found several ways to extend the function support beyond that, which may have an impact on the current way it's implemented. Is this something that makes sense to be included in 0.15, or should it wait to get stable and further extended before being merged? It appears to me that there are reasons for one and the other. However, given that we plan to drop support for (at least) Python 2.3 after this release, I feel that we should try to get out a version that doesn't require much fixing and maintenance. Changing the implementation of functions at this point seems like a rather deep cut. If we agree that it's in a state that we feel comfortable with, I wouldn't object to merging it. Otherwise, I'd suggest to wait until after the release. Comments? Stefan

5 9

Yet another Python to C compiler
by Vitja Makarov Aug. 12, 2011

Aug. 12, 2011

Hi! Recently I've found one more Python to C compiler, that translates python bytecode into C source. And author says about 100% Python compatibility. The project is a signle 800KB python file. http://code.google.com/p/2c-python/ I was wondering when found that 2c beats Cython in some benchmarks. For instance, it's about 2 times faster than Cython in pystone test (pure python mode). Think we should investigate performance differences and make cython faster. -- vitja.

2 2

Re: [Cython] [cython-users] Calling gil-requiring function not allowed without gil
by Stefan Behnel Aug. 11, 2011

Aug. 11, 2011

[moving this away from cython-users] Dag Sverre Seljebotn, 11.08.2011 14:53: > On 08/11/2011 02:13 PM, Stefan Behnel wrote: >> Dag Sverre Seljebotn, 11.08.2011 13:58: >>> On 08/11/2011 01:43 PM, Ting Zhou wrote: >>>> here is my pyx file. When I compile it, it says "Calling gil-requiring >>>> function not allowed without gil" at calling dabs(x[i]). Why my dabs >>>> function is a gil-requiring function? >>>> >>>> ==================================== >>>> from cython.parallel import * >>>> import numpy as np >>>> cimport numpy as np >>>> cimport cython >>>> >>>> cdef double dabs(double x): >>> >>> This should be >>> >>> cdef double dabs(double x) nogil: >> >> Note that Cython cannot infer this automatically. Even a trivial >> function may require an exclusive global lock for some reason, and it's >> common to use the GIL for that. So the programmer must be explicit here. > > Are you still against this mini-CEP?: > > with cython.global_lock(): > ... > > Where global_lock() is GIL-requiring noop. > > This > > a) Improves code readability vastly. Having a critical section take effect > because of the *lack* of a keyword is just very odd to anyone who's not > shoulder deep in CPython internals The GIL is more than a critical section. It's just there - and that's a good thing. Just take it as a part of the runtime environment, and disable it when you are sure you can *safely* do without it. A global lock makes life a lot easier, as it prevents unconditional (and unexpected) concurrency. > b) Allows inferring 'nogil' No it doesn't, because existing code doesn't use it. Only because you use one little statement in one part of your sources doesn't mean you know what you're doing, and that Cython can happily assume reversed semantics for the rest. Don't forget that threading is a dangerous abstraction, hard to get right and likely even the most error prone concurrency mechanism in widespread use. Not everything is "trivially parallelisable", and even those cases that are still produce enough problems. Actually, it's worth asking if even the prange loop was a good idea, given the amount of confusion driven traffic that it currently produces on the mailing list. I'm not arguing against it, but it seems to be a trickier concept than it appears at first sight. IMHO, that's worth working on *before* we introduce even more hard-to-understand concepts and hard-to-control compiler trickery. Stefan

2 1

CmdLine.py refactoring
by Stefano Aug. 9, 2011

Aug. 9, 2011

Hi, I've cloned GitHub repo and started working on it. First, I'm rewriting Compiler/CmdLine.py to use the optparse/argparse module. Then I'll try to address the Bugzilla ticket mentioned in the source file. Question: optparse has been deprecated since 2.7 (which adopts optparse module). So, which one to choose? Old one, or new one? Or should I work on a fallback system? --SKS

4 9

CommandLine.py refactoring
by Stefano Aug. 7, 2011

Aug. 7, 2011

> Then I'll try to address the Bugzilla ticket mentioned in the source file. Well, I meant Trac ticket. --SKS

1 0

Cython 0.15 released
by robertwb Aug. 5, 2011

Aug. 5, 2011

We are excited to announce the release of Cython 0.15, which is a huge step forward in achieving full Python language coverage as well as many new features, optimizations, and bugfixes. Download: http://cython.org/ or http://pypi.python.org/pypi/Cython == Major Features == * Generators (yield) - Cython has full support for generators, generator expressions and coroutines. http://www.python.org/dev/peps/pep-0342/ * The nonlocal keyword is supported. * Re-acquiring the gil: with gil - works as expected within a nogil context. * OpenMP support: http://docs.cython.org/0.15/src/userguide/parallelism.html * Control flow analysis prunes dead code and emits warnings and errors about uninitialised variables. * Debugger command cy set to assign values of expressions to Cython variables and cy exec counterpart cy_eval(). * Exception chaining http://www.python.org/dev/peps/pep-3134/ * Relative imports http://www.python.org/dev/peps/pep-0328/ * The with statement has its own dedicated and faster C implementation. * Improved pure syntax including cython.cclass, cython.cfunc, and cython.ccall. http://docs.cython.org/0.15/src/tutorial/pure.html * Support for del. * Boundschecking directives implemented for builtin Python sequence types. * Several updates and additions to the shipped standard library pxd files https://github.com/cython/cython/tree/master/Cython/Includes * Forward declaration of types is no longer required for circular references. Note: this will be the last release to support Python 2.3; Python 2.4 will be supported for at least one more release. == General improvements and bug fixes == This release contains over a thousand commits including hundreds of bugfixes and optimizations. The bug tracker has not been as heavily used this release cycle, but is still useful http://trac.cython.org/cython_trac/query?status=closed&group=component&orde… == Incompatible changes == * Uninitialized variables are no longer initialized to None and accessing them has the same semantics as standard Python. * globals() now returns a read-only dict of the Cython module's globals, rather than the globals of the first non-Cython module in the stack * Many C++ exceptions are now special cases to give closer Python counterparts. This means that except+ functions that formally raised generic RuntimeErrors may raise something else such as ArithmaticError. == Known regressions == * The inlined generator expressions (introduced in Cython 0.13) were disabled in favour of full generator expression support. This induces a performance regression for cases that were previously inlined. == Contributors == Many thanks to: Francesc Alted, Haoyu Bai, Stefan Behnel, Robert Bradshaw, Lars Buitinck, Lisandro Dalcin, John Ehresman, Mark Florisson, Christoph Gohlke, Jason Grout, Chris Lasher, Vitja Makarov, Brent Pedersen, Dag Sverre Seljebotn, Nathaniel Smith, and Pauli Virtanen

1 0

object to struct conversion
by mark florisson Aug. 5, 2011

Aug. 5, 2011

Is there any specific reason objects cannot coerce to structs (from e.g. dicts?). It would be convenient for memoryviews, then you could assign dicts (or any mapping) to items in the memoryview from Python space. You could also have structs as argument to def functions etc. Any objection to this addition? The implementation is pretty straightforward.

3 7

Ctypes object <-> pointer conversion.
by Robert Bradshaw Aug. 5, 2011

Aug. 5, 2011

This is getting a bit OT but is worth discussing, so I'm starting a new thread. On Thu, Aug 4, 2011 at 2:51 PM, Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: > Dag Sverre Seljebotn wrote: > >> - One idea is coercion of C pointers to ctypes Python objects and back >> again. > > Some way of requesting this manually might be useful, but > I don't think I'd like it to happen automatically. Slinging > raw pointers around in Python isn't something to be done > lightly -- even if all the code that dereferences it is > in Cython, there are problems with managing the lifetime > of whatever it references. You know, C has the same problem with slinging around raw pointers and managing their lifetimes :). Of course, with C, the user is all to painfully aware of the situation. I think this would be most useful for providing data from Python (even an interactive prompt) to a Cython module. I agree about it being dangerous to do implicitly, but something explicit like def entry_point(ctypes.CData x): cdef double* result = func(<int*>x) return <ctypes.CData>result # or cytpes.CData(result, option=...) to control deallocation could be really nice. Maybe CData could even be parameterized. - Robert

2 2

Willing to contribute
by Stefano Aug. 4, 2011

Aug. 4, 2011

Hi everybody, I've been a enthusiast Python programmer for 3 years now, and I've hailed Cython project with great interest. Now, I'm willing to contribute. I've would say I'm an quite an expert Python programmer and I've some-year-long experience with C, modern C++ and its cousin Java. Sidenote, I'm a student at Politecnico di Milano, Italy. Everything started with trying to speed up a Mandelbrot generator, which heavily relies on complex numbers. Eventually I've found in "Mini Projects (or something similar)" wiki that C99 `complex' is still to be implemented. So, I decide to make a step further and try to hop into development. I've would be grateful if a core developer could point me in the right direction concerning coding style, repositories, deadlines (I'm still a student, so I may have 'offline' days this winter) and other issues. I may start with something easy (`complex' implementation, or command line API restructuration), and I'd like someone from the team to discuss what I'll be implementing (ok, just for the first days), just to see if "I'm doing it right" (TM) and, mostly, if I'm the right man or I'm just making you loose your time. You can see an example of what I did in the past here: http://code.google.com/p/libsmth/ - for instance (still a WIP). I really appreciate your work :) --SKS

6 5