Mailman 3 June 2012 - cython-devel

Re: [Cython] Bringing Cython and PyPy closer together
by Stefan Behnel March 29, 2015

March 29, 2015

[copied here from PyPy mailing list] Stefan Behnel, 15.02.2012 12:32: > The current state of the discussion seems to be that PyPy provides ways to > talk to C code, but nothing as complete as CPython's C-API in the sense > that it allows efficient two-way communication between C code and Python > objects. Thus, we need to either improve this or look for alternatives. > > In order to get us more focussed on what can be done and what the > implications are, so that we may eventually be able to decide what should > be done, I started a Wiki page for a PyPy backend CEP (Cython Enhancement > Proposal). > > http://wiki.cython.org/enhancements/pypy The discussion so far makes me rather certain that the most promising short-term solution is to make Cython generate C code that PyPy's cpyext can handle. This should get us a rather broad set of running code somewhat quickly, while requiring the least design-from-scratch type of work in a direction that does not yet allow us to see if it will really make existing code work or not. On top of the basic cpyext interface, it should then be easy to implement obvious optimisations like native C level calls to Cython wrapped functions from PyPy (and potentially also the other direction) and otherwise avoid boxing/unboxing where unnecessary, e.g. for builtins. After all, it all boils down to native code at some point and I'm sure there are various ways to exploit that. Also, going this route will help both projects to get to know each other better. I think that's a required basis if we really aim for designing a more high-level interface at some point. The first steps I see are: - get Cython's test suite to run on PyPy - analyse the failing tests and decide how to fix them - adapt the Cython generated C code accordingly, special casing for PyPy where required Here is a "getting started" guide that tells you how testing works in Cython: http://wiki.cython.org/HackerGuide Once we have the test suite runnable, we can set up a PyPy instance on our CI server to get feed-back on any advances. https://sage.math.washington.edu:8091/hudson/ So, any volunteers or otherwise interested parties to help in getting this to work? Anyone in for financial support? Stefan

2 9

planning for 0.17
by Stefan Behnel July 22, 2012

July 22, 2012

Hi, I'd like to get an idea of what's still open for 0.17. Mark mentioned some open memoryview issues on his list and I know that there are still issues with PyPy, some of which could get fixed in a reasonable time frame. Also, Jenkins isn't all that happy yet. https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/ What's the current state of the master branch for everyone? Anything that you're working on and/or that you think should go in but isn't yet? I would like to see 0.17 released some time next month, if possible. I don't currently see any real blockers, so that might be doable. The release notes look ok so far, but the bug tracker list is really short in comparison. Please add to both as you see fit. http://wiki.cython.org/ReleaseNotes-0.17 http://trac.cython.org/cython_trac/query?status=closed&group=component&orde… Stefan

3 11

Re: [Cython] [cython-users] C++: how to handle failures of 'new'?
by Stefan Behnel July 19, 2012

July 19, 2012

[moving this to cython-devel as it's getting technical] Robert Bradshaw, 28.06.2012 21:46: > On Thu, Jun 28, 2012 at 11:38 AM, Stefan Behnel wrote: >> currently, when I write "new CppClass()" in Cython, it generates a straight >> call to the "new" operator. It doesn't do any error handling. And the >> current documentation doesn't even mention this case. >> >> Is there a "standard" way to handle this? It seems that C++ has different >> ways to deal with failures here but raises an exception by default. Would >> you declare the constructor(s) with an "except +MemoryError"? Is there a >> reason Cython shouldn't be doing this automatically (if nothing else was >> declared) ? > > I think it certainly makes sense to declare the default constructor as > "except +" (and std::bad_alloc should become MemoryError), Right. The code in the constructor can raise other exceptions that must also be handled properly. An explicit "except +" will handle that. > but whether > to implicitly annotate declared constructors is less clear, especially > as there's no way to un-annotate them. I agree, but sadly, it's the default behaviour that is wrong. I'm sure we made lots of users run into this trap already. I fixed the documentation for now, but the bottom line is that we require users to take care of proper declarations themselves. Otherwise, the code that we generate is incorrect, although it's 100% certain that an allocation error can occur, even if the constructor code doesn't raise any exceptions itself. Apparently, changing the behaviour of the "new" operator requires a special annotation "std::nothrow", which then returns NULL on allocation failures. You can pass that from Cython by hacking up a cname, e.g. Rectangle "(std::nothrow) Rectangle" (int w, int h) I'm sure there are users out there who figured this out (I mean, I did...) and use it in their code, so I agree that this isn't easy to handle because Cython simply wouldn't know what the actual error behaviour is for a given constructor and how to correctly detect an error. This problem applies only to heap allocation in that form. However, stack allocation and the new exttype field allocation suffer from similar problems when the default constructor raises an exception. Exttype fields are a particularly nasty case because the user has no control over the allocation. A C++ exception in the C++ class constructor would terminate the exttype constructor unexpectedly and thus leak resources (in the best case - no idea how CPython reacts if you throw a C++ exception through its type instantiation code). Similarly, a C++ exception in the constructor of a stack allocated object would then originate from the function entry code and potentially hit the Python function wrapper etc. Again, potentially leaking resources or worse. To me, this sounds like we should do something about it. At least for the implicit calls to the default constructor, we should generate "except +" code automatically because there is no other way to handle them safely. Stefan

5 33

Automatic C++ conversions
by Robert Bradshaw July 2, 2012

July 2, 2012

I've been looking how painful it is to constantly convert between Python objects and string in C++. Yes, it's easy to write a utility, but this should be as natural (if not more so, as the length is explicit) than bytes <-> char*. Several other of the libcpp classes (vector, map) have natural Python analogues too. What would people think about making it possible to declare these in a C++ file? Being able to make arbitrary mappings anywhere between types is contextless global state that I'd rather avoid, but perhaps special methods defined on the class such as cdef extern from "<string>" namespace "std": cdef cppclass string: def __object__(sting s): return s.c_str()[s.size()] def __create__(object o): return string(<char*>o, len(o)) ... (names open to suggestions) Then one could write cdef extern from *: string c_func(string) def f(x): return c_func(x) - Robert

3 9

Hash-based vtables
by Dag Sverre Seljebotn June 30, 2012

June 30, 2012

Me and Robert had a long discussion on the NumFOCUS list about this already, but I figured it was better to continue it and provide more in-depth benchmark results here. It's basically a new idea of how to provide a vtable based on perfect hashing, which should be a lot simpler to implement than what I first imagined. I'll write down some context first, if you're familiar with this skip ahead a bit.. This means that you can do fast dispatches *without* the messy business of binding vtable slots at compile time. To be concrete, this might e.g. take the form def f(obj): obj.method(3.4) # try to find a vtable with "void method(double)" in it or, a more typed approach, # File A cdef class MyImpl: def double method(double x): return x * x # File B # Here we never know about MyImpl, hence "duck-typed" @cython.interface class MyIntf: def double method(double x): pass def f(MyIntf obj): # obj *can* be MyImpl instance, or whatever else that supports # that interface obj.method(3.4) Now, the idea to implement this is: a) Both caller and callee pre-hash name/argument string "mymethod:iidd" to 64 bits of hash data (probably lower 64 bits of md5) b) Callee (MyImpl) generates a vtable of its methods by *perfect* hashing. What you do is define a final hash fh as a function of the pre-hash ph, for instance fh = ((ph >> vtable.r1) ^ (ph >> vtable.r2) ^ (ph >> vtable.r3)) & vtable.m (Me and Robert are benchmarking different functions to use here.) By playing with r1, r2, r3, you have 64**3 choices of hash function, and will be able to pick a combination which gives *no* (or very few) collisions. c) Caller then combines the pre-hash generated at compile-time, with r1, r2, r3, m stored in the vtable header, in order to find the final location in the hash-table. The exciting thing is that in benchmark, the performance penalty is actually very slight over a C++-style v-table. (Of course you can cache a proper vtable, but the fact that you get so close without caring about caching means that this can be done much faster.) Back to my and Robert's discussion on benchmarks: I've uploaded benchmarks here: https://github.com/dagss/hashvtable/tree/master/dispatchbench I've changed the benchmark taking to give more robust numbers (at least for me), you want to look at the 'min' column. I changed the benchmark a bit so that it benchmarks a *callsite*. So we don't pass 'h' on the stack, but either a) looks it up in a global variable (default), or b) it's a compile-time constant (immediate in assembly) (compile with -DIMHASH). Similarly, the ID is either an "interned" global variable, or an immediate (-DIMID). The results are very different on my machine depending on this aspect. My conclusions: - Both three shifts with masking, two shifts with a "fallback slot" (allowing for a single collision), three shifts, two shifts with two masks allows for constructing good vtables. In the case of only two shifts, one colliding method gets the twoshift+fback performance and the rest gets the twoshift performance. - Performance is really more affected by whether hashes are immediates or global variables than the hash function. This is in contrast to the interning vs. key benchmarks -- so I think that if we looked up the vtable through PyTypeObject, rather than getting the vtable directly, the loads of the global variables could potentially be masked by that. - My conclusion: Just use lower bits of md5 *both* for the hashing and the ID-ing (don't bother with any interning), and compile the thing as a 64-bit immediate. This can cause crashes/stack smashes etc. if there's lower-64bit-of-md5 collisions, but a) the probability is incredibly small, b) it would only matter in situations that should cause an AttributeError anyway, c) if we really care, we can always use an interning-like mechanism to validate on module loading that its hashes doesn't collide with other hashes (and raise an exception "Congratulations, you've discovered a phenomenal md5 collision, get in touch with cython devs and we'll work around it right away"). The RTTI (i.e. the char*) is also put in there, but is not used for comparison and is not interned. At least, that's what I think we should do for duck-style vtables. Do we then go to all the pain of defining key-encoding, interning etc. just for SEP 201? Isn't it easier to just mandate a md5 dependency and be done with it? (After all, md5 usually comes with Python in the md5 and hashlib modules) direct: Early-binding index: Call slot 0 (C++-style vtable/function pointer) noshift: h & m1 oneshift: (h >> r1) & m1 twoshift: ((h >> r1) ^ (h >> r2)) & m1 twoshift+fback: hash doesn't threeshift: ((h >> r1) ^ (h >> r2) ^ (h >> r3)) & m1 doublemask: ((h >> r1) & m1) ^ ((h >> r2) & m2) doublemask2: ((h >> r1) & m1) ^ ((h & m2) >> r2) Default distutils build (-O2): ------------------------------ Hash globalvar, ids globalvar direct: min=5.38e-09 mean=5.45e-09 std=3.79e-11 val=1600000000.000000 index: min=5.38e-09 mean=5.44e-09 std=3.09e-11 val=1200000000.000000 noshift: min=5.99e-09 mean=6.14e-09 std=6.63e-11 val=1200000000.000000 oneshift: min=6.47e-09 mean=6.53e-09 std=3.21e-11 val=1200000000.000000 twoshift: min=7.00e-09 mean=7.08e-09 std=3.73e-11 val=1200000000.000000 twoshift+fback: min=7.54e-09 mean=7.61e-09 std=4.46e-11 val=1200000000.000000 threeshift: min=7.54e-09 mean=7.64e-09 std=3.79e-11 val=1200000000.000000 doublemask: min=7.56e-09 mean=7.64e-09 std=3.02e-11 val=1200000000.000000 doublemask2: min=7.55e-09 mean=7.62e-09 std=3.24e-11 val=1200000000.000000 hash immediate, ids globalvar direct: min=5.38e-09 mean=5.45e-09 std=3.87e-11 val=1600000000.000000 index: min=5.40e-09 mean=5.45e-09 std=2.92e-11 val=1200000000.000000 noshift: min=5.38e-09 mean=5.44e-09 std=3.48e-11 val=1200000000.000000 oneshift: min=5.90e-09 mean=5.99e-09 std=4.05e-11 val=1200000000.000000 twoshift: min=6.09e-09 mean=6.17e-09 std=3.52e-11 val=1200000000.000000 twoshift+fback: min=7.00e-09 mean=7.08e-09 std=3.64e-11 val=1200000000.000000 threeshift: min=6.47e-09 mean=6.55e-09 std=6.04e-11 val=1200000000.000000 doublemask: min=6.46e-09 mean=6.50e-09 std=3.37e-11 val=1200000000.000000 doublemask2: min=6.46e-09 mean=6.51e-09 std=3.04e-11 val=1200000000.000000 all immediate: direct: min=5.39e-09 mean=5.50e-09 std=5.22e-11 val=1600000000.000000 index: min=5.38e-09 mean=5.51e-09 std=6.25e-11 val=1200000000.000000 noshift: min=5.38e-09 mean=5.51e-09 std=6.90e-11 val=1200000000.000000 oneshift: min=5.40e-09 mean=5.51e-09 std=5.35e-11 val=1200000000.000000 twoshift: min=5.94e-09 mean=6.06e-09 std=5.91e-11 val=1200000000.000000 twoshift+fback: min=7.06e-09 mean=7.19e-09 std=5.39e-11 val=1200000000.000000 threeshift: min=5.96e-09 mean=6.07e-09 std=5.54e-11 val=1200000000.000000 doublemask: min=5.88e-09 mean=6.01e-09 std=6.06e-11 val=1200000000.000000 doublemask2: min=5.94e-09 mean=6.05e-09 std=6.16e-11 val=1200000000.000000 -O3 build --------- all globalvars: direct: min=1.61e-09 mean=1.63e-09 std=1.40e-11 val=1600000000.000000 index: min=5.38e-09 mean=5.43e-09 std=2.82e-11 val=1200000000.000000 noshift: min=6.04e-09 mean=6.13e-09 std=4.76e-11 val=1200000000.000000 oneshift: min=6.46e-09 mean=6.54e-09 std=3.82e-11 val=1200000000.000000 twoshift: min=7.01e-09 mean=7.06e-09 std=3.41e-11 val=1200000000.000000 twoshift+fback: min=7.57e-09 mean=7.64e-09 std=3.47e-11 val=1200000000.000000 threeshift: min=7.54e-09 mean=7.63e-09 std=4.17e-11 val=1200000000.000000 doublemask: min=7.54e-09 mean=7.61e-09 std=3.64e-11 val=1200000000.000000 doublemask2: min=7.55e-09 mean=7.63e-09 std=3.35e-11 val=1200000000.000000 hash immediate, ids globalvar: direct: min=1.61e-09 mean=1.66e-09 std=3.30e-11 val=1600000000.000000 index: min=5.40e-09 mean=5.50e-09 std=4.94e-11 val=1200000000.000000 noshift: min=5.38e-09 mean=5.49e-09 std=6.02e-11 val=1200000000.000000 oneshift: min=5.95e-09 mean=6.06e-09 std=6.64e-11 val=1200000000.000000 twoshift: min=5.96e-09 mean=6.13e-09 std=7.22e-11 val=1200000000.000000 twoshift+fback: min=7.02e-09 mean=7.18e-09 std=7.04e-11 val=1200000000.000000 threeshift: min=6.52e-09 mean=6.65e-09 std=6.43e-11 val=1200000000.000000 doublemask: min=6.50e-09 mean=6.62e-09 std=5.28e-11 val=1200000000.000000 doublemask2: min=6.52e-09 mean=6.63e-09 std=5.23e-11 val=1200000000.000000 all immediate: direct: min=1.61e-09 mean=1.62e-09 std=9.77e-12 val=1600000000.000000 index: min=5.38e-09 mean=5.39e-09 std=1.71e-11 val=1200000000.000000 noshift: min=5.38e-09 mean=5.40e-09 std=2.41e-11 val=1200000000.000000 oneshift: min=5.38e-09 mean=5.40e-09 std=1.81e-11 val=1200000000.000000 twoshift: min=5.92e-09 mean=5.93e-09 std=1.43e-11 val=1200000000.000000 twoshift+fback: min=7.00e-09 mean=7.01e-09 std=2.20e-11 val=1200000000.000000 threeshift: min=5.92e-09 mean=5.94e-09 std=1.99e-11 val=1200000000.000000 doublemask: min=5.79e-09 mean=5.82e-09 std=2.32e-11 val=1200000000.000000 doublemask2: min=5.92e-09 mean=5.94e-09 std=2.25e-11 val=1200000000.000000

5 54

Potential bug: hole in "C <-> Python" conversion
by Dieter Maurer June 29, 2012

June 29, 2012

I have cdef extern from *: ctypedef char const_unsigned_char "const unsigned char" cdef const_unsigned_char *c_data = data leads to "Cannot convert Python object to 'const_unsigned_char *'" while "cdef char *c_data = data" works. Should the "ctypedef char const_unsigned_char" not ensure that "char" and "const_unsigned_char" are used as synonyms? -- Dieter

2 4

Feature request: generate signature information for use by "inspect"
by Dieter Maurer June 28, 2012

June 28, 2012

Python's "inspect" module is a great help to get valuable information about a package. Many higher level tools (e.g. the "help" builtin and "pydoc") are based on it. I have just recognized a deficiency of "cython" generated modules with respect to "inspect" support: "inspect" cannot determine the signatures for Python functions defined in "Cython" source. I understand that this might be a limitation of Python's "C" interface. In this case, I suggest to enhance the function's docstring by signature information. I now transform manually my docstrings def <rv> <f>(<signature>): """<header> <detail> """ into: def <rv> <f>(<signature>): """<f>(<signature>) -> <rv>: <header> <detail> """ and would be happy to get something similar automatically. -- Dieter

2 2

new FFI library for Python
by Stefan Behnel June 25, 2012

June 25, 2012

Hi, the PyPy folks have come up with a new FFI library (called cffi) for CPython (and eventually PyPy, obviously). http://cffi.readthedocs.org/ It borrows from LuaJIT's FFI in that it parses C declarations at runtime. It then builds a C extension to access the external code, i.e. it requires a C compiler at runtime (when running in CPython). Just thought this might be interesting. Stefan

5 7

static type checking in Python
by Stefan Behnel June 25, 2012

June 25, 2012

Hi, there's some work going on regarding static type analysis and checking of Python programs, here's the mailing list for it: https://groups.google.com/group/python-static-type-checking I think this is somewhat related to Cython. After all, they are trying to figure out static type information from source code - although apparently rather in order to find bugs than to speed things up. But the one doesn't necessarily exclude the other. Stefan

1 0

new Jenkins setup
by Stefan Behnel June 23, 2012

June 23, 2012

Hi, I moved the Jenkins installation out of the USB disk and into my home directory. The USB disk has proven very fragile in the past, so this will make us more independent from reboots and disk failures. To keep the builds fast, the workspaces have moved into a ramdisk, which is limited to 20 GB. This is less than the previous directory size, so I changed the job configs to delete redundant data after the builds, namely the unpacked CPython directories and the Cython installation. Those are still available in the job workspaces in form of the archives that the build jobs copy over at the beginning. So, to reproduce test failures on the Jenkins server, you can just unpack them manually. Currently, we are way below the limit (<5GB), but the developer branches haven't been built yet. It looks like no job takes more than 1GB when it runs, so the 6 active jobs that we run in parallel will not take more than 6GB in total. That leaves some 14 GB for the resident jobs. Those tend to stay around 30-100MB each (the Cython matrix jobs are more like 30-50MB per configuration), so we can keep quite a lot of jobs in the ramdisk. I'll keep an eye on them from time to time, but I think we'll be fine with that for a while. Still, please take a bit of care when you make changes to build jobs that you do not leave unnecessarily large sets of redundant data lying around at the end. Anything that we clearly no longer need after the build and that can be deleted will keep space free for running jobs and things like ccache. Stefan

1 0