Mailman 3 July 2012 - cython-devel

Odd behavior with std::string and .decode()
by Barry Warsaw July 19, 2012

July 19, 2012

I'm currently exploring using Cython to provide new Python 3 bindings for Xapian. I'm pretty much a Cython n00b but the documentation is great, and I was able to pretty quickly get something really simple working. I'm using Cython 0.15 on Ubuntu 12.04 with Python 3.2 and Xapian 1.2.12. I've pushed my current branch to github: https://github.com/warsaw/xapian/tree/py3/xapian-bindings/python3 There you'll see my xapianlib.pxd and xapian.pyx files. Where I'm seeing some odd behavior is in trying to expose the Xapian::TermGenerator.get_description() method. This returns a std::string and I'm trying to create a `description` property that coerces this to unicode before returning it to Python. Here's the relevant code: -----snip snip----- cdef class TermGenerator: cdef xapianlib.TermGenerator * _this def __cinit__(self): self._this = new xapianlib.TermGenerator() def __dealloc__(self): del self._this property description: def __get__(self): as_bytes = <char *>self._this.get_description().c_str() #return as_bytes return as_bytes.decode('utf-8') -----snip snip----- I'm sure I'm doing something naive or stupid, but the problem is that as written above, .description is returning nonsense. % python Python 3.2.3 (default, May 3 2012, 15:51:42) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import xapian >>> tg = xapian.TermGenerator() >>> tg.description '\x00\x00\x00\x00_\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' If instead, I return just the bytes object (i.e. what .get_description().c_str() returns), then I get more like what I expect. % python Python 3.2.3 (default, May 3 2012, 15:51:42) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import xapian >>> tg = xapian.TermGenerator() >>> tg.description b'Xapian::TermGenerator(stem=Xapian::Stem(none), doc=Document(Xapian::Document::Internal()), termpos=0)' >>> tg.description.decode('utf-8') 'Xapian::TermGenerator(stem=Xapian::Stem(none), doc=Document(Xapian::Document::Internal()), termpos=0)' I looked at the generated code in the first example, but didn't really see anything obvious. There are no NULs in the char* description afaict. I haven't yet tested Cython 0.16 or 0.17 to see if this behaves differently. Is this a bug or am I doing something stupid? Cheers, -Barry

2 4

known? 0.16 on varioius platforms: ValueError: Buffer dtype mismatch, expected 'char' but got 'unsigned char' in ...
by Yaroslav Halchenko July 19, 2012

July 19, 2012

Hi everyone! 1. uploaded 0.16 to Debian experimental... many platforms fail with smth like ,-- | ====================================================================== | FAIL: Doctest: memslice.__test__.test_padded_structs | ---------------------------------------------------------------------- | Traceback (most recent call last): | File "/usr/lib/python2.6/doctest.py", line 2163, in runTest | raise self.failureException(self.format_failure(new.getvalue())) | AssertionError: Failed doctest test for memslice.__test__.test_padded_structs | File "/build/buildd-cython_0.16-1-armhf-FpFrv7/cython-0.16/build/work-dir/run/cpp/memslice.so", line unknown line number, in test_padded_structs | | ---------------------------------------------------------------------- | File "/build/buildd-cython_0.16-1-armhf-FpFrv7/cython-0.16/build/work-dir/run/cpp/memslice.so", line ?, in memslice.__test__.test_padded_structs | Failed example: | test_padded_structs() | Exception raised: | Traceback (most recent call last): | File "/usr/lib/python2.6/doctest.py", line 1253, in __run | compileflags, 1) in test.globs | File "<doctest memslice.__test__.test_padded_structs[0]>", line 1, in <module> | test_padded_structs() | File "memslice.pyx", line 42, in memslice.testcase.wrapper (memslice.cpp:3729) | File "memslice.pyx", line 1766, in memslice.test_padded_structs (memslice.cpp:27137) | File "memslice.pyx", line 1779, in memslice._test_padded (memslice.cpp:27284) | ValueError: Buffer dtype mismatch, expected 'char' but got 'unsigned char' in 'ArrayStruct.chars' `--- more here: https://buildd.debian.org/status/package.php?p=cython&suite=experimental I have tested on a sparc box and current master seems to not show this particular problem (see below though) -- does anyone remember this particular issue and commit which fixed it? 2. related question -- what is the way with runtests.py to run a particular unittest? not the full suite (e.g. my specifying memslice in cmdline), but this particular one, e.g.: ,--- | (sid)yoh@vagus:~/deb/cython/cython$ PYTHONPATH=$PWD/install/lib/python2.7/site-packages/ python runtests.py --no-refnanny -v -v --exclude="parallel" --work-dir=build/work-dir memslice.__test__.test_padded_structs | Python 2.7.3 (default, Jun 18 2012, 19:31:31) | [GCC 4.6.3] | | Running tests against Cython 0.17pre b0a428d4d438b1ce3e1f5d805b09899377d1ecb5 | Backends: c,cpp | | | ---------------------------------------------------------------------- | Ran 0 tests in 0.001s | | OK | ALL DONE `--- so I could bisect reasonably fast whenever needed... ? 3. FWIW there are some similar and novel failures on a sparc box on current master: http://www.onerussian.com/tmp/tests-cython-sparc-0.16rc1-490-gb0a428d.log I wondered -- do you have any brief instructions on how to setup a local build bot to contribute back to the cython's Jenkins setup? Then I could put a sparc to provide you with CI updates for this platform -- or do you have a similar one already? -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik

1 1

2d buffer interface with aligned data
by Christian Heimes July 18, 2012

July 18, 2012

Hello, I'm the author of https://bitbucket.org/tiran/smc.freeimage , a Cython wrapper of the FreeImage and LCMS libraries. FreeImage supports a broad variety of image formats and pixel formats from standard RGB up to RGBAF and complex numbers. Now I like to add support for buffer interface to my code to support zero-copy access of pixel data from NumPy. The wiki describes several ways to access objects through the new and old Python buffer interface. However I'm unable to find an example how to *implement* the buffer interface. Memory alignment makes the buffer interface more complicated, too. FreeImage stores the raw pixels non-contiguously. The lines are memory aligned (usually 32bit). For example a standard RGB image with 1 byte per pixel and size of 2*3 pixels has a memory layout of BGRBGRxxBGRBGRxxBGRBGRxx on a little-endian machine. 'xx' are the two alignment bytes. Here is an excerpt from the FreeImage manual on how to access the raw pixel data. The above example has width: 2, height: 3, pitch: 8 and bpp: 3. unsigned width = FreeImage_GetWidth(dib); // 2 unsigned height = FreeImage_GetHeight(dib); // 3 unsigned pitch = FreeImage_GetPitch(dib); // 8 unsigned bpp = FreeImage_GetBPP(dib); // 3 BYTE *bits = (BYTE*)FreeImage_GetBits(dib); for(y = 0; y < height; y++) { BYTE *pixel = (BYTE*)bits; for(x = 0; x < width; x++) { pixel[FI_RGBA_RED] = 128; pixel[FI_RGBA_GREEN] = 128; pixel[FI_RGBA_BLUE] = 128; pixel += bpp; } // next line bits += pitch; } Please help, Christian

4 8

Local type inference, first success
by Vitja Makarov July 13, 2012

July 13, 2012

Hi! I've been a little bit busy last months, now I'm back. I made first steps on implementing local type inference, TI for assignments not only for their names. Here is sample program that works: (_entry_split) vitja@mchome:~/work/cython-vitek-git/zzz$ cat tf.pyx # cython: infer_types.verbose=True from cython cimport typeof def foo(c): a = "abc" print a, typeof(a) if c: a = 123 else: a = 1.123 print a, typeof(a) (_entry_split) vitja@mchome:~/work/cython-vitek-git/zzz$ make tf.c python ../cython.py -v tf.pyx -o tf.c Compiling /home/vitja/work/cython-vitek-git/zzz/tf.pyx Gonna split entries: entry: Entry(name=a, type=<unspecified>) ... group: set([NameAssignment(entry=Entry(name=a, type=<unspecified>)), NameAssignment(entry=Entry(name=a, type=<unspecified>))]) ... ... (<FileSourceDescriptor:/home/vitja/work/cython-vitek-git/zzz/tf.pyx>, 8, 10) ... ... (<FileSourceDescriptor:/home/vitja/work/cython-vitek-git/zzz/tf.pyx>, 10, 10) ... group: set([NameAssignment(entry=Entry(name=a, type=<unspecified>))]) ... ... (<FileSourceDescriptor:/home/vitja/work/cython-vitek-git/zzz/tf.pyx>, 5, 6) note: tf.pyx:5:6: inferred 'a.#1' to be of type 'Python object' note: tf.pyx:5:6: inferred 'a.#0' to be of type 'double' -- vitja.

1 1

errors in C++ tests
by Stefan Behnel July 8, 2012

July 8, 2012

Hi, I'm seeing C++ compiler errors errors in the C++ tests. https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests/BACKEND… """ cpp_operators.cpp: In function ‘void __Pyx_CppExn2PyErr()’: cpp_operators.cpp:442: error: expected unqualified-id before ‘&’ token cpp_operators.cpp:442: error: expected `)' before ‘&’ token """ The failing code line is this: """ } catch (const std::bad_alloc& exn) { """ That line looks ok to me. It started failing after I enabled C++ error handling for default constructors, which then lead to the above function being generated into the module code. Does anyone know what to make of this? Stefan

2 6

Higher fidelity translations from C++ exception to Python exception?
by Barry Warsaw July 7, 2012

July 7, 2012

So if you have code like the following: cdef class Database: cdef open(self, path) except +raise_py_error: something_that_can_throw_a_cpp_exception(path) you can write cdef int raise_py_error(): raise Something to kind of turn a C++ exception into a Python exception. The problem appears to be that you cannot include in the Python exception any information contained in the C++ exception, because there's no access to the C++ exception that was thrown. The generated code does a `catch(...)` so you lose that useful information. AFAICT, there's no way to find out within the catch clause (or anything called by that clause) what C++ exception occurred. It would sure be useful if raise_py_error() was passed the exception instance that was caught. Then at least our handler could extract some useful information to relay to the Python level. E.g. in the case above, it might tell us that the `path` is nonexistent. Cheers, -Barry

4 4

Cython+PyPy benchmarks
by Stefan Behnel July 5, 2012

July 5, 2012

Hi, I set up a Jenkins job to run a couple of (simple) benchmarks comparing Cython's current performance under CPython and PyPy. Note that these are C-API intensive benchmarks by design. https://sage.math.washington.edu:8091/hudson/job/cython-devel-cybenchmarks-… Basically, PyPy's cpyext is currently about 100-200x slower than CPython's native C-API for these kinds of benchmarks. That's because it hasn't been optimised in any way, correctness and completeness are still the main goals in its development (and they're not there yet). The one major performance issue in cpyext is currently the creation and deallocation of the PyObject representation for each object, which obviously has a huge impact on everything. I profiled the nbody benchmark and it showed that almost 80% of the runtime is currently spent in creating and discarding PyObject instances. Here's the call graph: http://cython.org/callgrind-pypy-nbody.png The up side of this is that there is likely a lot of low hanging fruit in cpyext (plus some more tweaks in Cython), given that no optimisation at all has been done so far. It shouldn't be too hard to drop the factor substantially. I also think we should add a couple of more C-ish benchmarks to see how much overhead there really is for less C-API intensive code. Stefan

1 0

Cython methods for C/C++ types
by Stefan Behnel July 3, 2012

July 3, 2012

Hi, the discussion on allowing for automatic user provided ways to coerce between Python types and C++ types got me thinking that this might hide a rather interesting general feature: methods for low-level types. I faintly remember that this idea has come up in our discussions before, but here's a draft CEP for it: http://wiki.cython.org/enhancements/ctypemethods Basically, it would allow writing this in .pxd files: """ cdef extern from "...": cdef cppclass MyClass: cdef int cython_method_here(self): return 1 ctypedef double mydouble: cdef double cython_method_here(self): return self ** 2 cdef struct mystruct: int x, y cdef int cython_method_here(self): return self.x + self.y cdef union myunion: int a double b cdef int cython_method_here(self): return self.a if ImSureImAnInt else <int>self.b """ The C code for these methods would then be generated into the modules that use these types, similar to what we allow with "__getbuffer__()". Calls would be direct function calls as with "final" methods. I think it fits into what's there in a very natural way. Stefan

2 3

e: Automatic C++ conversions
by Yaroslav Halchenko July 3, 2012

July 3, 2012

I was thinking about updating Debian package for cython 0.16 but ran into the failing unittests so decided first to check with experts: anything obvious which comes to mind from seeing those? ... compiling (c) and running sequential_parallel ... sequential_parallel.c: In function '__pyx_pf_19sequential_parallel_56test_chunksize': sequential_parallel.c:12405:7: warning: variable '__pyx_t_6' set but not used [-Wunused-but-set-variable] sequential_parallel.c:12404:7: warning: variable '__pyx_t_5' set but not used [-Wunused-but-set-variable] sequential_parallel.c:12402:7: warning: variable '__pyx_t_3' set but not used [-Wunused-but-set-variable] sequential_parallel.c: In function '__pyx_pw_19sequential_parallel_25test_nested_break_continue': sequential_parallel.c:5887:29: warning: '__pyx_v_j' may be used uninitialized in this function [-Wmaybe-uninitialized] sequential_parallel.c:5590:7: note: '__pyx_v_j' was declared here sequential_parallel.c: In function '__pyx_f_19sequential_parallel_parallel_exc_cpdef_unnested.isra.29': sequential_parallel.c:8915:3: warning: '__pyx_r' may be used uninitialized in this function [-Wmaybe-uninitialized] sequential_parallel.c:8681:7: note: '__pyx_r' was declared here sequential_parallel.c: In function '__pyx_f_19sequential_parallel_parallel_exc_cpdef.isra.30': sequential_parallel.c:8614:3: warning: '__pyx_r' may be used uninitialized in this function [-Wmaybe-uninitialized] sequential_parallel.c:8259:7: note: '__pyx_r' was declared here sequential_parallel.c: In function '__pyx_pw_19sequential_parallel_33test_parallel_exc_cdef': sequential_parallel.c:8222:78: warning: '__pyx_r' may be used uninitialized in this function [-Wuninitialized] sequential_parallel.c:7946:7: note: '__pyx_r' was declared here sequential_parallel.c:8233:69: warning: '__pyx_r' may be used uninitialized in this function [-Wuninitialized] sequential_parallel.c:7578:7: note: '__pyx_r' was declared here Fatal Python error: deletion of interned string failed Compiling /tmp/buildd/.cython/inline/_cython_inline_a998a12d3d1450b2368ca75e07a27942.pyx because it changed. Cythonizing /tmp/buildd/.cython/inline/_cython_inline_a998a12d3d1450b2368ca75e07a27942.pyx ERROR compiling (cpp) and running sequential_parallel ... sequential_parallel.cpp: In function 'PyObject* __pyx_pf_19sequential_parallel_56test_chunksize(PyObject*)': sequential_parallel.cpp:12402:7: warning: variable '__pyx_t_3' set but not used [-Wunused-but-set-variable] sequential_parallel.cpp:12404:7: warning: variable '__pyx_t_5' set but not used [-Wunused-but-set-variable] sequential_parallel.cpp:12405:7: warning: variable '__pyx_t_6' set but not used [-Wunused-but-set-variable] sequential_parallel.cpp: In function 'PyObject* __pyx_pw_19sequential_parallel_25test_nested_break_continue(PyObject*, PyObject*)': sequential_parallel.cpp:5887:39: warning: '__pyx_v_j' may be used uninitialized in this function [-Wmaybe-uninitialized] sequential_parallel.cpp:5590:7: note: '__pyx_v_j' was declared here sequential_parallel.cpp: In function 'int _ZL48__pyx_f_19sequential_parallel_parallel_exc_cpdefi.isra.30()': sequential_parallel.cpp:8614:10: warning: '__pyx_r' may be used uninitialized in this function [-Wmaybe-uninitialized] sequential_parallel.cpp:8259:7: note: '__pyx_r' was declared here sequential_parallel.cpp: In function 'int _ZL57__pyx_f_19sequential_parallel_parallel_exc_cpdef_unnestedi.isra.29()': sequential_parallel.cpp:8915:10: warning: '__pyx_r' may be used uninitialized in this function [-Wmaybe-uninitialized] sequential_parallel.cpp:8681:7: note: '__pyx_r' was declared here sequential_parallel.cpp: In function 'PyObject* __pyx_pw_19sequential_parallel_33test_parallel_exc_cdef(PyObject*, PyObject*)': sequential_parallel.cpp:8222:75: warning: '__pyx_r' may be used uninitialized in this function [-Wuninitialized] sequential_parallel.cpp:7946:7: note: '__pyx_r' was declared here sequential_parallel.cpp:8233:66: warning: '__pyx_r' may be used uninitialized in this function [-Wuninitialized] sequential_parallel.cpp:7578:7: note: '__pyx_r' was declared here ERROR ... ====================================================================== ERROR: compiling (c) and running parallel ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 707, in run self.run_tests(result) File "runtests.py", line 723, in run_tests self.run_doctests(self.module, result) File "runtests.py", line 729, in run_doctests run_forked_test(result, run_test, self.shortDescription(), self.fork) File "runtests.py", line 778, in run_forked_test (module_name, result_code & 255)) Exception: Tests in module 'parallel' were unexpectedly killed by signal 11 ====================================================================== ERROR: compiling (cpp) and running parallel ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 707, in run self.run_tests(result) File "runtests.py", line 723, in run_tests self.run_doctests(self.module, result) File "runtests.py", line 729, in run_doctests run_forked_test(result, run_test, self.shortDescription(), self.fork) File "runtests.py", line 778, in run_forked_test (module_name, result_code & 255)) Exception: Tests in module 'parallel' were unexpectedly killed by signal 11 ====================================================================== ERROR: compiling (c) and running sequential_parallel ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 707, in run self.run_tests(result) File "runtests.py", line 723, in run_tests self.run_doctests(self.module, result) File "runtests.py", line 729, in run_doctests run_forked_test(result, run_test, self.shortDescription(), self.fork) File "runtests.py", line 778, in run_forked_test (module_name, result_code & 255)) Exception: Tests in module 'sequential_parallel' were unexpectedly killed by signal 6 ====================================================================== ERROR: compiling (cpp) and running sequential_parallel ---------------------------------------------------------------------- Traceback (most recent call last): File "runtests.py", line 707, in run self.run_tests(result) File "runtests.py", line 723, in run_tests self.run_doctests(self.module, result) File "runtests.py", line 729, in run_doctests run_forked_test(result, run_test, self.shortDescription(), self.fork) File "runtests.py", line 778, in run_forked_test (module_name, result_code & 255)) Exception: Tests in module 'sequential_parallel' were unexpectedly killed by signal 11 ---------------------------------------------------------------------- Ran 6358 tests in 954.316s FAILED (errors=4) -- Yaroslav O. Halchenko Postdoctoral Fellow, Department of Psychological and Brain Sciences Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik

3 3

Automatic C++ conversions
by Robert Bradshaw July 2, 2012

July 2, 2012

I've been looking how painful it is to constantly convert between Python objects and string in C++. Yes, it's easy to write a utility, but this should be as natural (if not more so, as the length is explicit) than bytes <-> char*. Several other of the libcpp classes (vector, map) have natural Python analogues too. What would people think about making it possible to declare these in a C++ file? Being able to make arbitrary mappings anywhere between types is contextless global state that I'd rather avoid, but perhaps special methods defined on the class such as cdef extern from "<string>" namespace "std": cdef cppclass string: def __object__(sting s): return s.c_str()[s.size()] def __create__(object o): return string(<char*>o, len(o)) ... (names open to suggestions) Then one could write cdef extern from *: string c_func(string) def f(x): return c_func(x) - Robert

3 9