Mailman 3 February 2011 - cython-devel

Re: [Cython] Fwd: Re: Cython builds on various Debian platforms
by Lisandro Dalcin Feb. 19, 2011

Feb. 19, 2011

2011/2/18 Yaroslav Halchenko <lists(a)onerussian.com>: > > awesome! I will test it asap -- just let me know if there would be > another commit for above or I should tune up $HOME ;) > https://github.com/cython/cython/commit/475e9a085654252d5a274dab2118b746e8b… >> We are very near to be crystal clear... > > yeap > And now we should be crystal clear, finally! Could you confirm? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

2 4

Fwd: Re: Cython builds on various Debian platforms
by Stefan Behnel Feb. 18, 2011

Feb. 18, 2011

[forwarding to the list, from Yaroslav Halchenko] On Tue, 15 Feb 2011, Stefan Behnel wrote: > Cool. That looks good. As you mentioned, it doesn't include the test > runs yet, but it would be really great if you could get those > running. ok - added testing during build, so now need to assure that everything passes and all necessary dependencies are there. So, there is that failing one I have mentioned before: ,--- | ====================================================================== | FAIL: test_embed (__main__.EmbedTest) | ---------------------------------------------------------------------- | Traceback (most recent call last): | File "runtests.py", line 886, in test_embed | "make PYTHON='%s' LIBDIR1='%s' test > make.output" % (sys.executable, libdir)) == 0) | AssertionError `--- what could be done about it or should it be excluded? -- =------------------------------------------------------------------= Keep in touch www.onerussian.com Yaroslav Halchenko www.ohloh.net/accounts/yarikoptic

3 3

Re: [Cython] Fixing NumPy support for Python 3 (Stefan, please help!)
by Stefan Behnel Feb. 18, 2011

Feb. 18, 2011

Lisandro Dalcin, 18.02.2011 15:38: > On 18 February 2011 02:19, Stefan Behnel<stefan_ml(a)behnel.de> wrote: >> Lisandro Dalcin, 17.02.2011 17:24: >>> >>> >>> OK, I've found an alternative workaround. What do you think? >>> >>> diff --git a/Cython/Compiler/Interpreter.py >>> b/Cython/Compiler/Interpreter.py >>> index 83cb184..9fb5fe5 100644 >>> --- a/Cython/Compiler/Interpreter.py >>> +++ b/Cython/Compiler/Interpreter.py >>> @@ -6,6 +6,7 @@ For now this only covers parse tree to value conversion of >>> compile-time values. >>> """ >>> >>> +import sys >>> from Nodes import * >>> from ExprNodes import * >>> from Errors import CompileError >>> @@ -44,6 +45,10 @@ def interpret_compiletime_options(optlist, optdict, >>> type_env=None, type_args=()) >>> else: >>> raise CompileError(node.pos, "Type not allowed here.") >>> else: >>> + if (sys.version_info[0]>=3 and >>> + isinstance(node, StringNode) and >>> + node.unicode_value is not None): >>> + return (node.unicode_value, node.pos) >>> return (node.compile_time_value(empty_scope), node.pos) >>> >>> if optlist: >>> @@ -52,6 +57,7 @@ def interpret_compiletime_options(optlist, optdict, >>> type_env=None, type_args=()) >>> assert isinstance(optdict, DictNode) >>> new_optdict = {} >>> for item in optdict.key_value_pairs: >>> - new_optdict[item.key.value] = interpret(item.value, >>> item.key.value) >>> + new_key, dummy = interpret(item.key, None) >>> + new_optdict[new_key] = interpret(item.value, item.key.value) >>> optdict = new_optdict >>> return (optlist, new_optdict) >> >> This still isn't something that looks right. It just does the same thing at >> a different place. Actually, I'm not sure there is a way to "get it right". >> > > IIUC, the purpose of that code is to bring stuff from .pyx code to > Python world at Cython runtime, that code is trying to "coerce" a > DictNode object to a regular Python dict to be used later for > implementing compiler directives. So, if your pyx/pxd says > {"mode":"strided"} , and that is going to be compiler directive > options, you need to map StringNode -> __builtin__.str (and then use > node.value in Py3 or node.unicode_value in Py3). > > Stefan, look at numpy.pxd, you will see $cdef __cythonbufferdefaults__ > = {"mode": "strided"}$. W need to "convert" that declaration in a > regular Python dict to be used in Buffer.py, in such a way that > $"mode" in option_dict$ or $option_dict["mode"]$ work. Currently, in > Python 3, the key,value pair endup being bytes, then things fails... > > I still think that what I'm doing is not so bad. After all, I'm > hacking a method called interpret_compiletime_options(), the purpose > of its output is not code generation. Hmm, right. I guess that'll do then. Please go ahead and push this in. If it helps NumPy users in Py3, it's worth getting out rather sooner than later. Stefan

1 0

How to update python grammar in Windows?
by Thomson Feb. 18, 2011

Feb. 18, 2011

I tried to add a statement to Python, but I don't know how to update the grammar header files such as graminit.c&graminit.h after I updated the grammar file (Garmmar/Garmmar). It seems pgenmain.c could do this, but these is no pgen.exe in the build output of Visual Studio. How can I update the grammar files in Windows? Thanks, Thomson

2 1

python 2.7/3.x and numpy-dev (Dag, I need a quick comment)
by Lisandro Dalcin Feb. 18, 2011

Feb. 18, 2011

I'm working on a patch to get old, recent, and dev NumPy working in 2.7/3.x. So far, I had success, but I still have two failures like the one pasted below. Dag, could you elaborate a bit about the purpose of __Pyx_BufFmt_CheckString() ? It is just a validity check for pep 3118 format strings? Do you expect the failure below to be hard to fix? Just in case, the format string that triggers the failure is: >>> memoryview(np.zeros((1,), dtype=np.dtype('b,i', align=False))).format 'T{b:f0:=i:f1:}' ====================================================================== FAIL: numpy_test () Doctest: numpy_test ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/python/3.2/lib/python3.2/doctest.py", line 2113, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for numpy_test File "/u/dalcinl/Devel/Cython/cython/BUILD/run/c/numpy_test.cpython-32dm.so", line 1, in numpy_test ---------------------------------------------------------------------- File "/u/dalcinl/Devel/Cython/cython/BUILD/run/c/numpy_test.cpython-32dm.so", line 155, in numpy_test Failed example: print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) Exception raised: Traceback (most recent call last): File "/usr/local/python/3.2/lib/python3.2/doctest.py", line 1248, in __run compileflags, 1), test.globs) File "<doctest numpy_test[64]>", line 1, in <module> print(test_packed_align(np.zeros((1,), dtype=np.dtype('b,i', align=False)))) File "numpy_test.pyx", line 404, in numpy_test.test_packed_align (numpy_test.c:6367) ValueError: Buffer packing mode currently only allowed at beginning of format string (this is a defect) -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

2 1

del command
by Prashant Saxena Feb. 18, 2011

Feb. 18, 2011

Hello, python 2.6.6 Cython-0.14.1 xp, mac, linux When I am using "del" command in my scripts and compiling using pure python mode, I am getting an error: "Deletion of local or C global name not supported" There are couple of places where I have to delete an object. I am using a hack to solve this but I am not sure it's correct way. In a simple python script (delobj.py), I am writing a function: def deleteObject(obj): del obj I am importing this script in all the scripts I am compiling using cython and there I am using: delobj.deleteObject(obj) this is working fine and no errors. Is this is the correct way? Regards Prashant

2 1

Re: [Cython] Fixing NumPy support for Python 3 (Stefan, please help!)
by Stefan Behnel Feb. 18, 2011

Feb. 18, 2011

Lisandro Dalcin, 17.02.2011 17:24: > On 17 February 2011 12:16, Stefan Behnel wrote: >> Lisandro Dalcin, 17.02.2011 15:32: >>> >>> Stefan, what do you think about the patch below? This hunk is part of >>> a series of fixes required to get numpy-dev working under Python 3.2. >>> The root of the issue is that __cythonbufferdefaults__ keys&values >>> end-up being "bytes" (this coercion is triggered in Interpreter.py). >>> >>> >>> diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py >>> index 5b339da..b72deef 100755 >>> --- a/Cython/Compiler/ExprNodes.py >>> +++ b/Cython/Compiler/ExprNodes.py >>> @@ -12,6 +12,7 @@ cython.declare(error=object, warning=object, >>> warn_once=object, >>> Builtin=object, Symtab=object, Utils=object, >>> find_coercion_error >>> debug_disposal_code=object, debug_temp_alloc=object, >>> debug_coerc >>> >>> +import sys >>> import operator >>> >>> from Errors import error, warning, warn_once, InternalError, CompileError >>> @@ -1136,6 +1137,8 @@ class StringNode(PyConstNode): >>> return self.result_code >>> >>> def compile_time_value(self, env): >>> + if sys.version_info[0]>= 3 and self.unicode_value: >> >> You must use "self.unicode_value is not None" here, it may be the empty >> string. >> >>> + return self.unicode_value >>> return self.value >> >> Ok, that's a tricky one. Just because the compilation is running in Py3 >> doesn't mean that the correct compile time value is a Unicode string - we >> don't know what it'll be used for. > > OK, I've found an alternative workaround. What do you think? > > diff --git a/Cython/Compiler/Interpreter.py b/Cython/Compiler/Interpreter.py > index 83cb184..9fb5fe5 100644 > --- a/Cython/Compiler/Interpreter.py > +++ b/Cython/Compiler/Interpreter.py > @@ -6,6 +6,7 @@ For now this only covers parse tree to value conversion of > compile-time values. > """ > > +import sys > from Nodes import * > from ExprNodes import * > from Errors import CompileError > @@ -44,6 +45,10 @@ def interpret_compiletime_options(optlist, optdict, > type_env=None, type_args=()) > else: > raise CompileError(node.pos, "Type not allowed here.") > else: > + if (sys.version_info[0]>=3 and > + isinstance(node, StringNode) and > + node.unicode_value is not None): > + return (node.unicode_value, node.pos) > return (node.compile_time_value(empty_scope), node.pos) > > if optlist: > @@ -52,6 +57,7 @@ def interpret_compiletime_options(optlist, optdict, > type_env=None, type_args=()) > assert isinstance(optdict, DictNode) > new_optdict = {} > for item in optdict.key_value_pairs: > - new_optdict[item.key.value] = interpret(item.value, item.key.value) > + new_key, dummy = interpret(item.key, None) > + new_optdict[new_key] = interpret(item.value, item.key.value) > optdict = new_optdict > return (optlist, new_optdict) This still isn't something that looks right. It just does the same thing at a different place. Actually, I'm not sure there is a way to "get it right". >> Doing the above will do the wrong thing >> e.g. in this case: >> >> DEF const_x = "abc" >> cdef str x = const_x >> >> The problem is: it is broken already, returning self.value is wrong because >> it drops available type information by returning plain bytes instead of str. >> And simply returning self.unicode_value in Py3 doesn't fix that. > > I see... So the correct compile time value for StringNode should > depend on options.language_level, right? Hmmm. I'm not sure that would solve it. Py2 str has the property of changing type depending on the runtime environment. So this is actually independent of the language_level (-3 has easy semantics here). I mean, it won't even solve the problem at hand, because the code could still be Py2 but require a unicode string value because it gets *compiled* under Python 3. It shouldn't depend on the compile time environment at all, but the NumPy problem shows that it has to sometimes. I think we have to find a way to keep the double bytes/unicode string identity alive during runtime processing, up to the point where we can (or have to) decide what to make of it. Stefan

1 0

Python C-api ref count semantics
by Chris Colbert Feb. 18, 2011

Feb. 18, 2011

What is the rule of thumb when declaring functions from python's C-api when comes to ref counting? If I define a test case like so: cdef extern from "Python.h": object PyWeakref_NewRef(object, object) object PyWeakref_GET_OBJECT(object) class Foo(object): pass cdef class Test: cdef object obj cdef object wr def __init__(self): self.obj = Foo() self.wr = PyWeakref_NewRef(self.obj, None) def get_ref(self): return PyWeakref_GET_OBJECT(self.wr) I get these random python fatal errors: In [8]: %timeit -n 1000 t.get_ref() 1000 loops, best of 3: 224 ns per loop In [9]: %timeit -n 1000 t.get_ref() Fatal Python error: deallocating None Abort trap However, if I redefine the macro signature and getter function to this: from cpython cimport PyObject cdef extern from "Python.h": object PyWeakref_NewRef(object, object) PyObject* PyWeakref_GET_OBJECT(object) class Foo(object): pass cdef class Test: cdef object obj cdef object wr def __init__(self): self.obj = Foo() self.wr = PyWeakref_NewRef(self.obj, None) def clear_obj(self): self.obj = None def get_ref(self): return <object>PyWeakref_GET_OBJECT(self.wr) Then it runs without issue. I can other gather is has to due the incref/decref going on in the generated C code. Should be doing something on my end to manually manage ref counts when using the C-Api?

2 1

Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
by W. Trevor King Feb. 17, 2011

Feb. 17, 2011

This thread is coming over to cython-dev (and the new cython-devel) from cython-users because it turns out it will probably require chaning the Cython code. To get everyone who hasn't been following on cython-users up to speed, here's a summary of what I'm trying to do: That's what I was trying to give with this: On Wed, Feb 09, 2011 at 12:23:25PM -0500, W. Trevor King wrote: > I'm wrapping an external C library with Cython, so I have `mylib.pxd`: > > cdef extern from 'mylib.h' > enum: CONST_A > enum: CONST_B > ... > > where I declare each constant macro from the library's header `mylib.h`: > > #define CONST_A 1 > #define CONST_B 2 > ... > > Now I want to expose those constants in Python, so I have `expose.pyx`: > > cimport mylib > > CONST_A = mylib.CONST_A > CONST_B = mylib.CONST_B > ... > > But the last part seems pretty silly. I'd like to do something like > > cimport mylib > import sys > > for name in dir(mylib): > setattr(sys.modules[__name__], name, getattr(mylib, name)) > > which compiles fine, but fails to import with... Looking into the Cython internals, everything defined in mylib.pxd is stored as `Entry`s in a `ModuleScope`, and... On Wed, Feb 16, 2011 at 03:55:19PM -0800, Robert Bradshaw wrote: > On Wed, Feb 16, 2011 at 8:17 AM, W. Trevor King <wking(a)drexel.edu> wrote: > > What I'm missing is a way to bind the ModuleScope namespace to a name > > in expose.pyx so that commands like `dir(mylib)` and `getattr(mylib, > > name)` will work in expose.pyx. > > You have also hit into the thorny issue that .pxd files are used for > many things. They may be pure C library declarations with no Python > module backing, they may be declarations of (externally implemented) > Python modules (such as numpy.pxd), or they may be declarations for > Cython-implemented modules. > > It seems like it would be easier to generate some kind of wrapper > > class (PxdModule?) for mylib when it is cimported (at compile time), > > and then further interactions would take care of themselves (at run > > time). > > Would such an object be created anew for every module that cimports > the declaration file? Hmm, That doesn't sound very nice, does it. However, .pxd files declaring C libraries have no Python-space presence, so that was my initial idea. > I have toyed with the idea of subclassing the module object itself for > better support of C-level attributes from the Python (and Cython) > namespaces. Sorry, I don't understand "better support of C-level attributes". Can you give an example? > Here's another idea, what if extern blocks could contain cpdef > declarations, which would automatically generate a Python-level > wrappers for the declared members (if possible, otherwise an error)? Ah, this sounds good! Of the three .pxd roles you list above, external Python modules (e.g. numpy) and Cython-implemented modules (e.g. matched .pxd/.pyx) both already have a presence in Python-space. What's missing is a way to give (where possible) declarations of external C libraries a Python presence. cpdef fills this hole nicely, since its whole purpose is to expose Python interfaces to C-based elements. A side effect of this cpdef change would be that now even bare .pxd files (no matching .pyx) would have a Python presence, so You could do something like cimport mylib as mylib_c import mylib as mylib_py import sys # Access through Python for name in dir(mylib_py): setattr(sys.modules[__name__], name, getattr(mylib_py, name)) # Direct C access cdef get_a(): return mylib_c.CONST_A Where the Python access would be the new feature, and list all cpdef-ed stuff. However, from Parsing.py:2369: error(pos, "C struct/union/enum cannot be declared cpdef") From pyrex_differences.rst: If a function is declared :keyword:`cpdef` it can be called from and overridden by both extension and normal python subclasses. I believe the reason that cpdef-ed enums and similar are currently illegal is confusion between "can be called from Python" and "can be overridden from Python". I think these should be just like methods already are, in that you can "override" a method by subclassing it, but not by rebinding the name in the base class: >>> import pyximport; pyximport.install() >>> import rectangle as R >>> r = R.Rectangle(1, 2, 3, 4) >>> r.area = lambda(self): r.x1' Traceback (most recent call last): File "<string>", line 1, in <module> AttributeError: 'rectangle.Rectangle' object attribute 'area' is read-only Where rectangle.pyx is a minorly patched version of the last example from early_binding_for_speed.rst [1] and `area` is a cpdef-ed method. Why can't enums share this handling, with the enum taking the place of the method and the enum's module taking the place of the class? After all, enums have a Python-side tyoe (int or long). Unions don't really have a Python parallel, but structs do, so long as you can select which attributes should have (writable) Python interfaces. If we change the struct declaration syntax to be closer to the `cdef class` declaration syntax: cpdef struct Foo: cpdef public int intA cpdef readonly int intB cdef void *ptr We would both declare the important members of the C struct (as we can already do in Cython) and also have Cython automatically generate a Python class wrapping the struct (because of `cpdef struct`). The Python class would have: * Cython-generated getter/setter for intA (because of `cpdef public`) using the standard Python<->int coercion. * Similar Cython-generated getter for int B (because of `cpdef readonly`). * No Python access to ptr (standard C-access still possible through Cython). Doing something crazy like `cdef public void *ptr` would raise a compile-time error. I'm definately willing to help out with this (if someone will point me in the right direction), as the enum stuff would fix my original problem, and the struct stuff would allow me to rip out a bunch of boilerplate like cdef class Foo (object): cdef mylib.Foo _Foo def _intA_get(self): return self._Foo.intA def _intA_set(self, value): self._Foo.intA = value intA = property(fget=_intA_get, fset=_intA_set) def _intB_get(self): return self._Foo.intB intB = property(fget=_intB_get) from my wrapper code. Thanks, Trevor [1] While testing my overriding method example, I found a small typo in cython-docs' early_binding_for_speed.rst. Patch attached. -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt

2 3

Fixing NumPy support for Python 3 (Stefan, please help!)
by Lisandro Dalcin Feb. 17, 2011

Feb. 17, 2011

Stefan, what do you think about the patch below? This hunk is part of a series of fixes required to get numpy-dev working under Python 3.2. The root of the issue is that __cythonbufferdefaults__ keys&values end-up being "bytes" (this coercion is triggered in Interpreter.py). diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py index 5b339da..b72deef 100755 --- a/Cython/Compiler/ExprNodes.py +++ b/Cython/Compiler/ExprNodes.py @@ -12,6 +12,7 @@ cython.declare(error=object, warning=object, warn_once=object, Builtin=object, Symtab=object, Utils=object, find_coercion_error debug_disposal_code=object, debug_temp_alloc=object, debug_coerc +import sys import operator from Errors import error, warning, warn_once, InternalError, CompileError @@ -1136,6 +1137,8 @@ class StringNode(PyConstNode): return self.result_code def compile_time_value(self, env): + if sys.version_info[0] >= 3 and self.unicode_value: + return self.unicode_value return self.value -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

2 1