Mailman 3 August 2011 - cython-devel

Fwd: PEP 393 Summer of Code Project
by Stefan Behnel Aug. 22, 2011

Aug. 22, 2011

Sounds like he's at a point where this becomes interesting for us. Stefan -------- Original-Message -------- Subject: PEP 393 Summer of Code Project Date: Mon, 22 Aug 2011 14:58:51 -0400 From: Torsten Becker <torsten.becker...(a)gmail.com> To: python-dev...(a)python.org Hello all, I have implemented an initial version of PEP 393 -- "Flexible String Representation" as part of my Google Summer of Code project. My patch is hosted as a repository on bitbucket [1] and I created a related issue on the bug tracker [2]. I posted documentation for the current state of the development in the wiki [3]. Current tests show a potential reduction of memory by about 20% and CPU by 50% for a join micro benchmark. Starting a new interpreter still causes 3244 calls to create compatibility Py_UNICODE representations, 263 strings are created using the old API while 62719 are created using the new API. More measurements are on the wiki page [3]. If there is interest, I would like to continue working on the patch with the goal of getting it into Python 3.3. Any and all feedback is welcome. Regards, Torsten [1]: http://www.python.org/dev/peps/pep-0393 [2]: http://bugs.python.org/issue12819 [3]: http://wiki.python.org/moin/SummerOfCode/2011/PEP393

1 0

Entry.utility_code_definitions vs. Entry.utility_code
by Stefan Behnel Aug. 22, 2011

Aug. 22, 2011

Hi, is there a reason for having the two? The "uc_definition" is set to instances of "CythonUtilityCode" in UtilityCode.py, whereas the other is used to keep references to "UtilityCode" instances, but both inject their code in the same way, it appears. The first also has dedicated support in the pipeline, whereas support for the latter is spread across the syntax tree nodes. It seems cleaner to me to use the first approach, i.e. to simply walk all used entries at some late point in the pipeline and to inject their utility code, rather than to leave that responsibility to each node that uses them. Anything I misunderstood here? If not, any objections to merging the latter into the first? Stefan

2 4

Problems with decorated methods in cdef classes
by Stefan Behnel Aug. 22, 2011

Aug. 22, 2011

Hi, I've taken another stab at #593 and changed the way decorators are currently evaluated. http://trac.cython.org/cython_trac/ticket/593 https://github.com/cython/cython/commit/c40ff48f84b5e5841e4e2d2c6dcce3e6494… We previously had @deco def func(): pass turn into this: def func(): pass func = deco(func) Note how this binds the name more than once and even looks it up in between, which is problematic in class scopes and some other special cases. For example, this doesn't work: class Foo(object): @property def x(self): ... @x.setter def x(self, value): ... because "x.setter" is looked up *after* binding the second function "x" to its method name, thus overwriting the initial property. The correct way to do it is to create the function object in a temp, pass it through the decorator call chain, and then assign whatever result this produces to the method name. This works nicely, but it triggered a crash in problematic code of another test case, namely "closure_decorators_T478". That test does this: def print_args(func): def f(*args, **kwds): print "args", args, "kwds", kwds return func(*args, **kwds) return f cdef class Num: @print_args def is_prime(self, bint print_factors=False): ... Now, the problem is that Cython considers "is_prime" to be a method of a cdef class, although it actually is not. It's only an arbitrary function that happens to be defined inside of a cdef class body and that happens to be *called* by a method, namely "f". It now crashes for me because the "self" argument is not being passed into is_prime() as a C method argument when called by the wrapper function - and that's correct, because it's not a method call but a regular function call at that point. The correct way to fix this is to turn all decorated methods in cdef classes into plain functions. However, this has huge drawbacks, especially that the first argument ('self') can no longer be typed as the surrounding extension type. But, after all, you could do this: def swap_args(func): def f(*args): return func(*args[::-1]) return f cdef class Num: @swap_args def is_prime(arg, self): ... I'm not sure what to make of this. Does it make sense to go this route? Or does anyone see a way to make this "mostly" work, e.g. by somehow restricting cdef classes and their methods? Or should we just add runtime checks to prevent bad behaviour of decorators? Stefan

3 4

Re: [Cython] [cython-users] Calling gil-requiring function not allowed without gil
by Stefan Behnel Aug. 20, 2011

Aug. 20, 2011

[second try in moving this discussion to cython-devel] Dag Sverre Seljebotn, 12.08.2011 08:50: > On 08/12/2011 06:44 AM, Robert Bradshaw wrote: >> On Thu, Aug 11, 2011 at 5:53 AM, Dag Sverre Seljebotn >> <d.s.seljebotn(a)astro.uio.no> wrote: >>> On 08/11/2011 02:13 PM, Stefan Behnel wrote: >>>> >>>> Dag Sverre Seljebotn, 11.08.2011 13:58: >>>>> >>>>> On 08/11/2011 01:43 PM, Ting Zhou wrote: >>>>>> >>>>>> here is my pyx file. When I compile it, it says "Calling gil-requiring >>>>>> function not allowed without gil" at calling dabs(x[i]). Why my dabs >>>>>> function is a gil-requiring function? >>>>>> >>>>>> ==================================== >>>>>> from cython.parallel import * >>>>>> import numpy as np >>>>>> cimport numpy as np >>>>>> cimport cython >>>>>> >>>>>> cdef double dabs(double x): >>>>> >>>>> This should be >>>>> >>>>> cdef double dabs(double x) nogil: >>>> >>>> Note that Cython cannot infer this automatically. Even a trivial >>>> function may require an exclusive global lock for some reason, and it's >>>> common to use the GIL for that. So the programmer must be explicit here. >>> >>> Are you still against this mini-CEP?: >>> >>> with cython.global_lock(): >>> ... >>> >>> Where global_lock() is GIL-requiring noop. >> >> Just reading this, it's not immediately obvious what this means. (I >> thought at first this was different syntax for "with gil"...) > > True. "cython.synchronized"? The synchronized keyword in Java does not > quite the same thing but almost. > >>> This >>> >>> a) Improves code readability vastly. Having a critical section take effect >>> because of the *lack* of a keyword is just very odd to anyone who's not >>> shoulder deep in CPython internals >> >> I'm not following you here. The only way to run into this is if you >> have explicitly release it. Presumably you can learn both keywords at >> the same time. Perhaps extern function could be nogil by default. > > I'll try to explain again. Consider code like this: > > cdef call_c(): > magic_c_function(3) > > This code is perfectly fine even if magic_c_function is not reentrant -- > but that's hardly well documented! So it's conceivable that another > programmer comes along (who doesn't know the C library well) and decides > that "somebody has just been too lazy to add the nogil specifier" and slaps > it on -- at which point you have a bug. > > This is more to the point in creating readable code IMO: > > cdef call_c(): > with cython.synchronized(): > magic_c_function(3) I think I'm as confused as Robert here. Is that the GIL or some separate lock? If the latter, where would that lock be stored? Module-wide? While there are certainly use cases for that, I think it's much more common to either a) use the GIL or b) use an explicit lock at some well defined point, e.g. at an object or class level. I don't think we disagree when I say that locking should be explicit, but that includes the "thing" that keeps the lock during its lifetime. > Also, consider your classical race conditions: > > cdef init(): > global resource > if resource == NULL: > resource = malloc(sizeof(resource_t)) ... > > This is safe code. Well, it's safe code as long as there is no Python code interaction in between. Calling back into the interpreter may trigger a thread switch. Admittedly, this is not obvious and tends to introduce bugs. I learned that the hard way in lxml, actually twice. If your "synchronised" directive refers to the GIL, then it would suffer from that problem as well. I think that's very undesirable for an explicit critical section statement. Stefan

3 15

except clause on nogil functions
by mark florisson Aug. 20, 2011

Aug. 20, 2011

Hey, I think I found a rather serious bug: if an error label is used in a nogil function, it tries to build a traceback. So if the GIL is released you will immediately segfault, and otherwise it will work fine! Here is a snippet: cdef int with_gil_func() except 0 with gil: raise Exception("error!") cdef int nogil_func() nogil except 0: with_gil_func() with nogil: nogil_func() [0] [00:03] ~ ➤ python -c 'import test' [1] 6888 segmentation fault python -c 'import test' I've tried as far back as 0.12 but all versions down to 0.12 have it. BTW, it's really weird that the except clause on the function has to come after nogil but before with gil, and otherwise it's a syntax error. Should we fix it and do a bugfix release?

3 3

What now?
by Stefano Aug. 18, 2011

Aug. 18, 2011

Hi, now that I've nailed Cython code, I'd like to get into something more funny. Currently, I'm working on a set of macros to seamlessy integrate Cython into CMake build process (in fact, I love CMake). But, I'd like to work also on something more essential, so... Is there any updated TODO list? (ToDo.txt in git repo dates back to more than 8 months ago). I may also work on the wiki.... Cheers, --SKS

5 7

Entry lookup order
by Vitja Makarov Aug. 16, 2011

Aug. 16, 2011

Trying to silence compiler warning about taking address of variable I've found again that when CreateControlFlow is ran lhs entries are resolved and rhs aren't. As I understand analyse_declarations() creates scope entries for lhs. At this point we don't know much about scope name reference belongs to. Next analyse_expressions() fills the rest entries. Sine CreateControlFlow couldn't be run after analyse_expressions() is it correct to lookup for entry in the current scope? -- vitja.

1 0

Re: [Cython] [cython-users] Re: Segfault in PyThread_release_lock
by mark florisson Aug. 16, 2011

Aug. 16, 2011

On 15 August 2011 02:22, Nikolaus Rath <Nikolaus(a)rath.org> wrote: > mark florisson <markflorisson88(a)gmail.com> writes: >> On 14 August 2011 17:50, Nikolaus Rath <Nikolaus(a)rath.org> wrote: >>> mark florisson <markflorisson88(a)gmail.com> writes: >>>> On 12 August 2011 20:23, Nikolaus Rath <Nikolaus(a)rath.org> wrote: >>>>> Hello, >>>>> >>>>> The following segfault is completely incomprehensible to me: >>>>> >>>>> (gdb) bt full >>>>> #0 sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34 >>>>> No locals. >>>>> #1 0x000000000050e3e7 in PyThread_release_lock (lock=0x0) at ../Python/thread_pthread.h:374 >>>>> thelock = 0x0 >>>>> status = 0 >>>>> error = 0 >>>>> #2 0x00000000004c4f7b in PyEval_ReleaseLock () at ../Python/ceval.c:262 >>>>> No locals. >>>>> #3 0x00000000004fbeef in PyThreadState_DeleteCurrent () at ../Python/pystate.c:319 >>>>> tstate = 0x925da0 >>>>> #4 0x00000000004fc6a3 in PyGILState_Release (oldstate=PyGILState_UNLOCKED) >>>>> at ../Python/pystate.c:651 >>>>> tcur = 0x925da0 >>>>> __PRETTY_FUNCTION__ = "PyGILState_Release" >>>>> [...] >>>>> Why am I getting an error so deep in the python internals? >>>> >>>> The error is deep because Python needs to do certain things. So what >>>> it's doing here is destructing your thread state, which means that was >>>> the last call to PyGILState_Ensure(). This means your function was >>>> called from another thread than a Python thread. It looks to me like >>>> the GIL is not initialized (__pyx_gilstate_save = >>>> PyGILState_UNLOCKED). Are you sure you have called >>>> PyEval_EvalInitThreads()? >>> >>> No, I have not. But isn't Cython supposed to do this automatically when >>> entering the function if I define the function as "with gil"? >> >> No, it acquires the GIL, but it does not attempt to initialize it. >> Seeing how many people have issues with it, I'm thinking that perhaps >> we should if the module contains with gil functions, or with gil >> blocks in nogil functions. If you're using cython.parallel and with >> gil blocks, the GIL will be initialized at module-load time. Note that >> initializing the GIL slows down serially executed Python code, so >> that's why Cython doesn't initialize it unconditionally. It will have >> to be called at module-load time, as it should be called in the main >> python thread (or from any other python thread started by the >> thread(ing) module). > > Oh, wow. That's a tricky one. And it explains why I have the segfault > only in a reduced testcase, but not in the full blown application (the > application probably creates a Python thread before calling into Cython > code). I banged my head against the wall for quite some time already > because of that. > > Thanks a lot! It would certainly have cost me a *lot* of time if'd had > to figure that out myself. > > I wouldn't mind if Cython inserted the call automatically, but a big > warning in > http://docs.cython.org/src/userguide/external_C_code.html#acquiring-and-rel… > (under the "acquiring the GIL") might already be enough. > > > Best, > > -Nikolaus > > -- > »Time flies like an arrow, fruit flies like a Banana.« > > PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C > Definitely, I'll do that one of these days. I do think it's rather common to have with gil functions called as a callback from another (non-python) thread. I'm not so sure about with gil blocks in nogil functions. @Cython-dev: Do we merely want to update the docs, or do we want to initialize the GIL for either case, or only for the with gil functions? I'm not entirely sure what the overhead is for single-threaded code, but I'd say we need to initialize it for both cases.

4 11

Class scope lookup order
by Vitja Makarov Aug. 15, 2011

Aug. 15, 2011

When creating python-class dict it seems that CPython first looks at dict then at globals: A = 1 class X: A = A def y(): A = 3 class Y: A = A return Y Y = y() print(X.A, Y.A) Will print: 1, 1 And not: 1, 3 I didn't find documentation for this but if I'm correct that should be easy to fix issue in Cython. Now for this code cython reports compile-time error: local variable referenced before assignment -- vitja.

2 11

Cython bug ?
by Romain Guillebert Aug. 14, 2011

Aug. 14, 2011

Hi I tried to compiled Demos/primes.pyx using the ctypes backend and I think I've found a bug : The Entry for the kmax parameter does not set is_arg to 1. However if I turn the def into a cdef, it works fine. Is this a bug or a known hack ? Thanks Romain

3 5