From stefan_ml at behnel.de Sun Jul 3 20:33:13 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 03 Jul 2011 20:33:13 +0200 Subject: [Cython] Fwd: Re: GCC Python FE Message-ID: <4E10B5E9.9010401@behnel.de> Hi, just got this back as a reply from the author of a GCC Python frontend. This project flew completely below my radar so far, even though it's the second GSoC being invested here. From the Python wiki page, it looks like static Python compilers are starting to sum up these days. IMHO, it would be better to get these consolidated a bit, rather than having people reinvent wheels all over the place. http://wiki.python.org/moin/PythonImplementations Stefan -------- Original-Message -------- Subject: Re: GCC Python FE From: Philip Herron On 3 July 2011 16:57, Stefan Behnel wrote: > I just stumbled over this wiki page: > > http://gcc.gnu.org/wiki/PythonFrontEnd > > I'm a Cython core developer, and as such, I wonder what the purpose of this > project was, and what it lead to. Well i was inspired by PHC http://www.phpcompiler.org/ which was a similar idea but for php, the purpose was mostly i though it was interesting and wanted to see how feasible it is to do something like this plus i wanted to learn more, i've written compilers for simple more static languages and made my own interpreter before and the buzz word at the moment seems to be jit compilers but i still think compiling like this could still be a good step forward at least for my understanding. And in the end it could possibly open up different opportunities to use python in a different way be it on low memory systems or even a different way of embeding python into programs but we will see how feasible that is when its more mature. Well last year gsoc 2010 i spent mostly figuring out how i was going to implement a lot of things and getting more comfortable with GCC internals. But i am currently doing this for gsoc 2011 again http://www.google-melange.com/gsoc/project/google/gsoc2011/redbrain1123/11001 > > The project is marked as "part of GSoC 2010", so I guess there was some kind > of outcome. Could you give me an idea of how it went and what was achieved? > Well from last year's Gsoc i successfully got dynamic typing working, but its mostly similar to how cython does it, by wrapping things into the py_obect_ type where it stores the data and has hooks to its binary protocol and other standard hooks like print and destory etc or whatever it is, haven't looked at the cython code in sometime. Dynamic typing is kind of handled at runtime still because i cant really avoid it, but i am happy that i am working on a constant folding pass because i cant rely on gcc's own constant folding code to handle this because i have to handle it in my own front-end IL. But besides that i am working on object orientation and calls at the moment. Where i take something like: class foo: def __init__ (self) self.x = 1 And have to generate the type then generate the methods like: struct main.foo { gpy_object_t * x; } gpy_object_t * main.foo.__init__ (struct main.foo * self) { self->x = fold_int (1); ... } So i am just wrapping it into a normal object like you would in any other language because its fairly trivial its awkward in that python you can do self->x where x wasn't pre-declared it seems so i have to iterate several passes over my IL so i can generate the type then generate the methods. But this is all currently being worked on its mostly done just cleaning up lots of code and fixing some bugs leaving polymorphism for now will work on it when i come to it but i am more worried about how i will get calls working and imports across multiple files. But i am planning on using GCC's LTO to help me get around that, since at link time i have the opportunity to see code from everywhere to figure out where calls should be placed etc. But i wont get time to work on that for a few weeks yet as i need to finish what i am doing first. I am fairly happy with what i have achieved so far as its a lot of work for me on my own to do in 2 summers although i spent a lot of time though my university work just thinking how i will make things work so i can concentrate on implementation now. Its mostly a passion of mine but in the end i just want to understand things more and it seems this has taught me alot of how dynamic languages work and i enjoy it so i keep working on it. Although university is nearly over so i guess i will need to find a job xD lol. Hope this answers your questions feel free to ask more if your interested. :) --Phil From vitja.makarov at gmail.com Mon Jul 4 20:43:52 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 4 Jul 2011 22:43:52 +0400 Subject: [Cython] Hudson is down? Message-ID: It seems that hudson server is down. -- vitja. From stefan_ml at behnel.de Tue Jul 5 06:12:03 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 Jul 2011 06:12:03 +0200 Subject: [Cython] Hudson is down? In-Reply-To: References: Message-ID: <4E128F13.4040808@behnel.de> Vitja Makarov, 04.07.2011 20:43: > It seems that hudson server is down. It's back now. Looks like the sage.math machine was restarted. Stefan From vitja.makarov at gmail.com Tue Jul 5 08:21:39 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 5 Jul 2011 10:21:39 +0400 Subject: [Cython] PEP 3135 -- New Super Message-ID: I was thinking about implementing new super() with no arguments. The problem is where to store __class__, I see two options here: 1. Add func_class member to CyFunction, this way __class__ will be private and not visible for inner functions: 2. Put it into closure Actually, I like the first one. And I don't think that __class__ should be use somewhere outside super() -- vitja. From stefan_ml at behnel.de Tue Jul 5 09:00:49 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 Jul 2011 09:00:49 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: Message-ID: <4E12B6A1.3090805@behnel.de> Vitja Makarov, 05.07.2011 08:21: > I was thinking about implementing new super() with no arguments. Please do :) If you start working on it, please assign the ticket to you: http://trac.cython.org/cython_trac/ticket/696 > The problem is where to store __class__, I see two options here: > > 1. Add func_class member to CyFunction, this way __class__ will be > private and not visible for inner functions: > 2. Put it into closure The second option has the advantage of requiring the field only when super() is used, whereas the first impacts all functions. I would expect that programs commonly have a lot more functions than specifically methods that use a no-argument call to super(), so this may make a difference. OTOH, not all methods have a closure, so creating one just to store the "__class__" field is very wasteful, in terms of both runtime and memory overhead. A lot more wasteful than paying 8 bytes of memory for each function, with no additional time overhead. > And I don't think that __class__ should be use somewhere outside super() Agreed. CPython simply uses a compile time heuristic ("is there a function call to something global named 'super'?") when creating this field, so it's strictly reserved for this use case. BTW, I like the irony in the fact that CPython essentially gives Cython semantics to the "super" builtin here, by (partially) evaluating it at compile time. Stefan From vitja.makarov at gmail.com Tue Jul 5 09:17:09 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 5 Jul 2011 11:17:09 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: <4E12B6A1.3090805@behnel.de> References: <4E12B6A1.3090805@behnel.de> Message-ID: 2011/7/5 Stefan Behnel : > Vitja Makarov, 05.07.2011 08:21: >> >> I was thinking about implementing new super() with no arguments. > > Please do :) > > If you start working on it, please assign the ticket to you: > > http://trac.cython.org/cython_trac/ticket/696 > Ok, I'll do this if I start. > >> The problem is where to store __class__, I see two options here: >> >> 1. Add func_class member to CyFunction, this way __class__ will be >> private and not visible for inner functions: >> 2. Put it into closure > > The second option has the advantage of requiring the field only when super() > is used, whereas the first impacts all functions. > > I would expect that programs commonly have a lot more functions than > specifically methods that use a no-argument call to super(), so this may > make a difference. > So, now classes are created the following way: class_dict = {} class_dict.foo = foo_func class = CreateClass(class_dict) So after class is created I should check its dict for CyFunction members (maybe only ones that actually require __class__) and set __class__: for value in class.__dict__.itervalues(): if isinstance(value, CyFunction) and value.func_class is WantClass: value.func_class = class > OTOH, not all methods have a closure, so creating one just to store the > "__class__" field is very wasteful, in terms of both runtime and memory > overhead. A lot more wasteful than paying 8 bytes of memory for each > function, with no additional time overhead. > Going this way it only requires to initialize closure: closure.func_class = class Btw, first way requires cyfunction signature change, it would accept cyfunction object as first argument. This also could help to solve default args problem. > >> And I don't think that __class__ should be use somewhere outside super() > > Agreed. CPython simply uses a compile time heuristic ("is there a function > call to something global named 'super'?") when creating this field, so it's > strictly reserved for this use case. > > BTW, I like the irony in the fact that CPython essentially gives Cython > semantics to the "super" builtin here, by (partially) evaluating it at > compile time. > Yeah, I think Cython super with no args should be a little bit faster then classic one. -- vitja. From stefan_ml at behnel.de Tue Jul 5 10:04:14 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 Jul 2011 10:04:14 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> Message-ID: <4E12C57E.1060307@behnel.de> Vitja Makarov, 05.07.2011 09:17: > 2011/7/5 Stefan Behnel: >> Vitja Makarov, 05.07.2011 08:21: >>> I was thinking about implementing new super() with no arguments. >> http://trac.cython.org/cython_trac/ticket/696 >> >>> The problem is where to store __class__, I see two options here: >>> >>> 1. Add func_class member to CyFunction, this way __class__ will be >>> private and not visible for inner functions: >>> 2. Put it into closure >> >> The second option has the advantage of requiring the field only when super() >> is used, whereas the first impacts all functions. >> >> I would expect that programs commonly have a lot more functions than >> specifically methods that use a no-argument call to super(), so this may >> make a difference. >> > > So, now classes are created the following way: > > class_dict = {} > class_dict.foo = foo_func > class = CreateClass(class_dict) > > So after class is created I should check its dict for CyFunction > members (maybe only ones that actually require __class__) > and set __class__: > > for value in class.__dict__.itervalues(): > if isinstance(value, CyFunction) and value.func_class is WantClass: > value.func_class = class Remember that no-args super() can only be used in functions that are literally written inside of a class body, so we actually know at compile time which functions need this field. We can thus do better than a generic loop over all fields. We even have the function object pointers directly available in the module init function where we create the class body. BTW, we also need a way to make this work for cdef classes. No idea how different that would be. >> OTOH, not all methods have a closure, so creating one just to store the >> "__class__" field is very wasteful, in terms of both runtime and memory >> overhead. A lot more wasteful than paying 8 bytes of memory for each >> function, with no additional time overhead. > > Going this way it only requires to initialize closure: Yes, and that's costly. > Btw, first way requires cyfunction signature change, it would accept > cyfunction object as first argument. We currently pass the binding (i.e. owning) object, right? > This also could help to solve default args problem. And potentially other problems, too. Think of heap allocated modules, for example. http://trac.cython.org/cython_trac/ticket/173 http://trac.cython.org/cython_trac/ticket/218 Seeing this, I'm all for using a field in CyFunction. >>> And I don't think that __class__ should be use somewhere outside super() >> >> Agreed. CPython simply uses a compile time heuristic ("is there a function >> call to something global named 'super'?") when creating this field, so it's >> strictly reserved for this use case. >> >> BTW, I like the irony in the fact that CPython essentially gives Cython >> semantics to the "super" builtin here, by (partially) evaluating it at >> compile time. > > Yeah, I think Cython super with no args should be a little bit faster > then classic one. I think speed isn't really all that important here. Calling Python methods is costly enough anyway. IMO, the main reason for the heuristic is to prevent class objects from being kept alive by their methods, except for the single case where super() is used. Keeping a class alive just because one of its methods is still used somewhere can be very costly, depending on the content of the class dict. It also creates a reference cycle, which is another costly thing in CPython as it requires a GC run over the whole class dict to get cleaned up. The situation for modules is at least slightly different, as modules do not tend to get unloaded, so there's always a reference to them from sys.modules. If that ever gets removed, it's really ok if it takes time to clean things up. Stefan From vitja.makarov at gmail.com Tue Jul 5 10:37:40 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 5 Jul 2011 12:37:40 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: <4E12C57E.1060307@behnel.de> References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> Message-ID: 2011/7/5 Stefan Behnel : > Vitja Makarov, 05.07.2011 09:17: >> >> 2011/7/5 Stefan Behnel: >>> >>> Vitja Makarov, 05.07.2011 08:21: >>>> >>>> I was thinking about implementing new super() with no arguments. >>> >>> http://trac.cython.org/cython_trac/ticket/696 >>> >>>> The problem is where to store __class__, I see two options here: >>>> >>>> 1. Add func_class member to CyFunction, this way __class__ will be >>>> private and not visible for inner functions: >>>> 2. Put it into closure >>> >>> The second option has the advantage of requiring the field only when >>> super() >>> is used, whereas the first impacts all functions. >>> >>> I would expect that programs commonly have a lot more functions than >>> specifically methods that use a no-argument call to super(), so this may >>> make a difference. >>> >> >> So, now classes are created the following way: >> >> class_dict = {} >> class_dict.foo = foo_func >> class = CreateClass(class_dict) >> >> So after class is created I should check its dict for CyFunction >> members (maybe only ones that actually require __class__) >> and set __class__: >> >> for value in class.__dict__.itervalues(): >> ? ?if isinstance(value, CyFunction) and value.func_class is WantClass: >> ? ? ? ?value.func_class = class > > Remember that no-args super() can only be used in functions that are > literally written inside of a class body, so we actually know at compile > time which functions need this field. We can thus do better than a generic > loop over all fields. We even have the function object pointers directly > available in the module init function where we create the class body. > Yes, but now cyfunction references are lost as they were in temps. > BTW, we also need a way to make this work for cdef classes. No idea how > different that would be. > I think super() for cdef classes should be much easy to implement as we already know cname for a class at compile time. That would be simple transform. > >>> OTOH, not all methods have a closure, so creating one just to store the >>> "__class__" field is very wasteful, in terms of both runtime and memory >>> overhead. A lot more wasteful than paying 8 bytes of memory for each >>> function, with no additional time overhead. >> >> Going this way it only requires to initialize closure: > > Yes, and that's costly. > > >> Btw, first way requires cyfunction signature change, it would accept >> cyfunction object as first argument. > > We currently pass the binding (i.e. owning) object, right? > Right. When we pass CyFunction closure should be moved into its members. > >> This also could help to solve default args problem. > > And potentially other problems, too. Think of heap allocated modules, for > example. > > http://trac.cython.org/cython_trac/ticket/173 > http://trac.cython.org/cython_trac/ticket/218 > > Seeing this, I'm all for using a field in CyFunction. > > >> Yeah, I think Cython super with no args should be a little bit faster >> then classic one. > > I think speed isn't really all that important here. Calling Python methods > is costly enough anyway. > > IMO, the main reason for the heuristic is to prevent class objects from > being kept alive by their methods, except for the single case where super() > is used. Keeping a class alive just because one of its methods is still used > somewhere can be very costly, depending on the content of the class dict. It > also creates a reference cycle, which is another costly thing in CPython as > it requires a GC run over the whole class dict to get cleaned up. > > The situation for modules is at least slightly different, as modules do not > tend to get unloaded, so there's always a reference to them from > sys.modules. If that ever gets removed, it's really ok if it takes time to > clean things up. > Yes, but I hope that classes are rarely deleted. And I'm afraid that we can't avoid cycles when implementing __class__ for pure-python classes. -- vitja. From stefan_ml at behnel.de Tue Jul 5 10:49:42 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 05 Jul 2011 10:49:42 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> Message-ID: <4E12D026.9040206@behnel.de> Vitja Makarov, 05.07.2011 10:37: > 2011/7/5 Stefan Behnel: >> IMO, the main reason for the heuristic is to prevent class objects from >> being kept alive by their methods, except for the single case where super() >> is used. Keeping a class alive just because one of its methods is still used >> somewhere can be very costly, depending on the content of the class dict. It >> also creates a reference cycle, which is another costly thing in CPython as >> it requires a GC run over the whole class dict to get cleaned up. >> >> The situation for modules is at least slightly different, as modules do not >> tend to get unloaded, so there's always a reference to them from >> sys.modules. If that ever gets removed, it's really ok if it takes time to >> clean things up. > > Yes, but I hope that classes are rarely deleted. > And I'm afraid that we can't avoid cycles when implementing __class__ > for pure-python classes. As I said, it's fine if super() is used, but not otherwise. Basically, no-args super() is just a shortcut for "super(TheClass, self)", with the difference that the long form looks up the name at runtime, whereas the short form keeps a reference to the type at module setup time. So, both need the class at runtime, but have different characteristics otherwise. (but if the code is not using super(), there is no need to have the methods own a reference to their class) Stefan From robertwb at math.washington.edu Tue Jul 5 23:31:09 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 5 Jul 2011 14:31:09 -0700 Subject: [Cython] Hudson is down? In-Reply-To: <4E128F13.4040808@behnel.de> References: <4E128F13.4040808@behnel.de> Message-ID: On Mon, Jul 4, 2011 at 9:12 PM, Stefan Behnel wrote: > Vitja Makarov, 04.07.2011 20:43: >> >> It seems that hudson server is down. > > It's back now. Looks like the sage.math machine was restarted. Yep, sage.math was restarted. Thanks for getting this back up and running. - Robert From stefan_ml at behnel.de Wed Jul 6 02:36:32 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 06 Jul 2011 02:36:32 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: <4E12C57E.1060307@behnel.de> References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> Message-ID: <4E13AE10.3040104@behnel.de> Stefan Behnel, 05.07.2011 10:04: > Vitja Makarov, 05.07.2011 09:17: >> 2011/7/5 Stefan Behnel: >>> Vitja Makarov, 05.07.2011 08:21: >>>> I was thinking about implementing new super() with no arguments. >>> http://trac.cython.org/cython_trac/ticket/696 >>> >>>> The problem is where to store __class__, I see two options here: >>>> >>>> 1. Add func_class member to CyFunction, this way __class__ will be >>>> private and not visible for inner functions: >>>> 2. Put it into closure >>> >>> The second option has the advantage of requiring the field only when >>> super() >>> is used, whereas the first impacts all functions. >>> >>> I would expect that programs commonly have a lot more functions than >>> specifically methods that use a no-argument call to super(), so this may >>> make a difference. >>> >> >> So, now classes are created the following way: >> >> class_dict = {} >> class_dict.foo = foo_func >> class = CreateClass(class_dict) >> >> So after class is created I should check its dict for CyFunction >> members (maybe only ones that actually require __class__) >> and set __class__: >> >> for value in class.__dict__.itervalues(): >> if isinstance(value, CyFunction) and value.func_class is WantClass: >> value.func_class = class >> >> Btw, first way requires cyfunction signature change, it would accept >> cyfunction object as first argument. > > We currently pass the binding (i.e. owning) object, right? So, how would this work for methods? We need to pass the 'self' object there, which the CyFunction doesn't know. If anything, it only knows the class it was defined in, which doesn't help here. Stefan From romain.py at gmail.com Wed Jul 6 05:10:35 2011 From: romain.py at gmail.com (Romain Guillebert) Date: Wed, 6 Jul 2011 05:10:35 +0200 Subject: [Cython] Cython backend aiming PyPy Status Message-ID: <20110706031034.GA31841@ubuntu> Hi I created a blog post summarizing what I've done the last few weeks on the Cython backend aiming PyPy. It's located at this URL : http://rguillebert.blogspot.com/2011/07/cython-backend-aiming-pypy-status.html Cheers Romain From stefan_ml at behnel.de Wed Jul 6 08:01:16 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 06 Jul 2011 08:01:16 +0200 Subject: [Cython] Cython backend aiming PyPy Status In-Reply-To: <20110706031034.GA31841@ubuntu> References: <20110706031034.GA31841@ubuntu> Message-ID: <4E13FA2C.4020300@behnel.de> Romain Guillebert, 06.07.2011 05:10: > I created a blog post summarizing what I've done the last few weeks on > the Cython backend aiming PyPy. > > It's located at this URL : > http://rguillebert.blogspot.com/2011/07/cython-backend-aiming-pypy-status.html Congrats, that sounds pretty usable already. Instead of stepping directly into writing dedicated tests for your existing code, you should first try to get Cython's normal test suite running. That will give you a huge set of tests right away. One remark for this list: given that you simply reuse the type analysis steps in the ctypes pipeline (which is totally the right thing to do!), Cython should be careful what it does during these steps. Traditionally (and especially prehistorically), the type analysis step has been a point where the tree is changed in ways that didn't quite fit anywhere else. From now on, we need to take care not to break the ctypes backend by applying C-only tree changes here. Another important reason for getting the test suite to run, so that we can hand the regression testing over to Jenkins. Stefan From vitja.makarov at gmail.com Wed Jul 6 09:05:12 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 6 Jul 2011 11:05:12 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: <4E13AE10.3040104@behnel.de> References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> Message-ID: 2011/7/6 Stefan Behnel : > Stefan Behnel, 05.07.2011 10:04: >> >> Vitja Makarov, 05.07.2011 09:17: >>> >>> 2011/7/5 Stefan Behnel: >>>> >>>> Vitja Makarov, 05.07.2011 08:21: >>>>> >>>>> I was thinking about implementing new super() with no arguments. >>>> >>>> http://trac.cython.org/cython_trac/ticket/696 >>>> >>>>> The problem is where to store __class__, I see two options here: >>>>> >>>>> 1. Add func_class member to CyFunction, this way __class__ will be >>>>> private and not visible for inner functions: >>>>> 2. Put it into closure >>>> >>>> The second option has the advantage of requiring the field only when >>>> super() >>>> is used, whereas the first impacts all functions. >>>> >>>> I would expect that programs commonly have a lot more functions than >>>> specifically methods that use a no-argument call to super(), so this may >>>> make a difference. >>>> >>> >>> So, now classes are created the following way: >>> >>> class_dict = {} >>> class_dict.foo = foo_func >>> class = CreateClass(class_dict) >>> >>> So after class is created I should check its dict for CyFunction >>> members (maybe only ones that actually require __class__) >>> and set __class__: >>> >>> for value in class.__dict__.itervalues(): >>> if isinstance(value, CyFunction) and value.func_class is WantClass: >>> value.func_class = class >>> >>> Btw, first way requires cyfunction signature change, it would accept >>> cyfunction object as first argument. >> >> We currently pass the binding (i.e. owning) object, right? > > So, how would this work for methods? We need to pass the 'self' object > there, which the CyFunction doesn't know. If anything, it only knows the > class it was defined in, which doesn't help here. > >From PEP: "super() is equivalent to: super(__class__, )" So we only have func_class (that is inside cyfunction object) and first function argument of method to super() I tried to implement super() for cdef classes and it worked. It works very simple: cdef class Foo: def meth(self): super().meth() # is transformed into super(Foo, self) -- vitja. From stefan_ml at behnel.de Wed Jul 6 09:22:07 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 06 Jul 2011 09:22:07 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> Message-ID: <4E140D1F.5000809@behnel.de> Vitja Makarov, 06.07.2011 09:05: > 2011/7/6 Stefan Behnel: >> Stefan Behnel, 05.07.2011 10:04: >>> >>> Vitja Makarov, 05.07.2011 09:17: >>>> >>>> 2011/7/5 Stefan Behnel: >>>>> >>>>> Vitja Makarov, 05.07.2011 08:21: >>>>>> >>>>>> I was thinking about implementing new super() with no arguments. >>>>> >>>>> http://trac.cython.org/cython_trac/ticket/696 >>>>> >>>>>> The problem is where to store __class__, I see two options here: >>>>>> >>>>>> 1. Add func_class member to CyFunction, this way __class__ will be >>>>>> private and not visible for inner functions: >>>>>> 2. Put it into closure >>>>> >>>>> The second option has the advantage of requiring the field only when >>>>> super() >>>>> is used, whereas the first impacts all functions. >>>>> >>>>> I would expect that programs commonly have a lot more functions than >>>>> specifically methods that use a no-argument call to super(), so this may >>>>> make a difference. >>>>> >>>> >>>> So, now classes are created the following way: >>>> >>>> class_dict = {} >>>> class_dict.foo = foo_func >>>> class = CreateClass(class_dict) >>>> >>>> So after class is created I should check its dict for CyFunction >>>> members (maybe only ones that actually require __class__) >>>> and set __class__: >>>> >>>> for value in class.__dict__.itervalues(): >>>> if isinstance(value, CyFunction) and value.func_class is WantClass: >>>> value.func_class = class >>>> >>>> Btw, first way requires cyfunction signature change, it would accept >>>> cyfunction object as first argument. >>> >>> We currently pass the binding (i.e. owning) object, right? >> >> So, how would this work for methods? We need to pass the 'self' object >> there, which the CyFunction doesn't know. If anything, it only knows the >> class it was defined in, which doesn't help here. > > From PEP: "super() is equivalent to: super(__class__,)" I wasn't speaking of super(). What I meant, was: how do we pass 'self' when we pass the CyFunction object as the first argument? Stefan From vitja.makarov at gmail.com Wed Jul 6 10:01:19 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 6 Jul 2011 12:01:19 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: <4E140D1F.5000809@behnel.de> References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: 2011/7/6 Stefan Behnel : > Vitja Makarov, 06.07.2011 09:05: >> >> 2011/7/6 Stefan Behnel: >>> >>> Stefan Behnel, 05.07.2011 10:04: >>>> >>>> Vitja Makarov, 05.07.2011 09:17: >>>>> >>>>> 2011/7/5 Stefan Behnel: >>>>>> >>>>>> Vitja Makarov, 05.07.2011 08:21: >>>>>>> >>>>>>> I was thinking about implementing new super() with no arguments. >>>>>> >>>>>> http://trac.cython.org/cython_trac/ticket/696 >>>>>> >>>>>>> The problem is where to store __class__, I see two options here: >>>>>>> >>>>>>> 1. Add func_class member to CyFunction, this way __class__ will be >>>>>>> private and not visible for inner functions: >>>>>>> 2. Put it into closure >>>>>> >>>>>> The second option has the advantage of requiring the field only when >>>>>> super() >>>>>> is used, whereas the first impacts all functions. >>>>>> >>>>>> I would expect that programs commonly have a lot more functions than >>>>>> specifically methods that use a no-argument call to super(), so this >>>>>> may >>>>>> make a difference. >>>>>> >>>>> >>>>> So, now classes are created the following way: >>>>> >>>>> class_dict = {} >>>>> class_dict.foo = foo_func >>>>> class = CreateClass(class_dict) >>>>> >>>>> So after class is created I should check its dict for CyFunction >>>>> members (maybe only ones that actually require __class__) >>>>> and set __class__: >>>>> >>>>> for value in class.__dict__.itervalues(): >>>>> if isinstance(value, CyFunction) and value.func_class is WantClass: >>>>> value.func_class = class >>>>> >>>>> Btw, first way requires cyfunction signature change, it would accept >>>>> cyfunction object as first argument. >>>> >>>> We currently pass the binding (i.e. owning) object, right? >>> >>> So, how would this work for methods? We need to pass the 'self' object >>> there, which the CyFunction doesn't know. If anything, it only knows the >>> class it was defined in, which doesn't help here. >> >> From PEP: "super() is equivalent to: super(__class__,)" > > I wasn't speaking of super(). What I meant, was: how do we pass 'self' when > we pass the CyFunction object as the first argument? > Oh, ok. Now we pass closure or nothing in self. So method's self is passed via tuple. Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we can override this and change signature of cyfunction to: PyObject func(CyFunction *func, PyObject *self, PyObject *args, PyObject *kwargs); This way we should implement new instancemethod type. -- vitja. From vitja.makarov at gmail.com Wed Jul 6 21:24:45 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 6 Jul 2011 23:24:45 +0400 Subject: [Cython] Speedup hudson job with ccache Message-ID: I tried to speedup runtests with ccache How to run: CYTHON_RUNTESTS_CCACHE="ccache" python runtests.py Running 'CYTHON_RUNTESTS_CCACHE="ccache" python runtests.py --no-cpp' first time takes 490 seconds on my PC, second time is 214, speed up is 2.2x, cache size is 16MB https://github.com/vitek/cython/commit/c2fb58c780d8727f95353d3f0c28028977a8661c -- vitja. From robertwb at math.washington.edu Wed Jul 6 21:32:48 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 6 Jul 2011 12:32:48 -0700 Subject: [Cython] Speedup hudson job with ccache In-Reply-To: References: Message-ID: Very cool. We should get ccache on sage.math to start using it on hudson. On Wed, Jul 6, 2011 at 12:24 PM, Vitja Makarov wrote: > I tried to speedup runtests with ccache > > How to run: > CYTHON_RUNTESTS_CCACHE="ccache" python ?runtests.py > > Running 'CYTHON_RUNTESTS_CCACHE="ccache" python ?runtests.py --no-cpp' > first time takes 490 seconds on my PC, > second time is 214, speed up is 2.2x, cache size is 16MB > > https://github.com/vitek/cython/commit/c2fb58c780d8727f95353d3f0c28028977a8661c > > -- > vitja. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Thu Jul 7 08:24:41 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jul 2011 08:24:41 +0200 Subject: [Cython] Speedup hudson job with ccache In-Reply-To: References: Message-ID: <4E155129.4010406@behnel.de> Robert Bradshaw, 06.07.2011 21:32: > We should get ccache on sage.math to start using it on hudson. It's just a single binary when installed. I have a copy in my ~/ccache/bin. Changing the Jenkins jobs now. Stefan From vitja.makarov at gmail.com Thu Jul 7 08:32:40 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 7 Jul 2011 10:32:40 +0400 Subject: [Cython] Speedup hudson job with ccache In-Reply-To: <4E155129.4010406@behnel.de> References: <4E155129.4010406@behnel.de> Message-ID: 2011/7/7 Stefan Behnel > Robert Bradshaw, 06.07.2011 21:32: > > We should get ccache on sage.math to start using it on hudson. >> > > It's just a single binary when installed. I have a copy in my ~/ccache/bin. > Changing the Jenkins jobs now. > > Is that already used? -- vitja. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Thu Jul 7 08:41:33 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jul 2011 08:41:33 +0200 Subject: [Cython] Speedup hudson job with ccache In-Reply-To: References: <4E155129.4010406@behnel.de> Message-ID: <4E15551D.9080304@behnel.de> Vitja Makarov, 07.07.2011 08:32: > 2011/7/7 Stefan Behnel > >> Robert Bradshaw, 06.07.2011 21:32: >> >> We should get ccache on sage.math to start using it on hudson. >>> >> >> It's just a single binary when installed. I have a copy in my ~/ccache/bin. >> Changing the Jenkins jobs now. >> >> > Is that already used? Actually, no. I noticed that it would a) require changes to all test jobs, and b) prevent us from getting clean test runs. I prefer a safe and clean run over a fast one. Also, the test runner clears the target directory on startup. I wonder where you got your numbers from... Stefan From vitja.makarov at gmail.com Thu Jul 7 08:54:24 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 7 Jul 2011 10:54:24 +0400 Subject: [Cython] Speedup hudson job with ccache In-Reply-To: <4E15551D.9080304@behnel.de> References: <4E155129.4010406@behnel.de> <4E15551D.9080304@behnel.de> Message-ID: 2011/7/7 Stefan Behnel > Vitja Makarov, 07.07.2011 08:32: > >> 2011/7/7 Stefan Behnel >> >> Robert Bradshaw, 06.07.2011 21:32: >>> >>> We should get ccache on sage.math to start using it on hudson. >>> >>>> >>>> >>> It's just a single binary when installed. I have a copy in my >>> ~/ccache/bin. >>> Changing the Jenkins jobs now. >>> >>> >>> Is that already used? >> > > Actually, no. I noticed that it would a) require changes to all test jobs, > and b) prevent us from getting clean test runs. I prefer a safe and clean > run over a fast one. > > a) you can set CYTHON_RUNTESTS_CCACHE globaly somewhere inside ~/.profile > Also, the test runner clears the target directory on startup. I wonder > where you got your numbers from... > > By default ccache stores object files inside ~/.ccache directory, how it works: 1. It runs preprocessor (cpp), then adds compiler version, compilation flags and so on. 2. Calculates checksum (md5? I don't know) 3. Check if it's already in the cache, no compilation is required http://ccache.samba.org/ So I think that would be safe to use ccache here. -- vitja. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitja.makarov at gmail.com Thu Jul 7 09:05:16 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 7 Jul 2011 11:05:16 +0400 Subject: [Cython] Speedup hudson job with ccache In-Reply-To: References: <4E155129.4010406@behnel.de> <4E15551D.9080304@behnel.de> Message-ID: 2011/7/7 Vitja Makarov > > > 2011/7/7 Stefan Behnel > >> Vitja Makarov, 07.07.2011 08:32: >> >>> 2011/7/7 Stefan Behnel >>> >>> Robert Bradshaw, 06.07.2011 21:32: >>>> >>>> We should get ccache on sage.math to start using it on hudson. >>>> >>>>> >>>>> >>>> It's just a single binary when installed. I have a copy in my >>>> ~/ccache/bin. >>>> Changing the Jenkins jobs now. >>>> >>>> >>>> Is that already used? >>> >> >> Actually, no. I noticed that it would a) require changes to all test jobs, >> and b) prevent us from getting clean test runs. I prefer a safe and clean >> run over a fast one. >> >> > a) you can set CYTHON_RUNTESTS_CCACHE globaly somewhere inside ~/.profile > > > >> Also, the test runner clears the target directory on startup. I wonder >> where you got your numbers from... >> >> > By default ccache stores object files inside ~/.ccache directory, how it > works: > > 1. It runs preprocessor (cpp), then adds compiler version, compilation > flags and so on. > 2. Calculates checksum (md5? I don't know) > 3. Check if it's already in the cache, no compilation is required > > http://ccache.samba.org/ > > So I think that would be safe to use ccache here. > > How it works: http://ccache.samba.org/manual.html#_how_ccache_works Ccache >=3.0 has new "direct mode" think we should disable this, I was running ccache 2.4 that allways do full preprocessing -- vitja. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at math.washington.edu Thu Jul 7 09:19:36 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 Jul 2011 00:19:36 -0700 Subject: [Cython] Speedup hudson job with ccache In-Reply-To: References: <4E155129.4010406@behnel.de> <4E15551D.9080304@behnel.de> Message-ID: On Wed, Jul 6, 2011 at 11:54 PM, Vitja Makarov wrote: > > > 2011/7/7 Stefan Behnel >> >> Vitja Makarov, 07.07.2011 08:32: >>> >>> 2011/7/7 Stefan Behnel >>> >>>> Robert Bradshaw, 06.07.2011 21:32: >>>> >>>> ?We should get ccache on sage.math to start using it on hudson. >>>>> >>>> >>>> It's just a single binary when installed. I have a copy in my >>>> ~/ccache/bin. >>>> Changing the Jenkins jobs now. >>>> >>>> >>> Is that already used? >> >> Actually, no. I noticed that it would a) require changes to all test jobs, >> and b) prevent us from getting clean test runs. I prefer a safe and clean >> run over a fast one. >> > > a) you can set CYTHON_RUNTESTS_CCACHE globaly somewhere inside ~/.profile I think you can set global hudson environment variables as well, which would probably be preferable. >> Also, the test runner clears the target directory on startup. This isn't an issue, if it can, it just copies the .so files from the cache (with new timestamps). >> I wonder >> where you got your numbers from... > > By default ccache stores object files inside ~/.ccache directory, how it > works: > 1. It runs preprocessor (cpp), then adds compiler version, compilation flags > and so on. > 2. Calculates checksum (md5? I don't know) > 3. Check if it's already in the cache, no compilation is required > http://ccache.samba.org/ > So I think that would be safe to use ccache here. Yes, ccache should be totally safe here, unless it's fundamentally broken. - Robert From stefan_ml at behnel.de Thu Jul 7 09:29:14 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jul 2011 09:29:14 +0200 Subject: [Cython] Speedup hudson job with ccache In-Reply-To: References: <4E155129.4010406@behnel.de> <4E15551D.9080304@behnel.de> Message-ID: <4E15604A.8080805@behnel.de> Vitja Makarov, 07.07.2011 08:54: > 2011/7/7 Stefan Behnel > >> Vitja Makarov, 07.07.2011 08:32: >> >>> 2011/7/7 Stefan Behnel >>> >>> Robert Bradshaw, 06.07.2011 21:32: >>>> >>>> We should get ccache on sage.math to start using it on hudson. >>>> >>>>> >>>>> >>>> It's just a single binary when installed. I have a copy in my >>>> ~/ccache/bin. >>>> Changing the Jenkins jobs now. >>>> >>>> >>>> Is that already used? >>> >> >> Actually, no. I noticed that it would a) require changes to all test jobs, >> and b) prevent us from getting clean test runs. I prefer a safe and clean >> run over a fast one. >> >> > a) you can set CYTHON_RUNTESTS_CCACHE globaly somewhere inside ~/.profile ;) I actually set it globally in Jenkins. However, that doesn't automatically remove the "rm -fr *" in the job configs - I initially thought that would be a problem, but, apparently, it isn't. >> Also, the test runner clears the target directory on startup. I wonder >> where you got your numbers from... >> > By default ccache stores object files inside ~/.ccache directory Hmm, I guess changing that directory into a job local directory would still require changing each job... Are there any concurrency issues with ccache? Anyway, I can't see a .ccache in my home directory, but if that's how it works, it should at least be properly enabled now. Stefan From stefan_ml at behnel.de Thu Jul 7 10:46:07 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jul 2011 10:46:07 +0200 Subject: [Cython] Speedup hudson job with ccache In-Reply-To: <4E15604A.8080805@behnel.de> References: <4E155129.4010406@behnel.de> <4E15551D.9080304@behnel.de> <4E15604A.8080805@behnel.de> Message-ID: <4E15724F.8070905@behnel.de> Stefan Behnel, 07.07.2011 09:29: > Vitja Makarov, 07.07.2011 08:54: >> 2011/7/7 Stefan Behnel >> >>> Vitja Makarov, 07.07.2011 08:32: >>> >>>> 2011/7/7 Stefan Behnel >>>> >>>> Robert Bradshaw, 06.07.2011 21:32: >>>>> >>>>> We should get ccache on sage.math to start using it on hudson. >>>>> >>>>>> >>>>>> >>>>> It's just a single binary when installed. I have a copy in my >>>>> ~/ccache/bin. >>>>> Changing the Jenkins jobs now. >>>>> >>>>> >>>>> Is that already used? >>>> >>> >>> Actually, no. I noticed that it would a) require changes to all test jobs, >>> and b) prevent us from getting clean test runs. I prefer a safe and clean >>> run over a fast one. >>> >>> >> a) you can set CYTHON_RUNTESTS_CCACHE globaly somewhere inside ~/.profile > > ;) I actually set it globally in Jenkins. However, that doesn't > automatically remove the "rm -fr *" in the job configs - I initially > thought that would be a problem, but, apparently, it isn't. > > >>> Also, the test runner clears the target directory on startup. I wonder >>> where you got your numbers from... >>> >> By default ccache stores object files inside ~/.ccache directory > > Hmm, I guess changing that directory into a job local directory would still > require changing each job... Are there any concurrency issues with ccache? > > Anyway, I can't see a .ccache in my home directory, but if that's how it > works, it should at least be properly enabled now. Clearly makes a difference. https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py27-c/buildTimeTrend The .ccache directory is also there now, size is ~750M. I'll move it to the local scratch disc, CCACHE_DIR should do that. Does it make sense to delete it once a week, or should that be done more often? Stefan From vitja.makarov at gmail.com Thu Jul 7 11:00:47 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 7 Jul 2011 13:00:47 +0400 Subject: [Cython] Speedup hudson job with ccache In-Reply-To: <4E15724F.8070905@behnel.de> References: <4E155129.4010406@behnel.de> <4E15551D.9080304@behnel.de> <4E15604A.8080805@behnel.de> <4E15724F.8070905@behnel.de> Message-ID: 2011/7/7 Stefan Behnel : > Stefan Behnel, 07.07.2011 09:29: >> >> Vitja Makarov, 07.07.2011 08:54: >>> >>> 2011/7/7 Stefan Behnel >>> >>>> Vitja Makarov, 07.07.2011 08:32: >>>> >>>>> 2011/7/7 Stefan Behnel >>>>> >>>>> Robert Bradshaw, 06.07.2011 21:32: >>>>>> >>>>>> We should get ccache on sage.math to start using it on hudson. >>>>>> >>>>>>> >>>>>>> >>>>>> It's just a single binary when installed. I have a copy in my >>>>>> ~/ccache/bin. >>>>>> Changing the Jenkins jobs now. >>>>>> >>>>>> >>>>>> Is that already used? >>>>> >>>> >>>> Actually, no. I noticed that it would a) require changes to all test >>>> jobs, >>>> and b) prevent us from getting clean test runs. I prefer a safe and >>>> clean >>>> run over a fast one. >>>> >>>> >>> a) you can set CYTHON_RUNTESTS_CCACHE globaly somewhere inside ~/.profile >> >> ;) I actually set it globally in Jenkins. However, that doesn't >> automatically remove the "rm -fr *" in the job configs - I initially >> thought that would be a problem, but, apparently, it isn't. >> >> >>>> Also, the test runner clears the target directory on startup. I wonder >>>> where you got your numbers from... >>>> >>> By default ccache stores object files inside ~/.ccache directory >> >> Hmm, I guess changing that directory into a job local directory would >> still >> require changing each job... Are there any concurrency issues with ccache? >> >> Anyway, I can't see a .ccache in my home directory, but if that's how it >> works, it should at least be properly enabled now. > > Clearly makes a difference. > > https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py27-c/buildTimeTrend > > The .ccache directory is also there now, size is ~750M. I'll move it to the > local scratch disc, CCACHE_DIR should do that. Does it make sense to delete > it once a week, or should that be done more often? > Probably it's better to increase cache size, that will be shared between projects: $ ccache -s 5G (I don't know what's the right size to go) >From man ccache: """ CACHE SIZE MANAGEMENT By default ccache has a one gigabyte limit on the cache size and no maximum number of files. You can set a different limit using the "ccache -M" and "ccache -F" options, which set the size and number of files limits. When these limits are reached ccache will reduce the cache to 20% below the numbers you specified in order to avoid doing the cache clean opera? tion too often. """ Btw it's a good idea to cleanup cache once a week: $ ccache --cleanup (or --clear?) -- vitja. From stefan_ml at behnel.de Thu Jul 7 11:24:18 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jul 2011 11:24:18 +0200 Subject: [Cython] Speedup hudson job with ccache In-Reply-To: References: <4E155129.4010406@behnel.de> <4E15551D.9080304@behnel.de> <4E15604A.8080805@behnel.de> <4E15724F.8070905@behnel.de> Message-ID: <4E157B42.6090105@behnel.de> Vitja Makarov, 07.07.2011 11:00: > 2011/7/7 Stefan Behnel: >> Stefan Behnel, 07.07.2011 09:29: >> The .ccache directory is also there now, size is ~750M. I'll move it to the >> local scratch disc, CCACHE_DIR should do that. Does it make sense to delete >> it once a week, or should that be done more often? > > Probably it's better to increase cache size, that will be shared > between projects: > > $ ccache -s 5G (I don't know what's the right size to go) Done. I think 5GB is fine, it's a rather large disk. > Btw it's a good idea to cleanup cache once a week: > > $ ccache --cleanup (or --clear?) Added this to my crontab. Stefan From stefan_ml at behnel.de Thu Jul 7 15:28:36 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 07 Jul 2011 15:28:36 +0200 Subject: [Cython] hg-git shows 25 heads? Message-ID: <4E15B484.9040501@behnel.de> Hi, there's something broken with the repo recently. Even in a fresh checkout, I see 25 heads, most of which were supposed to be merges into the master branch (mainly button merges). The github web site doesn't show these, it only has three branches. This keeps me from pushing. A hg pull currently says that it gets 735 objects *each time*, without any visible changes, and a push says it would override a changeset. Any idea what may have gone wrong? Stefan From vitja.makarov at gmail.com Thu Jul 7 17:09:08 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 7 Jul 2011 19:09:08 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: 2011/7/6 Vitja Makarov : > 2011/7/6 Stefan Behnel : >> Vitja Makarov, 06.07.2011 09:05: >>> >>> 2011/7/6 Stefan Behnel: >>>> >>>> Stefan Behnel, 05.07.2011 10:04: >>>>> >>>>> Vitja Makarov, 05.07.2011 09:17: >>>>>> >>>>>> 2011/7/5 Stefan Behnel: >>>>>>> >>>>>>> Vitja Makarov, 05.07.2011 08:21: >>>>>>>> >>>>>>>> I was thinking about implementing new super() with no arguments. >>>>>>> >>>>>>> http://trac.cython.org/cython_trac/ticket/696 >>>>>>> >>>>>>>> The problem is where to store __class__, I see two options here: >>>>>>>> >>>>>>>> 1. Add func_class member to CyFunction, this way __class__ will be >>>>>>>> private and not visible for inner functions: >>>>>>>> 2. Put it into closure >>>>>>> >>>>>>> The second option has the advantage of requiring the field only when >>>>>>> super() >>>>>>> is used, whereas the first impacts all functions. >>>>>>> >>>>>>> I would expect that programs commonly have a lot more functions than >>>>>>> specifically methods that use a no-argument call to super(), so this >>>>>>> may >>>>>>> make a difference. >>>>>>> >>>>>> >>>>>> So, now classes are created the following way: >>>>>> >>>>>> class_dict = {} >>>>>> class_dict.foo = foo_func >>>>>> class = CreateClass(class_dict) >>>>>> >>>>>> So after class is created I should check its dict for CyFunction >>>>>> members (maybe only ones that actually require __class__) >>>>>> and set __class__: >>>>>> >>>>>> for value in class.__dict__.itervalues(): >>>>>> if isinstance(value, CyFunction) and value.func_class is WantClass: >>>>>> value.func_class = class >>>>>> >>>>>> Btw, first way requires cyfunction signature change, it would accept >>>>>> cyfunction object as first argument. >>>>> >>>>> We currently pass the binding (i.e. owning) object, right? >>>> >>>> So, how would this work for methods? We need to pass the 'self' object >>>> there, which the CyFunction doesn't know. If anything, it only knows the >>>> class it was defined in, which doesn't help here. >>> >>> From PEP: "super() is equivalent to: super(__class__,)" >> >> I wasn't speaking of super(). What I meant, was: how do we pass 'self' when >> we pass the CyFunction object as the first argument? >> > > > Oh, ok. Now we pass closure or nothing in self. So method's self is > passed via tuple. > Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we > can override this and change signature of cyfunction to: > > PyObject func(CyFunction *func, PyObject *self, PyObject *args, > PyObject *kwargs); > > This way we should implement new instancemethod type. > I've add support for new super() for cdef classes here: https://github.com/vitek/cython/compare/_new_super Pure python classes would require my _bindings branch. -- vitja. From robertwb at math.washington.edu Thu Jul 7 18:49:39 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 7 Jul 2011 09:49:39 -0700 Subject: [Cython] Speedup hudson job with ccache In-Reply-To: <4E157B42.6090105@behnel.de> References: <4E155129.4010406@behnel.de> <4E15551D.9080304@behnel.de> <4E15604A.8080805@behnel.de> <4E15724F.8070905@behnel.de> <4E157B42.6090105@behnel.de> Message-ID: On Thu, Jul 7, 2011 at 2:24 AM, Stefan Behnel wrote: > Vitja Makarov, 07.07.2011 11:00: >> >> 2011/7/7 Stefan Behnel: >>> >>> Stefan Behnel, 07.07.2011 09:29: >>> The .ccache directory is also there now, size is ~750M. I'll move it to >>> the >>> local scratch disc, CCACHE_DIR should do that. Does it make sense to >>> delete >>> it once a week, or should that be done more often? >> >> Probably it's better to increase cache size, that will be shared >> between projects: >> >> $ ccache -s 5G (I don't know what's the right size to go) > > Done. I think 5GB is fine, it's a rather large disk. This is well worth the time/load savings. I'll look at enabling this for Sage too, which should be a big win (it's been done before). >> Btw it's a good idea to cleanup cache once a week: >> >> $ ccache --cleanup (or --clear?) Why would this be needed, doesn't the cache get recycled naturally on its own? - Robert From pav at iki.fi Thu Jul 7 19:52:49 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 7 Jul 2011 17:52:49 +0000 (UTC) Subject: [Cython] hg-git shows 25 heads? References: <4E15B484.9040501@behnel.de> Message-ID: On Thu, 07 Jul 2011 15:28:36 +0200, Stefan Behnel wrote: > there's something broken with the repo recently. Even in a fresh > checkout, I see 25 heads, most of which were supposed to be merges into > the master branch (mainly button merges). The github web site doesn't > show these, it only has three branches. > > This keeps me from pushing. A hg pull currently says that it gets 735 > objects *each time*, without any visible changes, and a push says it > would override a changeset. > > Any idea what may have gone wrong? It seems that the Github's automatic merge feature creates hidden branches in the namespace refs/pull/* Git does not fetch these by default unless you tell it to, git fetch origin refs/*:refs/* The hidden branches are there, but you never see them in normal usage, and they are never pulled so they do not take space in clones. But maybe Mercurial's git plugin gets confused because of them? I'm not familiar with how it works, but maybe you can instruct it to track only branches in the default namespace refs/heads/* Pauli From d.s.seljebotn at astro.uio.no Thu Jul 7 20:10:30 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Thu, 07 Jul 2011 20:10:30 +0200 Subject: [Cython] hg-git shows 25 heads? In-Reply-To: References: <4E15B484.9040501@behnel.de> Message-ID: <4E15F696.9080308@astro.uio.no> On 07/07/2011 07:52 PM, Pauli Virtanen wrote: > On Thu, 07 Jul 2011 15:28:36 +0200, Stefan Behnel wrote: >> there's something broken with the repo recently. Even in a fresh >> checkout, I see 25 heads, most of which were supposed to be merges into >> the master branch (mainly button merges). The github web site doesn't >> show these, it only has three branches. >> >> This keeps me from pushing. A hg pull currently says that it gets 735 >> objects *each time*, without any visible changes, and a push says it >> would override a changeset. >> >> Any idea what may have gone wrong? > > It seems that the Github's automatic merge feature creates hidden > branches in the namespace > > refs/pull/* > > Git does not fetch these by default unless you tell it to, > > git fetch origin refs/*:refs/* > > The hidden branches are there, but you never see them in normal usage, > and they are never pulled so they do not take space in clones. > > But maybe Mercurial's git plugin gets confused because of them? > I'm not familiar with how it works, but maybe you can instruct it > to track only branches in the default namespace refs/heads/* Sounds like this should be a Github or hg-git bug report. 5 minutes of Googling didn't turn up anything... Do you think it would be a bad idea to simply delete these branches, something like git push origin :refs/* ? Dag Sverre From pav at iki.fi Thu Jul 7 21:06:42 2011 From: pav at iki.fi (Pauli Virtanen) Date: Thu, 7 Jul 2011 19:06:42 +0000 (UTC) Subject: [Cython] hg-git shows 25 heads? References: <4E15B484.9040501@behnel.de> <4E15F696.9080308@astro.uio.no> Message-ID: On Thu, 07 Jul 2011 20:10:30 +0200, Dag Sverre Seljebotn wrote: [clip] > 5 minutes of Googling didn't turn up anything... Do you think it would > be a bad idea to simply delete these branches, something like > > git push origin :refs/* That drops *all* branches -- not recommended :) As far as I see, git push origin :refs/pull/* should not break anything. However, depending on how the stuff is set up on the Github side, either the merge button stops working (would be an improvement to the state of matters, IMHO), or alternatively, the branches come back immediately whenever someone views pull requests. So I don't think doing this would help much. Maybe best to ask the Github guys what the heck is their creation doing here, and try to see if hg-git can be made to work properly. Pauli From markflorisson88 at gmail.com Thu Jul 7 21:44:16 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Thu, 7 Jul 2011 21:44:16 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: On 6 July 2011 10:01, Vitja Makarov wrote: > 2011/7/6 Stefan Behnel : >> Vitja Makarov, 06.07.2011 09:05: >>> >>> 2011/7/6 Stefan Behnel: >>>> >>>> Stefan Behnel, 05.07.2011 10:04: >>>>> >>>>> Vitja Makarov, 05.07.2011 09:17: >>>>>> >>>>>> 2011/7/5 Stefan Behnel: >>>>>>> >>>>>>> Vitja Makarov, 05.07.2011 08:21: >>>>>>>> >>>>>>>> I was thinking about implementing new super() with no arguments. >>>>>>> >>>>>>> http://trac.cython.org/cython_trac/ticket/696 >>>>>>> >>>>>>>> The problem is where to store __class__, I see two options here: >>>>>>>> >>>>>>>> 1. Add func_class member to CyFunction, this way __class__ will be >>>>>>>> private and not visible for inner functions: >>>>>>>> 2. Put it into closure >>>>>>> >>>>>>> The second option has the advantage of requiring the field only when >>>>>>> super() >>>>>>> is used, whereas the first impacts all functions. >>>>>>> >>>>>>> I would expect that programs commonly have a lot more functions than >>>>>>> specifically methods that use a no-argument call to super(), so this >>>>>>> may >>>>>>> make a difference. >>>>>>> >>>>>> >>>>>> So, now classes are created the following way: >>>>>> >>>>>> class_dict = {} >>>>>> class_dict.foo = foo_func >>>>>> class = CreateClass(class_dict) >>>>>> >>>>>> So after class is created I should check its dict for CyFunction >>>>>> members (maybe only ones that actually require __class__) >>>>>> and set __class__: >>>>>> >>>>>> for value in class.__dict__.itervalues(): >>>>>> if isinstance(value, CyFunction) and value.func_class is WantClass: >>>>>> value.func_class = class >>>>>> >>>>>> Btw, first way requires cyfunction signature change, it would accept >>>>>> cyfunction object as first argument. >>>>> >>>>> We currently pass the binding (i.e. owning) object, right? >>>> >>>> So, how would this work for methods? We need to pass the 'self' object >>>> there, which the CyFunction doesn't know. If anything, it only knows the >>>> class it was defined in, which doesn't help here. >>> >>> From PEP: "super() is equivalent to: super(__class__,)" >> >> I wasn't speaking of super(). What I meant, was: how do we pass 'self' when >> we pass the CyFunction object as the first argument? >> > > > Oh, ok. Now we pass closure or nothing in self. So method's self is > passed via tuple. > Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we > can override this and change signature of cyfunction to: > > PyObject func(CyFunction *func, PyObject *self, PyObject *args, > PyObject *kwargs); > > This way we should implement new instancemethod type. Would it be easier to make scope objects attributes of functions? Then you could still utilize PyCFunction_Call without needing to check the argument flags and such right? > -- > vitja. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From markflorisson88 at gmail.com Thu Jul 7 21:47:13 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Thu, 7 Jul 2011 21:47:13 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: On 7 July 2011 17:09, Vitja Makarov wrote: > 2011/7/6 Vitja Makarov : >> 2011/7/6 Stefan Behnel : >>> Vitja Makarov, 06.07.2011 09:05: >>>> >>>> 2011/7/6 Stefan Behnel: >>>>> >>>>> Stefan Behnel, 05.07.2011 10:04: >>>>>> >>>>>> Vitja Makarov, 05.07.2011 09:17: >>>>>>> >>>>>>> 2011/7/5 Stefan Behnel: >>>>>>>> >>>>>>>> Vitja Makarov, 05.07.2011 08:21: >>>>>>>>> >>>>>>>>> I was thinking about implementing new super() with no arguments. >>>>>>>> >>>>>>>> http://trac.cython.org/cython_trac/ticket/696 >>>>>>>> >>>>>>>>> The problem is where to store __class__, I see two options here: >>>>>>>>> >>>>>>>>> 1. Add func_class member to CyFunction, this way __class__ will be >>>>>>>>> private and not visible for inner functions: >>>>>>>>> 2. Put it into closure >>>>>>>> >>>>>>>> The second option has the advantage of requiring the field only when >>>>>>>> super() >>>>>>>> is used, whereas the first impacts all functions. >>>>>>>> >>>>>>>> I would expect that programs commonly have a lot more functions than >>>>>>>> specifically methods that use a no-argument call to super(), so this >>>>>>>> may >>>>>>>> make a difference. >>>>>>>> >>>>>>> >>>>>>> So, now classes are created the following way: >>>>>>> >>>>>>> class_dict = {} >>>>>>> class_dict.foo = foo_func >>>>>>> class = CreateClass(class_dict) >>>>>>> >>>>>>> So after class is created I should check its dict for CyFunction >>>>>>> members (maybe only ones that actually require __class__) >>>>>>> and set __class__: >>>>>>> >>>>>>> for value in class.__dict__.itervalues(): >>>>>>> if isinstance(value, CyFunction) and value.func_class is WantClass: >>>>>>> value.func_class = class >>>>>>> >>>>>>> Btw, first way requires cyfunction signature change, it would accept >>>>>>> cyfunction object as first argument. >>>>>> >>>>>> We currently pass the binding (i.e. owning) object, right? >>>>> >>>>> So, how would this work for methods? We need to pass the 'self' object >>>>> there, which the CyFunction doesn't know. If anything, it only knows the >>>>> class it was defined in, which doesn't help here. >>>> >>>> From PEP: "super() is equivalent to: super(__class__,)" >>> >>> I wasn't speaking of super(). What I meant, was: how do we pass 'self' when >>> we pass the CyFunction object as the first argument? >>> >> >> >> Oh, ok. Now we pass closure or nothing in self. So method's self is >> passed via tuple. >> Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we >> can override this and change signature of cyfunction to: >> >> PyObject func(CyFunction *func, PyObject *self, PyObject *args, >> PyObject *kwargs); >> >> This way we should implement new instancemethod type. >> > > I've add support for new super() for cdef classes here: > https://github.com/vitek/cython/compare/_new_super > > Pure python classes would require my _bindings branch. > I also need that branch to continue my work on fused types. The naming suggests I can't pull from it and start building on top of it? Did you have any plans to review it Stefan? Otherwise I won't mind giving it a go. > -- > vitja. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From vitja.makarov at gmail.com Thu Jul 7 22:15:31 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 8 Jul 2011 00:15:31 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: 2011/7/7 mark florisson > On 6 July 2011 10:01, Vitja Makarov wrote: > > 2011/7/6 Stefan Behnel : > >> Vitja Makarov, 06.07.2011 09:05: > >>> > >>> 2011/7/6 Stefan Behnel: > >>>> > >>>> Stefan Behnel, 05.07.2011 10:04: > >>>>> > >>>>> Vitja Makarov, 05.07.2011 09:17: > >>>>>> > >>>>>> 2011/7/5 Stefan Behnel: > >>>>>>> > >>>>>>> Vitja Makarov, 05.07.2011 08:21: > >>>>>>>> > >>>>>>>> I was thinking about implementing new super() with no arguments. > >>>>>>> > >>>>>>> http://trac.cython.org/cython_trac/ticket/696 > >>>>>>> > >>>>>>>> The problem is where to store __class__, I see two options here: > >>>>>>>> > >>>>>>>> 1. Add func_class member to CyFunction, this way __class__ will be > >>>>>>>> private and not visible for inner functions: > >>>>>>>> 2. Put it into closure > >>>>>>> > >>>>>>> The second option has the advantage of requiring the field only > when > >>>>>>> super() > >>>>>>> is used, whereas the first impacts all functions. > >>>>>>> > >>>>>>> I would expect that programs commonly have a lot more functions > than > >>>>>>> specifically methods that use a no-argument call to super(), so > this > >>>>>>> may > >>>>>>> make a difference. > >>>>>>> > >>>>>> > >>>>>> So, now classes are created the following way: > >>>>>> > >>>>>> class_dict = {} > >>>>>> class_dict.foo = foo_func > >>>>>> class = CreateClass(class_dict) > >>>>>> > >>>>>> So after class is created I should check its dict for CyFunction > >>>>>> members (maybe only ones that actually require __class__) > >>>>>> and set __class__: > >>>>>> > >>>>>> for value in class.__dict__.itervalues(): > >>>>>> if isinstance(value, CyFunction) and value.func_class is WantClass: > >>>>>> value.func_class = class > >>>>>> > >>>>>> Btw, first way requires cyfunction signature change, it would accept > >>>>>> cyfunction object as first argument. > >>>>> > >>>>> We currently pass the binding (i.e. owning) object, right? > >>>> > >>>> So, how would this work for methods? We need to pass the 'self' object > >>>> there, which the CyFunction doesn't know. If anything, it only knows > the > >>>> class it was defined in, which doesn't help here. > >>> > >>> From PEP: "super() is equivalent to: super(__class__,)" > >> > >> I wasn't speaking of super(). What I meant, was: how do we pass 'self' > when > >> we pass the CyFunction object as the first argument? > >> > > > > > > Oh, ok. Now we pass closure or nothing in self. So method's self is > > passed via tuple. > > Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we > > can override this and change signature of cyfunction to: > > > > PyObject func(CyFunction *func, PyObject *self, PyObject *args, > > PyObject *kwargs); > > > > This way we should implement new instancemethod type. > > Would it be easier to make scope objects attributes of functions? Then > you could still utilize PyCFunction_Call without needing to check the > argument flags and such right? > > Sure, scope object is already functions attribute now it's stored inside self. But, instancemethods desc_get builds new args tuple with self as first element. We can implement cython version of instance method and change method signature a little bit. -- vitja. -------------- next part -------------- An HTML attachment was scrubbed... URL: From markflorisson88 at gmail.com Thu Jul 7 22:30:26 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Thu, 7 Jul 2011 22:30:26 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: On 7 July 2011 22:15, Vitja Makarov wrote: > > > 2011/7/7 mark florisson >> >> On 6 July 2011 10:01, Vitja Makarov wrote: >> > 2011/7/6 Stefan Behnel : >> >> Vitja Makarov, 06.07.2011 09:05: >> >>> >> >>> 2011/7/6 Stefan Behnel: >> >>>> >> >>>> Stefan Behnel, 05.07.2011 10:04: >> >>>>> >> >>>>> Vitja Makarov, 05.07.2011 09:17: >> >>>>>> >> >>>>>> 2011/7/5 Stefan Behnel: >> >>>>>>> >> >>>>>>> Vitja Makarov, 05.07.2011 08:21: >> >>>>>>>> >> >>>>>>>> I was thinking about implementing new super() with no arguments. >> >>>>>>> >> >>>>>>> http://trac.cython.org/cython_trac/ticket/696 >> >>>>>>> >> >>>>>>>> The problem is where to store __class__, I see two options here: >> >>>>>>>> >> >>>>>>>> 1. Add func_class member to CyFunction, this way __class__ will >> >>>>>>>> be >> >>>>>>>> private and not visible for inner functions: >> >>>>>>>> 2. Put it into closure >> >>>>>>> >> >>>>>>> The second option has the advantage of requiring the field only >> >>>>>>> when >> >>>>>>> super() >> >>>>>>> is used, whereas the first impacts all functions. >> >>>>>>> >> >>>>>>> I would expect that programs commonly have a lot more functions >> >>>>>>> than >> >>>>>>> specifically methods that use a no-argument call to super(), so >> >>>>>>> this >> >>>>>>> may >> >>>>>>> make a difference. >> >>>>>>> >> >>>>>> >> >>>>>> So, now classes are created the following way: >> >>>>>> >> >>>>>> class_dict = {} >> >>>>>> class_dict.foo = foo_func >> >>>>>> class = CreateClass(class_dict) >> >>>>>> >> >>>>>> So after class is created I should check its dict for CyFunction >> >>>>>> members (maybe only ones that actually require __class__) >> >>>>>> and set __class__: >> >>>>>> >> >>>>>> for value in class.__dict__.itervalues(): >> >>>>>> if isinstance(value, CyFunction) and value.func_class is WantClass: >> >>>>>> value.func_class = class >> >>>>>> >> >>>>>> Btw, first way requires cyfunction signature change, it would >> >>>>>> accept >> >>>>>> cyfunction object as first argument. >> >>>>> >> >>>>> We currently pass the binding (i.e. owning) object, right? >> >>>> >> >>>> So, how would this work for methods? We need to pass the 'self' >> >>>> object >> >>>> there, which the CyFunction doesn't know. If anything, it only knows >> >>>> the >> >>>> class it was defined in, which doesn't help here. >> >>> >> >>> From PEP: "super() is equivalent to: super(__class__,)" >> >> >> >> I wasn't speaking of super(). What I meant, was: how do we pass 'self' >> >> when >> >> we pass the CyFunction object as the first argument? >> >> >> > >> > >> > Oh, ok. Now we pass closure or nothing in self. So method's self is >> > passed via tuple. >> > Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we >> > can override this and change signature of cyfunction to: >> > >> > PyObject func(CyFunction *func, PyObject *self, PyObject *args, >> > PyObject *kwargs); >> > >> > This way we should implement new instancemethod type. >> >> Would it be easier to make scope objects attributes of functions? Then >> you could still utilize PyCFunction_Call without needing to check the >> argument flags and such right? >> > > Sure, scope object is already functions attribute now it's stored inside > self. > But, instancemethods desc_get builds new args tuple with self as first > element. > We can implement cython version of instance method and change method > signature a little bit. Right, and you pass it in as the first argument to the C function. However, if you want to change the signature, you have to override PyCFunction_Call, which, if I'm not mistaken, means you will have to interpret the flags from the PyMethodDef and call the C function in the right way (e.g. with or without the args tuple and kwargs) and check if it's being called correctly. If you leave it as an attribute of the function, you can pass in the function and access the scope object from the function object in the closure C function. So I think changing the signature might be a bit harder? > -- > vitja. > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > From vitja.makarov at gmail.com Thu Jul 7 22:39:54 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 8 Jul 2011 00:39:54 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: 2011/7/8 mark florisson > On 7 July 2011 22:15, Vitja Makarov wrote: > > > > > > 2011/7/7 mark florisson > >> > >> On 6 July 2011 10:01, Vitja Makarov wrote: > >> > 2011/7/6 Stefan Behnel : > >> >> Vitja Makarov, 06.07.2011 09:05: > >> >>> > >> >>> 2011/7/6 Stefan Behnel: > >> >>>> > >> >>>> Stefan Behnel, 05.07.2011 10:04: > >> >>>>> > >> >>>>> Vitja Makarov, 05.07.2011 09:17: > >> >>>>>> > >> >>>>>> 2011/7/5 Stefan Behnel: > >> >>>>>>> > >> >>>>>>> Vitja Makarov, 05.07.2011 08:21: > >> >>>>>>>> > >> >>>>>>>> I was thinking about implementing new super() with no > arguments. > >> >>>>>>> > >> >>>>>>> http://trac.cython.org/cython_trac/ticket/696 > >> >>>>>>> > >> >>>>>>>> The problem is where to store __class__, I see two options > here: > >> >>>>>>>> > >> >>>>>>>> 1. Add func_class member to CyFunction, this way __class__ will > >> >>>>>>>> be > >> >>>>>>>> private and not visible for inner functions: > >> >>>>>>>> 2. Put it into closure > >> >>>>>>> > >> >>>>>>> The second option has the advantage of requiring the field only > >> >>>>>>> when > >> >>>>>>> super() > >> >>>>>>> is used, whereas the first impacts all functions. > >> >>>>>>> > >> >>>>>>> I would expect that programs commonly have a lot more functions > >> >>>>>>> than > >> >>>>>>> specifically methods that use a no-argument call to super(), so > >> >>>>>>> this > >> >>>>>>> may > >> >>>>>>> make a difference. > >> >>>>>>> > >> >>>>>> > >> >>>>>> So, now classes are created the following way: > >> >>>>>> > >> >>>>>> class_dict = {} > >> >>>>>> class_dict.foo = foo_func > >> >>>>>> class = CreateClass(class_dict) > >> >>>>>> > >> >>>>>> So after class is created I should check its dict for CyFunction > >> >>>>>> members (maybe only ones that actually require __class__) > >> >>>>>> and set __class__: > >> >>>>>> > >> >>>>>> for value in class.__dict__.itervalues(): > >> >>>>>> if isinstance(value, CyFunction) and value.func_class is > WantClass: > >> >>>>>> value.func_class = class > >> >>>>>> > >> >>>>>> Btw, first way requires cyfunction signature change, it would > >> >>>>>> accept > >> >>>>>> cyfunction object as first argument. > >> >>>>> > >> >>>>> We currently pass the binding (i.e. owning) object, right? > >> >>>> > >> >>>> So, how would this work for methods? We need to pass the 'self' > >> >>>> object > >> >>>> there, which the CyFunction doesn't know. If anything, it only > knows > >> >>>> the > >> >>>> class it was defined in, which doesn't help here. > >> >>> > >> >>> From PEP: "super() is equivalent to: super(__class__,)" > >> >> > >> >> I wasn't speaking of super(). What I meant, was: how do we pass > 'self' > >> >> when > >> >> we pass the CyFunction object as the first argument? > >> >> > >> > > >> > > >> > Oh, ok. Now we pass closure or nothing in self. So method's self is > >> > passed via tuple. > >> > Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call we > >> > can override this and change signature of cyfunction to: > >> > > >> > PyObject func(CyFunction *func, PyObject *self, PyObject *args, > >> > PyObject *kwargs); > >> > > >> > This way we should implement new instancemethod type. > >> > >> Would it be easier to make scope objects attributes of functions? Then > >> you could still utilize PyCFunction_Call without needing to check the > >> argument flags and such right? > >> > > > > Sure, scope object is already functions attribute now it's stored inside > > self. > > But, instancemethods desc_get builds new args tuple with self as first > > element. > > We can implement cython version of instance method and change method > > signature a little bit. > > Right, and you pass it in as the first argument to the C function. > However, if you want to change the signature, you have to override > PyCFunction_Call, which, if I'm not mistaken, means you will have to > interpret the flags from the PyMethodDef and call the C function in > the right way (e.g. with or without the args tuple and kwargs) and > check if it's being called correctly. If you leave it as an attribute > of the function, you can pass in the function and access the scope > object from the function object in the closure C function. So I think > changing the signature might be a bit harder? > > Yes, but how to handle instantmethods self arg? We can't store it inside function object as it is different for each instance. btw I've already moved pycyfunction_call into cython.. -- vitja. -------------- next part -------------- An HTML attachment was scrubbed... URL: From markflorisson88 at gmail.com Thu Jul 7 22:50:20 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Thu, 7 Jul 2011 22:50:20 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: On 7 July 2011 22:39, Vitja Makarov wrote: > > > 2011/7/8 mark florisson >> >> On 7 July 2011 22:15, Vitja Makarov wrote: >> > >> > >> > 2011/7/7 mark florisson >> >> >> >> On 6 July 2011 10:01, Vitja Makarov wrote: >> >> > 2011/7/6 Stefan Behnel : >> >> >> Vitja Makarov, 06.07.2011 09:05: >> >> >>> >> >> >>> 2011/7/6 Stefan Behnel: >> >> >>>> >> >> >>>> Stefan Behnel, 05.07.2011 10:04: >> >> >>>>> >> >> >>>>> Vitja Makarov, 05.07.2011 09:17: >> >> >>>>>> >> >> >>>>>> 2011/7/5 Stefan Behnel: >> >> >>>>>>> >> >> >>>>>>> Vitja Makarov, 05.07.2011 08:21: >> >> >>>>>>>> >> >> >>>>>>>> I was thinking about implementing new super() with no >> >> >>>>>>>> arguments. >> >> >>>>>>> >> >> >>>>>>> http://trac.cython.org/cython_trac/ticket/696 >> >> >>>>>>> >> >> >>>>>>>> The problem is where to store __class__, I see two options >> >> >>>>>>>> here: >> >> >>>>>>>> >> >> >>>>>>>> 1. Add func_class member to CyFunction, this way __class__ >> >> >>>>>>>> will >> >> >>>>>>>> be >> >> >>>>>>>> private and not visible for inner functions: >> >> >>>>>>>> 2. Put it into closure >> >> >>>>>>> >> >> >>>>>>> The second option has the advantage of requiring the field only >> >> >>>>>>> when >> >> >>>>>>> super() >> >> >>>>>>> is used, whereas the first impacts all functions. >> >> >>>>>>> >> >> >>>>>>> I would expect that programs commonly have a lot more functions >> >> >>>>>>> than >> >> >>>>>>> specifically methods that use a no-argument call to super(), so >> >> >>>>>>> this >> >> >>>>>>> may >> >> >>>>>>> make a difference. >> >> >>>>>>> >> >> >>>>>> >> >> >>>>>> So, now classes are created the following way: >> >> >>>>>> >> >> >>>>>> class_dict = {} >> >> >>>>>> class_dict.foo = foo_func >> >> >>>>>> class = CreateClass(class_dict) >> >> >>>>>> >> >> >>>>>> So after class is created I should check its dict for CyFunction >> >> >>>>>> members (maybe only ones that actually require __class__) >> >> >>>>>> and set __class__: >> >> >>>>>> >> >> >>>>>> for value in class.__dict__.itervalues(): >> >> >>>>>> if isinstance(value, CyFunction) and value.func_class is >> >> >>>>>> WantClass: >> >> >>>>>> value.func_class = class >> >> >>>>>> >> >> >>>>>> Btw, first way requires cyfunction signature change, it would >> >> >>>>>> accept >> >> >>>>>> cyfunction object as first argument. >> >> >>>>> >> >> >>>>> We currently pass the binding (i.e. owning) object, right? >> >> >>>> >> >> >>>> So, how would this work for methods? We need to pass the 'self' >> >> >>>> object >> >> >>>> there, which the CyFunction doesn't know. If anything, it only >> >> >>>> knows >> >> >>>> the >> >> >>>> class it was defined in, which doesn't help here. >> >> >>> >> >> >>> From PEP: "super() is equivalent to: super(__class__,)" >> >> >> >> >> >> I wasn't speaking of super(). What I meant, was: how do we pass >> >> >> 'self' >> >> >> when >> >> >> we pass the CyFunction object as the first argument? >> >> >> >> >> > >> >> > >> >> > Oh, ok. Now we pass closure or nothing in self. So method's self is >> >> > passed via tuple. >> >> > Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call >> >> > we >> >> > can override this and change signature of cyfunction to: >> >> > >> >> > PyObject func(CyFunction *func, PyObject *self, PyObject *args, >> >> > PyObject *kwargs); >> >> > >> >> > This way we should implement new instancemethod type. >> >> >> >> Would it be easier to make scope objects attributes of functions? Then >> >> you could still utilize PyCFunction_Call without needing to check the >> >> argument flags and such right? >> >> >> > >> > Sure, scope object is already functions attribute now it's stored inside >> > self. >> > But, instancemethods desc_get builds new args tuple with self as first >> > element. >> > We can implement cython version of instance method and change method >> > signature a little bit. >> >> Right, and you pass it in as the first argument to the C function. >> However, if you want to change the signature, you have to override >> PyCFunction_Call, which, if I'm not mistaken, means you will have to >> interpret the flags from the PyMethodDef and call the C function in >> the right way (e.g. with or without the args tuple and kwargs) and >> check if it's being called correctly. If you leave it as an attribute >> of the function, you can pass in the function and access the scope >> object from the function object in the closure C function. So I think >> changing the signature might be a bit harder? >> > > Yes, but how to handle instantmethods self arg? > We can't store it inside function object as it is different for each > instance. In descr_get you create and return a new CyFunction with a __self__ set (note, not m_self) and in __call__ you put __self__ in the args tuple and the function as m_self, and then call the CyFunction using PyCFunction_Call. But copying and modifying PyCFunction_Call works just as well I suppose :) Anyway, the important question was whether I could pull from your branch or whether I should wait for the merge. > btw I've already moved pycyfunction_call into cython.. > -- > vitja. > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > From vitja.makarov at gmail.com Thu Jul 7 23:13:04 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 8 Jul 2011 01:13:04 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: 2011/7/8 mark florisson > On 7 July 2011 22:39, Vitja Makarov wrote: > > > > > > 2011/7/8 mark florisson > >> > >> On 7 July 2011 22:15, Vitja Makarov wrote: > >> > > >> > > >> > 2011/7/7 mark florisson > >> >> > >> >> On 6 July 2011 10:01, Vitja Makarov wrote: > >> >> > 2011/7/6 Stefan Behnel : > >> >> >> Vitja Makarov, 06.07.2011 09:05: > >> >> >>> > >> >> >>> 2011/7/6 Stefan Behnel: > >> >> >>>> > >> >> >>>> Stefan Behnel, 05.07.2011 10:04: > >> >> >>>>> > >> >> >>>>> Vitja Makarov, 05.07.2011 09:17: > >> >> >>>>>> > >> >> >>>>>> 2011/7/5 Stefan Behnel: > >> >> >>>>>>> > >> >> >>>>>>> Vitja Makarov, 05.07.2011 08:21: > >> >> >>>>>>>> > >> >> >>>>>>>> I was thinking about implementing new super() with no > >> >> >>>>>>>> arguments. > >> >> >>>>>>> > >> >> >>>>>>> http://trac.cython.org/cython_trac/ticket/696 > >> >> >>>>>>> > >> >> >>>>>>>> The problem is where to store __class__, I see two options > >> >> >>>>>>>> here: > >> >> >>>>>>>> > >> >> >>>>>>>> 1. Add func_class member to CyFunction, this way __class__ > >> >> >>>>>>>> will > >> >> >>>>>>>> be > >> >> >>>>>>>> private and not visible for inner functions: > >> >> >>>>>>>> 2. Put it into closure > >> >> >>>>>>> > >> >> >>>>>>> The second option has the advantage of requiring the field > only > >> >> >>>>>>> when > >> >> >>>>>>> super() > >> >> >>>>>>> is used, whereas the first impacts all functions. > >> >> >>>>>>> > >> >> >>>>>>> I would expect that programs commonly have a lot more > functions > >> >> >>>>>>> than > >> >> >>>>>>> specifically methods that use a no-argument call to super(), > so > >> >> >>>>>>> this > >> >> >>>>>>> may > >> >> >>>>>>> make a difference. > >> >> >>>>>>> > >> >> >>>>>> > >> >> >>>>>> So, now classes are created the following way: > >> >> >>>>>> > >> >> >>>>>> class_dict = {} > >> >> >>>>>> class_dict.foo = foo_func > >> >> >>>>>> class = CreateClass(class_dict) > >> >> >>>>>> > >> >> >>>>>> So after class is created I should check its dict for > CyFunction > >> >> >>>>>> members (maybe only ones that actually require __class__) > >> >> >>>>>> and set __class__: > >> >> >>>>>> > >> >> >>>>>> for value in class.__dict__.itervalues(): > >> >> >>>>>> if isinstance(value, CyFunction) and value.func_class is > >> >> >>>>>> WantClass: > >> >> >>>>>> value.func_class = class > >> >> >>>>>> > >> >> >>>>>> Btw, first way requires cyfunction signature change, it would > >> >> >>>>>> accept > >> >> >>>>>> cyfunction object as first argument. > >> >> >>>>> > >> >> >>>>> We currently pass the binding (i.e. owning) object, right? > >> >> >>>> > >> >> >>>> So, how would this work for methods? We need to pass the 'self' > >> >> >>>> object > >> >> >>>> there, which the CyFunction doesn't know. If anything, it only > >> >> >>>> knows > >> >> >>>> the > >> >> >>>> class it was defined in, which doesn't help here. > >> >> >>> > >> >> >>> From PEP: "super() is equivalent to: super(__class__,)" > >> >> >> > >> >> >> I wasn't speaking of super(). What I meant, was: how do we pass > >> >> >> 'self' > >> >> >> when > >> >> >> we pass the CyFunction object as the first argument? > >> >> >> > >> >> > > >> >> > > >> >> > Oh, ok. Now we pass closure or nothing in self. So method's self is > >> >> > passed via tuple. > >> >> > Instancemethod do this for us. Now CyFucntion uses PyCFunction_Call > >> >> > we > >> >> > can override this and change signature of cyfunction to: > >> >> > > >> >> > PyObject func(CyFunction *func, PyObject *self, PyObject *args, > >> >> > PyObject *kwargs); > >> >> > > >> >> > This way we should implement new instancemethod type. > >> >> > >> >> Would it be easier to make scope objects attributes of functions? > Then > >> >> you could still utilize PyCFunction_Call without needing to check the > >> >> argument flags and such right? > >> >> > >> > > >> > Sure, scope object is already functions attribute now it's stored > inside > >> > self. > >> > But, instancemethods desc_get builds new args tuple with self as first > >> > element. > >> > We can implement cython version of instance method and change method > >> > signature a little bit. > >> > >> Right, and you pass it in as the first argument to the C function. > >> However, if you want to change the signature, you have to override > >> PyCFunction_Call, which, if I'm not mistaken, means you will have to > >> interpret the flags from the PyMethodDef and call the C function in > >> the right way (e.g. with or without the args tuple and kwargs) and > >> check if it's being called correctly. If you leave it as an attribute > >> of the function, you can pass in the function and access the scope > >> object from the function object in the closure C function. So I think > >> changing the signature might be a bit harder? > >> > > > > Yes, but how to handle instantmethods self arg? > > We can't store it inside function object as it is different for each > > instance. > > In descr_get you create and return a new CyFunction with a __self__ > set (note, not m_self) and in __call__ you put __self__ in the args > tuple and the function as m_self, and then call the CyFunction using > PyCFunction_Call. But copying and modifying PyCFunction_Call works > just as well I suppose :) > Yes, that would work) But I think CyFunction is too heavy for methods. I've just realized that descr_get is called each time you access method, each time you call it and so on. > Anyway, the important question was whether I could pull from your > branch or whether I should wait for the merge. > > It's better not to use my branch this way as it's marked with underscore and sometimes I do forced pushes. I hope it will be merged soon. There is one more thing left CPython uses pool for frequently used objects to avoid unnecessary memory allocations. Probably we should implement something like that too. You are going to subclass cyfunction and then override its tp_call method, right? -- vitja. -------------- next part -------------- An HTML attachment was scrubbed... URL: From markflorisson88 at gmail.com Thu Jul 7 23:23:15 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Thu, 7 Jul 2011 23:23:15 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: On 7 July 2011 23:13, Vitja Makarov wrote: > > > 2011/7/8 mark florisson >> >> On 7 July 2011 22:39, Vitja Makarov wrote: >> > >> > >> > 2011/7/8 mark florisson >> >> >> >> On 7 July 2011 22:15, Vitja Makarov wrote: >> >> > >> >> > >> >> > 2011/7/7 mark florisson >> >> >> >> >> >> On 6 July 2011 10:01, Vitja Makarov wrote: >> >> >> > 2011/7/6 Stefan Behnel : >> >> >> >> Vitja Makarov, 06.07.2011 09:05: >> >> >> >>> >> >> >> >>> 2011/7/6 Stefan Behnel: >> >> >> >>>> >> >> >> >>>> Stefan Behnel, 05.07.2011 10:04: >> >> >> >>>>> >> >> >> >>>>> Vitja Makarov, 05.07.2011 09:17: >> >> >> >>>>>> >> >> >> >>>>>> 2011/7/5 Stefan Behnel: >> >> >> >>>>>>> >> >> >> >>>>>>> Vitja Makarov, 05.07.2011 08:21: >> >> >> >>>>>>>> >> >> >> >>>>>>>> I was thinking about implementing new super() with no >> >> >> >>>>>>>> arguments. >> >> >> >>>>>>> >> >> >> >>>>>>> http://trac.cython.org/cython_trac/ticket/696 >> >> >> >>>>>>> >> >> >> >>>>>>>> The problem is where to store __class__, I see two options >> >> >> >>>>>>>> here: >> >> >> >>>>>>>> >> >> >> >>>>>>>> 1. Add func_class member to CyFunction, this way __class__ >> >> >> >>>>>>>> will >> >> >> >>>>>>>> be >> >> >> >>>>>>>> private and not visible for inner functions: >> >> >> >>>>>>>> 2. Put it into closure >> >> >> >>>>>>> >> >> >> >>>>>>> The second option has the advantage of requiring the field >> >> >> >>>>>>> only >> >> >> >>>>>>> when >> >> >> >>>>>>> super() >> >> >> >>>>>>> is used, whereas the first impacts all functions. >> >> >> >>>>>>> >> >> >> >>>>>>> I would expect that programs commonly have a lot more >> >> >> >>>>>>> functions >> >> >> >>>>>>> than >> >> >> >>>>>>> specifically methods that use a no-argument call to super(), >> >> >> >>>>>>> so >> >> >> >>>>>>> this >> >> >> >>>>>>> may >> >> >> >>>>>>> make a difference. >> >> >> >>>>>>> >> >> >> >>>>>> >> >> >> >>>>>> So, now classes are created the following way: >> >> >> >>>>>> >> >> >> >>>>>> class_dict = {} >> >> >> >>>>>> class_dict.foo = foo_func >> >> >> >>>>>> class = CreateClass(class_dict) >> >> >> >>>>>> >> >> >> >>>>>> So after class is created I should check its dict for >> >> >> >>>>>> CyFunction >> >> >> >>>>>> members (maybe only ones that actually require __class__) >> >> >> >>>>>> and set __class__: >> >> >> >>>>>> >> >> >> >>>>>> for value in class.__dict__.itervalues(): >> >> >> >>>>>> if isinstance(value, CyFunction) and value.func_class is >> >> >> >>>>>> WantClass: >> >> >> >>>>>> value.func_class = class >> >> >> >>>>>> >> >> >> >>>>>> Btw, first way requires cyfunction signature change, it would >> >> >> >>>>>> accept >> >> >> >>>>>> cyfunction object as first argument. >> >> >> >>>>> >> >> >> >>>>> We currently pass the binding (i.e. owning) object, right? >> >> >> >>>> >> >> >> >>>> So, how would this work for methods? We need to pass the 'self' >> >> >> >>>> object >> >> >> >>>> there, which the CyFunction doesn't know. If anything, it only >> >> >> >>>> knows >> >> >> >>>> the >> >> >> >>>> class it was defined in, which doesn't help here. >> >> >> >>> >> >> >> >>> From PEP: "super() is equivalent to: >> >> >> >>> super(__class__,)" >> >> >> >> >> >> >> >> I wasn't speaking of super(). What I meant, was: how do we pass >> >> >> >> 'self' >> >> >> >> when >> >> >> >> we pass the CyFunction object as the first argument? >> >> >> >> >> >> >> > >> >> >> > >> >> >> > Oh, ok. Now we pass closure or nothing in self. So method's self >> >> >> > is >> >> >> > passed via tuple. >> >> >> > Instancemethod do this for us. Now CyFucntion uses >> >> >> > PyCFunction_Call >> >> >> > we >> >> >> > can override this and change signature of cyfunction to: >> >> >> > >> >> >> > PyObject func(CyFunction *func, PyObject *self, PyObject *args, >> >> >> > PyObject *kwargs); >> >> >> > >> >> >> > This way we should implement new instancemethod type. >> >> >> >> >> >> Would it be easier to make scope objects attributes of functions? >> >> >> Then >> >> >> you could still utilize PyCFunction_Call without needing to check >> >> >> the >> >> >> argument flags and such right? >> >> >> >> >> > >> >> > Sure, scope object is already functions attribute now it's stored >> >> > inside >> >> > self. >> >> > But, instancemethods desc_get builds new args tuple with self as >> >> > first >> >> > element. >> >> > We can implement cython version of instance method and change method >> >> > signature a little bit. >> >> >> >> Right, and you pass it in as the first argument to the C function. >> >> However, if you want to change the signature, you have to override >> >> PyCFunction_Call, which, if I'm not mistaken, means you will have to >> >> interpret the flags from the PyMethodDef and call the C function in >> >> the right way (e.g. with or without the args tuple and kwargs) and >> >> check if it's being called correctly. If you leave it as an attribute >> >> of the function, you can pass in the function and access the scope >> >> object from the function object in the closure C function. So I think >> >> changing the signature might be a bit harder? >> >> >> > >> > Yes, but how to handle instantmethods self arg? >> > We can't store it inside function object as it is different for each >> > instance. >> >> In descr_get you create and return a new CyFunction with a __self__ >> set (note, not m_self) and in __call__ you put __self__ in the args >> tuple and the function as m_self, and then call the CyFunction using >> PyCFunction_Call. But copying and modifying PyCFunction_Call works >> just as well I suppose :) > > Yes, that would work) But I think CyFunction is too heavy for methods. I thought the python-like behaviour of them was appealing enough to consider defaulting to it? Otherwise you'd only have to use them for super(), __class__ and closures. > I've just realized that descr_get is called each time you access method, > each time you call it and so on. > >> >> Anyway, the important question was whether I could pull from your >> branch or whether I should wait for the merge. >> > > It's better not to use my branch this way as it's marked with underscore and > sometimes I do forced pushes. > I hope it will be merged soon. > There is one more thing left CPython uses pool for?frequently used objects > to avoid?unnecessary memory allocations. > Probably we should implement something like that too. > > You are going to subclass cyfunction and then override its tp_call method, > right? Yes, exactly. The thing is that I only need to use it in extension methods, where the self for the method is passed in as m_self instead of in the args tuple. But I don't think a signature change will inconvenience that, so I think it's fine :) > -- > vitja. > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > From L.J.Buitinck at uva.nl Thu Jul 7 23:25:23 2011 From: L.J.Buitinck at uva.nl (Lars Buitinck) Date: Thu, 7 Jul 2011 23:25:23 +0200 Subject: [Cython] BUG: Cython's dies with AttributeError Message-ID: Dear developers, I just got an error message from Cython (current Git). The error can be reproduced by putting cdef foo(): pass in a file called foo.pyx, and compiling that. (I know there's an error in the file as well.) Would you kindly look into this bug? The error message was: foo.pyx:1:9: Compiler crash in PostParse ModuleNode.body = StatListNode(foo.pyx:1:5) StatListNode.stats[0] = CFuncDefNode(foo.pyx:1:5, modifiers = [...]/0, visibility = u'private') CFuncDefNode.declarator = CFuncDeclaratorNode(foo.pyx:1:9, calling_convention = '') CFuncDeclaratorNode.base = CNameDeclaratorNode(foo.pyx:1:9, calling_convention = u'', name = u'foo') Compiler crash traceback from this point on: File "Visitor.py", line 173, in Cython.Compiler.Visitor.TreeVisitor._visitchild (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:3550) File "Visitor.py", line 282, in Cython.Compiler.Visitor.CythonTransform.visit_Node (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:5242) File "Visitor.py", line 234, in Cython.Compiler.Visitor.VisitorTransform.visitchildren (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:4395) File "Visitor.py", line 202, in Cython.Compiler.Visitor.TreeVisitor._visitchildren (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:4076) AttributeError: 'CNameDeclaratorNode' object has no attribute 'base' foo.pyx:1:9: Compiler crash in PostParse ModuleNode.body = StatListNode(foo.pyx:1:5) StatListNode.stats[0] = CFuncDefNode(foo.pyx:1:5, modifiers = [...]/0, visibility = u'private') CFuncDefNode.declarator = CFuncDeclaratorNode(foo.pyx:1:9, calling_convention = '') CFuncDeclaratorNode.base = CNameDeclaratorNode(foo.pyx:1:9, calling_convention = u'', name = u'foo') Compiler crash traceback from this point on: File "Visitor.py", line 173, in Cython.Compiler.Visitor.TreeVisitor._visitchild (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:3550) File "Visitor.py", line 282, in Cython.Compiler.Visitor.CythonTransform.visit_Node (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:5242) File "Visitor.py", line 234, in Cython.Compiler.Visitor.VisitorTransform.visitchildren (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:4395) File "Visitor.py", line 202, in Cython.Compiler.Visitor.TreeVisitor._visitchildren (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:4076) AttributeError: 'CNameDeclaratorNode' object has no attribute 'base' -- Lars Buitinck Scientific programmer, ILPS University of Amsterdam From vitja.makarov at gmail.com Tue Jul 12 09:46:21 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 12 Jul 2011 11:46:21 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: <4E140D1F.5000809@behnel.de> References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: 2011/7/6 Stefan Behnel > Vitja Makarov, 06.07.2011 09:05: > > 2011/7/6 Stefan Behnel: >> >>> Stefan Behnel, 05.07.2011 10:04: >>> >>>> >>>> Vitja Makarov, 05.07.2011 09:17: >>>> >>>>> >>>>> 2011/7/5 Stefan Behnel: >>>>> >>>>>> >>>>>> Vitja Makarov, 05.07.2011 08:21: >>>>>> >>>>>>> >>>>>>> I was thinking about implementing new super() with no arguments. >>>>>>> >>>>>> >>>>>> http://trac.cython.org/cython_**trac/ticket/696 >>>>>> >>>>>> The problem is where to store __class__, I see two options here: >>>>>>> >>>>>>> 1. Add func_class member to CyFunction, this way __class__ will be >>>>>>> private and not visible for inner functions: >>>>>>> 2. Put it into closure >>>>>>> >>>>>> >>>>>> The second option has the advantage of requiring the field only when >>>>>> super() >>>>>> is used, whereas the first impacts all functions. >>>>>> >>>>>> I would expect that programs commonly have a lot more functions than >>>>>> specifically methods that use a no-argument call to super(), so this >>>>>> may >>>>>> make a difference. >>>>>> >>>>>> >>>>> So, now classes are created the following way: >>>>> >>>>> class_dict = {} >>>>> class_dict.foo = foo_func >>>>> class = CreateClass(class_dict) >>>>> >>>>> So after class is created I should check its dict for CyFunction >>>>> members (maybe only ones that actually require __class__) >>>>> and set __class__: >>>>> >>>>> for value in class.__dict__.itervalues(): >>>>> if isinstance(value, CyFunction) and value.func_class is WantClass: >>>>> value.func_class = class >>>>> >>>>> Btw, first way requires cyfunction signature change, it would accept >>>>> cyfunction object as first argument. >>>>> >>>> >>>> We currently pass the binding (i.e. owning) object, right? >>>> >>> >>> So, how would this work for methods? We need to pass the 'self' object >>> there, which the CyFunction doesn't know. If anything, it only knows the >>> class it was defined in, which doesn't help here. >>> >> >> From PEP: "super() is equivalent to: super(__class__,)" >> > > I wasn't speaking of super(). What I meant, was: how do we pass 'self' when > we pass the CyFunction object as the first argument? > > > About cdef classes: probably it's better to transform super().method(...) into direct form, e.g. BaseClass.method(self, ...) That would be more cdef-like than actually calling super. Btw, super() should still be used if no method call pattern is matched. -- vitja. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Tue Jul 12 10:16:39 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 12 Jul 2011 10:16:39 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: <4E1C02E7.2000701@behnel.de> Vitja Makarov, 12.07.2011 09:46: > About cdef classes: probably it's better to > transform super().method(...) into direct form, e.g. BaseClass.method(self, > ...) Except when it doesn't work. ;) A / \ B C \ / D The MRO here is D-B-A-C. If C unconditionally calls A.method(), A's implementation will be called twice, as it was already called by B. http://www.python.org/download/releases/2.2/descrintro/#mro Stefan From vitja.makarov at gmail.com Tue Jul 12 11:46:03 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 12 Jul 2011 13:46:03 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: <4E1C02E7.2000701@behnel.de> References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> <4E1C02E7.2000701@behnel.de> Message-ID: 2011/7/12 Stefan Behnel > > Vitja Makarov, 12.07.2011 09:46: >> >> About cdef classes: probably it's better to >> transform super().method(...) into direct form, e.g. BaseClass.method(self, >> ...) > > Except when it doesn't work. ;) > > ? A > ?/ \ > ?B ?C > ?\ / > ? D > > The MRO here is D-B-A-C. If C unconditionally calls A.method(), A's implementation will be called twice, as it was already called by B. > > http://www.python.org/download/releases/2.2/descrintro/#mro According to your link mro in the example is DBCA ;) Is that a problem for cdef classes? Cdef class have only one base, isn't it? -- vitja. From markflorisson88 at gmail.com Tue Jul 12 12:11:00 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 12 Jul 2011 12:11:00 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> <4E1C02E7.2000701@behnel.de> Message-ID: On 12 July 2011 11:46, Vitja Makarov wrote: > 2011/7/12 Stefan Behnel >> >> Vitja Makarov, 12.07.2011 09:46: >>> >>> About cdef classes: probably it's better to >>> transform super().method(...) into direct form, e.g. BaseClass.method(self, >>> ...) >> >> Except when it doesn't work. ;) >> >> ? A >> ?/ \ >> ?B ?C >> ?\ / >> ? D >> >> The MRO here is D-B-A-C. If C unconditionally calls A.method(), A's implementation will be called twice, as it was already called by B. >> >> http://www.python.org/download/releases/2.2/descrintro/#mro > > According to your link mro in the example is DBCA ;) > > Is that a problem for cdef classes? Cdef class have only one base, isn't it? That is irrelevant, if B calls A's method then C's method will be skipped. If it would use super() it would call C's method, and C's method would call A's method. > -- > vitja. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Tue Jul 12 12:42:44 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 12 Jul 2011 12:42:44 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> <4E1C02E7.2000701@behnel.de> Message-ID: <4E1C2524.2050809@behnel.de> mark florisson, 12.07.2011 12:11: > On 12 July 2011 11:46, Vitja Makarov wrote: >> 2011/7/12 Stefan Behnel >>> >>> Vitja Makarov, 12.07.2011 09:46: >>>> >>>> About cdef classes: probably it's better to >>>> transform super().method(...) into direct form, e.g. BaseClass.method(self, >>>> ...) >>> >>> Except when it doesn't work. ;) >>> >>> A >>> / \ >>> B C >>> \ / >>> D >>> >>> The MRO here is D-B-A-C. If C unconditionally calls A.method(), A's implementation will be called twice, as it was already called by B. >>> >>> http://www.python.org/download/releases/2.2/descrintro/#mro >> >> According to your link mro in the example is DBCA ;) Totally proves my point. ;) >> Is that a problem for cdef classes? Cdef class have only one base, isn't it? > > That is irrelevant, if B calls A's method then C's method will be > skipped. If it would use super() it would call C's method, and C's > method would call A's method. Exactly. So, calling directly into the base classes method implementation (i.e. issuing a direct C function call) makes a suitable optimisation for "final" classes, but not for the general case. Stefan From vitja.makarov at gmail.com Tue Jul 12 12:45:36 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 12 Jul 2011 14:45:36 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: <4E1C2524.2050809@behnel.de> References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> <4E1C02E7.2000701@behnel.de> <4E1C2524.2050809@behnel.de> Message-ID: 2011/7/12 Stefan Behnel : > mark florisson, 12.07.2011 12:11: >> >> On 12 July 2011 11:46, Vitja Makarov wrote: >>> >>> 2011/7/12 Stefan Behnel >>>> >>>> Vitja Makarov, 12.07.2011 09:46: >>>>> >>>>> About cdef classes: probably it's better to >>>>> transform super().method(...) into direct form, e.g. >>>>> BaseClass.method(self, >>>>> ...) >>>> >>>> Except when it doesn't work. ;) >>>> >>>> ? A >>>> ?/ \ >>>> ?B ?C >>>> ?\ / >>>> ? D >>>> >>>> The MRO here is D-B-A-C. If C unconditionally calls A.method(), A's >>>> implementation will be called twice, as it was already called by B. >>>> >>>> http://www.python.org/download/releases/2.2/descrintro/#mro >>> >>> According to your link mro in the example is DBCA ;) > > Totally proves my point. ;) > > >>> Is that a problem for cdef classes? Cdef class have only one base, isn't >>> it? >> >> That is irrelevant, if B calls A's method then C's method will be >> skipped. If it would use super() it would call C's method, and C's >> method would call A's method. > > Exactly. So, calling directly into the base classes method implementation > (i.e. issuing a direct C function call) makes a suitable optimisation for > "final" classes, but not for the general case. > Ohh, I got it. -- vitja. From markflorisson88 at gmail.com Thu Jul 14 13:24:15 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Thu, 14 Jul 2011 13:24:15 +0200 Subject: [Cython] runtests libgomp fail Message-ID: I added a test for nested parallelism with exceptions (with OpenMP nesting explicitly enabled), however if libgomp cannot create more threads it exits the process with exit status 1 and the message "libgomp: Thread creation failed: Resource temporarily unavailable". This then results in a red Hudson and I think it exits the testrunner (are the tests not run as separate processes?). So what should I do? Should I fork manually in my test and watch the exit status? The result is here: https://sage.math.washington.edu:8091/hudson/view/cython-mark/job/cython-mark-tests-py27-c/72/console From vitja.makarov at gmail.com Fri Jul 15 21:32:59 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 15 Jul 2011 23:32:59 +0400 Subject: [Cython] Strange cimport behaviour Message-ID: Hi! I've found strange bug. In my example cimport misses "fcntl.h" include: ((c2f2e12...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ cat ./fff.pyx cimport posix.unistd cimport posix.fcntl print posix.fcntl.O_RDWR ((c2f2e12...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ make fff.so /usr//bin/python ../cython.py --gdb -v fff.pyx -o fff.c Compiling /home/vitja/work/cython-vitek/zzz/fff.pyx gcc -O0 -g3 -fPIC -I/usr//include/python2.6 -g3 -W -c -o fff.o fff.c fff.c: In function 'initfff': fff.c:537: error: 'O_RDWR' undeclared (first use in this function) fff.c:537: error: (Each undeclared identifier is reported only once fff.c:537: error: for each function it appears in.) make: *** [fff.o] ?????? 1 -- vitja. From vitja.makarov at gmail.com Fri Jul 15 21:36:39 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 15 Jul 2011 23:36:39 +0400 Subject: [Cython] Strange cimport behaviour In-Reply-To: References: Message-ID: 15 ???? 2011??. 23:32 ???????????? Vitja Makarov ???????: > Hi! > > I've found strange bug. In my example cimport misses "fcntl.h" include: > > ((c2f2e12...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ cat ./fff.pyx > cimport posix.unistd > cimport posix.fcntl > > print posix.fcntl.O_RDWR > > ((c2f2e12...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ make fff.so > /usr//bin/python ?../cython.py --gdb -v ?fff.pyx -o fff.c > Compiling /home/vitja/work/cython-vitek/zzz/fff.pyx > gcc -O0 -g3 -fPIC -I/usr//include/python2.6 -g3 -W ? ? -c -o fff.o fff.c > fff.c: In function 'initfff': > fff.c:537: error: 'O_RDWR' undeclared (first use in this function) > fff.c:537: error: (Each undeclared identifier is reported only once > fff.c:537: error: for each function it appears in.) > make: *** [fff.o] ?????? 1 > > Btw, from posix cimport unistd, fcntl works as expected -- vitja. From L.J.Buitinck at uva.nl Sat Jul 16 01:30:34 2011 From: L.J.Buitinck at uva.nl (Lars Buitinck) Date: Sat, 16 Jul 2011 01:30:34 +0200 Subject: [Cython] Strange cimport behaviour In-Reply-To: References: Message-ID: 2011/7/15 Vitja Makarov : > I've found strange bug. In my example cimport misses "fcntl.h" include: > > ((c2f2e12...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ cat ./fff.pyx > cimport posix.unistd > cimport posix.fcntl > > print posix.fcntl.O_RDWR Fascinating; I can reproduce the error (on Scientific Linux 5.5), but it goes away when I reverse the order of the includes. -- Lars Buitinck Scientific programmer, ILPS University of Amsterdam From stefan_ml at behnel.de Mon Jul 18 08:40:24 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 18 Jul 2011 08:40:24 +0200 Subject: [Cython] Any news from the IronPython port? Message-ID: <4E23D558.5000104@behnel.de> Hi, subject says it all. The port to IronPython appears to run completely in stealth mode, so I wonder if it has become any usable yet, or if the project was already swept under the carpet and buried for good. Stefan From vitja.makarov at gmail.com Mon Jul 18 11:38:33 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 18 Jul 2011 13:38:33 +0400 Subject: [Cython] How to define C-consts in python module scope Message-ID: cdef enum: EV_READ = 1 EV_WRITE = 2 Is there a way to put this constants into module dict? I want to access this constants from pure python code, I tried this way: globals()['EV_READ'] = EV_READ globals()['EV_WRITE'] = EV_WRITE But I don't like it, is there any other way? -- vitja. From jmccampbell at enthought.com Mon Jul 18 16:45:04 2011 From: jmccampbell at enthought.com (Jason McCampbell) Date: Mon, 18 Jul 2011 09:45:04 -0500 Subject: [Cython] Any news from the IronPython port? In-Reply-To: <4E23D558.5000104@behnel.de> References: <4E23D558.5000104@behnel.de> Message-ID: Hi Stefan, Definitely not buried for good, though we haven't made a lot of changes recently. :) We used it for porting SciPy to .NET and re-wrote a large number of the SciPy C module implementations in Cython. It is generally stable and produces good code within the set of features that were needed (by no means has feature parity with the CPython version). In general, I have been quite happy with the results given that it is possible to generate interfaces for two Python implementations from a single source. Of course, it is not free. One can, in general, not take a NumPy-heavy Cython file and just generate source code for IronPython. Because IronPython and NumPy for .NET do not share any common C APIs we had to wrap some of the APIs and in other cases switch to using Python notation and/or call the new Python-independent NumPy core API (present only in the refactored version). Overall, I think it's a good start and holds some promise for generating re-targetable native wrappings, but there is still plenty of work to do to make it more accessible. Regards, Jason On Mon, Jul 18, 2011 at 1:40 AM, Stefan Behnel wrote: > Hi, > > subject says it all. The port to IronPython appears to run completely in > stealth mode, so I wonder if it has become any usable yet, or if the project > was already swept under the carpet and buried for good. > > Stefan > ______________________________**_________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/**mailman/listinfo/cython-devel > -- *Jason McCampbell* Enthought, Inc. 512.536.1057 (Office) 512.850.6069 (Mobile) jmccampbell at enthought.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From matthew.brett at gmail.com Mon Jul 18 17:25:08 2011 From: matthew.brett at gmail.com (Matthew Brett) Date: Mon, 18 Jul 2011 16:25:08 +0100 Subject: [Cython] Any news from the IronPython port? In-Reply-To: References: <4E23D558.5000104@behnel.de> Message-ID: Hi, On Mon, Jul 18, 2011 at 3:45 PM, Jason McCampbell wrote: > Hi Stefan, > Definitely not buried for good, though we haven't made a lot of changes > recently. :) ?We used it for porting SciPy to .NET and re-wrote a large > number of the SciPy C module implementations in Cython. That sounds very useful. Can these re-writings be proposed as merges into the scipy tree? Did you have time to benchmark the implementations? Thanks a lot, Matthew From dalcinl at gmail.com Mon Jul 18 18:14:11 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Mon, 18 Jul 2011 13:14:11 -0300 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On 18 July 2011 06:38, Vitja Makarov wrote: > cdef enum: > ? ? EV_READ ?= 1 > ? ? EV_WRITE = 2 > > Is there a way to put this constants into module dict? > I want to access this constants from pure python code, I tried this way: > > globals()['EV_READ'] = EV_READ > globals()['EV_WRITE'] = EV_WRITE > > But I don't like it, is there any other way? > cdef public enum: EV_READ = 1 EV_WRITE = 2 However, I do not like it, because I would like to use "public" for other meaning (API generation). Also note that using "public" will trigger the generation of a header file. Perhaps we should support "cpdef enum: ..." -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From sccolbert at gmail.com Mon Jul 18 19:02:40 2011 From: sccolbert at gmail.com (Chris Colbert) Date: Mon, 18 Jul 2011 13:02:40 -0400 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On Mon, Jul 18, 2011 at 12:14 PM, Lisandro Dalcin wrote: > On 18 July 2011 06:38, Vitja Makarov wrote: > > cdef enum: > > EV_READ = 1 > > EV_WRITE = 2 > > > > Is there a way to put this constants into module dict? > > I want to access this constants from pure python code, I tried this way: > > > > globals()['EV_READ'] = EV_READ > > globals()['EV_WRITE'] = EV_WRITE > > > > But I don't like it, is there any other way? > > > > cdef public enum: > EV_READ = 1 > EV_WRITE = 2 > > However, I do not like it, because I would like to use "public" for > other meaning (API generation). Also note that using "public" will > trigger the generation of a header file. > > Perhaps we should support "cpdef enum: ..." > > > cpdef enum would be awesome. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitja.makarov at gmail.com Mon Jul 18 19:07:22 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 18 Jul 2011 21:07:22 +0400 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: 2011/7/18 Lisandro Dalcin : > On 18 July 2011 06:38, Vitja Makarov wrote: >> cdef enum: >> ? ? EV_READ ?= 1 >> ? ? EV_WRITE = 2 >> >> Is there a way to put this constants into module dict? >> I want to access this constants from pure python code, I tried this way: >> >> globals()['EV_READ'] = EV_READ >> globals()['EV_WRITE'] = EV_WRITE >> >> But I don't like it, is there any other way? >> > > cdef public enum: > ? ?EV_READ ?= 1 > ? ?EV_WRITE = 2 > > However, I do not like it, because I would like to use "public" for > other meaning (API generation). Also note that using "public" will > trigger the generation of a header file. > This header breaks my code, I have ev.pyx that includes ev.h, actually I can change that to but I don't like it this way too.. And it actually produces bad code for me: /* "ev.pxd":3 * cimport libev * * cdef api enum: # <<<<<<<<<<<<<< * EV_NONE = libev.EV_NONE * EV_READ = libev.EV_READ */ enum { /* "ev.pxd":6 * EV_NONE = libev.EV_NONE * EV_READ = libev.EV_READ * EV_WRITE = libev.EV_WRITE # <<<<<<<<<<<<<< * * cdef: */ EV_NONE = EV_NONE, EV_READ = EV_READ, EV_WRITE = EV_WRITE }; > Perhaps we should support "cpdef enum: ..." > Yes, that would be nice. -- vitja. From vitja.makarov at gmail.com Mon Jul 18 19:33:54 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 18 Jul 2011 21:33:54 +0400 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: 2011/7/18 Chris Colbert : > > > On Mon, Jul 18, 2011 at 12:14 PM, Lisandro Dalcin wrote: >> >> On 18 July 2011 06:38, Vitja Makarov wrote: >> > cdef enum: >> > ? ? EV_READ ?= 1 >> > ? ? EV_WRITE = 2 >> > >> > Is there a way to put this constants into module dict? >> > I want to access this constants from pure python code, I tried this way: >> > >> > globals()['EV_READ'] = EV_READ >> > globals()['EV_WRITE'] = EV_WRITE >> > >> > But I don't like it, is there any other way? >> > >> >> cdef public enum: >> ? ?EV_READ ?= 1 >> ? ?EV_WRITE = 2 >> >> However, I do not like it, because I would like to use "public" for >> other meaning (API generation). Also note that using "public" will >> trigger the generation of a header file. >> >> Perhaps we should support "cpdef enum: ..." >> >> > > cpdef enum would be awesome. I've created pull request: https://github.com/cython/cython/pull/45 -- vitja. From robertwb at math.washington.edu Mon Jul 18 20:30:59 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 18 Jul 2011 11:30:59 -0700 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: Trevor King and I discussed this quite a while back, but every time I got around to looking at his code (I don't think he ever created a formal pull request) something came up. The idea was that we could support cpdef structs and extern functions as well. On Mon, Jul 18, 2011 at 2:38 AM, Vitja Makarov wrote: > cdef enum: > ? ? EV_READ ?= 1 > ? ? EV_WRITE = 2 > > Is there a way to put this constants into module dict? > I want to access this constants from pure python code, I tried this way: > > globals()['EV_READ'] = EV_READ > globals()['EV_WRITE'] = EV_WRITE > > But I don't like it, is there any other way? > > -- > vitja. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From greg.ewing at canterbury.ac.nz Tue Jul 19 01:34:01 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 19 Jul 2011 11:34:01 +1200 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: <4E24C2E9.9030807@canterbury.ac.nz> Chris Colbert wrote: > cdef public enum: > EV_READ = 1 > EV_WRITE = 2 > > However, I do not like it, because I would like to use "public" for > other meaning (API generation). My suggestion is cdef exposed enum: ... -- Greg From robertwb at math.washington.edu Tue Jul 19 05:52:35 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 18 Jul 2011 20:52:35 -0700 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: <4E24C2E9.9030807@canterbury.ac.nz> References: <4E24C2E9.9030807@canterbury.ac.nz> Message-ID: On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: > Chris Colbert wrote: > >> ? ?cdef public enum: >> ? ? ? EV_READ ?= 1 >> ? ? ? EV_WRITE = 2 >> >> ? ?However, I do not like it, because I would like to use "public" for >> ? ?other meaning (API generation). > > My suggestion is > > ?cdef exposed enum: > ? ?... I agree, public is an overloaded word. This meaning is analogous to its use for cdef class members. Perhaps we should use "api" for api generation, and public used for Python-level access, with cpdef being the preferred form for declaring Python-accessible types. - Robert From robertwb at math.washington.edu Tue Jul 19 05:57:45 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 18 Jul 2011 20:57:45 -0700 Subject: [Cython] Any news from the IronPython port? In-Reply-To: References: <4E23D558.5000104@behnel.de> Message-ID: On Mon, Jul 18, 2011 at 7:45 AM, Jason McCampbell wrote: > Hi Stefan, > Definitely not buried for good, though we haven't made a lot of changes > recently. :) ?We used it for porting SciPy to .NET and re-wrote a large > number of the SciPy C module implementations in Cython. ?It is generally > stable and produces good code within the set of features that were needed > (by no means has feature parity with the CPython version). > In general, I have been quite happy with the results given that it is > possible to generate interfaces for two Python implementations from a single > source. ?Of course, it is not free. ?One can, in general, not take a > NumPy-heavy Cython file and just generate source code for IronPython. > ?Because IronPython and NumPy for .NET do not share any common C APIs we had > to wrap some of the APIs and in other cases switch to using Python notation > and/or call the new Python-independent NumPy core API (present only in the > refactored version). > Overall, I think it's a good start and holds some promise for generating > re-targetable native wrappings, but there is still plenty of work to do to > make it more accessible. > Regards, > Jason Thanks for the status update--is the code available somewhere (e.g. as a forked git repo)? Is it something that would be worth merging, or at this point is it mostly hacked up to just do what you need it to for SciPy? - Robert From vitja.makarov at gmail.com Tue Jul 19 07:24:38 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 19 Jul 2011 09:24:38 +0400 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: 2011/7/18 Robert Bradshaw : > Trevor King and I discussed this quite a while back, but every time I > got around to looking at his code (I don't think he ever created a > formal pull request) something came up. The idea was that we could > support cpdef structs and extern functions as well. > That's interesting, I think I shouldn't hurry with my pull request. 2011/7/19 Robert Bradshaw : > On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >> My suggestion is >> >> ?cdef exposed enum: >> ? ?... > > I agree, public is an overloaded word. This meaning is analogous to > its use for cdef class members. Perhaps we should use "api" for api > generation, and public used for Python-level access, with cpdef being > the preferred form for declaring Python-accessible types. > It seems to me that cpdef is more cythonic but exposed could be used in this case: cdef extern from "ev.h": exposed enum: EV_READ EV_WRITE -- vitja. From robertwb at math.washington.edu Tue Jul 19 07:52:40 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 18 Jul 2011 22:52:40 -0700 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On Mon, Jul 18, 2011 at 10:24 PM, Vitja Makarov wrote: > 2011/7/18 Robert Bradshaw : >> Trevor King and I discussed this quite a while back, but every time I >> got around to looking at his code (I don't think he ever created a >> formal pull request) something came up. The idea was that we could >> support cpdef structs and extern functions as well. >> > > That's interesting, I think I shouldn't hurry with my pull request. > > 2011/7/19 Robert Bradshaw : >> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>> My suggestion is >>> >>> ?cdef exposed enum: >>> ? ?... >> >> I agree, public is an overloaded word. This meaning is analogous to >> its use for cdef class members. Perhaps we should use "api" for api >> generation, and public used for Python-level access, with cpdef being >> the preferred form for declaring Python-accessible types. >> > > It seems to me that cpdef is more cythonic but exposed could be used > in this case: > > cdef extern from "ev.h": > ? ?exposed enum: > ? ? ? ?EV_READ > ? ? ? ?EV_WRITE I'd rather avoid adding another keyword unless it is truly necessary. - Robert From vitja.makarov at gmail.com Tue Jul 19 08:26:25 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 19 Jul 2011 10:26:25 +0400 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: 2011/7/19 Robert Bradshaw : > On Mon, Jul 18, 2011 at 10:24 PM, Vitja Makarov wrote: >> 2011/7/18 Robert Bradshaw : >>> Trevor King and I discussed this quite a while back, but every time I >>> got around to looking at his code (I don't think he ever created a >>> formal pull request) something came up. The idea was that we could >>> support cpdef structs and extern functions as well. >>> >> >> That's interesting, I think I shouldn't hurry with my pull request. >> >> 2011/7/19 Robert Bradshaw : >>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>> My suggestion is >>>> >>>> ?cdef exposed enum: >>>> ? ?... >>> >>> I agree, public is an overloaded word. This meaning is analogous to >>> its use for cdef class members. Perhaps we should use "api" for api >>> generation, and public used for Python-level access, with cpdef being >>> the preferred form for declaring Python-accessible types. >>> >> >> It seems to me that cpdef is more cythonic but exposed could be used >> in this case: >> >> cdef extern from "ev.h": >> ? ?exposed enum: >> ? ? ? ?EV_READ >> ? ? ? ?EV_WRITE > > I'd rather avoid adding another keyword unless it is truly necessary. > I agree. Is Trevor's branch ready for merge? I see last commit was made in march https://github.com/wking/cython/commits/cdef-enums-stucts-and-unions -- vitja. From robertwb at math.washington.edu Tue Jul 19 11:50:05 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 19 Jul 2011 02:50:05 -0700 Subject: [Cython] BUG: Cython's dies with AttributeError In-Reply-To: References: Message-ID: On Thu, Jul 7, 2011 at 2:25 PM, Lars Buitinck wrote: > Dear developers, > > I just got an error message from Cython (current Git). The error can > be reproduced by putting > > cdef foo(): pass > > in a file called foo.pyx, and compiling that. (I know there's an error > in the file as well.) Would you kindly look into this bug? The error > message was: > > > foo.pyx:1:9: Compiler crash in PostParse I'm unable to reproduce this error, perhaps there's something missing in the example here? > ModuleNode.body = StatListNode(foo.pyx:1:5) > StatListNode.stats[0] = CFuncDefNode(foo.pyx:1:5, > ? ?modifiers = [...]/0, > ? ?visibility = u'private') > CFuncDefNode.declarator = CFuncDeclaratorNode(foo.pyx:1:9, > ? ?calling_convention = '') > CFuncDeclaratorNode.base = CNameDeclaratorNode(foo.pyx:1:9, > ? ?calling_convention = u'', > ? ?name = u'foo') > > Compiler crash traceback from this point on: > ?File "Visitor.py", line 173, in > Cython.Compiler.Visitor.TreeVisitor._visitchild > (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:3550) > ?File "Visitor.py", line 282, in > Cython.Compiler.Visitor.CythonTransform.visit_Node > (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:5242) > ?File "Visitor.py", line 234, in > Cython.Compiler.Visitor.VisitorTransform.visitchildren > (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:4395) > ?File "Visitor.py", line 202, in > Cython.Compiler.Visitor.TreeVisitor._visitchildren > (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:4076) > AttributeError: 'CNameDeclaratorNode' object has no attribute 'base' > foo.pyx:1:9: Compiler crash in PostParse > > ModuleNode.body = StatListNode(foo.pyx:1:5) > StatListNode.stats[0] = CFuncDefNode(foo.pyx:1:5, > ? ?modifiers = [...]/0, > ? ?visibility = u'private') > CFuncDefNode.declarator = CFuncDeclaratorNode(foo.pyx:1:9, > ? ?calling_convention = '') > CFuncDeclaratorNode.base = CNameDeclaratorNode(foo.pyx:1:9, > ? ?calling_convention = u'', > ? ?name = u'foo') > > Compiler crash traceback from this point on: > ?File "Visitor.py", line 173, in > Cython.Compiler.Visitor.TreeVisitor._visitchild > (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:3550) > ?File "Visitor.py", line 282, in > Cython.Compiler.Visitor.CythonTransform.visit_Node > (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:5242) > ?File "Visitor.py", line 234, in > Cython.Compiler.Visitor.VisitorTransform.visitchildren > (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:4395) > ?File "Visitor.py", line 202, in > Cython.Compiler.Visitor.TreeVisitor._visitchildren > (/home/s1254871/src/cython/Cython/Compiler/Visitor.c:4076) > AttributeError: 'CNameDeclaratorNode' object has no attribute 'base' > > > -- > Lars Buitinck > Scientific programmer, ILPS > University of Amsterdam > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From L.J.Buitinck at uva.nl Tue Jul 19 13:17:29 2011 From: L.J.Buitinck at uva.nl (Lars Buitinck) Date: Tue, 19 Jul 2011 13:17:29 +0200 Subject: [Cython] BUG: Cython's dies with AttributeError In-Reply-To: References: Message-ID: 2011/7/19 Robert Bradshaw : > On Thu, Jul 7, 2011 at 2:25 PM, Lars Buitinck wrote: >> foo.pyx:1:9: Compiler crash in PostParse > > I'm unable to reproduce this error, perhaps there's something missing > in the example here? I just pulled from your repo and I'm not getting the error message anymore. Thanks anyway! -- Lars Buitinck Scientific programmer, ILPS University of Amsterdam From dalcinl at gmail.com Wed Jul 20 00:02:48 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 19 Jul 2011 19:02:48 -0300 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On 19 July 2011 02:24, Vitja Makarov wrote: > 2011/7/18 Robert Bradshaw : >> Trevor King and I discussed this quite a while back, but every time I >> got around to looking at his code (I don't think he ever created a >> formal pull request) something came up. The idea was that we could >> support cpdef structs and extern functions as well. >> > > That's interesting, I think I shouldn't hurry with my pull request. > > 2011/7/19 Robert Bradshaw : >> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>> My suggestion is >>> >>> ?cdef exposed enum: >>> ? ?... >> >> I agree, public is an overloaded word. This meaning is analogous to >> its use for cdef class members. Perhaps we should use "api" for api >> generation, and public used for Python-level access, with cpdef being >> the preferred form for declaring Python-accessible types. >> > > It seems to me that cpdef is more cythonic but exposed could be used > in this case: > > cdef extern from "ev.h": > ? ?exposed enum: > ? ? ? ?EV_READ > ? ? ? ?EV_WRITE > And what about this? cdef extern from "ev.h": cpdef enum: EV_READ EV_WRITE BTW, how is this supposed to work with *.pxd files? I think the values will be exposed just in the matching implementation .pyx file, and not with cimport, right? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From vitja.makarov at gmail.com Wed Jul 20 00:46:37 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 20 Jul 2011 02:46:37 +0400 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: 2011/7/20 Lisandro Dalcin : > On 19 July 2011 02:24, Vitja Makarov wrote: >> 2011/7/18 Robert Bradshaw : >>> Trevor King and I discussed this quite a while back, but every time I >>> got around to looking at his code (I don't think he ever created a >>> formal pull request) something came up. The idea was that we could >>> support cpdef structs and extern functions as well. >>> >> >> That's interesting, I think I shouldn't hurry with my pull request. >> >> 2011/7/19 Robert Bradshaw : >>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>> My suggestion is >>>> >>>> ?cdef exposed enum: >>>> ? ?... >>> >>> I agree, public is an overloaded word. This meaning is analogous to >>> its use for cdef class members. Perhaps we should use "api" for api >>> generation, and public used for Python-level access, with cpdef being >>> the preferred form for declaring Python-accessible types. >>> >> >> It seems to me that cpdef is more cythonic but exposed could be used >> in this case: >> >> cdef extern from "ev.h": >> ? ?exposed enum: >> ? ? ? ?EV_READ >> ? ? ? ?EV_WRITE >> > > And what about this? > > cdef extern from "ev.h": > ? cpdef enum: > ? ? ? EV_READ > ? ? ? EV_WRITE > I'm fine with that, but cpdef seems to be a little bit confusing here. On the other hand "cdef enum" is already supported inside "cdef extern" block. I have implemented "cpdef enum" inside "cdef extern" block here: https://github.com/vitek/cython/commit/471c41a7b37728b1b8844ea13d64b892530f403d I'm afraid that this change could introduce some side effects, however it still passes all the tests > > BTW, how is this supposed to work with *.pxd files? I think the values > will be exposed just in the matching implementation .pyx file, and not > with cimport, right? > I thought about that too and I think it's correct behaviour. By the way it's already working as expected. -- vitja. From robertwb at math.washington.edu Wed Jul 20 01:48:05 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 19 Jul 2011 16:48:05 -0700 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin wrote: > On 19 July 2011 02:24, Vitja Makarov wrote: >> 2011/7/18 Robert Bradshaw : >>> Trevor King and I discussed this quite a while back, but every time I >>> got around to looking at his code (I don't think he ever created a >>> formal pull request) something came up. The idea was that we could >>> support cpdef structs and extern functions as well. >>> >> >> That's interesting, I think I shouldn't hurry with my pull request. >> >> 2011/7/19 Robert Bradshaw : >>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>> My suggestion is >>>> >>>> ?cdef exposed enum: >>>> ? ?... >>> >>> I agree, public is an overloaded word. This meaning is analogous to >>> its use for cdef class members. Perhaps we should use "api" for api >>> generation, and public used for Python-level access, with cpdef being >>> the preferred form for declaring Python-accessible types. >>> >> >> It seems to me that cpdef is more cythonic but exposed could be used >> in this case: >> >> cdef extern from "ev.h": >> ? ?exposed enum: >> ? ? ? ?EV_READ >> ? ? ? ?EV_WRITE >> > > And what about this? > > cdef extern from "ev.h": > ? cpdef enum: > ? ? ? EV_READ > ? ? ? EV_WRITE Yep, exactly. > BTW, how is this supposed to work with *.pxd files? I think the values > will be exposed just in the matching implementation .pyx file, and not > with cimport, right? It would be an error unless there's an corresponding .pyx file (which could be empty). The idea is that one could also define extern cpdef functions and structs and wrappers would be provided. - Robert From robertwb at math.washington.edu Wed Jul 20 02:32:51 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Tue, 19 Jul 2011 17:32:51 -0700 Subject: [Cython] Cython 0.15 release Message-ID: We're long overdue for a release, and this week would be a good one for me to push one out. Hudson https://sage.math.washington.edu:8091/hudson is looking in pretty good shape, and though I know we've got a big pile of stuff currently in progress, we've also got a big backlog of stuff to get out. I'd like to finish looking at https://github.com/cython/cython/pull/38 , are there any other changes that people want to urgently get in? Also, I've started http://wiki.cython.org/ReleaseNotes-0.15 , feel free to edit if you think anything should be highlighted. - Robert From drsalists at gmail.com Wed Jul 20 03:49:34 2011 From: drsalists at gmail.com (Dan Stromberg) Date: Tue, 19 Jul 2011 18:49:34 -0700 Subject: [Cython] Cython 0.15 release In-Reply-To: References: Message-ID: Isn't this the first release that supports yield? That's a rather big deal to me. On Tue, Jul 19, 2011 at 5:32 PM, Robert Bradshaw < robertwb at math.washington.edu> wrote: > We're long overdue for a release, and this week would be a good one > for me to push one out. Hudson > https://sage.math.washington.edu:8091/hudson is looking in pretty good > shape, and though I know we've got a big pile of stuff currently in > progress, we've also got a big backlog of stuff to get out. I'd like > to finish looking at https://github.com/cython/cython/pull/38 , are > there any other changes that people want to urgently get in? Also, > I've started http://wiki.cython.org/ReleaseNotes-0.15 , feel free to > edit if you think anything should be highlighted. > > - Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From markflorisson88 at gmail.com Wed Jul 20 10:51:16 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 20 Jul 2011 10:51:16 +0200 Subject: [Cython] Cython 0.15 release In-Reply-To: References: Message-ID: On 20 July 2011 02:32, Robert Bradshaw wrote: > We're long overdue for a release, and this week would be a good one > for me to push one out. Hudson > https://sage.math.washington.edu:8091/hudson is looking in pretty good > shape, and though I know we've got a big pile of stuff currently in > progress, we've also got a big backlog of stuff to get out. I'd like > to finish looking at https://github.com/cython/cython/pull/38 , are > there any other changes that people want to urgently get in? Also, > I've started http://wiki.cython.org/ReleaseNotes-0.15 , feel free to > edit if you think anything should be highlighted. I think cpdef enum is a rather small but still very useful change for many people. > > - Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Wed Jul 20 11:26:54 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 20 Jul 2011 11:26:54 +0200 Subject: [Cython] Cython 0.15 release In-Reply-To: References: Message-ID: <4E269F5E.7090907@behnel.de> mark florisson, 20.07.2011 10:51: > On 20 July 2011 02:32, Robert Bradshaw wrote: >> We're long overdue for a release, and this week would be a good one >> for me to push one out. Hudson >> https://sage.math.washington.edu:8091/hudson is looking in pretty good >> shape, and though I know we've got a big pile of stuff currently in >> progress, we've also got a big backlog of stuff to get out. I'd like >> to finish looking at https://github.com/cython/cython/pull/38 , are >> there any other changes that people want to urgently get in? Also, >> I've started http://wiki.cython.org/ReleaseNotes-0.15 , feel free to >> edit if you think anything should be highlighted. > > I think cpdef enum is a rather small but still very useful change for > many people. Did we agree on the right syntax yet? We should be sure about that before adding a new syntax feature to the language that we won't be able to change later on. Stefan From markflorisson88 at gmail.com Wed Jul 20 11:40:18 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 20 Jul 2011 11:40:18 +0200 Subject: [Cython] Cython 0.15 release In-Reply-To: <4E269F5E.7090907@behnel.de> References: <4E269F5E.7090907@behnel.de> Message-ID: On 20 July 2011 11:26, Stefan Behnel wrote: > mark florisson, 20.07.2011 10:51: >> >> On 20 July 2011 02:32, Robert Bradshaw wrote: >>> >>> We're long overdue for a release, and this week would be a good one >>> for me to push one out. Hudson >>> https://sage.math.washington.edu:8091/hudson is looking in pretty good >>> shape, and though I know we've got a big pile of stuff currently in >>> progress, we've also got a big backlog of stuff to get out. I'd like >>> to finish looking at https://github.com/cython/cython/pull/38 , are >>> there any other changes that people want to urgently get in? Also, >>> I've started http://wiki.cython.org/ReleaseNotes-0.15 , feel free to >>> edit if you think anything should be highlighted. >> >> I think cpdef enum is a rather small but still very useful change for >> many people. > > Did we agree on the right syntax yet? We should be sure about that before > adding a new syntax feature to the language that we won't be able to change > later on. I think we settled on the syntax in the thread called Cython .pxd introspection: listing defined constants, when Trevor was working on it: http://mail.python.org/pipermail/cython-devel/2011-February/000035.html It does look like the most sensible syntax to me, as cpdef is about exposing C/Cython-only stuff to Python. > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Wed Jul 20 11:47:25 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 20 Jul 2011 11:47:25 +0200 Subject: [Cython] Cython 0.15 release In-Reply-To: References: <4E269F5E.7090907@behnel.de> Message-ID: <4E26A42D.40301@behnel.de> mark florisson, 20.07.2011 11:40: > On 20 July 2011 11:26, Stefan Behnel wrote: >> mark florisson, 20.07.2011 10:51: >>> >>> On 20 July 2011 02:32, Robert Bradshaw wrote: >>>> >>>> We're long overdue for a release, and this week would be a good one >>>> for me to push one out. Hudson >>>> https://sage.math.washington.edu:8091/hudson is looking in pretty good >>>> shape, and though I know we've got a big pile of stuff currently in >>>> progress, we've also got a big backlog of stuff to get out. I'd like >>>> to finish looking at https://github.com/cython/cython/pull/38 , are >>>> there any other changes that people want to urgently get in? Also, >>>> I've started http://wiki.cython.org/ReleaseNotes-0.15 , feel free to >>>> edit if you think anything should be highlighted. >>> >>> I think cpdef enum is a rather small but still very useful change for >>> many people. >> >> Did we agree on the right syntax yet? We should be sure about that before >> adding a new syntax feature to the language that we won't be able to change >> later on. > > I think we settled on the syntax in the thread called Cython .pxd > introspection: listing defined constants, when Trevor was working on > it: http://mail.python.org/pipermail/cython-devel/2011-February/000035.html > > It does look like the most sensible syntax to me, as cpdef is about > exposing C/Cython-only stuff to Python. Ok. Does the current implementation raise syntax errors where appropriate, in order to prevent the syntax from allowing to be applied in unwanted places? What about the "cdef extern: ... cpdef enum" case? Is that to be allowed? I know I'm being pedantic here, but these things matter. Stefan From markflorisson88 at gmail.com Wed Jul 20 11:51:24 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 20 Jul 2011 11:51:24 +0200 Subject: [Cython] Cython 0.15 release In-Reply-To: <4E26A42D.40301@behnel.de> References: <4E269F5E.7090907@behnel.de> <4E26A42D.40301@behnel.de> Message-ID: On 20 July 2011 11:47, Stefan Behnel wrote: > mark florisson, 20.07.2011 11:40: >> >> On 20 July 2011 11:26, Stefan Behnel wrote: >>> >>> mark florisson, 20.07.2011 10:51: >>>> >>>> On 20 July 2011 02:32, Robert Bradshaw wrote: >>>>> >>>>> We're long overdue for a release, and this week would be a good one >>>>> for me to push one out. Hudson >>>>> https://sage.math.washington.edu:8091/hudson is looking in pretty good >>>>> shape, and though I know we've got a big pile of stuff currently in >>>>> progress, we've also got a big backlog of stuff to get out. I'd like >>>>> to finish looking at https://github.com/cython/cython/pull/38 , are >>>>> there any other changes that people want to urgently get in? Also, >>>>> I've started http://wiki.cython.org/ReleaseNotes-0.15 , feel free to >>>>> edit if you think anything should be highlighted. >>>> >>>> I think cpdef enum is a rather small but still very useful change for >>>> many people. >>> >>> Did we agree on the right syntax yet? We should be sure about that before >>> adding a new syntax feature to the language that we won't be able to >>> change >>> later on. >> >> I think we settled on the syntax in the thread called Cython .pxd >> introspection: listing defined constants, when Trevor was working on >> it: >> http://mail.python.org/pipermail/cython-devel/2011-February/000035.html >> >> It does look like the most sensible syntax to me, as cpdef is about >> exposing C/Cython-only stuff to Python. > > Ok. Does the current implementation raise syntax errors where appropriate, > in order to prevent the syntax from allowing to be applied in unwanted > places? What about the "cdef extern: ... cpdef enum" case? Is that to be > allowed? I don't know about the former, but I believe it will raise syntax errors whenever cdef enum would be invalid. I don't know if it checks whether there is a corresponding .pyx when it is cimported, perhaps Vitja can comment on that. Yes, especially cdef extern: cpdef enum is very useful (and implemented), when you want to expose C constants to Python. > I know I'm being pedantic here, but these things matter. > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From vitja.makarov at gmail.com Wed Jul 20 11:58:31 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 20 Jul 2011 13:58:31 +0400 Subject: [Cython] Cython 0.15 release In-Reply-To: References: <4E269F5E.7090907@behnel.de> <4E26A42D.40301@behnel.de> Message-ID: 2011/7/20 mark florisson : > On 20 July 2011 11:47, Stefan Behnel wrote: >> mark florisson, 20.07.2011 11:40: >>> >>> On 20 July 2011 11:26, Stefan Behnel wrote: >>>> >>>> mark florisson, 20.07.2011 10:51: >>>>> >>>>> On 20 July 2011 02:32, Robert Bradshaw wrote: >>>>>> >>>>>> We're long overdue for a release, and this week would be a good one >>>>>> for me to push one out. Hudson >>>>>> https://sage.math.washington.edu:8091/hudson is looking in pretty good >>>>>> shape, and though I know we've got a big pile of stuff currently in >>>>>> progress, we've also got a big backlog of stuff to get out. I'd like >>>>>> to finish looking at https://github.com/cython/cython/pull/38 , are >>>>>> there any other changes that people want to urgently get in? Also, >>>>>> I've started http://wiki.cython.org/ReleaseNotes-0.15 , feel free to >>>>>> edit if you think anything should be highlighted. >>>>> >>>>> I think cpdef enum is a rather small but still very useful change for >>>>> many people. >>>> >>>> Did we agree on the right syntax yet? We should be sure about that before >>>> adding a new syntax feature to the language that we won't be able to >>>> change >>>> later on. >>> >>> I think we settled on the syntax in the thread called Cython .pxd >>> introspection: listing defined constants, when Trevor was working on >>> it: >>> http://mail.python.org/pipermail/cython-devel/2011-February/000035.html >>> >>> It does look like the most sensible syntax to me, as cpdef is about >>> exposing C/Cython-only stuff to Python. >> >> Ok. Does the current implementation raise syntax errors where appropriate, >> in order to prevent the syntax from allowing to be applied in unwanted >> places? What about the "cdef extern: ... cpdef enum" case? Is that to be >> allowed? > > I don't know about the former, but I believe it will raise syntax > errors whenever cdef enum would be invalid. I don't know if it checks > whether there is a corresponding .pyx when it is cimported, perhaps > Vitja can comment on that. > Actually I don't. My change simply allows cpdef enum and doesn't do context checking. So it's better to write some tests for this. > Yes, especially cdef extern: cpdef enum is very useful (and > implemented), when you want to expose C constants to Python. > Yes, this works, but I do worry about it https://github.com/vitek/cython/commit/471c41a7b37728b1b8844ea13d64b892530f403d I've changed CDefExternNode.generate_execution_code() to call body.generate_execution_code() And I'm not sure if it generates unwanted code. -- vitja. From dalcinl at gmail.com Wed Jul 20 18:06:00 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 20 Jul 2011 13:06:00 -0300 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On 19 July 2011 20:48, Robert Bradshaw wrote: > On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin wrote: >> On 19 July 2011 02:24, Vitja Makarov wrote: >>> 2011/7/18 Robert Bradshaw : >>>> Trevor King and I discussed this quite a while back, but every time I >>>> got around to looking at his code (I don't think he ever created a >>>> formal pull request) something came up. The idea was that we could >>>> support cpdef structs and extern functions as well. >>>> >>> >>> That's interesting, I think I shouldn't hurry with my pull request. >>> >>> 2011/7/19 Robert Bradshaw : >>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>>> My suggestion is >>>>> >>>>> ?cdef exposed enum: >>>>> ? ?... >>>> >>>> I agree, public is an overloaded word. This meaning is analogous to >>>> its use for cdef class members. Perhaps we should use "api" for api >>>> generation, and public used for Python-level access, with cpdef being >>>> the preferred form for declaring Python-accessible types. >>>> >>> >>> It seems to me that cpdef is more cythonic but exposed could be used >>> in this case: >>> >>> cdef extern from "ev.h": >>> ? ?exposed enum: >>> ? ? ? ?EV_READ >>> ? ? ? ?EV_WRITE >>> >> >> And what about this? >> >> cdef extern from "ev.h": >> ? cpdef enum: >> ? ? ? EV_READ >> ? ? ? EV_WRITE > > Yep, exactly. > >> BTW, how is this supposed to work with *.pxd files? I think the values >> will be exposed just in the matching implementation .pyx file, and not >> with cimport, right? > > It would be an error unless there's an corresponding .pyx file (which > could be empty). The idea is that one could also define extern cpdef > functions and structs and wrappers would be provided. > It would be an error? What do you mean? if you cpdef enum in foo.pxd, and have foo.pyx, then the enumerations should be exposed in the 'foo' module. But then if you "cimport foo" in bar.pyx, that should succeed but the enumerations should not be exposed in the "bar" module... Am I missing something? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From markflorisson88 at gmail.com Wed Jul 20 18:51:24 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 20 Jul 2011 18:51:24 +0200 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On 20 July 2011 18:06, Lisandro Dalcin wrote: > On 19 July 2011 20:48, Robert Bradshaw wrote: >> On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin wrote: >>> On 19 July 2011 02:24, Vitja Makarov wrote: >>>> 2011/7/18 Robert Bradshaw : >>>>> Trevor King and I discussed this quite a while back, but every time I >>>>> got around to looking at his code (I don't think he ever created a >>>>> formal pull request) something came up. The idea was that we could >>>>> support cpdef structs and extern functions as well. >>>>> >>>> >>>> That's interesting, I think I shouldn't hurry with my pull request. >>>> >>>> 2011/7/19 Robert Bradshaw : >>>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>>>> My suggestion is >>>>>> >>>>>> ?cdef exposed enum: >>>>>> ? ?... >>>>> >>>>> I agree, public is an overloaded word. This meaning is analogous to >>>>> its use for cdef class members. Perhaps we should use "api" for api >>>>> generation, and public used for Python-level access, with cpdef being >>>>> the preferred form for declaring Python-accessible types. >>>>> >>>> >>>> It seems to me that cpdef is more cythonic but exposed could be used >>>> in this case: >>>> >>>> cdef extern from "ev.h": >>>> ? ?exposed enum: >>>> ? ? ? ?EV_READ >>>> ? ? ? ?EV_WRITE >>>> >>> >>> And what about this? >>> >>> cdef extern from "ev.h": >>> ? cpdef enum: >>> ? ? ? EV_READ >>> ? ? ? EV_WRITE >> >> Yep, exactly. >> >>> BTW, how is this supposed to work with *.pxd files? I think the values >>> will be exposed just in the matching implementation .pyx file, and not >>> with cimport, right? >> >> It would be an error unless there's an corresponding .pyx file (which >> could be empty). The idea is that one could also define extern cpdef >> functions and structs and wrappers would be provided. >> > > It would be an error? What do you mean? if you cpdef enum in foo.pxd, > and have foo.pyx, then the enumerations should be exposed in the 'foo' > module. But then if you "cimport foo" in bar.pyx, that should succeed > but the enumerations should not be exposed in the "bar" module... Am I > missing something? > I believe Robert confirmed what you said and added to that that it would be an error to have a cpdef extern enum in a .pxd without a corresponding .pyx file. > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From dalcinl at gmail.com Wed Jul 20 20:04:38 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 20 Jul 2011 15:04:38 -0300 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On 20 July 2011 13:51, mark florisson wrote: > On 20 July 2011 18:06, Lisandro Dalcin wrote: >> On 19 July 2011 20:48, Robert Bradshaw wrote: >>> On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin wrote: >>>> On 19 July 2011 02:24, Vitja Makarov wrote: >>>>> 2011/7/18 Robert Bradshaw : >>>>>> Trevor King and I discussed this quite a while back, but every time I >>>>>> got around to looking at his code (I don't think he ever created a >>>>>> formal pull request) something came up. The idea was that we could >>>>>> support cpdef structs and extern functions as well. >>>>>> >>>>> >>>>> That's interesting, I think I shouldn't hurry with my pull request. >>>>> >>>>> 2011/7/19 Robert Bradshaw : >>>>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>>>>> My suggestion is >>>>>>> >>>>>>> ?cdef exposed enum: >>>>>>> ? ?... >>>>>> >>>>>> I agree, public is an overloaded word. This meaning is analogous to >>>>>> its use for cdef class members. Perhaps we should use "api" for api >>>>>> generation, and public used for Python-level access, with cpdef being >>>>>> the preferred form for declaring Python-accessible types. >>>>>> >>>>> >>>>> It seems to me that cpdef is more cythonic but exposed could be used >>>>> in this case: >>>>> >>>>> cdef extern from "ev.h": >>>>> ? ?exposed enum: >>>>> ? ? ? ?EV_READ >>>>> ? ? ? ?EV_WRITE >>>>> >>>> >>>> And what about this? >>>> >>>> cdef extern from "ev.h": >>>> ? cpdef enum: >>>> ? ? ? EV_READ >>>> ? ? ? EV_WRITE >>> >>> Yep, exactly. >>> >>>> BTW, how is this supposed to work with *.pxd files? I think the values >>>> will be exposed just in the matching implementation .pyx file, and not >>>> with cimport, right? >>> >>> It would be an error unless there's an corresponding .pyx file (which >>> could be empty). The idea is that one could also define extern cpdef >>> functions and structs and wrappers would be provided. >>> >> >> It would be an error? What do you mean? if you cpdef enum in foo.pxd, >> and have foo.pyx, then the enumerations should be exposed in the 'foo' >> module. But then if you "cimport foo" in bar.pyx, that should succeed >> but the enumerations should not be exposed in the "bar" module... Am I >> missing something? >> > > I believe Robert confirmed what you said and added to that that it > would be an error to have a cpdef extern enum in a .pxd without a > corresponding .pyx file. > But what an "error" means? Cython is going to fail compilation? Suppose I develop a package, and at setup.py install time my package installs somewhere a .pxd full of definitions in such a way that third-party code can cimport it. Then, while writing the third party code, you cimport the my package .pxd, and obviously there is no .pyx for my package, as you are writing YOUR code. What's going to happen in this case? -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From markflorisson88 at gmail.com Wed Jul 20 20:32:08 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 20 Jul 2011 20:32:08 +0200 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On 20 July 2011 20:04, Lisandro Dalcin wrote: > On 20 July 2011 13:51, mark florisson wrote: >> On 20 July 2011 18:06, Lisandro Dalcin wrote: >>> On 19 July 2011 20:48, Robert Bradshaw wrote: >>>> On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin wrote: >>>>> On 19 July 2011 02:24, Vitja Makarov wrote: >>>>>> 2011/7/18 Robert Bradshaw : >>>>>>> Trevor King and I discussed this quite a while back, but every time I >>>>>>> got around to looking at his code (I don't think he ever created a >>>>>>> formal pull request) something came up. The idea was that we could >>>>>>> support cpdef structs and extern functions as well. >>>>>>> >>>>>> >>>>>> That's interesting, I think I shouldn't hurry with my pull request. >>>>>> >>>>>> 2011/7/19 Robert Bradshaw : >>>>>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>>>>>> My suggestion is >>>>>>>> >>>>>>>> ?cdef exposed enum: >>>>>>>> ? ?... >>>>>>> >>>>>>> I agree, public is an overloaded word. This meaning is analogous to >>>>>>> its use for cdef class members. Perhaps we should use "api" for api >>>>>>> generation, and public used for Python-level access, with cpdef being >>>>>>> the preferred form for declaring Python-accessible types. >>>>>>> >>>>>> >>>>>> It seems to me that cpdef is more cythonic but exposed could be used >>>>>> in this case: >>>>>> >>>>>> cdef extern from "ev.h": >>>>>> ? ?exposed enum: >>>>>> ? ? ? ?EV_READ >>>>>> ? ? ? ?EV_WRITE >>>>>> >>>>> >>>>> And what about this? >>>>> >>>>> cdef extern from "ev.h": >>>>> ? cpdef enum: >>>>> ? ? ? EV_READ >>>>> ? ? ? EV_WRITE >>>> >>>> Yep, exactly. >>>> >>>>> BTW, how is this supposed to work with *.pxd files? I think the values >>>>> will be exposed just in the matching implementation .pyx file, and not >>>>> with cimport, right? >>>> >>>> It would be an error unless there's an corresponding .pyx file (which >>>> could be empty). The idea is that one could also define extern cpdef >>>> functions and structs and wrappers would be provided. >>>> >>> >>> It would be an error? What do you mean? if you cpdef enum in foo.pxd, >>> and have foo.pyx, then the enumerations should be exposed in the 'foo' >>> module. But then if you "cimport foo" in bar.pyx, that should succeed >>> but the enumerations should not be exposed in the "bar" module... Am I >>> missing something? >>> >> >> I believe Robert confirmed what you said and added to that that it >> would be an error to have a cpdef extern enum in a .pxd without a >> corresponding .pyx file. >> > > But what an "error" means? Cython is going to fail compilation? > Suppose I develop a package, and at setup.py install time my package > installs somewhere a .pxd full of definitions in such a way that > third-party code can cimport it. Then, while writing the third party > code, you cimport the my package .pxd, and obviously there is no .pyx > for my package, as you are writing YOUR code. What's going to happen > in this case? If you want to use cpdef in your .pxd, it means you want to expose those to Python. This means they must be importable from Python, which means you need an extension module that exposes the constants, which means you need a .pyx. So you ship a (possibly empty) .pyx file along with your .pxd. If you don't want to expose them to Python but only to Cython, don't use cpdef. This compile time error would be issued whenever a user does a cimport of a .pxd file that has a cpdef enum with no corresponding .pyx file, i.e. the pxd is unusable, you wouldn't ship it in the first place. > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From dalcinl at gmail.com Wed Jul 20 21:13:21 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 20 Jul 2011 16:13:21 -0300 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On 20 July 2011 15:32, mark florisson wrote: > On 20 July 2011 20:04, Lisandro Dalcin wrote: >> On 20 July 2011 13:51, mark florisson wrote: >>> On 20 July 2011 18:06, Lisandro Dalcin wrote: >>>> On 19 July 2011 20:48, Robert Bradshaw wrote: >>>>> On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin wrote: >>>>>> On 19 July 2011 02:24, Vitja Makarov wrote: >>>>>>> 2011/7/18 Robert Bradshaw : >>>>>>>> Trevor King and I discussed this quite a while back, but every time I >>>>>>>> got around to looking at his code (I don't think he ever created a >>>>>>>> formal pull request) something came up. The idea was that we could >>>>>>>> support cpdef structs and extern functions as well. >>>>>>>> >>>>>>> >>>>>>> That's interesting, I think I shouldn't hurry with my pull request. >>>>>>> >>>>>>> 2011/7/19 Robert Bradshaw : >>>>>>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>>>>>>> My suggestion is >>>>>>>>> >>>>>>>>> ?cdef exposed enum: >>>>>>>>> ? ?... >>>>>>>> >>>>>>>> I agree, public is an overloaded word. This meaning is analogous to >>>>>>>> its use for cdef class members. Perhaps we should use "api" for api >>>>>>>> generation, and public used for Python-level access, with cpdef being >>>>>>>> the preferred form for declaring Python-accessible types. >>>>>>>> >>>>>>> >>>>>>> It seems to me that cpdef is more cythonic but exposed could be used >>>>>>> in this case: >>>>>>> >>>>>>> cdef extern from "ev.h": >>>>>>> ? ?exposed enum: >>>>>>> ? ? ? ?EV_READ >>>>>>> ? ? ? ?EV_WRITE >>>>>>> >>>>>> >>>>>> And what about this? >>>>>> >>>>>> cdef extern from "ev.h": >>>>>> ? cpdef enum: >>>>>> ? ? ? EV_READ >>>>>> ? ? ? EV_WRITE >>>>> >>>>> Yep, exactly. >>>>> >>>>>> BTW, how is this supposed to work with *.pxd files? I think the values >>>>>> will be exposed just in the matching implementation .pyx file, and not >>>>>> with cimport, right? >>>>> >>>>> It would be an error unless there's an corresponding .pyx file (which >>>>> could be empty). The idea is that one could also define extern cpdef >>>>> functions and structs and wrappers would be provided. >>>>> >>>> >>>> It would be an error? What do you mean? if you cpdef enum in foo.pxd, >>>> and have foo.pyx, then the enumerations should be exposed in the 'foo' >>>> module. But then if you "cimport foo" in bar.pyx, that should succeed >>>> but the enumerations should not be exposed in the "bar" module... Am I >>>> missing something? >>>> >>> >>> I believe Robert confirmed what you said and added to that that it >>> would be an error to have a cpdef extern enum in a .pxd without a >>> corresponding .pyx file. >>> >> >> But what an "error" means? Cython is going to fail compilation? >> Suppose I develop a package, and at setup.py install time my package >> installs somewhere a .pxd full of definitions in such a way that >> third-party code can cimport it. Then, while writing the third party >> code, you cimport the my package .pxd, and obviously there is no .pyx >> for my package, as you are writing YOUR code. What's going to happen >> in this case? > > If you want to use cpdef in your .pxd, it means you want to expose > those to Python. This means they must be importable from Python, which > means you need an extension module that exposes the constants, which > means you need a .pyx. So you ship a (possibly empty) .pyx file along > with your .pxd. If you don't want to expose them to Python but only to > Cython, don't use cpdef. This compile time error would be issued > whenever a user does a cimport of a .pxd file that has a cpdef enum > with no corresponding .pyx file, i.e. the pxd is unusable, you > wouldn't ship it in the first place. > So supose I want to write a small wrapper for dlopen(), then I do: # dl.pxd cdef from extern from "dlfcn.h": cpdef enum: RTLD_GLOBAL void * dlopen()(const char *filename, int flag); # dl.pyx def open(filename, mode): cdef void *handle = c_dlopen(fiename,handle) return handle Then the "dl" module namespace contains "dlopen" and "RTLD_GLOBAL" Now, suppose I code my setup.py to build the ext module and install dl.pxd as package data. Now a user runs "pip install dl" and installs my package. She wants to reuse my .pxd (installed somewhere) for calling dlopen() in its own Cython code: # user.pyx from dl cimport RTLD_GLOBAL, c_dlopen dlopen("somelibrary.so", RTLD_GLOBAL) So the user code is going to fail? Then I cannot take advantage of cpdef enum for developing my "dl" module, I have to go back to manually exposing the enumerations -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From markflorisson88 at gmail.com Wed Jul 20 21:27:17 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 20 Jul 2011 21:27:17 +0200 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On 20 July 2011 21:13, Lisandro Dalcin wrote: > On 20 July 2011 15:32, mark florisson wrote: >> On 20 July 2011 20:04, Lisandro Dalcin wrote: >>> On 20 July 2011 13:51, mark florisson wrote: >>>> On 20 July 2011 18:06, Lisandro Dalcin wrote: >>>>> On 19 July 2011 20:48, Robert Bradshaw wrote: >>>>>> On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin wrote: >>>>>>> On 19 July 2011 02:24, Vitja Makarov wrote: >>>>>>>> 2011/7/18 Robert Bradshaw : >>>>>>>>> Trevor King and I discussed this quite a while back, but every time I >>>>>>>>> got around to looking at his code (I don't think he ever created a >>>>>>>>> formal pull request) something came up. The idea was that we could >>>>>>>>> support cpdef structs and extern functions as well. >>>>>>>>> >>>>>>>> >>>>>>>> That's interesting, I think I shouldn't hurry with my pull request. >>>>>>>> >>>>>>>> 2011/7/19 Robert Bradshaw : >>>>>>>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>>>>>>>> My suggestion is >>>>>>>>>> >>>>>>>>>> ?cdef exposed enum: >>>>>>>>>> ? ?... >>>>>>>>> >>>>>>>>> I agree, public is an overloaded word. This meaning is analogous to >>>>>>>>> its use for cdef class members. Perhaps we should use "api" for api >>>>>>>>> generation, and public used for Python-level access, with cpdef being >>>>>>>>> the preferred form for declaring Python-accessible types. >>>>>>>>> >>>>>>>> >>>>>>>> It seems to me that cpdef is more cythonic but exposed could be used >>>>>>>> in this case: >>>>>>>> >>>>>>>> cdef extern from "ev.h": >>>>>>>> ? ?exposed enum: >>>>>>>> ? ? ? ?EV_READ >>>>>>>> ? ? ? ?EV_WRITE >>>>>>>> >>>>>>> >>>>>>> And what about this? >>>>>>> >>>>>>> cdef extern from "ev.h": >>>>>>> ? cpdef enum: >>>>>>> ? ? ? EV_READ >>>>>>> ? ? ? EV_WRITE >>>>>> >>>>>> Yep, exactly. >>>>>> >>>>>>> BTW, how is this supposed to work with *.pxd files? I think the values >>>>>>> will be exposed just in the matching implementation .pyx file, and not >>>>>>> with cimport, right? >>>>>> >>>>>> It would be an error unless there's an corresponding .pyx file (which >>>>>> could be empty). The idea is that one could also define extern cpdef >>>>>> functions and structs and wrappers would be provided. >>>>>> >>>>> >>>>> It would be an error? What do you mean? if you cpdef enum in foo.pxd, >>>>> and have foo.pyx, then the enumerations should be exposed in the 'foo' >>>>> module. But then if you "cimport foo" in bar.pyx, that should succeed >>>>> but the enumerations should not be exposed in the "bar" module... Am I >>>>> missing something? >>>>> >>>> >>>> I believe Robert confirmed what you said and added to that that it >>>> would be an error to have a cpdef extern enum in a .pxd without a >>>> corresponding .pyx file. >>>> >>> >>> But what an "error" means? Cython is going to fail compilation? >>> Suppose I develop a package, and at setup.py install time my package >>> installs somewhere a .pxd full of definitions in such a way that >>> third-party code can cimport it. Then, while writing the third party >>> code, you cimport the my package .pxd, and obviously there is no .pyx >>> for my package, as you are writing YOUR code. What's going to happen >>> in this case? >> >> If you want to use cpdef in your .pxd, it means you want to expose >> those to Python. This means they must be importable from Python, which >> means you need an extension module that exposes the constants, which >> means you need a .pyx. So you ship a (possibly empty) .pyx file along >> with your .pxd. If you don't want to expose them to Python but only to >> Cython, don't use cpdef. This compile time error would be issued >> whenever a user does a cimport of a .pxd file that has a cpdef enum >> with no corresponding .pyx file, i.e. the pxd is unusable, you >> wouldn't ship it in the first place. >> > > So supose I want to write a small wrapper for dlopen(), then I do: > > # dl.pxd > cdef from extern from "dlfcn.h": > ? ?cpdef enum: RTLD_GLOBAL > ? ?void * dlopen()(const char *filename, int flag); > > # dl.pyx > def open(filename, mode): > ? ?cdef void *handle = c_dlopen(fiename,handle) > ? ?return handle > > Then the "dl" module namespace contains "dlopen" and "RTLD_GLOBAL" > > Now, suppose I code my setup.py to build the ext module and install > dl.pxd as package data. > > Now a user runs "pip install dl" and installs my package. She wants to > reuse my .pxd (installed somewhere) for calling dlopen() in its own > Cython code: > > # user.pyx > from dl cimport RTLD_GLOBAL, c_dlopen > dlopen("somelibrary.so", RTLD_GLOBAL) > > > So the user code is going to fail? Then I cannot take advantage of > cpdef enum for developing my ?"dl" module, I have to go back to > manually exposing the enumerations > Why can't you ship both dl.pxd and dl.pyx as package data? > > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From dalcinl at gmail.com Wed Jul 20 21:44:08 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 20 Jul 2011 16:44:08 -0300 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On 20 July 2011 16:27, mark florisson wrote: > On 20 July 2011 21:13, Lisandro Dalcin wrote: >> On 20 July 2011 15:32, mark florisson wrote: >>> On 20 July 2011 20:04, Lisandro Dalcin wrote: >>>> On 20 July 2011 13:51, mark florisson wrote: >>>>> On 20 July 2011 18:06, Lisandro Dalcin wrote: >>>>>> On 19 July 2011 20:48, Robert Bradshaw wrote: >>>>>>> On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin wrote: >>>>>>>> On 19 July 2011 02:24, Vitja Makarov wrote: >>>>>>>>> 2011/7/18 Robert Bradshaw : >>>>>>>>>> Trevor King and I discussed this quite a while back, but every time I >>>>>>>>>> got around to looking at his code (I don't think he ever created a >>>>>>>>>> formal pull request) something came up. The idea was that we could >>>>>>>>>> support cpdef structs and extern functions as well. >>>>>>>>>> >>>>>>>>> >>>>>>>>> That's interesting, I think I shouldn't hurry with my pull request. >>>>>>>>> >>>>>>>>> 2011/7/19 Robert Bradshaw : >>>>>>>>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>>>>>>>>> My suggestion is >>>>>>>>>>> >>>>>>>>>>> ?cdef exposed enum: >>>>>>>>>>> ? ?... >>>>>>>>>> >>>>>>>>>> I agree, public is an overloaded word. This meaning is analogous to >>>>>>>>>> its use for cdef class members. Perhaps we should use "api" for api >>>>>>>>>> generation, and public used for Python-level access, with cpdef being >>>>>>>>>> the preferred form for declaring Python-accessible types. >>>>>>>>>> >>>>>>>>> >>>>>>>>> It seems to me that cpdef is more cythonic but exposed could be used >>>>>>>>> in this case: >>>>>>>>> >>>>>>>>> cdef extern from "ev.h": >>>>>>>>> ? ?exposed enum: >>>>>>>>> ? ? ? ?EV_READ >>>>>>>>> ? ? ? ?EV_WRITE >>>>>>>>> >>>>>>>> >>>>>>>> And what about this? >>>>>>>> >>>>>>>> cdef extern from "ev.h": >>>>>>>> ? cpdef enum: >>>>>>>> ? ? ? EV_READ >>>>>>>> ? ? ? EV_WRITE >>>>>>> >>>>>>> Yep, exactly. >>>>>>> >>>>>>>> BTW, how is this supposed to work with *.pxd files? I think the values >>>>>>>> will be exposed just in the matching implementation .pyx file, and not >>>>>>>> with cimport, right? >>>>>>> >>>>>>> It would be an error unless there's an corresponding .pyx file (which >>>>>>> could be empty). The idea is that one could also define extern cpdef >>>>>>> functions and structs and wrappers would be provided. >>>>>>> >>>>>> >>>>>> It would be an error? What do you mean? if you cpdef enum in foo.pxd, >>>>>> and have foo.pyx, then the enumerations should be exposed in the 'foo' >>>>>> module. But then if you "cimport foo" in bar.pyx, that should succeed >>>>>> but the enumerations should not be exposed in the "bar" module... Am I >>>>>> missing something? >>>>>> >>>>> >>>>> I believe Robert confirmed what you said and added to that that it >>>>> would be an error to have a cpdef extern enum in a .pxd without a >>>>> corresponding .pyx file. >>>>> >>>> >>>> But what an "error" means? Cython is going to fail compilation? >>>> Suppose I develop a package, and at setup.py install time my package >>>> installs somewhere a .pxd full of definitions in such a way that >>>> third-party code can cimport it. Then, while writing the third party >>>> code, you cimport the my package .pxd, and obviously there is no .pyx >>>> for my package, as you are writing YOUR code. What's going to happen >>>> in this case? >>> >>> If you want to use cpdef in your .pxd, it means you want to expose >>> those to Python. This means they must be importable from Python, which >>> means you need an extension module that exposes the constants, which >>> means you need a .pyx. So you ship a (possibly empty) .pyx file along >>> with your .pxd. If you don't want to expose them to Python but only to >>> Cython, don't use cpdef. This compile time error would be issued >>> whenever a user does a cimport of a .pxd file that has a cpdef enum >>> with no corresponding .pyx file, i.e. the pxd is unusable, you >>> wouldn't ship it in the first place. >>> >> >> So supose I want to write a small wrapper for dlopen(), then I do: >> >> # dl.pxd >> cdef from extern from "dlfcn.h": >> ? ?cpdef enum: RTLD_GLOBAL >> ? ?void * dlopen()(const char *filename, int flag); >> >> # dl.pyx >> def open(filename, mode): >> ? ?cdef void *handle = c_dlopen(fiename,handle) >> ? ?return handle >> >> Then the "dl" module namespace contains "dlopen" and "RTLD_GLOBAL" >> >> Now, suppose I code my setup.py to build the ext module and install >> dl.pxd as package data. >> >> Now a user runs "pip install dl" and installs my package. She wants to >> reuse my .pxd (installed somewhere) for calling dlopen() in its own >> Cython code: >> >> # user.pyx >> from dl cimport RTLD_GLOBAL, c_dlopen >> dlopen("somelibrary.so", RTLD_GLOBAL) >> >> >> So the user code is going to fail? Then I cannot take advantage of >> cpdef enum for developing my ?"dl" module, I have to go back to >> manually exposing the enumerations >> > > Why can't you ship both dl.pxd and dl.pyx as package data? > This is like asking to provide a C source in order to being able to #include "someheader.h" in other C sources. End users do not need implementation source code, they need declarations,interface,header files. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From markflorisson88 at gmail.com Wed Jul 20 21:53:10 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 20 Jul 2011 21:53:10 +0200 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On 20 July 2011 21:44, Lisandro Dalcin wrote: > On 20 July 2011 16:27, mark florisson wrote: >> On 20 July 2011 21:13, Lisandro Dalcin wrote: >>> On 20 July 2011 15:32, mark florisson wrote: >>>> On 20 July 2011 20:04, Lisandro Dalcin wrote: >>>>> On 20 July 2011 13:51, mark florisson wrote: >>>>>> On 20 July 2011 18:06, Lisandro Dalcin wrote: >>>>>>> On 19 July 2011 20:48, Robert Bradshaw wrote: >>>>>>>> On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin wrote: >>>>>>>>> On 19 July 2011 02:24, Vitja Makarov wrote: >>>>>>>>>> 2011/7/18 Robert Bradshaw : >>>>>>>>>>> Trevor King and I discussed this quite a while back, but every time I >>>>>>>>>>> got around to looking at his code (I don't think he ever created a >>>>>>>>>>> formal pull request) something came up. The idea was that we could >>>>>>>>>>> support cpdef structs and extern functions as well. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> That's interesting, I think I shouldn't hurry with my pull request. >>>>>>>>>> >>>>>>>>>> 2011/7/19 Robert Bradshaw : >>>>>>>>>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>>>>>>>>>> My suggestion is >>>>>>>>>>>> >>>>>>>>>>>> ?cdef exposed enum: >>>>>>>>>>>> ? ?... >>>>>>>>>>> >>>>>>>>>>> I agree, public is an overloaded word. This meaning is analogous to >>>>>>>>>>> its use for cdef class members. Perhaps we should use "api" for api >>>>>>>>>>> generation, and public used for Python-level access, with cpdef being >>>>>>>>>>> the preferred form for declaring Python-accessible types. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> It seems to me that cpdef is more cythonic but exposed could be used >>>>>>>>>> in this case: >>>>>>>>>> >>>>>>>>>> cdef extern from "ev.h": >>>>>>>>>> ? ?exposed enum: >>>>>>>>>> ? ? ? ?EV_READ >>>>>>>>>> ? ? ? ?EV_WRITE >>>>>>>>>> >>>>>>>>> >>>>>>>>> And what about this? >>>>>>>>> >>>>>>>>> cdef extern from "ev.h": >>>>>>>>> ? cpdef enum: >>>>>>>>> ? ? ? EV_READ >>>>>>>>> ? ? ? EV_WRITE >>>>>>>> >>>>>>>> Yep, exactly. >>>>>>>> >>>>>>>>> BTW, how is this supposed to work with *.pxd files? I think the values >>>>>>>>> will be exposed just in the matching implementation .pyx file, and not >>>>>>>>> with cimport, right? >>>>>>>> >>>>>>>> It would be an error unless there's an corresponding .pyx file (which >>>>>>>> could be empty). The idea is that one could also define extern cpdef >>>>>>>> functions and structs and wrappers would be provided. >>>>>>>> >>>>>>> >>>>>>> It would be an error? What do you mean? if you cpdef enum in foo.pxd, >>>>>>> and have foo.pyx, then the enumerations should be exposed in the 'foo' >>>>>>> module. But then if you "cimport foo" in bar.pyx, that should succeed >>>>>>> but the enumerations should not be exposed in the "bar" module... Am I >>>>>>> missing something? >>>>>>> >>>>>> >>>>>> I believe Robert confirmed what you said and added to that that it >>>>>> would be an error to have a cpdef extern enum in a .pxd without a >>>>>> corresponding .pyx file. >>>>>> >>>>> >>>>> But what an "error" means? Cython is going to fail compilation? >>>>> Suppose I develop a package, and at setup.py install time my package >>>>> installs somewhere a .pxd full of definitions in such a way that >>>>> third-party code can cimport it. Then, while writing the third party >>>>> code, you cimport the my package .pxd, and obviously there is no .pyx >>>>> for my package, as you are writing YOUR code. What's going to happen >>>>> in this case? >>>> >>>> If you want to use cpdef in your .pxd, it means you want to expose >>>> those to Python. This means they must be importable from Python, which >>>> means you need an extension module that exposes the constants, which >>>> means you need a .pyx. So you ship a (possibly empty) .pyx file along >>>> with your .pxd. If you don't want to expose them to Python but only to >>>> Cython, don't use cpdef. This compile time error would be issued >>>> whenever a user does a cimport of a .pxd file that has a cpdef enum >>>> with no corresponding .pyx file, i.e. the pxd is unusable, you >>>> wouldn't ship it in the first place. >>>> >>> >>> So supose I want to write a small wrapper for dlopen(), then I do: >>> >>> # dl.pxd >>> cdef from extern from "dlfcn.h": >>> ? ?cpdef enum: RTLD_GLOBAL >>> ? ?void * dlopen()(const char *filename, int flag); >>> >>> # dl.pyx >>> def open(filename, mode): >>> ? ?cdef void *handle = c_dlopen(fiename,handle) >>> ? ?return handle >>> >>> Then the "dl" module namespace contains "dlopen" and "RTLD_GLOBAL" >>> >>> Now, suppose I code my setup.py to build the ext module and install >>> dl.pxd as package data. >>> >>> Now a user runs "pip install dl" and installs my package. She wants to >>> reuse my .pxd (installed somewhere) for calling dlopen() in its own >>> Cython code: >>> >>> # user.pyx >>> from dl cimport RTLD_GLOBAL, c_dlopen >>> dlopen("somelibrary.so", RTLD_GLOBAL) >>> >>> >>> So the user code is going to fail? Then I cannot take advantage of >>> cpdef enum for developing my ?"dl" module, I have to go back to >>> manually exposing the enumerations >>> >> >> Why can't you ship both dl.pxd and dl.pyx as package data? >> > > This is like asking to provide a C source in order to being able to > #include "someheader.h" in other C sources. End users do not need > implementation source code, they need declarations,interface,header > files. > I personally don't really mind, but what about not allowing cpdef in .pxds for enums, but only in .pyx files: from mypxd cimport my_extern_enum cpdef enum my_extern_enum Of course, the usual cpdef enum my_enum: ... would still be allowed in .pyx files. > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From robertwb at math.washington.edu Wed Jul 20 23:24:50 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 20 Jul 2011 14:24:50 -0700 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On Wed, Jul 20, 2011 at 12:53 PM, mark florisson wrote: > On 20 July 2011 21:44, Lisandro Dalcin wrote: >> On 20 July 2011 16:27, mark florisson wrote: >>> On 20 July 2011 21:13, Lisandro Dalcin wrote: >>>> On 20 July 2011 15:32, mark florisson wrote: >>>>> On 20 July 2011 20:04, Lisandro Dalcin wrote: >>>>>> On 20 July 2011 13:51, mark florisson wrote: >>>>>>> On 20 July 2011 18:06, Lisandro Dalcin wrote: >>>>>>>> On 19 July 2011 20:48, Robert Bradshaw wrote: >>>>>>>>> On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin wrote: >>>>>>>>>> On 19 July 2011 02:24, Vitja Makarov wrote: >>>>>>>>>>> 2011/7/18 Robert Bradshaw : >>>>>>>>>>>> Trevor King and I discussed this quite a while back, but every time I >>>>>>>>>>>> got around to looking at his code (I don't think he ever created a >>>>>>>>>>>> formal pull request) something came up. The idea was that we could >>>>>>>>>>>> support cpdef structs and extern functions as well. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> That's interesting, I think I shouldn't hurry with my pull request. >>>>>>>>>>> >>>>>>>>>>> 2011/7/19 Robert Bradshaw : >>>>>>>>>>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>>>>>>>>>>> My suggestion is >>>>>>>>>>>>> >>>>>>>>>>>>> ?cdef exposed enum: >>>>>>>>>>>>> ? ?... >>>>>>>>>>>> >>>>>>>>>>>> I agree, public is an overloaded word. This meaning is analogous to >>>>>>>>>>>> its use for cdef class members. Perhaps we should use "api" for api >>>>>>>>>>>> generation, and public used for Python-level access, with cpdef being >>>>>>>>>>>> the preferred form for declaring Python-accessible types. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> It seems to me that cpdef is more cythonic but exposed could be used >>>>>>>>>>> in this case: >>>>>>>>>>> >>>>>>>>>>> cdef extern from "ev.h": >>>>>>>>>>> ? ?exposed enum: >>>>>>>>>>> ? ? ? ?EV_READ >>>>>>>>>>> ? ? ? ?EV_WRITE >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> And what about this? >>>>>>>>>> >>>>>>>>>> cdef extern from "ev.h": >>>>>>>>>> ? cpdef enum: >>>>>>>>>> ? ? ? EV_READ >>>>>>>>>> ? ? ? EV_WRITE >>>>>>>>> >>>>>>>>> Yep, exactly. >>>>>>>>> >>>>>>>>>> BTW, how is this supposed to work with *.pxd files? I think the values >>>>>>>>>> will be exposed just in the matching implementation .pyx file, and not >>>>>>>>>> with cimport, right? >>>>>>>>> >>>>>>>>> It would be an error unless there's an corresponding .pyx file (which >>>>>>>>> could be empty). The idea is that one could also define extern cpdef >>>>>>>>> functions and structs and wrappers would be provided. >>>>>>>>> >>>>>>>> >>>>>>>> It would be an error? What do you mean? if you cpdef enum in foo.pxd, >>>>>>>> and have foo.pyx, then the enumerations should be exposed in the 'foo' >>>>>>>> module. But then if you "cimport foo" in bar.pyx, that should succeed >>>>>>>> but the enumerations should not be exposed in the "bar" module... Am I >>>>>>>> missing something? >>>>>>>> >>>>>>> >>>>>>> I believe Robert confirmed what you said and added to that that it >>>>>>> would be an error to have a cpdef extern enum in a .pxd without a >>>>>>> corresponding .pyx file. >>>>>>> >>>>>> >>>>>> But what an "error" means? Cython is going to fail compilation? >>>>>> Suppose I develop a package, and at setup.py install time my package >>>>>> installs somewhere a .pxd full of definitions in such a way that >>>>>> third-party code can cimport it. Then, while writing the third party >>>>>> code, you cimport the my package .pxd, and obviously there is no .pyx >>>>>> for my package, as you are writing YOUR code. What's going to happen >>>>>> in this case? >>>>> >>>>> If you want to use cpdef in your .pxd, it means you want to expose >>>>> those to Python. This means they must be importable from Python, which >>>>> means you need an extension module that exposes the constants, which >>>>> means you need a .pyx. So you ship a (possibly empty) .pyx file along >>>>> with your .pxd. If you don't want to expose them to Python but only to >>>>> Cython, don't use cpdef. This compile time error would be issued >>>>> whenever a user does a cimport of a .pxd file that has a cpdef enum >>>>> with no corresponding .pyx file, i.e. the pxd is unusable, you >>>>> wouldn't ship it in the first place. Alternatively it would be a runtime error when trying to import the objects. I suppose if cimport does not inject anything into the python namespace, then "cimport" could just work and "import" would work iff the module is available at runtime. >>>> So supose I want to write a small wrapper for dlopen(), then I do: >>>> >>>> # dl.pxd >>>> cdef from extern from "dlfcn.h": >>>> ? ?cpdef enum: RTLD_GLOBAL >>>> ? ?void * dlopen()(const char *filename, int flag); >>>> >>>> # dl.pyx >>>> def open(filename, mode): >>>> ? ?cdef void *handle = c_dlopen(fiename,handle) >>>> ? ?return handle >>>> >>>> Then the "dl" module namespace contains "dlopen" and "RTLD_GLOBAL" >>>> >>>> Now, suppose I code my setup.py to build the ext module and install >>>> dl.pxd as package data. >>>> >>>> Now a user runs "pip install dl" and installs my package. She wants to >>>> reuse my .pxd (installed somewhere) for calling dlopen() in its own >>>> Cython code: >>>> >>>> # user.pyx >>>> from dl cimport RTLD_GLOBAL, c_dlopen >>>> dlopen("somelibrary.so", RTLD_GLOBAL) >>>> >>>> >>>> So the user code is going to fail? Then I cannot take advantage of >>>> cpdef enum for developing my ?"dl" module, I have to go back to >>>> manually exposing the enumerations >>>> >>> >>> Why can't you ship both dl.pxd and dl.pyx as package data? >>> >> >> This is like asking to provide a C source in order to being able to >> #include "someheader.h" in other C sources. End users do not need >> implementation source code, they need declarations,interface,header >> files. Because we all love link errors :). > I personally don't really mind, but what about not allowing cpdef in > .pxds for enums, but only in .pyx files: > > from mypxd cimport my_extern_enum > cpdef enum my_extern_enum > > Of course, the usual > > cpdef enum my_enum: > ? ... > > would still be allowed in .pyx files. Hmm... that's an idea, though it means declaring things twice. Also, what about big lists of enums in, e.g. numpy.pxd or the system headers? I'm perhaps OK with them being like cdef classes--the .pyx file is not needed at compile time it's a runtime error on import if the corresponding module is not available. - Robert From markflorisson88 at gmail.com Wed Jul 20 23:53:23 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 20 Jul 2011 23:53:23 +0200 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On 20 July 2011 23:24, Robert Bradshaw wrote: > On Wed, Jul 20, 2011 at 12:53 PM, mark florisson > wrote: >> On 20 July 2011 21:44, Lisandro Dalcin wrote: >>> On 20 July 2011 16:27, mark florisson wrote: >>>> On 20 July 2011 21:13, Lisandro Dalcin wrote: >>>>> On 20 July 2011 15:32, mark florisson wrote: >>>>>> On 20 July 2011 20:04, Lisandro Dalcin wrote: >>>>>>> On 20 July 2011 13:51, mark florisson wrote: >>>>>>>> On 20 July 2011 18:06, Lisandro Dalcin wrote: >>>>>>>>> On 19 July 2011 20:48, Robert Bradshaw wrote: >>>>>>>>>> On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin wrote: >>>>>>>>>>> On 19 July 2011 02:24, Vitja Makarov wrote: >>>>>>>>>>>> 2011/7/18 Robert Bradshaw : >>>>>>>>>>>>> Trevor King and I discussed this quite a while back, but every time I >>>>>>>>>>>>> got around to looking at his code (I don't think he ever created a >>>>>>>>>>>>> formal pull request) something came up. The idea was that we could >>>>>>>>>>>>> support cpdef structs and extern functions as well. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> That's interesting, I think I shouldn't hurry with my pull request. >>>>>>>>>>>> >>>>>>>>>>>> 2011/7/19 Robert Bradshaw : >>>>>>>>>>>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>>>>>>>>>>>> My suggestion is >>>>>>>>>>>>>> >>>>>>>>>>>>>> ?cdef exposed enum: >>>>>>>>>>>>>> ? ?... >>>>>>>>>>>>> >>>>>>>>>>>>> I agree, public is an overloaded word. This meaning is analogous to >>>>>>>>>>>>> its use for cdef class members. Perhaps we should use "api" for api >>>>>>>>>>>>> generation, and public used for Python-level access, with cpdef being >>>>>>>>>>>>> the preferred form for declaring Python-accessible types. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> It seems to me that cpdef is more cythonic but exposed could be used >>>>>>>>>>>> in this case: >>>>>>>>>>>> >>>>>>>>>>>> cdef extern from "ev.h": >>>>>>>>>>>> ? ?exposed enum: >>>>>>>>>>>> ? ? ? ?EV_READ >>>>>>>>>>>> ? ? ? ?EV_WRITE >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> And what about this? >>>>>>>>>>> >>>>>>>>>>> cdef extern from "ev.h": >>>>>>>>>>> ? cpdef enum: >>>>>>>>>>> ? ? ? EV_READ >>>>>>>>>>> ? ? ? EV_WRITE >>>>>>>>>> >>>>>>>>>> Yep, exactly. >>>>>>>>>> >>>>>>>>>>> BTW, how is this supposed to work with *.pxd files? I think the values >>>>>>>>>>> will be exposed just in the matching implementation .pyx file, and not >>>>>>>>>>> with cimport, right? >>>>>>>>>> >>>>>>>>>> It would be an error unless there's an corresponding .pyx file (which >>>>>>>>>> could be empty). The idea is that one could also define extern cpdef >>>>>>>>>> functions and structs and wrappers would be provided. >>>>>>>>>> >>>>>>>>> >>>>>>>>> It would be an error? What do you mean? if you cpdef enum in foo.pxd, >>>>>>>>> and have foo.pyx, then the enumerations should be exposed in the 'foo' >>>>>>>>> module. But then if you "cimport foo" in bar.pyx, that should succeed >>>>>>>>> but the enumerations should not be exposed in the "bar" module... Am I >>>>>>>>> missing something? >>>>>>>>> >>>>>>>> >>>>>>>> I believe Robert confirmed what you said and added to that that it >>>>>>>> would be an error to have a cpdef extern enum in a .pxd without a >>>>>>>> corresponding .pyx file. >>>>>>>> >>>>>>> >>>>>>> But what an "error" means? Cython is going to fail compilation? >>>>>>> Suppose I develop a package, and at setup.py install time my package >>>>>>> installs somewhere a .pxd full of definitions in such a way that >>>>>>> third-party code can cimport it. Then, while writing the third party >>>>>>> code, you cimport the my package .pxd, and obviously there is no .pyx >>>>>>> for my package, as you are writing YOUR code. What's going to happen >>>>>>> in this case? >>>>>> >>>>>> If you want to use cpdef in your .pxd, it means you want to expose >>>>>> those to Python. This means they must be importable from Python, which >>>>>> means you need an extension module that exposes the constants, which >>>>>> means you need a .pyx. So you ship a (possibly empty) .pyx file along >>>>>> with your .pxd. If you don't want to expose them to Python but only to >>>>>> Cython, don't use cpdef. This compile time error would be issued >>>>>> whenever a user does a cimport of a .pxd file that has a cpdef enum >>>>>> with no corresponding .pyx file, i.e. the pxd is unusable, you >>>>>> wouldn't ship it in the first place. > > Alternatively it would be a runtime error when trying to import the > objects. I suppose if cimport does not inject anything into the python > namespace, then "cimport" could just work and "import" would work iff > the module is available at runtime. > >>>>> So supose I want to write a small wrapper for dlopen(), then I do: >>>>> >>>>> # dl.pxd >>>>> cdef from extern from "dlfcn.h": >>>>> ? ?cpdef enum: RTLD_GLOBAL >>>>> ? ?void * dlopen()(const char *filename, int flag); >>>>> >>>>> # dl.pyx >>>>> def open(filename, mode): >>>>> ? ?cdef void *handle = c_dlopen(fiename,handle) >>>>> ? ?return handle >>>>> >>>>> Then the "dl" module namespace contains "dlopen" and "RTLD_GLOBAL" >>>>> >>>>> Now, suppose I code my setup.py to build the ext module and install >>>>> dl.pxd as package data. >>>>> >>>>> Now a user runs "pip install dl" and installs my package. She wants to >>>>> reuse my .pxd (installed somewhere) for calling dlopen() in its own >>>>> Cython code: >>>>> >>>>> # user.pyx >>>>> from dl cimport RTLD_GLOBAL, c_dlopen >>>>> dlopen("somelibrary.so", RTLD_GLOBAL) >>>>> >>>>> >>>>> So the user code is going to fail? Then I cannot take advantage of >>>>> cpdef enum for developing my ?"dl" module, I have to go back to >>>>> manually exposing the enumerations >>>>> >>>> >>>> Why can't you ship both dl.pxd and dl.pyx as package data? >>>> >>> >>> This is like asking to provide a C source in order to being able to >>> #include "someheader.h" in other C sources. End users do not need >>> implementation source code, they need declarations,interface,header >>> files. > > Because we all love link errors :). > >> I personally don't really mind, but what about not allowing cpdef in >> .pxds for enums, but only in .pyx files: >> >> from mypxd cimport my_extern_enum >> cpdef enum my_extern_enum >> >> Of course, the usual >> >> cpdef enum my_enum: >> ? ... >> >> would still be allowed in .pyx files. > > Hmm... that's an idea, though it means declaring things twice. Also, > what about big lists of enums in, e.g. numpy.pxd or the system > headers? You'd declare the enum twice, but the contents once. In the pxd you list the items, in the pyx you merely say "ok expose this entire thing to Python for me". > I'm perhaps OK with them being like cdef classes--the .pyx file is not > needed at compile time it's a runtime error on import if the > corresponding module is not available. With cdef classes you need the import to access the data, but with enums you don't so I think it's a little weird. I'd prefer a compile time error or a warning perhaps. I don't think installing source files is a bad idea (I'm usually disappointed when I only find .pyc files and not the actual source, same for .pxd). > - Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From robertwb at math.washington.edu Thu Jul 21 00:32:03 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 20 Jul 2011 15:32:03 -0700 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: On Wed, Jul 20, 2011 at 2:53 PM, mark florisson wrote: > On 20 July 2011 23:24, Robert Bradshaw wrote: >> On Wed, Jul 20, 2011 at 12:53 PM, mark florisson >> wrote: >>> On 20 July 2011 21:44, Lisandro Dalcin wrote: >>>> On 20 July 2011 16:27, mark florisson wrote: >>>>> On 20 July 2011 21:13, Lisandro Dalcin wrote: >>>>>> On 20 July 2011 15:32, mark florisson wrote: >>>>>>> On 20 July 2011 20:04, Lisandro Dalcin wrote: >>>>>>>> On 20 July 2011 13:51, mark florisson wrote: >>>>>>>>> On 20 July 2011 18:06, Lisandro Dalcin wrote: >>>>>>>>>> On 19 July 2011 20:48, Robert Bradshaw wrote: >>>>>>>>>>> On Tue, Jul 19, 2011 at 3:02 PM, Lisandro Dalcin wrote: >>>>>>>>>>>> On 19 July 2011 02:24, Vitja Makarov wrote: >>>>>>>>>>>>> 2011/7/18 Robert Bradshaw : >>>>>>>>>>>>>> Trevor King and I discussed this quite a while back, but every time I >>>>>>>>>>>>>> got around to looking at his code (I don't think he ever created a >>>>>>>>>>>>>> formal pull request) something came up. The idea was that we could >>>>>>>>>>>>>> support cpdef structs and extern functions as well. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> That's interesting, I think I shouldn't hurry with my pull request. >>>>>>>>>>>>> >>>>>>>>>>>>> 2011/7/19 Robert Bradshaw : >>>>>>>>>>>>>> On Mon, Jul 18, 2011 at 4:34 PM, Greg Ewing wrote: >>>>>>>>>>>>>>> My suggestion is >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ?cdef exposed enum: >>>>>>>>>>>>>>> ? ?... >>>>>>>>>>>>>> >>>>>>>>>>>>>> I agree, public is an overloaded word. This meaning is analogous to >>>>>>>>>>>>>> its use for cdef class members. Perhaps we should use "api" for api >>>>>>>>>>>>>> generation, and public used for Python-level access, with cpdef being >>>>>>>>>>>>>> the preferred form for declaring Python-accessible types. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> It seems to me that cpdef is more cythonic but exposed could be used >>>>>>>>>>>>> in this case: >>>>>>>>>>>>> >>>>>>>>>>>>> cdef extern from "ev.h": >>>>>>>>>>>>> ? ?exposed enum: >>>>>>>>>>>>> ? ? ? ?EV_READ >>>>>>>>>>>>> ? ? ? ?EV_WRITE >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> And what about this? >>>>>>>>>>>> >>>>>>>>>>>> cdef extern from "ev.h": >>>>>>>>>>>> ? cpdef enum: >>>>>>>>>>>> ? ? ? EV_READ >>>>>>>>>>>> ? ? ? EV_WRITE >>>>>>>>>>> >>>>>>>>>>> Yep, exactly. >>>>>>>>>>> >>>>>>>>>>>> BTW, how is this supposed to work with *.pxd files? I think the values >>>>>>>>>>>> will be exposed just in the matching implementation .pyx file, and not >>>>>>>>>>>> with cimport, right? >>>>>>>>>>> >>>>>>>>>>> It would be an error unless there's an corresponding .pyx file (which >>>>>>>>>>> could be empty). The idea is that one could also define extern cpdef >>>>>>>>>>> functions and structs and wrappers would be provided. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> It would be an error? What do you mean? if you cpdef enum in foo.pxd, >>>>>>>>>> and have foo.pyx, then the enumerations should be exposed in the 'foo' >>>>>>>>>> module. But then if you "cimport foo" in bar.pyx, that should succeed >>>>>>>>>> but the enumerations should not be exposed in the "bar" module... Am I >>>>>>>>>> missing something? >>>>>>>>>> >>>>>>>>> >>>>>>>>> I believe Robert confirmed what you said and added to that that it >>>>>>>>> would be an error to have a cpdef extern enum in a .pxd without a >>>>>>>>> corresponding .pyx file. >>>>>>>>> >>>>>>>> >>>>>>>> But what an "error" means? Cython is going to fail compilation? >>>>>>>> Suppose I develop a package, and at setup.py install time my package >>>>>>>> installs somewhere a .pxd full of definitions in such a way that >>>>>>>> third-party code can cimport it. Then, while writing the third party >>>>>>>> code, you cimport the my package .pxd, and obviously there is no .pyx >>>>>>>> for my package, as you are writing YOUR code. What's going to happen >>>>>>>> in this case? >>>>>>> >>>>>>> If you want to use cpdef in your .pxd, it means you want to expose >>>>>>> those to Python. This means they must be importable from Python, which >>>>>>> means you need an extension module that exposes the constants, which >>>>>>> means you need a .pyx. So you ship a (possibly empty) .pyx file along >>>>>>> with your .pxd. If you don't want to expose them to Python but only to >>>>>>> Cython, don't use cpdef. This compile time error would be issued >>>>>>> whenever a user does a cimport of a .pxd file that has a cpdef enum >>>>>>> with no corresponding .pyx file, i.e. the pxd is unusable, you >>>>>>> wouldn't ship it in the first place. >> >> Alternatively it would be a runtime error when trying to import the >> objects. I suppose if cimport does not inject anything into the python >> namespace, then "cimport" could just work and "import" would work iff >> the module is available at runtime. >> >>>>>> So supose I want to write a small wrapper for dlopen(), then I do: >>>>>> >>>>>> # dl.pxd >>>>>> cdef from extern from "dlfcn.h": >>>>>> ? ?cpdef enum: RTLD_GLOBAL >>>>>> ? ?void * dlopen()(const char *filename, int flag); >>>>>> >>>>>> # dl.pyx >>>>>> def open(filename, mode): >>>>>> ? ?cdef void *handle = c_dlopen(fiename,handle) >>>>>> ? ?return handle >>>>>> >>>>>> Then the "dl" module namespace contains "dlopen" and "RTLD_GLOBAL" >>>>>> >>>>>> Now, suppose I code my setup.py to build the ext module and install >>>>>> dl.pxd as package data. >>>>>> >>>>>> Now a user runs "pip install dl" and installs my package. She wants to >>>>>> reuse my .pxd (installed somewhere) for calling dlopen() in its own >>>>>> Cython code: >>>>>> >>>>>> # user.pyx >>>>>> from dl cimport RTLD_GLOBAL, c_dlopen >>>>>> dlopen("somelibrary.so", RTLD_GLOBAL) >>>>>> >>>>>> >>>>>> So the user code is going to fail? Then I cannot take advantage of >>>>>> cpdef enum for developing my ?"dl" module, I have to go back to >>>>>> manually exposing the enumerations >>>>>> >>>>> >>>>> Why can't you ship both dl.pxd and dl.pyx as package data? >>>>> >>>> >>>> This is like asking to provide a C source in order to being able to >>>> #include "someheader.h" in other C sources. End users do not need >>>> implementation source code, they need declarations,interface,header >>>> files. >> >> Because we all love link errors :). >> >>> I personally don't really mind, but what about not allowing cpdef in >>> .pxds for enums, but only in .pyx files: >>> >>> from mypxd cimport my_extern_enum >>> cpdef enum my_extern_enum >>> >>> Of course, the usual >>> >>> cpdef enum my_enum: >>> ? ... >>> >>> would still be allowed in .pyx files. >> >> Hmm... that's an idea, though it means declaring things twice. Also, >> what about big lists of enums in, e.g. numpy.pxd or the system >> headers? > > You'd declare the enum twice, but the contents once. In the pxd you > list the items, in the pyx you merely say "ok expose this entire thing > to Python for me". It's not as bad for enums, but would be more of a pain for automatically wrapped functions/structs. And redundancy is redundancy. >> I'm perhaps OK with them being like cdef classes--the .pyx file is not >> needed at compile time it's a runtime error on import if the >> corresponding module is not available. > > With cdef classes you need the import to access the data, but with > enums you don't so I think it's a little weird. Alternatively, we could force you to import them iff used in a Python context, though I'd rather have it unconditional. Think cpdef structs/functions converting ito Python objects--even enums might be nice to convert to Python as a new type with an __int__ function rather than raw ints so they print nice and can be accessed like my_enum_type.[tab]). > I'd prefer a compile > time error or a warning perhaps. I don't think installing source files > is a bad idea (I'm usually disappointed when I only find .pyc files > and not the actual source, same for .pxd). I have similar sentiments and I prefer compile time errors as well, but that might be too restrictive. - Robert From robertwb at math.washington.edu Thu Jul 21 02:19:24 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 20 Jul 2011 17:19:24 -0700 Subject: [Cython] Cython 0.15 release In-Reply-To: <4E26A42D.40301@behnel.de> References: <4E269F5E.7090907@behnel.de> <4E26A42D.40301@behnel.de> Message-ID: On Wed, Jul 20, 2011 at 2:47 AM, Stefan Behnel wrote: > mark florisson, 20.07.2011 11:40: >> >> On 20 July 2011 11:26, Stefan Behnel wrote: >>> >>> mark florisson, 20.07.2011 10:51: >>>> >>>> On 20 July 2011 02:32, Robert Bradshaw wrote: >>>>> >>>>> We're long overdue for a release, and this week would be a good one >>>>> for me to push one out. Hudson >>>>> https://sage.math.washington.edu:8091/hudson is looking in pretty good >>>>> shape, and though I know we've got a big pile of stuff currently in >>>>> progress, we've also got a big backlog of stuff to get out. I'd like >>>>> to finish looking at https://github.com/cython/cython/pull/38 , are >>>>> there any other changes that people want to urgently get in? Also, >>>>> I've started http://wiki.cython.org/ReleaseNotes-0.15 , feel free to >>>>> edit if you think anything should be highlighted. >>>> >>>> I think cpdef enum is a rather small but still very useful change for >>>> many people. >>> >>> Did we agree on the right syntax yet? We should be sure about that before >>> adding a new syntax feature to the language that we won't be able to >>> change >>> later on. >> >> I think we settled on the syntax in the thread called Cython .pxd >> introspection: listing defined constants, when Trevor was working on >> it: >> http://mail.python.org/pipermail/cython-devel/2011-February/000035.html >> >> It does look like the most sensible syntax to me, as cpdef is about >> exposing C/Cython-only stuff to Python. > > Ok. Does the current implementation raise syntax errors where appropriate, > in order to prevent the syntax from allowing to be applied in unwanted > places? What about the "cdef extern: ... cpdef enum" case? Is that to be > allowed? > > I know I'm being pedantic here, but these things matter. In any case, it sounds like there's enough uncertainty to not rush this into the release. - Robert From stefan_ml at behnel.de Thu Jul 21 04:41:23 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 21 Jul 2011 04:41:23 +0200 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: References: Message-ID: <4E2791D3.90601@behnel.de> Hi, I think we're getting way off course with this discussion. The following is a good way to get back into a useful direction. Robert Bradshaw, 20.07.2011 23:24: > it would be a runtime error when trying to import the > objects. I suppose if cimport does not inject anything into the python > namespace, then "cimport" could just work and "import" would work iff > the module is available at runtime. +1 Stefan From dalcinl at gmail.com Thu Jul 21 04:53:07 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Wed, 20 Jul 2011 23:53:07 -0300 Subject: [Cython] How to define C-consts in python module scope In-Reply-To: <4E2791D3.90601@behnel.de> References: <4E2791D3.90601@behnel.de> Message-ID: On 20 July 2011 23:41, Stefan Behnel wrote: > Hi, > > I think we're getting way off course with this discussion. The following is > a good way to get back into a useful direction. > > Robert Bradshaw, 20.07.2011 23:24: >> >> it would be a runtime error when trying to import the >> objects. I suppose if cimport does not inject anything into the python >> namespace, then "cimport" could just work and "import" would work iff >> the module is available at runtime. > > +1 > +1, too... That was my whole point! -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From vitja.makarov at gmail.com Thu Jul 21 06:27:29 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 21 Jul 2011 08:27:29 +0400 Subject: [Cython] Strange cimport behaviour In-Reply-To: References: Message-ID: 2011/7/16 Lars Buitinck : > 2011/7/15 Vitja Makarov : >> I've found strange bug. In my example cimport misses "fcntl.h" include: >> >> ((c2f2e12...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ cat ./fff.pyx >> cimport posix.unistd >> cimport posix.fcntl >> >> print posix.fcntl.O_RDWR > > Fascinating; I can reproduce the error (on Scientific Linux 5.5), but > it goes away when I reverse the order of the includes. > What's about this issue? Is that easy to fix before release or it's better to create a ticket? -- vitja. From robertwb at math.washington.edu Thu Jul 21 06:34:06 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 20 Jul 2011 21:34:06 -0700 Subject: [Cython] Strange cimport behaviour In-Reply-To: References: Message-ID: On Wed, Jul 20, 2011 at 9:27 PM, Vitja Makarov wrote: > 2011/7/16 Lars Buitinck : >> 2011/7/15 Vitja Makarov : >>> I've found strange bug. In my example cimport misses "fcntl.h" include: >>> >>> ((c2f2e12...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ cat ./fff.pyx >>> cimport posix.unistd >>> cimport posix.fcntl >>> >>> print posix.fcntl.O_RDWR >> >> Fascinating; I can reproduce the error (on Scientific Linux 5.5), but >> it goes away when I reverse the order of the includes. >> > > What's about this issue? Is that easy to fix before release or it's > better to create a ticket? I'm not sure what we can do here--we make sure to emit the #include statements in the same order as they are encountered in the Cython sources because C is sensitive to this kind of thing, but we can't really "fix" C. I suppose we could cimport posix.fcntl from within posix.unistd to force an ordering. - Robert From vitja.makarov at gmail.com Thu Jul 21 06:40:58 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 21 Jul 2011 08:40:58 +0400 Subject: [Cython] Strange cimport behaviour In-Reply-To: References: Message-ID: 2011/7/21 Robert Bradshaw : > On Wed, Jul 20, 2011 at 9:27 PM, Vitja Makarov wrote: >> 2011/7/16 Lars Buitinck : >>> 2011/7/15 Vitja Makarov : >>>> I've found strange bug. In my example cimport misses "fcntl.h" include: >>>> >>>> ((c2f2e12...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ cat ./fff.pyx >>>> cimport posix.unistd >>>> cimport posix.fcntl >>>> >>>> print posix.fcntl.O_RDWR >>> >>> Fascinating; I can reproduce the error (on Scientific Linux 5.5), but >>> it goes away when I reverse the order of the includes. >>> >> >> What's about this issue? Is that easy to fix before release or it's >> better to create a ticket? > > I'm not sure what we can do here--we make sure to emit the #include > statements in the same order as they are encountered in the Cython > sources because C is sensitive to this kind of thing, but we can't > really "fix" C. I suppose we could cimport posix.fcntl from within > posix.unistd to force an ordering. > That's not C bug. That's cython bug. Actualy fcntl.h isn't included at all. That seems to be a problem with cimport and packages. As I said "from posix cimport unistd, fcntl" works just fine. -- vitja. From robertwb at math.washington.edu Thu Jul 21 07:30:29 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 20 Jul 2011 22:30:29 -0700 Subject: [Cython] Strange cimport behaviour In-Reply-To: References: Message-ID: On Wed, Jul 20, 2011 at 9:40 PM, Vitja Makarov wrote: > 2011/7/21 Robert Bradshaw : >> On Wed, Jul 20, 2011 at 9:27 PM, Vitja Makarov wrote: >>> 2011/7/16 Lars Buitinck : >>>> 2011/7/15 Vitja Makarov : >>>>> I've found strange bug. In my example cimport misses "fcntl.h" include: >>>>> >>>>> ((c2f2e12...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ cat ./fff.pyx >>>>> cimport posix.unistd >>>>> cimport posix.fcntl >>>>> >>>>> print posix.fcntl.O_RDWR >>>> >>>> Fascinating; I can reproduce the error (on Scientific Linux 5.5), but >>>> it goes away when I reverse the order of the includes. >>>> >>> >>> What's about this issue? Is that easy to fix before release or it's >>> better to create a ticket? >> >> I'm not sure what we can do here--we make sure to emit the #include >> statements in the same order as they are encountered in the Cython >> sources because C is sensitive to this kind of thing, but we can't >> really "fix" C. I suppose we could cimport posix.fcntl from within >> posix.unistd to force an ordering. >> > > That's not C bug. That's cython bug. Actualy fcntl.h isn't included at all. > That seems to be a problem with cimport and packages. > > As I said "from posix cimport unistd, fcntl" works just fine. I thought you meant that cimport posix.fcntl cimport posix.unistd worked but cimport posix.unistd cimport posix.fcntl didn't. I'm seeing the bug now. (There are of course cases where one does care about import order.) - Robert From robertwb at math.washington.edu Thu Jul 21 08:58:37 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 20 Jul 2011 23:58:37 -0700 Subject: [Cython] Strange cimport behaviour In-Reply-To: References: Message-ID: On Wed, Jul 20, 2011 at 10:30 PM, Robert Bradshaw wrote: > On Wed, Jul 20, 2011 at 9:40 PM, Vitja Makarov wrote: >> 2011/7/21 Robert Bradshaw : >>> On Wed, Jul 20, 2011 at 9:27 PM, Vitja Makarov wrote: >>>> 2011/7/16 Lars Buitinck : >>>>> 2011/7/15 Vitja Makarov : >>>>>> I've found strange bug. In my example cimport misses "fcntl.h" include: >>>>>> >>>>>> ((c2f2e12...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ cat ./fff.pyx >>>>>> cimport posix.unistd >>>>>> cimport posix.fcntl >>>>>> >>>>>> print posix.fcntl.O_RDWR >>>>> >>>>> Fascinating; I can reproduce the error (on Scientific Linux 5.5), but >>>>> it goes away when I reverse the order of the includes. >>>>> >>>> >>>> What's about this issue? Is that easy to fix before release or it's >>>> better to create a ticket? >>> >>> I'm not sure what we can do here--we make sure to emit the #include >>> statements in the same order as they are encountered in the Cython >>> sources because C is sensitive to this kind of thing, but we can't >>> really "fix" C. I suppose we could cimport posix.fcntl from within >>> posix.unistd to force an ordering. >>> >> >> That's not C bug. That's cython bug. Actualy fcntl.h isn't included at all. >> That seems to be a problem with cimport and packages. >> >> As I said "from posix cimport unistd, fcntl" works just fine. > > I thought you meant that > > ? ?cimport posix.fcntl > ? ?cimport posix.unistd > > worked but > > ? ?cimport posix.unistd > ? ?cimport posix.fcntl > > didn't. I'm seeing the bug now. (There are of course cases where one > does care about import order.) https://github.com/cython/cython/commit/55d5e576935d83c6bdadc593e36793aecffe0bae From vitja.makarov at gmail.com Thu Jul 21 09:47:58 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 21 Jul 2011 11:47:58 +0400 Subject: [Cython] Strange cimport behaviour In-Reply-To: References: Message-ID: 2011/7/21 Robert Bradshaw : > On Wed, Jul 20, 2011 at 10:30 PM, Robert Bradshaw > wrote: >> On Wed, Jul 20, 2011 at 9:40 PM, Vitja Makarov wrote: >>> 2011/7/21 Robert Bradshaw : >>>> On Wed, Jul 20, 2011 at 9:27 PM, Vitja Makarov wrote: >>>>> 2011/7/16 Lars Buitinck : >>>>>> 2011/7/15 Vitja Makarov : >>>>>>> I've found strange bug. In my example cimport misses "fcntl.h" include: >>>>>>> >>>>>>> ((c2f2e12...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ cat ./fff.pyx >>>>>>> cimport posix.unistd >>>>>>> cimport posix.fcntl >>>>>>> >>>>>>> print posix.fcntl.O_RDWR >>>>>> >>>>>> Fascinating; I can reproduce the error (on Scientific Linux 5.5), but >>>>>> it goes away when I reverse the order of the includes. >>>>>> >>>>> >>>>> What's about this issue? Is that easy to fix before release or it's >>>>> better to create a ticket? >>>> >>>> I'm not sure what we can do here--we make sure to emit the #include >>>> statements in the same order as they are encountered in the Cython >>>> sources because C is sensitive to this kind of thing, but we can't >>>> really "fix" C. I suppose we could cimport posix.fcntl from within >>>> posix.unistd to force an ordering. >>>> >>> >>> That's not C bug. That's cython bug. Actualy fcntl.h isn't included at all. >>> That seems to be a problem with cimport and packages. >>> >>> As I said "from posix cimport unistd, fcntl" works just fine. >> >> I thought you meant that >> >> ? ?cimport posix.fcntl >> ? ?cimport posix.unistd >> >> worked but >> >> ? ?cimport posix.unistd >> ? ?cimport posix.fcntl >> >> didn't. I'm seeing the bug now. (There are of course cases where one >> does care about import order.) > > https://github.com/cython/cython/commit/55d5e576935d83c6bdadc593e36793aecffe0bae Cool, thanks! -- vitja. From robertwb at math.washington.edu Thu Jul 21 10:07:56 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 21 Jul 2011 01:07:56 -0700 Subject: [Cython] runtests libgomp fail In-Reply-To: References: Message-ID: On Thu, Jul 14, 2011 at 4:24 AM, mark florisson wrote: > I added a test for nested parallelism with exceptions (with OpenMP > nesting explicitly enabled), however if libgomp cannot create more > threads it exits the process with exit status 1 and the message > "libgomp: Thread creation failed: Resource temporarily unavailable". > This then results in a red Hudson and I think it exits the testrunner > (are the tests not run as separate processes?). So what should I do? > Should I fork manually in my test and watch the exit status? > > The result is here: > https://sage.math.washington.edu:8091/hudson/view/cython-mark/job/cython-mark-tests-py27-c/72/console Yes, I think it's worth forking and checking the exit code in this case so we can continue on. That's not to say we shouldn't see if we can't solve the underlying flakiness. - Robert From robertwb at math.washington.edu Thu Jul 21 11:21:15 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 21 Jul 2011 02:21:15 -0700 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: On Thu, Jul 7, 2011 at 2:13 PM, Vitja Makarov wrote: > > > 2011/7/8 mark florisson >> >> On 7 July 2011 22:39, Vitja Makarov wrote: >> > >> > >> > 2011/7/8 mark florisson >> >> >> >> On 7 July 2011 22:15, Vitja Makarov wrote: >> >> > >> >> > >> >> > 2011/7/7 mark florisson >> >> >> >> >> >> On 6 July 2011 10:01, Vitja Makarov wrote: >> >> >> > 2011/7/6 Stefan Behnel : >> >> >> >> Vitja Makarov, 06.07.2011 09:05: >> >> >> >>> >> >> >> >>> 2011/7/6 Stefan Behnel: >> >> >> >>>> >> >> >> >>>> Stefan Behnel, 05.07.2011 10:04: >> >> >> >>>>> >> >> >> >>>>> Vitja Makarov, 05.07.2011 09:17: >> >> >> >>>>>> >> >> >> >>>>>> 2011/7/5 Stefan Behnel: >> >> >> >>>>>>> >> >> >> >>>>>>> Vitja Makarov, 05.07.2011 08:21: >> >> >> >>>>>>>> >> >> >> >>>>>>>> I was thinking about implementing new super() with no >> >> >> >>>>>>>> arguments. >> >> >> >>>>>>> >> >> >> >>>>>>> http://trac.cython.org/cython_trac/ticket/696 >> >> >> >>>>>>> >> >> >> >>>>>>>> The problem is where to store __class__, I see two options >> >> >> >>>>>>>> here: >> >> >> >>>>>>>> >> >> >> >>>>>>>> 1. Add func_class member to CyFunction, this way __class__ >> >> >> >>>>>>>> will >> >> >> >>>>>>>> be >> >> >> >>>>>>>> private and not visible for inner functions: >> >> >> >>>>>>>> 2. Put it into closure >> >> >> >>>>>>> >> >> >> >>>>>>> The second option has the advantage of requiring the field >> >> >> >>>>>>> only >> >> >> >>>>>>> when >> >> >> >>>>>>> super() >> >> >> >>>>>>> is used, whereas the first impacts all functions. >> >> >> >>>>>>> >> >> >> >>>>>>> I would expect that programs commonly have a lot more >> >> >> >>>>>>> functions >> >> >> >>>>>>> than >> >> >> >>>>>>> specifically methods that use a no-argument call to super(), >> >> >> >>>>>>> so >> >> >> >>>>>>> this >> >> >> >>>>>>> may >> >> >> >>>>>>> make a difference. >> >> >> >>>>>>> >> >> >> >>>>>> >> >> >> >>>>>> So, now classes are created the following way: >> >> >> >>>>>> >> >> >> >>>>>> class_dict = {} >> >> >> >>>>>> class_dict.foo = foo_func >> >> >> >>>>>> class = CreateClass(class_dict) >> >> >> >>>>>> >> >> >> >>>>>> So after class is created I should check its dict for >> >> >> >>>>>> CyFunction >> >> >> >>>>>> members (maybe only ones that actually require __class__) >> >> >> >>>>>> and set __class__: >> >> >> >>>>>> >> >> >> >>>>>> for value in class.__dict__.itervalues(): >> >> >> >>>>>> if isinstance(value, CyFunction) and value.func_class is >> >> >> >>>>>> WantClass: >> >> >> >>>>>> value.func_class = class >> >> >> >>>>>> >> >> >> >>>>>> Btw, first way requires cyfunction signature change, it would >> >> >> >>>>>> accept >> >> >> >>>>>> cyfunction object as first argument. >> >> >> >>>>> >> >> >> >>>>> We currently pass the binding (i.e. owning) object, right? >> >> >> >>>> >> >> >> >>>> So, how would this work for methods? We need to pass the 'self' >> >> >> >>>> object >> >> >> >>>> there, which the CyFunction doesn't know. If anything, it only >> >> >> >>>> knows >> >> >> >>>> the >> >> >> >>>> class it was defined in, which doesn't help here. >> >> >> >>> >> >> >> >>> From PEP: "super() is equivalent to: >> >> >> >>> super(__class__,)" >> >> >> >> >> >> >> >> I wasn't speaking of super(). What I meant, was: how do we pass >> >> >> >> 'self' >> >> >> >> when >> >> >> >> we pass the CyFunction object as the first argument? >> >> >> >> >> >> >> > >> >> >> > >> >> >> > Oh, ok. Now we pass closure or nothing in self. So method's self >> >> >> > is >> >> >> > passed via tuple. >> >> >> > Instancemethod do this for us. Now CyFucntion uses >> >> >> > PyCFunction_Call >> >> >> > we >> >> >> > can override this and change signature of cyfunction to: >> >> >> > >> >> >> > PyObject func(CyFunction *func, PyObject *self, PyObject *args, >> >> >> > PyObject *kwargs); >> >> >> > >> >> >> > This way we should implement new instancemethod type. >> >> >> >> >> >> Would it be easier to make scope objects attributes of functions? >> >> >> Then >> >> >> you could still utilize PyCFunction_Call without needing to check >> >> >> the >> >> >> argument flags and such right? >> >> >> >> >> > >> >> > Sure, scope object is already functions attribute now it's stored >> >> > inside >> >> > self. >> >> > But, instancemethods desc_get builds new args tuple with self as >> >> > first >> >> > element. >> >> > We can implement cython version of instance method and change method >> >> > signature a little bit. >> >> >> >> Right, and you pass it in as the first argument to the C function. >> >> However, if you want to change the signature, you have to override >> >> PyCFunction_Call, which, if I'm not mistaken, means you will have to >> >> interpret the flags from the PyMethodDef and call the C function in >> >> the right way (e.g. with or without the args tuple and kwargs) and >> >> check if it's being called correctly. If you leave it as an attribute >> >> of the function, you can pass in the function and access the scope >> >> object from the function object in the closure C function. So I think >> >> changing the signature might be a bit harder? >> >> >> > >> > Yes, but how to handle instantmethods self arg? >> > We can't store it inside function object as it is different for each >> > instance. >> >> In descr_get you create and return a new CyFunction with a __self__ >> set (note, not m_self) and in __call__ you put __self__ in the args >> tuple and the function as m_self, and then call the CyFunction using >> PyCFunction_Call. But copying and modifying PyCFunction_Call works >> just as well I suppose :) > > Yes, that would work) But I think CyFunction is too heavy for methods. > I've just realized that descr_get is called each time you access method, > each time you call it and so on. True, but this is also the case for CPython methods and method descriptors. There's also an inefficiency in calling bound methods in that a new tuple needs to be created with the bound parameter as the first component. Perhaps we could do away with that as well with specialized unpacking code. It would simultaneously be a step forward and backwards in terms of compatibility, but we could do away with ever generating PyCFunctions, methods, and method descriptors. Perhaps a CEP is in order to nail down the behavior of the CyFunction object. - Robert From L.J.Buitinck at uva.nl Thu Jul 21 11:28:23 2011 From: L.J.Buitinck at uva.nl (Lars Buitinck) Date: Thu, 21 Jul 2011 11:28:23 +0200 Subject: [Cython] Strange cimport behaviour In-Reply-To: References: Message-ID: 2011/7/21 Robert Bradshaw : > On Wed, Jul 20, 2011 at 10:30 PM, Robert Bradshaw > wrote: >> On Wed, Jul 20, 2011 at 9:40 PM, Vitja Makarov wrote: >>> 2011/7/21 Robert Bradshaw : >>>> I'm not sure what we can do here--we make sure to emit the #include >>>> statements in the same order as they are encountered in the Cython >>>> sources because C is sensitive to this kind of thing, but we can't >>>> really "fix" C. I suppose we could cimport posix.fcntl from within >>>> posix.unistd to force an ordering. > > https://github.com/cython/cython/commit/55d5e576935d83c6bdadc593e36793aecffe0bae I'm not familiar enough with the Cython internals to understand this change; are #includes now generated in the C code in the order of the corresponding cimports? -- Lars Buitinck Scientific programmer, ILPS University of Amsterdam From robertwb at math.washington.edu Thu Jul 21 12:16:34 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 21 Jul 2011 03:16:34 -0700 Subject: [Cython] Strange cimport behaviour In-Reply-To: References: Message-ID: On Thu, Jul 21, 2011 at 2:28 AM, Lars Buitinck wrote: > 2011/7/21 Robert Bradshaw : >> On Wed, Jul 20, 2011 at 10:30 PM, Robert Bradshaw >> wrote: >>> On Wed, Jul 20, 2011 at 9:40 PM, Vitja Makarov wrote: >>>> 2011/7/21 Robert Bradshaw : >>>>> I'm not sure what we can do here--we make sure to emit the #include >>>>> statements in the same order as they are encountered in the Cython >>>>> sources because C is sensitive to this kind of thing, but we can't >>>>> really "fix" C. I suppose we could cimport posix.fcntl from within >>>>> posix.unistd to force an ordering. >> >> https://github.com/cython/cython/commit/55d5e576935d83c6bdadc593e36793aecffe0bae > > I'm not familiar enough with the Cython internals to understand this > change; are #includes now generated in the C code in the order of the > corresponding cimports? Yes. Essentially, imagine every time you see a "cdef extern from 'header.h'" you add 'header.h' to your ordered list of #includes (if it's not already there), and when you encounter a cimport, you "step into" it and gather all the #includes it has before moving on. - Robert From stefan_ml at behnel.de Thu Jul 21 14:39:21 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 21 Jul 2011 14:39:21 +0200 Subject: [Cython] Strange cimport behaviour In-Reply-To: References: Message-ID: <4E281DF9.1070503@behnel.de> Lars Buitinck, 21.07.2011 11:28: > 2011/7/21 Robert Bradshaw: >> https://github.com/cython/cython/commit/55d5e576935d83c6bdadc593e36793aecffe0bae > > I'm not familiar enough with the Cython internals to understand this > change; are #includes now generated in the C code in the order of the > corresponding cimports? That's been the case for several releases already. The change above only fixes a glitch where the #include was not generated at all. Stefan From stefan_ml at behnel.de Thu Jul 21 18:29:40 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 21 Jul 2011 18:29:40 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> Message-ID: <4E2853F4.8010703@behnel.de> Robert Bradshaw, 21.07.2011 11:21: > On Thu, Jul 7, 2011 at 2:13 PM, Vitja Makarov wrote: >> 2011/7/8 mark florisson >>> In descr_get you create and return a new CyFunction with a __self__ >>> set (note, not m_self) and in __call__ you put __self__ in the args >>> tuple and the function as m_self, and then call the CyFunction using >>> PyCFunction_Call. But copying and modifying PyCFunction_Call works >>> just as well I suppose :) >> >> Yes, that would work) But I think CyFunction is too heavy for methods. >> I've just realized that descr_get is called each time you access method, >> each time you call it and so on. > > True, but this is also the case for CPython methods and method > descriptors. There's also an inefficiency in calling bound methods in > that a new tuple needs to be created with the bound parameter as the > first component. Perhaps we could do away with that as well with > specialized unpacking code. It would simultaneously be a step forward > and backwards in terms of compatibility, but we could do away with > ever generating PyCFunctions, methods, and method descriptors. That sounds like a really cool optimisation path. We could get rid of one indirection by replacing tp_call (originally PyCFunction_Call) directly by the real function entry point, i.e. the Python function wrapper (assuming the DefNode refactoring has taken place by then). It would also (somewhat) simplify the argument unpacking code, as that would no longer have to care about the METH_O and METH_NOARGS special cases. They'd be handled naturally inside of the Python call wrapper. > Perhaps > a CEP is in order to nail down the behavior of the CyFunction object. +1, totally helps in figuring out the details. Stefan From vitja.makarov at gmail.com Thu Jul 21 20:57:55 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 21 Jul 2011 22:57:55 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: <4E2853F4.8010703@behnel.de> References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> <4E2853F4.8010703@behnel.de> Message-ID: 2011/7/21 Stefan Behnel : > Robert Bradshaw, 21.07.2011 11:21: >> >> On Thu, Jul 7, 2011 at 2:13 PM, Vitja Makarov wrote: >>> >>> 2011/7/8 mark florisson >>>> >>>> In descr_get you create and return a new CyFunction with a __self__ >>>> set (note, not m_self) and in __call__ you put __self__ in the args >>>> tuple and the function as m_self, and then call the CyFunction using >>>> PyCFunction_Call. But copying and modifying PyCFunction_Call works >>>> just as well I suppose :) >>> >>> Yes, that would work) But I think CyFunction is too heavy for methods. >>> I've just realized that descr_get is called each time you access method, >>> each time you call it and so on. >> >> True, but this is also the case for CPython methods and method >> descriptors. There's also an inefficiency in calling bound methods in >> that a new tuple needs to be created with the bound parameter as the >> first component. Perhaps we could do away with that as well with >> specialized unpacking code. It would simultaneously be a step forward >> and backwards in terms of compatibility, but we could do away with >> ever generating PyCFunctions, methods, and method descriptors. > Yeah, I was thinking about that too. > That sounds like a really cool optimisation path. We could get rid of one > indirection by replacing tp_call (originally PyCFunction_Call) directly by > the real function entry point, i.e. the Python function wrapper (assuming > the DefNode refactoring has taken place by then). > +1 I think DefNode refactoring should be done first (Did we froget about that?) > It would also (somewhat) simplify the argument unpacking code, as that would > no longer have to care about the METH_O and METH_NOARGS special cases. > They'd be handled naturally inside of the Python call wrapper. > > >> Perhaps >> a CEP is in order to nail down the behavior of the CyFunction object. > > +1, totally helps in figuring out the details. CEP for refactored DefNode would be a good idea too. As I rember the idea is to turn def node into cdef node with wrapper call function just like cpdef buf without C-level access. So here we have to decide how to how to match python function's signature and cdef one. -- vitja. From robertwb at math.washington.edu Fri Jul 22 01:00:39 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 21 Jul 2011 16:00:39 -0700 Subject: [Cython] Cython 0.15 release In-Reply-To: References: Message-ID: Stefan, Are the lxml failures at https://sage.math.washington.edu:8091/hudson/job/cython-devel-lxml-trunk-py27/ expected? - Robert From robertwb at math.washington.edu Fri Jul 22 01:14:06 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 21 Jul 2011 16:14:06 -0700 Subject: [Cython] Cython 0.15 release candidate Message-ID: Cython has seen an enormous amount of development since 0.14.1. If you are not already using the latest version from the development repository, we encourage you to try out the release candidate: http://cython.org/release/Cython-0.15rc0.tar.gz - Robert From robertwb at math.washington.edu Fri Jul 22 01:33:29 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 21 Jul 2011 16:33:29 -0700 Subject: [Cython] [cython-users] Cython 0.15 release candidate In-Reply-To: References: Message-ID: On Thu, Jul 21, 2011 at 4:24 PM, ??? wrote: > what is the new feature of cython 0.15? The release notes are still pretty bare-bones, but see: http://wiki.cython.org/ReleaseNotes-0.15 > 2011/7/22 Robert Bradshaw >> >> Cython has seen an enormous amount of development since 0.14.1. If you >> are not already using the latest version from the development >> repository, we encourage you to try out the release candidate: >> http://cython.org/release/Cython-0.15rc0.tar.gz >> >> - Robert > > From cgohlke at uci.edu Fri Jul 22 04:43:52 2011 From: cgohlke at uci.edu (Christoph Gohlke) Date: Thu, 21 Jul 2011 19:43:52 -0700 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: References: Message-ID: <4E28E3E8.6000000@uci.edu> On 7/21/2011 4:14 PM, Robert Bradshaw wrote: > Cython has seen an enormous amount of development since 0.14.1. If you > are not already using the latest version from the development > repository, we encourage you to try out the release candidate: > http://cython.org/release/Cython-0.15rc0.tar.gz > > - Robert Hi, I get a test error on Windows with msvc9 due to the "with gil" statement producing invalid C code (at least for msvc9). Below is the offending code in with_gil.c. Moving the declaration of the __pyx_gilstate_save variable a couple of lines up fixes this. A patch is attached. Christoph /* "with_gil.pyx":347 * raise Exception("This will be overridden") * finally: * with gil: # <<<<<<<<<<<<<< * raise Exception("Override exception!") * */ /*finally:*/ { int __pyx_why; PyObject *__pyx_exc_type, *__pyx_exc_value, *__pyx_exc_tb; int __pyx_exc_lineno; __pyx_exc_type = 0; __pyx_exc_value = 0; __pyx_exc_tb = 0; __pyx_exc_lineno = 0; #ifdef WITH_THREAD PyGILState_STATE __pyx_gilstate_save; #endif __pyx_why = 0; goto __pyx_L10; -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: with_gil.diff URL: From dalcinl at gmail.com Fri Jul 22 04:52:26 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 21 Jul 2011 23:52:26 -0300 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: <4E28E3E8.6000000@uci.edu> References: <4E28E3E8.6000000@uci.edu> Message-ID: On 21 July 2011 23:43, Christoph Gohlke wrote: > > > On 7/21/2011 4:14 PM, Robert Bradshaw wrote: >> >> Cython has seen an enormous amount of development since 0.14.1. If you >> are not already using the latest version from the development >> repository, we encourage you to try out the release candidate: >> http://cython.org/release/Cython-0.15rc0.tar.gz >> >> - Robert > > Hi, > > I get a test error on Windows with msvc9 due to the "with gil" statement > producing invalid C code (at least for msvc9). Below is the offending code > in with_gil.c. Moving the declaration of the __pyx_gilstate_save variable a > couple of lines up fixes this. A patch is attached. > > Christoph > > > ? ? ? ?/* "with_gil.pyx":347 > ?* ? ? ? ? ? ? ? ? raise Exception("This will be overridden") > ?* ? ? ? ? finally: > ?* ? ? ? ? ? ? with gil: ? ? ? ? ? ? # <<<<<<<<<<<<<< > ?* ? ? ? ? ? ? ? ? raise Exception("Override exception!") > ?* > ?*/ > ? ? ? ?/*finally:*/ { > ? ? ? ? ?int __pyx_why; > ? ? ? ? ?PyObject *__pyx_exc_type, *__pyx_exc_value, *__pyx_exc_tb; > ? ? ? ? ?int __pyx_exc_lineno; > ? ? ? ? ?__pyx_exc_type = 0; __pyx_exc_value = 0; __pyx_exc_tb = 0; > __pyx_exc_lineno = 0; > ? ? ? ? ?#ifdef WITH_THREAD > ? ? ? ? ?PyGILState_STATE __pyx_gilstate_save; > ? ? ? ? ?#endif > ? ? ? ? ?__pyx_why = 0; goto __pyx_L10; > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > Your patch is OK. However now I'm wondering why not to initialize the exc variables to 0 directly on declaration, like this: PyObject *__pyx_exc_type = 0, *__pyx_exc_value =0, *__pyx_exc_tb = 0; (anyway, I still prefer declaring the gilstate variable first) -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From stefan_ml at behnel.de Fri Jul 22 06:25:06 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 06:25:06 +0200 Subject: [Cython] Cython 0.15 release In-Reply-To: References: Message-ID: <4E28FBA2.9020005@behnel.de> Robert Bradshaw, 22.07.2011 01:00: > Are the lxml failures at > https://sage.math.washington.edu:8091/hudson/job/cython-devel-lxml-trunk-py27/ > expected? Yes, the "test_xmlschema_import_file" test keeps crashing with a high probability. Some kind of threading related bug, no idea how to reproduce it in a debuggable way... Just ignore those two jobs for the time being. I keep looking at the tests from time to time, because it's not *always* that bug that kills them. Stefan From stefan_ml at behnel.de Fri Jul 22 07:58:26 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 07:58:26 +0200 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> <4E2853F4.8010703@behnel.de> Message-ID: <4E291182.4020704@behnel.de> Vitja Makarov, 21.07.2011 20:57: > 2011/7/21 Stefan Behnel: >> Robert Bradshaw, 21.07.2011 11:21: >>> >>> On Thu, Jul 7, 2011 at 2:13 PM, Vitja Makarov wrote: >>>> >>>> 2011/7/8 mark florisson >>>>> >>>>> In descr_get you create and return a new CyFunction with a __self__ >>>>> set (note, not m_self) and in __call__ you put __self__ in the args >>>>> tuple and the function as m_self, and then call the CyFunction using >>>>> PyCFunction_Call. But copying and modifying PyCFunction_Call works >>>>> just as well I suppose :) >>>> >>>> Yes, that would work) But I think CyFunction is too heavy for methods. >>>> I've just realized that descr_get is called each time you access method, >>>> each time you call it and so on. >>> >>> True, but this is also the case for CPython methods and method >>> descriptors. There's also an inefficiency in calling bound methods in >>> that a new tuple needs to be created with the bound parameter as the >>> first component. Perhaps we could do away with that as well with >>> specialized unpacking code. It would simultaneously be a step forward >>> and backwards in terms of compatibility, but we could do away with >>> ever generating PyCFunctions, methods, and method descriptors. >> > > Yeah, I was thinking about that too. > >> That sounds like a really cool optimisation path. We could get rid of one >> indirection by replacing tp_call (originally PyCFunction_Call) directly by >> the real function entry point, i.e. the Python function wrapper (assuming >> the DefNode refactoring has taken place by then). >> > > +1 I think DefNode refactoring should be done first (Did we froget about that?) > >> It would also (somewhat) simplify the argument unpacking code, as that would >> no longer have to care about the METH_O and METH_NOARGS special cases. >> They'd be handled naturally inside of the Python call wrapper. >> >> >>> Perhaps >>> a CEP is in order to nail down the behavior of the CyFunction object. >> >> +1, totally helps in figuring out the details. > > CEP for refactored DefNode would be a good idea too. Well, there's still the part I wrote for the generator CEP: http://wiki.cython.org/enhancements/generators#Pythonfunctionrefactoring > As I rember the idea is to turn def node into cdef node with wrapper > call function just like cpdef buf without C-level access. > So here we have to decide how to how to match python function's > signature and cdef one. I think that's mostly trivial - the argument unpacking code already does the mapping for us. Stefan From stefan_ml at behnel.de Fri Jul 22 08:02:06 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 08:02:06 +0200 Subject: [Cython] hg-git shows 25 heads? In-Reply-To: References: <4E15B484.9040501@behnel.de> <4E15F696.9080308@astro.uio.no> Message-ID: <4E29125E.5000904@behnel.de> Pauli Virtanen, 07.07.2011 21:06: > On Thu, 07 Jul 2011 20:10:30 +0200, Dag Sverre Seljebotn wrote: > [clip] >> 5 minutes of Googling didn't turn up anything... Do you think it would >> be a bad idea to simply delete these branches, something like >> >> git push origin :refs/* > > That drops *all* branches -- not recommended :) > > As far as I see, > > git push origin :refs/pull/* > > should not break anything. However, depending on how the stuff is set > up on the Github side, either the merge button stops working (would be > an improvement to the state of matters, IMHO), or alternatively, > the branches come back immediately whenever someone views pull requests. > > So I don't think doing this would help much. Maybe best to ask the Github > guys what the heck is their creation doing here, and try to see if hg-git > can be made to work properly. I asked and they already fixed it. There isn't a release yet, but the latest hg-git sources work nicely. Stefan From vitja.makarov at gmail.com Fri Jul 22 08:13:07 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 22 Jul 2011 10:13:07 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: <4E291182.4020704@behnel.de> References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> <4E2853F4.8010703@behnel.de> <4E291182.4020704@behnel.de> Message-ID: 2011/7/22 Stefan Behnel : > Vitja Makarov, 21.07.2011 20:57: >> >> 2011/7/21 Stefan Behnel: >>> >>> Robert Bradshaw, 21.07.2011 11:21: >>>> >>>> On Thu, Jul 7, 2011 at 2:13 PM, Vitja Makarov wrote: >>>>> >>>>> 2011/7/8 mark florisson >>>>>> >>>>>> In descr_get you create and return a new CyFunction with a __self__ >>>>>> set (note, not m_self) and in __call__ you put __self__ in the args >>>>>> tuple and the function as m_self, and then call the CyFunction using >>>>>> PyCFunction_Call. But copying and modifying PyCFunction_Call works >>>>>> just as well I suppose :) >>>>> >>>>> Yes, that would work) But I think CyFunction is too heavy for methods. >>>>> I've just realized that descr_get is called each time you access >>>>> method, >>>>> each time you call it and so on. >>>> >>>> True, but this is also the case for CPython methods and method >>>> descriptors. There's also an inefficiency in calling bound methods in >>>> that a new tuple needs to be created with the bound parameter as the >>>> first component. Perhaps we could do away with that as well with >>>> specialized unpacking code. It would simultaneously be a step forward >>>> and backwards in terms of compatibility, but we could do away with >>>> ever generating PyCFunctions, methods, and method descriptors. >>> >> >> Yeah, I was thinking about that too. >> >>> That sounds like a really cool optimisation path. We could get rid of one >>> indirection by replacing tp_call (originally PyCFunction_Call) directly >>> by >>> the real function entry point, i.e. the Python function wrapper (assuming >>> the DefNode refactoring has taken place by then). >>> >> >> +1 I think DefNode refactoring should be done first (Did we froget about >> that?) >> >>> It would also (somewhat) simplify the argument unpacking code, as that >>> would >>> no longer have to care about the METH_O and METH_NOARGS special cases. >>> They'd be handled naturally inside of the Python call wrapper. >>> >>> >>>> Perhaps >>>> a CEP is in order to nail down the behavior of the CyFunction object. >>> >>> +1, totally helps in figuring out the details. >> >> CEP for refactored DefNode would be a good idea too. > > Well, there's still the part I wrote for the generator CEP: > > http://wiki.cython.org/enhancements/generators#Pythonfunctionrefactoring > > >> As I rember the idea is to turn def node into cdef node with wrapper >> call function just like cpdef buf without C-level access. >> So here we have to decide how to how to match python function's >> signature and cdef one. > > I think that's mostly trivial - the argument unpacking code already does the > mapping for us. > So the wrapper could pass all the args (including defaults) to underlaying C-function. That solves the problem. -- vitja. From vitja.makarov at gmail.com Fri Jul 22 09:00:01 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 22 Jul 2011 11:00:01 +0400 Subject: [Cython] PEP 3135 -- New Super In-Reply-To: <4E2853F4.8010703@behnel.de> References: <4E12B6A1.3090805@behnel.de> <4E12C57E.1060307@behnel.de> <4E13AE10.3040104@behnel.de> <4E140D1F.5000809@behnel.de> <4E2853F4.8010703@behnel.de> Message-ID: 2011/7/21 Stefan Behnel : > Robert Bradshaw, 21.07.2011 11:21: >> >> On Thu, Jul 7, 2011 at 2:13 PM, Vitja Makarov wrote: >>> >>> 2011/7/8 mark florisson >>>> >>>> In descr_get you create and return a new CyFunction with a __self__ >>>> set (note, not m_self) and in __call__ you put __self__ in the args >>>> tuple and the function as m_self, and then call the CyFunction using >>>> PyCFunction_Call. But copying and modifying PyCFunction_Call works >>>> just as well I suppose :) >>> >>> Yes, that would work) But I think CyFunction is too heavy for methods. >>> I've just realized that descr_get is called each time you access method, >>> each time you call it and so on. >> >> True, but this is also the case for CPython methods and method >> descriptors. There's also an inefficiency in calling bound methods in >> that a new tuple needs to be created with the bound parameter as the >> first component. Perhaps we could do away with that as well with >> specialized unpacking code. It would simultaneously be a step forward >> and backwards in terms of compatibility, but we could do away with >> ever generating PyCFunctions, methods, and method descriptors. > > That sounds like a really cool optimisation path. We could get rid of one > indirection by replacing tp_call (originally PyCFunction_Call) directly by > the real function entry point, i.e. the Python function wrapper (assuming > the DefNode refactoring has taken place by then). > To replace tp_call with direct entry we have to create new type for each function, it's better to make tp_call call indirectly our wrapper. -- vitja. From markflorisson88 at gmail.com Fri Jul 22 10:54:33 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 22 Jul 2011 10:54:33 +0200 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: <4E28E3E8.6000000@uci.edu> References: <4E28E3E8.6000000@uci.edu> Message-ID: On 22 July 2011 04:43, Christoph Gohlke wrote: > > > On 7/21/2011 4:14 PM, Robert Bradshaw wrote: >> >> Cython has seen an enormous amount of development since 0.14.1. If you >> are not already using the latest version from the development >> repository, we encourage you to try out the release candidate: >> http://cython.org/release/Cython-0.15rc0.tar.gz >> >> - Robert > > Hi, > > I get a test error on Windows with msvc9 due to the "with gil" statement > producing invalid C code (at least for msvc9). Below is the offending code > in with_gil.c. Moving the declaration of the __pyx_gilstate_save variable a > couple of lines up fixes this. A patch is attached. > > Christoph > > > ? ? ? ?/* "with_gil.pyx":347 > ?* ? ? ? ? ? ? ? ? raise Exception("This will be overridden") > ?* ? ? ? ? finally: > ?* ? ? ? ? ? ? with gil: ? ? ? ? ? ? # <<<<<<<<<<<<<< > ?* ? ? ? ? ? ? ? ? raise Exception("Override exception!") > ?* > ?*/ > ? ? ? ?/*finally:*/ { > ? ? ? ? ?int __pyx_why; > ? ? ? ? ?PyObject *__pyx_exc_type, *__pyx_exc_value, *__pyx_exc_tb; > ? ? ? ? ?int __pyx_exc_lineno; > ? ? ? ? ?__pyx_exc_type = 0; __pyx_exc_value = 0; __pyx_exc_tb = 0; > __pyx_exc_lineno = 0; > ? ? ? ? ?#ifdef WITH_THREAD > ? ? ? ? ?PyGILState_STATE __pyx_gilstate_save; > ? ? ? ? ?#endif > ? ? ? ? ?__pyx_why = 0; goto __pyx_L10; > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > Ah, thanks a bunch Christoph! I'll apply it in a bit (if Lisandro hasn't already). From markflorisson88 at gmail.com Fri Jul 22 10:58:23 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 22 Jul 2011 10:58:23 +0200 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: References: <4E28E3E8.6000000@uci.edu> Message-ID: On 22 July 2011 04:52, Lisandro Dalcin wrote: > On 21 July 2011 23:43, Christoph Gohlke wrote: >> >> >> On 7/21/2011 4:14 PM, Robert Bradshaw wrote: >>> >>> Cython has seen an enormous amount of development since 0.14.1. If you >>> are not already using the latest version from the development >>> repository, we encourage you to try out the release candidate: >>> http://cython.org/release/Cython-0.15rc0.tar.gz >>> >>> - Robert >> >> Hi, >> >> I get a test error on Windows with msvc9 due to the "with gil" statement >> producing invalid C code (at least for msvc9). Below is the offending code >> in with_gil.c. Moving the declaration of the __pyx_gilstate_save variable a >> couple of lines up fixes this. A patch is attached. >> >> Christoph >> >> >> ? ? ? ?/* "with_gil.pyx":347 >> ?* ? ? ? ? ? ? ? ? raise Exception("This will be overridden") >> ?* ? ? ? ? finally: >> ?* ? ? ? ? ? ? with gil: ? ? ? ? ? ? # <<<<<<<<<<<<<< >> ?* ? ? ? ? ? ? ? ? raise Exception("Override exception!") >> ?* >> ?*/ >> ? ? ? ?/*finally:*/ { >> ? ? ? ? ?int __pyx_why; >> ? ? ? ? ?PyObject *__pyx_exc_type, *__pyx_exc_value, *__pyx_exc_tb; >> ? ? ? ? ?int __pyx_exc_lineno; >> ? ? ? ? ?__pyx_exc_type = 0; __pyx_exc_value = 0; __pyx_exc_tb = 0; >> __pyx_exc_lineno = 0; >> ? ? ? ? ?#ifdef WITH_THREAD >> ? ? ? ? ?PyGILState_STATE __pyx_gilstate_save; >> ? ? ? ? ?#endif >> ? ? ? ? ?__pyx_why = 0; goto __pyx_L10; >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> >> > > Your patch is OK. However now I'm wondering why not to initialize the > exc variables to 0 directly on declaration, like this: > > PyObject *__pyx_exc_type = 0, *__pyx_exc_value =0, *__pyx_exc_tb = 0; > > (anyway, I still prefer declaring the gilstate variable first) I think the reason for that was to avoid compiler warnings about unused variables in certain cases. Amazing by the way, how it still doesn't support C99. > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169 > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From markflorisson88 at gmail.com Fri Jul 22 11:46:55 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 22 Jul 2011 11:46:55 +0200 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: <4E28E3E8.6000000@uci.edu> References: <4E28E3E8.6000000@uci.edu> Message-ID: On 22 July 2011 04:43, Christoph Gohlke wrote: > > > On 7/21/2011 4:14 PM, Robert Bradshaw wrote: >> >> Cython has seen an enormous amount of development since 0.14.1. If you >> are not already using the latest version from the development >> repository, we encourage you to try out the release candidate: >> http://cython.org/release/Cython-0.15rc0.tar.gz >> >> - Robert > > Hi, > > I get a test error on Windows with msvc9 due to the "with gil" statement > producing invalid C code (at least for msvc9). Below is the offending code > in with_gil.c. Moving the declaration of the __pyx_gilstate_save variable a > couple of lines up fixes this. A patch is attached. > > Christoph > > > ? ? ? ?/* "with_gil.pyx":347 > ?* ? ? ? ? ? ? ? ? raise Exception("This will be overridden") > ?* ? ? ? ? finally: > ?* ? ? ? ? ? ? with gil: ? ? ? ? ? ? # <<<<<<<<<<<<<< > ?* ? ? ? ? ? ? ? ? raise Exception("Override exception!") > ?* > ?*/ > ? ? ? ?/*finally:*/ { > ? ? ? ? ?int __pyx_why; > ? ? ? ? ?PyObject *__pyx_exc_type, *__pyx_exc_value, *__pyx_exc_tb; > ? ? ? ? ?int __pyx_exc_lineno; > ? ? ? ? ?__pyx_exc_type = 0; __pyx_exc_value = 0; __pyx_exc_tb = 0; > __pyx_exc_lineno = 0; > ? ? ? ? ?#ifdef WITH_THREAD > ? ? ? ? ?PyGILState_STATE __pyx_gilstate_save; > ? ? ? ? ?#endif > ? ? ? ? ?__pyx_why = 0; goto __pyx_L10; > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > Christoph, do I have your permission to commit the patch with your name and email address? From markflorisson88 at gmail.com Fri Jul 22 12:12:17 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 22 Jul 2011 12:12:17 +0200 Subject: [Cython] Utility Codes and templates Message-ID: For my work on the _memview branch (and also on fused types) I noticed that UtilityCodes started weighing heavily on me in their current form, so I wrote a little loader in the _memview branch: https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 The idea is simple: you put your utility codes in Cython/Utility in .pyx, .c, .h files etc, and then load them. It works for both prototypes and implementations, for UtilityCode and CythonUtilityCode: myutility.c // UtilityProto: MyUtility header code here // UtilityCode: MyUtility implementation code here You can add as many other utilities as you like to the same file. You can then load it using UtilityCode.load_utility_from_file("myutility.c", "MyUtility") Of course you can pass in any other arguments, like proto_block, name, etc. You can additionally pass it a dict for formatting (for both the prototypes and the implementation). It will return a UtilityCode instance ready for use. You can also simply retrieve a utility code as a string, where it returns (proto, implementation). As debated before, an actual template library would be really convenient. Dag and I had a discussion about it and he suggested Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3, and is pure-python. It supports all the good things like iteration, template inheritance, etc. Now I'm not sure whether it supports python 2.3 as it doesn't compile on my system, but it does support 2.4 (confirmation for 2.3 would be appreciated). On a side note, I'd be perfectly happy to drop support for 2.3, it's kind of a chore. The documentation for Tempita can be found here: http://pythonpaste.org/tempita/ That way we might rid ourselves of a lot of code.putln() and move those to template utilities instead (or at least prevent writing more of those). What do you guys think? BTW, I will refrain from moving any utility codes other than the ones in MemoryView.py, CythonScope.py and Buffer.py, to avoid any possible future conflicts when merging into mainline. From d.s.seljebotn at astro.uio.no Fri Jul 22 12:43:54 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 22 Jul 2011 12:43:54 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: Message-ID: <4E29546A.7080007@astro.uio.no> On 07/22/2011 12:12 PM, mark florisson wrote: > For my work on the _memview branch (and also on fused types) I noticed > that UtilityCodes started weighing heavily on me in their current > form, so I wrote a little loader in the _memview branch: > > https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 > > The idea is simple: you put your utility codes in Cython/Utility in > .pyx, .c, .h files etc, and then load them. It works for both > prototypes and implementations, for UtilityCode and CythonUtilityCode: > > myutility.c > > // UtilityProto: MyUtility > header code here > > // UtilityCode: MyUtility > implementation code here > > You can add as many other utilities as you like to the same file. You > can then load it using > > UtilityCode.load_utility_from_file("myutility.c", "MyUtility") > > Of course you can pass in any other arguments, like proto_block, name, > etc. You can additionally pass it a dict for formatting (for both the > prototypes and the implementation). It will return a UtilityCode > instance ready for use. > You can also simply retrieve a utility code as a string, where it > returns (proto, implementation). > > As debated before, an actual template library would be really > convenient. Dag and I had a discussion about it and he suggested > Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3, > and is pure-python. It supports all the good things like iteration, > template inheritance, etc. Now I'm not sure whether it supports python > 2.3 as it doesn't compile on my system, but it does support 2.4 > (confirmation for 2.3 would be appreciated). On a side note, I'd be > perfectly happy to drop support for 2.3, it's kind of a chore. > The documentation for Tempita can be found here: http://pythonpaste.org/tempita/ Also important: (master) ~/code/tempita $ sloccount tempita/ SLOC Directory SLOC-by-Language (Sorted) 1050 tempita python=1050 That is, 1050 source code lines. So it can easily (and should probably) be bundled with Cython, in the same way Plex is. Dag Sverre From stefan_ml at behnel.de Fri Jul 22 13:10:22 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 13:10:22 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: Message-ID: <4E295A9E.6090800@behnel.de> mark florisson, 22.07.2011 12:12: > For my work on the _memview branch (and also on fused types) I noticed > that UtilityCodes started weighing heavily on me in their current > form, so I wrote a little loader in the _memview branch: > > https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 > > The idea is simple: you put your utility codes in Cython/Utility in > .pyx, .c, .h files etc, and then load them. It works for both > prototypes and implementations, for UtilityCode and CythonUtilityCode: > > myutility.c > > // UtilityProto: MyUtility > header code here > > // UtilityCode: MyUtility > implementation code here > > You can add as many other utilities as you like to the same file. You > can then load it using > > UtilityCode.load_utility_from_file("myutility.c", "MyUtility") Why not have exactly one per file? They can't (or at least shouldn't) be interdependent anyway, since they're always loaded and injected separately. Having one per file makes it easy to take the file name and grep for it. > Of course you can pass in any other arguments, like proto_block, name, > etc. You can additionally pass it a dict for formatting (for both the > prototypes and the implementation) Dict? Why not keyword arguments? I'd prefer an interface like this: UtilityCode.load_from_file("myutility", some_format_arg="somename", some_repeat_arg = 2) That would automatically look up the corresponding files Cython/UtilityCode/myutility.{pyx,c,cpp,h} and take whichever it finds (first), then run the template engine on it. I don't think we need to support separate arguments for prototype and code section, they can just use different names. Keep it simple. > It will return a UtilityCode instance ready for use. > You can also simply retrieve a utility code as a string, where it > returns (proto, implementation). > > As debated before, an actual template library would be really > convenient. Dag and I had a discussion about it and he suggested > Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3, > and is pure-python. It supports all the good things like iteration, > template inheritance, etc. Now I'm not sure whether it supports python > 2.3 as it doesn't compile on my system, but it does support 2.4 > (confirmation for 2.3 would be appreciated). On a side note, I'd be > perfectly happy to drop support for 2.3, it's kind of a chore. > The documentation for Tempita can be found here: http://pythonpaste.org/tempita/ > > That way we might rid ourselves of a lot of code.putln() and move > those to template utilities instead (or at least prevent writing more > of those). What do you guys think? I'm fine with using a template engine for the more involved cases (which are rare enough). However, I'd prefer not adding a new dependency, but just shipping a tiny single-module engine with Cython, e.g. Templite or pyratemp (just found them, never used them). http://www.joonis.de/content/TemplitePythonTemplatingEngine http://www.simple-is-better.org/template/pyratemp.html There's a whole bunch of engines listed here: http://wiki.python.org/moin/Templating Stefan From stefan_ml at behnel.de Fri Jul 22 13:26:31 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 13:26:31 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: Message-ID: <4E295E67.5020705@behnel.de> mark florisson, 22.07.2011 12:12: > For my work on the _memview branch (and also on fused types) I noticed > that UtilityCodes started weighing heavily on me in their current > form, so I wrote a little loader in the _memview branch: > > https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 > > The idea is simple: you put your utility codes in Cython/Utility in > .pyx, .c, .h files etc, and then load them. It works for both > prototypes and implementations, for UtilityCode and CythonUtilityCode: > > myutility.c > > // UtilityProto: MyUtility > header code here > > // UtilityCode: MyUtility > implementation code here Actually, if we use a template engine anyway, why not simplify this further? We could just evaluate the template twice, once with target="proto" and once with target="impl", and use the conditionals of the template language to output the right code section depending on that parameter. Stefan From d.s.seljebotn at astro.uio.no Fri Jul 22 13:31:48 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 22 Jul 2011 13:31:48 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E295A9E.6090800@behnel.de> References: <4E295A9E.6090800@behnel.de> Message-ID: <4E295FA4.1020300@astro.uio.no> On 07/22/2011 01:10 PM, Stefan Behnel wrote: > mark florisson, 22.07.2011 12:12: >> For my work on the _memview branch (and also on fused types) I noticed >> that UtilityCodes started weighing heavily on me in their current >> form, so I wrote a little loader in the _memview branch: >> >> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >> >> >> The idea is simple: you put your utility codes in Cython/Utility in >> .pyx, .c, .h files etc, and then load them. It works for both >> prototypes and implementations, for UtilityCode and CythonUtilityCode: >> >> myutility.c >> >> // UtilityProto: MyUtility >> header code here >> >> // UtilityCode: MyUtility >> implementation code here >> >> You can add as many other utilities as you like to the same file. You >> can then load it using >> >> UtilityCode.load_utility_from_file("myutility.c", "MyUtility") > > Why not have exactly one per file? They can't (or at least shouldn't) be > interdependent anyway, since they're always loaded and injected > separately. Having one per file makes it easy to take the file name and > grep for it. > > >> Of course you can pass in any other arguments, like proto_block, name, >> etc. You can additionally pass it a dict for formatting (for both the >> prototypes and the implementation) > > Dict? Why not keyword arguments? > > I'd prefer an interface like this: > > UtilityCode.load_from_file("myutility", > some_format_arg="somename", some_repeat_arg = 2) > > That would automatically look up the corresponding files > Cython/UtilityCode/myutility.{pyx,c,cpp,h} > and take whichever it finds (first), then run the template engine on it. > > I don't think we need to support separate arguments for prototype and > code section, they can just use different names. Keep it simple. > > >> It will return a UtilityCode instance ready for use. >> You can also simply retrieve a utility code as a string, where it >> returns (proto, implementation). >> >> As debated before, an actual template library would be really >> convenient. Dag and I had a discussion about it and he suggested >> Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3, >> and is pure-python. It supports all the good things like iteration, >> template inheritance, etc. Now I'm not sure whether it supports python >> 2.3 as it doesn't compile on my system, but it does support 2.4 >> (confirmation for 2.3 would be appreciated). On a side note, I'd be >> perfectly happy to drop support for 2.3, it's kind of a chore. >> The documentation for Tempita can be found here: >> http://pythonpaste.org/tempita/ >> >> That way we might rid ourselves of a lot of code.putln() and move >> those to template utilities instead (or at least prevent writing more >> of those). What do you guys think? > > I'm fine with using a template engine for the more involved cases (which > are rare enough). However, I'd prefer not adding a new dependency, but > just shipping a tiny single-module engine with Cython, e.g. Templite or > pyratemp (just found them, never used them). Tempita *IS* intended to be a "tiny single-module engine". Well, it is spread across several source files, but it wouldn't be intrusive to merge them into one file (if that's the qualififying criterion). I submit that: a) For the templating we need, there's probably a dozen out there that are "good enough" and small and simple. b) So, there's not any point in spending valuable time finding the "optimal solution". Good enough is good enough. c) Still, we shouldn't pick one at random, because there might be a subtlety that makes it impracticaly in real world usage. Thus, I think there's a strong case for going by personal recommendation by somebody who's actually used something, and just go with that. If anybody else has used something else and is happy with it, I'm happy with going with that instead. Tempita: - Is small enough for bundling - Has the recommendation of me and Robert Kern (earlier on this list) - Has been used by me for doing a bunch of templating in Cython, C and Fortran code Dag Sverre > > http://www.joonis.de/content/TemplitePythonTemplatingEngine > > http://www.simple-is-better.org/template/pyratemp.html > > There's a whole bunch of engines listed here: > > http://wiki.python.org/moin/Templating > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel From markflorisson88 at gmail.com Fri Jul 22 13:45:18 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 22 Jul 2011 13:45:18 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E295A9E.6090800@behnel.de> References: <4E295A9E.6090800@behnel.de> Message-ID: On 22 July 2011 13:10, Stefan Behnel wrote: > mark florisson, 22.07.2011 12:12: >> >> For my work on the _memview branch (and also on fused types) I noticed >> that UtilityCodes started weighing heavily on me in their current >> form, so I wrote a little loader in the _memview branch: >> >> >> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >> >> The idea is simple: you put your utility codes in Cython/Utility in >> .pyx, .c, .h files etc, and then load them. It works for both >> prototypes and implementations, for UtilityCode and CythonUtilityCode: >> >> myutility.c >> >> // UtilityProto: MyUtility >> header code here >> >> // UtilityCode: MyUtility >> implementation code here >> >> You can add as many other utilities as you like to the same file. You >> can then load it using >> >> ? ? UtilityCode.load_utility_from_file("myutility.c", "MyUtility") > > Why not have exactly one per file? They can't (or at least shouldn't) be > interdependent anyway, since they're always loaded and injected separately. > Having one per file makes it easy to take the file name and grep for it. Putting them in one file does not make them interdependent, they are still loaded separately. Once the first utility is loaded from the file, it loads all the utilities as strings from the file and caches them. You don't want to enforce one utility per file as many utilities are pretty small, and you want to group utilities based on their purpose. (r)Grepping will be just as easy. You may still want to pass the filename as you might want a .c and a corresponding .h (or a .pyx and a corresponding .pxd), or simply one .pyx and one .c, one for the Cython utility code and one for the C utility code. Of course, we could introduce a method load_utility() which takes just "MyUtility" and loads it from MyUtility.*, whichever one it finds first. > >> Of course you can pass in any other arguments, like proto_block, name, >> etc. You can additionally pass it a dict for formatting (for both the >> prototypes and the implementation) > > Dict? Why not keyword arguments? Because (Cython)UtilityCode already takes a bunch of keyword arguments and you want to keep them separate. In any case you could always add whatever convenience method you want. I'll add one to pass in arguments like that. > I'd prefer an interface like this: > > ?UtilityCode.load_from_file("myutility", > ? ? ? ? ?some_format_arg="somename", some_repeat_arg = 2) > > That would automatically look up the corresponding files > ? ?Cython/UtilityCode/myutility.{pyx,c,cpp,h} > and take whichever it finds (first), then run the template engine on it. > > I don't think we need to support separate arguments for prototype and code > section, they can just use different names. Keep it simple. Sure, that's also fine. > >> It will return a UtilityCode instance ready for use. >> You can also simply retrieve a utility code as a string, where it >> returns (proto, implementation). >> >> As debated before, an actual template library would be really >> convenient. Dag and I had a discussion about it and he suggested >> Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3, >> and is pure-python. It supports all the good things like iteration, >> template inheritance, etc. Now I'm not sure whether it supports python >> 2.3 as it doesn't compile on my system, but it does support 2.4 >> (confirmation for 2.3 would be appreciated). On a side note, I'd be >> perfectly happy to drop support for 2.3, it's kind of a chore. >> The documentation for Tempita can be found here: >> http://pythonpaste.org/tempita/ >> >> That way we might rid ourselves of a lot of code.putln() and move >> those to template utilities instead (or at least prevent writing more >> of those). What do you guys think? > > I'm fine with using a template engine for the more involved cases (which are > rare enough). However, I'd prefer not adding a new dependency, but just > shipping a tiny single-module engine with Cython, e.g. Templite or pyratemp > (just found them, never used them). It's far from rare, you could then use it not only for utility codes, but for general code generation. > http://www.joonis.de/content/TemplitePythonTemplatingEngine > > http://www.simple-is-better.org/template/pyratemp.html > > There's a whole bunch of engines listed here: > > http://wiki.python.org/moin/Templating > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From markflorisson88 at gmail.com Fri Jul 22 13:45:46 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 22 Jul 2011 13:45:46 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E295FA4.1020300@astro.uio.no> References: <4E295A9E.6090800@behnel.de> <4E295FA4.1020300@astro.uio.no> Message-ID: On 22 July 2011 13:31, Dag Sverre Seljebotn wrote: > On 07/22/2011 01:10 PM, Stefan Behnel wrote: >> >> mark florisson, 22.07.2011 12:12: >>> >>> For my work on the _memview branch (and also on fused types) I noticed >>> that UtilityCodes started weighing heavily on me in their current >>> form, so I wrote a little loader in the _memview branch: >>> >>> >>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>> >>> >>> The idea is simple: you put your utility codes in Cython/Utility in >>> .pyx, .c, .h files etc, and then load them. It works for both >>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>> >>> myutility.c >>> >>> // UtilityProto: MyUtility >>> header code here >>> >>> // UtilityCode: MyUtility >>> implementation code here >>> >>> You can add as many other utilities as you like to the same file. You >>> can then load it using >>> >>> UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >> >> Why not have exactly one per file? They can't (or at least shouldn't) be >> interdependent anyway, since they're always loaded and injected >> separately. Having one per file makes it easy to take the file name and >> grep for it. >> >> >>> Of course you can pass in any other arguments, like proto_block, name, >>> etc. You can additionally pass it a dict for formatting (for both the >>> prototypes and the implementation) >> >> Dict? Why not keyword arguments? >> >> I'd prefer an interface like this: >> >> UtilityCode.load_from_file("myutility", >> some_format_arg="somename", some_repeat_arg = 2) >> >> That would automatically look up the corresponding files >> Cython/UtilityCode/myutility.{pyx,c,cpp,h} >> and take whichever it finds (first), then run the template engine on it. >> >> I don't think we need to support separate arguments for prototype and >> code section, they can just use different names. Keep it simple. >> >> >>> It will return a UtilityCode instance ready for use. >>> You can also simply retrieve a utility code as a string, where it >>> returns (proto, implementation). >>> >>> As debated before, an actual template library would be really >>> convenient. Dag and I had a discussion about it and he suggested >>> Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3, >>> and is pure-python. It supports all the good things like iteration, >>> template inheritance, etc. Now I'm not sure whether it supports python >>> 2.3 as it doesn't compile on my system, but it does support 2.4 >>> (confirmation for 2.3 would be appreciated). On a side note, I'd be >>> perfectly happy to drop support for 2.3, it's kind of a chore. >>> The documentation for Tempita can be found here: >>> http://pythonpaste.org/tempita/ >>> >>> That way we might rid ourselves of a lot of code.putln() and move >>> those to template utilities instead (or at least prevent writing more >>> of those). What do you guys think? >> >> I'm fine with using a template engine for the more involved cases (which >> are rare enough). However, I'd prefer not adding a new dependency, but >> just shipping a tiny single-module engine with Cython, e.g. Templite or >> pyratemp (just found them, never used them). > > Tempita *IS* intended to be a "tiny single-module engine". Well, it is > spread across several source files, but it wouldn't be intrusive to merge > them into one file (if that's the qualififying criterion). > > I submit that: > > ?a) For the templating we need, there's probably a dozen out there that are > "good enough" and small and simple. > > ?b) So, there's not any point in spending valuable time finding the "optimal > solution". Good enough is good enough. > > ?c) Still, we shouldn't pick one at random, because there might be a > subtlety that makes it impracticaly in real world usage. > > Thus, I think there's a strong case for going by personal recommendation by > somebody who's actually used something, and just go with that. > > If anybody else has used something else and is happy with it, I'm happy with > going with that instead. Tempita: > > ?- Is small enough for bundling > ?- Has the recommendation of me and Robert Kern (earlier on this list) > ?- Has been used by me for doing a bunch of templating in Cython, C and > Fortran code > > Dag Sverre +1, it looks pretty neat to me. >> >> http://www.joonis.de/content/TemplitePythonTemplatingEngine >> >> http://www.simple-is-better.org/template/pyratemp.html >> >> There's a whole bunch of engines listed here: >> >> http://wiki.python.org/moin/Templating >> >> Stefan >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From d.s.seljebotn at astro.uio.no Fri Jul 22 13:54:49 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 22 Jul 2011 13:54:49 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E295A9E.6090800@behnel.de> References: <4E295A9E.6090800@behnel.de> Message-ID: <4E296509.6010704@astro.uio.no> On 07/22/2011 01:10 PM, Stefan Behnel wrote: > mark florisson, 22.07.2011 12:12: >> For my work on the _memview branch (and also on fused types) I noticed >> that UtilityCodes started weighing heavily on me in their current >> form, so I wrote a little loader in the _memview branch: >> >> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >> >> >> The idea is simple: you put your utility codes in Cython/Utility in >> .pyx, .c, .h files etc, and then load them. It works for both >> prototypes and implementations, for UtilityCode and CythonUtilityCode: >> >> myutility.c >> >> // UtilityProto: MyUtility >> header code here >> >> // UtilityCode: MyUtility >> implementation code here >> >> You can add as many other utilities as you like to the same file. You >> can then load it using >> >> UtilityCode.load_utility_from_file("myutility.c", "MyUtility") > > Why not have exactly one per file? They can't (or at least shouldn't) be > interdependent anyway, since they're always loaded and injected > separately. Having one per file makes it easy to take the file name and > grep for it. > > >> Of course you can pass in any other arguments, like proto_block, name, >> etc. You can additionally pass it a dict for formatting (for both the >> prototypes and the implementation) > > Dict? Why not keyword arguments? > > I'd prefer an interface like this: > > UtilityCode.load_from_file("myutility", > some_format_arg="somename", some_repeat_arg = 2) > > That would automatically look up the corresponding files > Cython/UtilityCode/myutility.{pyx,c,cpp,h} > and take whichever it finds (first), then run the template engine on it. > > I don't think we need to support separate arguments for prototype and > code section, they can just use different names. Keep it simple. > > >> It will return a UtilityCode instance ready for use. >> You can also simply retrieve a utility code as a string, where it >> returns (proto, implementation). >> >> As debated before, an actual template library would be really >> convenient. Dag and I had a discussion about it and he suggested >> Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3, >> and is pure-python. It supports all the good things like iteration, >> template inheritance, etc. Now I'm not sure whether it supports python >> 2.3 as it doesn't compile on my system, but it does support 2.4 >> (confirmation for 2.3 would be appreciated). On a side note, I'd be >> perfectly happy to drop support for 2.3, it's kind of a chore. >> The documentation for Tempita can be found here: >> http://pythonpaste.org/tempita/ >> >> That way we might rid ourselves of a lot of code.putln() and move >> those to template utilities instead (or at least prevent writing more >> of those). What do you guys think? > > I'm fine with using a template engine for the more involved cases (which > are rare enough). However, I'd prefer not adding a new dependency, but > just shipping a tiny single-module engine with Cython, e.g. Templite or > pyratemp (just found them, never used them). BTW, I don't think anybody is suggesting having an *external* dependency that users would need to go fetch themselves....that would be insane. Dag Sverre From stefan_ml at behnel.de Fri Jul 22 14:38:27 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 14:38:27 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E295A9E.6090800@behnel.de> Message-ID: <4E296F43.5070409@behnel.de> mark florisson, 22.07.2011 13:45: > On 22 July 2011 13:10, Stefan Behnel wrote: >> mark florisson, 22.07.2011 12:12: >>> >>> For my work on the _memview branch (and also on fused types) I noticed >>> that UtilityCodes started weighing heavily on me in their current >>> form, so I wrote a little loader in the _memview branch: >>> >>> >>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>> >>> The idea is simple: you put your utility codes in Cython/Utility in >>> .pyx, .c, .h files etc, and then load them. It works for both >>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>> >>> myutility.c >>> >>> // UtilityProto: MyUtility >>> header code here >>> >>> // UtilityCode: MyUtility >>> implementation code here >>> >>> You can add as many other utilities as you like to the same file. You >>> can then load it using >>> >>> UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >> >> Why not have exactly one per file? They can't (or at least shouldn't) be >> interdependent anyway, since they're always loaded and injected separately. >> Having one per file makes it easy to take the file name and grep for it. > > Putting them in one file does not make them interdependent I meant the opposite direction. They shouldn't depend on each other anyway, so there's no need to put them in one file. > Once the first utility is loaded from the > file, it loads all the utilities as strings from the file and caches > them. Premature optimisation? ;) > You don't want to enforce one utility per file as many utilities > are pretty small, and you want to group utilities based on their > purpose. It substantially complicates the file handling, though. > You may still want to pass > the filename as you might want a .c and a corresponding .h (or a .pyx > and a corresponding .pxd), or simply one .pyx and one .c, one for the > Cython utility code and one for the C utility code. Do you have a use case for that? For example, when would you need a .h file? AFAICT, that's unprecedented in Cython. Plus, I would expect that files with different extensions would also have different names as they contain different things. > Of course, we could introduce a method load_utility() which takes just > "MyUtility" and loads it from MyUtility.*, whichever one it finds > first. I'm not talking about convenience functionality here, it's more of a basic design question. >>> Of course you can pass in any other arguments, like proto_block, name, >>> etc. You can additionally pass it a dict for formatting (for both the >>> prototypes and the implementation) >> >> Dict? Why not keyword arguments? > > Because (Cython)UtilityCode already takes a bunch of keyword arguments But none that would still matter when you put the code into an external file. IMHO, the file should be as self contained as possible, with the exception of externally provided parameters and (obviously) the decision where the resulting utility code will eventually be injected. >> I'd prefer an interface like this: >> >> UtilityCode.load_from_file("myutility", >> some_format_arg="somename", some_repeat_arg = 2) >> >> That would automatically look up the corresponding files >> Cython/UtilityCode/myutility.{pyx,c,cpp,h} >> and take whichever it finds (first), then run the template engine on it. >> >> I don't think we need to support separate arguments for prototype and code >> section, they can just use different names. Keep it simple. > > Sure, that's also fine. Well, it's a much simpler interface to start with. >>> It will return a UtilityCode instance ready for use. >>> You can also simply retrieve a utility code as a string, where it >>> returns (proto, implementation). >>> >>> As debated before, an actual template library would be really >>> convenient. Dag and I had a discussion about it and he suggested >>> Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3, >>> and is pure-python. It supports all the good things like iteration, >>> template inheritance, etc. Now I'm not sure whether it supports python >>> 2.3 as it doesn't compile on my system, but it does support 2.4 >>> (confirmation for 2.3 would be appreciated). On a side note, I'd be >>> perfectly happy to drop support for 2.3, it's kind of a chore. >>> The documentation for Tempita can be found here: >>> http://pythonpaste.org/tempita/ >>> >>> That way we might rid ourselves of a lot of code.putln() and move >>> those to template utilities instead (or at least prevent writing more >>> of those). What do you guys think? >> >> I'm fine with using a template engine for the more involved cases (which are >> rare enough). However, I'd prefer not adding a new dependency, but just >> shipping a tiny single-module engine with Cython, e.g. Templite or pyratemp >> (just found them, never used them). > > It's far from rare, you could then use it not only for utility codes, > but for general code generation. At least I find it rare. I never felt the need to use a template engine, simply because most of the generated code is either static (utility code) or so highly parametrised that it's better to generate the code programmatically than to use a template. There are exceptions, sure, but I consider them exceptions, and thus rare. Stefan From stefan_ml at behnel.de Fri Jul 22 15:04:12 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 15:04:12 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E296509.6010704@astro.uio.no> References: <4E295A9E.6090800@behnel.de> <4E296509.6010704@astro.uio.no> Message-ID: <4E29754C.3010609@behnel.de> Dag Sverre Seljebotn, 22.07.2011 13:54: > On 07/22/2011 01:10 PM, Stefan Behnel wrote: >> I'm fine with using a template engine for the more involved cases (which >> are rare enough). However, I'd prefer not adding a new dependency, but >> just shipping a tiny single-module engine with Cython, e.g. Templite or >> pyratemp (just found them, never used them). > > BTW, I don't think anybody is suggesting having an *external* dependency > that users would need to go fetch themselves....that would be insane. Sure. I also don't mind having a small engine in a package, but just putting a single file in the source tree (and overwriting it with a new version at need) would be a perfectly small and nonintrusive addition. And given that there really are engines of that size (one file, <500 lines), it's not so unrealistic either. That being said, if you can recommend Tempita (which IMHO qualifies as "small enough"), I won't object to it. Stefan From markflorisson88 at gmail.com Fri Jul 22 15:07:18 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 22 Jul 2011 15:07:18 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E296F43.5070409@behnel.de> References: <4E295A9E.6090800@behnel.de> <4E296F43.5070409@behnel.de> Message-ID: On 22 July 2011 14:38, Stefan Behnel wrote: > mark florisson, 22.07.2011 13:45: >> >> On 22 July 2011 13:10, Stefan Behnel wrote: >>> >>> mark florisson, 22.07.2011 12:12: >>>> >>>> For my work on the _memview branch (and also on fused types) I noticed >>>> that UtilityCodes started weighing heavily on me in their current >>>> form, so I wrote a little loader in the _memview branch: >>>> >>>> >>>> >>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>> >>>> The idea is simple: you put your utility codes in Cython/Utility in >>>> .pyx, .c, .h files etc, and then load them. It works for both >>>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>>> >>>> myutility.c >>>> >>>> // UtilityProto: MyUtility >>>> header code here >>>> >>>> // UtilityCode: MyUtility >>>> implementation code here >>>> >>>> You can add as many other utilities as you like to the same file. You >>>> can then load it using >>>> >>>> ? ? UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>> >>> Why not have exactly one per file? They can't (or at least shouldn't) be >>> interdependent anyway, since they're always loaded and injected >>> separately. >>> Having one per file makes it easy to take the file name and grep for it. >> >> Putting them in one file does not make them interdependent > > I meant the opposite direction. They shouldn't depend on each other anyway, > so there's no need to put them in one file. Say we have one struct definition, and two utilities that depend on it, but not directly on one another. I want all of them in the same file, because they are *related*, just not all dependent on each other. For instance, I have a memviewslice (which is actually a struct) with different "methods". Depending on what methods are used they are included. I certainly want all of them in the same file, but as separate utilities. > >> Once the first utility is loaded from the >> file, it loads all the utilities as strings from the file and caches >> them. > > Premature optimisation? ;) Not at all, I have one file with multiple templates, and you can't seek to their starting position. Why should you re-read the file every time you load just one of them? But sure, I haven't timed it. But I'm sure it will make a difference if I put it on a network share. > >> You don't want to enforce one utility per file as many utilities >> are pretty small, and you want to group utilities based on their >> purpose. > > It substantially complicates the file handling, though. Complicates? In what way? It means I don't have a gazillion files, but just a MemoryView.c and a MemoryView.pyx (actually I renamed the former to avoid confusion, as people may think it's the cythonization of the .pyx). That way I can easily oversee the code, and people will have no trouble locating it. > >> You may still want to pass >> the filename as you might want a .c and a corresponding .h (or a .pyx >> and a corresponding .pxd), or simply one .pyx and one .c, one for the >> Cython utility code and one for the C utility code. > > Do you have a use case for that? For example, when would you need a .h file? > AFAICT, that's unprecedented in Cython. Plus, I would expect that files with > different extensions would also have different names as they contain > different things. > >> Of course, we could introduce a method load_utility() which takes just >> "MyUtility" and loads it from MyUtility.*, whichever one it finds >> first. > > I'm not talking about convenience functionality here, it's more of a basic > design question. > >>>> Of course you can pass in any other arguments, like proto_block, name, >>>> etc. You can additionally pass it a dict for formatting (for both the >>>> prototypes and the implementation) >>> >>> Dict? Why not keyword arguments? >> >> Because (Cython)UtilityCode already takes a bunch of keyword arguments > > But none that would still matter when you put the code into an external > file. IMHO, the file should be as self contained as possible, with the > exception of externally provided parameters and (obviously) the decision > where the resulting utility code will eventually be injected. No, it matters. The load() method takes any additional keyword arguments that are passed to (Cython)UtilityCode. e.g. you might want to pass in the variable 'name' or 'init' into your template, but those are already taken. > >>> I'd prefer an interface like this: >>> >>> ?UtilityCode.load_from_file("myutility", >>> ? ? ? ? ?some_format_arg="somename", some_repeat_arg = 2) >>> >>> That would automatically look up the corresponding files >>> ? ?Cython/UtilityCode/myutility.{pyx,c,cpp,h} >>> and take whichever it finds (first), then run the template engine on it. >>> >>> I don't think we need to support separate arguments for prototype and >>> code >>> section, they can just use different names. Keep it simple. >> >> Sure, that's also fine. > > Well, it's a much simpler interface to start with. > > >>>> It will return a UtilityCode instance ready for use. >>>> You can also simply retrieve a utility code as a string, where it >>>> returns (proto, implementation). >>>> >>>> As debated before, an actual template library would be really >>>> convenient. Dag and I had a discussion about it and he suggested >>>> Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3, >>>> and is pure-python. It supports all the good things like iteration, >>>> template inheritance, etc. Now I'm not sure whether it supports python >>>> 2.3 as it doesn't compile on my system, but it does support 2.4 >>>> (confirmation for 2.3 would be appreciated). On a side note, I'd be >>>> perfectly happy to drop support for 2.3, it's kind of a chore. >>>> The documentation for Tempita can be found here: >>>> http://pythonpaste.org/tempita/ >>>> >>>> That way we might rid ourselves of a lot of code.putln() and move >>>> those to template utilities instead (or at least prevent writing more >>>> of those). What do you guys think? >>> >>> I'm fine with using a template engine for the more involved cases (which >>> are >>> rare enough). However, I'd prefer not adding a new dependency, but just >>> shipping a tiny single-module engine with Cython, e.g. Templite or >>> pyratemp >>> (just found them, never used them). >> >> It's far from rare, you could then use it not only for utility codes, >> but for general code generation. > > At least I find it rare. I never felt the need to use a template engine, > simply because most of the generated code is either static (utility code) or > so highly parametrised that it's better to generate the code > programmatically than to use a template. There are exceptions, sure, but I > consider them exceptions, and thus rare. > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Fri Jul 22 15:07:52 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 15:07:52 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E29754C.3010609@behnel.de> References: <4E295A9E.6090800@behnel.de> <4E296509.6010704@astro.uio.no> <4E29754C.3010609@behnel.de> Message-ID: <4E297628.6000908@behnel.de> Stefan Behnel, 22.07.2011 15:04: > Dag Sverre Seljebotn, 22.07.2011 13:54: >> On 07/22/2011 01:10 PM, Stefan Behnel wrote: >>> I'm fine with using a template engine for the more involved cases (which >>> are rare enough). However, I'd prefer not adding a new dependency, but >>> just shipping a tiny single-module engine with Cython, e.g. Templite or >>> pyratemp (just found them, never used them). >> >> BTW, I don't think anybody is suggesting having an *external* dependency >> that users would need to go fetch themselves....that would be insane. > > Sure. I also don't mind having a small engine in a package, but just > putting a single file in the source tree (and overwriting it with a new > version at need) would be a perfectly small and nonintrusive addition. And > given that there really are engines of that size (one file, <500 lines), > it's not so unrealistic either. > > That being said, if you can recommend Tempita (which IMHO qualifies as > "small enough"), I won't object to it. ... although, isn't it unfortunate that it uses "{{...}}" for template code? How well readable is that when used in C code? Stefan From stefan_ml at behnel.de Fri Jul 22 15:54:34 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 15:54:34 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E295A9E.6090800@behnel.de> <4E296F43.5070409@behnel.de> Message-ID: <4E29811A.9070109@behnel.de> mark florisson, 22.07.2011 15:07: > On 22 July 2011 14:38, Stefan Behnel wrote: >> mark florisson, 22.07.2011 13:45: >>> >>> On 22 July 2011 13:10, Stefan Behnel wrote: >>>> >>>> mark florisson, 22.07.2011 12:12: >>>>> >>>>> For my work on the _memview branch (and also on fused types) I noticed >>>>> that UtilityCodes started weighing heavily on me in their current >>>>> form, so I wrote a little loader in the _memview branch: >>>>> >>>>> >>>>> >>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>>> >>>>> The idea is simple: you put your utility codes in Cython/Utility in >>>>> .pyx, .c, .h files etc, and then load them. It works for both >>>>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>>>> >>>>> myutility.c >>>>> >>>>> // UtilityProto: MyUtility >>>>> header code here >>>>> >>>>> // UtilityCode: MyUtility >>>>> implementation code here >>>>> >>>>> You can add as many other utilities as you like to the same file. You >>>>> can then load it using >>>>> >>>>> UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>>> >>>> Why not have exactly one per file? They can't (or at least shouldn't) be >>>> interdependent anyway, since they're always loaded and injected >>>> separately. >>>> Having one per file makes it easy to take the file name and grep for it. >>> >>> Putting them in one file does not make them interdependent >> >> I meant the opposite direction. They shouldn't depend on each other anyway, >> so there's no need to put them in one file. > > Say we have one struct definition, and two utilities that depend on > it, but not directly on one another. I want all of them in the same > file, because they are *related*, just not all dependent on each > other. Ok, I see the use case now. How do you mark the dependencies within the file? And how do you mark dependencies towards other code in other files? I.e. how do you map the parameters that UtilityCode() current takes into the file format? You sure don't want to keep those in the Python code that injects (or loads) the utility code, do you? >>> Once the first utility is loaded from the >>> file, it loads all the utilities as strings from the file and caches >>> them. >> >> Premature optimisation? ;) > > Not at all, I have one file with multiple templates, and you can't > seek to their starting position. Why should you re-read the file every > time you load just one of them? You seem to be subscribed to misinterpreting me. ;) *Iff* you accept that multiple implementations should be in one file, then this approach makes sense. I was merely suggesting that the performance is not a reason by itself for putting multiple implementations into one file. >>> You don't want to enforce one utility per file as many utilities >>> are pretty small, and you want to group utilities based on their >>> purpose. >> >> It substantially complicates the file handling, though. > > Complicates? In what way? In the sense that you have to do something to split the file into separate sections before being able to apply the templating engine to it. So you have several levels of marker semantics in the file format, one to separate the implementations, one to separate the parts of an implementation, and one level for the templating language. I call that "complicating the file handling". However, given that you always need to express metadata in some way (e.g. for dependencies), I don't think you can do better than with two levels anyway, so a third won't hurt *that* much. >>>>> Of course you can pass in any other arguments, like proto_block, name, >>>>> etc. You can additionally pass it a dict for formatting (for both the >>>>> prototypes and the implementation) >>>> >>>> Dict? Why not keyword arguments? >>> >>> Because (Cython)UtilityCode already takes a bunch of keyword arguments >> >> But none that would still matter when you put the code into an external >> file. IMHO, the file should be as self contained as possible, with the >> exception of externally provided parameters and (obviously) the decision >> where the resulting utility code will eventually be injected. > > No, it matters. The load() method takes any additional keyword > arguments that are passed to (Cython)UtilityCode. e.g. you might want > to pass in the variable 'name' or 'init' into your template, but those > are already taken. But why would you need to pass those into the UtilityCode instantiation? As I said, the templates should be as self-contained as possible, just as they are today, as written in the Python code. Passing "init" as a parameter to UtilityCode() doesn't make sense to me. (not sure what exactly the "name" parameter is there for, though) Stefan From stefan_ml at behnel.de Fri Jul 22 16:08:22 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 16:08:22 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E29811A.9070109@behnel.de> References: <4E295A9E.6090800@behnel.de> <4E296F43.5070409@behnel.de> <4E29811A.9070109@behnel.de> Message-ID: <4E298456.9050001@behnel.de> Stefan Behnel, 22.07.2011 15:54: > However, given that you always need to express metadata in some way (e.g. > for dependencies), I don't think you can do better than with two levels > anyway, so a third won't hurt *that* much. Actually, I take that back. Two levels are easy (think of Windows-INI style files), three levels start to get tricky. Stefan From markflorisson88 at gmail.com Fri Jul 22 16:11:41 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 22 Jul 2011 16:11:41 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E29811A.9070109@behnel.de> References: <4E295A9E.6090800@behnel.de> <4E296F43.5070409@behnel.de> <4E29811A.9070109@behnel.de> Message-ID: On 22 July 2011 15:54, Stefan Behnel wrote: > mark florisson, 22.07.2011 15:07: >> >> On 22 July 2011 14:38, Stefan Behnel wrote: >>> >>> mark florisson, 22.07.2011 13:45: >>>> >>>> On 22 July 2011 13:10, Stefan Behnel wrote: >>>>> >>>>> mark florisson, 22.07.2011 12:12: >>>>>> >>>>>> For my work on the _memview branch (and also on fused types) I noticed >>>>>> that UtilityCodes started weighing heavily on me in their current >>>>>> form, so I wrote a little loader in the _memview branch: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>>>> >>>>>> The idea is simple: you put your utility codes in Cython/Utility in >>>>>> .pyx, .c, .h files etc, and then load them. It works for both >>>>>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>>>>> >>>>>> myutility.c >>>>>> >>>>>> // UtilityProto: MyUtility >>>>>> header code here >>>>>> >>>>>> // UtilityCode: MyUtility >>>>>> implementation code here >>>>>> >>>>>> You can add as many other utilities as you like to the same file. You >>>>>> can then load it using >>>>>> >>>>>> ? ? UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>>>> >>>>> Why not have exactly one per file? They can't (or at least shouldn't) >>>>> be >>>>> interdependent anyway, since they're always loaded and injected >>>>> separately. >>>>> Having one per file makes it easy to take the file name and grep for >>>>> it. >>>> >>>> Putting them in one file does not make them interdependent >>> >>> I meant the opposite direction. They shouldn't depend on each other >>> anyway, >>> so there's no need to put them in one file. >> >> Say we have one struct definition, and two utilities that depend on >> it, but not directly on one another. I want all of them in the same >> file, because they are *related*, just not all dependent on each >> other. > > Ok, I see the use case now. How do you mark the dependencies within the > file? And how do you mark dependencies towards other code in other files? > I.e. how do you map the parameters that UtilityCode() current takes into the > file format? > > You sure don't want to keep those in the Python code that injects (or loads) > the utility code, do you? > That's what I currently do. You load the utilities with the normal arguments, except that the implementation and prototypes are automatically fetched from a utility code file. So you do my_utility = UtilityCode.load("MyUtility", proto_block='...', requires=[my_other_utility]) It wouldn't be hard to support requirements (and other options) in utility code files though. Do you think we should? >>>> Once the first utility is loaded from the >>>> file, it loads all the utilities as strings from the file and caches >>>> them. >>> >>> Premature optimisation? ;) >> >> Not at all, I have one file with multiple templates, and you can't >> seek to their starting position. Why should you re-read the file every >> time you load just one of them? > > You seem to be subscribed to misinterpreting me. ;) > > *Iff* you accept that multiple implementations should be in one file, then > this approach makes sense. I was merely suggesting that the performance is > not a reason by itself for putting multiple implementations into one file. > Ah, sure, I agree then :) I want this feature for my sanity, not for optimization. >>>> You don't want to enforce one utility per file as many utilities >>>> are pretty small, and you want to group utilities based on their >>>> purpose. >>> >>> It substantially complicates the file handling, though. >> >> Complicates? In what way? > > In the sense that you have to do something to split the file into separate > sections before being able to apply the templating engine to it. So you have > several levels of marker semantics in the file format, one to separate the > implementations, one to separate the parts of an implementation, and one > level for the templating language. I call that "complicating the file > handling". > > However, given that you always need to express metadata in some way (e.g. > for dependencies), I don't think you can do better than with two levels > anyway, so a third won't hurt *that* much. > Actually, the prototype and the implementation do not need to be grouped. i.e. it is valid to first list all prototypes and then all implementations. So yeah, there would be two levels instead of one, but that's what you want, because you want different parameters for different utilities in the file, and you may load them at different times or even multiple times with different parameters. >>>>>> Of course you can pass in any other arguments, like proto_block, name, >>>>>> etc. You can additionally pass it a dict for formatting (for both the >>>>>> prototypes and the implementation) >>>>> >>>>> Dict? Why not keyword arguments? >>>> >>>> Because (Cython)UtilityCode already takes a bunch of keyword arguments >>> >>> But none that would still matter when you put the code into an external >>> file. IMHO, the file should be as self contained as possible, with the >>> exception of externally provided parameters and (obviously) the decision >>> where the resulting utility code will eventually be injected. >> >> No, it matters. The load() method takes any additional keyword >> arguments that are passed to (Cython)UtilityCode. e.g. you might want >> to pass in the variable 'name' or 'init' into your template, but those >> are already taken. > > But why would you need to pass those into the UtilityCode instantiation? As > I said, the templates should be as self-contained as possible, just as they > are today, as written in the Python code. Passing "init" as a parameter to > UtilityCode() doesn't make sense to me. (not sure what exactly the "name" > parameter is there for, though) Well this was just a gentle start. Fortunately the init parameter is only used twice in the entire codebase. > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From markflorisson88 at gmail.com Fri Jul 22 16:13:19 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 22 Jul 2011 16:13:19 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E298456.9050001@behnel.de> References: <4E295A9E.6090800@behnel.de> <4E296F43.5070409@behnel.de> <4E29811A.9070109@behnel.de> <4E298456.9050001@behnel.de> Message-ID: On 22 July 2011 16:08, Stefan Behnel wrote: > Stefan Behnel, 22.07.2011 15:54: >> >> However, given that you always need to express metadata in some way (e.g. >> for dependencies), I don't think you can do better than with two levels >> anyway, so a third won't hurt *that* much. > > Actually, I take that back. Two levels are easy (think of Windows-INI style > files), three levels start to get tricky. Fortunate that it is two levels then. > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Fri Jul 22 16:31:45 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 16:31:45 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E295A9E.6090800@behnel.de> <4E296F43.5070409@behnel.de> <4E29811A.9070109@behnel.de> Message-ID: <4E2989D1.2060801@behnel.de> mark florisson, 22.07.2011 16:11: > On 22 July 2011 15:54, Stefan Behnel wrote: >> mark florisson, 22.07.2011 15:07: >>> >>> On 22 July 2011 14:38, Stefan Behnel wrote: >>>> >>>> mark florisson, 22.07.2011 13:45: >>>>> >>>>> On 22 July 2011 13:10, Stefan Behnel wrote: >>>>>> >>>>>> mark florisson, 22.07.2011 12:12: >>>>>>> >>>>>>> For my work on the _memview branch (and also on fused types) I noticed >>>>>>> that UtilityCodes started weighing heavily on me in their current >>>>>>> form, so I wrote a little loader in the _memview branch: >>>>>>> >>>>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>>>>> >>>>>>> The idea is simple: you put your utility codes in Cython/Utility in >>>>>>> .pyx, .c, .h files etc, and then load them. It works for both >>>>>>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>>>>>> >>>>>>> myutility.c >>>>>>> >>>>>>> // UtilityProto: MyUtility >>>>>>> header code here >>>>>>> >>>>>>> // UtilityCode: MyUtility >>>>>>> implementation code here >>>>>>> >>>>>>> You can add as many other utilities as you like to the same file. You >>>>>>> can then load it using >>>>>>> >>>>>>> UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>>>>> >>>>>> Why not have exactly one per file? They can't (or at least shouldn't) >>>>>> be >>>>>> interdependent anyway, since they're always loaded and injected >>>>>> separately. >>>>>> Having one per file makes it easy to take the file name and grep for >>>>>> it. >>>>> >>>>> Putting them in one file does not make them interdependent >>>> >>>> I meant the opposite direction. They shouldn't depend on each other >>>> anyway, >>>> so there's no need to put them in one file. >>> >>> Say we have one struct definition, and two utilities that depend on >>> it, but not directly on one another. I want all of them in the same >>> file, because they are *related*, just not all dependent on each >>> other. >> >> Ok, I see the use case now. How do you mark the dependencies within the >> file? And how do you mark dependencies towards other code in other files? >> I.e. how do you map the parameters that UtilityCode() current takes into the >> file format? >> >> You sure don't want to keep those in the Python code that injects (or loads) >> the utility code, do you? > > That's what I currently do. You load the utilities with the normal > arguments, except that the implementation and prototypes are > automatically fetched from a utility code file. So you do > > my_utility = UtilityCode.load("MyUtility", proto_block='...', > requires=[my_other_utility]) Ah, ok, I hadn't noticed that in your commit. That doesn't make much sense. It's the implementation that knows what it depends on, not the code that uses it. It's not uncommon for utility code to be used in multiple places, it's bad to require all of them to agree on the same configuration for that code. That's why I'm saying that the external file should be self-contained in this regard. The dependencies must be explicit and right where the code is. > It wouldn't be hard to support requirements (and other options) in > utility code files though. Do you think we should? Sure. The code that injects utility code should only have to know its name and what cnames (functions etc.) it provides, and shouldn't have to know anything about its implementation details. Exactly as it works now. >>>>> You don't want to enforce one utility per file as many utilities >>>>> are pretty small, and you want to group utilities based on their >>>>> purpose. >>>> >>>> It substantially complicates the file handling, though. >>> >>> Complicates? In what way? >> >> In the sense that you have to do something to split the file into separate >> sections before being able to apply the templating engine to it. So you have >> several levels of marker semantics in the file format, one to separate the >> implementations, one to separate the parts of an implementation, and one >> level for the templating language. I call that "complicating the file >> handling". >> >> However, given that you always need to express metadata in some way (e.g. >> for dependencies), I don't think you can do better than with two levels >> anyway, so a third won't hurt *that* much. > > Actually, the prototype and the implementation do not need to be > grouped. i.e. it is valid to first list all prototypes and then all > implementations. I understand that, and it makes sense to support that. > So yeah, there would be two levels instead of one, Well, three instead of two. But I see how you want to mingle the third level into the second. That should work. > but that's what you want, because you want different parameters for > different utilities in the file, and you may load them at different > times or even multiple times with different parameters. Right, that's why it would be easier with separate files. Stefan From markflorisson88 at gmail.com Fri Jul 22 16:45:26 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 22 Jul 2011 16:45:26 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E2989D1.2060801@behnel.de> References: <4E295A9E.6090800@behnel.de> <4E296F43.5070409@behnel.de> <4E29811A.9070109@behnel.de> <4E2989D1.2060801@behnel.de> Message-ID: On 22 July 2011 16:31, Stefan Behnel wrote: > mark florisson, 22.07.2011 16:11: >> >> On 22 July 2011 15:54, Stefan Behnel wrote: >>> >>> mark florisson, 22.07.2011 15:07: >>>> >>>> On 22 July 2011 14:38, Stefan Behnel wrote: >>>>> >>>>> mark florisson, 22.07.2011 13:45: >>>>>> >>>>>> On 22 July 2011 13:10, Stefan Behnel wrote: >>>>>>> >>>>>>> mark florisson, 22.07.2011 12:12: >>>>>>>> >>>>>>>> For my work on the _memview branch (and also on fused types) I >>>>>>>> noticed >>>>>>>> that UtilityCodes started weighing heavily on me in their current >>>>>>>> form, so I wrote a little loader in the _memview branch: >>>>>>>> >>>>>>>> >>>>>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>>>>>> >>>>>>>> The idea is simple: you put your utility codes in Cython/Utility in >>>>>>>> .pyx, .c, .h files etc, and then load them. It works for both >>>>>>>> prototypes and implementations, for UtilityCode and >>>>>>>> CythonUtilityCode: >>>>>>>> >>>>>>>> myutility.c >>>>>>>> >>>>>>>> // UtilityProto: MyUtility >>>>>>>> header code here >>>>>>>> >>>>>>>> // UtilityCode: MyUtility >>>>>>>> implementation code here >>>>>>>> >>>>>>>> You can add as many other utilities as you like to the same file. >>>>>>>> You >>>>>>>> can then load it using >>>>>>>> >>>>>>>> ? ? UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>>>>>> >>>>>>> Why not have exactly one per file? They can't (or at least shouldn't) >>>>>>> be >>>>>>> interdependent anyway, since they're always loaded and injected >>>>>>> separately. >>>>>>> Having one per file makes it easy to take the file name and grep for >>>>>>> it. >>>>>> >>>>>> Putting them in one file does not make them interdependent >>>>> >>>>> I meant the opposite direction. They shouldn't depend on each other >>>>> anyway, >>>>> so there's no need to put them in one file. >>>> >>>> Say we have one struct definition, and two utilities that depend on >>>> it, but not directly on one another. I want all of them in the same >>>> file, because they are *related*, just not all dependent on each >>>> other. >>> >>> Ok, I see the use case now. How do you mark the dependencies within the >>> file? And how do you mark dependencies towards other code in other files? >>> I.e. how do you map the parameters that UtilityCode() current takes into >>> the >>> file format? >>> >>> You sure don't want to keep those in the Python code that injects (or >>> loads) >>> the utility code, do you? >> >> That's what I currently do. You load the utilities with the normal >> arguments, except that the implementation and prototypes are >> automatically fetched from a utility code file. So you do >> >> my_utility = UtilityCode.load("MyUtility", proto_block='...', >> requires=[my_other_utility]) > > Ah, ok, I hadn't noticed that in your commit. That doesn't make much sense. > It's the implementation that knows what it depends on, not the code that > uses it. It's not uncommon for utility code to be used in multiple places, > it's bad to require all of them to agree on the same configuration for that > code. That's why I'm saying that the external file should be self-contained > in this regard. The dependencies must be explicit and right where the code > is. > Point taken, it was mainly implemented this way because it was easiest and still premature. I'll support at least the requires clause. >> It wouldn't be hard to support requirements (and other options) in >> utility code files though. Do you think we should? > > Sure. The code that injects utility code should only have to know its name > and what cnames (functions etc.) it provides, and shouldn't have to know > anything about its implementation details. Exactly as it works now. > This is about loading the utility code, not about injecting them. You usually assign them to global variables, just like before. >>>>>> You don't want to enforce one utility per file as many utilities >>>>>> are pretty small, and you want to group utilities based on their >>>>>> purpose. >>>>> >>>>> It substantially complicates the file handling, though. >>>> >>>> Complicates? In what way? >>> >>> In the sense that you have to do something to split the file into >>> separate >>> sections before being able to apply the templating engine to it. So you >>> have >>> several levels of marker semantics in the file format, one to separate >>> the >>> implementations, one to separate the parts of an implementation, and one >>> level for the templating language. I call that "complicating the file >>> handling". >>> >>> However, given that you always need to express metadata in some way (e.g. >>> for dependencies), I don't think you can do better than with two levels >>> anyway, so a third won't hurt *that* much. >> >> Actually, the prototype and the implementation do not need to be >> grouped. i.e. it is valid to first list all prototypes and then all >> implementations. > > I understand that, and it makes sense to support that. > > >> So yeah, there would be two levels instead of one, > > Well, three instead of two. But I see how you want to mingle the third level > into the second. That should work. > > >> but that's what you want, because you want different parameters for >> different utilities in the file, and you may load them at different >> times or even multiple times with different parameters. > > Right, that's why it would be easier with separate files. Unless you think passing in a filename is hard, separating the utilities in different files doesn't make it easier in any way. For my utilities I simply created a function load_memview_cy_utility() that eats the name and parameters and loads it from the right file. I think this two-line function is well-worth the convenience of the grouping. > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Fri Jul 22 16:49:57 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 16:49:57 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E295A9E.6090800@behnel.de> <4E296F43.5070409@behnel.de> <4E29811A.9070109@behnel.de> <4E298456.9050001@behnel.de> Message-ID: <4E298E15.7060405@behnel.de> mark florisson, 22.07.2011 16:13: > On 22 July 2011 16:08, Stefan Behnel wrote: >> Stefan Behnel, 22.07.2011 15:54: >>> >>> However, given that you always need to express metadata in some way (e.g. >>> for dependencies), I don't think you can do better than with two levels >>> anyway, so a third won't hurt *that* much. >> >> Actually, I take that back. Two levels are easy (think of Windows-INI style >> files), three levels start to get tricky. > > Fortunate that it is two levels then. BTW, what about actually using an ini style file format? Something like the following may work, but I guess RawConfigParser won't be able to properly pass through the C code... Stefan [utilitycode] export=MyCode1,MyCode2 [MyCode1] depends=file1:OtherCode1,file1:OtherCode2,MyCode2 [MyCode2] depends=file2:OtherCode3 [MyCode1:proto] static void ...(); /*proto*/ [Mycode2:proto] static int ...(); /*proto*/ [Mycode2:impl] static int ...() { ... } [Mycode1:impl] static void ...() { ... } From markflorisson88 at gmail.com Fri Jul 22 16:57:51 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 22 Jul 2011 16:57:51 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E298E15.7060405@behnel.de> References: <4E295A9E.6090800@behnel.de> <4E296F43.5070409@behnel.de> <4E29811A.9070109@behnel.de> <4E298456.9050001@behnel.de> <4E298E15.7060405@behnel.de> Message-ID: On 22 July 2011 16:49, Stefan Behnel wrote: > mark florisson, 22.07.2011 16:13: >> >> On 22 July 2011 16:08, Stefan Behnel wrote: >>> >>> Stefan Behnel, 22.07.2011 15:54: >>>> >>>> However, given that you always need to express metadata in some way >>>> (e.g. >>>> for dependencies), I don't think you can do better than with two levels >>>> anyway, so a third won't hurt *that* much. >>> >>> Actually, I take that back. Two levels are easy (think of Windows-INI >>> style >>> files), three levels start to get tricky. >> >> Fortunate that it is two levels then. > > BTW, what about actually using an ini style file format? Something like the > following may work, but I guess RawConfigParser won't be able to properly > pass through the C code... > > Stefan > > > [utilitycode] > export=MyCode1,MyCode2 > > [MyCode1] > depends=file1:OtherCode1,file1:OtherCode2,MyCode2 > > [MyCode2] > depends=file2:OtherCode3 > > [MyCode1:proto] > static void ...(); /*proto*/ > > [Mycode2:proto] > static int ...(); /*proto*/ > > [Mycode2:impl] > static int ...() { > ... > } > > [Mycode1:impl] > static void ...() { > ... > } I prefer comments, in .pyx files you use # and in .c files you use //. Works nicely with syntax highlighting. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From d.s.seljebotn at astro.uio.no Fri Jul 22 17:46:27 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 22 Jul 2011 17:46:27 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E297628.6000908@behnel.de> References: <4E295A9E.6090800@behnel.de> <4E296509.6010704@astro.uio.no> <4E29754C.3010609@behnel.de> <4E297628.6000908@behnel.de> Message-ID: <4E299B53.1010604@astro.uio.no> On 07/22/2011 03:07 PM, Stefan Behnel wrote: > Stefan Behnel, 22.07.2011 15:04: >> Dag Sverre Seljebotn, 22.07.2011 13:54: >>> On 07/22/2011 01:10 PM, Stefan Behnel wrote: >>>> I'm fine with using a template engine for the more involved cases >>>> (which >>>> are rare enough). However, I'd prefer not adding a new dependency, but >>>> just shipping a tiny single-module engine with Cython, e.g. Templite or >>>> pyratemp (just found them, never used them). >>> >>> BTW, I don't think anybody is suggesting having an *external* dependency >>> that users would need to go fetch themselves....that would be insane. >> >> Sure. I also don't mind having a small engine in a package, but just >> putting a single file in the source tree (and overwriting it with a new >> version at need) would be a perfectly small and nonintrusive addition. >> And >> given that there really are engines of that size (one file, <500 lines), >> it's not so unrealistic either. >> >> That being said, if you can recommend Tempita (which IMHO qualifies as >> "small enough"), I won't object to it. > > ... although, isn't it unfortunate that it uses "{{...}}" for template > code? How well readable is that when used in C code? At least in emacs, pretty much any template syntax will mess up things pretty badly, especially automatic indentation and so on. What I do is using mmm-mode, so that everything within {{ and }} is highlighted with python-mode, and the rest with c-mode. I'll share my configuration later. I'm merely stating why this has never been an issue for me, I'm NOT suggesting everyone should use emacs (or that plain text editing isn't convenient); you do have a point. Keep in mind that a syntax must work well with utility code written in Cython as well. Dag Sverre From d.s.seljebotn at astro.uio.no Fri Jul 22 17:49:07 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 22 Jul 2011 17:49:07 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E299B53.1010604@astro.uio.no> References: <4E295A9E.6090800@behnel.de> <4E296509.6010704@astro.uio.no> <4E29754C.3010609@behnel.de> <4E297628.6000908@behnel.de> <4E299B53.1010604@astro.uio.no> Message-ID: <4E299BF3.2080800@astro.uio.no> On 07/22/2011 05:46 PM, Dag Sverre Seljebotn wrote: > On 07/22/2011 03:07 PM, Stefan Behnel wrote: >> Stefan Behnel, 22.07.2011 15:04: >>> Dag Sverre Seljebotn, 22.07.2011 13:54: >>>> On 07/22/2011 01:10 PM, Stefan Behnel wrote: >>>>> I'm fine with using a template engine for the more involved cases >>>>> (which >>>>> are rare enough). However, I'd prefer not adding a new dependency, but >>>>> just shipping a tiny single-module engine with Cython, e.g. >>>>> Templite or >>>>> pyratemp (just found them, never used them). >>>> >>>> BTW, I don't think anybody is suggesting having an *external* >>>> dependency >>>> that users would need to go fetch themselves....that would be insane. >>> >>> Sure. I also don't mind having a small engine in a package, but just >>> putting a single file in the source tree (and overwriting it with a new >>> version at need) would be a perfectly small and nonintrusive addition. >>> And >>> given that there really are engines of that size (one file, <500 lines), >>> it's not so unrealistic either. >>> >>> That being said, if you can recommend Tempita (which IMHO qualifies as >>> "small enough"), I won't object to it. >> >> ... although, isn't it unfortunate that it uses "{{...}}" for template >> code? How well readable is that when used in C code? > > At least in emacs, pretty much any template syntax will mess up things > pretty badly, especially automatic indentation and so on. > > What I do is using mmm-mode, so that everything within {{ and }} is > highlighted with python-mode, and the rest with c-mode. I'll share my > configuration later. > > I'm merely stating why this has never been an issue for me, I'm NOT > suggesting everyone should use emacs (or that plain text editing isn't > convenient); you do have a point. > > Keep in mind that a syntax must work well with utility code written in > Cython as well. OTOH one *might* want to have code.put_tempita(..) # or similar in Python code, and in that case plain text readability is very important. Of course, hacking Tempita to make it %[ ]% instead is a five minute job. But I'm happy with any other engine that somebody have actually tried out extensively. Dag Sverre From stefan_ml at behnel.de Fri Jul 22 17:58:16 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 17:58:16 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E299BF3.2080800@astro.uio.no> References: <4E295A9E.6090800@behnel.de> <4E296509.6010704@astro.uio.no> <4E29754C.3010609@behnel.de> <4E297628.6000908@behnel.de> <4E299B53.1010604@astro.uio.no> <4E299BF3.2080800@astro.uio.no> Message-ID: <4E299E18.2020203@behnel.de> Dag Sverre Seljebotn, 22.07.2011 17:49: > On 07/22/2011 05:46 PM, Dag Sverre Seljebotn wrote: >> On 07/22/2011 03:07 PM, Stefan Behnel wrote: >>> Stefan Behnel, 22.07.2011 15:04: >>>> Dag Sverre Seljebotn, 22.07.2011 13:54: >>>>> On 07/22/2011 01:10 PM, Stefan Behnel wrote: >>>>>> I'm fine with using a template engine for the more involved cases >>>>>> (which >>>>>> are rare enough). However, I'd prefer not adding a new dependency, but >>>>>> just shipping a tiny single-module engine with Cython, e.g. >>>>>> Templite or >>>>>> pyratemp (just found them, never used them). >>>>> >>>>> BTW, I don't think anybody is suggesting having an *external* >>>>> dependency >>>>> that users would need to go fetch themselves....that would be insane. >>>> >>>> Sure. I also don't mind having a small engine in a package, but just >>>> putting a single file in the source tree (and overwriting it with a new >>>> version at need) would be a perfectly small and nonintrusive addition. >>>> And >>>> given that there really are engines of that size (one file, <500 lines), >>>> it's not so unrealistic either. >>>> >>>> That being said, if you can recommend Tempita (which IMHO qualifies as >>>> "small enough"), I won't object to it. >>> >>> ... although, isn't it unfortunate that it uses "{{...}}" for template >>> code? How well readable is that when used in C code? >> >> At least in emacs, pretty much any template syntax will mess up things >> pretty badly, especially automatic indentation and so on. >> >> What I do is using mmm-mode, so that everything within {{ and }} is >> highlighted with python-mode, and the rest with c-mode. I'll share my >> configuration later. >> >> I'm merely stating why this has never been an issue for me, I'm NOT >> suggesting everyone should use emacs (or that plain text editing isn't >> convenient); you do have a point. >> >> Keep in mind that a syntax must work well with utility code written in >> Cython as well. > > OTOH one *might* want to have > > code.put_tempita(..) # or similar > > in Python code, and in that case plain text readability is very important. > > Of course, hacking Tempita to make it %[ ]% instead is a five minute job. > But I'm happy with any other engine that somebody have actually tried out > extensively. Tempita accepts the delimiters as a parameter to the Template class, so that turns out to be a minor issue - we just have to come up with some good characters ourselves. Stefan From stefan_ml at behnel.de Fri Jul 22 18:04:44 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 18:04:44 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E299BF3.2080800@astro.uio.no> References: <4E295A9E.6090800@behnel.de> <4E296509.6010704@astro.uio.no> <4E29754C.3010609@behnel.de> <4E297628.6000908@behnel.de> <4E299B53.1010604@astro.uio.no> <4E299BF3.2080800@astro.uio.no> Message-ID: <4E299F9C.4010906@behnel.de> Dag Sverre Seljebotn, 22.07.2011 17:49: > On 07/22/2011 05:46 PM, Dag Sverre Seljebotn wrote: >> On 07/22/2011 03:07 PM, Stefan Behnel wrote: >>> Stefan Behnel, 22.07.2011 15:04: >>>> Dag Sverre Seljebotn, 22.07.2011 13:54: >>>>> On 07/22/2011 01:10 PM, Stefan Behnel wrote: >>>>>> I'm fine with using a template engine for the more involved cases >>>>>> (which >>>>>> are rare enough). However, I'd prefer not adding a new dependency, but >>>>>> just shipping a tiny single-module engine with Cython, e.g. >>>>>> Templite or >>>>>> pyratemp (just found them, never used them). >>>>> >>>>> BTW, I don't think anybody is suggesting having an *external* >>>>> dependency >>>>> that users would need to go fetch themselves....that would be insane. >>>> >>>> Sure. I also don't mind having a small engine in a package, but just >>>> putting a single file in the source tree (and overwriting it with a new >>>> version at need) would be a perfectly small and nonintrusive addition. >>>> And >>>> given that there really are engines of that size (one file, <500 lines), >>>> it's not so unrealistic either. >>>> >>>> That being said, if you can recommend Tempita (which IMHO qualifies as >>>> "small enough"), I won't object to it. >>> >>> ... although, isn't it unfortunate that it uses "{{...}}" for template >>> code? How well readable is that when used in C code? >> >> At least in emacs, pretty much any template syntax will mess up things >> pretty badly, especially automatic indentation and so on. >> >> What I do is using mmm-mode, so that everything within {{ and }} is >> highlighted with python-mode, and the rest with c-mode. I'll share my >> configuration later. >> >> I'm merely stating why this has never been an issue for me, I'm NOT >> suggesting everyone should use emacs (or that plain text editing isn't >> convenient); you do have a point. >> >> Keep in mind that a syntax must work well with utility code written in >> Cython as well. > > OTOH one *might* want to have > > code.put_tempita(..) # or similar > > in Python code, and in that case plain text readability is very important. > > Of course, hacking Tempita to make it %[ ]% instead is a five minute job. > But I'm happy with any other engine that somebody have actually tried out > extensively. Tempita has a syntax for comments - what about using the regular C and Cython comments as template syntax delimiters? That way, you'd write /*# some comment */ to get an actual comment in C code, and /* for x in y */ for template code. For Cython code, we could use "##", or maybe "#<" and ">#". That should work pretty well with editors. Stefan From hoytak at stat.washington.edu Fri Jul 22 18:48:04 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Fri, 22 Jul 2011 09:48:04 -0700 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E299F9C.4010906@behnel.de> References: <4E295A9E.6090800@behnel.de> <4E296509.6010704@astro.uio.no> <4E29754C.3010609@behnel.de> <4E297628.6000908@behnel.de> <4E299B53.1010604@astro.uio.no> <4E299BF3.2080800@astro.uio.no> <4E299F9C.4010906@behnel.de> Message-ID: I know I'm a little late to the discussion, but I've been meaning to ask about a feature like this myself. As a point of information, Mako (http://www.makotemplates.org/) might be a bit heavyweight for embedding in cython (I don't know for sure; haven't looked at the source). One selling point, however, is that it is already used with cython code in scipy (see, e.g., https://github.com/scipy/scipy/blob/master/scipy/interpolate/interpnd.pyx). --Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From stefan_ml at behnel.de Fri Jul 22 19:04:27 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 22 Jul 2011 19:04:27 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E295A9E.6090800@behnel.de> <4E296509.6010704@astro.uio.no> <4E29754C.3010609@behnel.de> <4E297628.6000908@behnel.de> <4E299B53.1010604@astro.uio.no> <4E299BF3.2080800@astro.uio.no> <4E299F9C.4010906@behnel.de> Message-ID: <4E29AD9B.7030908@behnel.de> Hoyt Koepke, 22.07.2011 18:48: > I know I'm a little late to the discussion, but I've been meaning to > ask about a feature like this myself. It's not exactly a feature, rather an internal developer tool. That being said, once it's there, it can also form a basis for end users who want to do Cython code generation. To make this a feature, we could see if we can set up a suitable entry point somewhere, such as a template class that we preconfigure for Cython code. However, I guess it would be best to see the requirements for that emerge outside of Cython first. > As a point of information, Mako (http://www.makotemplates.org/) might > be a bit heavyweight for embedding in cython (I don't know for sure; > haven't looked at the source). One > selling point, however, is that it is already used with cython code in > scipy (see, > e.g., https://github.com/scipy/scipy/blob/master/scipy/interpolate/interpnd.pyx). Well, SciPy isn't a dependency of Cython either, so I think that doesn't count much. Stefan From d.s.seljebotn at astro.uio.no Fri Jul 22 20:02:19 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 22 Jul 2011 20:02:19 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E299F9C.4010906@behnel.de> References: <4E295A9E.6090800@behnel.de> <4E296509.6010704@astro.uio.no> <4E29754C.3010609@behnel.de> <4E297628.6000908@behnel.de> <4E299B53.1010604@astro.uio.no> <4E299BF3.2080800@astro.uio.no> <4E299F9C.4010906@behnel.de> Message-ID: <4E29BB2B.80606@astro.uio.no> On 07/22/2011 06:04 PM, Stefan Behnel wrote: > Dag Sverre Seljebotn, 22.07.2011 17:49: >> On 07/22/2011 05:46 PM, Dag Sverre Seljebotn wrote: >>> On 07/22/2011 03:07 PM, Stefan Behnel wrote: >>>> Stefan Behnel, 22.07.2011 15:04: >>>>> Dag Sverre Seljebotn, 22.07.2011 13:54: >>>>>> On 07/22/2011 01:10 PM, Stefan Behnel wrote: >>>>>>> I'm fine with using a template engine for the more involved cases >>>>>>> (which >>>>>>> are rare enough). However, I'd prefer not adding a new >>>>>>> dependency, but >>>>>>> just shipping a tiny single-module engine with Cython, e.g. >>>>>>> Templite or >>>>>>> pyratemp (just found them, never used them). >>>>>> >>>>>> BTW, I don't think anybody is suggesting having an *external* >>>>>> dependency >>>>>> that users would need to go fetch themselves....that would be insane. >>>>> >>>>> Sure. I also don't mind having a small engine in a package, but just >>>>> putting a single file in the source tree (and overwriting it with a >>>>> new >>>>> version at need) would be a perfectly small and nonintrusive addition. >>>>> And >>>>> given that there really are engines of that size (one file, <500 >>>>> lines), >>>>> it's not so unrealistic either. >>>>> >>>>> That being said, if you can recommend Tempita (which IMHO qualifies as >>>>> "small enough"), I won't object to it. >>>> >>>> ... although, isn't it unfortunate that it uses "{{...}}" for template >>>> code? How well readable is that when used in C code? >>> >>> At least in emacs, pretty much any template syntax will mess up things >>> pretty badly, especially automatic indentation and so on. >>> >>> What I do is using mmm-mode, so that everything within {{ and }} is >>> highlighted with python-mode, and the rest with c-mode. I'll share my >>> configuration later. >>> >>> I'm merely stating why this has never been an issue for me, I'm NOT >>> suggesting everyone should use emacs (or that plain text editing isn't >>> convenient); you do have a point. >>> >>> Keep in mind that a syntax must work well with utility code written in >>> Cython as well. >> >> OTOH one *might* want to have >> >> code.put_tempita(..) # or similar >> >> in Python code, and in that case plain text readability is very >> important. >> >> Of course, hacking Tempita to make it %[ ]% instead is a five minute job. >> But I'm happy with any other engine that somebody have actually tried out >> extensively. > > Tempita has a syntax for comments - what about using the regular C and > Cython comments as template syntax delimiters? That way, you'd write > > /*# some comment */ > > to get an actual comment in C code, and > > /* for x in y */ > > for template code. For Cython code, we could use "##", or maybe "#<" and > ">#". That should work pretty well with editors. I really like the idea of making use of comments, but I don't think we should "overload" the C comments; both for human parsing and because it is helpful with comments in generated Cython code as well (at least I would like to see much more of that!) For Cython code, this doesn't work well for inline expressions: variable## = 34 So I propose C: /*$ $*/ Cython: "$ $" Or some other character than $ that we can have in common between C and Cython (! is taken by documentation systems, but : is an option...). Using a string in Cython should play better with at least some syntax highlighters, and we can simply make sure to use ' for all actual strings. Dag Sverre From hoytak at stat.washington.edu Fri Jul 22 20:02:39 2011 From: hoytak at stat.washington.edu (Hoyt Koepke) Date: Fri, 22 Jul 2011 11:02:39 -0700 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E29AD9B.7030908@behnel.de> References: <4E295A9E.6090800@behnel.de> <4E296509.6010704@astro.uio.no> <4E29754C.3010609@behnel.de> <4E297628.6000908@behnel.de> <4E299B53.1010604@astro.uio.no> <4E299BF3.2080800@astro.uio.no> <4E299F9C.4010906@behnel.de> <4E29AD9B.7030908@behnel.de> Message-ID: > Hoyt Koepke, 22.07.2011 18:48: >> >> I know I'm a little late to the discussion, but I've been meaning to >> ask about a feature like this myself. > > It's not exactly a feature, rather an internal developer tool. Ah, I see; didn't read things closely enough. Sorry for the OT post then. > Well, SciPy isn't a dependency of Cython either, so I think that doesn't > count much. True; my point was just that it was a use case of the user-space templating feature idea; but, given that this OT for this thread, I definitely agree with you :-) -- Hoyt ++++++++++++++++++++++++++++++++++++++++++++++++ + Hoyt Koepke + University of Washington Department of Statistics + http://www.stat.washington.edu/~hoytak/ + hoytak at gmail.com ++++++++++++++++++++++++++++++++++++++++++ From d.s.seljebotn at astro.uio.no Fri Jul 22 20:06:50 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 22 Jul 2011 20:06:50 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E295A9E.6090800@behnel.de> <4E296509.6010704@astro.uio.no> <4E29754C.3010609@behnel.de> <4E297628.6000908@behnel.de> <4E299B53.1010604@astro.uio.no> <4E299BF3.2080800@astro.uio.no> <4E299F9C.4010906@behnel.de> Message-ID: <4E29BC3A.6060309@astro.uio.no> On 07/22/2011 06:48 PM, Hoyt Koepke wrote: > I know I'm a little late to the discussion, but I've been meaning to > ask about a feature like this myself. > > As a point of information, Mako (http://www.makotemplates.org/) might > be a bit heavyweight for embedding in cython (I don't know for sure; > haven't looked at the source). One > selling point, however, is that it is already used with cython code in > scipy (see, > e.g., https://github.com/scipy/scipy/blob/master/scipy/interpolate/interpnd.pyx). OT: I spoke with Pauli about this and he just picked it because it's one he already knew. If they need to start using the template system during the build they'd probably switch to something like Tempita. And in SciPy for .NET there's thousands of lines of Tempita code written (well, generated) by yours truly, they may eventually be merged back, at which point the volume of Tempita code in SciPy trumps the volume of Mako code. Dag Sverre From robertwb at math.washington.edu Fri Jul 22 22:05:12 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 22 Jul 2011 13:05:12 -0700 Subject: [Cython] Utility Codes and templates In-Reply-To: References: Message-ID: On Fri, Jul 22, 2011 at 3:12 AM, mark florisson wrote: > For my work on the _memview branch (and also on fused types) I noticed > that UtilityCodes started weighing heavily on me in their current > form, so I wrote a little loader in the _memview branch: > > https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 > > The idea is simple: you put your utility codes in Cython/Utility in > .pyx, .c, .h files etc, and then load them. It works for both > prototypes and implementations, for UtilityCode and CythonUtilityCode: This sounds like it could be a nice way to organize our UtilityCode snippets. So far we haven't really needed any more templating than simple substitution, but for what you're doing I can see this being quite handy. This may also provide a more flexible way forward for supporting multiple backends. > myutility.c > > // UtilityProto: MyUtility > header code here > > // UtilityCode: MyUtility > implementation code here > > You can add as many other utilities as you like to the same file. You > can then load it using > > ? ?UtilityCode.load_utility_from_file("myutility.c", "MyUtility") I agree with you that having multiple related, named snippets in same file is worthwhile. What about ////////////////////// MyUtility.proto /////////////////////////// and ############ MyCyUtility ############## so the chunks are easy to see. > Of course you can pass in any other arguments, like proto_block, name, > etc. You can additionally pass it a dict for formatting (for both the > prototypes and the implementation). It will return a UtilityCode > instance ready for use. > You can also simply retrieve a utility code as a string, where it > returns (proto, implementation). > > As debated before, an actual template library would be really > convenient. Dag and I had a discussion about it and he suggested > Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3, > and is pure-python. It supports all the good things like iteration, > template inheritance, etc. Now I'm not sure whether it supports python > 2.3 as it doesn't compile on my system, but it does support 2.4 > (confirmation for 2.3 would be appreciated). On a side note, I'd be > perfectly happy to drop support for 2.3, it's kind of a chore. > The documentation for Tempita can be found here: http://pythonpaste.org/tempita/ Looks like a reasonable choice to me, it's small enough to ship and looks versatile enough to do everything we need. There's certainly not a shortage of options, but I'm happy with your and Dag's endorsement. In terms of delimiters, I prefer some form of strings over comments (often a value is expected, and Python's comments always extend to the line break) but sticking with the {{ }} syntax appeals to me even more. It's pretty standard, and decent syntax hilighters shouldn't choke on it (well, the surrounding code) too bad. > That way we might rid ourselves of a lot of code.putln() and move > those to template utilities instead (or at least prevent writing more > of those). What do you guys think? +1 > BTW, I will refrain from moving any utility codes other than the ones > in MemoryView.py, CythonScope.py and Buffer.py, to avoid any possible > future conflicts when merging into mainline. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From markflorisson88 at gmail.com Fri Jul 22 22:39:30 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 22 Jul 2011 22:39:30 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: Message-ID: On 22 July 2011 22:05, Robert Bradshaw wrote: > On Fri, Jul 22, 2011 at 3:12 AM, mark florisson > wrote: >> For my work on the _memview branch (and also on fused types) I noticed >> that UtilityCodes started weighing heavily on me in their current >> form, so I wrote a little loader in the _memview branch: >> >> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >> >> The idea is simple: you put your utility codes in Cython/Utility in >> .pyx, .c, .h files etc, and then load them. It works for both >> prototypes and implementations, for UtilityCode and CythonUtilityCode: > > This sounds like it could be a nice way to organize our UtilityCode > snippets. So far we haven't really needed any more templating than > simple substitution, but for what you're doing I can see this being > quite handy. This may also provide a more flexible way forward for > supporting multiple backends. > >> myutility.c >> >> // UtilityProto: MyUtility >> header code here >> >> // UtilityCode: MyUtility >> implementation code here >> >> You can add as many other utilities as you like to the same file. You >> can then load it using >> >> ? ?UtilityCode.load_utility_from_file("myutility.c", "MyUtility") > > I agree with you that having multiple related, named snippets in same > file is worthwhile. What about > > ////////////////////// MyUtility.proto /////////////////////////// > > and > > ############ MyCyUtility ############## > > so the chunks are easy to see. Sounds like a good idea. How would we handle the requires list in that case though? Or should we leave that to the Python code that loads it for now? >> Of course you can pass in any other arguments, like proto_block, name, >> etc. You can additionally pass it a dict for formatting (for both the >> prototypes and the implementation). It will return a UtilityCode >> instance ready for use. >> You can also simply retrieve a utility code as a string, where it >> returns (proto, implementation). >> >> As debated before, an actual template library would be really >> convenient. Dag and I had a discussion about it and he suggested >> Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3, >> and is pure-python. It supports all the good things like iteration, >> template inheritance, etc. Now I'm not sure whether it supports python >> 2.3 as it doesn't compile on my system, but it does support 2.4 >> (confirmation for 2.3 would be appreciated). On a side note, I'd be >> perfectly happy to drop support for 2.3, it's kind of a chore. >> The documentation for Tempita can be found here: http://pythonpaste.org/tempita/ > > Looks like a reasonable choice to me, it's small enough to ship and > looks versatile enough to do everything we need. There's certainly not > a shortage of options, but I'm happy with your and Dag's endorsement. > > In terms of delimiters, I prefer some form of strings over comments > (often a value is expected, and Python's comments always extend to the > line break) but sticking with the {{ }} syntax appeals to me even > more. It's pretty standard, and decent syntax hilighters shouldn't > choke on it (well, the surrounding code) too bad. Ok, that's fine with me. It would be easily changeable if we revert later anyways. I'll add tempita to Cython. >> That way we might rid ourselves of a lot of code.putln() and move >> those to template utilities instead (or at least prevent writing more >> of those). What do you guys think? > > +1 > >> BTW, I will refrain from moving any utility codes other than the ones >> in MemoryView.py, CythonScope.py and Buffer.py, to avoid any possible >> future conflicts when merging into mainline. >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From robertwb at math.washington.edu Fri Jul 22 23:44:20 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 22 Jul 2011 14:44:20 -0700 Subject: [Cython] Utility Codes and templates In-Reply-To: References: Message-ID: On Fri, Jul 22, 2011 at 1:39 PM, mark florisson wrote: > On 22 July 2011 22:05, Robert Bradshaw wrote: >> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >> wrote: >>> For my work on the _memview branch (and also on fused types) I noticed >>> that UtilityCodes started weighing heavily on me in their current >>> form, so I wrote a little loader in the _memview branch: >>> >>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>> >>> The idea is simple: you put your utility codes in Cython/Utility in >>> .pyx, .c, .h files etc, and then load them. It works for both >>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >> >> This sounds like it could be a nice way to organize our UtilityCode >> snippets. So far we haven't really needed any more templating than >> simple substitution, but for what you're doing I can see this being >> quite handy. This may also provide a more flexible way forward for >> supporting multiple backends. >> >>> myutility.c >>> >>> // UtilityProto: MyUtility >>> header code here >>> >>> // UtilityCode: MyUtility >>> implementation code here >>> >>> You can add as many other utilities as you like to the same file. You >>> can then load it using >>> >>> ? ?UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >> >> I agree with you that having multiple related, named snippets in same >> file is worthwhile. What about >> >> ////////////////////// MyUtility.proto /////////////////////////// >> >> and >> >> ############ MyCyUtility ############## >> >> so the chunks are easy to see. > > Sounds like a good idea. How would we handle the requires list in that > case though? Or should we leave that to the Python code that loads it > for now? For now, lets leave them with the Python code. Eventually we could do something similar to how we do Cython pragmas, i.e. parse the comments that occur at the top of the file (chunk) as having special meaning. >>> Of course you can pass in any other arguments, like proto_block, name, >>> etc. You can additionally pass it a dict for formatting (for both the >>> prototypes and the implementation). It will return a UtilityCode >>> instance ready for use. >>> You can also simply retrieve a utility code as a string, where it >>> returns (proto, implementation). >>> >>> As debated before, an actual template library would be really >>> convenient. Dag and I had a discussion about it and he suggested >>> Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3, >>> and is pure-python. It supports all the good things like iteration, >>> template inheritance, etc. Now I'm not sure whether it supports python >>> 2.3 as it doesn't compile on my system, but it does support 2.4 >>> (confirmation for 2.3 would be appreciated). On a side note, I'd be >>> perfectly happy to drop support for 2.3, it's kind of a chore. >>> The documentation for Tempita can be found here: http://pythonpaste.org/tempita/ >> >> Looks like a reasonable choice to me, it's small enough to ship and >> looks versatile enough to do everything we need. There's certainly not >> a shortage of options, but I'm happy with your and Dag's endorsement. >> >> In terms of delimiters, I prefer some form of strings over comments >> (often a value is expected, and Python's comments always extend to the >> line break) but sticking with the {{ }} syntax appeals to me even >> more. It's pretty standard, and decent syntax hilighters shouldn't >> choke on it (well, the surrounding code) too bad. > > Ok, that's fine with me. It would be easily changeable if we revert > later anyways. Yep, which makes it a much easier decision than trying to figure out a templating solution for the user-visible front-end. > I'll add tempita to Cython. Sounds good. (If anyone else has objections, bring them up now...) - Robert From vitja.makarov at gmail.com Sat Jul 23 07:03:56 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sat, 23 Jul 2011 09:03:56 +0400 Subject: [Cython] Utility Codes and templates In-Reply-To: References: Message-ID: 2011/7/23 Robert Bradshaw : > On Fri, Jul 22, 2011 at 1:39 PM, mark florisson > wrote: >> On 22 July 2011 22:05, Robert Bradshaw wrote: >>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >>> wrote: >>>> For my work on the _memview branch (and also on fused types) I noticed >>>> that UtilityCodes started weighing heavily on me in their current >>>> form, so I wrote a little loader in the _memview branch: >>>> >>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>> >>>> The idea is simple: you put your utility codes in Cython/Utility in >>>> .pyx, .c, .h files etc, and then load them. It works for both >>>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>> >>> This sounds like it could be a nice way to organize our UtilityCode >>> snippets. So far we haven't really needed any more templating than >>> simple substitution, but for what you're doing I can see this being >>> quite handy. This may also provide a more flexible way forward for >>> supporting multiple backends. >>> >>>> myutility.c >>>> >>>> // UtilityProto: MyUtility >>>> header code here >>>> >>>> // UtilityCode: MyUtility >>>> implementation code here >>>> >>>> You can add as many other utilities as you like to the same file. You >>>> can then load it using >>>> >>>> ? ?UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>> >>> I agree with you that having multiple related, named snippets in same >>> file is worthwhile. What about >>> >>> ////////////////////// MyUtility.proto /////////////////////////// >>> >>> and >>> >>> ############ MyCyUtility ############## >>> >>> so the chunks are easy to see. >> >> Sounds like a good idea. How would we handle the requires list in that >> case though? Or should we leave that to the Python code that loads it >> for now? > > For now, lets leave them with the Python code. Eventually we could do > something similar to how we do Cython pragmas, i.e. parse the comments > that occur at the top of the file (chunk) as having special meaning. > >>>> Of course you can pass in any other arguments, like proto_block, name, >>>> etc. You can additionally pass it a dict for formatting (for both the >>>> prototypes and the implementation). It will return a UtilityCode >>>> instance ready for use. >>>> You can also simply retrieve a utility code as a string, where it >>>> returns (proto, implementation). >>>> >>>> As debated before, an actual template library would be really >>>> convenient. Dag and I had a discussion about it and he suggested >>>> Tempita (by Ian Bicking), it is compatible with Python 2 and Python 3, >>>> and is pure-python. It supports all the good things like iteration, >>>> template inheritance, etc. Now I'm not sure whether it supports python >>>> 2.3 as it doesn't compile on my system, but it does support 2.4 >>>> (confirmation for 2.3 would be appreciated). On a side note, I'd be >>>> perfectly happy to drop support for 2.3, it's kind of a chore. >>>> The documentation for Tempita can be found here: http://pythonpaste.org/tempita/ >>> >>> Looks like a reasonable choice to me, it's small enough to ship and >>> looks versatile enough to do everything we need. There's certainly not >>> a shortage of options, but I'm happy with your and Dag's endorsement. >>> >>> In terms of delimiters, I prefer some form of strings over comments >>> (often a value is expected, and Python's comments always extend to the >>> line break) but sticking with the {{ }} syntax appeals to me even >>> more. It's pretty standard, and decent syntax hilighters shouldn't >>> choke on it (well, the surrounding code) too bad. >> >> Ok, that's fine with me. It would be easily changeable if we revert >> later anyways. > > Yep, which makes it a much easier decision than trying to figure out a > templating solution for the user-visible front-end. > >> I'll add tempita to Cython. > > Sounds good. (If anyone else has objections, bring them up now...) > Tempita looks cool, +1 -- vitja. From stefan_ml at behnel.de Sat Jul 23 09:37:04 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 23 Jul 2011 09:37:04 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: Message-ID: <4E2A7A20.4010105@behnel.de> Robert Bradshaw, 22.07.2011 23:44: > On Fri, Jul 22, 2011 at 1:39 PM, mark florisson > wrote: >> On 22 July 2011 22:05, Robert Bradshaw wrote: >>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >>> wrote: >>>> For my work on the _memview branch (and also on fused types) I noticed >>>> that UtilityCodes started weighing heavily on me in their current >>>> form, so I wrote a little loader in the _memview branch: >>>> >>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>> >>>> The idea is simple: you put your utility codes in Cython/Utility in >>>> .pyx, .c, .h files etc, and then load them. It works for both >>>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>> >>> This sounds like it could be a nice way to organize our UtilityCode >>> snippets. So far we haven't really needed any more templating than >>> simple substitution, but for what you're doing I can see this being >>> quite handy. This may also provide a more flexible way forward for >>> supporting multiple backends. It's unlikely to be useful for the ctypes backend, but C/C++ will benefit from it, and so may others. >>>> myutility.c >>>> >>>> // UtilityProto: MyUtility >>>> header code here >>>> >>>> // UtilityCode: MyUtility >>>> implementation code here >>>> >>>> You can add as many other utilities as you like to the same file. You >>>> can then load it using >>>> >>>> UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>> >>> I agree with you that having multiple related, named snippets in same >>> file is worthwhile. What about >>> >>> ////////////////////// MyUtility.proto /////////////////////////// >>> >>> and >>> >>> ############ MyCyUtility ############## Hmm, right. Cython utility code doesn't actually need any separate parts. I hadn't realised that. +1, the above is well visible and doesn't hurt the file syntax. >> Sounds like a good idea. How would we handle the requires list in that >> case though? Or should we leave that to the Python code that loads it >> for now? > > For now, lets leave them with the Python code. -1, I see no reason to do it the wrong way now only to clean it up later. Why not just do it right? It's not like that's so much more work. I can help out. > Eventually we could do > something similar to how we do Cython pragmas, i.e. parse the comments > that occur at the top of the file (chunk) as having special meaning. Requiring a short declaration at the top of the file makes it easier to find out what's in that file from a quick look at the top, rather than having to skip (or grep) through it. Yes, I think it makes sense to put this into some kind of file header. Coming back to my ini file proposal, what about a file syntax like this: /* [MyUtil1] requires = file1:abc [MyUtil2] ; nothing to declare ... (I think it's ; for comments - don't care) */ ///////////////// MyUtil1.proto ///////////////////// ... and the same for Cython code: """ [MyUtil1] requires = file1:abc [MyUtil2] requires = MyUtil1 """ ################### MyUtil1 ##################### ... The initial section is trivial to extract and to drop into RawConfigParser, and we are completely free to extend the metadata section later on, without having to touch the code as well. >>> In terms of delimiters, I prefer some form of strings over comments >>> (often a value is expected, and Python's comments always extend to the >>> line break) Yes, it's clearly a problem in Cython code. It would work very nicely for C code, though. I think we should use it there right from the start. >>> but sticking with the {{ }} syntax appeals to me even >>> more. It's pretty standard, and decent syntax hilighters shouldn't >>> choke on it (well, the surrounding code) too bad. >> >> Ok, that's fine with me. It would be easily changeable if we revert >> later anyways. > > Yep, which makes it a much easier decision than trying to figure out a > templating solution for the user-visible front-end. I'm fine with {{ }} for Cython code. It's just as good as anything else. >> I'll add tempita to Cython. > > Sounds good. (If anyone else has objections, bring them up now...) Tempita is fine with me. Stefan From markflorisson88 at gmail.com Sat Jul 23 10:03:16 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Sat, 23 Jul 2011 10:03:16 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E2A7A20.4010105@behnel.de> References: <4E2A7A20.4010105@behnel.de> Message-ID: On 23 July 2011 09:37, Stefan Behnel wrote: > Robert Bradshaw, 22.07.2011 23:44: >> >> On Fri, Jul 22, 2011 at 1:39 PM, mark florisson >> ?wrote: >>> >>> On 22 July 2011 22:05, Robert Bradshaw >>> ?wrote: >>>> >>>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >>>> ?wrote: >>>>> >>>>> For my work on the _memview branch (and also on fused types) I noticed >>>>> that UtilityCodes started weighing heavily on me in their current >>>>> form, so I wrote a little loader in the _memview branch: >>>>> >>>>> >>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>>> >>>>> The idea is simple: you put your utility codes in Cython/Utility in >>>>> .pyx, .c, .h files etc, and then load them. It works for both >>>>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>>> >>>> This sounds like it could be a nice way to organize our UtilityCode >>>> snippets. So far we haven't really needed any more templating than >>>> simple substitution, but for what you're doing I can see this being >>>> quite handy. This may also provide a more flexible way forward for >>>> supporting multiple backends. > > It's unlikely to be useful for the ctypes backend, but C/C++ will benefit > from it, and so may others. > > >>>>> myutility.c >>>>> >>>>> // UtilityProto: MyUtility >>>>> header code here >>>>> >>>>> // UtilityCode: MyUtility >>>>> implementation code here >>>>> >>>>> You can add as many other utilities as you like to the same file. You >>>>> can then load it using >>>>> >>>>> ? ?UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>>> >>>> I agree with you that having multiple related, named snippets in same >>>> file is worthwhile. What about >>>> >>>> ////////////////////// MyUtility.proto /////////////////////////// >>>> >>>> and >>>> >>>> ############ MyCyUtility ############## > > Hmm, right. Cython utility code doesn't actually need any separate parts. I > hadn't realised that. > > +1, the above is well visible and doesn't hurt the file syntax. > > >>> Sounds like a good idea. How would we handle the requires list in that >>> case though? Or should we leave that to the Python code that loads it >>> for now? >> >> For now, lets leave them with the Python code. > > -1, I see no reason to do it the wrong way now only to clean it up later. > Why not just do it right? It's not like that's so much more work. I can help > out. > > >> Eventually we could do >> something similar to how we do Cython pragmas, i.e. parse the comments >> that occur at the top of the file (chunk) as having special meaning. > > Requiring a short declaration at the top of the file makes it easier to find > out what's in that file from a quick look at the top, rather than having to > skip (or grep) through it. Yes, I think it makes sense to put this into some > kind of file header. > > Coming back to my ini file proposal, what about a file syntax like this: > > > /* > [MyUtil1] > requires = file1:abc > > [MyUtil2] > ; nothing to declare ... ?(I think it's ; for comments - don't care) > */ > > ///////////////// MyUtil1.proto ///////////////////// > ... > > > and the same for Cython code: > > """ > [MyUtil1] > requires = file1:abc > > [MyUtil2] > requires = MyUtil1 > """ > > ################### MyUtil1 ##################### > ... > > The initial section is trivial to extract and to drop into RawConfigParser, > and we are completely free to extend the metadata section later on, without > having to touch the code as well. > Looks good to me. Up to now you could also use comments at the top but they were just dropped. >>>> In terms of delimiters, I prefer some form of strings over comments >>>> (often a value is expected, and Python's comments always extend to the >>>> line break) > > Yes, it's clearly a problem in Cython code. It would work very nicely for C > code, though. I think we should use it there right from the start. > I concur that the {{ }} can show up like an ugly red in some situations. I do think comments are nicer. However for Cython templates you'd have to use triple-quoted strings in order to get a benefit (I assume you occasionally want something that stretches multiple lines). >>>> but sticking with the {{ }} syntax appeals to me even >>>> more. It's pretty standard, and decent syntax hilighters shouldn't >>>> choke on it (well, the surrounding code) too bad. >>> >>> Ok, that's fine with me. It would be easily changeable if we revert >>> later anyways. >> >> Yep, which makes it a much easier decision than trying to figure out a >> templating solution for the user-visible front-end. > > I'm fine with {{ }} for Cython code. It's just as good as anything else. > > >>> I'll add tempita to Cython. >> >> Sounds good. (If anyone else has objections, bring them up now...) > > Tempita is fine with me. > > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Sat Jul 23 11:47:07 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 23 Jul 2011 11:47:07 +0200 Subject: [Cython] Transformation of pxds In-Reply-To: <20110723045244.GA7492@ubuntu> References: <20110723045244.GA7492@ubuntu> Message-ID: <4E2A989B.4020103@behnel.de> Hi Romain, CC-ing cython-devel here, as this is not an entirely trivial question to answer. Romain Guillebert, 23.07.2011 06:52: > During this week, I've been trying to find a way to implement cimport, I > think that the cleanest way of implementing this is to compile pxds just > like pyxs are compiled. You are aware that there are two types of .pxd files? One next to a .pyx or .py file with the same name, which provides additional declarations for that source file and can also serve as a cimportable .pxd for other modules, thus providing a C-level API between modules, and a second type that only contains external declarations (well, and potentially inline functions etc.), usually for external C libraries or header files, but that does not correspond to a .pyx/.py source file. The idea of materialising the external .pxd files does sound appealing. I imagine that this means that the resulting .py file would basically just set up ctypes wrappers for external declarations and store them under the corresponding name, right? Then a cimport from an external .pxd file would turn into a regular import. Plus, you'd end up with a single place where the external library (if any, or potentially many libraries) is loaded and managed. Cool. Even the first kind of .pxd file would fit into this quite nicely. Here, the corresponding .pyx/.py file would be enough to represent the module, and the exported declarations in the .pxd file would just be ignored. Consequently, the .pxd file would only be used to provide declarations for the module itself and serve no other purpose. Thus, in both cases, there would simply be an importable .py file for each cimported .pxd file, be it user provided or generated. > I'm having trouble knowing where to implement that though. > > I think that I will implement that in the method process_pxd of Context > in Cython/Compiler/Main.py (and it will probably call compile_single) > but I don't know if it's a good design. > > What do you think ? I think it could be done on the fly, when cimporting the .pxd file. Or maybe it's better to just queue up the cimported files that need to get compiled, and then compile them after compiling the main source file. For the client side of the cimport mechanism, parsing the .pxd file is enough to figure out what it will eventually export, so there is no actual need to generate a .py file for it at the same time. I think the compilation will be a separate operation anyway, because the .pxd file is currently parsed into the context of the .pyx file being compiled. Hacking an additional compilation onto that step may not be trivial. I don't mind either way, i.e. if it's done while cimporting the .pxd or afterwards. In any case, this means that compiling a .pyx file can potentially generate multiple .py files, which means that there should be a way to reuse already generated files from other compiler runs (i.e. if you compile multiple .pyx files which cimport the same .pxd files, then each .pxd file only needs to get compiled once). Basically, this means that the compiler would check if there is already a .py file for a .pxd, and not generate it in that case. However, this is problematic, because we do not necessarily know if the .py file was user provided or generated... This makes me think that the ctypes backend may generally want to generate its output files into a separate target *directory*, rather than just providing a target file name. Stefan From wonjunchoi001 at gmail.com Mon Jul 25 07:21:08 2011 From: wonjunchoi001 at gmail.com (=?EUC-KR?B?w9a/+MHY?=) Date: Mon, 25 Jul 2011 14:21:08 +0900 Subject: [Cython] Cython 0.15 release In-Reply-To: <4E28FBA2.9020005@behnel.de> References: <4E28FBA2.9020005@behnel.de> Message-ID: Can I test the generators? 2011/7/22 Stefan Behnel > Robert Bradshaw, 22.07.2011 01:00: > > Are the lxml failures at >> https://sage.math.washington.**edu:8091/hudson/job/cython-** >> devel-lxml-trunk-py27/ >> expected? >> > > Yes, the "test_xmlschema_import_file" test keeps crashing with a high > probability. Some kind of threading related bug, no idea how to reproduce it > in a debuggable way... > > Just ignore those two jobs for the time being. I keep looking at the tests > from time to time, because it's not *always* that bug that kills them. > > Stefan > > ______________________________**_________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/**mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitja.makarov at gmail.com Mon Jul 25 07:41:27 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 25 Jul 2011 09:41:27 +0400 Subject: [Cython] Cython 0.15 release In-Reply-To: References: <4E28FBA2.9020005@behnel.de> Message-ID: 2011/7/25 ??? : > Can I test the generators? > > Sure, generators support is in master already: https://github.com/cython/cython -- vitja. From wonjunchoi001 at gmail.com Mon Jul 25 08:03:01 2011 From: wonjunchoi001 at gmail.com (=?EUC-KR?B?w9a/+MHY?=) Date: Mon, 25 Jul 2011 15:03:01 +0900 Subject: [Cython] Cython 0.15 release In-Reply-To: References: <4E28FBA2.9020005@behnel.de> Message-ID: I don't know where it is.. sorry 2011/7/25 Vitja Makarov > 2011/7/25 ??? : > > Can I test the generators? > > > > > > Sure, generators support is in master already: > > https://github.com/cython/cython > > -- > vitja. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Mon Jul 25 08:15:06 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 25 Jul 2011 08:15:06 +0200 Subject: [Cython] Cython 0.15 release In-Reply-To: References: <4E28FBA2.9020005@behnel.de> Message-ID: <4E2D09EA.9030507@behnel.de> ???, 25.07.2011 08:03: > 2011/7/25 Vitja Makarov >> 2011/7/25 ???: >>> Can I test the generators? >> >> Sure, generators support is in master already: >> >> https://github.com/cython/cython > > I don't know where it is.. sorry It's actually not that hard. You go to that URL, click on the big fat "Downloads" button, and then select the archive format you prefer. Next, you unpack it on your machine, change into the extracted directory and run "python setup.py install". Besides, this is a question for the cython-users mailing list, not the core developers mailing list. Stefan From wonjunchoi001 at gmail.com Mon Jul 25 08:17:27 2011 From: wonjunchoi001 at gmail.com (=?EUC-KR?B?w9a/+MHY?=) Date: Mon, 25 Jul 2011 15:17:27 +0900 Subject: [Cython] Cython 0.15 release In-Reply-To: <4E2D09EA.9030507@behnel.de> References: <4E28FBA2.9020005@behnel.de> <4E2D09EA.9030507@behnel.de> Message-ID: ok 2011/7/25 Stefan Behnel > ???, 25.07.2011 08:03: > >> 2011/7/25 Vitja Makarov >> >>> 2011/7/25 ???: >>> >>> Can I test the generators? >>>> >>> >>> Sure, generators support is in master already: >>> >>> https://github.com/cython/**cython >>> >> >> I don't know where it is.. sorry >> > > It's actually not that hard. You go to that URL, click on the big fat > "Downloads" button, and then select the archive format you prefer. Next, you > unpack it on your machine, change into the extracted directory and run > "python setup.py install". > > Besides, this is a question for the cython-users mailing list, not the core > developers mailing list. > > Stefan > > ______________________________**_________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/**mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vitja.makarov at gmail.com Mon Jul 25 08:32:07 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 25 Jul 2011 10:32:07 +0400 Subject: [Cython] Build failed in Jenkins: cython-devel-sdist #522 In-Reply-To: <1235336665.281311574059433.JavaMail.scoder@sage.math.washington.edu> References: <1032484410.271311573862438.JavaMail.scoder@sage.math.washington.edu> <1235336665.281311574059433.JavaMail.scoder@sage.math.washington.edu> Message-ID: 2011/7/25 Hudson on sage-math : > See > > ------------------------------------------ > Started by user scoder > Building on master > Checkout:workspace / - hudson.remoting.LocalChannel at 5aa4a4a9 > Using strategy: Default > Last Built Revision: Revision 6af313c91dddad29ac5b69e394c6a60031724b05 (origin/master) > Checkout:workspace / - hudson.remoting.LocalChannel at 5aa4a4a9 > Fetching changes from the remote Git repository > Fetching upstream changes from git://github.com/cython/cython > Commencing build of Revision 6af313c91dddad29ac5b69e394c6a60031724b05 (origin/master) > Checking out Revision 6af313c91dddad29ac5b69e394c6a60031724b05 (origin/master) > FATAL: Unable to produce a script file > hudson.util.IOException2: Failed to create a temp file on > ? ? ? ?at hudson.FilePath.createTextTempFile(FilePath.java:966) > ? ? ? ?at hudson.tasks.CommandInterpreter.createScriptFile(CommandInterpreter.java:104) > ? ? ? ?at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) > ? ? ? ?at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58) > ? ? ? ?at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) > ? ? ? ?at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:664) > ? ? ? ?at hudson.model.Build$RunnerImpl.build(Build.java:177) > ? ? ? ?at hudson.model.Build$RunnerImpl.doRun(Build.java:139) > ? ? ? ?at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:430) > ? ? ? ?at hudson.model.Run.run(Run.java:1376) > ? ? ? ?at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) > ? ? ? ?at hudson.model.ResourceController.execute(ResourceController.java:88) > ? ? ? ?at hudson.model.Executor.run(Executor.java:175) > Caused by: java.io.IOException: No space left on device > ? ? ? ?at java.io.FileOutputStream.writeBytes(Native Method) > ? ? ? ?at java.io.FileOutputStream.write(FileOutputStream.java:260) > ? ? ? ?at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202) > ? ? ? ?at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297) > ? ? ? ?at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130) > ? ? ? ?at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216) > ? ? ? ?at hudson.FilePath$12.invoke(FilePath.java:960) > ? ? ? ?at hudson.FilePath$12.invoke(FilePath.java:944) > ? ? ? ?at hudson.FilePath.act(FilePath.java:758) > ? ? ? ?at hudson.FilePath.act(FilePath.java:740) > ? ? ? ?at hudson.FilePath.createTextTempFile(FilePath.java:944) > ? ? ? ?... 12 more > Build step 'Execute shell' marked build as failure > Archiving artifacts > Recording fingerprints > > It seems like hudson server is running out of space. -- vitja. From stefan_ml at behnel.de Mon Jul 25 08:39:51 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 25 Jul 2011 08:39:51 +0200 Subject: [Cython] Build failed in Jenkins: cython-devel-sdist #522 In-Reply-To: References: <1032484410.271311573862438.JavaMail.scoder@sage.math.washington.edu> <1235336665.281311574059433.JavaMail.scoder@sage.math.washington.edu> Message-ID: <4E2D0FB7.3040705@behnel.de> Vitja Makarov, 25.07.2011 08:32: > It seems like hudson server is running out of space. It's actually the /tmp directory that's full here. Contains some 65GB from other jobs that are running on the machine. I wrote a message to the cluster users mailing list to have someone with appropriate rights clean it up. Stefan From vitja.makarov at gmail.com Mon Jul 25 08:41:45 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 25 Jul 2011 10:41:45 +0400 Subject: [Cython] Utility Codes and templates In-Reply-To: References: Message-ID: 2011/7/23 Robert Bradshaw : > On Fri, Jul 22, 2011 at 3:12 AM, mark florisson > wrote: >> For my work on the _memview branch (and also on fused types) I noticed >> that UtilityCodes started weighing heavily on me in their current >> form, so I wrote a little loader in the _memview branch: >> >> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >> >> The idea is simple: you put your utility codes in Cython/Utility in >> .pyx, .c, .h files etc, and then load them. It works for both >> prototypes and implementations, for UtilityCode and CythonUtilityCode: > > This sounds like it could be a nice way to organize our UtilityCode > snippets. So far we haven't really needed any more templating than > simple substitution, but for what you're doing I can see this being > quite handy. This may also provide a more flexible way forward for > supporting multiple backends. > >> myutility.c >> >> // UtilityProto: MyUtility >> header code here >> >> // UtilityCode: MyUtility >> implementation code here >> >> You can add as many other utilities as you like to the same file. You >> can then load it using >> >> ? ?UtilityCode.load_utility_from_file("myutility.c", "MyUtility") > > I agree with you that having multiple related, named snippets in same > file is worthwhile. What about > > ////////////////////// MyUtility.proto /////////////////////////// > > and > > ############ MyCyUtility ############## > > so the chunks are easy to see. > C++ comments looks ugly. May be it's better to have something like this: /* UtilityCode: MyUtility.proto */ and # UtilityCode: MyCyUtility That's also pretty easy to parse -- vitja. From stefan_ml at behnel.de Mon Jul 25 08:46:35 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 25 Jul 2011 08:46:35 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: Message-ID: <4E2D114B.1000509@behnel.de> Vitja Makarov, 25.07.2011 08:41: > 2011/7/23 Robert Bradshaw: >> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >> wrote: >>> For my work on the _memview branch (and also on fused types) I noticed >>> that UtilityCodes started weighing heavily on me in their current >>> form, so I wrote a little loader in the _memview branch: >>> >>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>> >>> The idea is simple: you put your utility codes in Cython/Utility in >>> .pyx, .c, .h files etc, and then load them. It works for both >>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >> >> This sounds like it could be a nice way to organize our UtilityCode >> snippets. So far we haven't really needed any more templating than >> simple substitution, but for what you're doing I can see this being >> quite handy. This may also provide a more flexible way forward for >> supporting multiple backends. >> >>> myutility.c >>> >>> // UtilityProto: MyUtility >>> header code here >>> >>> // UtilityCode: MyUtility >>> implementation code here >>> >>> You can add as many other utilities as you like to the same file. You >>> can then load it using >>> >>> UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >> >> I agree with you that having multiple related, named snippets in same >> file is worthwhile. What about >> >> ////////////////////// MyUtility.proto /////////////////////////// >> >> and >> >> ############ MyCyUtility ############## >> >> so the chunks are easy to see. >> > > C++ comments looks ugly. May be it's better to have something like this: > > /* UtilityCode: MyUtility.proto */ > and > > # UtilityCode: MyCyUtility > > That's also pretty easy to parse For the parser, it makes no difference. For a human, it does. A big fat marker like "/////////////////////" is hard to miss, whereas a tiny one like "/* ... */" is easily overlooked within a longer piece of code. Stefan From vitja.makarov at gmail.com Mon Jul 25 10:25:12 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Mon, 25 Jul 2011 12:25:12 +0400 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E2D114B.1000509@behnel.de> References: <4E2D114B.1000509@behnel.de> Message-ID: 2011/7/25 Stefan Behnel : > Vitja Makarov, 25.07.2011 08:41: >> >> 2011/7/23 Robert Bradshaw: >>> >>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >>> ?wrote: >>>> >>>> For my work on the _memview branch (and also on fused types) I noticed >>>> that UtilityCodes started weighing heavily on me in their current >>>> form, so I wrote a little loader in the _memview branch: >>>> >>>> >>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>> >>>> The idea is simple: you put your utility codes in Cython/Utility in >>>> .pyx, .c, .h files etc, and then load them. It works for both >>>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>> >>> This sounds like it could be a nice way to organize our UtilityCode >>> snippets. So far we haven't really needed any more templating than >>> simple substitution, but for what you're doing I can see this being >>> quite handy. This may also provide a more flexible way forward for >>> supporting multiple backends. >>> >>>> myutility.c >>>> >>>> // UtilityProto: MyUtility >>>> header code here >>>> >>>> // UtilityCode: MyUtility >>>> implementation code here >>>> >>>> You can add as many other utilities as you like to the same file. You >>>> can then load it using >>>> >>>> ? ?UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>> >>> I agree with you that having multiple related, named snippets in same >>> file is worthwhile. What about >>> >>> ////////////////////// MyUtility.proto /////////////////////////// >>> >>> and >>> >>> ############ MyCyUtility ############## >>> >>> so the chunks are easy to see. >>> >> >> C++ comments looks ugly. May be it's better to have something like this: >> >> /* UtilityCode: ?MyUtility.proto */ >> and >> >> # UtilityCode: MyCyUtility >> >> That's also pretty easy to parse > > For the parser, it makes no difference. For a human, it does. A big fat > marker like "/////////////////////" is hard to miss, whereas a tiny one like > "/* ... */" is easily overlooked within a longer piece of code. > Ok, but how many slashes should be there? I'd prefer something like: ########## UtilityCode: MyUtility ################ For both C and Cython -- vitja. From stefan_ml at behnel.de Mon Jul 25 12:00:38 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 25 Jul 2011 12:00:38 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E2D114B.1000509@behnel.de> Message-ID: <4E2D3EC6.5060801@behnel.de> Vitja Makarov, 25.07.2011 10:25: > 2011/7/25 Stefan Behnel: >> Vitja Makarov, 25.07.2011 08:41: >>> >>> 2011/7/23 Robert Bradshaw: >>>> >>>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >>>> wrote: >>>>> >>>>> For my work on the _memview branch (and also on fused types) I noticed >>>>> that UtilityCodes started weighing heavily on me in their current >>>>> form, so I wrote a little loader in the _memview branch: >>>>> >>>>> >>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>>> >>>>> The idea is simple: you put your utility codes in Cython/Utility in >>>>> .pyx, .c, .h files etc, and then load them. It works for both >>>>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>>> >>>> This sounds like it could be a nice way to organize our UtilityCode >>>> snippets. So far we haven't really needed any more templating than >>>> simple substitution, but for what you're doing I can see this being >>>> quite handy. This may also provide a more flexible way forward for >>>> supporting multiple backends. >>>> >>>>> myutility.c >>>>> >>>>> // UtilityProto: MyUtility >>>>> header code here >>>>> >>>>> // UtilityCode: MyUtility >>>>> implementation code here >>>>> >>>>> You can add as many other utilities as you like to the same file. You >>>>> can then load it using >>>>> >>>>> UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>>> >>>> I agree with you that having multiple related, named snippets in same >>>> file is worthwhile. What about >>>> >>>> ////////////////////// MyUtility.proto /////////////////////////// >>>> >>>> and >>>> >>>> ############ MyCyUtility ############## >>>> >>>> so the chunks are easy to see. >>>> >>> >>> C++ comments looks ugly. May be it's better to have something like this: >>> >>> /* UtilityCode: MyUtility.proto */ >>> and >>> >>> # UtilityCode: MyCyUtility >>> >>> That's also pretty easy to parse >> >> For the parser, it makes no difference. For a human, it does. A big fat >> marker like "/////////////////////" is hard to miss, whereas a tiny one like >> "/* ... */" is easily overlooked within a longer piece of code. >> > > Ok, but how many slashes should be there? > > I'd prefer something like: > > ########## UtilityCode: MyUtility ################ > > For both C and Cython I don't think the parser should care, and I don't think we should put much thought into this kind of detail, either. I'd suggest making the separator lines at most 78 characters long, and putting as many slashes or hashes in there as are needed to fill up the lines after writing the text content. What's important to the parser is that there's more than one hash/slash at start and end, and something parsable in between. The rest is up to those who write the file. Stefan From markflorisson88 at gmail.com Mon Jul 25 12:07:01 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Mon, 25 Jul 2011 12:07:01 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E2D3EC6.5060801@behnel.de> References: <4E2D114B.1000509@behnel.de> <4E2D3EC6.5060801@behnel.de> Message-ID: On 25 July 2011 12:00, Stefan Behnel wrote: > Vitja Makarov, 25.07.2011 10:25: >> >> 2011/7/25 Stefan Behnel: >>> >>> Vitja Makarov, 25.07.2011 08:41: >>>> >>>> 2011/7/23 Robert Bradshaw: >>>>> >>>>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >>>>> ? ?wrote: >>>>>> >>>>>> For my work on the _memview branch (and also on fused types) I noticed >>>>>> that UtilityCodes started weighing heavily on me in their current >>>>>> form, so I wrote a little loader in the _memview branch: >>>>>> >>>>>> >>>>>> >>>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>>>> >>>>>> The idea is simple: you put your utility codes in Cython/Utility in >>>>>> .pyx, .c, .h files etc, and then load them. It works for both >>>>>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>>>> >>>>> This sounds like it could be a nice way to organize our UtilityCode >>>>> snippets. So far we haven't really needed any more templating than >>>>> simple substitution, but for what you're doing I can see this being >>>>> quite handy. This may also provide a more flexible way forward for >>>>> supporting multiple backends. >>>>> >>>>>> myutility.c >>>>>> >>>>>> // UtilityProto: MyUtility >>>>>> header code here >>>>>> >>>>>> // UtilityCode: MyUtility >>>>>> implementation code here >>>>>> >>>>>> You can add as many other utilities as you like to the same file. You >>>>>> can then load it using >>>>>> >>>>>> ? ?UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>>>> >>>>> I agree with you that having multiple related, named snippets in same >>>>> file is worthwhile. What about >>>>> >>>>> ////////////////////// MyUtility.proto /////////////////////////// >>>>> >>>>> and >>>>> >>>>> ############ MyCyUtility ############## >>>>> >>>>> so the chunks are easy to see. >>>>> >>>> >>>> C++ comments looks ugly. May be it's better to have something like this: >>>> >>>> /* UtilityCode: ?MyUtility.proto */ >>>> and >>>> >>>> # UtilityCode: MyCyUtility >>>> >>>> That's also pretty easy to parse >>> >>> For the parser, it makes no difference. For a human, it does. A big fat >>> marker like "/////////////////////" is hard to miss, whereas a tiny one >>> like >>> "/* ... */" is easily overlooked within a longer piece of code. >>> >> >> Ok, but how many slashes should be there? >> >> I'd prefer something like: >> >> ########## UtilityCode: MyUtility ################ >> >> For both C and Cython > > I don't think the parser should care, and I don't think we should put much > thought into this kind of detail, either. I'd suggest making the separator > lines at most 78 characters long, and putting as many slashes or hashes in > there as are needed to fill up the lines after writing the text content. > What's important to the parser is that there's more than one hash/slash at > start and end, and something parsable in between. The rest is up to those > who write the file. Yeah, I simply mandated a minimum of 5 and a maximum of 30 on either side (symmetry is not enforced). It's now 'MyUtility' and 'MyUtility.proto'. If there's no objection to the ini-style header (with requirements and other metadata possibly), then I'll implement that one of these days. > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From romain.py at gmail.com Tue Jul 26 04:10:13 2011 From: romain.py at gmail.com (Romain Guillebert) Date: Tue, 26 Jul 2011 04:10:13 +0200 Subject: [Cython] Transformation of pxds In-Reply-To: <4E2A989B.4020103@behnel.de> References: <20110723045244.GA7492@ubuntu> <4E2A989B.4020103@behnel.de> Message-ID: <20110726021013.GA1447@ubuntu> Hi I can now compile pxd files, but I have encountered something rather strange : AnalyseExpressionsTransform turns : cimport foo def test(): foo.printf() into : cimport foo def test(): printf() It is due to the method analyse_as_cimported_attribute of AttributeNode in the file Cython/Compiler/ExprNodes.py Is this useful (since I don't think it has anything to do with expression analysis) and do you have an idea on how I can solve this (my first idea is : extend the class to disable the effect of this method but it's more a hack than anything else) ? Cheers Romain From robertwb at math.washington.edu Tue Jul 26 06:29:34 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 25 Jul 2011 21:29:34 -0700 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E2D114B.1000509@behnel.de> <4E2D3EC6.5060801@behnel.de> Message-ID: On Mon, Jul 25, 2011 at 3:07 AM, mark florisson wrote: > On 25 July 2011 12:00, Stefan Behnel wrote: >> Vitja Makarov, 25.07.2011 10:25: >>> >>> 2011/7/25 Stefan Behnel: >>>> >>>> Vitja Makarov, 25.07.2011 08:41: >>>>> >>>>> 2011/7/23 Robert Bradshaw: >>>>>> >>>>>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >>>>>> ? ?wrote: >>>>>>> >>>>>>> For my work on the _memview branch (and also on fused types) I noticed >>>>>>> that UtilityCodes started weighing heavily on me in their current >>>>>>> form, so I wrote a little loader in the _memview branch: >>>>>>> >>>>>>> >>>>>>> >>>>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>>>>> >>>>>>> The idea is simple: you put your utility codes in Cython/Utility in >>>>>>> .pyx, .c, .h files etc, and then load them. It works for both >>>>>>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>>>>> >>>>>> This sounds like it could be a nice way to organize our UtilityCode >>>>>> snippets. So far we haven't really needed any more templating than >>>>>> simple substitution, but for what you're doing I can see this being >>>>>> quite handy. This may also provide a more flexible way forward for >>>>>> supporting multiple backends. >>>>>> >>>>>>> myutility.c >>>>>>> >>>>>>> // UtilityProto: MyUtility >>>>>>> header code here >>>>>>> >>>>>>> // UtilityCode: MyUtility >>>>>>> implementation code here >>>>>>> >>>>>>> You can add as many other utilities as you like to the same file. You >>>>>>> can then load it using >>>>>>> >>>>>>> ? ?UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>>>>> >>>>>> I agree with you that having multiple related, named snippets in same >>>>>> file is worthwhile. What about >>>>>> >>>>>> ////////////////////// MyUtility.proto /////////////////////////// >>>>>> >>>>>> and >>>>>> >>>>>> ############ MyCyUtility ############## >>>>>> >>>>>> so the chunks are easy to see. >>>>>> >>>>> >>>>> C++ comments looks ugly. May be it's better to have something like this: >>>>> >>>>> /* UtilityCode: ?MyUtility.proto */ >>>>> and >>>>> >>>>> # UtilityCode: MyCyUtility >>>>> >>>>> That's also pretty easy to parse >>>> >>>> For the parser, it makes no difference. For a human, it does. A big fat >>>> marker like "/////////////////////" is hard to miss, whereas a tiny one >>>> like >>>> "/* ... */" is easily overlooked within a longer piece of code. >>>> >>> >>> Ok, but how many slashes should be there? >>> >>> I'd prefer something like: >>> >>> ########## UtilityCode: MyUtility ################ >>> >>> For both C and Cython >> >> I don't think the parser should care, and I don't think we should put much >> thought into this kind of detail, either. I'd suggest making the separator >> lines at most 78 characters long, and putting as many slashes or hashes in >> there as are needed to fill up the lines after writing the text content. >> What's important to the parser is that there's more than one hash/slash at >> start and end, and something parsable in between. The rest is up to those >> who write the file. > > Yeah, I simply mandated a minimum of 5 and a maximum of 30 on either > side (symmetry is not enforced). I don't even think we need a max. > It's now 'MyUtility' and > 'MyUtility.proto'. If there's no objection to the ini-style header > (with requirements and other metadata possibly), then I'll implement > that one of these days. One drawback with the ini style is that it detaches the metadata from the code block itself (and require duplicating their names). - Robert From vitja.makarov at gmail.com Tue Jul 26 06:48:10 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Tue, 26 Jul 2011 08:48:10 +0400 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E2D114B.1000509@behnel.de> <4E2D3EC6.5060801@behnel.de> Message-ID: 2011/7/26 Robert Bradshaw : > On Mon, Jul 25, 2011 at 3:07 AM, mark florisson > wrote: >> On 25 July 2011 12:00, Stefan Behnel wrote: >>> Vitja Makarov, 25.07.2011 10:25: >>>> >>>> 2011/7/25 Stefan Behnel: >>>>> >>>>> Vitja Makarov, 25.07.2011 08:41: >>>>>> >>>>>> 2011/7/23 Robert Bradshaw: >>>>>>> >>>>>>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >>>>>>> ? ?wrote: >>>>>>>> >>>>>>>> For my work on the _memview branch (and also on fused types) I noticed >>>>>>>> that UtilityCodes started weighing heavily on me in their current >>>>>>>> form, so I wrote a little loader in the _memview branch: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>>>>>> >>>>>>>> The idea is simple: you put your utility codes in Cython/Utility in >>>>>>>> .pyx, .c, .h files etc, and then load them. It works for both >>>>>>>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>>>>>> >>>>>>> This sounds like it could be a nice way to organize our UtilityCode >>>>>>> snippets. So far we haven't really needed any more templating than >>>>>>> simple substitution, but for what you're doing I can see this being >>>>>>> quite handy. This may also provide a more flexible way forward for >>>>>>> supporting multiple backends. >>>>>>> >>>>>>>> myutility.c >>>>>>>> >>>>>>>> // UtilityProto: MyUtility >>>>>>>> header code here >>>>>>>> >>>>>>>> // UtilityCode: MyUtility >>>>>>>> implementation code here >>>>>>>> >>>>>>>> You can add as many other utilities as you like to the same file. You >>>>>>>> can then load it using >>>>>>>> >>>>>>>> ? ?UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>>>>>> >>>>>>> I agree with you that having multiple related, named snippets in same >>>>>>> file is worthwhile. What about >>>>>>> >>>>>>> ////////////////////// MyUtility.proto /////////////////////////// >>>>>>> >>>>>>> and >>>>>>> >>>>>>> ############ MyCyUtility ############## >>>>>>> >>>>>>> so the chunks are easy to see. >>>>>>> >>>>>> >>>>>> C++ comments looks ugly. May be it's better to have something like this: >>>>>> >>>>>> /* UtilityCode: ?MyUtility.proto */ >>>>>> and >>>>>> >>>>>> # UtilityCode: MyCyUtility >>>>>> >>>>>> That's also pretty easy to parse >>>>> >>>>> For the parser, it makes no difference. For a human, it does. A big fat >>>>> marker like "/////////////////////" is hard to miss, whereas a tiny one >>>>> like >>>>> "/* ... */" is easily overlooked within a longer piece of code. >>>>> >>>> >>>> Ok, but how many slashes should be there? >>>> >>>> I'd prefer something like: >>>> >>>> ########## UtilityCode: MyUtility ################ >>>> >>>> For both C and Cython >>> >>> I don't think the parser should care, and I don't think we should put much >>> thought into this kind of detail, either. I'd suggest making the separator >>> lines at most 78 characters long, and putting as many slashes or hashes in >>> there as are needed to fill up the lines after writing the text content. >>> What's important to the parser is that there's more than one hash/slash at >>> start and end, and something parsable in between. The rest is up to those >>> who write the file. >> >> Yeah, I simply mandated a minimum of 5 and a maximum of 30 on either >> side (symmetry is not enforced). > > I don't even think we need a max. > >> It's now 'MyUtility' and >> 'MyUtility.proto'. If there's no objection to the ini-style header >> (with requirements and other metadata possibly), then I'll implement >> that one of these days. > > One drawback with the ini style is that it detaches the metadata from > the code block itself (and require duplicating their names). > Are we going to move all the existing utility code away from python code? I think that would seriously help with moving shared C-code into cython library (.h and .so). Some things like generator class implementation and cyfunction could be move into .so file. -- vitja. From robertwb at math.washington.edu Tue Jul 26 07:00:59 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 25 Jul 2011 22:00:59 -0700 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E2D114B.1000509@behnel.de> <4E2D3EC6.5060801@behnel.de> Message-ID: On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov wrote: > 2011/7/26 Robert Bradshaw : >> On Mon, Jul 25, 2011 at 3:07 AM, mark florisson >> wrote: >>> On 25 July 2011 12:00, Stefan Behnel wrote: >>>> Vitja Makarov, 25.07.2011 10:25: >>>>> >>>>> 2011/7/25 Stefan Behnel: >>>>>> >>>>>> Vitja Makarov, 25.07.2011 08:41: >>>>>>> >>>>>>> 2011/7/23 Robert Bradshaw: >>>>>>>> >>>>>>>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson >>>>>>>> ? ?wrote: >>>>>>>>> >>>>>>>>> For my work on the _memview branch (and also on fused types) I noticed >>>>>>>>> that UtilityCodes started weighing heavily on me in their current >>>>>>>>> form, so I wrote a little loader in the _memview branch: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110 >>>>>>>>> >>>>>>>>> The idea is simple: you put your utility codes in Cython/Utility in >>>>>>>>> .pyx, .c, .h files etc, and then load them. It works for both >>>>>>>>> prototypes and implementations, for UtilityCode and CythonUtilityCode: >>>>>>>> >>>>>>>> This sounds like it could be a nice way to organize our UtilityCode >>>>>>>> snippets. So far we haven't really needed any more templating than >>>>>>>> simple substitution, but for what you're doing I can see this being >>>>>>>> quite handy. This may also provide a more flexible way forward for >>>>>>>> supporting multiple backends. >>>>>>>> >>>>>>>>> myutility.c >>>>>>>>> >>>>>>>>> // UtilityProto: MyUtility >>>>>>>>> header code here >>>>>>>>> >>>>>>>>> // UtilityCode: MyUtility >>>>>>>>> implementation code here >>>>>>>>> >>>>>>>>> You can add as many other utilities as you like to the same file. You >>>>>>>>> can then load it using >>>>>>>>> >>>>>>>>> ? ?UtilityCode.load_utility_from_file("myutility.c", "MyUtility") >>>>>>>> >>>>>>>> I agree with you that having multiple related, named snippets in same >>>>>>>> file is worthwhile. What about >>>>>>>> >>>>>>>> ////////////////////// MyUtility.proto /////////////////////////// >>>>>>>> >>>>>>>> and >>>>>>>> >>>>>>>> ############ MyCyUtility ############## >>>>>>>> >>>>>>>> so the chunks are easy to see. >>>>>>>> >>>>>>> >>>>>>> C++ comments looks ugly. May be it's better to have something like this: >>>>>>> >>>>>>> /* UtilityCode: ?MyUtility.proto */ >>>>>>> and >>>>>>> >>>>>>> # UtilityCode: MyCyUtility >>>>>>> >>>>>>> That's also pretty easy to parse >>>>>> >>>>>> For the parser, it makes no difference. For a human, it does. A big fat >>>>>> marker like "/////////////////////" is hard to miss, whereas a tiny one >>>>>> like >>>>>> "/* ... */" is easily overlooked within a longer piece of code. >>>>>> >>>>> >>>>> Ok, but how many slashes should be there? >>>>> >>>>> I'd prefer something like: >>>>> >>>>> ########## UtilityCode: MyUtility ################ >>>>> >>>>> For both C and Cython >>>> >>>> I don't think the parser should care, and I don't think we should put much >>>> thought into this kind of detail, either. I'd suggest making the separator >>>> lines at most 78 characters long, and putting as many slashes or hashes in >>>> there as are needed to fill up the lines after writing the text content. >>>> What's important to the parser is that there's more than one hash/slash at >>>> start and end, and something parsable in between. The rest is up to those >>>> who write the file. >>> >>> Yeah, I simply mandated a minimum of 5 and a maximum of 30 on either >>> side (symmetry is not enforced). >> >> I don't even think we need a max. >> >>> It's now 'MyUtility' and >>> 'MyUtility.proto'. If there's no objection to the ini-style header >>> (with requirements and other metadata possibly), then I'll implement >>> that one of these days. >> >> One drawback with the ini style is that it detaches the metadata from >> the code block itself (and require duplicating their names). >> > > Are we going to move all the existing utility code away from python code? Yes, though I don't see a huge hurry to get rid of the old style. > I think that would seriously help with ?moving shared C-code into > cython library (.h and .so). > > Some things like generator class implementation and cyfunction could > be move into .so file. Yes, that would be good. Of course we don't want a single .so file for every tiny chunk of utility code, but one needs to be careful lumping them together as well (e.g. things may compile differently depending on the set of headers included in the .pyx file). To do this right I we need to compile things at a higher level than module-by-module with Cython.Build.cythonize or similar. Certainly CyFunction and the generator class are easy and high-benefit targets (compared to, e.g. tuple indexing utility code). - Robert From stefan_ml at behnel.de Tue Jul 26 08:39:04 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 26 Jul 2011 08:39:04 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E2D114B.1000509@behnel.de> <4E2D3EC6.5060801@behnel.de> Message-ID: <4E2E6108.7090001@behnel.de> Robert Bradshaw, 26.07.2011 06:29: > On Mon, Jul 25, 2011 at 3:07 AM, mark florisson >> It's now 'MyUtility' and >> 'MyUtility.proto'. If there's no objection to the ini-style header >> (with requirements and other metadata possibly), then I'll implement >> that one of these days. > > One drawback with the ini style is that it detaches the metadata from > the code block itself (and require duplicating their names). The names are duplicated in the file already, since impl and proto are separated and can be anywhere in the file (e.g., you could put all protos at the top and all implementation parts further down). The advantages of putting in a header is that a) it's immediately visible from the top of the file what it contains b) the dependencies on other files are explicit and clearly visible, again, from the top of the file c) the metadata is easy and quick to parse, as the header is trivially separated from the rest d) only the header needs to be parsed in order to know what can be found in it or what other files are required to compile it e) we don't need to write our own parser, and the overall file format is trivial to parse Stefan From robertwb at math.washington.edu Tue Jul 26 08:46:53 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 25 Jul 2011 23:46:53 -0700 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E2E6108.7090001@behnel.de> References: <4E2D114B.1000509@behnel.de> <4E2D3EC6.5060801@behnel.de> <4E2E6108.7090001@behnel.de> Message-ID: On Mon, Jul 25, 2011 at 11:39 PM, Stefan Behnel wrote: > Robert Bradshaw, 26.07.2011 06:29: >> >> On Mon, Jul 25, 2011 at 3:07 AM, mark florisson >>> >>> It's now 'MyUtility' and >>> 'MyUtility.proto'. If there's no objection to the ini-style header >>> (with requirements and other metadata possibly), then I'll implement >>> that one of these days. >> >> One drawback with the ini style is that it detaches the metadata from >> the code block itself (and require duplicating their names). > > The names are duplicated in the file already, since impl and proto are > separated and can be anywhere in the file (e.g., you could put all protos at > the top and all implementation parts further down). The advantages of > putting in a header is that > > a) it's immediately visible from the top of the file what it contains > > b) the dependencies on other files are explicit and clearly visible, again, > from the top of the file > > c) the metadata is easy and quick to parse, as the header is trivially > separated from the rest > > d) only the header needs to be parsed in order to know what can be found in > it or what other files are required to compile it > > e) we don't need to write our own parser, and the overall file format is > trivial to parse I prefer keeping the metadata close, but perhaps that's just a matter of opinion. Slurping in the entire file and parsing the block headers is pretty trivial as well, and with pre-fetching (on so many levels) I don't think there's a technical advantage for files so small. For those (likely common) cases where no metadata is required, I'm -1 on having to manually keep a "table of contents." - Robert From stefan_ml at behnel.de Tue Jul 26 08:50:09 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 26 Jul 2011 08:50:09 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E2D114B.1000509@behnel.de> <4E2D3EC6.5060801@behnel.de> Message-ID: <4E2E63A1.6080706@behnel.de> Robert Bradshaw, 26.07.2011 07:00: > On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov wrote: >> I think that would seriously help with moving shared C-code into >> cython library (.h and .so). >> >> Some things like generator class implementation and cyfunction could >> be move into .so file. > > Yes, that would be good. Well, *if* we find a way to make sure the additional modules get properly distributed. It's not obvious to me how to do that, certainly not in a cross-package way, and not even in a per-package way, as that would still require some kind of interaction with distutils and maybe even changes to the users' setup.py scripts (hopefully in a way that doesn't require them to know what exact additional modules will be generated). > Of course we don't want a single .so file for > every tiny chunk of utility code, but one needs to be careful lumping > them together as well (e.g. things may compile differently depending > on the set of headers included in the .pyx file). To do this right I > we need to compile things at a higher level than module-by-module with > Cython.Build.cythonize or similar. > > Certainly CyFunction and the generator class are easy and high-benefit > targets (compared to, e.g. tuple indexing utility code). Now that we have a way to group utility code in a single file, including automatically resolved dependencies to other files, it should be trivial to take such a file and to compile it into a separate module. Cython modules would then trivially export their public names anyway, and utility code in C files could have an additional Cython wrapper file that would simply depend on the C file and that properly exports the C functions through a Python API and inter-module C-API. We could even write empty utility files that only contain dependencies, thus grouping together multiple utility files to one larger module. Stefan From robertwb at math.washington.edu Tue Jul 26 08:57:51 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Mon, 25 Jul 2011 23:57:51 -0700 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E2E63A1.6080706@behnel.de> References: <4E2D114B.1000509@behnel.de> <4E2D3EC6.5060801@behnel.de> <4E2E63A1.6080706@behnel.de> Message-ID: On Mon, Jul 25, 2011 at 11:50 PM, Stefan Behnel wrote: > Robert Bradshaw, 26.07.2011 07:00: >> >> On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov wrote: >>> >>> I think that would seriously help with ?moving shared C-code into >>> cython library (.h and .so). >>> >>> Some things like generator class implementation and cyfunction could >>> be move into .so file. >> >> Yes, that would be good. > > Well, *if* we find a way to make sure the additional modules get properly > distributed. It's not obvious to me how to do that, certainly not in a > cross-package way, and not even in a per-package way, as that would still > require some kind of interaction with distutils and maybe even changes to > the users' setup.py scripts (hopefully in a way that doesn't require them to > know what exact additional modules will be generated). This is where the ext_modules = Cython.Build.cythonize(...) pattern could come in handy. We'd just add an extra module to the list. (The generated .c file could also be shipped, but might have to be listed explicitly.) >> Of course we don't want a single .so file for >> every tiny chunk of utility code, but one needs to be careful lumping >> them together as well (e.g. things may compile differently depending >> on the set of headers included in the .pyx file). To do this right I >> we need to compile things at a higher level than module-by-module with >> Cython.Build.cythonize or similar. >> >> Certainly CyFunction and the generator class are easy and high-benefit >> targets (compared to, e.g. tuple indexing utility code). > > Now that we have a way to group utility code in a single file, including > automatically resolved dependencies to other files, it should be trivial to > take such a file and to compile it into a separate module. Cython modules > would then trivially export their public names anyway, and utility code in C > files could have an additional Cython wrapper file that would simply depend > on the C file and that properly exports the C functions through a Python API > and inter-module C-API. > > We could even write empty utility files that only contain dependencies, thus > grouping together multiple utility files to one larger module. I was thinking about stuff like how we handle complex numbers where the pre-processor has a large role to play. Low-level stuff like this probably belongs in a .h file rather than a .so file though. I'm also wondering if there are any efficiency gains to be had from actually linking the resulting .so file at compile time rather than using the runtime inter-module C-API stuff. - Robert From stefan_ml at behnel.de Tue Jul 26 10:10:57 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 26 Jul 2011 10:10:57 +0200 Subject: [Cython] Transformation of pxds In-Reply-To: <20110726021013.GA1447@ubuntu> References: <20110723045244.GA7492@ubuntu> <4E2A989B.4020103@behnel.de> <20110726021013.GA1447@ubuntu> Message-ID: <4E2E7691.3060602@behnel.de> Romain Guillebert, 26.07.2011 04:10: > I can now compile pxd files, but I have encountered something rather > strange : AnalyseExpressionsTransform turns : > > cimport foo > > def test(): > foo.printf() > > into : > > cimport foo > > def test(): > printf() This makes sense from the POV of C, which has a flat namespace. It is true that this doesn't make much sense for the Python backend. Specifically, we do not want to flatten the .pxd namespace here, because the attribute name may already exist in the module namespace. Only the qualified name gives us proper and safe Python code. > It is due to the method analyse_as_cimported_attribute of AttributeNode > in the file Cython/Compiler/ExprNodes.py > > Is this useful (since I don't think it has anything to do with > expression analysis) and do you have an idea on how I can solve this (my > first idea is : extend the class to disable the effect of this method > but it's more a hack than anything else) ? Well, the current implementation is already a huge hack, see mutate_into_name_node(). The proper fix we envisioned for cases like this is to globally refactor the analyse_types() methods into methods that return self, or any other node that they deem more appropriate. The caller would then put the returned value in the place of the original node. This approach would allow analyse_as_cimported_attribute() to simply instantiate and return a new NameNode, instead of having to changing the type of itself locally. However, so far, no-one has done anything to get this refactoring done, likely because it's a lot of rather boring work. Regardless of this refactoring, I think it might be best to explicitly implement this method in a backend specific way. When targeting C, it should flatten the attribute access. When targeting Python, it should still determine the entry (and thus the type of the attribute) but should not build a NameNode. I assume the problem here is that the method does not actually know which backend it is targeting, right? The "env" parameter (i.e. the scope) won't know that, and I'm not sure it should know that. My suggestion is to let the module scope of the .pxd know that it's being read into (or as) a flat or prefixed namespace, and to propagate the information from there into the Entry objects it describes. Then the Entry of the attribute would know if it needs a namespace or not, and the analysis method could get its information directly from there. Does that make sense? Stefan From markflorisson88 at gmail.com Tue Jul 26 10:57:57 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 26 Jul 2011 10:57:57 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E2D114B.1000509@behnel.de> <4E2D3EC6.5060801@behnel.de> <4E2E6108.7090001@behnel.de> Message-ID: On 26 July 2011 08:46, Robert Bradshaw wrote: > On Mon, Jul 25, 2011 at 11:39 PM, Stefan Behnel wrote: >> Robert Bradshaw, 26.07.2011 06:29: >>> >>> On Mon, Jul 25, 2011 at 3:07 AM, mark florisson >>>> >>>> It's now 'MyUtility' and >>>> 'MyUtility.proto'. If there's no objection to the ini-style header >>>> (with requirements and other metadata possibly), then I'll implement >>>> that one of these days. >>> >>> One drawback with the ini style is that it detaches the metadata from >>> the code block itself (and require duplicating their names). >> >> The names are duplicated in the file already, since impl and proto are >> separated and can be anywhere in the file (e.g., you could put all protos at >> the top and all implementation parts further down). The advantages of >> putting in a header is that >> >> a) it's immediately visible from the top of the file what it contains >> >> b) the dependencies on other files are explicit and clearly visible, again, >> from the top of the file >> >> c) the metadata is easy and quick to parse, as the header is trivially >> separated from the rest >> >> d) only the header needs to be parsed in order to know what can be found in >> it or what other files are required to compile it >> >> e) we don't need to write our own parser, and the overall file format is >> trivial to parse > > I prefer keeping the metadata close, but perhaps that's just a matter > of opinion. Slurping in the entire file and parsing the block headers > is pretty trivial as well, and with pre-fetching (on so many levels) I > don't think there's a technical advantage for files so small. For > those (likely common) cases where no metadata is required, I'm -1 on > having to manually keep a "table of contents." Right, you woud only list metadata if there is something worthwhile to list. Otherwise you just omit it. (Perhaps that was not what Stefan had in mind though, I'm not sure). > - Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From markflorisson88 at gmail.com Tue Jul 26 10:58:16 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 26 Jul 2011 10:58:16 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E2E63A1.6080706@behnel.de> References: <4E2D114B.1000509@behnel.de> <4E2D3EC6.5060801@behnel.de> <4E2E63A1.6080706@behnel.de> Message-ID: On 26 July 2011 08:50, Stefan Behnel wrote: > Robert Bradshaw, 26.07.2011 07:00: >> >> On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov wrote: >>> >>> I think that would seriously help with ?moving shared C-code into >>> cython library (.h and .so). >>> >>> Some things like generator class implementation and cyfunction could >>> be move into .so file. >> >> Yes, that would be good. > > Well, *if* we find a way to make sure the additional modules get properly > distributed. It's not obvious to me how to do that, certainly not in a > cross-package way, and not even in a per-package way, as that would still > require some kind of interaction with distutils and maybe even changes to > the users' setup.py scripts (hopefully in a way that doesn't require them to > know what exact additional modules will be generated). > > >> Of course we don't want a single .so file for >> every tiny chunk of utility code, but one needs to be careful lumping >> them together as well (e.g. things may compile differently depending >> on the set of headers included in the .pyx file). To do this right I >> we need to compile things at a higher level than module-by-module with >> Cython.Build.cythonize or similar. >> >> Certainly CyFunction and the generator class are easy and high-benefit >> targets (compared to, e.g. tuple indexing utility code). > > Now that we have a way to group utility code in a single file, including > automatically resolved dependencies to other files, it should be trivial to > take such a file and to compile it into a separate module. Cython modules > would then trivially export their public names anyway, and utility code in C > files could have an additional Cython wrapper file that would simply depend > on the C file and that properly exports the C functions through a Python API > and inter-module C-API. > > We could even write empty utility files that only contain dependencies, thus > grouping together multiple utility files to one larger module. As you said earlier, you'd lose the inline functionality though. But I suppose we could put inline utilities in a header and non-inline utilities in .so files. I'm then not sure how to prevent compiler warnings for unused static inline functions though... > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Tue Jul 26 11:26:28 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 26 Jul 2011 11:26:28 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: References: <4E2D114B.1000509@behnel.de> <4E2D3EC6.5060801@behnel.de> <4E2E6108.7090001@behnel.de> Message-ID: <4E2E8844.4030701@behnel.de> mark florisson, 26.07.2011 10:57: > On 26 July 2011 08:46, Robert Bradshaw wrote: >> On Mon, Jul 25, 2011 at 11:39 PM, Stefan Behnel wrote: >>> Robert Bradshaw, 26.07.2011 06:29: >>>> >>>> On Mon, Jul 25, 2011 at 3:07 AM, mark florisson >>>>> >>>>> It's now 'MyUtility' and >>>>> 'MyUtility.proto'. If there's no objection to the ini-style header >>>>> (with requirements and other metadata possibly), then I'll implement >>>>> that one of these days. >>>> >>>> One drawback with the ini style is that it detaches the metadata from >>>> the code block itself (and require duplicating their names). >>> >>> The names are duplicated in the file already, since impl and proto are >>> separated and can be anywhere in the file (e.g., you could put all protos at >>> the top and all implementation parts further down). The advantages of >>> putting in a header is that >>> >>> a) it's immediately visible from the top of the file what it contains >>> >>> b) the dependencies on other files are explicit and clearly visible, again, >>> from the top of the file >>> >>> c) the metadata is easy and quick to parse, as the header is trivially >>> separated from the rest >>> >>> d) only the header needs to be parsed in order to know what can be found in >>> it or what other files are required to compile it >>> >>> e) we don't need to write our own parser, and the overall file format is >>> trivial to parse >> >> I prefer keeping the metadata close, but perhaps that's just a matter >> of opinion. Slurping in the entire file and parsing the block headers >> is pretty trivial as well, and with pre-fetching (on so many levels) I >> don't think there's a technical advantage for files so small. For >> those (likely common) cases where no metadata is required, I'm -1 on >> having to manually keep a "table of contents." > > Right, you woud only list metadata if there is something worthwhile to > list. Otherwise you just omit it. (Perhaps that was not what Stefan > had in mind though, I'm not sure). I don't care so much here, but I consider the file header an interface description. I.e. you could also include utility code in a file that is only available as dependencies to other utility code, simply by not listing it in the header. Allowing to skip the header makes the semantics a bit more ugly here. But as I said, I wouldn't mind if the header was optional, as long as that's easy enough to handle. Stefan From markflorisson88 at gmail.com Tue Jul 26 11:42:38 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 26 Jul 2011 11:42:38 +0200 Subject: [Cython] Utility Codes and templates In-Reply-To: <4E2E8844.4030701@behnel.de> References: <4E2D114B.1000509@behnel.de> <4E2D3EC6.5060801@behnel.de> <4E2E6108.7090001@behnel.de> <4E2E8844.4030701@behnel.de> Message-ID: On 26 July 2011 11:26, Stefan Behnel wrote: > mark florisson, 26.07.2011 10:57: >> >> On 26 July 2011 08:46, Robert Bradshaw >> ?wrote: >>> >>> On Mon, Jul 25, 2011 at 11:39 PM, Stefan Behnel >>> ?wrote: >>>> >>>> Robert Bradshaw, 26.07.2011 06:29: >>>>> >>>>> On Mon, Jul 25, 2011 at 3:07 AM, mark florisson >>>>>> >>>>>> It's now 'MyUtility' and >>>>>> 'MyUtility.proto'. If there's no objection to the ini-style header >>>>>> (with requirements and other metadata possibly), then I'll implement >>>>>> that one of these days. >>>>> >>>>> One drawback with the ini style is that it detaches the metadata from >>>>> the code block itself (and require duplicating their names). >>>> >>>> The names are duplicated in the file already, since impl and proto are >>>> separated and can be anywhere in the file (e.g., you could put all >>>> protos at >>>> the top and all implementation parts further down). The advantages of >>>> putting in a header is that >>>> >>>> a) it's immediately visible from the top of the file what it contains >>>> >>>> b) the dependencies on other files are explicit and clearly visible, >>>> again, >>>> from the top of the file >>>> >>>> c) the metadata is easy and quick to parse, as the header is trivially >>>> separated from the rest >>>> >>>> d) only the header needs to be parsed in order to know what can be found >>>> in >>>> it or what other files are required to compile it >>>> >>>> e) we don't need to write our own parser, and the overall file format is >>>> trivial to parse >>> >>> I prefer keeping the metadata close, but perhaps that's just a matter >>> of opinion. Slurping in the entire file and parsing the block headers >>> is pretty trivial as well, and with pre-fetching (on so many levels) I >>> don't think there's a technical advantage for files so small. For >>> those (likely common) cases where no metadata is required, I'm -1 on >>> having to manually keep a "table of contents." >> >> Right, you woud only list metadata if there is something worthwhile to >> list. Otherwise you just omit it. (Perhaps that was not what Stefan >> had in mind though, I'm not sure). > > I don't care so much here, but I consider the file header an interface > description. I.e. you could also include utility code in a file that is only > available as dependencies to other utility code, simply by not listing it in > the header. Allowing to skip the header makes the semantics a bit more ugly > here. > > But as I said, I wouldn't mind if the header was optional, as long as that's > easy enough to handle. Right, in that case I think the solution is quite appealing. > Stefan > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > From markflorisson88 at gmail.com Tue Jul 26 13:00:10 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Tue, 26 Jul 2011 13:00:10 +0200 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: <201107250808.19181.Arfrever.FTA@gmail.com> References: <201107250808.19181.Arfrever.FTA@gmail.com> Message-ID: On 25 July 2011 08:08, Arfrever Frehtes Taifersar Arahesis wrote: > There are 4 NumPy-related test errors with Python 3.1 and 3.2. > Output with Python 3.2: > > ====================================================================== > ERROR: runTest (__main__.CythonRunTestCase) > compiling (cpp) and running numpy_bufacc_T155 > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 183, in _compile > ? ?extra_postargs) > ?File "/usr/lib64/python3.2/distutils/ccompiler.py", line 911, in spawn > ? ?spawn(cmd, dry_run=self.dry_run) > ?File "/usr/lib64/python3.2/distutils/spawn.py", line 34, in spawn > ? ?_spawn_posix(cmd, search_path, dry_run=dry_run) > ?File "/usr/lib64/python3.2/distutils/spawn.py", line 165, in _spawn_posix > ? ?% (cmd[0], exit_status)) > distutils.errors.DistutilsExecError: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1 > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > ?File "runtests.py", line 678, in run > ? ?self.runCompileTest() > ?File "runtests.py", line 490, in runCompileTest > ? ?self.test_directory, self.expect_errors, self.annotate) > ?File "runtests.py", line 665, in compile > ? ?self.run_distutils(test_directory, module, workdir, incdir) > ?File "runtests.py", line 625, in run_distutils > ? ?build_extension.run() > ?File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 347, in run > ? ?self.build_extensions() > ?File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 456, in build_extensions > ? ?self.build_extension(ext) > ?File "runtests.py", line 257, in build_extension > ? ?_build_ext.build_extension(self, ext) > ?File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 511, in build_extension > ? ?depends=ext.depends) > ?File "/usr/lib64/python3.2/distutils/ccompiler.py", line 576, in compile > ? ?self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) > ?File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 188, in _compile > ? ?raise CompileError(msg) > distutils.errors.CompileError: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1 > > ====================================================================== > ERROR: runTest (__main__.CythonRunTestCase) > compiling (cpp) and running numpy_cimport > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 183, in _compile > ? ?extra_postargs) > ?File "/usr/lib64/python3.2/distutils/ccompiler.py", line 911, in spawn > ? ?spawn(cmd, dry_run=self.dry_run) > ?File "/usr/lib64/python3.2/distutils/spawn.py", line 34, in spawn > ? ?_spawn_posix(cmd, search_path, dry_run=dry_run) > ?File "/usr/lib64/python3.2/distutils/spawn.py", line 165, in _spawn_posix > ? ?% (cmd[0], exit_status)) > distutils.errors.DistutilsExecError: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1 > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > ?File "runtests.py", line 678, in run > ? ?self.runCompileTest() > ?File "runtests.py", line 490, in runCompileTest > ? ?self.test_directory, self.expect_errors, self.annotate) > ?File "runtests.py", line 665, in compile > ? ?self.run_distutils(test_directory, module, workdir, incdir) > ?File "runtests.py", line 625, in run_distutils > ? ?build_extension.run() > ?File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 347, in run > ? ?self.build_extensions() > ?File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 456, in build_extensions > ? ?self.build_extension(ext) > ?File "runtests.py", line 257, in build_extension > ? ?_build_ext.build_extension(self, ext) > ?File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 511, in build_extension > ? ?depends=ext.depends) > ?File "/usr/lib64/python3.2/distutils/ccompiler.py", line 576, in compile > ? ?self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) > ?File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 188, in _compile > ? ?raise CompileError(msg) > distutils.errors.CompileError: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1 > > ====================================================================== > ERROR: runTest (__main__.CythonRunTestCase) > compiling (cpp) and running numpy_parallel > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 183, in _compile > ? ?extra_postargs) > ?File "/usr/lib64/python3.2/distutils/ccompiler.py", line 911, in spawn > ? ?spawn(cmd, dry_run=self.dry_run) > ?File "/usr/lib64/python3.2/distutils/spawn.py", line 34, in spawn > ? ?_spawn_posix(cmd, search_path, dry_run=dry_run) > ?File "/usr/lib64/python3.2/distutils/spawn.py", line 165, in _spawn_posix > ? ?% (cmd[0], exit_status)) > distutils.errors.DistutilsExecError: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1 > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > ?File "runtests.py", line 678, in run > ? ?self.runCompileTest() > ?File "runtests.py", line 490, in runCompileTest > ? ?self.test_directory, self.expect_errors, self.annotate) > ?File "runtests.py", line 665, in compile > ? ?self.run_distutils(test_directory, module, workdir, incdir) > ?File "runtests.py", line 625, in run_distutils > ? ?build_extension.run() > ?File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 347, in run > ? ?self.build_extensions() > ?File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 456, in build_extensions > ? ?self.build_extension(ext) > ?File "runtests.py", line 257, in build_extension > ? ?_build_ext.build_extension(self, ext) > ?File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 511, in build_extension > ? ?depends=ext.depends) > ?File "/usr/lib64/python3.2/distutils/ccompiler.py", line 576, in compile > ? ?self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) > ?File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 188, in _compile > ? ?raise CompileError(msg) > distutils.errors.CompileError: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1 > > ====================================================================== > ERROR: runTest (__main__.CythonRunTestCase) > compiling (cpp) and running numpy_test > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 183, in _compile > ? ?extra_postargs) > ?File "/usr/lib64/python3.2/distutils/ccompiler.py", line 911, in spawn > ? ?spawn(cmd, dry_run=self.dry_run) > ?File "/usr/lib64/python3.2/distutils/spawn.py", line 34, in spawn > ? ?_spawn_posix(cmd, search_path, dry_run=dry_run) > ?File "/usr/lib64/python3.2/distutils/spawn.py", line 165, in _spawn_posix > ? ?% (cmd[0], exit_status)) > distutils.errors.DistutilsExecError: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1 > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > ?File "runtests.py", line 678, in run > ? ?self.runCompileTest() > ?File "runtests.py", line 490, in runCompileTest > ? ?self.test_directory, self.expect_errors, self.annotate) > ?File "runtests.py", line 665, in compile > ? ?self.run_distutils(test_directory, module, workdir, incdir) > ?File "runtests.py", line 625, in run_distutils > ? ?build_extension.run() > ?File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 347, in run > ? ?self.build_extensions() > ?File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 456, in build_extensions > ? ?self.build_extension(ext) > ?File "runtests.py", line 257, in build_extension > ? ?_build_ext.build_extension(self, ext) > ?File "/usr/lib64/python3.2/distutils/command/build_ext.py", line 511, in build_extension > ? ?depends=ext.depends) > ?File "/usr/lib64/python3.2/distutils/ccompiler.py", line 576, in compile > ? ?self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts) > ?File "/usr/lib64/python3.2/distutils/unixccompiler.py", line 188, in _compile > ? ?raise CompileError(msg) > distutils.errors.CompileError: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1 > > ====================================================================== > FAIL: test_nested_break_continue (line 331) (parallel.__test__) > Doctest: parallel.__test__.test_nested_break_continue (line 331) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python3.2/doctest.py", line 2118, in runTest > ? ?raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for parallel.__test__.test_nested_break_continue (line 331) > ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-3.2/run/c/parallel.cpython-32.so", line unknown line number, in test_nested_break_continue (line 331) > > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-3.2/run/c/parallel.cpython-32.so", line ?, in parallel.__test__.test_nested_break_continue (line 331) > Failed example: > ? ?test_nested_break_continue() > Expected: > ? ?6 7 6 7 > ? ?8 > Got: > ? ?6 0 -1160725808 -1160725808 > ? ?8 > > > ====================================================================== > FAIL: test_propagation (line 49) (parallel.__test__) > Doctest: parallel.__test__.test_propagation (line 49) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python3.2/doctest.py", line 2118, in runTest > ? ?raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for parallel.__test__.test_propagation (line 49) > ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-3.2/run/c/parallel.cpython-32.so", line unknown line number, in test_propagation (line 49) > > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-3.2/run/c/parallel.cpython-32.so", line ?, in parallel.__test__.test_propagation (line 49) > Failed example: > ? ?test_propagation() > Expected: > ? ?(9, 9, 9, 9, 450, 450) > Got: > ? ?(9, 0, 9, -1160725808, 0, 0) > > > ====================================================================== > FAIL: test_nested_break_continue (line 331) (parallel.__test__) > Doctest: parallel.__test__.test_nested_break_continue (line 331) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python3.2/doctest.py", line 2118, in runTest > ? ?raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for parallel.__test__.test_nested_break_continue (line 331) > ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-3.2/run/cpp/parallel.cpython-32.so", line unknown line number, in test_nested_break_continue (line 331) > > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-3.2/run/cpp/parallel.cpython-32.so", line ?, in parallel.__test__.test_nested_break_continue (line 331) > Failed example: > ? ?test_nested_break_continue() > Expected: > ? ?6 7 6 7 > ? ?8 > Got: > ? ?6 0 -1160725808 -1160725808 > ? ?8 > > > ====================================================================== > FAIL: test_propagation (line 49) (parallel.__test__) > Doctest: parallel.__test__.test_propagation (line 49) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python3.2/doctest.py", line 2118, in runTest > ? ?raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for parallel.__test__.test_propagation (line 49) > ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-3.2/run/cpp/parallel.cpython-32.so", line unknown line number, in test_propagation (line 49) > > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-3.2/run/cpp/parallel.cpython-32.so", line ?, in parallel.__test__.test_propagation (line 49) > Failed example: > ? ?test_propagation() > Expected: > ? ?(9, 9, 9, 9, 450, 450) > Got: > ? ?(9, 0, 9, -1160725808, 0, 0) > > > ---------------------------------------------------------------------- > Ran 5528 tests in 1476.755s > > FAILED (failures=4, errors=4) > ALL DONE > > -- > Arfrever Frehtes Taifersar Arahesis > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > Unfortunately the output of the testrunner is rather useless here. Could you run the tests with --no-cleanup and attach the .c and .cpp files (found in build/run/c/numpy_cimport.c, etc)? I think something is up with your numpy version and headers. The numpy tests should be automatically skipped if numpy is not available. As for the parallel tests, the values of lastprivates were racy after breaking, returning or propagating exceptions. I pushed a fix to https://github.com/markflorisson88/cython/commit/d9aceb672910796c3952340ab4b5d61c6634254b . If you want and have the time Dag, you could have a look. I'll push it to cython master later today. From arfrever.fta at gmail.com Wed Jul 27 03:26:35 2011 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Wed, 27 Jul 2011 03:26:35 +0200 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: <4E2D0A74.1070001@behnel.de> References: <201107250808.19181.Arfrever.FTA@gmail.com> <4E2D0A74.1070001@behnel.de> Message-ID: <201107270326.37108.Arfrever.FTA@gmail.com> 2011-07-25 08:17:24 Stefan Behnel napisa?(a): > Arfrever Frehtes Taifersar Arahesis, 25.07.2011 08:08: > > There are 4 NumPy-related test errors with Python 3.1 and 3.2. > > I assume that's NumPy 1.5? Could you provide the C compiler output from the > test logs? > > This is on 64bit Linux, right? What gcc version? NumPy 1.6.1, 64bit Linux, GCC 4.5.2. -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From stefan_ml at behnel.de Wed Jul 27 18:32:09 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 27 Jul 2011 18:32:09 +0200 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) Message-ID: <4E303D89.2060304@behnel.de> Hi, quick question before raising a poll on the users mailing list. Would anyone mind dropping support for CPython 2.3? 1) it's long out of maintenance, even the last security release dates back to early 2008 2) there have been seven main releases of CPython since then, four of which were in the 2.x line, starting with 2.4 in late 2004 - even 2.5 was released five years ago 3) it produces weird and annoying errors in Jenkins, or rather none at all most of the time, since the test suite does not run the doctests on 2.3 anyway 4) the new code that was written by Vitja and Mark would be (or would have been) cleaner with decorators and other 'recent' Python features There are two sides to this: dropping support for running Cython in 2.3 and dropping support for compiling the generated code in 2.3. The first is the more interesting one. It's not strictly required to do both, we could continue to support it at the C level, but given how badly tested Cython is on that version anyway, I think stating that the generated code is 2.3 compatible is already hand waving today. So we may even just let the C code support fade out silently until someone actually notices. Actually, even 2.4 is a candidate for dropping support for running Cython on it. The last release dates back to December 2008, and its lack of 64 bit support makes it severly less attractive than even 2.5, which is also going out of security-fix maintenance now. Comments? Stefan From vitja.makarov at gmail.com Wed Jul 27 18:46:01 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Wed, 27 Jul 2011 20:46:01 +0400 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: <4E303D89.2060304@behnel.de> References: <4E303D89.2060304@behnel.de> Message-ID: 2011/7/27 Stefan Behnel : > Hi, > > quick question before raising a poll on the users mailing list. > > Would anyone mind dropping support for CPython 2.3? > > 1) it's long out of maintenance, even the last security release dates back > to early 2008 > > 2) there have been seven main releases of CPython since then, four of which > were in the 2.x line, starting with 2.4 in late 2004 - even 2.5 was released > five years ago > > 3) it produces weird and annoying errors in Jenkins, or rather none at all > most of the time, since the test suite does not run the doctests on 2.3 > anyway > > 4) the new code that was written by Vitja and Mark would be (or would have > been) cleaner with decorators and other 'recent' Python features > > There are two sides to this: dropping support for running Cython in 2.3 and > dropping support for compiling the generated code in 2.3. The first is the > more interesting one. It's not strictly required to do both, we could > continue to support it at the C level, but given how badly tested Cython is > on that version anyway, I think stating that the generated code is 2.3 > compatible is already hand waving today. So we may even just let the C code > support fade out silently until someone actually notices. > > Actually, even 2.4 is a candidate for dropping support for running Cython on > it. The last release dates back to December 2008, and its lack of 64 bit > support makes it severly less attractive than even 2.5, which is also going > out of security-fix maintenance now. > > Comments? > +1 I think we should completely drop 2.3 and 2.4 support. I hope nobody use it with recent Cython versions. -- vitja. From markflorisson88 at gmail.com Wed Jul 27 18:51:18 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Wed, 27 Jul 2011 18:51:18 +0200 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: References: <4E303D89.2060304@behnel.de> Message-ID: On 27 July 2011 18:46, Vitja Makarov wrote: > 2011/7/27 Stefan Behnel : >> Hi, >> >> quick question before raising a poll on the users mailing list. >> >> Would anyone mind dropping support for CPython 2.3? >> >> 1) it's long out of maintenance, even the last security release dates back >> to early 2008 >> >> 2) there have been seven main releases of CPython since then, four of which >> were in the 2.x line, starting with 2.4 in late 2004 - even 2.5 was released >> five years ago >> >> 3) it produces weird and annoying errors in Jenkins, or rather none at all >> most of the time, since the test suite does not run the doctests on 2.3 >> anyway >> >> 4) the new code that was written by Vitja and Mark would be (or would have >> been) cleaner with decorators and other 'recent' Python features >> >> There are two sides to this: dropping support for running Cython in 2.3 and >> dropping support for compiling the generated code in 2.3. The first is the >> more interesting one. It's not strictly required to do both, we could >> continue to support it at the C level, but given how badly tested Cython is >> on that version anyway, I think stating that the generated code is 2.3 >> compatible is already hand waving today. So we may even just let the C code >> support fade out silently until someone actually notices. >> >> Actually, even 2.4 is a candidate for dropping support for running Cython on >> it. The last release dates back to December 2008, and its lack of 64 bit >> support makes it severly less attractive than even 2.5, which is also going >> out of security-fix maintenance now. >> >> Comments? >> > > +1 > > I think we should completely drop 2.3 and 2.4 support. I hope nobody > use it with recent Cython versions. > > -- > vitja. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > +1 from me too, especially 2.3 is a royal pain in the ass. I want my any, all, decorators and I want my str.partition and even str.rsplit. I also can't run 2.3 on my system as it won't compile, so it always makes my hudson red. I'm also pretty ok with 2.4 if anyone wants to keep it, but would certainly not mind dropping it. From drsalists at gmail.com Wed Jul 27 20:01:25 2011 From: drsalists at gmail.com (Dan Stromberg) Date: Wed, 27 Jul 2011 11:01:25 -0700 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: <4E303D89.2060304@behnel.de> References: <4E303D89.2060304@behnel.de> Message-ID: We probably all want to use newer Python features. I believe CentOS just recently released an OS that includes a recent Python that wouldn't be hit by this; it'll probably be a while before CentOS and RHEL people don't desire 2.4 support. But in my own use, I target 2.5 and up - if a system only has 2.4 or 2.3, I compile a newer interpreter. On Wed, Jul 27, 2011 at 9:32 AM, Stefan Behnel wrote: > Hi, > > quick question before raising a poll on the users mailing list. > > Would anyone mind dropping support for CPython 2.3? > > 1) it's long out of maintenance, even the last security release dates back > to early 2008 > > 2) there have been seven main releases of CPython since then, four of which > were in the 2.x line, starting with 2.4 in late 2004 - even 2.5 was released > five years ago > > 3) it produces weird and annoying errors in Jenkins, or rather none at all > most of the time, since the test suite does not run the doctests on 2.3 > anyway > > 4) the new code that was written by Vitja and Mark would be (or would have > been) cleaner with decorators and other 'recent' Python features > > There are two sides to this: dropping support for running Cython in 2.3 and > dropping support for compiling the generated code in 2.3. The first is the > more interesting one. It's not strictly required to do both, we could > continue to support it at the C level, but given how badly tested Cython is > on that version anyway, I think stating that the generated code is 2.3 > compatible is already hand waving today. So we may even just let the C code > support fade out silently until someone actually notices. > > Actually, even 2.4 is a candidate for dropping support for running Cython > on it. The last release dates back to December 2008, and its lack of 64 bit > support makes it severly less attractive than even 2.5, which is also going > out of security-fix maintenance now. > > Comments? > > Stefan > ______________________________**_________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/**mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at math.washington.edu Wed Jul 27 20:11:48 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 27 Jul 2011 11:11:48 -0700 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: References: <4E303D89.2060304@behnel.de> Message-ID: On Wed, Jul 27, 2011 at 9:51 AM, mark florisson wrote: > On 27 July 2011 18:46, Vitja Makarov wrote: >> 2011/7/27 Stefan Behnel : >>> Hi, >>> >>> quick question before raising a poll on the users mailing list. >>> >>> Would anyone mind dropping support for CPython 2.3? >>> >>> 1) it's long out of maintenance, even the last security release dates back >>> to early 2008 >>> >>> 2) there have been seven main releases of CPython since then, four of which >>> were in the 2.x line, starting with 2.4 in late 2004 - even 2.5 was released >>> five years ago >>> >>> 3) it produces weird and annoying errors in Jenkins, or rather none at all >>> most of the time, since the test suite does not run the doctests on 2.3 >>> anyway >>> >>> 4) the new code that was written by Vitja and Mark would be (or would have >>> been) cleaner with decorators and other 'recent' Python features >>> >>> There are two sides to this: dropping support for running Cython in 2.3 and >>> dropping support for compiling the generated code in 2.3. The first is the >>> more interesting one. It's not strictly required to do both, we could >>> continue to support it at the C level, but given how badly tested Cython is >>> on that version anyway, I think stating that the generated code is 2.3 >>> compatible is already hand waving today. So we may even just let the C code >>> support fade out silently until someone actually notices. >>> >>> Actually, even 2.4 is a candidate for dropping support for running Cython on >>> it. The last release dates back to December 2008, and its lack of 64 bit >>> support makes it severly less attractive than even 2.5, which is also going >>> out of security-fix maintenance now. >>> >>> Comments? >>> >> >> +1 >> >> I think we should completely drop 2.3 and 2.4 support. I hope nobody >> use it with recent Cython versions. I'm OK with dropping 2.3, but lets pose the question on cython-users as well first. If so, we'd declare 0.15 as the last release "supporting" 2.3. Dropping 2.4 seems to have less advantages and more disadvantages, but is worth inquiring about as well. - Robert From d.s.seljebotn at astro.uio.no Wed Jul 27 20:14:05 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 27 Jul 2011 20:14:05 +0200 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: <4E303D89.2060304@behnel.de> References: <4E303D89.2060304@behnel.de> Message-ID: <4E30556D.7030901@astro.uio.no> On 07/27/2011 06:32 PM, Stefan Behnel wrote: > Hi, > > quick question before raising a poll on the users mailing list. > > Would anyone mind dropping support for CPython 2.3? > > 1) it's long out of maintenance, even the last security release dates > back to early 2008 > > 2) there have been seven main releases of CPython since then, four of > which were in the 2.x line, starting with 2.4 in late 2004 - even 2.5 > was released five years ago > > 3) it produces weird and annoying errors in Jenkins, or rather none at > all most of the time, since the test suite does not run the doctests on > 2.3 anyway > > 4) the new code that was written by Vitja and Mark would be (or would > have been) cleaner with decorators and other 'recent' Python features > > There are two sides to this: dropping support for running Cython in 2.3 > and dropping support for compiling the generated code in 2.3. The first > is the more interesting one. It's not strictly required to do both, we > could continue to support it at the C level, but given how badly tested > Cython is on that version anyway, I think stating that the generated > code is 2.3 compatible is already hand waving today. So we may even just > let the C code support fade out silently until someone actually notices. > > Actually, even 2.4 is a candidate for dropping support for running > Cython on it. The last release dates back to December 2008, and its lack > of 64 bit support makes it severly less attractive than even 2.5, which > is also going out of security-fix maintenance now. > > Comments? An important aspect is that if you're really stuck on an old Python, you can always use an old version of Cython. I'm +1 at dropping 2.3 support. Let's not have it "fade out silently", I don't like hand waving much -- let's make sure to put it prominently in the release notes that it is not supported, drop the Jenkins builds for it, and not think twice about generating C that doesn't work under 2.3. In fact, how about an #ifdef explicitly generating an error under 2.3. I'm less sure about 2.4. The RHEL in use at our institute actually have Python 2.4 in /usr/bin -- granted, they're just about to upgrade. I'd say keep 2.4 for Cython 0.15[.x], put a deprecation warning in the release notes, and then remove support in Cython 0.16. That way, there's at least one Cython version out with all the new stuff for Python 2.4 that people can keep using. Dag Sverre From d.s.seljebotn at astro.uio.no Wed Jul 27 20:16:45 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Wed, 27 Jul 2011 20:16:45 +0200 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: References: <4E303D89.2060304@behnel.de> Message-ID: <4E30560D.6050307@astro.uio.no> On 07/27/2011 08:11 PM, Robert Bradshaw wrote: > On Wed, Jul 27, 2011 at 9:51 AM, mark florisson > wrote: >> On 27 July 2011 18:46, Vitja Makarov wrote: >>> 2011/7/27 Stefan Behnel: >>>> Hi, >>>> >>>> quick question before raising a poll on the users mailing list. >>>> >>>> Would anyone mind dropping support for CPython 2.3? >>>> >>>> 1) it's long out of maintenance, even the last security release dates back >>>> to early 2008 >>>> >>>> 2) there have been seven main releases of CPython since then, four of which >>>> were in the 2.x line, starting with 2.4 in late 2004 - even 2.5 was released >>>> five years ago >>>> >>>> 3) it produces weird and annoying errors in Jenkins, or rather none at all >>>> most of the time, since the test suite does not run the doctests on 2.3 >>>> anyway >>>> >>>> 4) the new code that was written by Vitja and Mark would be (or would have >>>> been) cleaner with decorators and other 'recent' Python features >>>> >>>> There are two sides to this: dropping support for running Cython in 2.3 and >>>> dropping support for compiling the generated code in 2.3. The first is the >>>> more interesting one. It's not strictly required to do both, we could >>>> continue to support it at the C level, but given how badly tested Cython is >>>> on that version anyway, I think stating that the generated code is 2.3 >>>> compatible is already hand waving today. So we may even just let the C code >>>> support fade out silently until someone actually notices. >>>> >>>> Actually, even 2.4 is a candidate for dropping support for running Cython on >>>> it. The last release dates back to December 2008, and its lack of 64 bit >>>> support makes it severly less attractive than even 2.5, which is also going >>>> out of security-fix maintenance now. >>>> >>>> Comments? >>>> >>> >>> +1 >>> >>> I think we should completely drop 2.3 and 2.4 support. I hope nobody >>> use it with recent Cython versions. > > I'm OK with dropping 2.3, but lets pose the question on cython-users > as well first. If so, we'd declare 0.15 as the last release > "supporting" 2.3. Dropping 2.4 seems to have less advantages and more > disadvantages, but is worth inquiring about as well. Stefan mentioned 2.3 being "all red" -- would you support 2.3 for Cython 0.15 even if it's a lot of work? If it's not much work I agree, I just don't think it's a valuable use of our time if things need fixing. It's not like 0.14.x will become unavailable. Dag Sverre From robertwb at math.washington.edu Wed Jul 27 20:28:11 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Wed, 27 Jul 2011 11:28:11 -0700 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: <4E30560D.6050307@astro.uio.no> References: <4E303D89.2060304@behnel.de> <4E30560D.6050307@astro.uio.no> Message-ID: On Wed, Jul 27, 2011 at 11:16 AM, Dag Sverre Seljebotn wrote: > On 07/27/2011 08:11 PM, Robert Bradshaw wrote: >> >> On Wed, Jul 27, 2011 at 9:51 AM, mark florisson >> ?wrote: >>> >>> On 27 July 2011 18:46, Vitja Makarov ?wrote: >>>> >>>> 2011/7/27 Stefan Behnel: >>>>> >>>>> Hi, >>>>> >>>>> quick question before raising a poll on the users mailing list. >>>>> >>>>> Would anyone mind dropping support for CPython 2.3? >>>>> >>>>> 1) it's long out of maintenance, even the last security release dates >>>>> back >>>>> to early 2008 >>>>> >>>>> 2) there have been seven main releases of CPython since then, four of >>>>> which >>>>> were in the 2.x line, starting with 2.4 in late 2004 - even 2.5 was >>>>> released >>>>> five years ago >>>>> >>>>> 3) it produces weird and annoying errors in Jenkins, or rather none at >>>>> all >>>>> most of the time, since the test suite does not run the doctests on 2.3 >>>>> anyway >>>>> >>>>> 4) the new code that was written by Vitja and Mark would be (or would >>>>> have >>>>> been) cleaner with decorators and other 'recent' Python features >>>>> >>>>> There are two sides to this: dropping support for running Cython in 2.3 >>>>> and >>>>> dropping support for compiling the generated code in 2.3. The first is >>>>> the >>>>> more interesting one. It's not strictly required to do both, we could >>>>> continue to support it at the C level, but given how badly tested >>>>> Cython is >>>>> on that version anyway, I think stating that the generated code is 2.3 >>>>> compatible is already hand waving today. So we may even just let the C >>>>> code >>>>> support fade out silently until someone actually notices. >>>>> >>>>> Actually, even 2.4 is a candidate for dropping support for running >>>>> Cython on >>>>> it. The last release dates back to December 2008, and its lack of 64 >>>>> bit >>>>> support makes it severly less attractive than even 2.5, which is also >>>>> going >>>>> out of security-fix maintenance now. >>>>> >>>>> Comments? >>>>> >>>> >>>> +1 >>>> >>>> I think we should completely drop 2.3 and 2.4 support. I hope nobody >>>> use it with recent Cython versions. >> >> I'm OK with dropping 2.3, but lets pose the question on cython-users >> as well first. If so, we'd declare 0.15 as the last release >> "supporting" 2.3. Dropping 2.4 seems to have less advantages and more >> disadvantages, but is worth inquiring about as well. > > Stefan mentioned 2.3 being "all red" -- would you support 2.3 for Cython > 0.15 even if it's a lot of work? If it's not much work I agree, I just don't > think it's a valuable use of our time if things need fixing. I haven't looked at how much fixing it needs, but it's a pretty good track record: https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py23-c/ > It's not like 0.14.x will become unavailable. Yes, that is a good point, though if it's (really) easy, I'd rather say "this is the last release supporting 2.3" rather than "oh, btw, we dropped support for 2.3." I also think we should have a big fat #error rather than letting 2.3 support just fade away (or worse, silently produce bad/incorrect code). - Robert From wonjunchoi001 at gmail.com Thu Jul 28 08:23:24 2011 From: wonjunchoi001 at gmail.com (=?EUC-KR?B?w9a/+MHY?=) Date: Thu, 28 Jul 2011 15:23:24 +0900 Subject: [Cython] Cython 0.15 release In-Reply-To: References: <4E28FBA2.9020005@behnel.de> <4E2D09EA.9030507@behnel.de> Message-ID: Can I make cython file from python and C using generators? 2011? 7? 25? ?? 3:17, ??? ?? ?: > ok > > > 2011/7/25 Stefan Behnel > >> ???, 25.07.2011 08:03: >> >>> 2011/7/25 Vitja Makarov >>> >>>> 2011/7/25 ???: >>>> >>>> Can I test the generators? >>>>> >>>> >>>> Sure, generators support is in master already: >>>> >>>> https://github.com/cython/**cython >>>> >>> >>> I don't know where it is.. sorry >>> >> >> It's actually not that hard. You go to that URL, click on the big fat >> "Downloads" button, and then select the archive format you prefer. Next, you >> unpack it on your machine, change into the extracted directory and run >> "python setup.py install". >> >> Besides, this is a question for the cython-users mailing list, not the >> core developers mailing list. >> >> Stefan >> >> ______________________________**_________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/**mailman/listinfo/cython-devel >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Thu Jul 28 08:26:41 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 28 Jul 2011 08:26:41 +0200 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: References: <4E303D89.2060304@behnel.de> <4E30560D.6050307@astro.uio.no> Message-ID: <4E310121.2050400@behnel.de> Robert Bradshaw, 27.07.2011 20:28: > Yes, that is a good point, though if it's (really) easy, I'd rather > say "this is the last release supporting 2.3" rather than "oh, btw, we > dropped support for 2.3." I wouldn't mind either way. Let's see what the users poll gives. > I also think we should have a big fat #error rather than letting 2.3 > support just fade away (or worse, silently produce bad/incorrect > code). With "fading out" I just meant that a) we don't even know right now if the C code we generate really works in Python 2.3 and b) we could just leave it that way and add a note to the release notes that it's no longer actively supported but on a "I feel lucky" basis. But I agree that it's better to let the C compilation fail loudly, so that users know what the problem is. Stefan From stefan_ml at behnel.de Thu Jul 28 08:29:41 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 28 Jul 2011 08:29:41 +0200 Subject: [Cython] Cython 0.15 release In-Reply-To: References: <4E28FBA2.9020005@behnel.de> <4E2D09EA.9030507@behnel.de> Message-ID: <4E3101D5.6090407@behnel.de> ???, 28.07.2011 08:23: > Can I make cython file from python and C using generators? Please ask your question on the cython-users mailing list, not on this list, which is the core developers mailing list. And please try to be specific about what you want to achieve. It's not clear to me at all from the question above what you are trying to do. Stefan From vitja.makarov at gmail.com Thu Jul 28 08:32:03 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Thu, 28 Jul 2011 10:32:03 +0400 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: <4E310121.2050400@behnel.de> References: <4E303D89.2060304@behnel.de> <4E30560D.6050307@astro.uio.no> <4E310121.2050400@behnel.de> Message-ID: 2011/7/28 Stefan Behnel : > Robert Bradshaw, 27.07.2011 20:28: >> >> Yes, that is a good point, though if it's (really) easy, I'd rather >> say "this is the last release supporting 2.3" rather than "oh, btw, we >> dropped support for 2.3." > > I wouldn't mind either way. Let's see what the users poll gives. > > >> I also think we should have a big fat #error rather than letting 2.3 >> support just fade away (or worse, silently produce bad/incorrect >> code). > > With "fading out" I just meant that a) we don't even know right now if the C > code we generate really works in Python 2.3 and b) we could just leave it > that way and add a note to the release notes that it's no longer actively > supported but on a "I feel lucky" basis. > > But I agree that it's better to let the C compilation fail loudly, so that > users know what the problem is. > That could be a warning not an error. It's fun: with cython you can use decorators, generators and more with py2.3, may be it's better to drop support after 0.15? I hope it would be easy to fix this error: https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py23-c/871/console -- vitja. From wonjunchoi001 at gmail.com Thu Jul 28 08:34:41 2011 From: wonjunchoi001 at gmail.com (=?EUC-KR?B?w9a/+MHY?=) Date: Thu, 28 Jul 2011 15:34:41 +0900 Subject: [Cython] Cython 0.15 release In-Reply-To: <4E3101D5.6090407@behnel.de> References: <4E28FBA2.9020005@behnel.de> <4E2D09EA.9030507@behnel.de> <4E3101D5.6090407@behnel.de> Message-ID: because of no answer on the cython users.. anyone knows about this? 2011/7/28 Stefan Behnel > ???, 28.07.2011 08:23: > > Can I make cython file from python and C using generators? >> > > Please ask your question on the cython-users mailing list, not on this > list, which is the core developers mailing list. And please try to be > specific about what you want to achieve. It's not clear to me at all from > the question above what you are trying to do. > > > Stefan > ______________________________**_________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/**mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.s.seljebotn at astro.uio.no Thu Jul 28 08:53:26 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Thu, 28 Jul 2011 08:53:26 +0200 Subject: [Cython] Cython 0.15 release In-Reply-To: References: <4E28FBA2.9020005@behnel.de> <4E2D09EA.9030507@behnel.de> <4E3101D5.6090407@behnel.de> Message-ID: <866964a3-e5ed-4681-889b-45448ab20bb4@email.android.com> The reason you do not get answers is because people do not understand your question. Please describe in detail what you want to do (on the users list). -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. "???" wrote: because of no answer on the cython users.. anyone knows about this? 2011/7/28 Stefan Behnel ???, 28.07.2011 08:23: Can I make cython file from python and C using generators? Please ask your question on the cython-users mailing list, not on this list, which is the core developers mailing list. And please try to be specific about what you want to achieve. It's not clear to me at all from the question above what you are trying to do. Stefan _______________________________________________ cython-devel mailing list cython-devel at python.org http://mail.python.org/mailman/listinfo/cython-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From wonjunchoi001 at gmail.com Thu Jul 28 09:31:08 2011 From: wonjunchoi001 at gmail.com (=?EUC-KR?B?w9a/+MHY?=) Date: Thu, 28 Jul 2011 16:31:08 +0900 Subject: [Cython] Cython 0.15 release In-Reply-To: <866964a3-e5ed-4681-889b-45448ab20bb4@email.android.com> References: <4E28FBA2.9020005@behnel.de> <4E2D09EA.9030507@behnel.de> <4E3101D5.6090407@behnel.de> <866964a3-e5ed-4681-889b-45448ab20bb4@email.android.com> Message-ID: I had posted several issues in detail on the list but there was no answer so I deleted it several times.. on the cython users list. I just want to hear the answer(like a "describe in detail") even though the answer guy doesn't know the solution. I think no answer is bothering act. It is easy to tell me rtfm or do not post in here. 2011/7/28 Dag Sverre Seljebotn > ** The reason you do not get answers is because people do not understand > your question. Please describe in detail what you want to do (on the users > list). > > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity. > > > "???" wrote: >> >> because of no answer on the cython users.. >> anyone knows about this? >> >> >> >> 2011/7/28 Stefan Behnel >> >>> ???, 28.07.2011 08:23: >>> >>> Can I make cython file from python and C using generators? >>>> >>> >>> Please ask your question on the cython-users mailing list, not on this >>> list, which is the core developers mailing list. And please try to be >>> specific about what you want to achieve. It's not clear to me at all from >>> the question above what you are trying to do. >>> >>> >>> Stefan >>> ______________________________**_________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> http://mail.python.org/**mailman/listinfo/cython-devel >>> >> >> > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Thu Jul 28 09:51:38 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 28 Jul 2011 09:51:38 +0200 Subject: [Cython] Cython 0.15 release In-Reply-To: References: <4E28FBA2.9020005@behnel.de> <4E2D09EA.9030507@behnel.de> <4E3101D5.6090407@behnel.de> <866964a3-e5ed-4681-889b-45448ab20bb4@email.android.com> Message-ID: <4E31150A.7090603@behnel.de> ???, 28.07.2011 09:31: > I had posted several issues in detail on the list > but there was no answer so I deleted it several times.. on the cython users > list. > I just want to hear the answer(like a "describe in detail") even though the > answer guy doesn't know the solution. > I think no answer is bothering act. It's much more "bothering" to ask questions that no-one understands, because a lot of nice and helpful people will waste time in reading them and trying to understand them before they resign. They want to help you, but you do not allow them to do it. It's not a problem if you have difficulties with the English language, which the users mailing list deploys. We try to be helpful anyway. It *is*, however, a problem if you do not provide the details that are necessary to understand what you want. It's in your own interest to make others understand. After all, it's *you* who wants a specific problem solved. This will help you in writing better questions: http://www.catb.org/~esr/faqs/smart-questions.html > It is easy to tell me rtfm We cannot tell you "RTFM" as long as we do not even know which manual you need to read. > or do not post in here. We tell you not to post on this list, because it is the wrong list. Please use the users mailing list for usage related questions. This is the last time I tell you this. Stefan From wonjunchoi001 at gmail.com Thu Jul 28 10:04:51 2011 From: wonjunchoi001 at gmail.com (=?EUC-KR?B?w9a/+MHY?=) Date: Thu, 28 Jul 2011 17:04:51 +0900 Subject: [Cython] Cython 0.15 release In-Reply-To: <4E31150A.7090603@behnel.de> References: <4E28FBA2.9020005@behnel.de> <4E2D09EA.9030507@behnel.de> <4E3101D5.6090407@behnel.de> <866964a3-e5ed-4681-889b-45448ab20bb4@email.android.com> <4E31150A.7090603@behnel.de> Message-ID: OK Nervermind! The problem is that I don't know my question is wrong unless someone tell me "what are you talking about?" or "Please describe in detail" and Sorry for posting in here. 2011/7/28 Stefan Behnel > ???, 28.07.2011 09:31: > > I had posted several issues in detail on the list >> but there was no answer so I deleted it several times.. on the cython >> users >> list. >> I just want to hear the answer(like a "describe in detail") even though >> the >> answer guy doesn't know the solution. >> I think no answer is bothering act. >> > > It's much more "bothering" to ask questions that no-one understands, > because a lot of nice and helpful people will waste time in reading them and > trying to understand them before they resign. They want to help you, but you > do not allow them to do it. > > It's not a problem if you have difficulties with the English language, > which the users mailing list deploys. We try to be helpful anyway. > > It *is*, however, a problem if you do not provide the details that are > necessary to understand what you want. It's in your own interest to make > others understand. After all, it's *you* who wants a specific problem > solved. > > This will help you in writing better questions: > > http://www.catb.org/~esr/faqs/**smart-questions.html > > > > It is easy to tell me rtfm >> > > We cannot tell you "RTFM" as long as we do not even know which manual you > need to read. > > > > or do not post in here. >> > > We tell you not to post on this list, because it is the wrong list. Please > use the users mailing list for usage related questions. > > This is the last time I tell you this. > > > Stefan > ______________________________**_________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/**mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Jul 29 07:06:20 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 29 Jul 2011 07:06:20 +0200 Subject: [Cython] GCC 4.6 unused-but-set-variable warnings In-Reply-To: <878vt54w22.fsf@inspiron.ap.columbia.edu> References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu> <4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org> <4DF5C85D.3010306@behnel.de> <878vt54w22.fsf@inspiron.ap.columbia.edu> Message-ID: <4E323FCC.8010002@behnel.de> [moving this here from cython-users] Nikolaus Rath, 13.06.2011 16:59: > Stefan Behnel writes: >> Nikolaus Rath, 13.06.2011 01:18: >>> Stefan Behnel writes: >>>> Nikolaus Rath, 03.06.2011 23:24: >>>>> Cython 0.14 generated code triggers lots of unused-but-set-variable >>>>> warnings (which are new in GCC 4.6). >>>> >>>> Hmm, that's annoying. We actually generate a lot of useless "dangling >>>> reset to 0" statements for safety (and also as a bit of a "now unused" >>>> hint to the C compiler). >>>> >>>> We could get rid of most of them based on control flow analysis, but I >>>> actually like having them there, just in case we accidentally generate >>>> broken code at some point. A NULL pointer is a lot safer than an >>>> arbitrarily dangling address, especially when it comes to CPython heap >>>> allocated memory (which doesn't necessarily get freed, so you may not >>>> get a quick segfault but a read from dead memory). >>> >>> I don't quite understand. The way to get rid of the warnings is to >>> remove the unnecessary variables. How can this result in a dangling >>> address? You'll get a syntax error from gcc about using an undeclared >>> variable. >> >> Ah, sorry. I misunderstood the meaning of the warning message as >> pointing at unnecessary assignments in general. Could you post a C >> code snippet (and the corresponding Cython code) that triggers this? I >> don't have gcc 4.6 available for testing. > > Sure. It's apparently triggered by unused function arguments: > > $ cat test.pyx > class Operations(object): > def handle_exc(self, fn, exc): > pass > > $ cython test.pyx > > $ gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c test.c -o test.o -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -Werror -Wall -Wextra -Wconversion -Wno-unused-parameter -Wno-sign-conversion -fno-strict-aliasing test.c: > test.c:454:13: warning: variable ?__pyx_v_exc? set but not used [-Wunused-but-set-variable] > test.c:453:13: warning: variable ?__pyx_v_fn? set but not used [-Wunused-but-set-variable] > test.c:452:13: warning: variable ?__pyx_v_self? set but not used [-Wunused-but-set-variable] http://trac.cython.org/cython_trac/ticket/704 I think this is something that the control flow analysis could deal with. Basically, every parameter that turns out to be unused in the function (and this most likely only happens with parameters) could be ignored, unless it requires a type conversion with potential side effects. In that (unlikely) case, we could still drop the final assignment to the variable itself. Stefan From vitja.makarov at gmail.com Fri Jul 29 08:34:35 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 29 Jul 2011 10:34:35 +0400 Subject: [Cython] GCC 4.6 unused-but-set-variable warnings In-Reply-To: <4E323FCC.8010002@behnel.de> References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu> <4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org> <4DF5C85D.3010306@behnel.de> <878vt54w22.fsf@inspiron.ap.columbia.edu> <4E323FCC.8010002@behnel.de> Message-ID: 2011/7/29 Stefan Behnel : > [moving this here from cython-users] > > Nikolaus Rath, 13.06.2011 16:59: >> >> Stefan Behnel writes: >>> >>> Nikolaus Rath, 13.06.2011 01:18: >>>> >>>> Stefan Behnel writes: >>>>> >>>>> Nikolaus Rath, 03.06.2011 23:24: >>>>>> >>>>>> Cython 0.14 generated code triggers lots of ?unused-but-set-variable >>>>>> warnings (which are new in GCC 4.6). >>>>> >>>>> Hmm, that's annoying. We actually generate a lot of useless "dangling >>>>> reset to 0" statements for safety (and also as a bit of a "now unused" >>>>> hint to the C compiler). >>>>> >>>>> We could get rid of most of them based on control flow analysis, but I >>>>> actually like having them there, just in case we accidentally generate >>>>> broken code at some point. A NULL pointer is a lot safer than an >>>>> arbitrarily dangling address, especially when it comes to CPython heap >>>>> allocated memory (which doesn't necessarily get freed, so you may not >>>>> get a quick segfault but a read from dead memory). >>>> >>>> I don't quite understand. The way to get rid of the warnings is to >>>> remove the unnecessary variables. How can this result in a dangling >>>> address? You'll get a syntax error from gcc about using an undeclared >>>> variable. >>> >>> Ah, sorry. I misunderstood the meaning of the warning message as >>> pointing at unnecessary assignments in general. Could you post a C >>> code snippet (and the corresponding Cython code) that triggers this? I >>> don't have gcc 4.6 available for testing. >> >> Sure. It's apparently triggered by unused function arguments: >> >> $ cat test.pyx >> class Operations(object): >> ? ? def handle_exc(self, fn, exc): >> ? ? ? ? pass >> >> $ cython test.pyx >> >> $ gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall >> -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c test.c -o test.o >> -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -Werror -Wall -Wextra >> -Wconversion -Wno-unused-parameter -Wno-sign-conversion -fno-strict-aliasing >> test.c: >> test.c:454:13: warning: variable ?__pyx_v_exc? set but not used >> [-Wunused-but-set-variable] >> test.c:453:13: warning: variable ?__pyx_v_fn? set but not used >> [-Wunused-but-set-variable] >> test.c:452:13: warning: variable ?__pyx_v_self? set but not used >> [-Wunused-but-set-variable] > > http://trac.cython.org/cython_trac/ticket/704 > > I think this is something that the control flow analysis could deal with. > Basically, every parameter that turns out to be unused in the function (and > this most likely only happens with parameters) could be ignored, unless it > requires a type conversion with potential side effects. In that (unlikely) > case, we could still drop the final assignment to the variable itself. > There is entry.cf_unused flag it's set when the whole entry is not used. -- vitja. From vitja.makarov at gmail.com Fri Jul 29 08:38:04 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 29 Jul 2011 10:38:04 +0400 Subject: [Cython] GCC 4.6 unused-but-set-variable warnings In-Reply-To: References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu> <4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org> <4DF5C85D.3010306@behnel.de> <878vt54w22.fsf@inspiron.ap.columbia.edu> <4E323FCC.8010002@behnel.de> Message-ID: 2011/7/29 Vitja Makarov : > 2011/7/29 Stefan Behnel : >> [moving this here from cython-users] >> >> Nikolaus Rath, 13.06.2011 16:59: >>> >>> Stefan Behnel writes: >>>> >>>> Nikolaus Rath, 13.06.2011 01:18: >>>>> >>>>> Stefan Behnel writes: >>>>>> >>>>>> Nikolaus Rath, 03.06.2011 23:24: >>>>>>> >>>>>>> Cython 0.14 generated code triggers lots of ?unused-but-set-variable >>>>>>> warnings (which are new in GCC 4.6). >>>>>> >>>>>> Hmm, that's annoying. We actually generate a lot of useless "dangling >>>>>> reset to 0" statements for safety (and also as a bit of a "now unused" >>>>>> hint to the C compiler). >>>>>> >>>>>> We could get rid of most of them based on control flow analysis, but I >>>>>> actually like having them there, just in case we accidentally generate >>>>>> broken code at some point. A NULL pointer is a lot safer than an >>>>>> arbitrarily dangling address, especially when it comes to CPython heap >>>>>> allocated memory (which doesn't necessarily get freed, so you may not >>>>>> get a quick segfault but a read from dead memory). >>>>> >>>>> I don't quite understand. The way to get rid of the warnings is to >>>>> remove the unnecessary variables. How can this result in a dangling >>>>> address? You'll get a syntax error from gcc about using an undeclared >>>>> variable. >>>> >>>> Ah, sorry. I misunderstood the meaning of the warning message as >>>> pointing at unnecessary assignments in general. Could you post a C >>>> code snippet (and the corresponding Cython code) that triggers this? I >>>> don't have gcc 4.6 available for testing. >>> >>> Sure. It's apparently triggered by unused function arguments: >>> >>> $ cat test.pyx >>> class Operations(object): >>> ? ? def handle_exc(self, fn, exc): >>> ? ? ? ? pass >>> >>> $ cython test.pyx >>> >>> $ gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall >>> -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c test.c -o test.o >>> -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -Werror -Wall -Wextra >>> -Wconversion -Wno-unused-parameter -Wno-sign-conversion -fno-strict-aliasing >>> test.c: >>> test.c:454:13: warning: variable ?__pyx_v_exc? set but not used >>> [-Wunused-but-set-variable] >>> test.c:453:13: warning: variable ?__pyx_v_fn? set but not used >>> [-Wunused-but-set-variable] >>> test.c:452:13: warning: variable ?__pyx_v_self? set but not used >>> [-Wunused-but-set-variable] >> >> http://trac.cython.org/cython_trac/ticket/704 >> >> I think this is something that the control flow analysis could deal with. >> Basically, every parameter that turns out to be unused in the function (and >> this most likely only happens with parameters) could be ignored, unless it >> requires a type conversion with potential side effects. In that (unlikely) >> case, we could still drop the final assignment to the variable itself. >> > > There is entry.cf_unused flag it's set when the whole entry is not used. > ((6af313c...)) vitja at vitja-laptop:~/work/cython-vitek$ python cython.py -X warn.unused_arg=True test.pyx warning: test.pyx:2:19: Unused argument 'self' warning: test.pyx:2:25: Unused argument 'fn' warning: test.pyx:2:29: Unused argument 'exc' -- vitja. From stefan_ml at behnel.de Fri Jul 29 08:50:59 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 29 Jul 2011 08:50:59 +0200 Subject: [Cython] GCC 4.6 unused-but-set-variable warnings In-Reply-To: References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu> <4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org> <4DF5C85D.3010306@behnel.de> <878vt54w22.fsf@inspiron.ap.columbia.edu> <4E323FCC.8010002@behnel.de> Message-ID: <4E325853.5070207@behnel.de> Vitja Makarov, 29.07.2011 08:38: > 2011/7/29 Vitja Makarov: >> 2011/7/29 Stefan Behnel: >>> Nikolaus Rath, 13.06.2011 16:59: >>>> $ gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall >>>> -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c test.c -o test.o >>>> -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -Werror -Wall -Wextra >>>> -Wconversion -Wno-unused-parameter -Wno-sign-conversion -fno-strict-aliasing >>>> test.c: >>>> test.c:454:13: warning: variable ?__pyx_v_exc? set but not used >>>> [-Wunused-but-set-variable] >>>> test.c:453:13: warning: variable ?__pyx_v_fn? set but not used >>>> [-Wunused-but-set-variable] >>>> test.c:452:13: warning: variable ?__pyx_v_self? set but not used >>>> [-Wunused-but-set-variable] >>> >>> http://trac.cython.org/cython_trac/ticket/704 >>> >>> I think this is something that the control flow analysis could deal with. >>> Basically, every parameter that turns out to be unused in the function (and >>> this most likely only happens with parameters) could be ignored, unless it >>> requires a type conversion with potential side effects. In that (unlikely) >>> case, we could still drop the final assignment to the variable itself. >> >> There is entry.cf_unused flag it's set when the whole entry is not used. > > ((6af313c...)) vitja at vitja-laptop:~/work/cython-vitek$ python > cython.py -X warn.unused_arg=True test.pyx > warning: test.pyx:2:19: Unused argument 'self' > warning: test.pyx:2:25: Unused argument 'fn' > warning: test.pyx:2:29: Unused argument 'exc' Cool. That's what I expected. :) Want to give it a try? The argument unpacking code has traditionally been my domain, and it's somewhat complex, but it should still be easy to stick this in. Note that there is more than one place where this needs to be handled, as the code is structured into different cases based on the type of signature. Otherwise, I can see if I find a bit of time over the weekend. Stefan From vitja.makarov at gmail.com Fri Jul 29 10:08:06 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 29 Jul 2011 12:08:06 +0400 Subject: [Cython] GCC 4.6 unused-but-set-variable warnings In-Reply-To: <4E325853.5070207@behnel.de> References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu> <4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org> <4DF5C85D.3010306@behnel.de> <878vt54w22.fsf@inspiron.ap.columbia.edu> <4E323FCC.8010002@behnel.de> <4E325853.5070207@behnel.de> Message-ID: 2011/7/29 Stefan Behnel : > Vitja Makarov, 29.07.2011 08:38: >> >> 2011/7/29 Vitja Makarov: >>> >>> 2011/7/29 Stefan Behnel: >>>> >>>> Nikolaus Rath, 13.06.2011 16:59: >>>>> >>>>> $ gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall >>>>> -Wstrict-prototypes -fPIC -I/usr/include/python2.6 -c test.c -o test.o >>>>> -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -Werror -Wall -Wextra >>>>> -Wconversion -Wno-unused-parameter -Wno-sign-conversion >>>>> -fno-strict-aliasing >>>>> test.c: >>>>> test.c:454:13: warning: variable ?__pyx_v_exc? set but not used >>>>> [-Wunused-but-set-variable] >>>>> test.c:453:13: warning: variable ?__pyx_v_fn? set but not used >>>>> [-Wunused-but-set-variable] >>>>> test.c:452:13: warning: variable ?__pyx_v_self? set but not used >>>>> [-Wunused-but-set-variable] >>>> >>>> http://trac.cython.org/cython_trac/ticket/704 >>>> >>>> I think this is something that the control flow analysis could deal >>>> with. >>>> Basically, every parameter that turns out to be unused in the function >>>> (and >>>> this most likely only happens with parameters) could be ignored, unless >>>> it >>>> requires a type conversion with potential side effects. In that >>>> (unlikely) >>>> case, we could still drop the final assignment to the variable itself. >>> >>> There is entry.cf_unused flag it's set when the whole entry is not used. >> >> ((6af313c...)) vitja at vitja-laptop:~/work/cython-vitek$ python >> cython.py -X warn.unused_arg=True test.pyx >> warning: test.pyx:2:19: Unused argument 'self' >> warning: test.pyx:2:25: Unused argument 'fn' >> warning: test.pyx:2:29: Unused argument 'exc' > > Cool. That's what I expected. :) > > Want to give it a try? The argument unpacking code has traditionally been my > domain, and it's somewhat complex, but it should still be easy to stick this > in. Note that there is more than one place where this needs to be handled, > as the code is structured into different cases based on the type of > signature. > > Otherwise, I can see if I find a bit of time over the weekend. > I will try to find some time. Btw this issue isn't critical and even isn't a bug at all. -- vitja. From stefan_ml at behnel.de Fri Jul 29 10:23:27 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 29 Jul 2011 10:23:27 +0200 Subject: [Cython] GCC 4.6 unused-but-set-variable warnings In-Reply-To: References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu> <4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org> <4DF5C85D.3010306@behnel.de> <878vt54w22.fsf@inspiron.ap.columbia.edu> <4E323FCC.8010002@behnel.de> <4E325853.5070207@behnel.de> Message-ID: <4E326DFF.2090307@behnel.de> Vitja Makarov, 29.07.2011 10:08: > this issue isn't critical and even isn't a bug at all. Agreed. It's nothing that needs to be done for 0.15. I just thought you might be interested. :D Stefan From markflorisson88 at gmail.com Fri Jul 29 10:33:24 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 29 Jul 2011 10:33:24 +0200 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: <201107250804.10446.Arfrever.FTA@gmail.com> References: <201107250804.10446.Arfrever.FTA@gmail.com> Message-ID: On 25 July 2011 08:03, Arfrever Frehtes Taifersar Arahesis wrote: > There are 4 test failures with Python 2.6, 2.7, 3.1 and 3.2. > Output with Python 2.7: > > ====================================================================== > FAIL: test_nested_break_continue (line 331) (parallel.__test__) > Doctest: parallel.__test__.test_nested_break_continue (line 331) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest > ? ?raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for parallel.__test__.test_nested_break_continue (line 331) > ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line unknown line number, in test_nested_break_continue (line 331) > > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line ?, in parallel.__test__.test_nested_break_continue (line 331) > Failed example: > ? ?test_nested_break_continue() > Expected: > ? ?6 7 6 7 > ? ?8 > Got: > ? ?6 0 -1160725808 -1160725808 > ? ?8 > > > ====================================================================== > FAIL: test_propagation (line 49) (parallel.__test__) > Doctest: parallel.__test__.test_propagation (line 49) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest > ? ?raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for parallel.__test__.test_propagation (line 49) > ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line unknown line number, in test_propagation (line 49) > > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line ?, in parallel.__test__.test_propagation (line 49) > Failed example: > ? ?test_propagation() > Expected: > ? ?(9, 9, 9, 9, 450, 450) > Got: > ? ?(9, 0, 9, -1160725808, 0, 0) > > > ====================================================================== > FAIL: test_nested_break_continue (line 331) (parallel.__test__) > Doctest: parallel.__test__.test_nested_break_continue (line 331) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest > ? ?raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for parallel.__test__.test_nested_break_continue (line 331) > ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line unknown line number, in test_nested_break_continue (line 331) > > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line ?, in parallel.__test__.test_nested_break_continue (line 331) > Failed example: > ? ?test_nested_break_continue() > Expected: > ? ?6 7 6 7 > ? ?8 > Got: > ? ?6 0 -1160725808 -1160725808 > ? ?8 > > > ====================================================================== > FAIL: test_propagation (line 49) (parallel.__test__) > Doctest: parallel.__test__.test_propagation (line 49) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest > ? ?raise self.failureException(self.format_failure(new.getvalue())) > AssertionError: Failed doctest test for parallel.__test__.test_propagation (line 49) > ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line unknown line number, in test_propagation (line 49) > > ---------------------------------------------------------------------- > File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line ?, in parallel.__test__.test_propagation (line 49) > Failed example: > ? ?test_propagation() > Expected: > ? ?(9, 9, 9, 9, 450, 450) > Got: > ? ?(9, 0, 9, -1160725808, 0, 0) > > > ---------------------------------------------------------------------- > Ran 5554 tests in 1509.246s > > FAILED (failures=4) > ALL DONE > > -- > Arfrever Frehtes Taifersar Arahesis > > _______________________________________________ > Cython-dev mailing list > Cython-dev at codespeak.net > http://codespeak.net/mailman/listinfo/cython-dev > > Unfortunately due to a bug in gcc 4.5, Dag and I have concluded that we must disable nested parallelism support in Cython, as there is no suitable workaround. (The issue is that the bug doesn't propagate lastprivates, which results in random values for inner privates.) The reduction fails in the test due to yet another bug, which can be worked around easily. So Robert, please wait with the next release candidate until I have disabled nesting and the corresponding tests and documentation. From vitja.makarov at gmail.com Fri Jul 29 10:44:04 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 29 Jul 2011 12:44:04 +0400 Subject: [Cython] GCC 4.6 unused-but-set-variable warnings In-Reply-To: <4E326DFF.2090307@behnel.de> References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu> <4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org> <4DF5C85D.3010306@behnel.de> <878vt54w22.fsf@inspiron.ap.columbia.edu> <4E323FCC.8010002@behnel.de> <4E325853.5070207@behnel.de> <4E326DFF.2090307@behnel.de> Message-ID: 2011/7/29 Stefan Behnel : > Vitja Makarov, 29.07.2011 10:08: >> >> this issue isn't critical and even isn't a bug at all. > > Agreed. It's nothing that needs to be done for 0.15. I just thought you > might be interested. :D > Yeah, I tried to do this once but I've found some problems with buffer variables. What to do about local variables: def foo(): a = 1 'a' is unused here -- vitja. From stefan_ml at behnel.de Fri Jul 29 10:50:35 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 29 Jul 2011 10:50:35 +0200 Subject: [Cython] GCC 4.6 unused-but-set-variable warnings In-Reply-To: References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu> <4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org> <4DF5C85D.3010306@behnel.de> <878vt54w22.fsf@inspiron.ap.columbia.edu> <4E323FCC.8010002@behnel.de> <4E325853.5070207@behnel.de> <4E326DFF.2090307@behnel.de> Message-ID: <4E32745B.7040501@behnel.de> Vitja Makarov, 29.07.2011 10:44: > 2011/7/29 Stefan Behnel: >> Vitja Makarov, 29.07.2011 10:08: >>> >>> this issue isn't critical and even isn't a bug at all. >> >> Agreed. It's nothing that needs to be done for 0.15. I just thought you >> might be interested. :D >> > > Yeah, I tried to do this once but I've found some problems with buffer > variables. > > What to do about local variables: > > def foo(): > a = 1 > > 'a' is unused here That's up to the user to fix. However, there may be restrictions regarding the signature (inheritance etc.) that the users cannot control, so unused *parameters* must not produce warnings. Stefan From vitja.makarov at gmail.com Fri Jul 29 10:55:09 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Fri, 29 Jul 2011 12:55:09 +0400 Subject: [Cython] GCC 4.6 unused-but-set-variable warnings In-Reply-To: <4E32745B.7040501@behnel.de> References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu> <4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org> <4DF5C85D.3010306@behnel.de> <878vt54w22.fsf@inspiron.ap.columbia.edu> <4E323FCC.8010002@behnel.de> <4E325853.5070207@behnel.de> <4E326DFF.2090307@behnel.de> <4E32745B.7040501@behnel.de> Message-ID: 2011/7/29 Stefan Behnel : > Vitja Makarov, 29.07.2011 10:44: >> >> 2011/7/29 Stefan Behnel: >>> >>> Vitja Makarov, 29.07.2011 10:08: >>>> >>>> this issue isn't critical and even isn't a bug at all. >>> >>> Agreed. It's nothing that needs to be done for 0.15. I just thought you >>> might be interested. :D >>> >> >> Yeah, I tried to do this once but I've found some problems with buffer >> variables. >> >> What to do about local variables: >> >> def foo(): >> ? ? a = 1 >> >> 'a' is unused here > > That's up to the user to fix. However, there may be restrictions regarding > the signature (inheritance etc.) that the users cannot control, so unused > *parameters* must not produce warnings. > Sure. Because of that there is separate warn.unused_args option :) -- vitja. From stefan_ml at behnel.de Fri Jul 29 11:11:43 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 29 Jul 2011 11:11:43 +0200 Subject: [Cython] GCC 4.6 unused-but-set-variable warnings In-Reply-To: References: <87oc2eehgv.fsf@inspiron.ap.columbia.edu> <4DF51F03.8010806@behnel.de> <87d3iihc5p.fsf@vostro.rath.org> <4DF5C85D.3010306@behnel.de> <878vt54w22.fsf@inspiron.ap.columbia.edu> <4E323FCC.8010002@behnel.de> <4E325853.5070207@behnel.de> <4E326DFF.2090307@behnel.de> <4E32745B.7040501@behnel.de> Message-ID: <4E32794F.5040007@behnel.de> Vitja Makarov, 29.07.2011 10:55: > 2011/7/29 Stefan Behnel: >> Vitja Makarov, 29.07.2011 10:44: >>> >>> 2011/7/29 Stefan Behnel: >>>> >>>> Vitja Makarov, 29.07.2011 10:08: >>>>> >>>>> this issue isn't critical and even isn't a bug at all. >>>> >>>> Agreed. It's nothing that needs to be done for 0.15. I just thought you >>>> might be interested. :D >>>> >>> >>> Yeah, I tried to do this once but I've found some problems with buffer >>> variables. >>> >>> What to do about local variables: >>> >>> def foo(): >>> a = 1 >>> >>> 'a' is unused here >> >> That's up to the user to fix. However, there may be restrictions regarding >> the signature (inheritance etc.) that the users cannot control, so unused >> *parameters* must not produce warnings. >> > > Sure. Because of that there is separate warn.unused_args option :) With the caveat that gcc 4.6 produces a warning with -Wall for them, because it cannot know that they originally were parameters in the Cython code. Stefan From arfrever.fta at gmail.com Fri Jul 29 19:18:38 2011 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Fri, 29 Jul 2011 19:18:38 +0200 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: References: <201107250808.19181.Arfrever.FTA@gmail.com> Message-ID: <201107291918.39717.Arfrever.FTA@gmail.com> 2011-07-26 13:00:10 mark florisson napisa?(a): > Unfortunately the output of the testrunner is rather useless here. > Could you run the tests with --no-cleanup and attach the .c and .cpp > files (found in build/run/c/numpy_cimport.c, etc)? I think something > is up with your numpy version and headers. The numpy tests should be > automatically skipped if numpy is not available. I have found the following lines in earlier part of output: runTest (__main__.CythonRunTestCase) compiling (c) and running numpy_bufacc_T155 ... numpy_bufacc_T155.c: In function ?PyInit_numpy_bufacc_T155?: numpy_bufacc_T155.c:3652:5: warning: ?return? with no value, in function returning non-void numpy_bufacc_T155 () Doctest: numpy_bufacc_T155 ... ok runTest (__main__.CythonRunTestCase) compiling (cpp) and running numpy_bufacc_T155 ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ numpy_bufacc_T155.cpp: In function ?PyObject* PyInit_numpy_bufacc_T155()?: numpy_bufacc_T155.cpp:3652:5: error: return-statement with no value, in function returning ?PyObject*? ERROR runTest (__main__.CythonRunTestCase) compiling (c) and running numpy_cimport ... numpy_cimport.c: In function ?PyInit_numpy_cimport?: numpy_cimport.c:3327:5: warning: ?return? with no value, in function returning non-void numpy_cimport () Doctest: numpy_cimport ... ok runTest (__main__.CythonRunTestCase) compiling (cpp) and running numpy_cimport ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ numpy_cimport.cpp: In function ?PyObject* PyInit_numpy_cimport()?: numpy_cimport.cpp:3327:5: error: return-statement with no value, in function returning ?PyObject*? ERROR runTest (__main__.CythonRunTestCase) compiling (c) and running numpy_parallel ... numpy_parallel.c: In function ?PyInit_numpy_parallel?: numpy_parallel.c:3824:5: warning: ?return? with no value, in function returning non-void test_parallel_numpy_arrays (line 11) (numpy_parallel.__test__) Doctest: numpy_parallel.__test__.test_parallel_numpy_arrays (line 11) ... ok runTest (__main__.CythonRunTestCase) compiling (cpp) and running numpy_parallel ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ numpy_parallel.cpp: In function ?PyObject* PyInit_numpy_parallel()?: numpy_parallel.cpp:3824:5: error: return-statement with no value, in function returning ?PyObject*? ERROR runTest (__main__.CythonRunTestCase) compiling (c) and running numpy_test ... numpy_test.c: In function ?PyInit_numpy_test?: numpy_test.c:11604:5: warning: ?return? with no value, in function returning non-void numpy_test () Doctest: numpy_test ... ok runTest (__main__.CythonRunTestCase) compiling (cpp) and running numpy_test ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ numpy_test.cpp: In function ?PyObject* PyInit_numpy_test()?: numpy_test.cpp:11604:5: error: return-statement with no value, in function returning ?PyObject*? ERROR Relevant source files: http://people.apache.org/~Arfrever/numpy_bufacc_T155.c http://people.apache.org/~Arfrever/numpy_bufacc_T155.cpp http://people.apache.org/~Arfrever/numpy_cimport.c http://people.apache.org/~Arfrever/numpy_cimport.cpp http://people.apache.org/~Arfrever/numpy_parallel.c http://people.apache.org/~Arfrever/numpy_parallel.cpp http://people.apache.org/~Arfrever/numpy_test.c http://people.apache.org/~Arfrever/numpy_test.cpp -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From robertwb at math.washington.edu Fri Jul 29 20:06:31 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 29 Jul 2011 11:06:31 -0700 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: References: <201107250804.10446.Arfrever.FTA@gmail.com> Message-ID: On Fri, Jul 29, 2011 at 1:33 AM, mark florisson wrote: > On 25 July 2011 08:03, Arfrever Frehtes Taifersar Arahesis > wrote: >> There are 4 test failures with Python 2.6, 2.7, 3.1 and 3.2. >> Output with Python 2.7: >> >> ====================================================================== >> FAIL: test_nested_break_continue (line 331) (parallel.__test__) >> Doctest: parallel.__test__.test_nested_break_continue (line 331) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest >> ? ?raise self.failureException(self.format_failure(new.getvalue())) >> AssertionError: Failed doctest test for parallel.__test__.test_nested_break_continue (line 331) >> ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line unknown line number, in test_nested_break_continue (line 331) >> >> ---------------------------------------------------------------------- >> File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line ?, in parallel.__test__.test_nested_break_continue (line 331) >> Failed example: >> ? ?test_nested_break_continue() >> Expected: >> ? ?6 7 6 7 >> ? ?8 >> Got: >> ? ?6 0 -1160725808 -1160725808 >> ? ?8 >> >> >> ====================================================================== >> FAIL: test_propagation (line 49) (parallel.__test__) >> Doctest: parallel.__test__.test_propagation (line 49) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest >> ? ?raise self.failureException(self.format_failure(new.getvalue())) >> AssertionError: Failed doctest test for parallel.__test__.test_propagation (line 49) >> ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line unknown line number, in test_propagation (line 49) >> >> ---------------------------------------------------------------------- >> File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line ?, in parallel.__test__.test_propagation (line 49) >> Failed example: >> ? ?test_propagation() >> Expected: >> ? ?(9, 9, 9, 9, 450, 450) >> Got: >> ? ?(9, 0, 9, -1160725808, 0, 0) >> >> >> ====================================================================== >> FAIL: test_nested_break_continue (line 331) (parallel.__test__) >> Doctest: parallel.__test__.test_nested_break_continue (line 331) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest >> ? ?raise self.failureException(self.format_failure(new.getvalue())) >> AssertionError: Failed doctest test for parallel.__test__.test_nested_break_continue (line 331) >> ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line unknown line number, in test_nested_break_continue (line 331) >> >> ---------------------------------------------------------------------- >> File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line ?, in parallel.__test__.test_nested_break_continue (line 331) >> Failed example: >> ? ?test_nested_break_continue() >> Expected: >> ? ?6 7 6 7 >> ? ?8 >> Got: >> ? ?6 0 -1160725808 -1160725808 >> ? ?8 >> >> >> ====================================================================== >> FAIL: test_propagation (line 49) (parallel.__test__) >> Doctest: parallel.__test__.test_propagation (line 49) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest >> ? ?raise self.failureException(self.format_failure(new.getvalue())) >> AssertionError: Failed doctest test for parallel.__test__.test_propagation (line 49) >> ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line unknown line number, in test_propagation (line 49) >> >> ---------------------------------------------------------------------- >> File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line ?, in parallel.__test__.test_propagation (line 49) >> Failed example: >> ? ?test_propagation() >> Expected: >> ? ?(9, 9, 9, 9, 450, 450) >> Got: >> ? ?(9, 0, 9, -1160725808, 0, 0) >> >> >> ---------------------------------------------------------------------- >> Ran 5554 tests in 1509.246s >> >> FAILED (failures=4) >> ALL DONE >> >> -- >> Arfrever Frehtes Taifersar Arahesis >> >> _______________________________________________ >> Cython-dev mailing list >> Cython-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/cython-dev >> >> > > Unfortunately due to a bug in gcc 4.5, Dag and I have concluded that > we must disable nested parallelism support in Cython, as there is no > suitable workaround. (The issue is that the bug doesn't propagate > lastprivates, which results in random values for inner privates.) The > reduction fails in the test due to yet another bug, which can be > worked around easily. Nice sleuthing, and thanks Arfrever for the bug report. Glad these were caught. > So Robert, please wait with the next release candidate until I have > disabled nesting and the corresponding tests and documentation. Any ETA here? (I was ironically thinking about pushing out another RC today, but there's no huge hurry.) - Robert From robertwb at math.washington.edu Fri Jul 29 20:10:27 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 29 Jul 2011 11:10:27 -0700 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: References: <4E303D89.2060304@behnel.de> <4E30560D.6050307@astro.uio.no> <4E310121.2050400@behnel.de> Message-ID: On Wed, Jul 27, 2011 at 11:32 PM, Vitja Makarov wrote: > 2011/7/28 Stefan Behnel : >> Robert Bradshaw, 27.07.2011 20:28: >>> >>> Yes, that is a good point, though if it's (really) easy, I'd rather >>> say "this is the last release supporting 2.3" rather than "oh, btw, we >>> dropped support for 2.3." >> >> I wouldn't mind either way. Let's see what the users poll gives. >> >> >>> I also think we should have a big fat #error rather than letting 2.3 >>> support just fade away (or worse, silently produce bad/incorrect >>> code). >> >> With "fading out" I just meant that a) we don't even know right now if the C >> code we generate really works in Python 2.3 and b) we could just leave it >> that way and add a note to the release notes that it's no longer actively >> supported but on a "I feel lucky" basis. >> >> But I agree that it's better to let the C compilation fail loudly, so that >> users know what the problem is. >> > > That could be a warning not an error. > > It's fun: with cython you can use decorators, generators and more with py2.3, > may be it's better to drop support after 0.15? > > I hope it would be easy to fix this error: > https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-py23-c/871/console I'll look into it. If it's trivial I'll fix it and we'll call 2.3 supported, if not we'll not worry about it. In general, it sounds like people are fine with dropping support for 2.3, and we've got one request for holding onto 2.4 a little bit longer (but dropping that is on the horizon too). - Robert From markflorisson88 at gmail.com Fri Jul 29 20:24:35 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 29 Jul 2011 20:24:35 +0200 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: References: <201107250804.10446.Arfrever.FTA@gmail.com> Message-ID: On 29 July 2011 20:06, Robert Bradshaw wrote: > On Fri, Jul 29, 2011 at 1:33 AM, mark florisson > wrote: >> On 25 July 2011 08:03, Arfrever Frehtes Taifersar Arahesis >> wrote: >>> There are 4 test failures with Python 2.6, 2.7, 3.1 and 3.2. >>> Output with Python 2.7: >>> >>> ====================================================================== >>> FAIL: test_nested_break_continue (line 331) (parallel.__test__) >>> Doctest: parallel.__test__.test_nested_break_continue (line 331) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest >>> ? ?raise self.failureException(self.format_failure(new.getvalue())) >>> AssertionError: Failed doctest test for parallel.__test__.test_nested_break_continue (line 331) >>> ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line unknown line number, in test_nested_break_continue (line 331) >>> >>> ---------------------------------------------------------------------- >>> File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line ?, in parallel.__test__.test_nested_break_continue (line 331) >>> Failed example: >>> ? ?test_nested_break_continue() >>> Expected: >>> ? ?6 7 6 7 >>> ? ?8 >>> Got: >>> ? ?6 0 -1160725808 -1160725808 >>> ? ?8 >>> >>> >>> ====================================================================== >>> FAIL: test_propagation (line 49) (parallel.__test__) >>> Doctest: parallel.__test__.test_propagation (line 49) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest >>> ? ?raise self.failureException(self.format_failure(new.getvalue())) >>> AssertionError: Failed doctest test for parallel.__test__.test_propagation (line 49) >>> ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line unknown line number, in test_propagation (line 49) >>> >>> ---------------------------------------------------------------------- >>> File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line ?, in parallel.__test__.test_propagation (line 49) >>> Failed example: >>> ? ?test_propagation() >>> Expected: >>> ? ?(9, 9, 9, 9, 450, 450) >>> Got: >>> ? ?(9, 0, 9, -1160725808, 0, 0) >>> >>> >>> ====================================================================== >>> FAIL: test_nested_break_continue (line 331) (parallel.__test__) >>> Doctest: parallel.__test__.test_nested_break_continue (line 331) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest >>> ? ?raise self.failureException(self.format_failure(new.getvalue())) >>> AssertionError: Failed doctest test for parallel.__test__.test_nested_break_continue (line 331) >>> ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line unknown line number, in test_nested_break_continue (line 331) >>> >>> ---------------------------------------------------------------------- >>> File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line ?, in parallel.__test__.test_nested_break_continue (line 331) >>> Failed example: >>> ? ?test_nested_break_continue() >>> Expected: >>> ? ?6 7 6 7 >>> ? ?8 >>> Got: >>> ? ?6 0 -1160725808 -1160725808 >>> ? ?8 >>> >>> >>> ====================================================================== >>> FAIL: test_propagation (line 49) (parallel.__test__) >>> Doctest: parallel.__test__.test_propagation (line 49) >>> ---------------------------------------------------------------------- >>> Traceback (most recent call last): >>> ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest >>> ? ?raise self.failureException(self.format_failure(new.getvalue())) >>> AssertionError: Failed doctest test for parallel.__test__.test_propagation (line 49) >>> ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line unknown line number, in test_propagation (line 49) >>> >>> ---------------------------------------------------------------------- >>> File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line ?, in parallel.__test__.test_propagation (line 49) >>> Failed example: >>> ? ?test_propagation() >>> Expected: >>> ? ?(9, 9, 9, 9, 450, 450) >>> Got: >>> ? ?(9, 0, 9, -1160725808, 0, 0) >>> >>> >>> ---------------------------------------------------------------------- >>> Ran 5554 tests in 1509.246s >>> >>> FAILED (failures=4) >>> ALL DONE >>> >>> -- >>> Arfrever Frehtes Taifersar Arahesis >>> >>> _______________________________________________ >>> Cython-dev mailing list >>> Cython-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/cython-dev >>> >>> >> >> Unfortunately due to a bug in gcc 4.5, Dag and I have concluded that >> we must disable nested parallelism support in Cython, as there is no >> suitable workaround. (The issue is that the bug doesn't propagate >> lastprivates, which results in random values for inner privates.) The >> reduction fails in the test due to yet another bug, which can be >> worked around easily. > > Nice sleuthing, and thanks Arfrever for the bug report. Glad these were caught. > >> So Robert, please wait with the next release candidate until I have >> disabled nesting and the corresponding tests and documentation. > > Any ETA here? (I was ironically thinking about pushing out another RC > today, but there's no huge hurry.) > > - Robert > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > I pushed to master, but the docs are out of date, so I'm updating them. I also want to run the testsuite with gcc-4.5, and finally have a look at those numpy issues. At least the bug is fixed now in 4.5: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49897 . A pretty quick fix! From markflorisson88 at gmail.com Fri Jul 29 20:51:11 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Fri, 29 Jul 2011 20:51:11 +0200 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: <201107291918.39717.Arfrever.FTA@gmail.com> References: <201107250808.19181.Arfrever.FTA@gmail.com> <201107291918.39717.Arfrever.FTA@gmail.com> Message-ID: On 29 July 2011 19:18, Arfrever Frehtes Taifersar Arahesis wrote: > 2011-07-26 13:00:10 mark florisson napisa?(a): >> Unfortunately the output of the testrunner is rather useless here. >> Could you run the tests with --no-cleanup and attach the .c and .cpp >> files (found in build/run/c/numpy_cimport.c, etc)? I think something >> is up with your numpy version and headers. The numpy tests should be >> automatically skipped if numpy is not available. > > I have found the following lines in earlier part of output: > > runTest (__main__.CythonRunTestCase) > compiling (c) and running numpy_bufacc_T155 ... numpy_bufacc_T155.c: In function ?PyInit_numpy_bufacc_T155?: > numpy_bufacc_T155.c:3652:5: warning: ?return? with no value, in function returning non-void > numpy_bufacc_T155 () > Doctest: numpy_bufacc_T155 ... ok > runTest (__main__.CythonRunTestCase) > compiling (cpp) and running numpy_bufacc_T155 ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ > numpy_bufacc_T155.cpp: In function ?PyObject* PyInit_numpy_bufacc_T155()?: > numpy_bufacc_T155.cpp:3652:5: error: return-statement with no value, in function returning ?PyObject*? > ERROR > runTest (__main__.CythonRunTestCase) > compiling (c) and running numpy_cimport ... numpy_cimport.c: In function ?PyInit_numpy_cimport?: > numpy_cimport.c:3327:5: warning: ?return? with no value, in function returning non-void > numpy_cimport () > Doctest: numpy_cimport ... ok > runTest (__main__.CythonRunTestCase) > compiling (cpp) and running numpy_cimport ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ > numpy_cimport.cpp: In function ?PyObject* PyInit_numpy_cimport()?: > numpy_cimport.cpp:3327:5: error: return-statement with no value, in function returning ?PyObject*? > ERROR > runTest (__main__.CythonRunTestCase) > compiling (c) and running numpy_parallel ... numpy_parallel.c: In function ?PyInit_numpy_parallel?: > numpy_parallel.c:3824:5: warning: ?return? with no value, in function returning non-void > test_parallel_numpy_arrays (line 11) (numpy_parallel.__test__) > Doctest: numpy_parallel.__test__.test_parallel_numpy_arrays (line 11) ... ok > runTest (__main__.CythonRunTestCase) > compiling (cpp) and running numpy_parallel ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ > numpy_parallel.cpp: In function ?PyObject* PyInit_numpy_parallel()?: > numpy_parallel.cpp:3824:5: error: return-statement with no value, in function returning ?PyObject*? > ERROR > runTest (__main__.CythonRunTestCase) > compiling (c) and running numpy_test ... numpy_test.c: In function ?PyInit_numpy_test?: > numpy_test.c:11604:5: warning: ?return? with no value, in function returning non-void > numpy_test () > Doctest: numpy_test ... ok > runTest (__main__.CythonRunTestCase) > compiling (cpp) and running numpy_test ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ > numpy_test.cpp: In function ?PyObject* PyInit_numpy_test()?: > numpy_test.cpp:11604:5: error: return-statement with no value, in function returning ?PyObject*? > ERROR > > Relevant source files: > > http://people.apache.org/~Arfrever/numpy_bufacc_T155.c > http://people.apache.org/~Arfrever/numpy_bufacc_T155.cpp > http://people.apache.org/~Arfrever/numpy_cimport.c > http://people.apache.org/~Arfrever/numpy_cimport.cpp > http://people.apache.org/~Arfrever/numpy_parallel.c > http://people.apache.org/~Arfrever/numpy_parallel.cpp > http://people.apache.org/~Arfrever/numpy_test.c > http://people.apache.org/~Arfrever/numpy_test.cpp > > -- > Arfrever Frehtes Taifersar Arahesis > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > > The compiler complains about the init function of the module, not sure why, but the line numbers don't seem to correspond with the files you provided. The tests run fine for me on both osx, fedora, and ubuntu, all with different gcc versions, so I'm assuming something is wrong with your setup. Thanks for your report though! Feel free to retry using the latest checkout from master (https://github.com/cython/cython). Robert, I'm done with documentation and updates. From robertwb at math.washington.edu Fri Jul 29 20:58:04 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Fri, 29 Jul 2011 11:58:04 -0700 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: References: <201107250804.10446.Arfrever.FTA@gmail.com> Message-ID: On Fri, Jul 29, 2011 at 11:24 AM, mark florisson wrote: > On 29 July 2011 20:06, Robert Bradshaw wrote: >> On Fri, Jul 29, 2011 at 1:33 AM, mark florisson >> wrote: >>> On 25 July 2011 08:03, Arfrever Frehtes Taifersar Arahesis >>> wrote: >>>> There are 4 test failures with Python 2.6, 2.7, 3.1 and 3.2. >>>> Output with Python 2.7: >>>> >>>> ====================================================================== >>>> FAIL: test_nested_break_continue (line 331) (parallel.__test__) >>>> Doctest: parallel.__test__.test_nested_break_continue (line 331) >>>> ---------------------------------------------------------------------- >>>> Traceback (most recent call last): >>>> ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest >>>> ? ?raise self.failureException(self.format_failure(new.getvalue())) >>>> AssertionError: Failed doctest test for parallel.__test__.test_nested_break_continue (line 331) >>>> ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line unknown line number, in test_nested_break_continue (line 331) >>>> >>>> ---------------------------------------------------------------------- >>>> File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line ?, in parallel.__test__.test_nested_break_continue (line 331) >>>> Failed example: >>>> ? ?test_nested_break_continue() >>>> Expected: >>>> ? ?6 7 6 7 >>>> ? ?8 >>>> Got: >>>> ? ?6 0 -1160725808 -1160725808 >>>> ? ?8 >>>> >>>> >>>> ====================================================================== >>>> FAIL: test_propagation (line 49) (parallel.__test__) >>>> Doctest: parallel.__test__.test_propagation (line 49) >>>> ---------------------------------------------------------------------- >>>> Traceback (most recent call last): >>>> ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest >>>> ? ?raise self.failureException(self.format_failure(new.getvalue())) >>>> AssertionError: Failed doctest test for parallel.__test__.test_propagation (line 49) >>>> ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line unknown line number, in test_propagation (line 49) >>>> >>>> ---------------------------------------------------------------------- >>>> File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/c/parallel.so", line ?, in parallel.__test__.test_propagation (line 49) >>>> Failed example: >>>> ? ?test_propagation() >>>> Expected: >>>> ? ?(9, 9, 9, 9, 450, 450) >>>> Got: >>>> ? ?(9, 0, 9, -1160725808, 0, 0) >>>> >>>> >>>> ====================================================================== >>>> FAIL: test_nested_break_continue (line 331) (parallel.__test__) >>>> Doctest: parallel.__test__.test_nested_break_continue (line 331) >>>> ---------------------------------------------------------------------- >>>> Traceback (most recent call last): >>>> ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest >>>> ? ?raise self.failureException(self.format_failure(new.getvalue())) >>>> AssertionError: Failed doctest test for parallel.__test__.test_nested_break_continue (line 331) >>>> ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line unknown line number, in test_nested_break_continue (line 331) >>>> >>>> ---------------------------------------------------------------------- >>>> File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line ?, in parallel.__test__.test_nested_break_continue (line 331) >>>> Failed example: >>>> ? ?test_nested_break_continue() >>>> Expected: >>>> ? ?6 7 6 7 >>>> ? ?8 >>>> Got: >>>> ? ?6 0 -1160725808 -1160725808 >>>> ? ?8 >>>> >>>> >>>> ====================================================================== >>>> FAIL: test_propagation (line 49) (parallel.__test__) >>>> Doctest: parallel.__test__.test_propagation (line 49) >>>> ---------------------------------------------------------------------- >>>> Traceback (most recent call last): >>>> ?File "/usr/lib64/python2.7/doctest.py", line 2166, in runTest >>>> ? ?raise self.failureException(self.format_failure(new.getvalue())) >>>> AssertionError: Failed doctest test for parallel.__test__.test_propagation (line 49) >>>> ?File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line unknown line number, in test_propagation (line 49) >>>> >>>> ---------------------------------------------------------------------- >>>> File "/var/tmp/portage/dev-python/cython-0.15_rc0/work/Cython-0.15rc0/tests-2.7/run/cpp/parallel.so", line ?, in parallel.__test__.test_propagation (line 49) >>>> Failed example: >>>> ? ?test_propagation() >>>> Expected: >>>> ? ?(9, 9, 9, 9, 450, 450) >>>> Got: >>>> ? ?(9, 0, 9, -1160725808, 0, 0) >>>> >>>> >>>> ---------------------------------------------------------------------- >>>> Ran 5554 tests in 1509.246s >>>> >>>> FAILED (failures=4) >>>> ALL DONE >>>> >>>> -- >>>> Arfrever Frehtes Taifersar Arahesis >>>> >>>> _______________________________________________ >>>> Cython-dev mailing list >>>> Cython-dev at codespeak.net >>>> http://codespeak.net/mailman/listinfo/cython-dev >>>> >>>> >>> >>> Unfortunately due to a bug in gcc 4.5, Dag and I have concluded that >>> we must disable nested parallelism support in Cython, as there is no >>> suitable workaround. (The issue is that the bug doesn't propagate >>> lastprivates, which results in random values for inner privates.) The >>> reduction fails in the test due to yet another bug, which can be >>> worked around easily. >> >> Nice sleuthing, and thanks Arfrever for the bug report. Glad these were caught. >> >>> So Robert, please wait with the next release candidate until I have >>> disabled nesting and the corresponding tests and documentation. >> >> Any ETA here? (I was ironically thinking about pushing out another RC >> today, but there's no huge hurry.) >> >> - Robert >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> > > I pushed to master, but the docs are out of date, so I'm updating > them. Thanks. > I also want to run the testsuite with gcc-4.5, and finally have > a look at those numpy issues. > > At least the bug is fixed now in 4.5: > http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49897 . A pretty quick > fix! Wow, that was fast. - Robert From arfrever.fta at gmail.com Fri Jul 29 22:14:43 2011 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Fri, 29 Jul 2011 22:14:43 +0200 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: References: <201107291918.39717.Arfrever.FTA@gmail.com> Message-ID: <201107292214.44324.Arfrever.FTA@gmail.com> 2011-07-29 20:51:11 mark florisson napisa?(a): > On 29 July 2011 19:18, Arfrever Frehtes Taifersar Arahesis > wrote: > > 2011-07-26 13:00:10 mark florisson napisa?(a): > >> Unfortunately the output of the testrunner is rather useless here. > >> Could you run the tests with --no-cleanup and attach the .c and .cpp > >> files (found in build/run/c/numpy_cimport.c, etc)? I think something > >> is up with your numpy version and headers. The numpy tests should be > >> automatically skipped if numpy is not available. > > > > I have found the following lines in earlier part of output: > > > > runTest (__main__.CythonRunTestCase) > > compiling (c) and running numpy_bufacc_T155 ... numpy_bufacc_T155.c: In function ?PyInit_numpy_bufacc_T155?: > > numpy_bufacc_T155.c:3652:5: warning: ?return? with no value, in function returning non-void > > numpy_bufacc_T155 () > > Doctest: numpy_bufacc_T155 ... ok > > runTest (__main__.CythonRunTestCase) > > compiling (cpp) and running numpy_bufacc_T155 ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ > > numpy_bufacc_T155.cpp: In function ?PyObject* PyInit_numpy_bufacc_T155()?: > > numpy_bufacc_T155.cpp:3652:5: error: return-statement with no value, in function returning ?PyObject*? > > ERROR > > runTest (__main__.CythonRunTestCase) > > compiling (c) and running numpy_cimport ... numpy_cimport.c: In function ?PyInit_numpy_cimport?: > > numpy_cimport.c:3327:5: warning: ?return? with no value, in function returning non-void > > numpy_cimport () > > Doctest: numpy_cimport ... ok > > runTest (__main__.CythonRunTestCase) > > compiling (cpp) and running numpy_cimport ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ > > numpy_cimport.cpp: In function ?PyObject* PyInit_numpy_cimport()?: > > numpy_cimport.cpp:3327:5: error: return-statement with no value, in function returning ?PyObject*? > > ERROR > > runTest (__main__.CythonRunTestCase) > > compiling (c) and running numpy_parallel ... numpy_parallel.c: In function ?PyInit_numpy_parallel?: > > numpy_parallel.c:3824:5: warning: ?return? with no value, in function returning non-void > > test_parallel_numpy_arrays (line 11) (numpy_parallel.__test__) > > Doctest: numpy_parallel.__test__.test_parallel_numpy_arrays (line 11) ... ok > > runTest (__main__.CythonRunTestCase) > > compiling (cpp) and running numpy_parallel ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ > > numpy_parallel.cpp: In function ?PyObject* PyInit_numpy_parallel()?: > > numpy_parallel.cpp:3824:5: error: return-statement with no value, in function returning ?PyObject*? > > ERROR > > runTest (__main__.CythonRunTestCase) > > compiling (c) and running numpy_test ... numpy_test.c: In function ?PyInit_numpy_test?: > > numpy_test.c:11604:5: warning: ?return? with no value, in function returning non-void > > numpy_test () > > Doctest: numpy_test ... ok > > runTest (__main__.CythonRunTestCase) > > compiling (cpp) and running numpy_test ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ > > numpy_test.cpp: In function ?PyObject* PyInit_numpy_test()?: > > numpy_test.cpp:11604:5: error: return-statement with no value, in function returning ?PyObject*? > > ERROR > > > > Relevant source files: > > > > http://people.apache.org/~Arfrever/numpy_bufacc_T155.c > > http://people.apache.org/~Arfrever/numpy_bufacc_T155.cpp > > http://people.apache.org/~Arfrever/numpy_cimport.c > > http://people.apache.org/~Arfrever/numpy_cimport.cpp > > http://people.apache.org/~Arfrever/numpy_parallel.c > > http://people.apache.org/~Arfrever/numpy_parallel.cpp > > http://people.apache.org/~Arfrever/numpy_test.c > > http://people.apache.org/~Arfrever/numpy_test.cpp > > The compiler complains about the init function of the module, not sure > why, but the line numbers don't seem to correspond with the files you > provided. The tests run fine for me on both osx, fedora, and ubuntu, > all with different gcc versions, so I'm assuming something is wrong > with your setup. Actually line numbers correspond to the files provided by me. You probably tested with Python 2 instead of Python 3. The files generated by Cython contain this line: import_umath(); numpy/__ufunc_api.h defines (see 'return' at the end): #define import_umath() { UFUNC_NOFPE if (_import_umath() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "numpy.core.umath failed to import"); return; }} pyport.h of Python 2.* defines: #define PyMODINIT_FUNC void pyport.h of Python 3.* defines: #define PyMODINIT_FUNC PyObject* -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From d.s.seljebotn at astro.uio.no Fri Jul 29 23:23:46 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Fri, 29 Jul 2011 23:23:46 +0200 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: <201107292214.44324.Arfrever.FTA@gmail.com> References: <201107291918.39717.Arfrever.FTA@gmail.com> <201107292214.44324.Arfrever.FTA@gmail.com> Message-ID: Looks very much like a numpy-on-py3 bug to me. (NumPy has an RC up currently as well, it'd be good to test them together but don't think I have time this weekend...) Do you know if the problem is there with Cython 0.14? Even if not, it could simply be that the test cases now exercise more of NumPy and nothing to do with Cython... Try copying over the numpy test cases from 0.15 and run them with Cython 0.14 perhaps (sorry that I don't have time to look into this myself now). At any rate, I think this is unlikely to be a Cython regression. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Arfrever Frehtes Taifersar Arahesis wrote: 2011-07-29 20:51:11 mark florisson napisa?(a): > On 29 July 2011 19:18, Arfrever Frehtes Taifersar Arahesis > wrote: > > 2011-07-26 13:00:10 mark florisson napisa?(a): > >> Unfortunately the output of the testrunner is rather useless here. > >> Could you run the tests with --no-cleanup and attach the .c and .cpp > >> files (found in build/run/c/numpy_cimport.c, etc)? I think something > >> is up with your numpy version and headers. The numpy tests should be > >> automatically skipped if numpy is not available. > > > > I have found the following lines in earlier part of output: > > > > runTest (__main__.CythonRunTestCase) > > compiling (c) and running numpy_bufacc_T155 ... numpy_bufacc_T155.c: In function ?PyInit_numpy_bufacc_T155?: > > numpy_bufacc_T155.c:3652:5: warning: ?return? with no value, in function returning non-void > > numpy_bufacc_T155 () > > Doctest: numpy_bufacc_T155 ... ok > > runTest (__main__.CythonRunTestCase) > > compiling (cpp) and running numpy_bufacc_T155 ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ > > numpy_bufacc_T155.cpp: In function ?PyObject* PyInit_numpy_bufacc_T155()?: > > numpy_bufacc_T155.cpp:3652:5: error: return-statement with no value, in function returning ?PyObject*? > > ERROR > > runTest (__main__.CythonRunTestCase) > > compiling (c) and running numpy_cimport ... numpy_cimport.c: In function ?PyInit_numpy_cimport?: > > numpy_cimport.c:3327:5: warning: ?return? with no value, in function returning non-void > > numpy_cimport () > > Doctest: numpy_cimport ... ok > > runTest (__main__.CythonRunTestCase) > > compiling (cpp) and running numpy_cimport ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ > > numpy_cimport.cpp: In function ?PyObject* PyInit_numpy_cimport()?: > > numpy_cimport.cpp:3327:5: error: return-statement with no value, in function returning ?PyObject*? > > ERROR > > runTest (__main__.CythonRunTestCase) > > compiling (c) and running numpy_parallel ... numpy_parallel.c: In function ?PyInit_numpy_parallel?: > > numpy_parallel.c:3824:5: warning: ?return? with no value, in function returning non-void > > test_parallel_numpy_arrays (line 11) (numpy_parallel.__test__) > > Doctest: numpy_parallel.__test__.test_parallel_numpy_arrays (line 11) ... ok > > runTest (__main__.CythonRunTestCase) > > compiling (cpp) and running numpy_parallel ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ > > numpy_parallel.cpp: In function ?PyObject* PyInit_numpy_parallel()?: > > numpy_parallel.cpp:3824:5: error: return-statement with no value, in function returning ?PyObject*? > > ERROR > > runTest (__main__.CythonRunTestCase) > > compiling (c) and running numpy_test ... numpy_test.c: In function ?PyInit_numpy_test?: > > numpy_test.c:11604:5: warning: ?return? with no value, in function returning non-void > > numpy_test () > > Doctest: numpy_test ... ok > > runTest (__main__.CythonRunTestCase) > > compiling (cpp) and running numpy_test ... cc1plus: warning: command line option "-Wpointer-sign" is valid for C/ObjC but not for C++ > > numpy_test.cpp: In function ?PyObject* PyInit_numpy_test()?: > > numpy_test.cpp:11604:5: error: return-statement with no value, in function returning ?PyObject*? > > ERROR > > > > Relevant source files: > > > > http://people.apache.org/~Arfrever/numpy_bufacc_T155.c > > http://people.apache.org/~Arfrever/numpy_bufacc_T155.cpp > > http://people.apache.org/~Arfrever/numpy_cimport.c > > http://people.apache.org/~Arfrever/numpy_cimport.cpp > > http://people.apache.org/~Arfrever/numpy_parallel.c > > http://people.apache.org/~Arfrever/numpy_parallel.cpp > > http://people.apache.org/~Arfrever/numpy_test.c > > http://people.apache.org/~Arfrever/numpy_test.cpp > > The compiler complains about the init function of the module, not sure > why, but the line numbers don't seem to correspond with the files you > provided. The tests run fine for me on both osx, fedora, and ubuntu, > all with different gcc versions, so I'm assuming something is wrong > with your setup. Actually line numbers correspond to the files provided by me. You probably tested with Python 2 instead of Python 3. The files generated by Cython contain this line: import_umath(); numpy/__ufunc_api.h defines (see 'return' at the end): #define import_umath() { UFUNC_NOFPE if (_import_umath() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "numpy.core.umath failed to import"); return; }} pyport.h of Python 2.* defines: #define PyMODINIT_FUNC void pyport.h of Python 3.* defines: #define PyMODINIT_FUNC PyObject* -- Arfrever Frehtes Taifersar Arahesis_____________________________________________ cython-devel mailing list cython-devel at python.org http://mail.python.org/mailman/listinfo/cython-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sat Jul 30 10:43:19 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 30 Jul 2011 10:43:19 +0200 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: References: <4E303D89.2060304@behnel.de> <4E30560D.6050307@astro.uio.no> <4E310121.2050400@behnel.de> Message-ID: <4E33C427.5020705@behnel.de> Robert Bradshaw, 29.07.2011 20:10: > On Wed, Jul 27, 2011 at 11:32 PM, Vitja Makarov wrote: >> It's fun: with cython you can use decorators, generators and more with py2.3, >> may be it's better to drop support after 0.15? I think Vitja has a point here. Cython now supports major Python language features that were not even remotely planned when the CPython versions it runs on were released. > In general, it sounds like people are fine with dropping support for > 2.3, and we've got one request for holding onto 2.4 a little bit > longer (but dropping that is on the horizon too). I just noticed that the 2.4 hg branch of CPython was closed in March, so it's definitely dead and gone. Dropping support for 2.4 has a couple of advantages at the C level. Exceptions turned into types in 2.5 and Py_ssize_t was introduced. So, making 2.5 the oldest supported version would allow us to drop some legacy code and start enabling optimisations for that version. Not to mention that sets are builtins since 2.4, which would drop the need for another bunch of legacy code. Other new features in 2.5: - some extensions to distutils, the lack of which currently account for some of the warts in our setup.py script. - relative imports - with statement - coroutines (yield as an expression) - unified try-except-finally All of these would have helped in more than one place to make Cython's code cleaner, simpler and safer. This version also was the first one to use an AST for parsing (not sure if that's interesting for us), and it was the first to ship with ctypes/libffi, which keeps being considered as a future basis for certain advanced Cython features. All of this makes me think that we should drop support for 2.4 soon, preferably right after 0.15 is out, and then have some fun ripping out dead code. Stefan From cournape at gmail.com Sat Jul 30 10:52:39 2011 From: cournape at gmail.com (David Cournapeau) Date: Sat, 30 Jul 2011 17:52:39 +0900 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: <4E33C427.5020705@behnel.de> References: <4E303D89.2060304@behnel.de> <4E30560D.6050307@astro.uio.no> <4E310121.2050400@behnel.de> <4E33C427.5020705@behnel.de> Message-ID: On Sat, Jul 30, 2011 at 5:43 PM, Stefan Behnel wrote: > > This version also was the first one to use an AST for parsing (not sure if > that's interesting for us), and it was the first to ship with ctypes/libffi, > which keeps being considered as a future basis for certain advanced Cython > features. Note that ctypes may not be available on every platform (python can be built without ctypes). > > All of this makes me think that we should drop support for 2.4 soon, > preferably right after 0.15 is out, and then have some fun ripping out dead > code. I would prefer keeping python 2.4 support for a while, especially for the generated code side of things. Python 2.4 is still surprisingly common. Working around it for C extensions can be pretty daunting. cheers, David From stefan_ml at behnel.de Sat Jul 30 11:07:55 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 30 Jul 2011 11:07:55 +0200 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: References: <4E303D89.2060304@behnel.de> <4E30560D.6050307@astro.uio.no> <4E310121.2050400@behnel.de> <4E33C427.5020705@behnel.de> Message-ID: <4E33C9EB.2010204@behnel.de> David Cournapeau, 30.07.2011 10:52: > Python 2.4 is still surprisingly > common. Working around it for C extensions can be pretty daunting. The same applies to Cython, obviously, although I do see the advantage of doing it in one place, and specifically in Cython, instead of dropping it on the users. Stefan From stefan_ml at behnel.de Sat Jul 30 11:37:46 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 30 Jul 2011 11:37:46 +0200 Subject: [Cython] Vitja's CyFunction branch Message-ID: <4E33D0EA.9020504@behnel.de> Hi, I wonder what we should do with Vitja's CyFunction branch. He mentioned issues with it in the past (I remember that there was one specific changeset that he considered questionable), and it seems that we found several ways to extend the function support beyond that, which may have an impact on the current way it's implemented. Is this something that makes sense to be included in 0.15, or should it wait to get stable and further extended before being merged? It appears to me that there are reasons for one and the other. However, given that we plan to drop support for (at least) Python 2.3 after this release, I feel that we should try to get out a version that doesn't require much fixing and maintenance. Changing the implementation of functions at this point seems like a rather deep cut. If we agree that it's in a state that we feel comfortable with, I wouldn't object to merging it. Otherwise, I'd suggest to wait until after the release. Comments? Stefan From vitja.makarov at gmail.com Sat Jul 30 11:55:53 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sat, 30 Jul 2011 13:55:53 +0400 Subject: [Cython] Vitja's CyFunction branch In-Reply-To: <4E33D0EA.9020504@behnel.de> References: <4E33D0EA.9020504@behnel.de> Message-ID: 2011/7/30 Stefan Behnel : > Hi, > > I wonder what we should do with Vitja's CyFunction branch. He mentioned > issues with it in the past (I remember that there was one specific changeset > that he considered questionable), and it seems that we found several ways to > extend the function support beyond that, which may have an impact on the > current way it's implemented. > > Is this something that makes sense to be included in 0.15, or should it wait > to get stable and further extended before being merged? > > It appears to me that there are reasons for one and the other. However, > given that we plan to drop support for (at least) Python 2.3 after this > release, I feel that we should try to get out a version that doesn't require > much fixing and maintenance. Changing the implementation of functions at > this point seems like a rather deep cut. If we agree that it's in a state > that we feel comfortable with, I wouldn't object to merging it. Otherwise, > I'd suggest to wait until after the release. > > Comments? > That's not such a big change, CyFunction replcaces old PyCFunction_binding type and shouldn't break anything. -- vitja. From cournape at gmail.com Sat Jul 30 12:21:53 2011 From: cournape at gmail.com (David Cournapeau) Date: Sat, 30 Jul 2011 19:21:53 +0900 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: <4E33C9EB.2010204@behnel.de> References: <4E303D89.2060304@behnel.de> <4E30560D.6050307@astro.uio.no> <4E310121.2050400@behnel.de> <4E33C427.5020705@behnel.de> <4E33C9EB.2010204@behnel.de> Message-ID: On Sat, Jul 30, 2011 at 6:07 PM, Stefan Behnel wrote: > David Cournapeau, 30.07.2011 10:52: >> >> Python 2.4 is still surprisingly >> common. Working around it for C extensions can be pretty daunting. > > The same applies to Cython, obviously, although I do see the advantage of > doing it in one place, and specifically in Cython, instead of dropping it on > the users. I was not very clear: I meant that if you were to use python 2.5 or above on a platform where only 2.4 is available, you would need to build a lot of C extensions by yourself, which is out of reach to the vast majority of our users. Typically, I had to use centos 5.x 2 years ago, and even for me, it was quite difficult to build a python 2.6 environment I was comfortable with (keeping in mind that a lot of "big-lab environments" do not give you admin access). Centos 5.x end of life is in 2014. cheers, David From markflorisson88 at gmail.com Sat Jul 30 15:26:13 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Sat, 30 Jul 2011 15:26:13 +0200 Subject: [Cython] Vitja's CyFunction branch In-Reply-To: References: <4E33D0EA.9020504@behnel.de> Message-ID: On 30 July 2011 11:55, Vitja Makarov wrote: > 2011/7/30 Stefan Behnel : >> Hi, >> >> I wonder what we should do with Vitja's CyFunction branch. He mentioned >> issues with it in the past (I remember that there was one specific changeset >> that he considered questionable), and it seems that we found several ways to >> extend the function support beyond that, which may have an impact on the >> current way it's implemented. >> >> Is this something that makes sense to be included in 0.15, or should it wait >> to get stable and further extended before being merged? >> >> It appears to me that there are reasons for one and the other. However, >> given that we plan to drop support for (at least) Python 2.3 after this >> release, I feel that we should try to get out a version that doesn't require >> much fixing and maintenance. Changing the implementation of functions at >> this point seems like a rather deep cut. If we agree that it's in a state >> that we feel comfortable with, I wouldn't object to merging it. Otherwise, >> I'd suggest to wait until after the release. >> >> Comments? >> > > That's not such a big change, CyFunction replcaces old > PyCFunction_binding type and shouldn't break anything. > > > -- > vitja. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > +1 on merging, I think it's been ready for weeks. I see no problem if it passes the testsuite, and I need it for fused types (although admittedly I'm taking a vacation after this weekend). From d.s.seljebotn at astro.uio.no Sat Jul 30 16:20:14 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Sat, 30 Jul 2011 16:20:14 +0200 Subject: [Cython] Vitja's CyFunction branch In-Reply-To: References: <4E33D0EA.9020504@behnel.de> Message-ID: My opinion is that we create a (short-lived) branch for the release, and continue development (ignoring the release) on master. If CyFunction solves no problems that blocks a release, I am -1 on merging it into the release branch. But we shouldn't keep things in pull requests just because we're waiting for a release, it is better to use a dedicated release branch. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. mark florisson wrote: On 30 July 2011 11:55, Vitja Makarov wrote: > 2011/7/30 Stefan Behnel : >> Hi, >> >> I wonder what we should do with Vitja's CyFunction branch. He mentioned >> issues with it in the past (I remember that there was one specific changeset >> that he considered questionable), and it seems that we found several ways to >> extend the function support beyond that, which may have an impact on the >> current way it's implemented. >> >> Is this something that makes sense to be included in 0.15, or should it wait >> to get stable and further extended before being merged? >> >> It appears to me that there are reasons for one and the other. However, >> given that we plan to drop support for (at least) Python 2.3 after this >> release, I feel that we should try to get out a version that doesn't require >> much fixing and maintenance. Changing the implementation of functions at >> this point seems like a rather deep cut. If we agree that it's in a state >> that we feel comfortable with, I wouldn't object to merging it. Otherwise, >> I'd suggest to wait until after the release. >> >> Comments? >> > > That's not such a big change, CyFunction replcaces old > PyCFunction_binding type and shouldn't break anything. > > > -- > vitja. >_____________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > +1 on merging, I think it's been ready for weeks. I see no problem if it passes the testsuite, and I need it for fused types (although admittedly I'm taking a vacation after this weekend)._____________________________________________ cython-devel mailing list cython-devel at python.org http://mail.python.org/mailman/listinfo/cython-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at math.washington.edu Sat Jul 30 18:49:54 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sat, 30 Jul 2011 09:49:54 -0700 Subject: [Cython] Vitja's CyFunction branch In-Reply-To: References: <4E33D0EA.9020504@behnel.de> Message-ID: On Sat, Jul 30, 2011 at 7:20 AM, Dag Sverre Seljebotn wrote: > My opinion is that we create a (short-lived) branch for the release, and > continue development (ignoring the release) on master. > > If CyFunction solves no problems that blocks a release, I am -1 on merging > it into the release branch. My thoughts exactly. I was initially wanting to get it in, but it was to substantial of a change to drop in at the last moment. > But we shouldn't keep things in pull requests > just because we're waiting for a release, it is better to use a dedicated > release branch. +1. The only reason I haven't pushed a release branch is that last time I did that it kept getting the mainline development pulled into it (I think accidentally getting rid of mercurial heads). To be clear, don't let a pending release prevent us from merging stuff into main--I'll be careful when I make the next RC to make sure it's all conservative. - Robert From arfrever.fta at gmail.com Sat Jul 30 19:57:11 2011 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Sat, 30 Jul 2011 19:57:11 +0200 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: References: <201107292214.44324.Arfrever.FTA@gmail.com> Message-ID: <201107301957.12161.Arfrever.FTA@gmail.com> 2011-07-29 23:23:46 Dag Sverre Seljebotn napisa?(a): > Looks very much like a numpy-on-py3 bug to me. I have reported: http://projects.scipy.org/numpy/ticket/1919 (NumPy-related tests in Cython pass with Python 3.*, when NumPy has been built with the patch from this ticket applied.) -- Arfrever Frehtes Taifersar Arahesis -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part. URL: From stefan_ml at behnel.de Sat Jul 30 20:39:48 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 30 Jul 2011 20:39:48 +0200 Subject: [Cython] Vitja's CyFunction branch In-Reply-To: References: <4E33D0EA.9020504@behnel.de> Message-ID: <4E344FF4.3060000@behnel.de> Robert Bradshaw, 30.07.2011 18:49: > The only reason I haven't pushed a release branch is that last > time I did that it kept getting the mainline development pulled into > it That was just an accident on my side when I wasn't aware of the new branch you had created. Won't happen again. Just open a new branch when you deem it appropriate. Stefan From dalcinl at gmail.com Sat Jul 30 20:40:20 2011 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Sat, 30 Jul 2011 14:40:20 -0400 Subject: [Cython] Should we drop support for CPython 2.3? (and maybe even 2.4?) In-Reply-To: References: <4E303D89.2060304@behnel.de> <4E30560D.6050307@astro.uio.no> <4E310121.2050400@behnel.de> <4E33C427.5020705@behnel.de> <4E33C9EB.2010204@behnel.de> Message-ID: On 30 July 2011 06:21, David Cournapeau wrote: > On Sat, Jul 30, 2011 at 6:07 PM, Stefan Behnel wrote: >> David Cournapeau, 30.07.2011 10:52: >>> >>> Python 2.4 is still surprisingly >>> common. Working around it for C extensions can be pretty daunting. >> >> The same applies to Cython, obviously, although I do see the advantage of >> doing it in one place, and specifically in Cython, instead of dropping it on >> the users. > > I was not very clear: I meant that if you were to use python 2.5 or > above on a platform where only 2.4 is available, you would need to > build a lot of C extensions by yourself, which is out of reach to the > vast majority of our users. > > Typically, I had to use centos 5.x 2 years ago, and even for me, it > was quite difficult to build a python 2.6 environment I was > comfortable with (keeping in mind that a lot of "big-lab environments" > do not give you admin access). Centos 5.x end of life is in 2014. > I'm with David here. I do not object dropping support for 2.3, but would like 2.4 to be supported. -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 From vitja.makarov at gmail.com Sun Jul 31 08:35:49 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sun, 31 Jul 2011 10:35:49 +0400 Subject: [Cython] Cython 0.15 release In-Reply-To: References: Message-ID: 2011/7/20 Robert Bradshaw : > We're long overdue for a release, and this week would be a good one > for me to push one out. Hudson > https://sage.math.washington.edu:8091/hudson is looking in pretty good > shape, and though I know we've got a big pile of stuff currently in > progress, we've also got a big backlog of stuff to get out. I'd like > to finish looking at https://github.com/cython/cython/pull/38 , are > there any other changes that people want to urgently get in? Also, > I've started http://wiki.cython.org/ReleaseNotes-0.15 , feel free to > edit if you think anything should be highlighted. > Anybody noticed problem with py3k: doctest.DocTestCase.Doctest: tp_new.__test__.make_new_untyped (line 132) It seems that error message have changed in new python version, may be it's better to use #doctest: +ELLIPSIS there? -- vitja. From robertwb at math.washington.edu Sun Jul 31 09:37:08 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 31 Jul 2011 00:37:08 -0700 Subject: [Cython] Cython 0.15 release In-Reply-To: References: Message-ID: On Sat, Jul 30, 2011 at 11:35 PM, Vitja Makarov wrote: > 2011/7/20 Robert Bradshaw : >> We're long overdue for a release, and this week would be a good one >> for me to push one out. Hudson >> https://sage.math.washington.edu:8091/hudson is looking in pretty good >> shape, and though I know we've got a big pile of stuff currently in >> progress, we've also got a big backlog of stuff to get out. I'd like >> to finish looking at https://github.com/cython/cython/pull/38 , are >> there any other changes that people want to urgently get in? Also, >> I've started http://wiki.cython.org/ReleaseNotes-0.15 , feel free to >> edit if you think anything should be highlighted. >> > > Anybody noticed problem with py3k: > > doctest.DocTestCase.Doctest: tp_new.__test__.make_new_untyped (line 132) > > It seems that error message have changed in new python version, > may be it's better to use #doctest: +ELLIPSIS there? Yes, that'd make sense. From robertwb at math.washington.edu Sun Jul 31 09:42:33 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 31 Jul 2011 00:42:33 -0700 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: <201107301957.12161.Arfrever.FTA@gmail.com> References: <201107292214.44324.Arfrever.FTA@gmail.com> <201107301957.12161.Arfrever.FTA@gmail.com> Message-ID: On Sat, Jul 30, 2011 at 10:57 AM, Arfrever Frehtes Taifersar Arahesis wrote: > 2011-07-29 23:23:46 Dag Sverre Seljebotn napisa?(a): >> Looks very much like a numpy-on-py3 bug to me. > > I have reported: > http://projects.scipy.org/numpy/ticket/1919 > (NumPy-related tests in Cython pass with Python 3.*, when NumPy has been built with the patch > from this ticket applied.) Thanks for looking into this. From robertwb at math.washington.edu Sun Jul 31 09:45:15 2011 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Sun, 31 Jul 2011 00:45:15 -0700 Subject: [Cython] Cython 0.15 release candidate In-Reply-To: References: Message-ID: On Thu, Jul 21, 2011 at 4:14 PM, Robert Bradshaw wrote: > Cython has seen an enormous amount of development since 0.14.1. If you > are not already using the latest version from the development > repository, we encourage you to try out the release candidate: > http://cython.org/release/Cython-0.15rc0.tar.gz Thanks for all the feedback. Here's another release candidate: http://cython.org/release/Cython-0.15rc0.tar.gz Unless something major is found, this is likely to be the release. - Robert From stefan_ml at behnel.de Sun Jul 31 10:06:54 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 31 Jul 2011 10:06:54 +0200 Subject: [Cython] Cython 0.15 release In-Reply-To: References: Message-ID: <4E350D1E.7010506@behnel.de> Vitja Makarov, 31.07.2011 08:35: > Anybody noticed problem with py3k: > > doctest.DocTestCase.Doctest: tp_new.__test__.make_new_untyped (line 132) > > It seems that error message have changed in new python version, That happens from time to time, yes. In case we provide a similar error message for optimised cases, it would be good to check if it makes sense to adapt it accordingly. Error messages in CPython tend to get better over time, so it's sometimes a good idea to follow. > may be it's better to use #doctest: +ELLIPSIS there? Yes, we already do that in lots of similar places. Stefan From stefan_ml at behnel.de Sun Jul 31 10:53:00 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 31 Jul 2011 10:53:00 +0200 Subject: [Cython] problematic code in C++ pyregr tests Message-ID: <4E3517EC.40807@behnel.de> Hi, I noticed a couple of problems in the latest pyregr test run for py3k. https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py3k-cpp/951/consoleFull 1) test_contextlib.cpp:16926: warning: deprecated conversion from string constant to ?char*? This repeated warning refers (I assume) to each literal in this code: """ char* __pyx_import_star_type_names[] = { "__pyx_scope_struct_2_test_contextmanager_finally", "__pyx_scope_struct_19_test_contextmanager_as_decorator", [...] "__pyx_scope_struct_6_test_contextmanager_except", "__pyx_scope_struct_12_testWithCondition", 0 }; """ This looks like a standard C++ WTF to me. I noticed that Lisandro keeps casting string literals in his code, so I wonder if that's something we need to do here as well? 2) test_cgi.cpp:2494: error: ?None? was not declared in this scope There seems to be a problem with the dict-loop optimisation. This is how the following code gets translated: """ * for k, v in dict(form).items(): # <<<<<<<<<<<<<< __Pyx_INCREF(None); __Pyx_XDECREF(__pyx_t_1); __pyx_t_1 = None; __pyx_t_5 = 0; __pyx_t_6 = PyDict_Size(__pyx_t_1); while (1) { /* __pyx_t_7 allocated */ /* __pyx_t_8 allocated */ if (unlikely(__pyx_t_6 != PyDict_Size(__pyx_t_1))) { PyErr_SetString(PyExc_RuntimeError, "dictionary changed size during iteration"); {__pyx_filename = __pyx_f[0]; __pyx_lineno = 114; """ Looks like the literal doesn't get evaluated. I'll look into this. 3) warning: TestClassUtilityCode:18:10: 'cpdef_method' redeclared warning: TestClassUtilityCode:29:10: 'cpdef_cname_method' redeclared 4) test_pep263.cpp:520: error: redefinition of ?PyObject* __pyx_kp_b_1? test_pep263.cpp:519: error: ?PyObject* __pyx_kp_b_1? previously declared here test_pep263.cpp:526: error: redefinition of ?PyObject* __pyx_kp_b_2? test_pep263.cpp:525: error: ?PyObject* __pyx_kp_b_2? previously declared here Triggered by this code, which refers to the same byte string literals twice in both cases: """ * self.assertEqual( # <<<<<<<<<<<<<< * "".encode("utf-8"), * b'\xd0\x9f\xd0\xb8\xd1\x82\xd0\xbe\xd0\xbd' */ /* __pyx_t_1 allocated */ __pyx_t_1 = PyObject_GetAttr(__pyx_v_self, __pyx_n_s__assertEqual); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 9; __pyx_clineno = __LINE__; goto __pyx_L1_error;} __Pyx_GOTREF(__pyx_t_1); /* __pyx_t_2 allocated */ __pyx_t_2 = PyTuple_New(2); if (unlikely(!__pyx_t_2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 9; __pyx_clineno = __LINE__; goto __pyx_L1_error;} __Pyx_GOTREF(((PyObject *)__pyx_t_2)); __Pyx_INCREF(((PyObject *)__pyx_kp_b_1)); PyTuple_SET_ITEM(__pyx_t_2, 0, ((PyObject *)__pyx_kp_b_1)); __Pyx_GIVEREF(((PyObject *)__pyx_kp_b_1)); __Pyx_INCREF(((PyObject *)__pyx_kp_b_1)); PyTuple_SET_ITEM(__pyx_t_2, 1, ((PyObject *)__pyx_kp_b_1)); __Pyx_GIVEREF(((PyObject *)__pyx_kp_b_1)); * self.assertEqual( # <<<<<<<<<<<<<< * "\".encode("utf-8"), * b'\\\xd0\x9f' */ /* __pyx_t_3 allocated */ __pyx_t_3 = PyObject_GetAttr(__pyx_v_self, __pyx_n_s__assertEqual); if (unlikely(!__pyx_t_3)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 13; __pyx_clineno = __LINE__; goto __pyx_L1_error;} __Pyx_GOTREF(__pyx_t_3); /* __pyx_t_2 allocated */ __pyx_t_2 = PyTuple_New(2); if (unlikely(!__pyx_t_2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 13; __pyx_clineno = __LINE__; goto __pyx_L1_error;} __Pyx_GOTREF(((PyObject *)__pyx_t_2)); __Pyx_INCREF(((PyObject *)__pyx_kp_b_2)); PyTuple_SET_ITEM(__pyx_t_2, 0, ((PyObject *)__pyx_kp_b_2)); __Pyx_GIVEREF(((PyObject *)__pyx_kp_b_2)); __Pyx_INCREF(((PyObject *)__pyx_kp_b_2)); PyTuple_SET_ITEM(__pyx_t_2, 1, ((PyObject *)__pyx_kp_b_2)); __Pyx_GIVEREF(((PyObject *)__pyx_kp_b_2)); """ 5) there are also several compiler crashes in that log 6) several tests bail out with """ File "runtests.py", line 900, in run_test except (unittest.SkipTest, support.ResourceDenied): AttributeError: 'module' object has no attribute 'ResourceDenied' """ I guess that's an incompatibility of our test runner with the latest Py3k. 7) g++: /.../workspace/BUILD/pyregr/cpp/test_inspect.o: No such file or directory g++: no input files There are loads of these - not sure what might trigger them. As I said, I'll look into the dict looping bug. If any of the other bugs is obvious to someone and not too hard to fix, it would be nice to get it done for the release. However, I think none of them is a real blocker. Stefan From vitja.makarov at gmail.com Sun Jul 31 11:14:01 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sun, 31 Jul 2011 13:14:01 +0400 Subject: [Cython] problematic code in C++ pyregr tests In-Reply-To: <4E3517EC.40807@behnel.de> References: <4E3517EC.40807@behnel.de> Message-ID: 2011/7/31 Stefan Behnel : > Hi, > > I noticed a couple of problems in the latest pyregr test run for py3k. > > https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py3k-cpp/951/consoleFull > > 1) test_contextlib.cpp:16926: warning: deprecated conversion from string > constant to ?char*? > > This repeated warning refers (I assume) to each literal in this code: > > """ > char* __pyx_import_star_type_names[] = { > ?"__pyx_scope_struct_2_test_contextmanager_finally", > ?"__pyx_scope_struct_19_test_contextmanager_as_decorator", > ?[...] > ?"__pyx_scope_struct_6_test_contextmanager_except", > ?"__pyx_scope_struct_12_testWithCondition", > ?0 > }; > """ > > This looks like a standard C++ WTF to me. I noticed that Lisandro keeps > casting string literals in his code, so I wonder if that's something we need > to do here as well? > That's really annoying. I was bothering this in CyFunction branch. I think we should introduce new macro here: #define __Pyx_STRCONST(x) ((char *) (x)) > > 6) several tests bail out with > > """ > File "runtests.py", line 900, in run_test > ? ?except (unittest.SkipTest, support.ResourceDenied): > AttributeError: 'module' object has no attribute 'ResourceDenied' > """ > > I guess that's an incompatibility of our test runner with the latest Py3k. > That's because latest py3k has test_support module again: try: from test import test_support as support except ImportError: # Py3k from test import support Think I should change the order -- vitja. From vitja.makarov at gmail.com Sun Jul 31 12:56:01 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sun, 31 Jul 2011 14:56:01 +0400 Subject: [Cython] Entry.is_arg question Message-ID: What is the meaning of Entry.is_arg flag? For generic python arguments it's set to 0, it makes it very tricky to determinate whether entry is argument: https://github.com/cython/cython/blob/master/Cython/Compiler/FlowControl.py#L515 Can we have is_args flag set for all kind of args? And have all the args inside scope.arg_entries? -- vitja. From vitja.makarov at gmail.com Sun Jul 31 21:29:30 2011 From: vitja.makarov at gmail.com (Vitja Makarov) Date: Sun, 31 Jul 2011 23:29:30 +0400 Subject: [Cython] OpenMP problem Message-ID: I've tried openmp support with simple example: from cython.parallel cimport prange def mul(values): ret = 1 for i in prange(values): ret *= i return ret And cython crashes: ((b04e040...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ make mul.so /usr//bin/python ../cython.py --gdb -v mul.pyx -o mul.c Compiling /home/vitja/work/cython-vitek/zzz/mul.pyx Error compiling Cython file: ------------------------------------------------------------ ... from cython.parallel cimport prange def mul(values): ret = 1 for i in prange(values): ^ ------------------------------------------------------------ mul.pyx:5:26: stop argument must be numeric or a pointer (perhaps if a numeric literal is too big, use 1000LL) Error compiling Cython file: ------------------------------------------------------------ ... from cython.parallel cimport prange def mul(values): ret = 1 for i in prange(values): ^ ------------------------------------------------------------ mul.pyx:5:19: Compiler crash in AnalyseExpressionsTransform ModuleNode.body = StatListNode(mul.pyx:1:0) StatListNode.stats[0] = DefNode(mul.pyx:3:0, modifiers = [...]/0, name = u'mul', num_required_args = 1, reqd_kw_flags_cname = '0', used = True) File 'Nodes.py', line 337, in analyse_expressions: StatListNode(mul.pyx:4:4, is_terminator = True) File 'Nodes.py', line 6553, in analyse_expressions: ParallelRangeNode(mul.pyx:5:19, assigned_nodes = [...]/1, is_parallel = True, is_prange = True, valid_keyword_arguments = [...]/3) Compiler crash traceback from this point on: File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line 6553, in analyse_expressions self.index_type, node.type) File "/home/vitja/work/cython-vitek/Cython/Compiler/PyrexTypes.py", line 2624, in widest_numeric_type elif type1.rank < type2.rank: AttributeError: 'PyObjectType' object has no attribute 'rank' make: *** [mul.c] ?????? 1 rm mul.c ((b04e040...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ -- vitja. From markflorisson88 at gmail.com Sun Jul 31 21:49:43 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 31 Jul 2011 21:49:43 +0200 Subject: [Cython] OpenMP problem In-Reply-To: References: Message-ID: 2011/7/31 Vitja Makarov : > I've tried openmp support with simple example: > > from cython.parallel cimport prange > > def mul(values): > ? ?ret = 1 > ? ?for i in prange(values): > ? ? ? ?ret *= i > ? ?return ret > > And cython crashes: > > ((b04e040...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ make mul.so > /usr//bin/python ?../cython.py --gdb -v ?mul.pyx -o mul.c > Compiling /home/vitja/work/cython-vitek/zzz/mul.pyx > > Error compiling Cython file: > ------------------------------------------------------------ > ... > from cython.parallel cimport prange > > def mul(values): > ? ?ret = 1 > ? ?for i in prange(values): > ? ? ? ? ? ? ? ? ? ? ? ? ^ > ------------------------------------------------------------ > > mul.pyx:5:26: stop argument must be numeric or a pointer (perhaps if a > numeric literal is too big, use 1000LL) > > Error compiling Cython file: > ------------------------------------------------------------ > ... > from cython.parallel cimport prange > > def mul(values): > ? ?ret = 1 > ? ?for i in prange(values): > ? ? ? ? ? ? ? ? ?^ > ------------------------------------------------------------ > > mul.pyx:5:19: Compiler crash in AnalyseExpressionsTransform > > ModuleNode.body = StatListNode(mul.pyx:1:0) > StatListNode.stats[0] = DefNode(mul.pyx:3:0, > ? ?modifiers = [...]/0, > ? ?name = u'mul', > ? ?num_required_args = 1, > ? ?reqd_kw_flags_cname = '0', > ? ?used = True) > File 'Nodes.py', line 337, in analyse_expressions: StatListNode(mul.pyx:4:4, > ? ?is_terminator = True) > File 'Nodes.py', line 6553, in analyse_expressions: > ParallelRangeNode(mul.pyx:5:19, > ? ?assigned_nodes = [...]/1, > ? ?is_parallel = True, > ? ?is_prange = True, > ? ?valid_keyword_arguments = [...]/3) > > Compiler crash traceback from this point on: > ?File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line > 6553, in analyse_expressions > ? ?self.index_type, node.type) > ?File "/home/vitja/work/cython-vitek/Cython/Compiler/PyrexTypes.py", > line 2624, in widest_numeric_type > ? ?elif type1.rank < type2.rank: > AttributeError: 'PyObjectType' object has no attribute 'rank' > make: *** [mul.c] ?????? 1 > rm mul.c > ((b04e040...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ > > > > -- > vitja. > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > http://mail.python.org/mailman/listinfo/cython-devel > Thanks for the report, I pushed a fix: https://github.com/cython/cython/commit/e9f9fde70d5d96f418244a0640a63a966a2457c1 . Seems like I forgot an error test for that. From markflorisson88 at gmail.com Sun Jul 31 22:08:26 2011 From: markflorisson88 at gmail.com (mark florisson) Date: Sun, 31 Jul 2011 22:08:26 +0200 Subject: [Cython] OpenMP problem In-Reply-To: References: Message-ID: On 31 July 2011 21:49, mark florisson wrote: > 2011/7/31 Vitja Makarov : >> I've tried openmp support with simple example: >> >> from cython.parallel cimport prange >> >> def mul(values): >> ? ?ret = 1 >> ? ?for i in prange(values): >> ? ? ? ?ret *= i >> ? ?return ret >> >> And cython crashes: >> >> ((b04e040...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ make mul.so >> /usr//bin/python ?../cython.py --gdb -v ?mul.pyx -o mul.c >> Compiling /home/vitja/work/cython-vitek/zzz/mul.pyx >> >> Error compiling Cython file: >> ------------------------------------------------------------ >> ... >> from cython.parallel cimport prange >> >> def mul(values): >> ? ?ret = 1 >> ? ?for i in prange(values): >> ? ? ? ? ? ? ? ? ? ? ? ? ^ >> ------------------------------------------------------------ >> >> mul.pyx:5:26: stop argument must be numeric or a pointer (perhaps if a >> numeric literal is too big, use 1000LL) >> >> Error compiling Cython file: >> ------------------------------------------------------------ >> ... >> from cython.parallel cimport prange >> >> def mul(values): >> ? ?ret = 1 >> ? ?for i in prange(values): >> ? ? ? ? ? ? ? ? ?^ >> ------------------------------------------------------------ >> >> mul.pyx:5:19: Compiler crash in AnalyseExpressionsTransform >> >> ModuleNode.body = StatListNode(mul.pyx:1:0) >> StatListNode.stats[0] = DefNode(mul.pyx:3:0, >> ? ?modifiers = [...]/0, >> ? ?name = u'mul', >> ? ?num_required_args = 1, >> ? ?reqd_kw_flags_cname = '0', >> ? ?used = True) >> File 'Nodes.py', line 337, in analyse_expressions: StatListNode(mul.pyx:4:4, >> ? ?is_terminator = True) >> File 'Nodes.py', line 6553, in analyse_expressions: >> ParallelRangeNode(mul.pyx:5:19, >> ? ?assigned_nodes = [...]/1, >> ? ?is_parallel = True, >> ? ?is_prange = True, >> ? ?valid_keyword_arguments = [...]/3) >> >> Compiler crash traceback from this point on: >> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line >> 6553, in analyse_expressions >> ? ?self.index_type, node.type) >> ?File "/home/vitja/work/cython-vitek/Cython/Compiler/PyrexTypes.py", >> line 2624, in widest_numeric_type >> ? ?elif type1.rank < type2.rank: >> AttributeError: 'PyObjectType' object has no attribute 'rank' >> make: *** [mul.c] ?????? 1 >> rm mul.c >> ((b04e040...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ >> >> >> >> -- >> vitja. >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> > > Thanks for the report, I pushed a fix: > https://github.com/cython/cython/commit/e9f9fde70d5d96f418244a0640a63a966a2457c1 > . Seems like I forgot an error test for that. > I pushed it to release, should I rebase master on release now? From d.s.seljebotn at astro.uio.no Sun Jul 31 22:55:19 2011 From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn) Date: Sun, 31 Jul 2011 22:55:19 +0200 Subject: [Cython] OpenMP problem In-Reply-To: References: Message-ID: <761acc07-d451-413e-94cb-39b591f76736@email.android.com> That sounds risky; I think we should consider master as non-rebaseable except in emergencies. So fixes should be pushed to release and then merged into master. Of course, for something like this with no development depending on it one can just delay the merge for a day or two in case more fixes show up. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. mark florisson wrote: On 31 July 2011 21:49, mark florisson wrote: > 2011/7/31 Vitja Makarov : >> I've tried openmp support with simple example: >> >> from cython.parallel cimport prange >> >> def mul(values): >> ret = 1 >> for i in prange(values): >> ret *= i >> return ret >> >> And cython crashes: >> >> ((b04e040...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ make mul.so >> /usr//bin/python ../cython.py --gdb -v mul.pyx -o mul.c >> Compiling /home/vitja/work/cython-vitek/zzz/mul.pyx >> >> Error compiling Cython file: >>_____________________________________________ >> ... >> from cython.parallel cimport prange >> >> def mul(values): >> ret = 1 >> for i in prange(values): >> ^ >>_____________________________________________ >> >> mul.pyx:5:26: stop argument must be numeric or a pointer (perhaps if a >> numeric literal is too big, use 1000LL) >> >> Error compiling Cython file: >>_____________________________________________ >> ... >> from cython.parallel cimport prange >> >> def mul(values): >> ret = 1 >> for i in prange(values): >> ^ >>_____________________________________________ >> >> mul.pyx:5:19: Compiler crash in AnalyseExpressionsTransform >> >> ModuleNode.body = StatListNode(mul.pyx:1:0) >> StatListNode.stats[0] = DefNode(mul.pyx:3:0, >> modifiers = [...]/0, >> name = u'mul', >> num_required_args = 1, >> reqd_kw_flags_cname = '0', >> used = True) >> File 'Nodes.py', line 337, in analyse_expressions: StatListNode(mul.pyx:4:4, >> is_terminator = True) >> File 'Nodes.py', line 6553, in analyse_expressions: >> ParallelRangeNode(mul.pyx:5:19, >> assigned_nodes = [...]/1, >> is_parallel = True, >> is_prange = True, >> valid_keyword_arguments = [...]/3) >> >> Compiler crash traceback from this point on: >> File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line >> 6553, in analyse_expressions >> self.index_type, node.type) >> File "/home/vitja/work/cython-vitek/Cython/Compiler/PyrexTypes.py", >> line 2624, in widest_numeric_type >> elif type1.rank < type2.rank: >> AttributeError: 'PyObjectType' object has no attribute 'rank' >> make: *** [mul.c] ?????? 1 >> rm mul.c >> ((b04e040...)) vitja at vitja-laptop:~/work/cython-vitek/zzz$ >> >> >> >> -- >> vitja. >>_____________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> http://mail.python.org/mailman/listinfo/cython-devel >> > > Thanks for the report, I pushed a fix: > https://github.com/cython/cython/commit/e9f9fde70d5d96f418244a0640a63a966a2457c1 > . Seems like I forgot an error test for that. > I pushed it to release, should I rebase master on release now?_____________________________________________ cython-devel mailing list cython-devel at python.org http://mail.python.org/mailman/listinfo/cython-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: