From rmcgibbo at gmail.com Wed Jul 1 11:12:34 2015 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Wed, 1 Jul 2015 02:12:34 -0700 Subject: [Cython] Use of long type for intermediate integral variables Message-ID: Hi, (First time poster. Apologies in advance if I'm not following the appropriate protocol, or if this has been posted already) I noticed an issue on Windows when debugging an issue in scipy , but I think it might be a little more general. In some places in the generated code, it looks like intermediate integral variables are declared with type long, even when long is too small to hold necessary value. For example, with the code pasted below, the value n+1 is stored in a variable of type long (using Cython 0.22.1) before being supplied to F.__getitem__. This is especially pertinent on Windows (32 bit and 64 bit) and 32-bit linux, where longs are 32-bits, so you get an overflow for a program like the example below. The result is that it prints 1 instead of the expected value, 2**53+1 = 9007199254740993. But this same issue comes up basically whenever you do arithmetic on an array index in 64-bit Windows, for indices larger than 2**31-1, since sizeof(long) << sizeof(void*). ``` from libc.stdint cimport int64_t class F(object): def __getitem__(self, i): print(i) cdef int64_t n = 2**53 f = F() f[n+1] ``` -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Thu Jul 2 08:30:15 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 02 Jul 2015 08:30:15 +0200 Subject: [Cython] Use of long type for intermediate integral variables In-Reply-To: References: Message-ID: <5594DA77.2090306@behnel.de> Robert McGibbon schrieb am 01.07.2015 um 11:12: > I noticed an issue on Windows when debugging an issue in scipy > , but I think it might be a > little more general. In some places in the generated code, it looks like > intermediate integral variables are declared with type long, even when long > is too small to hold necessary value. For example, with the code pasted > below, the value n+1 is stored in a variable of type long (using Cython > 0.22.1) before being supplied to F.__getitem__. > > This is especially pertinent on Windows (32 bit and 64 bit) and 32-bit > linux, where longs are 32-bits, so you get an overflow for a program like > the example below. The result is that it prints 1 instead of the expected > value, 2**53+1 = 9007199254740993. But this same issue comes up basically > whenever you do arithmetic on an array index in 64-bit Windows, for indices > larger than 2**31-1, since sizeof(long) << sizeof(void*). > > ``` > from libc.stdint cimport int64_t > > class F(object): > def __getitem__(self, i): > print(i) > > cdef int64_t n = 2**53 > f = F() > f[n+1] > ``` Thanks for the report and the investigation. I can imagine why this is the case. "libc.stdint.int64_t" is hand-wavingly declared as "long" and the literal 1 is also of type "long" in Cython, so it infers that using "long" is good enough to hold the result of the sum. You can work around this by casting the 1 to , but that's clumsy and error prone. The problem is that Cython doesn't know the exact type of typedefs at translation time, only the C compilers will eventually know and might have diverging ideas about it. Your specific issue could be helped by preferring typedefs over standard integer types in the decision which type to use for arithmetic expressions, but that would then break the case where the typedef-ed type happens to be smaller than the standard one, e.g. cdef extern from "...": ctypedef long long SomeInt # declared large enough, just in case def test(SomeInt x): cdef long long y = 1 return x + y If Cython inferred "int64" for the type of the result, the C code would be correct if sizeof(SomeInt) >= sizeof(long long), but not if it's smaller. Also, what should happen in expressions that used two different user provided typedefs of the same declared base type? The decision here must necessarily be arbitrary. So, I agree that what you have found is a problem, it feels like fixing it by preferring typedefs would generally be a good idea, but on the other hand, it might break existing code (which usually means that it *will* break someone's code), and it would not fix all possible problematic cases. Not an easy decision... Stefan From rmcgibbo at gmail.com Thu Jul 2 09:49:54 2015 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Thu, 2 Jul 2015 00:49:54 -0700 Subject: [Cython] Use of long type for intermediate integral variables In-Reply-To: <5594DA77.2090306@behnel.de> References: <5594DA77.2090306@behnel.de> Message-ID: > "libc.stdint.int64_t" is hand-wavingly declared as "long" There are some deeper issues in the rest of your message, but as a preliminary matter, isn't this a clear error for linux-32 and windows? -Robert On Wed, Jul 1, 2015 at 11:30 PM, Stefan Behnel wrote: > Robert McGibbon schrieb am 01.07.2015 um 11:12: > > I noticed an issue on Windows when debugging an issue in scipy > > , but I think it might be a > > little more general. In some places in the generated code, it looks like > > intermediate integral variables are declared with type long, even when > long > > is too small to hold necessary value. For example, with the code pasted > > below, the value n+1 is stored in a variable of type long (using Cython > > 0.22.1) before being supplied to F.__getitem__. > > > > This is especially pertinent on Windows (32 bit and 64 bit) and 32-bit > > linux, where longs are 32-bits, so you get an overflow for a program like > > the example below. The result is that it prints 1 instead of the expected > > value, 2**53+1 = 9007199254740993. But this same issue comes up basically > > whenever you do arithmetic on an array index in 64-bit Windows, for > indices > > larger than 2**31-1, since sizeof(long) << sizeof(void*). > > > > ``` > > from libc.stdint cimport int64_t > > > > class F(object): > > def __getitem__(self, i): > > print(i) > > > > cdef int64_t n = 2**53 > > f = F() > > f[n+1] > > ``` > > Thanks for the report and the investigation. I can imagine why this is the > case. "libc.stdint.int64_t" is hand-wavingly declared as "long" and the > literal 1 is also of type "long" in Cython, so it infers that using "long" > is good enough to hold the result of the sum. > > You can work around this by casting the 1 to , but that's clumsy > and error prone. The problem is that Cython doesn't know the exact type of > typedefs at translation time, only the C compilers will eventually know and > might have diverging ideas about it. Your specific issue could be helped by > preferring typedefs over standard integer types in the decision which type > to use for arithmetic expressions, but that would then break the case where > the typedef-ed type happens to be smaller than the standard one, e.g. > > cdef extern from "...": > ctypedef long long SomeInt # declared large enough, just in case > > def test(SomeInt x): > cdef long long y = 1 > return x + y > > If Cython inferred "int64" for the type of the result, the C code would be > correct if sizeof(SomeInt) >= sizeof(long long), but not if it's smaller. > > Also, what should happen in expressions that used two different user > provided typedefs of the same declared base type? The decision here must > necessarily be arbitrary. > > So, I agree that what you have found is a problem, it feels like fixing it > by preferring typedefs would generally be a good idea, but on the other > hand, it might break existing code (which usually means that it *will* > break someone's code), and it would not fix all possible problematic cases. > > Not an easy decision... > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Thu Jul 2 09:58:41 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 02 Jul 2015 09:58:41 +0200 Subject: [Cython] Use of long type for intermediate integral variables In-Reply-To: References: <5594DA77.2090306@behnel.de> Message-ID: <5594EF31.8020008@behnel.de> Robert McGibbon schrieb am 02.07.2015 um 09:49: >> "libc.stdint.int64_t" is hand-wavingly declared as "long" > > There are some deeper issues in the rest of your message, but as a > preliminary matter, isn't this a clear error for linux-32 and windows? No, it's not. That's just what Cython sees. The C compiler then sees the exact platform specific type. And as the vast amount of Cython code out there shows, it's usually not a problem in practice. Cython is designed to handle most of these "platform specific type" issues at C compile time rather than C code generation time. But as your example shows, it can't always hide the details entirely. And there can be bugs. Stefan From rmcgibbo at gmail.com Thu Jul 2 21:08:13 2015 From: rmcgibbo at gmail.com (Robert McGibbon) Date: Thu, 2 Jul 2015 12:08:13 -0700 Subject: [Cython] Use of long type for intermediate integral variables In-Reply-To: <5594EF31.8020008@behnel.de> References: <5594DA77.2090306@behnel.de> <5594EF31.8020008@behnel.de> Message-ID: Right, okay. I think I understand. -Robert On Thu, Jul 2, 2015 at 12:58 AM, Stefan Behnel wrote: > Robert McGibbon schrieb am 02.07.2015 um 09:49: > >> "libc.stdint.int64_t" is hand-wavingly declared as "long" > > > > There are some deeper issues in the rest of your message, but as a > > preliminary matter, isn't this a clear error for linux-32 and windows? > > No, it's not. That's just what Cython sees. The C compiler then sees the > exact platform specific type. And as the vast amount of Cython code out > there shows, it's usually not a problem in practice. Cython is designed to > handle most of these "platform specific type" issues at C compile time > rather than C code generation time. But as your example shows, it can't > always hide the details entirely. And there can be bugs. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From insertinterestingnamehere at gmail.com Thu Jul 2 22:11:42 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Thu, 02 Jul 2015 20:11:42 +0000 Subject: [Cython] Use of long type for intermediate integral variables In-Reply-To: References: <5594DA77.2090306@behnel.de> <5594EF31.8020008@behnel.de> Message-ID: On Thu, Jul 2, 2015 at 1:08 PM Robert McGibbon wrote: > Right, okay. I think I understand. > > -Robert > > On Thu, Jul 2, 2015 at 12:58 AM, Stefan Behnel > wrote: > >> Robert McGibbon schrieb am 02.07.2015 um 09:49: >> >> "libc.stdint.int64_t" is hand-wavingly declared as "long" >> > >> > There are some deeper issues in the rest of your message, but as a >> > preliminary matter, isn't this a clear error for linux-32 and windows? >> >> No, it's not. That's just what Cython sees. The C compiler then sees the >> exact platform specific type. And as the vast amount of Cython code out >> there shows, it's usually not a problem in practice. Cython is designed to >> handle most of these "platform specific type" issues at C compile time >> rather than C code generation time. But as your example shows, it can't >> always hide the details entirely. And there can be bugs. >> >> Stefan >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> https://mail.python.org/mailman/listinfo/cython-devel >> > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel This is an interesting discussion. Thanks. For the particular case of int64_t, wouldn't it still be wise to make it a typedef of long long rather than just long so as to consistently get the correct size on platforms where long is only 32 bits? Thanks! -Ian Henriksen -------------- next part -------------- An HTML attachment was scrubbed... URL: From lrq3000 at gmail.com Fri Jul 3 03:16:22 2015 From: lrq3000 at gmail.com (Stephen LARROQUE) Date: Fri, 3 Jul 2015 03:16:22 +0200 Subject: [Cython] cython-devel Digest, Vol 53, Issue 9 In-Reply-To: References: Message-ID: Thank you very much Stefan. I have just read the ticket, and it seems that Guido van Rossum consider this to be a feature, not a bug (???), so I guess they won't fix it. For me it would be a nice plus to have this problem fixed, but I can live with my current workaround, so I'm not sure if Cython should spend dev time to fix that if that's expensive. At least, I think the workaround should be mentioned in the github wiki so that others may implement finite fields extension types without MemoryError issues. Should I add a wiki page? 2015-06-19 12:00 GMT+02:00 : > Send cython-devel mailing list submissions to > cython-devel at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/cython-devel > or, via email, send a message with subject or body 'help' to > cython-devel-request at python.org > > You can reach the person managing the list at > cython-devel-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of cython-devel digest..." > > > Today's Topics: > > 1. Re: Bug: Extension Type inheriting from int cause a > MemoryError (Stefan Behnel) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 19 Jun 2015 08:38:32 +0200 > From: Stefan Behnel > To: cython-devel at python.org > Subject: Re: [Cython] Bug: Extension Type inheriting from int cause a > MemoryError > Message-ID: <5583B8E8.9050902 at behnel.de> > Content-Type: text/plain; charset=utf-8 > > Stephen LARROQUE schrieb am 15.06.2015 um 12:34: > > I am trying to make an extension type inheriting from int or cython.int > (to > > do arithmetic operations in Galois Fields). However, this causes a > > MemoryError because it seems such extension type is not freed correctly. > > Other than that, it works perfectly well. > > > > cdef class ExtendedInt(int): pass > > > > for j in xrange(10000000): > > ExtendedInt(j) > > It looks like a bug in the "int" type in Python 2.x. Python 2.7 has this > code in intobject.c: > > """ > static void > int_dealloc(PyIntObject *v) > { > if (PyInt_CheckExact(v)) { > Py_TYPE(v) = (struct _typeobject *)free_list; > free_list = v; > } > else > Py_TYPE(v)->tp_free((PyObject *)v); > } > > static void > int_free(PyIntObject *v) > { > Py_TYPE(v) = (struct _typeobject *)free_list; > free_list = v; > } > """ > > Your extension type automatically inherits the "int_free" slot function > from its base type, so the "else" case in "int_dealloc()" will in fact call > "int_free()" and append the object to the free list despite *not* being > exactly of type PyInt. Then, when creating a new ExtendedInt instance, > Python's int-subtype instantiation code *ignores* the free list and instead > creates a completely new object. > > Thus, the free list keeps growing until it fills all memory and the process > dies. I created a CPython ticket. > > https://bugs.python.org/issue24469 > > I guess we could hack up a work around for this in Cython somehow (Python > 2.7 is an easy target being a dead end, after all), but let's wait what the > CPython devs have to say about this. Note, however, that any fix in a > future CPython 2.7.x release will not fix it in earlier Python 2.x > versions, so my guess is that we'd end up having to add a work-around on > our side anyway. > > Stefan > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > > > ------------------------------ > > End of cython-devel Digest, Vol 53, Issue 9 > ******************************************* > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at math.washington.edu Fri Jul 3 05:45:51 2015 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 2 Jul 2015 20:45:51 -0700 Subject: [Cython] Use of long type for intermediate integral variables In-Reply-To: References: <5594DA77.2090306@behnel.de> <5594EF31.8020008@behnel.de> Message-ID: I've fixed this particular case by making our integer ranking more consistent: signedness is ignored for anything but chars and typedefs are always preferred (as a last tiebreaker) over non-typedefs (rather than arbitrarily picking the first or second argument). https://github.com/cython/cython/commit/61ee8e544bd8802dfc313d832f5da97baf755c3d I've also upgraded the definition of int64_t to be long long to avoid issues with int64_t + unsigned long. There's still an issue of int32_t + unsigned int for any platforms where sizeof(int) == 16. -Robert On Thu, Jul 2, 2015 at 1:11 PM, Ian Henriksen wrote: > On Thu, Jul 2, 2015 at 1:08 PM Robert McGibbon wrote: >> >> Right, okay. I think I understand. >> >> -Robert >> >> On Thu, Jul 2, 2015 at 12:58 AM, Stefan Behnel >> wrote: >>> >>> Robert McGibbon schrieb am 02.07.2015 um 09:49: >>> >> "libc.stdint.int64_t" is hand-wavingly declared as "long" >>> > >>> > There are some deeper issues in the rest of your message, but as a >>> > preliminary matter, isn't this a clear error for linux-32 and windows? >>> >>> No, it's not. That's just what Cython sees. The C compiler then sees the >>> exact platform specific type. And as the vast amount of Cython code out >>> there shows, it's usually not a problem in practice. Cython is designed >>> to >>> handle most of these "platform specific type" issues at C compile time >>> rather than C code generation time. But as your example shows, it can't >>> always hide the details entirely. And there can be bugs. >>> >>> Stefan >>> >>> _______________________________________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> https://mail.python.org/mailman/listinfo/cython-devel >> >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> https://mail.python.org/mailman/listinfo/cython-devel > > > This is an interesting discussion. Thanks. > For the particular case of int64_t, wouldn't it still be wise to > make it a typedef of long long rather than just long so as to consistently > get the correct size on platforms where long > is only 32 bits? > Thanks! > -Ian Henriksen > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > From robertwb at math.washington.edu Fri Jul 3 05:50:05 2015 From: robertwb at math.washington.edu (Robert Bradshaw) Date: Thu, 2 Jul 2015 20:50:05 -0700 Subject: [Cython] Use of long type for intermediate integral variables In-Reply-To: References: <5594DA77.2090306@behnel.de> <5594EF31.8020008@behnel.de> Message-ID: To clarify, any choice of int64_t + unsigned long will be wrong on one platform or the others, but getting the wrong sign seems preferable to getting the wrong number of bits. On Thu, Jul 2, 2015 at 8:45 PM, Robert Bradshaw wrote: > I've fixed this particular case by making our integer ranking more > consistent: signedness is ignored for anything but chars and typedefs > are always preferred (as a last tiebreaker) over non-typedefs (rather > than arbitrarily picking the first or second argument). > https://github.com/cython/cython/commit/61ee8e544bd8802dfc313d832f5da97baf755c3d > > I've also upgraded the definition of int64_t to be long long to avoid > issues with int64_t + unsigned long. There's still an issue of int32_t > + unsigned int for any platforms where sizeof(int) == 16. > > -Robert > > On Thu, Jul 2, 2015 at 1:11 PM, Ian Henriksen > wrote: >> On Thu, Jul 2, 2015 at 1:08 PM Robert McGibbon wrote: >>> >>> Right, okay. I think I understand. >>> >>> -Robert >>> >>> On Thu, Jul 2, 2015 at 12:58 AM, Stefan Behnel >>> wrote: >>>> >>>> Robert McGibbon schrieb am 02.07.2015 um 09:49: >>>> >> "libc.stdint.int64_t" is hand-wavingly declared as "long" >>>> > >>>> > There are some deeper issues in the rest of your message, but as a >>>> > preliminary matter, isn't this a clear error for linux-32 and windows? >>>> >>>> No, it's not. That's just what Cython sees. The C compiler then sees the >>>> exact platform specific type. And as the vast amount of Cython code out >>>> there shows, it's usually not a problem in practice. Cython is designed >>>> to >>>> handle most of these "platform specific type" issues at C compile time >>>> rather than C code generation time. But as your example shows, it can't >>>> always hide the details entirely. And there can be bugs. >>>> >>>> Stefan >>>> >>>> _______________________________________________ >>>> cython-devel mailing list >>>> cython-devel at python.org >>>> https://mail.python.org/mailman/listinfo/cython-devel >>> >>> >>> _______________________________________________ >>> cython-devel mailing list >>> cython-devel at python.org >>> https://mail.python.org/mailman/listinfo/cython-devel >> >> >> This is an interesting discussion. Thanks. >> For the particular case of int64_t, wouldn't it still be wise to >> make it a typedef of long long rather than just long so as to consistently >> get the correct size on platforms where long >> is only 32 bits? >> Thanks! >> -Ian Henriksen >> >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> https://mail.python.org/mailman/listinfo/cython-devel >> From insertinterestingnamehere at gmail.com Fri Jul 3 06:46:40 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Fri, 03 Jul 2015 04:46:40 +0000 Subject: [Cython] Use of long type for intermediate integral variables In-Reply-To: References: <5594DA77.2090306@behnel.de> <5594EF31.8020008@behnel.de> Message-ID: On Thu, Jul 2, 2015 at 9:50 PM Robert Bradshaw wrote: > To clarify, any choice of int64_t + unsigned long will be wrong on one > platform or the others, but getting the wrong sign seems preferable to > getting the wrong number of bits. > That makes sens. Thanks for looking at it! -Ian Henriksen > > On Thu, Jul 2, 2015 at 8:45 PM, Robert Bradshaw > wrote: > > I've fixed this particular case by making our integer ranking more > > consistent: signedness is ignored for anything but chars and typedefs > > are always preferred (as a last tiebreaker) over non-typedefs (rather > > than arbitrarily picking the first or second argument). > > > https://github.com/cython/cython/commit/61ee8e544bd8802dfc313d832f5da97baf755c3d > > > > I've also upgraded the definition of int64_t to be long long to avoid > > issues with int64_t + unsigned long. There's still an issue of int32_t > > + unsigned int for any platforms where sizeof(int) == 16. > > > > -Robert > > > > On Thu, Jul 2, 2015 at 1:11 PM, Ian Henriksen > > wrote: > >> On Thu, Jul 2, 2015 at 1:08 PM Robert McGibbon > wrote: > >>> > >>> Right, okay. I think I understand. > >>> > >>> -Robert > >>> > >>> On Thu, Jul 2, 2015 at 12:58 AM, Stefan Behnel > >>> wrote: > >>>> > >>>> Robert McGibbon schrieb am 02.07.2015 um 09:49: > >>>> >> "libc.stdint.int64_t" is hand-wavingly declared as "long" > >>>> > > >>>> > There are some deeper issues in the rest of your message, but as a > >>>> > preliminary matter, isn't this a clear error for linux-32 and > windows? > >>>> > >>>> No, it's not. That's just what Cython sees. The C compiler then sees > the > >>>> exact platform specific type. And as the vast amount of Cython code > out > >>>> there shows, it's usually not a problem in practice. Cython is > designed > >>>> to > >>>> handle most of these "platform specific type" issues at C compile time > >>>> rather than C code generation time. But as your example shows, it > can't > >>>> always hide the details entirely. And there can be bugs. > >>>> > >>>> Stefan > >>>> > >>>> _______________________________________________ > >>>> cython-devel mailing list > >>>> cython-devel at python.org > >>>> https://mail.python.org/mailman/listinfo/cython-devel > >>> > >>> > >>> _______________________________________________ > >>> cython-devel mailing list > >>> cython-devel at python.org > >>> https://mail.python.org/mailman/listinfo/cython-devel > >> > >> > >> This is an interesting discussion. Thanks. > >> For the particular case of int64_t, wouldn't it still be wise to > >> make it a typedef of long long rather than just long so as to > consistently > >> get the correct size on platforms where long > >> is only 32 bits? > >> Thanks! > >> -Ian Henriksen > >> > >> _______________________________________________ > >> cython-devel mailing list > >> cython-devel at python.org > >> https://mail.python.org/mailman/listinfo/cython-devel > >> > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From codewarrior0 at gmail.com Fri Jul 3 02:26:26 2015 From: codewarrior0 at gmail.com (David Vierra) Date: Thu, 02 Jul 2015 14:26:26 -1000 Subject: [Cython] Use of long type for intermediate integral variables In-Reply-To: References: <5594DA77.2090306@behnel.de> <5594EF31.8020008@behnel.de> Message-ID: <5595D6B2.8030503@gmail.com> Very informative discussion. I see that Cython doesn't actually know the underlying types of typedefs because it doesn't parse any C headers. Cython doesn't even care about the exact length of the underlying type, just that it can order the types by rank to find which of two types is widest. When you write `ctypedef`, you're just telling Cython that two types are equivalent. I second the opinion that the stdint.h typedefs in the standard library should have accurate definitions. This way `ctypedefs` that resolve to a stdint type will not be too wrong. It might be safe just to change the ctypedefs for int64_t to `long long` - per the C99 standard, this type is *always* at least 64-bits long. (I don't know if all compilers adhere to this part of the spec and don't feel like getting a list of every compiler's opinion...but I know for sure it's 64 bits on MSVC, which doesn't claim to implement C99 nor even provide a stdint.h in the 2008 version.) David Vierra MCEdit, a Minecraft World Editor http://www.mcedit.net http://twitter.com/codewarrior0 On 7/2/2015 10:11 AM, Ian Henriksen wrote: > On Thu, Jul 2, 2015 at 1:08 PM Robert McGibbon > wrote: > > Right, okay. I think I understand. > > -Robert > > On Thu, Jul 2, 2015 at 12:58 AM, Stefan Behnel > > wrote: > > Robert McGibbon schrieb am 02.07.2015 um 09:49: > >> "libc.stdint.int64_t" is hand-wavingly declared as "long" > > > > There are some deeper issues in the rest of your message, > but as a > > preliminary matter, isn't this a clear error for linux-32 > and windows? > > No, it's not. That's just what Cython sees. The C compiler > then sees the > exact platform specific type. And as the vast amount of Cython > code out > there shows, it's usually not a problem in practice. Cython is > designed to > handle most of these "platform specific type" issues at C > compile time > rather than C code generation time. But as your example shows, > it can't > always hide the details entirely. And there can be bugs. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > > > This is an interesting discussion. Thanks. > For the particular case of int64_t, wouldn't it still be wise to > make it a typedef of long long rather than just long so as to > consistently get the correct size on platforms where long > is only 32 bits? > Thanks! > -Ian Henriksen > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Jul 3 08:54:48 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 03 Jul 2015 08:54:48 +0200 Subject: [Cython] Use of long type for intermediate integral variables In-Reply-To: <5595D6B2.8030503@gmail.com> References: <5594DA77.2090306@behnel.de> <5594EF31.8020008@behnel.de> <5595D6B2.8030503@gmail.com> Message-ID: <559631B8.9040708@behnel.de> David Vierra schrieb am 03.07.2015 um 02:26: > I see that Cython doesn't actually know the > underlying types of typedefs because it doesn't parse any C headers. And even if it did parse header files, it still wouldn't know the properties of the underlying types because they are platform specific and only turn into concrete types and sizes at C compile time. Stefan From insertinterestingnamehere at gmail.com Sat Jul 4 00:43:51 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Fri, 03 Jul 2015 22:43:51 +0000 Subject: [Cython] Multidimensional indexing of C++ objects Message-ID: Hi everyone, I'm a GSOC student working to make a Cython API for DyND. DyND is a relatively new n-dimensional array library in C++ that is based on NumPy. A full set of Python bindings (created using Cython) are provided as a separate package. The goal of my project is to make it so that DyND arrays can be used easily within Cython so that an n-dimensional array object can be used without any of the corresponding Python overhead. Currently, there isn't a good way to assign to multidimensional slices within Cython. Since the indexing operator in C++ is limited to a single argument, we use the call operator to represent multidimensional indexing, and then use a proxy class to perform assignment to a slice. Currently, in C++, assigning to a slice along the second axis of a DyND array looks like this: a(irange(), 1).vals() = 0; Unfortunately, in Cython, only the index operator can be used for assignment, so following the C++ syntax isn't currently possible. Does anyone know of a good way to address this? I'm willing to spend some time implementing a new feature if we can reach a consensus on a good way to deal with this. Here are some possible solutions I've thought of: 1. We could allow assignment to C++ method and function calls that return references. This has the advantage that it matches the existing syntax in C++ for dealing with C++ objects. Though Cython is a Python-like language, the ability to manipulate C++ objects directly is a key part of its feature set. Since the native way to do things like multidimensional indexing in C++ is via the call operator, it seems sensible to allow assignment to C++-level call operations in Cython as well. This could be enabled via a Cython compiler directive and be disabled by default. Using a compiler directive like this would result in an interface similar to the one already used for cdivision, wrap-around indexing, and index bounds checking. The user would avoid unexpected results by default, but be able to get the needed functionality simply by enabling it. 2. We could recommend that all assignment operations of this nature be wrapped in a fake method that wraps the assignment in it's c-level name. This has the advantage that it works in current and past versions of Cython, but it is a rather unusual hack. For example, something like the following would work right now: # declared as a method in a pxd file: void assign "vals() = "(int value) except + # used in a pyx file to perform assignment to a slice of an array a: a(irange(), 1).assign(0) For DyND, at least for now, this would be a workable solution since the difference lies primarily in the placement of the parenthesis and the presence of the assignment operator. The syntax is less clear than it could be, but it would work. On the other hand, other libraries may not be so lucky since this involves replacing assignment to a slice with a method call. For example, the expression template libraries Eigen and Blaze-lib would encounter incompatibility to varying degrees if someone were to try using them within Cython. This method also has the disadvantage that it creates an interface that is fundamentally different from both the Python and C++ interfaces. I have also considered, writing a proxy class that can serve as an effective temporary value while a multidimensional index is constructed from a series of calls to operator[]. This is a reasonable approach, but it leads to unnecessary code bloat. It also complicates the interface exposed to users, since operator[] would be needed for left hand values and operator() would be needed for right hand values. This would also make it so that users that want to use these C++ classes in Cython would have to include and link against another set of headers and libraries to be able to use the proxy class. The burden of maintainability for Python bindings created in this way would be greater as well. This also isn't a viable approach for using any C++ class that overloads both operators. Another option I have considered is allowing Cython's indexing operator to dispatch to a different function. Currently, user-defined cname entries for overloaded operators are not used. If this were changed for the indexing operator, indexing could be performed at the C++ level using some other method. This doesn't look like a viable approach though, since, for this to really work, users would need some way to call different methods when a C++ object is being indexed and when it is being assigned to. Using operator[] for left-hand values and operator() for right-hand values is a possible solution, but that isn't a very consistent interface. Doing this would also increase the complexity of the existing code for indexing in the Cython compiler and could lead to name collisions for classes that overload both operator[] and operator(). Are any of these acceptable ways to go forward? Does anyone have any better ideas? My preference would definitely be toward allowing C++ calls returning references to be used as lvalues, but I'd really appreciate any alternative solutions. Thanks! -Ian Henriksen -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sat Jul 4 08:37:07 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 04 Jul 2015 08:37:07 +0200 Subject: [Cython] Multidimensional indexing of C++ objects In-Reply-To: References: Message-ID: <55977F13.5050407@behnel.de> Hi Ian! Ian Henriksen schrieb am 04.07.2015 um 00:43: > I'm a GSOC student working to make a Cython API for DyND. DyND > is a relatively new n-dimensional > array library in C++ that is based on NumPy. A full set of Python bindings > (created using Cython) are provided as a separate package. The goal of my > project is to make it so that DyND arrays can be used easily within Cython > so that an n-dimensional array object can be used without any of the > corresponding Python overhead. > > Currently, there isn't a good way to assign to multidimensional slices > within Cython. Since the indexing operator in C++ is limited to a single > argument, we use the call operator to represent multidimensional indexing, > and then use a proxy class to perform assignment to a slice. > Currently, in C++, assigning to a slice along the second axis of a DyND > array looks like this: > > a(irange(), 1).vals() = 0; > > Unfortunately, in Cython, only the index operator can be used for > assignment, so following the C++ syntax isn't currently possible. Does > anyone know of a good way to address this? Just an idea, don't know how feasible this is, but we could allow inline special methods in C++ class declarations that implement Python protocols. Example: cdef extern from ...: cppclass Array2D: int operator[] except + int getItemAt(ssize_t x, ssize_t y) except + cdef inline __getitem__(self, Py_ssize_t x, Py_ssize_t y): return self.getItemAt(x, y) def test(): cdef Array2D a return a[1, 2] Cython could then translate an item access on an Array2D instance into the corresponding special "method" call. Drawbacks: 1) The example above would conflict with the C++ [] operator, so it would be ambiguous which one is being used in Cython code. Not sure if there's a use case for making both available to Cython code, but that would be difficult to achieve if the need arises. 2) It doesn't solve the general problem of assigning to C++ expressions, especially because it does not extend the syntax allowed by Cython which would still limit what you can do in these fake special methods. Regarding your proposals, I'd be happy if we could avoid adding syntax support for assigning to function calls. And I agree that the cname assignment hack is really just a big hack. It shouldn't be relied on. Stefan From michael at ensslin.cc Sat Jul 4 13:18:24 2015 From: michael at ensslin.cc (=?windows-1252?Q?Michael_En=DFlin?=) Date: Sat, 04 Jul 2015 13:18:24 +0200 Subject: [Cython] Multidimensional indexing of C++ objects In-Reply-To: <55977F13.5050407@behnel.de> References: <55977F13.5050407@behnel.de> Message-ID: <5597C100.2090702@ensslin.cc> Drawback 1) could be solved by declaring the operator as cdef inline operator []() instead of cdef inline __getitem__() -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From stefan_ml at behnel.de Sat Jul 4 13:34:37 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 04 Jul 2015 13:34:37 +0200 Subject: [Cython] Multidimensional indexing of C++ objects In-Reply-To: <5597C100.2090702@ensslin.cc> References: <55977F13.5050407@behnel.de> <5597C100.2090702@ensslin.cc> Message-ID: <5597C4CD.1050300@behnel.de> Michael En?lin schrieb am 04.07.2015 um 13:18: > Drawback 1) could be solved by declaring the operator as > > cdef inline operator []() > > instead of > > cdef inline __getitem__() But that would even shadow the operator entirely. The __getitem__ method is meant to provide a Python/Cython-level operator implementation, whereas operator[] declares the C++ level operation. The inline method might even want to use the C++ operator, or do its entirely own thing. Both are really separate levels. OTOH, if users actually provide their own implementation, I guess their intention is rather clear about *not* wanting the original C++ operator for some reason, even if it's implemented. So maybe shadowing isn't actually wrong here. Can't say if it really makes a difference which of the two inline methods above we allow or which one would appear cleaner. Adapting the operator interfaces in Cython should be possible in both cases. Stefan From josh at ayers.pw Tue Jul 7 17:11:39 2015 From: josh at ayers.pw (Josh Ayers) Date: Tue, 07 Jul 2015 08:11:39 -0700 Subject: [Cython] PyUnicode_Tailmatch Message-ID: <1436281899.3294972.317405161.314ABEDD@webmail.messagingengine.com> Cython devs, In the function __Pyx_PyUnicode_Tailmatch, the return type of PyUnicode_Tailmatch is assumed to be in int. See line 543 of Cython/Utility/StringTools.c. PyUnicode_Tailmatch actually returns a Py_ssize_t. See: https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Tailmatch. For reference, there was previously an error in the Python documentation of this function. The documentation mistakenly said it returned an int. This was fixed via Python issue 22580: https://bugs.python.org/issue22580. Thanks, Josh Ayers From robertwb at gmail.com Tue Jul 7 19:05:30 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 7 Jul 2015 10:05:30 -0700 Subject: [Cython] PyUnicode_Tailmatch In-Reply-To: <1436281899.3294972.317405161.314ABEDD@webmail.messagingengine.com> References: <1436281899.3294972.317405161.314ABEDD@webmail.messagingengine.com> Message-ID: Fixed (though I'm inclined to agree with the assessment that this choice of return type was a bug in the first place). https://github.com/cython/cython/commit/0a5890216d29d7bce941c9ab5cb0cb818eed643d On Tue, Jul 7, 2015 at 8:11 AM, Josh Ayers wrote: > Cython devs, > > In the function __Pyx_PyUnicode_Tailmatch, the return type of > PyUnicode_Tailmatch is assumed to be in int. See line 543 of > Cython/Utility/StringTools.c. > > PyUnicode_Tailmatch actually returns a Py_ssize_t. See: > https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Tailmatch. > > For reference, there was previously an error in the Python documentation > of this function. The documentation mistakenly said it returned an int. > This was fixed via Python issue 22580: > https://bugs.python.org/issue22580. > > Thanks, > Josh Ayers > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel From stefan_ml at behnel.de Tue Jul 7 21:07:52 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 07 Jul 2015 21:07:52 +0200 Subject: [Cython] PyUnicode_Tailmatch In-Reply-To: References: <1436281899.3294972.317405161.314ABEDD@webmail.messagingengine.com> Message-ID: <559C2388.5020400@behnel.de> Robert Bradshaw schrieb am 07.07.2015 um 19:05: > On Tue, Jul 7, 2015 at 8:11 AM, Josh Ayers wrote: >> In the function __Pyx_PyUnicode_Tailmatch, the return type of >> PyUnicode_Tailmatch is assumed to be in int. See line 543 of >> Cython/Utility/StringTools.c. >> >> PyUnicode_Tailmatch actually returns a Py_ssize_t. See: >> https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Tailmatch. >> >> For reference, there was previously an error in the Python documentation >> of this function. The documentation mistakenly said it returned an int. >> This was fixed via Python issue 22580: >> https://bugs.python.org/issue22580. > > Fixed (though I'm inclined to agree with the assessment that this > choice of return type was a bug in the first place). > > https://github.com/cython/cython/commit/0a5890216d29d7bce941c9ab5cb0cb818eed643d That's not a complete fix, though, because it would also be necessary to widen the types wherever these functions are called. I therefore prefer an explicit and local downcast to "int" over letting this API quirk spread. https://github.com/cython/cython/commit/9962ade0f048dac953b974ff9dddf087ed2b8fab Stefan From robertwb at gmail.com Tue Jul 7 21:30:38 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 7 Jul 2015 12:30:38 -0700 Subject: [Cython] PyUnicode_Tailmatch In-Reply-To: <559C2388.5020400@behnel.de> References: <1436281899.3294972.317405161.314ABEDD@webmail.messagingengine.com> <559C2388.5020400@behnel.de> Message-ID: On Tue, Jul 7, 2015 at 12:07 PM, Stefan Behnel wrote: > Robert Bradshaw schrieb am 07.07.2015 um 19:05: >> On Tue, Jul 7, 2015 at 8:11 AM, Josh Ayers wrote: >>> In the function __Pyx_PyUnicode_Tailmatch, the return type of >>> PyUnicode_Tailmatch is assumed to be in int. See line 543 of >>> Cython/Utility/StringTools.c. >>> >>> PyUnicode_Tailmatch actually returns a Py_ssize_t. See: >>> https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Tailmatch. >>> >>> For reference, there was previously an error in the Python documentation >>> of this function. The documentation mistakenly said it returned an int. >>> This was fixed via Python issue 22580: >>> https://bugs.python.org/issue22580. >> >> Fixed (though I'm inclined to agree with the assessment that this >> choice of return type was a bug in the first place). >> >> https://github.com/cython/cython/commit/0a5890216d29d7bce941c9ab5cb0cb818eed643d > > That's not a complete fix, though, because it would also be necessary to > widen the types wherever these functions are called. I therefore prefer an > explicit and local downcast to "int" over letting this API quirk spread. > > https://github.com/cython/cython/commit/9962ade0f048dac953b974ff9dddf087ed2b8fab I did look at the callsites--they were all conversion to a boolean or the implementation of an explicit call to PyUnicode_Tailmatch in user code. - Robert From insertinterestingnamehere at gmail.com Wed Jul 8 03:50:56 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Wed, 08 Jul 2015 01:50:56 +0000 Subject: [Cython] Multidimensional indexing of C++ objects In-Reply-To: <55977F13.5050407@behnel.de> References: <55977F13.5050407@behnel.de> Message-ID: On Sat, Jul 4, 2015 at 12:43 AM Stefan Behnel wrote: > Hi Ian! > > Ian Henriksen schrieb am 04.07.2015 um 00:43: > > I'm a GSOC student working to make a Cython API for DyND. DyND > > is a relatively new n-dimensional > > array library in C++ that is based on NumPy. A full set of Python > bindings > > (created using Cython) are provided as a separate package. The goal of my > > project is to make it so that DyND arrays can be used easily within > Cython > > so that an n-dimensional array object can be used without any of the > > corresponding Python overhead. > > > > Currently, there isn't a good way to assign to multidimensional slices > > within Cython. Since the indexing operator in C++ is limited to a single > > argument, we use the call operator to represent multidimensional > indexing, > > and then use a proxy class to perform assignment to a slice. > > Currently, in C++, assigning to a slice along the second axis of a DyND > > array looks like this: > > > > a(irange(), 1).vals() = 0; > > > > Unfortunately, in Cython, only the index operator can be used for > > assignment, so following the C++ syntax isn't currently possible. Does > > anyone know of a good way to address this? > > Just an idea, don't know how feasible this is, but we could allow inline > special methods in C++ class declarations that implement Python protocols. > Example: > > cdef extern from ...: > cppclass Array2D: > int operator[] except + > int getItemAt(ssize_t x, ssize_t y) except + > > cdef inline __getitem__(self, Py_ssize_t x, Py_ssize_t y): > return self.getItemAt(x, y) > > def test(): > cdef Array2D a > return a[1, 2] > > Cython could then translate an item access on an Array2D instance into the > corresponding special "method" call. > > Drawbacks: > > 1) The example above would conflict with the C++ [] operator, so it would > be ambiguous which one is being used in Cython code. Not sure if there's a > use case for making both available to Cython code, but that would be > difficult to achieve if the need arises. > > 2) It doesn't solve the general problem of assigning to C++ expressions, > especially because it does not extend the syntax allowed by Cython which > would still limit what you can do in these fake special methods. > > Regarding your proposals, I'd be happy if we could avoid adding syntax > support for assigning to function calls. And I agree that the cname > assignment hack is really just a big hack. It shouldn't be relied on. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel Yes, both this idea and the modified version that redefines operator[] are similar to the idea I had about respecting the cname entries for operator[]. This method would certainly expose a more flexible API for modules that want to do this. It may work in my case, but I worry that getting this into Cython would further complicate the (already lengthy) indexing logic. I'm still uneasy about exporting an API that is fundamentally different from the existing Python and C++ APIs, but making a way to use Python's syntax could help with that. Is there a good way to make a method like this accept Python-like indexing syntax? It would be confusing to put a code definition like this inside an extern block too. Could this syntax be adapted to work outside the extern block while still showing its connection to the original cppclass? Thanks for looking at it! -Ian -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Wed Jul 8 19:01:53 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 08 Jul 2015 19:01:53 +0200 Subject: [Cython] Multidimensional indexing of C++ objects In-Reply-To: References: <55977F13.5050407@behnel.de> Message-ID: <559D5781.9050907@behnel.de> Ian Henriksen schrieb am 08.07.2015 um 03:50: > On Sat, Jul 4, 2015 at 12:43 AM Stefan Behnel wrote: >> Ian Henriksen schrieb am 04.07.2015 um 00:43: >>> I'm a GSOC student working to make a Cython API for DyND. DyND >>> is a relatively new n-dimensional >>> array library in C++ that is based on NumPy. A full set of Python >> bindings >>> (created using Cython) are provided as a separate package. The goal of my >>> project is to make it so that DyND arrays can be used easily within >> Cython >>> so that an n-dimensional array object can be used without any of the >>> corresponding Python overhead. >>> >>> Currently, there isn't a good way to assign to multidimensional slices >>> within Cython. Since the indexing operator in C++ is limited to a single >>> argument, we use the call operator to represent multidimensional >> indexing, >>> and then use a proxy class to perform assignment to a slice. >>> Currently, in C++, assigning to a slice along the second axis of a DyND >>> array looks like this: >>> >>> a(irange(), 1).vals() = 0; >>> >>> Unfortunately, in Cython, only the index operator can be used for >>> assignment, so following the C++ syntax isn't currently possible. Does >>> anyone know of a good way to address this? >> >> Just an idea, don't know how feasible this is, but we could allow inline >> special methods in C++ class declarations that implement Python protocols. >> Example: >> >> cdef extern from ...: >> cppclass Array2D: >> int operator[] except + >> int getItemAt(ssize_t x, ssize_t y) except + >> >> cdef inline __getitem__(self, Py_ssize_t x, Py_ssize_t y): >> return self.getItemAt(x, y) >> >> def test(): >> cdef Array2D a >> return a[1, 2] >> >> Cython could then translate an item access on an Array2D instance into the >> corresponding special "method" call. >> >> Drawbacks: >> >> 1) The example above would conflict with the C++ [] operator, so it would >> be ambiguous which one is being used in Cython code. Not sure if there's a >> use case for making both available to Cython code, but that would be >> difficult to achieve if the need arises. >> >> 2) It doesn't solve the general problem of assigning to C++ expressions, >> especially because it does not extend the syntax allowed by Cython which >> would still limit what you can do in these fake special methods. >> >> Regarding your proposals, I'd be happy if we could avoid adding syntax >> support for assigning to function calls. And I agree that the cname >> assignment hack is really just a big hack. It shouldn't be relied on. > > Yes, both this idea and the modified version that redefines operator[] are > similar to the idea I had about respecting the cname entries for > operator[]. This method would certainly expose a more flexible API for > modules that want to do this. It may work in my case, but I worry that > getting this into Cython would further complicate the (already lengthy) > indexing logic. The main problem with the logic in IndexNode is that it predates the infrastructure change that allows node replacements in the analyse_types() methods. It should eventually be split into separate nodes that do different things, e.g. integer indexing into C arrays, Python object item access, C++ operator[] usage, buffer/memory view indexing, memory view slicing, you name it. In any case, adding new functionality can now be done by creating a new node rather than complicating the type analysis code. And any further refactoring would be warmly appreciated. :) > I'm still uneasy about exporting an API that is > fundamentally different from the existing Python and C++ APIs, but making a > way to use Python's syntax could help with that. Is there a good way to > make a method like this accept Python-like indexing syntax? It would be > confusing to put a code definition like this inside an extern block too. > Could this syntax be adapted to work outside the extern block while still > showing its connection to the original cppclass? The feature of providing inline functions in .pxd files already exists, as does the feature of adding functionality to external extension types by implementing special methods in their declaration. See, for example, the buffer protocol support for old NumPy arrays that we implemented in numpy/__init__pxd (look for "__getbuffer__") or the helper functions in cpython/array.pxd. Allowing to override __getitem__() in an extern C++ class declaration would really only be one step further. The question is whether __getitem__() is the right abstraction to use here as it also only accepts a single argument as input. That would be a tuple in Python for multi-dimensional lookup. It would be nice if the index arguments (e.g. x,y,z for 3 dimensions) could be explicit in the method signature instead, potentially using default arguments if less dimensions should be allowed. Stefan From jdemeyer at cage.ugent.be Mon Jul 13 15:56:32 2015 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Mon, 13 Jul 2015 15:56:32 +0200 Subject: [Cython] Regression: cdef append() no longer works Message-ID: <55A3C390.7090401@cage.ugent.be> Code like the following no longer works correctly: cdef class Foo: def __init__(self): self.append(1) cdef append(self, x): return x It seems that Cython now always assumes that append() is a special method: it incorrectly uses __Pyx_PyObject_Append. PS: it's not really clear if bug reports like these should go on this mailing list or on Trac (my impression is that Cython's Trac is not really used). What's the recommended procedure? From robertwb at gmail.com Tue Jul 14 02:13:12 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Mon, 13 Jul 2015 17:13:12 -0700 Subject: [Cython] Regression: cdef append() no longer works In-Reply-To: <55A3C390.7090401@cage.ugent.be> References: <55A3C390.7090401@cage.ugent.be> Message-ID: On Mon, Jul 13, 2015 at 6:56 AM, Jeroen Demeyer wrote: > Code like the following no longer works correctly: > > cdef class Foo: > def __init__(self): > self.append(1) > > cdef append(self, x): > return x > > It seems that Cython now always assumes that append() is a special method: > it incorrectly uses __Pyx_PyObject_Append. Fixed: https://github.com/cython/cython/commit/f75cad4297e437e9d9ab9e0ad43b8af6b351ba9a > PS: it's not really clear if bug reports like these should go on this > mailing list or on Trac (my impression is that Cython's Trac is not really > used). What's the recommended procedure? Trac is certainly not as actively used as projects like Sage, though we do use it. For something simple like this I'd send a mail the list and then if it's not fixed right away make sure to file a bug so it doesn't get lost. From stefan_ml at behnel.de Tue Jul 14 08:15:30 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 14 Jul 2015 08:15:30 +0200 Subject: [Cython] Regression: cdef append() no longer works In-Reply-To: References: <55A3C390.7090401@cage.ugent.be> Message-ID: <55A4A902.8070609@behnel.de> Robert Bradshaw schrieb am 14.07.2015 um 02:13: > On Mon, Jul 13, 2015 at 6:56 AM, Jeroen Demeyer wrote: >> PS: it's not really clear if bug reports like these should go on this >> mailing list or on Trac (my impression is that Cython's Trac is not really >> used). What's the recommended procedure? > > Trac is certainly not as actively used as projects like Sage, though > we do use it. For something simple like this I'd send a mail the list > and then if it's not fixed right away make sure to file a bug so it > doesn't get lost. +1, that works best for me, too. Stefan From markflorisson88 at gmail.com Tue Jul 14 20:34:52 2015 From: markflorisson88 at gmail.com (Mark Florisson) Date: Tue, 14 Jul 2015 19:34:52 +0100 Subject: [Cython] overflow bug? Message-ID: Hi, I think this might be a bug (python 3.3 + cython 0.22.1): def f(term=b"12345"): val = int('987278186585') # The below line does not work, because it treats 1 as a constant integer # in the C code (32 bit on my machine). Using 1L does work however. val -= 1 << (len(term) * 8) return val print(f()) This works in pure-python, but Cython generates '1 << __pyx_t_somevar', which I think treats the '1' as an integer (causing it to overflow). Using '1L' works in the Cython code however (but that may be just my platform). Cheers, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From insertinterestingnamehere at gmail.com Wed Jul 15 01:32:20 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Tue, 14 Jul 2015 23:32:20 +0000 Subject: [Cython] Handling of cascaded assignment Message-ID: I came across this while working on allowing proper overloading of the assignment operator. The problem is, essentially, that Cython does not follow the C/C++ ordering for cascaded assignment, even when dealing with objects at the C/C++ level. For example, calling the test function from the following Cython file and running the following C++ file should both give the same output: def test(): cdef: long long a = (2LL << 36) - 257 int b char c b = c = a print a, b, c #include int main(){ long long a = (2LL << 36) - 257; int b; char c; b = c = a; std::cout << a << std::endl; std::cout << b << std::endl; std::cout << (int)c << std::endl; } The Cython version prints 137438953215 -257 -1 and the C++ version prints 137438953215 -1 -1 To mimic C/C++, it should go through the arguments from right to left and perform each assignment operation in that order, i.e. c = a b = c As it stands, the current code does: b = a c = a This does appear to follow what Python does. For example, the following gives the same output as the Cython version: import numpy as np a = np.array([(2 << 36) - 257], np.int64) b = np.empty(1, np.int32) c = np.empty(1, np.int8) b[:] = c[:] = a print a[0], b[0], c[0] Is this a deliberate design decision? If this weren't already included in the language, I'd be inclined to not allow cascaded assignment to C objects at all since it can give such surprising results. On the other hand I don't see any particularly clean way to deal with this incompatibility between C/C++ and Python. Thanks! -Ian Henriksen -------------- next part -------------- An HTML attachment was scrubbed... URL: From insertinterestingnamehere at gmail.com Wed Jul 15 03:30:39 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Wed, 15 Jul 2015 01:30:39 +0000 Subject: [Cython] Multidimensional indexing of C++ objects In-Reply-To: <559D5781.9050907@behnel.de> References: <55977F13.5050407@behnel.de> <559D5781.9050907@behnel.de> Message-ID: On Wed, Jul 8, 2015 at 11:20 AM Stefan Behnel wrote: > Ian Henriksen schrieb am 08.07.2015 um 03:50: > > On Sat, Jul 4, 2015 at 12:43 AM Stefan Behnel wrote: > >> Ian Henriksen schrieb am 04.07.2015 um 00:43: > >>> I'm a GSOC student working to make a Cython API for DyND. DyND > >>> is a relatively new n-dimensional > >>> array library in C++ that is based on NumPy. A full set of Python > >> bindings > >>> (created using Cython) are provided as a separate package. The goal of > my > >>> project is to make it so that DyND arrays can be used easily within > >> Cython > >>> so that an n-dimensional array object can be used without any of the > >>> corresponding Python overhead. > >>> > >>> Currently, there isn't a good way to assign to multidimensional slices > >>> within Cython. Since the indexing operator in C++ is limited to a > single > >>> argument, we use the call operator to represent multidimensional > >> indexing, > >>> and then use a proxy class to perform assignment to a slice. > >>> Currently, in C++, assigning to a slice along the second axis of a DyND > >>> array looks like this: > >>> > >>> a(irange(), 1).vals() = 0; > >>> > >>> Unfortunately, in Cython, only the index operator can be used for > >>> assignment, so following the C++ syntax isn't currently possible. Does > >>> anyone know of a good way to address this? > >> > >> Just an idea, don't know how feasible this is, but we could allow inline > >> special methods in C++ class declarations that implement Python > protocols. > >> Example: > >> > >> cdef extern from ...: > >> cppclass Array2D: > >> int operator[] except + > >> int getItemAt(ssize_t x, ssize_t y) except + > >> > >> cdef inline __getitem__(self, Py_ssize_t x, Py_ssize_t y): > >> return self.getItemAt(x, y) > >> > >> def test(): > >> cdef Array2D a > >> return a[1, 2] > >> > >> Cython could then translate an item access on an Array2D instance into > the > >> corresponding special "method" call. > >> > >> Drawbacks: > >> > >> 1) The example above would conflict with the C++ [] operator, so it > would > >> be ambiguous which one is being used in Cython code. Not sure if > there's a > >> use case for making both available to Cython code, but that would be > >> difficult to achieve if the need arises. > >> > >> 2) It doesn't solve the general problem of assigning to C++ expressions, > >> especially because it does not extend the syntax allowed by Cython which > >> would still limit what you can do in these fake special methods. > >> > >> Regarding your proposals, I'd be happy if we could avoid adding syntax > >> support for assigning to function calls. And I agree that the cname > >> assignment hack is really just a big hack. It shouldn't be relied on. > > > > Yes, both this idea and the modified version that redefines operator[] > are > > similar to the idea I had about respecting the cname entries for > > operator[]. This method would certainly expose a more flexible API for > > modules that want to do this. It may work in my case, but I worry that > > getting this into Cython would further complicate the (already lengthy) > > indexing logic. > > The main problem with the logic in IndexNode is that it predates the > infrastructure change that allows node replacements in the analyse_types() > methods. It should eventually be split into separate nodes that do > different things, e.g. integer indexing into C arrays, Python object item > access, C++ operator[] usage, buffer/memory view indexing, memory view > slicing, you name it. > > In any case, adding new functionality can now be done by creating a new > node rather than complicating the type analysis code. And any further > refactoring would be warmly appreciated. :) > > > > I'm still uneasy about exporting an API that is > > fundamentally different from the existing Python and C++ APIs, but > making a > > way to use Python's syntax could help with that. Is there a good way to > > make a method like this accept Python-like indexing syntax? It would be > > confusing to put a code definition like this inside an extern block too. > > Could this syntax be adapted to work outside the extern block while still > > showing its connection to the original cppclass? > > The feature of providing inline functions in .pxd files already exists, as > does the feature of adding functionality to external extension types by > implementing special methods in their declaration. See, for example, the > buffer protocol support for old NumPy arrays that we implemented in > numpy/__init__pxd (look for "__getbuffer__") or the helper functions in > cpython/array.pxd. > > Allowing to override __getitem__() in an extern C++ class declaration would > really only be one step further. The question is whether __getitem__() is > the right abstraction to use here as it also only accepts a single argument > as input. That would be a tuple in Python for multi-dimensional lookup. It > would be nice if the index arguments (e.g. x,y,z for 3 dimensions) could be > explicit in the method signature instead, potentially using default > arguments if less dimensions should be allowed. > > Stefan > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel Okay, I see what you mean in the numpy pxd file. With regards to which abstraction to use, __getitem__ and __setitem__ would work best, but we would need to allow a greater number of arguments. We could force users to use something like a C++ vector or tuple, but that creates a bunch of useless temporaries (which we can only hope the C++ compiler will handle properly) and will cause all kinds of grief when mixing code compiled with different standard libraries. That's not the end of the world, but I'd prefer to avoid incompatibilities like that wherever possible. Using __getitem__ and __setitem__ does have the advantage that it would maintain a separation between using an indexing operation on the right or left of an assignment so that libraries could handle the two cases separately. That has the potential to make the Cython API exposed to users simpler than the C++ API in some cases, though overwriting operator[] comes with its own set of problems. Using operator[] as a name for a Cython-level indexing operation would work too. Since we're already overloading the syntax, the name is available. On the other hand, there don't appear to be any additional benefits other than leaving __getitem__ and __setitem__ alone. I feel like using the same name as is used in C++ could be a potential point of confusion. Rewriting a C++ class's handling of indexing in code that primarily provides a wrapper is plenty confusing on its own. Using different names would at least help accentuate the change. With regards to defining these special methods, how would one go about using the C++ version of the syntax? I've been trying to make it so that all the Python compatibility functions in dynd-python can be loaded dynamically via cimport so that the users will only have to find and include the headers from the original C++ library. With the additional special methods like this, the only way I can see to do this is to define C++ functions that operate on the C++ objects, provide their declarations and make Cython shims in a pyx file, then use the Cython shims inside the pxd file to define the special __getitem__ and __setitem__ methods. Even though most users won't see it, that's an awful lot of code for any library writer that wants to expose an interface like this. Do you see any better solution? Thanks for the great suggestions! -Ian Henriksen -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Wed Jul 15 07:09:42 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 15 Jul 2015 07:09:42 +0200 Subject: [Cython] Handling of cascaded assignment In-Reply-To: References: Message-ID: <55A5EB16.8070902@behnel.de> Ian Henriksen schrieb am 15.07.2015 um 01:32: > The problem is, essentially, that Cython does not follow the C/C++ > ordering for cascaded assignment, even when dealing with objects at > the C/C++ level. > [...] > This does appear to follow what Python does. > [...] > Is this a deliberate design decision? Yes. Cython is Python with C/C++ data types, not the other way round. Stefan From alex0player at gmail.com Wed Jul 15 13:51:03 2015 From: alex0player at gmail.com (Alex S.) Date: Wed, 15 Jul 2015 14:51:03 +0300 Subject: [Cython] Fwd: [BUG] Cython failing performance when Numpy is not installed In-Reply-To: References: Message-ID: Hello, being unable to acquire the Trac account (Robert Bradshaw's email is not listed on his site), I write here. The issue was described here: http://trac.cython.org/ticket/847 but was, however, misidentified. The wrapper for a function which receives "fused slice" (D[:]) arguments automatically includes the following code: > cdef type ndarray > try: > import numpy > ndarray = numpy.ndarray > except (ImportError, AttributeError, TypeError): > ndarray = None (Compiler/FusedNode.py:468). When numpy is not installed, each import tries to search for it in the system path. (When it is, it's just cached). This severely degrades the performance. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Brian.McAndrews at chicagotrading.com Thu Jul 16 23:13:36 2015 From: Brian.McAndrews at chicagotrading.com (Brian McAndrews) Date: Thu, 16 Jul 2015 21:13:36 +0000 Subject: [Cython] Problem with declaring a map with an enum type. Message-ID: <29040F2D79C0E742B2D46E059088835AEB9FEAF2@CH12WXEX00.chicagotrading.com> I have a C enum type that I'm attempting to put in a C++ map: Here's what I got: colors.h: enum Color { GREEN,BLUE,RED }; val.pyx: from libcpp.string cimport string from libcpp.map cimport map cdef extern from "colors.h": cpdef enum _Color "Color": GREEN,BLUE,RED cdef class TestMe: cdef public map[string, _Color] vals I get a compile error (VSC++ 2012) VAL.cpp(1239) : error C2440: '=' : cannot convert from 'long' to 'Color' Conversion to enumeration type requires an explicit cast (static_cast, C-style cast or function-style cast) The offending statement is this: __pyx_t_9 = PyInt_AsLong(__pyx_v_value); if (unlikely(PyErr_Occurred())) {__pyx_filename = __pyx_f[1]; __pyx_lineno = 202; __pyx_clineno = __LINE__; goto __pyx_L1_error;} In the __pyx_convert_map_from_py_std_3a__3a_string__and_enum__Color function. The cast from PyInt_AsLong(...) to an enum Color is not specified when using maps. If I don't put it in a map (just as a member variable), the cast is performed. Has anyone seen this before? Thanks, Brian This electronic mail message and any attached files contain information intended for the exclusive use of the individual or entity to whom it is addressed and may contain information that is proprietary, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any viewing, copying, disclosure or distribution of this information may be subject to legal restriction or sanction. Please notify the sender, by electronic mail or telephone, of any unintended recipients and delete the original message without making any copies. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Fri Jul 17 10:02:50 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 17 Jul 2015 10:02:50 +0200 Subject: [Cython] Fwd: [BUG] Cython failing performance when Numpy is not installed In-Reply-To: References: Message-ID: <55A8B6AA.3080306@behnel.de> Alex S. schrieb am 15.07.2015 um 13:51: > The issue was described here: http://trac.cython.org/ticket/847 but was, > however, misidentified. The wrapper for a function which receives "fused > slice" (D[:]) arguments automatically includes the following code: >> cdef type ndarray >> try: >> import numpy >> ndarray = numpy.ndarray >> except (ImportError, AttributeError, TypeError): >> ndarray = None > (Compiler/FusedNode.py:468). > When numpy is not installed, each import tries to search for it in the > system path. (When it is, it's just cached). This severely degrades the > performance. Sounds like enabling the absolute_import future import might help at least a little. https://github.com/cython/cython/commit/ac4f248b4109cb17b2da53dc7b38e359c5dee612 But yes, this is definitely a problem. In general, the way extension types are handled in the fused types dispatch code (and that includes numpy arrays) would seriously benefit from optimisation, including cached one-time imports. Stefan From robertwb at gmail.com Sat Jul 18 09:39:32 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Sat, 18 Jul 2015 00:39:32 -0700 Subject: [Cython] Problem with declaring a map with an enum type. In-Reply-To: <29040F2D79C0E742B2D46E059088835AEB9FEAF2@CH12WXEX00.chicagotrading.com> References: <29040F2D79C0E742B2D46E059088835AEB9FEAF2@CH12WXEX00.chicagotrading.com> Message-ID: On Thu, Jul 16, 2015 at 2:13 PM, Brian McAndrews wrote: > I have a C enum type that I?m attempting to put in a C++ map: Interesting, thanks for the report. Fixed at https://github.com/cython/cython/commit/427771298da04f0f5bb61659ddf4a61bd0f02af4 > Here?s what I got: > > > > colors.h: > > enum Color > > { > > GREEN,BLUE,RED > > }; > > > > val.pyx: > > from libcpp.string cimport string > > from libcpp.map cimport map > > > > cdef extern from "colors.h": > > cpdef enum _Color "Color": > > GREEN,BLUE,RED > > > > cdef class TestMe: > > cdef public map[string, _Color] vals > > > > > > I get a compile error (VSC++ 2012) > > VAL.cpp(1239) : error C2440: '=' : cannot convert from 'long' to 'Color' > > Conversion to enumeration type requires an explicit cast > (static_cast, C-style cast or function-style cast) > > > > The offending statement is this: > > __pyx_t_9 = PyInt_AsLong(__pyx_v_value); if (unlikely(PyErr_Occurred())) > {__pyx_filename = __pyx_f[1]; __pyx_lineno = 202; __pyx_clineno = __LINE__; > goto __pyx_L1_error;} > > > > In the __pyx_convert_map_from_py_std_3a__3a_string__and_enum__Color > function. > > > > The cast from PyInt_AsLong(?) to an enum Color is not specified when using > maps. > > > > If I don?t put it in a map (just as a member variable), the cast is > performed. > > > > Has anyone seen this before? > > > > Thanks, > > Brian > > > > > > > > This electronic mail message and any attached files contain information > intended for the exclusive use of the individual or entity to whom it is > addressed and may contain information that is proprietary, confidential > and/or exempt from disclosure under applicable law. If you are not the > intended recipient, you are hereby notified that any viewing, copying, > disclosure or distribution of this information may be subject to legal > restriction or sanction. Please notify the sender, by electronic mail or > telephone, of any unintended recipients and delete the original message > without making any copies. > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Mon Jul 20 09:50:12 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 20 Jul 2015 09:50:12 +0200 Subject: [Cython] Cython 0.23 beta 1 released Message-ID: <55ACA834.3000304@behnel.de> Hi everyone, I just uploaded a first beta for the upcoming Cython 0.23. There have been many major changes and fixes, so please give it some testing with your own code - especially on less commonly tested platforms, namely MS Windows. You can get the signed release from here: http://cython.org/release/ http://cython.org/release/Cython-0.23.beta1.tar.gz http://cython.org/release/Cython-0.23.beta1.zip SHA1 sums: 45347674bf14ebfce879dd1bf15af64387a39f8c Cython-0.23.beta1.tar.gz 0db13cdd8a11f83158e5c72f5e84b89e574af215 Cython-0.23.beta1.zip Changelog follows below. Since we're in beta phase now, new features should not generally be added for the release any more, but we can make exceptions for currently existing pull requests if you can provide a good motivation. If you think we forgot something, sending a timely reply to this email or reviving an existing mailing list thread or pull request is a good idea. Note that the coverage analysis support is still incomplete as the plugin API of the coverage.py tool itself is still subject to modifications. Have fun, Stefan Changelog: ========== Features added -------------- * PEP 492 (async/await) was implemented. See https://www.python.org/dev/peps/pep-0492/ * PEP 448 (Additional Unpacking Generalizations) was implemented. See https://www.python.org/dev/peps/pep-0448/ * Support for coverage.py 4.0+ can be enabled by adding the plugin "Cython.Coverage" to the ".coveragerc" config file. * Annotated HTML source pages can integrate (XML) coverage reports. * Tracing is supported in ``nogil`` functions/sections and module init code. * When generators are used in a Cython module and the module imports the modules "inspect" and/or "asyncio", Cython enables interoperability by patching these modules during the import to recognise Cython's internal generator and coroutine types. This can be disabled by C compiling the module with "-D CYTHON_PATCH_ASYNCIO=0" or "-D CYTHON_PATCH_INSPECT=0" * When generators or coroutines are used in a Cython module, their types are registered with the ``Generator`` and ``Coroutine`` ABCs in the ``collections`` or ``collections.abc`` stdlib module at import time to enable interoperability with code that needs to detect and process Python generators/coroutines. These ABCs were added in CPython 3.5 and are available for older Python versions through the ``backports_abc`` module on PyPI. See https://bugs.python.org/issue24018 * Adding/subtracting/dividing/modulus and equality comparisons with constant Python floats and small integers are faster. * Binary and/or/xor/rshift operations with small constant Python integers are faster. * When called on generator expressions, the builtins ``all()``, ``any()``, ``dict()``, ``list()``, ``set()``, ``sorted()`` and ``unicode.join()`` avoid the generator iteration overhead by inlining a part of their functionality into the for-loop. * Keyword argument dicts are no longer copied on function entry when they are not being used or only passed through to other function calls (e.g. in wrapper functions). * The ``PyTypeObject`` declaration in ``cpython.object`` was extended. * ``wraparound()`` and ``boundscheck()`` are available as no-ops in pure Python mode. * Const iterators were added to the provided C++ STL declarations. * ``NULL`` is allowed as default argument when embedding signatures. This fixes ticket 843. * When compiling with ``--embed``, the internal module name is changed to ``__main__`` to allow arbitrary program names, including those that would be invalid for modules. Note that this prevents reuse of the generated C code as an importable module. * External C++ classes that overload the assignment operator can be used. Patch by Ian Henriksen. Bugs fixed ---------- * Calling "yield from" from Python on a Cython generator that returned a value triggered a crash in CPython. This is now being worked around. See https://bugs.python.org/issue23996 * Language level 3 did not enable true division (a.k.a. float division) for integer operands. * Functions with fused argument types that included a generic 'object' fallback could end up using that fallback also for other explicitly listed object types. * Relative cimports could accidentally fall back to trying an absolute cimport on failure. * The result of calling a C struct constructor no longer requires an intermediate assignment when coercing to a Python dict. * C++ exception declarations with mapping functions could fail to compile when pre-declared in .pxd files. * ``cpdef void`` methods are now permitted. * ``abs(cint)`` could fail to compile in MSVC and used sub-optimal code in C++. Patch by David Vierra, original patch by Michael En?lin. * Buffer index calculations using index variables with small C integer types could overflow for large buffer sizes. Original patch by David Vierra. * C unions use a saner way to coerce from and to Python dicts. * When compiling a module ``foo.pyx``, the directories in ``sys.path`` are no longer searched when looking for ``foo.pxd``. Patch by Jeroen Demeyer. * Memory leaks in the embedding main function were fixed. Original patch by Michael En?lin. * Some complex Python expressions could fail to compile inside of finally clauses. Other changes ------------- * Changed mangling scheme in header files generated by ``cdef api`` declarations. From jdemeyer at cage.ugent.be Mon Jul 20 14:16:30 2015 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Mon, 20 Jul 2015 14:16:30 +0200 Subject: [Cython] Cython 0.23 beta 1 released In-Reply-To: <55ACA834.3000304@behnel.de> References: <55ACA834.3000304@behnel.de> Message-ID: <55ACE69E.1030508@cage.ugent.be> Hello, The changelog should mention something about the very recent changes to "enum". Some bad code in Sage (a literal integer pretending to be an enum) needed to be fixed because of this change. Jeroen. From insertinterestingnamehere at gmail.com Mon Jul 20 23:06:17 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Mon, 20 Jul 2015 21:06:17 +0000 Subject: [Cython] Non-type template parameters Message-ID: Hi all, I've spent a little time working on adding support for non-type template parameters. In doing this, it has been very easy to add support for doing something like the following: cdef extern from "add_const.hpp" nogil: int myfunc1[i](int) def test(): print myfunc1[2](a) The downside of this is that it does not let the Cython compiler distinguish between different kinds of template parameters. Stricter checking could be added using a syntax like this: cdef extern from "add_const.hpp" nogil: int myfunc1[int i](int) def test(): print myfunc1[2](a) The downsides here are that the syntax doesn't really match the existing template syntax. It will also complicate the Cython codebase since we'll have to go to greater lengths to allow or disallow all the different special cases for templates. Which version would be preferable? On a similar note, for variadic templates, would we prefer something like cdef extern from "my_variadic.hpp" nogil: T myfunc2[T,...](T, ...) or something like: cdef extern from "my_variadic.hpp" nogil: T myfunc2[T, Types...](T, Types... args) Again, the latter syntax is more explicit, but it will require much more complicated code in Cython. It also doesn't match the existing syntax very well. The former syntax matches the existing syntax for templates better, but will make it hard for Cython to raise errors early on in compilation. I'd greatly appreciate any input on the best syntax for either use-case. Regards, -Ian Henriksen -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex0player at gmail.com Tue Jul 21 12:47:26 2015 From: alex0player at gmail.com (Alex S.) Date: Tue, 21 Jul 2015 13:47:26 +0300 Subject: [Cython] Fwd: [BUG] Cython failing performance when Numpy is not installed In-Reply-To: <55A8B6AA.3080306@behnel.de> References: <55A8B6AA.3080306@behnel.de> Message-ID: 2015-07-17 11:02 GMT+03:00 Stefan Behnel : > Alex S. schrieb am 15.07.2015 um 13:51: > > The issue was described here: http://trac.cython.org/ticket/847 but was, > > however, misidentified. The wrapper for a function which receives "fused > > slice" (D[:]) arguments automatically includes the following code: > >> cdef type ndarray > >> try: > >> import numpy > >> ndarray = numpy.ndarray > >> except (ImportError, AttributeError, TypeError): > >> ndarray = None > > (Compiler/FusedNode.py:468). > > When numpy is not installed, each import tries to search for it in the > > system path. (When it is, it's just cached). This severely degrades the > > performance. > > Sounds like enabling the absolute_import future import might help at least > a little. > > > https://github.com/cython/cython/commit/ac4f248b4109cb17b2da53dc7b38e359c5dee612 > > But yes, this is definitely a problem. In general, the way extension types > are handled in the fused types dispatch code (and that includes numpy > arrays) would seriously benefit from optimisation, including cached > one-time imports. > > Stefan > > Thank you for your response! There's a similar problem in the Utility/MemoryView.py{,x} file concerning converting values from Python objects on index assignment. ("import struct") Is there any reason while this feature should not be on for all such code? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdemeyer at cage.ugent.be Thu Jul 23 23:14:28 2015 From: jdemeyer at cage.ugent.be (Jeroen Demeyer) Date: Thu, 23 Jul 2015 23:14:28 +0200 Subject: [Cython] Cython 0.23 beta 1 released In-Reply-To: <55ACE69E.1030508@cage.ugent.be> References: <55ACA834.3000304@behnel.de> <55ACE69E.1030508@cage.ugent.be> Message-ID: <55B15934.5050301@cage.ugent.be> On 2015-07-20 14:16, Jeroen Demeyer wrote: > Hello, > > The changelog should mention something about the very recent changes to > "enum". Some bad code in Sage (a literal integer pretending to be an > enum) needed to be fixed because of this change. I forgot to mention that, apart from this enum issue, everything works fine in Sage. Jeroen. From carlosjosepita at gmail.com Sat Jul 25 15:23:33 2015 From: carlosjosepita at gmail.com (Carlos Pita) Date: Sat, 25 Jul 2015 13:23:33 +0000 (UTC) Subject: [Cython] [RFE] Add dummy compiler directive decorators for pure python mode References: <552421E6.7080307@behnel.de> <5527C562.7060108@behnel.de> Message-ID: > > Not sure, but would it be desirable for the decorators to be less dummy and > > for RuntimeCompiledFunction to take the flags into account when compiling > > on the fly? > > Can you provide a pull request, including tests? I don't have much time these days but I gave some thoughts to this and I would like to hear your opinion about the following proposal before start coding: 1. The compiler-directives, locals, etc. decorators will be less dummy in two ways: - They will know how to print a repr of themselves. - They will register themselves under some "private" attribute (say _cython_decorators, do you have any preference here?) of the decorated function. 2. Besides get_body(f), RuntimeCompiledFunction will use a new function get_cython_decorators(f), which returns the concatenated repr of the registered decorators. 3. This block of code will be passed as a new argument (say cython_decorators) to cython_inline. The module code template will now be: module_code = """ %(module_body)s %(cimports)s %(cython_decorators)s def __invoke(%(params)s): %(func_body)s """ What do you think? Cheers -- Carlos From robertwb at gmail.com Tue Jul 28 09:35:51 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 28 Jul 2015 00:35:51 -0700 Subject: [Cython] Non-type template parameters In-Reply-To: References: Message-ID: Sorry for not getting back sooner... responses below. On Mon, Jul 20, 2015 at 2:06 PM, Ian Henriksen wrote: > Hi all, > I've spent a little time working on adding support for non-type > template parameters. In doing this, it has been very easy to add > support for doing something like the following: > > cdef extern from "add_const.hpp" nogil: > int myfunc1[i](int) > def test(): > print myfunc1[2](a) > > The downside of this is that it does not let the Cython compiler > distinguish between different kinds of template parameters. > > Stricter checking could be added using a syntax like this: > > cdef extern from "add_const.hpp" nogil: > int myfunc1[int i](int) > def test(): > print myfunc1[2](a) > > The downsides here are that the syntax doesn't really match the > existing template syntax. It will also complicate the Cython codebase > since we'll have to go to greater lengths to allow or disallow all the > different special cases for templates. > > Which version would be preferable? I think I'd prefer the [int i] syntax. Hopefully it shouldn't complicate things too much. > On a similar note, for variadic templates, would we prefer something > like > > cdef extern from "my_variadic.hpp" nogil: > T myfunc2[T,...](T, ...) > > or something like: > > cdef extern from "my_variadic.hpp" nogil: > T myfunc2[T, Types...](T, Types... args) > > Again, the latter syntax is more explicit, but it will require much more > complicated code in Cython. It also doesn't match the existing syntax > very well. The former syntax matches the existing syntax for templates > better, but will make it hard for Cython to raise errors early on in > compilation. Hmm... this is a tougher call. Let's go with the former for now. > I'd greatly appreciate any input on the best syntax for either use-case. > > Regards, > > -Ian Henriksen > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > From robertwb at gmail.com Tue Jul 28 09:24:46 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 28 Jul 2015 00:24:46 -0700 Subject: [Cython] [RFE] Add dummy compiler directive decorators for pure python mode In-Reply-To: References: <552421E6.7080307@behnel.de> <5527C562.7060108@behnel.de> Message-ID: On Sat, Jul 25, 2015 at 6:23 AM, Carlos Pita wrote: >> > Not sure, but would it be desirable for the decorators to be less dummy and >> > for RuntimeCompiledFunction to take the flags into account when compiling >> > on the fly? >> >> Can you provide a pull request, including tests? > > I don't have much time these days but I gave some thoughts to this and I > would like to hear your opinion about the following proposal before start > coding: > > 1. The compiler-directives, locals, etc. decorators will be less dummy in > two ways: > - They will know how to print a repr of themselves. > - They will register themselves under some "private" attribute (say > _cython_decorators, do you have any preference here?) of the decorated function. > > 2. Besides get_body(f), RuntimeCompiledFunction will use a new function > get_cython_decorators(f), which returns the concatenated repr of the > registered decorators. > > 3. This block of code will be passed as a new argument (say > cython_decorators) to cython_inline. The module code template will now be: > > module_code = """ > %(module_body)s > %(cimports)s > %(cython_decorators)s > def __invoke(%(params)s): > %(func_body)s > """ > > What do you think? +1 From insertinterestingnamehere at gmail.com Tue Jul 28 20:18:42 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Tue, 28 Jul 2015 18:18:42 +0000 Subject: [Cython] Non-type template parameters In-Reply-To: References: Message-ID: On Tue, Jul 28, 2015 at 1:36 AM Robert Bradshaw wrote: > Sorry for not getting back sooner... responses below. > > On Mon, Jul 20, 2015 at 2:06 PM, Ian Henriksen > wrote: > > Hi all, > > I've spent a little time working on adding support for non-type > > template parameters. In doing this, it has been very easy to add > > support for doing something like the following: > > > > cdef extern from "add_const.hpp" nogil: > > int myfunc1[i](int) > > def test(): > > print myfunc1[2](a) > > > > The downside of this is that it does not let the Cython compiler > > distinguish between different kinds of template parameters. > > > > Stricter checking could be added using a syntax like this: > > > > cdef extern from "add_const.hpp" nogil: > > int myfunc1[int i](int) > > def test(): > > print myfunc1[2](a) > > > > The downsides here are that the syntax doesn't really match the > > existing template syntax. It will also complicate the Cython codebase > > since we'll have to go to greater lengths to allow or disallow all the > > different special cases for templates. > > > > Which version would be preferable? > > I think I'd prefer the [int i] syntax. Hopefully it shouldn't > complicate things too much. > Okay, I think I see a way to make that work. On the other hand, since there weren't any replies here, I've already nearly finished implementing the first syntax. I'll spend another hour or two finishing it off later today and submit a PR so you can look it over. I originally favored the first syntax because it minimizes the number of fancy template features (SFINAE, for example) we have to worry about on the Cython end. I'm still open to discuss it though. > > > On a similar note, for variadic templates, would we prefer something > > like > > > > cdef extern from "my_variadic.hpp" nogil: > > T myfunc2[T,...](T, ...) > > > > or something like: > > > > cdef extern from "my_variadic.hpp" nogil: > > T myfunc2[T, Types...](T, Types... args) > > > > Again, the latter syntax is more explicit, but it will require much more > > complicated code in Cython. It also doesn't match the existing syntax > > very well. The former syntax matches the existing syntax for templates > > better, but will make it hard for Cython to raise errors early on in > > compilation. > > Hmm... this is a tougher call. Let's go with the former for now. > I like the former a lot more. It will keep the syntax simpler on our end and I haven't been able to find any case that it doesn't cover. This will also be significantly easier to implement. I'll take a look at it soon. Thanks! -Ian Henriksen > > > I'd greatly appreciate any input on the best syntax for either use-case. > > > > Regards, > > > > -Ian Henriksen > > > > _______________________________________________ > > cython-devel mailing list > > cython-devel at python.org > > https://mail.python.org/mailman/listinfo/cython-devel > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertwb at gmail.com Tue Jul 28 20:36:36 2015 From: robertwb at gmail.com (Robert Bradshaw) Date: Tue, 28 Jul 2015 11:36:36 -0700 Subject: [Cython] Non-type template parameters In-Reply-To: References: Message-ID: On Tue, Jul 28, 2015 at 11:18 AM, Ian Henriksen wrote: > > On Tue, Jul 28, 2015 at 1:36 AM Robert Bradshaw wrote: >> >> Sorry for not getting back sooner... responses below. >> >> On Mon, Jul 20, 2015 at 2:06 PM, Ian Henriksen >> wrote: >> > Hi all, >> > I've spent a little time working on adding support for non-type >> > template parameters. In doing this, it has been very easy to add >> > support for doing something like the following: >> > >> > cdef extern from "add_const.hpp" nogil: >> > int myfunc1[i](int) >> > def test(): >> > print myfunc1[2](a) >> > >> > The downside of this is that it does not let the Cython compiler >> > distinguish between different kinds of template parameters. >> > >> > Stricter checking could be added using a syntax like this: >> > >> > cdef extern from "add_const.hpp" nogil: >> > int myfunc1[int i](int) >> > def test(): >> > print myfunc1[2](a) >> > >> > The downsides here are that the syntax doesn't really match the >> > existing template syntax. It will also complicate the Cython codebase >> > since we'll have to go to greater lengths to allow or disallow all the >> > different special cases for templates. >> > >> > Which version would be preferable? >> >> I think I'd prefer the [int i] syntax. Hopefully it shouldn't >> complicate things too much. > > Okay, I think I see a way to make that work. On the other hand, since there > weren't any replies here, I've already nearly finished implementing the > first > syntax. I'll spend another hour or two finishing it off later today and > submit a PR > so you can look it over. I originally favored the first syntax because it > minimizes > the number of fancy template features (SFINAE, for example) we have to worry > about on the Cython end. I'm still open to discuss it though. I think this falls into the "explicit is better than implicit" bucket. That and getting obscure template errors that could have been caught at Cython compile time will be very nice. >> > On a similar note, for variadic templates, would we prefer something >> > like >> > >> > cdef extern from "my_variadic.hpp" nogil: >> > T myfunc2[T,...](T, ...) >> > >> > or something like: >> > >> > cdef extern from "my_variadic.hpp" nogil: >> > T myfunc2[T, Types...](T, Types... args) >> > >> > Again, the latter syntax is more explicit, but it will require much more >> > complicated code in Cython. It also doesn't match the existing syntax >> > very well. The former syntax matches the existing syntax for templates >> > better, but will make it hard for Cython to raise errors early on in >> > compilation. >> >> Hmm... this is a tougher call. Let's go with the former for now. > > > I like the former a lot more. It will keep the syntax simpler on our end and > I > haven't been able to find any case that it doesn't cover. This will also be > significantly easier to implement. I'll take a look at it soon. Sounds good. There's also the question of default arguments. Let's adopt the [type T = *] syntax for that. >> > I'd greatly appreciate any input on the best syntax for either use-case. >> > >> > Regards, >> > >> > -Ian Henriksen >> > >> > _______________________________________________ >> > cython-devel mailing list >> > cython-devel at python.org >> > https://mail.python.org/mailman/listinfo/cython-devel >> > >> _______________________________________________ >> cython-devel mailing list >> cython-devel at python.org >> https://mail.python.org/mailman/listinfo/cython-devel > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel > From stefan_ml at behnel.de Wed Jul 29 20:02:57 2015 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 29 Jul 2015 20:02:57 +0200 Subject: [Cython] Cython 0.23 beta 2 released Message-ID: <55B91551.5000504@behnel.de> Hi everyone, I just uploaded a second beta for the upcoming Cython 0.23. Please give it some more testing as there were substantial code changes since the first beta. You can get the signed release from here: http://cython.org/release/ http://cython.org/release/Cython-0.23b2.tar.gz http://cython.org/release/Cython-0.23b2.zip SHA1 sums: 1d12e335116ba3fcbe731d83ccc5ee5f0f6e5371 Cython-0.23b2.tar.gz 6f063e6b08b6dbecbadd44ed6c6150bc799fa230 Cython-0.23b2.zip Changes since beta 1 (in addition to general bug fixes): * The builtin ``type`` type is now declared as PyTypeObject in source, allowing for extern functions taking type parameters to have the correct C signatures. Note that this might break code that uses ``type`` just for passing around Python types in typed variables. Removing the type declaration provides a backwards compatible fix. * Support operator bool() for C++ classes so they can be used in if statements. * Installation under CPython 3.3+ no longer requires a pass of the 2to3 tool. This also makes it possible to run Cython in Python 3.3+ from a source checkout without installing it first. Patch by Petr Viktorin. * Coverage analysis and async/await were adapted to recent changes in coverage.py and CPython. Complete changelog follows below. Since we're in beta phase now, new features should not generally be added for the release any more, but we can make exceptions for currently existing pull requests if you can provide a good motivation. If you think we forgot something, sending a timely reply to this email or reviving an existing mailing list thread or pull request is a good idea. Have fun, Stefan Features added -------------- * PEP 492 (async/await) was implemented. See https://www.python.org/dev/peps/pep-0492/ * PEP 448 (Additional Unpacking Generalizations) was implemented. See https://www.python.org/dev/peps/pep-0448/ * Support for coverage.py 4.0+ can be enabled by adding the plugin "Cython.Coverage" to the ".coveragerc" config file. * Annotated HTML source pages can integrate (XML) coverage reports. * Tracing is supported in ``nogil`` functions/sections and module init code. * When generators are used in a Cython module and the module imports the modules "inspect" and/or "asyncio", Cython enables interoperability by patching these modules during the import to recognise Cython's internal generator and coroutine types. This can be disabled by C compiling the module with "-D CYTHON_PATCH_ASYNCIO=0" or "-D CYTHON_PATCH_INSPECT=0" * When generators or coroutines are used in a Cython module, their types are registered with the ``Generator`` and ``Coroutine`` ABCs in the ``collections`` or ``collections.abc`` stdlib module at import time to enable interoperability with code that needs to detect and process Python generators/coroutines. These ABCs were added in CPython 3.5 and are available for older Python versions through the ``backports_abc`` module on PyPI. See https://bugs.python.org/issue24018 * Adding/subtracting/dividing/modulus and equality comparisons with constant Python floats and small integers are faster. * Binary and/or/xor/rshift operations with small constant Python integers are faster. * When called on generator expressions, the builtins ``all()``, ``any()``, ``dict()``, ``list()``, ``set()``, ``sorted()`` and ``unicode.join()`` avoid the generator iteration overhead by inlining a part of their functionality into the for-loop. * Keyword argument dicts are no longer copied on function entry when they are not being used or only passed through to other function calls (e.g. in wrapper functions). * The ``PyTypeObject`` declaration in ``cpython.object`` was extended. * The builtin ``type`` type is now declared as PyTypeObject in source, allowing for extern functions taking type parameters to have the correct C signatures. Note that this might break code that uses ``type`` just for passing around Python types in typed variables. Removing the type declaration provides a backwards compatible fix. * ``wraparound()`` and ``boundscheck()`` are available as no-ops in pure Python mode. * Const iterators were added to the provided C++ STL declarations. * ``NULL`` is allowed as default argument when embedding signatures. This fixes ticket 843. * When compiling with ``--embed``, the internal module name is changed to ``__main__`` to allow arbitrary program names, including those that would be invalid for modules. Note that this prevents reuse of the generated C code as an importable module. * External C++ classes that overload the assignment operator can be used. Patch by Ian Henriksen. * Support operator bool() for C++ classes so they can be used in if statements. Bugs fixed ---------- * Calling "yield from" from Python on a Cython generator that returned a value triggered a crash in CPython. This is now being worked around. See https://bugs.python.org/issue23996 * Language level 3 did not enable true division (a.k.a. float division) for integer operands. * Functions with fused argument types that included a generic 'object' fallback could end up using that fallback also for other explicitly listed object types. * Relative cimports could accidentally fall back to trying an absolute cimport on failure. * The result of calling a C struct constructor no longer requires an intermediate assignment when coercing to a Python dict. * C++ exception declarations with mapping functions could fail to compile when pre-declared in .pxd files. * ``cpdef void`` methods are now permitted. * ``abs(cint)`` could fail to compile in MSVC and used sub-optimal code in C++. Patch by David Vierra, original patch by Michael En?lin. * Buffer index calculations using index variables with small C integer types could overflow for large buffer sizes. Original patch by David Vierra. * C unions use a saner way to coerce from and to Python dicts. * When compiling a module ``foo.pyx``, the directories in ``sys.path`` are no longer searched when looking for ``foo.pxd``. Patch by Jeroen Demeyer. * Memory leaks in the embedding main function were fixed. Original patch by Michael En?lin. * Some complex Python expressions could fail to compile inside of finally clauses. Other changes ------------- * Changed mangling scheme in header files generated by ``cdef api`` declarations. * Installation under CPython 3.3+ no longer requires a pass of the 2to3 tool. This also makes it possible to run Cython in Python 3.3+ from a source checkout without installing it first. Patch by Petr Viktorin. From insertinterestingnamehere at gmail.com Wed Jul 29 23:56:37 2015 From: insertinterestingnamehere at gmail.com (Ian Henriksen) Date: Wed, 29 Jul 2015 21:56:37 +0000 Subject: [Cython] Non-type template parameters In-Reply-To: References: Message-ID: On Tue, Jul 28, 2015 at 12:37 PM Robert Bradshaw wrote: > On Tue, Jul 28, 2015 at 11:18 AM, Ian Henriksen > wrote: > > > > On Tue, Jul 28, 2015 at 1:36 AM Robert Bradshaw > wrote: > >> > >> Sorry for not getting back sooner... responses below. > >> > >> On Mon, Jul 20, 2015 at 2:06 PM, Ian Henriksen > >> wrote: > >> > Hi all, > >> > I've spent a little time working on adding support for non-type > >> > template parameters. In doing this, it has been very easy to add > >> > support for doing something like the following: > >> > > >> > cdef extern from "add_const.hpp" nogil: > >> > int myfunc1[i](int) > >> > def test(): > >> > print myfunc1[2](a) > >> > > >> > The downside of this is that it does not let the Cython compiler > >> > distinguish between different kinds of template parameters. > >> > > >> > Stricter checking could be added using a syntax like this: > >> > > >> > cdef extern from "add_const.hpp" nogil: > >> > int myfunc1[int i](int) > >> > def test(): > >> > print myfunc1[2](a) > >> > > >> > The downsides here are that the syntax doesn't really match the > >> > existing template syntax. It will also complicate the Cython codebase > >> > since we'll have to go to greater lengths to allow or disallow all the > >> > different special cases for templates. > >> > > >> > Which version would be preferable? > >> > >> I think I'd prefer the [int i] syntax. Hopefully it shouldn't > >> complicate things too much. > > > > Okay, I think I see a way to make that work. On the other hand, since > there > > weren't any replies here, I've already nearly finished implementing the > > first > > syntax. I'll spend another hour or two finishing it off later today and > > submit a PR > > so you can look it over. I originally favored the first syntax because it > > minimizes > > the number of fancy template features (SFINAE, for example) we have to > worry > > about on the Cython end. I'm still open to discuss it though. > > I think this falls into the "explicit is better than implicit" bucket. > That and getting obscure template errors that could have been caught > at Cython compile time will be very nice. > > >> > On a similar note, for variadic templates, would we prefer something > >> > like > >> > > >> > cdef extern from "my_variadic.hpp" nogil: > >> > T myfunc2[T,...](T, ...) > >> > > >> > or something like: > >> > > >> > cdef extern from "my_variadic.hpp" nogil: > >> > T myfunc2[T, Types...](T, Types... args) > >> > > >> > Again, the latter syntax is more explicit, but it will require much > more > >> > complicated code in Cython. It also doesn't match the existing syntax > >> > very well. The former syntax matches the existing syntax for templates > >> > better, but will make it hard for Cython to raise errors early on in > >> > compilation. > >> > >> Hmm... this is a tougher call. Let's go with the former for now. > > > > > > I like the former a lot more. It will keep the syntax simpler on our end > and > > I > > haven't been able to find any case that it doesn't cover. This will also > be > > significantly easier to implement. I'll take a look at it soon. > > Sounds good. > > There's also the question of default arguments. Let's adopt the [type > T = *] syntax for that. > > >> > I'd greatly appreciate any input on the best syntax for either > use-case. > >> > > >> > Regards, > >> > > >> > -Ian Henriksen > >> > > >> > _______________________________________________ > >> > cython-devel mailing list > >> > cython-devel at python.org > >> > https://mail.python.org/mailman/listinfo/cython-devel > >> > > >> _______________________________________________ > >> cython-devel mailing list > >> cython-devel at python.org > >> https://mail.python.org/mailman/listinfo/cython-devel > > > > > > _______________________________________________ > > cython-devel mailing list > > cython-devel at python.org > > https://mail.python.org/mailman/listinfo/cython-devel > > > _______________________________________________ > cython-devel mailing list > cython-devel at python.org > https://mail.python.org/mailman/listinfo/cython-devel Yep, I can see why that syntax makes sense. I'm a little worried I may not be able to finish adding all of this in the near future. Adding all the of this type checking for templates goes a long way toward implementing the full syntax for declaring them. That said, it seems sensible to catch errors early if possible. Thanks! -Ian Henriksen -------------- next part -------------- An HTML attachment was scrubbed... URL: