Cython devs,
In the function __Pyx_PyUnicode_Tailmatch, the return type of
PyUnicode_Tailmatch is assumed to be in int. See line 543 of
Cython/Utility/StringTools.c.
PyUnicode_Tailmatch actually returns a Py_ssize_t. See:
https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Tailmatch.
For reference, there was previously an error in the Python documentation
of this function. The documentation mistakenly said it returned an int.
This was fixed via Python issue 22580:
https://bugs.python.org/issue22580.
Thanks,
Josh Ayers
Hi,
(First time poster. Apologies in advance if I'm not following the
appropriate protocol, or if this has been posted already)
I noticed an issue on Windows when debugging an issue in scipy
<https://github.com/scipy/scipy/issues/4907>, but I think it might be a
little more general. In some places in the generated code, it looks like
intermediate integral variables are declared with type long, even when long
is too small to hold necessary value. For example, with the code pasted
below, the value n+1 is stored in a variable of type long (using Cython
0.22.1) before being supplied to F.__getitem__.
This is especially pertinent on Windows (32 bit and 64 bit) and 32-bit
linux, where longs are 32-bits, so you get an overflow for a program like
the example below. The result is that it prints 1 instead of the expected
value, 2**53+1 = 9007199254740993. But this same issue comes up basically
whenever you do arithmetic on an array index in 64-bit Windows, for indices
larger than 2**31-1, since sizeof(long) << sizeof(void*).
```
from libc.stdint cimport int64_t
class F(object):
def __getitem__(self, i):
print(i)
cdef int64_t n = 2**53
f = F()
f[n+1]
```
Thank you very much Stefan.
I have just read the ticket, and it seems that Guido van Rossum consider
this to be a feature, not a bug (???), so I guess they won't fix it.
For me it would be a nice plus to have this problem fixed, but I can live
with my current workaround, so I'm not sure if Cython should spend dev time
to fix that if that's expensive.
At least, I think the workaround should be mentioned in the github wiki so
that others may implement finite fields extension types without MemoryError
issues. Should I add a wiki page?
2015-06-19 12:00 GMT+02:00 <cython-devel-request(a)python.org>:
> Send cython-devel mailing list submissions to
> cython-devel(a)python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://mail.python.org/mailman/listinfo/cython-devel
> or, via email, send a message with subject or body 'help' to
> cython-devel-request(a)python.org
>
> You can reach the person managing the list at
> cython-devel-owner(a)python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of cython-devel digest..."
>
>
> Today's Topics:
>
> 1. Re: Bug: Extension Type inheriting from int cause a
> MemoryError (Stefan Behnel)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 19 Jun 2015 08:38:32 +0200
> From: Stefan Behnel <stefan_ml(a)behnel.de>
> To: cython-devel(a)python.org
> Subject: Re: [Cython] Bug: Extension Type inheriting from int cause a
> MemoryError
> Message-ID: <5583B8E8.9050902(a)behnel.de>
> Content-Type: text/plain; charset=utf-8
>
> Stephen LARROQUE schrieb am 15.06.2015 um 12:34:
> > I am trying to make an extension type inheriting from int or cython.int
> (to
> > do arithmetic operations in Galois Fields). However, this causes a
> > MemoryError because it seems such extension type is not freed correctly.
> > Other than that, it works perfectly well.
> >
> > cdef class ExtendedInt(int): pass
> >
> > for j in xrange(10000000):
> > ExtendedInt(j)
>
> It looks like a bug in the "int" type in Python 2.x. Python 2.7 has this
> code in intobject.c:
>
> """
> static void
> int_dealloc(PyIntObject *v)
> {
> if (PyInt_CheckExact(v)) {
> Py_TYPE(v) = (struct _typeobject *)free_list;
> free_list = v;
> }
> else
> Py_TYPE(v)->tp_free((PyObject *)v);
> }
>
> static void
> int_free(PyIntObject *v)
> {
> Py_TYPE(v) = (struct _typeobject *)free_list;
> free_list = v;
> }
> """
>
> Your extension type automatically inherits the "int_free" slot function
> from its base type, so the "else" case in "int_dealloc()" will in fact call
> "int_free()" and append the object to the free list despite *not* being
> exactly of type PyInt. Then, when creating a new ExtendedInt instance,
> Python's int-subtype instantiation code *ignores* the free list and instead
> creates a completely new object.
>
> Thus, the free list keeps growing until it fills all memory and the process
> dies. I created a CPython ticket.
>
> https://bugs.python.org/issue24469
>
> I guess we could hack up a work around for this in Cython somehow (Python
> 2.7 is an easy target being a dead end, after all), but let's wait what the
> CPython devs have to say about this. Note, however, that any fix in a
> future CPython 2.7.x release will not fix it in earlier Python 2.x
> versions, so my guess is that we'd end up having to add a work-around on
> our side anyway.
>
> Stefan
>
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> cython-devel mailing list
> cython-devel(a)python.org
> https://mail.python.org/mailman/listinfo/cython-devel
>
>
> ------------------------------
>
> End of cython-devel Digest, Vol 53, Issue 9
> *******************************************
>
