Mailman 3 October 2011 - Python-Dev

Re: [Python-Dev] cpython: PyUnicode_FromKindAndData() raises a ValueError if the kind is unknown
by Georg Brandl 06 Oct '11

06 Oct '11

On 10/02/11 01:14, victor.stinner wrote: > http://hg.python.org/cpython/rev/9124a00df142 > changeset: 72573:9124a00df142 > parent: 72571:fa0b1e50270f > user: Victor Stinner <victor.stinner(a)haypocalc.com> > date: Sat Oct 01 23:48:37 2011 +0200 > summary: > PyUnicode_FromKindAndData() raises a ValueError if the kind is unknown > > files: > Objects/unicodeobject.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c > --- a/Objects/unicodeobject.c > +++ b/Objects/unicodeobject.c > @@ -1211,7 +1211,7 @@ > case PyUnicode_4BYTE_KIND: > return _PyUnicode_FromUCS4(buffer, size); > } > - assert(0); > + PyErr_SetString(PyExc_ValueError, "invalid kind"); > return NULL; > } Is that really a ValueError? It should only be a ValueError if the user could trigger that error. Otherwise it should be a SystemError. Georg

7 11

Re: [Python-Dev] [Python-checkins] cpython: Fix find_module_path(): make the string ready
by "Martin v. Löwis" 06 Oct '11

06 Oct '11

> + if (PyUnicode_READY(path_unicode)) > + return -1; > + I think we need to discuss/reconsider the return value of PyUnicode_READY. It's defined to give -1 on error currently. If that sounds good, then the check for error should be a check that it is -1. Regards, Martin

1 0

Re: [Python-Dev] [Python-checkins] cpython: When expandtabs() would be a no-op, don't create a duplicate string
by Éric Araujo 06 Oct '11

06 Oct '11

Hi, > http://hg.python.org/cpython/rev/447f521ac6d9 > user: Antoine Pitrou <solipsis(a)pitrou.net> > date: Tue Oct 04 16:04:01 2011 +0200 > summary: > When expandtabs() would be a no-op, don't create a duplicate string > > diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py > --- a/Lib/test/test_unicode.py > +++ b/Lib/test/test_unicode.py > @@ -1585,6 +1585,10 @@ > return > self.assertRaises(OverflowError, 't\tt\t'.expandtabs, sys.maxsize) > > + def test_expandtabs_optimization(self): > + s = 'abc' > + self.assertIs(s.expandtabs(), s) Shouldn’t that be marked CPython-specific?

1 0

Using PEP384 Stable ABI for the lzma extension module
by Amaury Forgeot d'Arc 06 Oct '11

06 Oct '11

Hi, Has someone already tried to *really* use Py_LIMITED_API for some "serious" extension module? I wanted to give it a try for the _lzma module (see issue 6715) because liblzma does not compile with Microsoft compilers; an alternative could be to use mingw to (pre)build _lzma.pyd, which would link with a static liblzma.a also compiled with mingw. Mixing compilers in a Python process is one of the reasons of PEP384, so I added #define Py_LIMITED_API on top of the module, and "fixed" the issues one by one: - Py_LIMITED_API is incompatible with --with-pydebug, and compilation stops. I skipped the check to continue. - I replaced PyBytes_GET_SIZE() with Py_SIZE(), which is OK, and PyBytes_AS_STRING() with PyBytes_AsString(), which may have a slight performance impact. - I replaced Py_TYPE(self)->tp_free((PyObject *)self); with PyObject_Del(self), I hope this is the same thing (for a non-GC object) - _PyBytes_Resize() is missing; I moved it under a Py_LIMITED_API section. - For the "y*" argument spec, the Py_buffer structure is required (only for two fields: buf and len), as well as PyBuffer_Release() - PyType_FromSpec() does not call PyType_Ready(), which caused crashes in __new__. Now the module seems to work correctly and passes tests... at least on Linux in a standard environment. I will do other tests on Windows. What do you think about using the stable ABI even in shipped extensions? Have you already used it somewhere else? Cheers, -- Amaury Forgeot d'Arc

6 15

Re: [Python-Dev] [Python-checkins] cpython: Document requierements of Unicode kinds
by Terry Reedy 06 Oct '11

06 Oct '11

On 10/5/2011 1:43 PM, victor.stinner wrote: > http://hg.python.org/cpython/rev/055174308822 > changeset: 72699:055174308822 > user: Victor Stinner<victor.stinner(a)haypocalc.com> > date: Wed Oct 05 01:31:05 2011 +0200 > summary: > Document requierements of Unicode kinds > > files: > Include/unicodeobject.h | 24 ++++++++++++++++++++---- > 1 files changed, 20 insertions(+), 4 deletions(-) > > > diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h > --- a/Include/unicodeobject.h > +++ b/Include/unicodeobject.h > @@ -288,10 +288,26 @@ > unsigned int interned:2; > /* Character size: > > - PyUnicode_WCHAR_KIND (0): wchar_t* > - PyUnicode_1BYTE_KIND (1): Py_UCS1* > - PyUnicode_2BYTE_KIND (2): Py_UCS2* > - PyUnicode_4BYTE_KIND (3): Py_UCS4* > + - PyUnicode_WCHAR_KIND (0): > + > + * character type = wchar_t (16 or 32 bits, depending on the > + platform) > + > + - PyUnicode_1BYTE_KIND (1): > + > + * character type = Py_UCS1 (8 bits, unsigned) > + * if ascii is 1, at least one character must be in range > + U+80-U+FF, otherwise all characters must be in range U+00-U+7F Given that 1==True, this looks backwards. > + > + - PyUnicode_2BYTE_KIND (2): > + > + * character type = Py_UCS2 (16 bits, unsigned) > + * at least one character must be in range U+0100-U+1FFFF /U+1FFFF/U+FFFF/ ? Terry

2 2

Heads up: Apple llvm gcc 4.2 miscompiles PEP 393
by martin＠v.loewis.de 05 Oct '11

05 Oct '11

The gcc that Apple ships with the Lion SDK (not sure what Xcode version that is) miscompiles Python now. I've reported this to Apple as bug 10143715; not sure whether there is a public link to this bug report. In essence, the code typedef struct { long length; long hash; int state; int *wstr; } PyASCIIObject; typedef struct { PyASCIIObject _base; long utf8_length; char *utf8; long wstr_length; } PyCompactUnicodeObject; void *_PyUnicode_compact_data(void *unicode) { return ((((PyASCIIObject*)unicode)->state & 0x20) ? ((void*)((PyASCIIObject*)(unicode) + 1)) : ((void*)((PyCompactUnicodeObject*)(unicode) + 1))); } miscompiles (with -O2 -fomit-frame-pointer) to __PyUnicode_compact_data: Leh_func_begin1: leaq 32(%rdi), %rax ret The compiler version is gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00) This unconditionally assumes that sizeof(PyASCIIObject) needs to be added to unicode, independent of whether the state bit is set or not. I'm not aware of a work-around in the code. My work-around is to use gcc-4.0, which is still available on my system from an earlier Xcode installation (in /Developer-3.2.6) Regards, Martin

5 7

Re: [Python-Dev] [Python-checkins] cpython: Optimize string slicing to use the new API
by "Martin v. Löwis" 05 Oct '11

05 Oct '11

Am 04.10.11 19:50, schrieb Antoine Pitrou: > On Tue, 04 Oct 2011 19:49:09 +0200 > "Martin v. Löwis"<martin(a)v.loewis.de> wrote: > >>> + result = PyUnicode_New(slicelength, PyUnicode_MAX_CHAR_VALUE(self)); >> >> This is incorrect: the maxchar of the slice might be smaller than the >> maxchar of the input string. > > I thought that heuristic would be good enough. I'll try to fix it. No - strings must always be in the canonical form. For example, PyUnicode_RichCompare considers string unequal if they have different kinds. As a consequence, your slice result may not compare equal to a canonical variant of itself.

2 2

Re: [Python-Dev] [Python-checkins] cpython: Migrate str.expandtabs to the new API
by "Martin v. Löwis" 04 Oct '11

04 Oct '11

> Migrate str.expandtabs to the new API This needs if (PyUnicode_READY(self) == -1) return NULL; right after the ParseTuple call. In most cases, the check will be a noop. But if it's not, omitting it will make expandtabs have no effect, since the string length will be 0 (in a debug build, you also get an assertion failure). Regards, Martin

2 1

Re: [Python-Dev] [Python-checkins] cpython: fix compiler warnings
by Victor Stinner 04 Oct '11

04 Oct '11

Le 04/10/2011 01:34, benjamin.peterson a écrit : > http://hg.python.org/cpython/rev/afb60b190f1c > changeset: 72633:afb60b190f1c > user: Benjamin Peterson<benjamin(a)python.org> > date: Mon Oct 03 19:34:12 2011 -0400 > summary: > fix compiler warnings > > +++ b/Objects/unicodeobject.c > @@ -369,6 +369,12 @@ > } > return 1; > } > +#else > +static int > +_PyUnicode_CheckConsistency(void *op) > +{ > + return 1; > +} > #endif Oh no, please don't do that. Calling _PyUnicode_CheckConsistency() is reserved to debug builds. In release mode, we should not check string consistency (it would slow down Python). Yes, there was a warning: Objects/unicodeobject.c:539:13: warning: statement with no effect _PyUnicode_CHECK(unicode); I added these checks recently to ensure that strings are consistent just before exiting (to help me to track down a bug). The right fix is just to replace _PyUnicode_CHECK(unicode) by assert(_PyUnicode_CHECK(unicode)). Victor

3 3

Re: [Python-Dev] cpython: PyUnicode_Join() checks output length in debug mode
by Georg Brandl 04 Oct '11

04 Oct '11

On 10/03/11 23:35, victor.stinner wrote: > http://hg.python.org/cpython/rev/bfd8b5d35f9c > changeset: 72623:bfd8b5d35f9c > user: Victor Stinner <victor.stinner(a)haypocalc.com> > date: Mon Oct 03 23:36:02 2011 +0200 > summary: > PyUnicode_Join() checks output length in debug mode > > PyUnicode_CopyCharacters() may copies less character than requested size, if > the input string is smaller than the argument. (This is very unlikely, but who > knows!?) > > Avoid also calling PyUnicode_CopyCharacters() if the string is empty. > > files: > Objects/unicodeobject.c | 34 +++++++++++++++++++--------- > 1 files changed, 23 insertions(+), 11 deletions(-) > > > diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c > --- a/Objects/unicodeobject.c > +++ b/Objects/unicodeobject.c > @@ -8890,20 +8890,32 @@ > > /* Catenate everything. */ > for (i = 0, res_offset = 0; i < seqlen; ++i) { > - Py_ssize_t itemlen; > + Py_ssize_t itemlen, copied; > item = items[i]; > + /* Copy item, and maybe the separator. */ > + if (i && seplen != 0) { > + copied = PyUnicode_CopyCharacters(res, res_offset, > + sep, 0, seplen); > + if (copied < 0) > + goto onError; > +#ifdef Py_DEBUG > + res_offset += copied; > +#else > + res_offset += seplen; > +#endif > + } > itemlen = PyUnicode_GET_LENGTH(item); > - /* Copy item, and maybe the separator. */ > - if (i) { > - if (PyUnicode_CopyCharacters(res, res_offset, > - sep, 0, seplen) < 0) > + if (itemlen != 0) { > + copied = PyUnicode_CopyCharacters(res, res_offset, > + item, 0, itemlen); > + if (copied < 0) > goto onError; > - res_offset += seplen; > - } > - if (PyUnicode_CopyCharacters(res, res_offset, > - item, 0, itemlen) < 0) > - goto onError; > - res_offset += itemlen; > +#ifdef Py_DEBUG > + res_offset += copied; > +#else > + res_offset += itemlen; > +#endif > + } > } > assert(res_offset == PyUnicode_GET_LENGTH(res)); I don't understand this change. Why would you not always add "copied" once you already have it? It seems to be the more correct version anyway. Georg

2 1