Mailman 3 November 2011 - Python-Dev

file.readinto performance regression in Python 3.2 vs. 2.7?
by Eli Bendersky 25 Nov '11

25 Nov '11

Hi there, I was doing some experiments with the buffer interface of bytearray today, for the purpose of quickly reading a file's contents into a bytearray which I can then modify. I decided to do some benchmarking and ran into surprising results. Here are the functions I was timing: def justread(): # Just read a file's contents into a string/bytes object f = open(FILENAME, 'rb') s = f.read() def readandcopy(): # Read a file's contents and copy them into a bytearray. # An extra copy is done here. f = open(FILENAME, 'rb') b = bytearray(f.read()) def readinto(): # Read a file's contents directly into a bytearray, # hopefully employing its buffer interface f = open(FILENAME, 'rb') b = bytearray(os.path.getsize(FILENAME)) f.readinto(b) FILENAME is the name of a 3.6MB text file. It is read in binary mode, however, for fullest compatibility between 2.x and 3.x Now, running this under Python 2.7.2 I got these results ($1 just reflects the executable name passed to a bash script I wrote to automate these runs): $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.justread()' 1000 loops, best of 3: 461 usec per loop $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.readandcopy()' 100 loops, best of 3: 2.81 msec per loop $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.readinto()' 1000 loops, best of 3: 697 usec per loop Which make sense. The readinto() approach is much faster than copying the read buffer into the bytearray. But with Python 3.2.2 (built from the 3.2 branch today): $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.justread()' 1000 loops, best of 3: 336 usec per loop $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.readandcopy()' 100 loops, best of 3: 2.62 msec per loop $1 -m timeit -s'import fileread_bytearray' 'fileread_bytearray.readinto()' 100 loops, best of 3: 2.69 msec per loop Oops, readinto takes the same time as copying. This is a real shame, because readinto in conjunction with the buffer interface was supposed to avoid the redundant copy. Is there a real performance regression here, is this a well-known issue, or am I just missing something obvious? Eli P.S. The machine is quad-core i7-2820QM, running 64-bit Ubuntu 10.04

7 25

webmaster@python.org address not working
by Jesus Cea 25 Nov '11

25 Nov '11

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 When mailing there, I get this error. Not sure where to report. """ Final-Recipient: rfc822; sdrees(a)sdrees.de Original-Recipient: rfc822;webmaster(a)python.org Action: failed Status: 5.1.1 Remote-MTA: dns; stefan.zinzdrees.de Diagnostic-Code: smtp; 550 5.1.1 <sdrees(a)sdrees.de>: Recipient address rejected: User unknown in local recipient table """ - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTs7fQZlgi5GaxT1NAQLxrQQAmph2w/KrLbwK34IVFKNKAn3P78uY19U1 yoUslB7J4u4IhqQHd5r/FD0v6q6W12t9H8UFNdKELc/zRnRWtE7wKI+3RAeBMAfe pTV6OY7kbGtUfDk9na8o6+oEQ7iZUWT1LbBtMpSusHBuif239RD3HMeaaJ6u/BFT TMmsu39qf2E= =ecRu -----END PGP SIGNATURE-----

3 2

PyUnicode_Resize
by Victor Stinner 23 Nov '11

23 Nov '11

Hi, In Python 3.2, PyUnicode_Resize() expects a number of Py_UNICODE units, whereas Python 3.3 expects a number of characters. It is tricky to convert a number of Py_UNICODE units to a number of characters, so it is diffcult to provide a backward compatibility PyUnicode_Resize() function taking a number of Py_UNICODE units in Python 3.3. Should we rename PyUnicode_Resize() in Python 3.3 to avoid surprising bugs? The issue only concerns Windows with non-BMP characters, so a very rare use case. The easiest solution is to do nothing in Python 3.3: the API changed, but it doesn't really matter. Developers just have to be careful on this particular issue (which is not well documented today). Victor

3 2

Python 3.4 Release Manager
by Larry Hastings 23 Nov '11

23 Nov '11

I've volunteered to be the Release Manager for Python 3.4. The FLUFL has already given it his Sloppy Wet Kiss Of Approval, and we talked to Georg and he was for it too. There's no formal process for selecting the RM, so I may already be stuck with the job, but I thought it best to pipe up on python-dev in case someone had a better idea. But look! I'm already practicing: NO YOU CAN'T CHECK THAT IN. How's that? Needs work? I look forward to seeing how the sausage is made, /larry/

9 8

Re: [Python-Dev] cpython: fix compiler warning by implementing this more cleverly
by Antoine Pitrou 23 Nov '11

23 Nov '11

On Tue, 22 Nov 2011 21:29:43 +0100 benjamin.peterson <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/77ab830930ae > changeset: 73697:77ab830930ae > user: Benjamin Peterson <benjamin(a)python.org> > date: Tue Nov 22 15:29:32 2011 -0500 > summary: > fix compiler warning by implementing this more cleverly You mean "more obscurely"? Obfuscating the original intent in order to disable a compiler warning doesn't seem very wise to me. Regards Antoine. > files: > Objects/unicodeobject.c | 7 +------ > 1 files changed, 1 insertions(+), 6 deletions(-) > > > diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c > --- a/Objects/unicodeobject.c > +++ b/Objects/unicodeobject.c > @@ -6164,12 +6164,7 @@ > kind = PyUnicode_KIND(unicode); > data = PyUnicode_DATA(unicode); > len = PyUnicode_GET_LENGTH(unicode); > - > - switch(kind) { > - case PyUnicode_1BYTE_KIND: expandsize = 4; break; > - case PyUnicode_2BYTE_KIND: expandsize = 6; break; > - case PyUnicode_4BYTE_KIND: expandsize = 10; break; > - } > + expandsize = kind * 2 + 2; > > if (len > PY_SSIZE_T_MAX / expandsize) > return PyErr_NoMemory(); > > -- > Repository URL: http://hg.python.org/cpython

5 7

Re: [Python-Dev] cpython: fix wrong credit and issue id given in previous commit
by Antoine Pitrou 23 Nov '11

23 Nov '11

On Tue, 22 Nov 2011 13:38:03 +0100 giampaolo.rodola <python-checkins(a)python.org> wrote: > diff --git a/Misc/ACKS b/Misc/ACKS > --- a/Misc/ACKS > +++ b/Misc/ACKS > @@ -11,7 +11,7 @@ > PS: In the standard Python distribution, this file is encoded in UTF-8 > and the list is in rough alphabetical order by last names. > > -Matt Mulsow > +Chris Clark > Rajiv Abraham "The list is in rough alphabetical order by last names". Regarfs Antoine.

3 3

Re: [Python-Dev] [Python-checkins] cpython: sort last committed name in alphabetical order
by Nadeem Vawda 23 Nov '11

23 Nov '11

Did you mean to also modify sched.py in this changeset?

2 2

PyUnicode_EncodeDecimal
by Victor Stinner 22 Nov '11

22 Nov '11

Hi, I'm trying to rewrite PyUnicode_EncodeDecimal() to upgrade it to the new Unicode API. The problem is that the function is not accessible in Python nor tested. Should we document and test it, leave it unchanged and deprecate it, or simply remove it? -- Python has a PyUnicode_EncodeDecimal() function. It was used in Python 2 by int, long and complex constructors. In Python 3, the function is no more used: it has been replaced by PyUnicode_TransformDecimalToASCII() in Python <= 3.2 and _PyUnicode_TransformDecimalAndSpaceToASCII() in Python 3.3. PyUnicode_EncodeDecimal() goes into an unlimited loop if there is more than one unencodable character. It's a known bug and there is a patch: http://bugs.python.org/issue13093 PyUnicode_EncodeDecimal() is undocumented and not tested: http://bugs.python.org/issue8646 Stefan Krah uses PyUnicode_EncodeDecimal() in its cdecimal project. See also "Malformed error message from float()" issue: http://bugs.python.org/issue10557 Python 3.3 has now 3 encoders to decimal: - PyUnicode_EncodeDecimal() - PyUnicode_TransformDecimalToASCII() - _PyUnicode_TransformDecimalAndSpaceToASCII() (new in 3.3) _PyUnicode_TransformDecimalAndSpaceToASCII() replaces also Unicode spaces with ASCII spaces. PyUnicode_EncodeDecimal() and PyUnicode_TransformDecimalToASCII() take Py_UNICODE* strings. PyUnicode_EncodeDecimal() requires an output buffer and it has no argument for the size of the output buffer. It is unsafe: it leads to buffer overflow if the buffer is too small. Victor

2 3

Committing PEP 3155
by Antoine Pitrou 21 Nov '11

21 Nov '11

Hello, I haven't seen any strong objections, so I would like to go ahead and commit PEP 3155 (*) soon. Is anyone against it? (*) "Qualified name for classes and functions" http://www.python.org/dev/peps/pep-3155/ Thank you Antoine.

6 8

Python 3, new-style classes and __class__
by Vinay Sajip 21 Nov '11

21 Nov '11

I was looking through the errors which occur when running the test suite of Django's py3k branch under Python 3, and one particular set of errors caught my eye which is unrelated to the bytes/str dance. These errors occur in some Django utility code, which supplies a SimpleLazyObject (new-style) class [1]. This implements a proxy, which is initialised using a callable. The callable returns the object to be wrapped, and it's called when needed to set up the wrapped instance. The SimpleLazyObject needs to pretend to be the class of the wrapped object, e.g. for equality tests. This pretending is done by declaring __class__ as a property in SimpleLazyObject which fetches and returns the __class__ attribute of the wrapped object. This approach doesn't work in Python 3, however: the property named __class__ doesn't show up in the class dict of SimpleLazyObject, and moreover, there are restrictions on what you can set __class__ to - e.g. Python complains if you try and set a __class__ attribute on the instance to anything other than a new-style class. What's the simplest way in Python 3 of implementing the equivalent approach to pretending to be a different class? Any pointers appreciated. Thanks and regards, Vinay Sajip [1] http://goo.gl/1Jlbj

3 9