Mailman 3 January 2014 - Python-Dev

Re: [Python-Dev] [Python-checkins] peps: Fill in PEP number (461).
by Brett Cannon 14 Jan '14

14 Jan '14

I think this was supposed to be 461, not 460 =) On Tue, Jan 14, 2014 at 2:12 PM, guido.van.rossum < python-checkins(a)python.org> wrote: > http://hg.python.org/peps/rev/a25f48998ad3 > changeset: 5346:a25f48998ad3 > user: Guido van Rossum <guido(a)dropbox.com> > date: Tue Jan 14 11:12:09 2014 -0800 > summary: > Fill in PEP number (461). > > files: > pep-0461.txt | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/pep-0461.txt b/pep-0461.txt > --- a/pep-0461.txt > +++ b/pep-0461.txt > @@ -1,4 +1,4 @@ > -PEP: XXX > +PEP: 460 > Title: Adding % and {} formatting to bytes > Version: $Revision$ > Last-Modified: $Date$ > > -- > Repository URL: http://hg.python.org/peps > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > https://mail.python.org/mailman/listinfo/python-checkins > >

1 0

Re: [Python-Dev] cpython (merge 2.7 -> 3.1): complain when nbytes > buflen to fix possible buffer overflow (closes #20246)
by Ned Deily 14 Jan '14

14 Jan '14

In article <3f3JH11tMgz7Lln(a)mail.python.org>, benjamin.peterson <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/715fd3d8ac93 > changeset: 88454:715fd3d8ac93 > branch: 3.1 > parent: 86777:b1ddcb220a7f > parent: 88453:87673659d8f7 > user: Benjamin Peterson <benjamin(a)python.org> > date: Mon Jan 13 23:06:14 2014 -0500 > summary: > complain when nbytes > buflen to fix possible buffer overflow (closes > #20246) Benjamin, I think you may have mistakenly merged from 2.7 to 3.1 here and then left the 3.1 branch open (i.e. unmerged to 3.2). -- Ned Deily, nad(a)acm.org

2 1

Test failures when running as root
by Chris Angelico 14 Jan '14

14 Jan '14

And now for something completely different. My root buildbot is finally now able to telnet out and get "Connection refused" errors. (For the curious, the VirtualBox "NAT" mode doesn't work properly, but the new "NAT Network" mode does. Why? I have no idea. But if anyone else is having the same problem, upgrade to the latest VirtualBox and set up a NAT Network. All I care is, it now works.) The test suite is now failing at another point, and this applies to 2.7, 3.3, and 3.x. ====================================================================== ERROR: test_initgroups (test.test_posix.PosixGroupsTester) ---------------------------------------------------------------------- Traceback (most recent call last): File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_posix.py", line 1143, in test_initgroups g = max(self.saved_groups) + 1 ValueError: max() arg is an empty sequence ---------------------------------------------------------------------- The saved_groups value comes from posix.getgroups(), and it's being used to try to get a group that this user doesn't have (I think). When I run Python as root, posix.getgroups() returns [0], but apparently it's not returning any groups when the test runs. So, two questions. Firstly, is this a problem that needs to be fixed in Python, or is it a configuration change that I made? It began failing recently, so possibly when I rebooted the VM as part of VirtualBox changes I mucked something up. And secondly, how can I run the tests manually? I can't find a binary inside the buildarea tree. Does it get deleted afterward? Apologies if these are dumb questions, hopefully they're a small distraction from PEP 460 arguments! ChrisA

4 7

Re: [Python-Dev] RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5
by Matt Billenstein 14 Jan '14

14 Jan '14

On Wed, Jan 08, 2014 at 07:12:06PM +0100, Stefan Behnel wrote: > Why can't someone write a third-party library that does what these projects > need, and that works in both Py2 and Py3, so that these projects can be > modified to use that library and thus get on with their porting to Py3? Apologies if this is out of place and slightly OT and soap-boxey... Does it not strike anyone here how odd it is that one would need a library to manipulate binary data in a programming language with "batteries included" on a binary computer? And maybe you can do it with existing facilities in both versions of Python, although in python3, I need to understand what bytes, format, ascii, and surrogateescape mean - among other things. I started in Python blissfully unaware of unicode - it was a different time for sure, but what I knew from C worked pretty much the same in Python - I could read some binary data out of a file, twiddle some bits, and write it back out again without any of these complexities - life was good and granted I was naive, but it made Python approachable for me and I enjoyed it. I stuck with it and learned about unicode and the complexities of encoding data and now I'm astonished at how many professional programmers don't know the slightest bit about it and how horribly munged some data you can consume on the web might be - I agree it's all quite a mess. So now I'm getting more serious about Python3 and my fear is that the development community (python3) has fractured from the user community (python2) in that they've built something that solves their problems (to oversimplify lets say a webapp) - sure, a bunch of stuff got fixed along the way and we gave the users division they would expect (3/2 == 1.5), but somewhere what I felt was more like a hobbyist language has become big and complex and "we need to protect our users from doing the wrong thing." And I think everyone was well intentioned - and python3 covers most of the bases, but working with binary data is not only a "wire-protocol programmer's" problem. Needing a library to wrap bytesthing.format('ascii', 'surrogateescape') or some such thing makes python3 less approachable for those who haven't learned that yet - which was almost all of us at some point when we started programming. I appreciate everyone's hard work - I'm confident the community will cross the 2-3 chasm and I hope we preserve the approachability I first came to love about Python when I started using it for all sorts of applications. thx m -- Matt Billenstein matt(a)vazor.com http://www.vazor.com/

35 104

PEP460 thoughts from a Mercurial dev
by Augie Fackler 13 Jan '14

13 Jan '14

(sorry for not piling on any existing threads - I don't subscribe to python-dev due to lack of time) Brett Cannon asked me to chime in - I haven't actually read the very long thread at this point, I'm just providing responses to things Brett mentioned: 1) What do we need in terms of functionality Best guess, %s, %d, and %f. I've not done a full audit of the code, but some limited looking over the grep hits for % in .py files suggests I'm right, and we could even do without %f (we only use that for 'hg --time' output, which we could do in unicode). We also need some way to emit raw bytes (in potentially mixed encodings, yes I know this is "doing it wrong") to stdout/stderr (example: someone changes a file from latin1 to utf8, and then wants to see the resulting diff). 2) Would having it as an external library that worked with Python 2 help? Probably, IF it came with 2.4 support (RHEL support, basically), and we could bundle it in our source tree. It's been extremely valuable to have the install only depend on a working C compiler and Python. 3) If this does go in, how long would it take us to port Mercurial to py3? Would it being in 3.5 hold us up? I'm honestly not sure. I'm still in the outermost layers of this yak shave: fixing cyclic imports. I'll know more when I can at least get 'hg version' to print its own version, because at that point the testsuite failures might be informative. I'd honestly _rather_ this went into 3.5 *and* got lots of validation by both us and twisted (the other folks that care?) before becoming set in stone by a release. Does that make sense? 4) Do we care if it's .format()/%, or could it be in the stdlib? It'd be really nice to not have to boil the oceans as far as editing everyplace in the codebase that does % today. If we do have to do that, it's not going to be much more helpful than something like: def maybestr(a): if isinstance(a, bytes): return a.decode('latin1) return a def sprintf(fmt, *args): (fmt.decode('latin1') % [maybestr(a) for a in args]).encode('latin1) or similar. That was (roughly) what I was figuring I'd do today without any formal bytes-string-formatting support. He also mentioned that some are calling for a shortened 3.5 release cycle - I'd rather not see that happen, for the aforementioned reason of wanting time to make sure this is Right - it'd be a shame to do the work and rush it out only to find something missing in an important way. Feel free to ask further questions - I'll try to respond promptly. AF (For those curious: my hg-on-py3 repo isn't published at the moment because I rebuilt the server it lived on and I forgot to publish it. I'll rectify that sometime this week, I hope, but it's really totally nonfunctional due to cyclic imports.)

5 10

Re: [Python-Dev] cpython (3.3): Update Sphinx toolchain.
by Georg Brandl 13 Jan '14

13 Jan '14

Am 11.01.2014 21:11, schrieb Terry Reedy: > On 1/11/2014 2:04 PM, georg.brandl wrote: >> http://hg.python.org/cpython/rev/87bdee4d633a >> changeset: 88413:87bdee4d633a >> branch: 3.3 >> parent: 88410:05e84d3ecd1e >> user: Georg Brandl <georg(a)python.org> >> date: Sat Jan 11 20:04:19 2014 +0100 >> summary: >> Update Sphinx toolchain. >> >> files: >> Doc/Makefile | 8 ++++---- >> 1 files changed, 4 insertions(+), 4 deletions(-) >> >> >> diff --git a/Doc/Makefile b/Doc/Makefile >> --- a/Doc/Makefile >> +++ b/Doc/Makefile >> @@ -41,19 +41,19 @@ >> checkout: >> @if [ ! -d tools/sphinx ]; then \ >> echo "Checking out Sphinx..."; \ >> - svn checkout $(SVNROOT)/external/Sphinx-1.0.7/sphinx tools/sphinx; \ >> + svn checkout $(SVNROOT)/external/Sphinx-1.2/sphinx tools/sphinx; \ >> fi > > Doc/make.bat needs to be similarly updated. Indeed, thanks for the reminder. Georg

3 6

Python advanced debug support (update frame code)
by Fabio Zadrozny 13 Jan '14

13 Jan '14

Hi Python-dev. I'm playing a bit on the concept on live-coding during a debug session and one of the most annoying things is that although I can reload the code for a function (using something close to xreload), it seems it's not possible to change the code for the current frame (i.e.: I need to get out of the function call and then back in to a call to the method from that frame to see the changes). I gave a look on the frameobject and it seems it would be possible to set frame.f_code to another code object -- and set the line number to the start of the new object, which would cover the most common situation, which would be restarting the current frame -- provided the arguments remain the same (which is close to what the java debugger in Eclipse does when it drops the current frame -- on Python, provided I'm not in a try..except block I can do even better setting the the frame.f_lineno, but without being able to change the frame f_code it loses a lot of its usefulness). So, I'd like to ask for feedback from people with more knowledge on whether it'd be actually feasible to change the frame.f_code and possible implications on doing that. Thanks, Fabio

2 1

Trying to focus the whole bytes/str formatting discussion
by Brett Cannon 13 Jan '14

13 Jan '14

I don't know about the rest of you but I feel like the discussion is heading off the rails (if it hasn't already jumped the tracks). Let's try to bring this back around to something actionable which people can focus their energy on as the amount of developer time spent arguing could have led to several coded-up solutions. I see it as a practicality-beats-purity vs. explicit-is-better-than-implicit. The PBP group want bytes.format() (just assume I include interpolation support if you want that) to work as close to a drop-in replacement for current str.format() use in Python 2 to ease porting. The argument is that code looks cleaner and the amount of changes in Python 2 code being ported to Python 3 is much smaller. THE EIBTI group are willing to support PEP 460 but beyond that don't want to have in Python itself anything for bytes.format() which takes in a string and spits out bytes. It's bytes in->bytes out and not bytes & str in->bytes out as the PBP group is after. The EIBTI group are arguing that letting str into bytes.format() and then automatically be converted to strict ASCII leads to conflating the text/bytes divide as well as being too magical, e.g. what if you actually wanted UTF-16 for you number string instead of ASCII; the EIBTI group **wants** to force people to make a decision. They are also less concerned with making users update Python 2 code to handle this as it already needs to be updated for other Python 3 things anyway. >From where I'm sitting, the EIBTI group and their PEP 460 proposal from Antoine (and no longer Victor) are not controversial. Everyone seems to agree that PEP 460 **at minimum** is acceptable and should happen for Python 3.5. The people with the uphill battle and something to prove are those arguing for str in->bytes out support in bytes.format(). The added features that the PBP group want are the ones being argued over. As the onus is on the PBP group to convince the EIBTI group (or Guido), I think the PBP group should code up a solution that does what they want and put it on PyPI to see what the community thinks. If the PBP group wants to convince the EIBTI group that str in->bytes out for bytes.format() is critical in getting a key group of users to start using Python 3 then I think that needs to be demonstrated through real-world usage by some people. If there is serious pickup of the solution from PyPI by projects then we can discuss integrating it into Python 3.5. That gives at least **one year** to come up with a solution which gets picked up by the community (standard requirement for stdlib inclusion). At worst some projects use the PyPI project and find it useful but it doesn't go into Python 3.5. At best lots of people find it useful enough that we add it to Python 3.5. But regardless, a PyPI project helps people **no matter what** the EIBTI group thinks. That's more forward momentum than this conversation currently has. This has split down philosophical lines and does not look to be tilting one way or the other by simply using words. I think it has reached the point that showing code is going to be the only way to tilt the favour towards the PBP group at this point. Guido has not spoken up so either he is ignoring it because he's busy, he doesn't care, or he's mulling things over still. Assuming he doesn't speak up then it comes down to getting a clear majority on the side of the PBP group and that is not going to happen the way this discussion is going. So, action items are: * Get PEP 460 pronounced on **as is** * A PyPI project containing PBP ideas and see if the community seizes on it or not (benefit to people regardless) * Do a separate PEP that builds on PEP 460 if people really want to continue down that road at this time Don't forget, we are talking about Python 3.5; we have not even hit Python 3.4rc1 yet so this level of arguing seems a bit premature and going nowhere.

4 3

RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5
by Juraj Sukop 13 Jan '14

13 Jan '14

(Sorry if this messes-up the thread order, it is meant as a reply to the original RFC.) Dear list, newbie here. After much hesitation I decided to put forward a use case which bothers me about the current proposal. Disclaimer: I happen to write a library which is directly influenced by this. As you may know, PDF operates over bytes and an integer or floating-point number is written down as-is, for example "100" or "1.23". However, the proposal drops "%d", "%f" and "%x" formats and the suggested workaround for writing down a number is to use ".encode('ascii')", which I think has two problems: One is that it needs to construct one additional object per formatting as opposed to Python 2; it is not uncommon for a PDF file to contain millions of numbers. The second problem is that, in my eyes, it is very counter-intuitive to require the use of str only to get formatting on bytes. Consider the case where a large bytes object is created out of many smaller bytes objects. If I wanted to format a part I had to use str instead. For example: content = b''.join([ b'header', b'some dictionary structure', b'part 1 abc', ('part 2 %.3f' % number).encode('ascii'), b'trailer']) In the case of PDF, the embedding of an image into PDF looks like: 10 0 obj << /Type /XObject /Width 100 /Height 100 /Alternates 15 0 R /Length 2167 >> stream ...binary image data... endstream endobj Because of the image it makes sense to store such structure inside bytes. On the other hand, there may well be another "obj" which contains the coordinates of Bezier paths: 11 0 obj ... stream 0.5 0.1 0.2 RG 300 300 m 300 400 400 400 400 300 c b endstream endobj To summarize, there are cases which mix "binary" and "text" and, in my opinion, dropping the bytes-formatting of numbers makes it more complicated than it was. I would appreciate any explanation on how: b'%.1f %.1f %.1f RG' % (r, g, b) is more confusing than: b'%s %s %s RG' % tuple(map(lambda x: (u'%.1f' % x).encode('ascii'), (r, g, b))) Similar situation exists for HTTP ("Content-Length: 123") and ASCII STL ("vertex 1.0 0.0 0.0"). Thanks and have a nice day, Juraj Sukop PS: In the case the proposal will not include the number formatting, it would be nice to list there a set of guidelines or examples on how to proceed with porting Python 2 formats to Python 3.

16 56

RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5
by Victor Stinner 12 Jan '14

12 Jan '14

Hi, bytes % args and bytes.format(args) are requested by Mercurial and Twisted projects. The issue #3982 was stuck because nobody proposed a complete definition of the "new" features. Here is a try as a PEP. The PEP is a draft with open questions. First, I'm not sure that both bytes%args and bytes.format(args) are needed. The implementation of .format() is more complex, so why not only adding bytes%args? Then, the following points must be decided to define the complete list of supported features (formatters): * Format integer to hexadecimal? ``%x`` and ``%X`` * Format integer to octal? ``%o`` * Format integer to binary? ``{!b}`` * Alignment? * Truncating? Truncate or raise an error? * format keywords? ``b'{arg}'.format(arg=5)`` * ``str % dict`` ? ``b'%(arg)s' % {'arg': 5)`` * Floating point number? * ``%i``, ``%u`` and ``%d`` formats for integer numbers? * Signed number? ``%+i`` and ``%-i`` HTML version of the PEP: http://www.python.org/dev/peps/pep-0460/ Inline copy: PEP: 460 Title: Add bytes % args and bytes.format(args) to Python 3.5 Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 6-Jan-2014 Python-Version: 3.5 Abstract ======== Add ``bytes % args`` operator and ``bytes.format(args)`` method to Python 3.5. Rationale ========= ``bytes % args`` and ``bytes.format(args)`` have been removed in Python 2. This operator and this method are requested by Mercurial and Twisted developers to ease porting their project on Python 3. Python 3 suggests to format text first and then encode to bytes. In some cases, it does not make sense because arguments are bytes strings. Typical usage is a network protocol which is binary, since data are send to and received from sockets. For example, SMTP, SIP, HTTP, IMAP, POP, FTP are ASCII commands interspersed with binary data. Using multiple ``bytes + bytes`` instructions is inefficient because it requires temporary buffers and copies which are slow and waste memory. Python 3.3 optimizes ``str2 += str2`` but not ``bytes2 += bytes1``. ``bytes % args`` and ``bytes.format(args)`` were asked since 2008, even before the first release of Python 3.0 (see issue #3982). ``struct.pack()`` is incomplete. For example, a number cannot be formatted as decimal and it does not support padding bytes string. Mercurial 2.8 still supports Python 2.4. Needed and excluded features ============================ Needed features * Bytes strings: bytes, bytearray and memoryview types * Format integer numbers as decimal * Padding with spaces and null bytes * "%s" should use the buffer protocol, not str() The feature set is minimal to keep the implementation as simple as possible to limit the cost of the implementation. ``str % args`` and ``str.format(args)`` are already complex and difficult to maintain, the code is heavily optimized. Excluded features: * no implicit conversion from Unicode to bytes (ex: encode to ASCII or to Latin1) * Locale support (``{!n}`` format for numbers). Locales are related to text and usually to an encoding. * ``repr()``, ``ascii()``: ``%r``, ``{!r}``, ``%a`` and ``{!a}`` formats. ``repr()`` and ``ascii()`` are used to debug, the output is displayed a terminal or a graphical widget. They are more related to text. * Attribute access: ``{obj.attr}`` * Indexing: ``{dict[key]}`` * Features of struct.pack(). For example, format a number as 32 bit unsigned integer in network endian. The ``struct.pack()`` can be used to prepare arguments, the implementation should be kept simple. * Features of int.to_bytes(). * Features of ctypes. * New format protocol like a new ``__bformat__()`` method. Since the * list of supported types is short, there is no need to add a new protocol. Other types must be explicitly casted. * Alternate format for integer. For example, ``'{|#x}'.format(0x123)`` to get ``0x123``. It is more related to debug, and the prefix can be easily be written in the format string (ex: ``0x%x``). * Relation with format() and the __format__() protocol. bytes.format() and str.format() are unrelated. Unknown: * Format integer to hexadecimal? ``%x`` and ``%X`` * Format integer to octal? ``%o`` * Format integer to binary? ``{!b}`` * Alignment? * Truncating? Truncate or raise an error? * format keywords? ``b'{arg}'.format(arg=5)`` * ``str % dict`` ? ``b'%(arg)s' % {'arg': 5)`` * Floating point number? * ``%i``, ``%u`` and ``%d`` formats for integer numbers? * Signed number? ``%+i`` and ``%-i`` bytes % args ============ Formatters: * ``"%c"``: one byte * ``"%s"``: integer or bytes strings * ``"%20s"`` pads to 20 bytes with spaces (``b' '``) * ``"%020s"`` pads to 20 bytes with zeros (``b'0'``) * ``"%\020s"`` pads to 20 bytes with null bytes (``b'\0'``) bytes.format(args) ================== Formatters: * ``"{!c}"``: one byte * ``"{!s}"``: integer or bytes strings * ``"{!.20s}"`` pads to 20 bytes with spaces (``b' '``) * ``"{!.020s}"`` pads to 20 bytes with zeros (``b'0'``) * ``"{!\020s}"`` pads to 20 bytes with null bytes (``b'\0'``) Examples ======== * ``b'a%sc%s' % (b'b', 4)`` gives ``b'abc4'`` * ``b'a{}c{}'.format(b'b', 4)`` gives ``b'abc4'`` * ``b'%c'`` % 88`` gives ``b'X``' * ``b'%%'`` gives ``b'%'`` Criticisms ========== * The development cost and maintenance cost. * In 3.3 encoding to ascii or latin1 is as fast as memcpy * Developers must work around the lack of bytes%args and bytes.format(args) anyway to support Python 3.0-3.4 * bytes.join() is consistently faster than format to join bytes strings. * Formatting functions can be implemented in a third party module References ========== * `Issue #3982: support .format for bytes <http://bugs.python.org/issue3982>`_ * `Mercurial project <http://mercurial.selenic.com/>`_ * `Twisted project <http://twistedmatrix.com/trac/>`_ * `Documentation of Python 2 formatting (str % args) <http://docs.python.org/2/library/stdtypes.html#string-formatting>`_ * `Documentation of Python 2 formatting (str.format) <http://docs.python.org/2/library/string.html#formatstrings>`_ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

35 115