Mailman 3 September 2010 - Python-Dev

Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices)
by Nick Coghlan 23 Sep '10

23 Sep '10

On Sun, Sep 19, 2010 at 4:18 AM, John Nagle <nagle(a)animats.com> wrote: > On 9/18/2010 2:29 AM, python-dev-request(a)python.org wrote: >> >> Polymorphic best practices [was: (Not) delaying the 3.2 release] > > If you're hung up on this, try writing the user-level documentation > first. Your target audience is a working-level Web programmer, not > someone who knows six programming languages and has a CS degree. > If the explanation is too complex, so is the design. > > Coding in this area is quite hard to do right. There are > issues with character set, HTML encoding, URL encoding, and > internationalized domain names. It's often done wrong; > I recently found a Google service which botched it. > Python libraries should strive to deliver textual data to the programmer > in clean Unicode. If someone needs the underlying wire representation > it should be available, but not the default. Even though URL byte sequences are defined as using only an ASCII subset, I'm currently inclined to add raw bytes supports to urlib.parse by providing parallel APIs (i.e. urlib.parse.urlsplitb, etc) rather than doing it implicitly in the normal functions. My rationale is as follows: - while URLs are *meant* to be encoded correctly as an ASCII subset, the real world isn't always quite so tidy (i.e. applications treat as URLs things that technically are not because the encoding is wrong) - separating the APIs forces the programmer to declare that they know they're working with the raw bytes off the wire to avoid the decode/encode overhead that comes with working in the Unicode domain - easier to change our minds later. Adding implicit bytes support to the normal names can be done any time, but removing it would require an extensive deprecation period Essentially, while I can see strong use cases for wanting to manipulate URLs in wire format, I *don't* see strong use cases for manipulating URLs without *knowing* whether they're in wire format (encoded bytes) or display format (Unicode text). For some APIs that work for arbitrary encodings (e.g. os.listdir) switching based on argument type seems like a reasonable idea. For those that may silently produce incorrect output for ASCII-incompatible encodings, the os.environ/os.environb seems like a better approach. I could probably be persuaded to merge the APIs, but the email6 precedent suggests to me that separating the APIs better reflects the mental model we're trying to encourage in programmers manipulating text (i.e. the difference between the raw octet sequence and the text character sequence/parsed data). Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

8 16

Advice sought on memory allocation latency reduction C1X standard proposal
by Niall Douglas 22 Sep '10

22 Sep '10

Dear Python Devs, I am hoping to gain feedback on an ISO C1X/C++ standard library proposal I hope to submit. It consists of a rationale (http://mallocv2.wordpress.com/) which shows how growth in RAM capacity is exponentially outgrowing the growth in RAM access speed. The consequences are profound: computer software which has always been written under the assumption of scarcity of RAM capacity will need to be retargeted to assume the scarcity of RAM access speed instead. The C1X proposal (http://mallocv2.wordpress.com/the-c-proposal-text/) enables four things of interest to Python: (i) aligned block resizing (ii) speculative in-place block resizing (iii) batch block allocation and (iv) the ability to reserve address space, thus avoiding the need to overallocate array storage. Aligned block resizing is especially useful to numpy. Where one has an array of aligned SSE vector quantities one cannot currently resize that block and guarantee that alignment will not be destroyed. With the new feature of non-relocating realloc() and being able to specify an alignment to realloc() one may avoid memory copying, and therefore reduce memory bandwidth utilisation and therefore overall memory access latencies. The ability to reserve address space and speculative in-place block resizing can be combined to allow Python to reserve an arbitrary amount of address space after the storage for an array object. Should the array then become extended, the speculative in-place block resizing can attempt to expand storage into that reserved space without having to relocate the contents of the storage. This again translates into much reduced memory copying as well as memory consumption, and once again reduces overall memory access latencies. Lastly, the batch allocation mechanism allows a sequence of allocations to be performed at once. I don't know of any attempts to have Python make use of similar functionality in Linux's system allocator, however Perl saw a 18% reduction in startup time (http://groups.google.com/group/perl-compiler/msg/31bca5297764002b). I am not familiar with Python's implementation outside working extensively with Boost.Python, so I was hoping that this list could advise me on what I might be forgetting, what problems there could be for Python with this design and/or any other general concerns and thoughts. I thank the list in advance for your time and consideration. Niall Douglas -- Technology & Consulting Services - ned Productions Limited. http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Company no: 472909.

1 0

Re: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices)
by Nick Coghlan 22 Sep '10

22 Sep '10

On Tue, Sep 21, 2010 at 3:03 PM, Stephen J. Turnbull <stephen(a)xemacs.org> wrote: > On the other hand, it is dangerous to provide a polymorphic API which > does that more extensive parsing, because a less than paranoid > programmer will have very likely allowed the parsed components to > escape from the context where their encodings can be reliably > determined. Remember, *it is unlikely that they will ever be punished > for their own lack of caution.* The person who is doomed is somebody > who tries to take that code and reuse it in a different context. Yeah, that's the original reasoning that had me leaning towards the parallel API approach. If I seem to be changing my mind a lot in this thread it's because I'm genuinely torn between the desire to make it easier to port existing 2.x code to 3.x by making the current API polymorphic and the fear that doing so will reintroduce some of the exact same bytes/text confusion that the bytes/str split is trying to get rid of. There's no real way for 2to3 to help with the porting issue either, since it has no way to determine the original intent of the 2.x code. I *think* avoiding the quote/unquote precedent and applying the rule "bytes in -> bytes out" will help with avoiding the worst of any potential encoding confusion problems though. At some point the programmer is going to have to invoke decode() if they want a string to pass to display functions and the like (or vice versa with encode()) so there are still limits to how far any poorly handled code will get before blowing up. (Basically, while the issue of programmers assuming 'latin-1' or 'utf-8' or similar ASCII friendly encodings when they shouldn't is real, I don't believe a polymorphic API here will make things any *worse* than what would happen with a parallel API) And if this turns out to be a disaster in practice: a) on my head be it; and b) we still have the option of the DeprecationWarning dance for bytes inputs to the existing functions and moving to a parallel API Still-trying-to-figure-out-what-moment-of-insanity-prompted-me-to-volunteer-to-tackle-this'ly, Nick.

7 9

Backup plan: WSGI 1 Addenda and wsgiref update for Py3
by P.J. Eby 21 Sep '10

21 Sep '10

While the Web-SIG is trying to hash out PEP 444, I thought it would be a good idea to have a backup plan that would allow the Python 3 stdlib to move forward, without needing a major new spec to settle out implementation questions. After all, even if PEP 333 is ultimately replaced by PEP 444, it's probably a good idea to have *some* sort of WSGI 1-ish thing available on Python 3, with bytes/unicode and other matters settled. In the past, I was waiting for some consensuses (consensi?) on Web-SIG about different approaches to Python 3, looking for some sort of definite, "yes, we all like this" response. However, I can see now that this just means it's my fault we don't have a spec yet. :-( So, unless any last-minute showstopper rebuttals show up this week, I've decided to go ahead officially bless nearly all of what Graham Dumpleton (who's not only the mod_wsgi author, but has put huge amounts of work into shepherding WSGI-on-Python3 proposals, WSGI amendments, etc.) has proposed, with a few minor exceptions. In other words: almost none of the following is my own original work; it's like 90% Graham's. Any praise for this belongs to him; the only thing that belongs to me is the blame for not doing this sooner! (Sorry Graham. You asked me to do this ages ago, and you were right.) Anyway, I'm posting this for comment to both Python-Dev and the Web-SIG. If you are commenting on the technical details of the amendments, please reply to the Web-SIG only. If you are commenting on the development agenda for wsgiref or other Python 3 library issues, please reply to Python-Dev only. That way, neither list will see off-topic discussions. Thanks! The Plan ======== I plan to update the proposal below per comments and feedback during this week, then update PEP 333 itself over the weekend or early next week, followed by a code review of Python 3's wsgiref, and implementation of needed changes (such as recoding os.environ to latin1-captured bytes in the CGI handler). To complete the changes, it is possible that I may need assistance from one or more developers who have more Python 3 experience. If after reading the proposed changes to the spec, you would like to volunteer to help with updating wsgiref to match, please let me know! The Proposal ============ Overview -------- 1. The primary purpose of this update is to provide a uniform porting pattern for moving Python 2 WSGI code to Python 3, meaning a pattern of changes that can be mechanically applied to as little code as practical, while still keeping the WSGI spec easy to programmatically validate (e.g. via ``wsgiref.validate``). The Python 3 specific changes are to use: * ``bytes`` for I/O streams in both directions * ``str`` for environ keys and values * ``bytes`` for arguments to start_response() and write() * text stream for wsgi.errors In other words, "strings in, bytes out" for headers, bytes for bodies. In general, only changes that don't break Python 2 WSGI implementations are allowed. The changes should also not break mod_wsgi on Python 3, but may make some Python 3 wsgi applications non-compliant, despite continuing to function on mod_wsgi. This is because mod_wsgi allows applications to output string headers and bodies, but I am ruling that option out because it forces every piece of middleware to have to be tested with arbitrary combinations of strings and bytes in order to test compliance. If you want your application to output strings rather than bytes, you can always use a decorator to do that. (And a sample one could be provided in wsgiref.) 2. The secondary purpose of the update is to address some long-standing open issues documented here: http://www.wsgi.org/wsgi/Amendments_1.0 As with the Python 3 changes, only changes that don't retroactively invalidate existing implementations are allowed. 3. There is no tertiary purpose. ;-) (By which I mean, all other kinds of changes are out-of-scope for this update.) 4. The section below labeled "A Note On String Types" is proposed for verbatim addition to the "Specification Overview" section in the PEP; the other sections below describe changes to be made inline at the appropriate part of the spec, and changes that were proposed but are rejected for inclusion in this amendment. A Note On String Types ---------------------- In general, HTTP deals with bytes, which means that this specification is mostly about handling bytes. However, the content of those bytes often has some kind of textual interpretation, and in Python, strings are the most convenient way to handle text. But in many Python versions and implementations, strings are Unicode, rather than bytes. This requires a careful balance between a usable API and correct translations between bytes and text in the context of HTTP... especially to support porting code between Python implementations with different ``str`` types. WSGI therefore defines two kinds of "string": * "Native" strings (which are always implemented using the type named ``str``) * "Bytestrings" (which are implemented using the ``bytes`` type in Python 3, and ``str`` elsewhere) So, even though HTTP is in some sense "really just bytes", there are many API conveniences to be had by using whatever Python's default ``str`` type is. Do not be confused however: even if Python's ``str`` is actually Unicode under the hood, the *content* of a native string is still restricted to bytes! See the section on `Unicode Issues`_ later in this document. In short: where you see the word "string" in this document, it refers to a "native" string, i.e., an object of type ``str``, whether it is internally implemented as bytes or unicode. Where you see references to "bytestring", this should be read as "an object of type ``bytes`` under Python 3, or type ``str`` under Python 2". Clarifications (To be made in-line) ----------------------------------- The following amendments are clarifications to parts of the existing spec that proved over the years to be ambiguous or insufficiently specified, as well as some attempts to correct practical errors. (Note: many of these issues cannot be completely fixed in WSGI 1 without breaking existing implementations, and so the text below has notations such as "(MUST in WSGI 2)" to indicate where any replacement spec for WSGI 1 should strengthen them.) * If an application returns a body iterator, a server (or middleware) MAY stop iterating over it and discard the remainder of the output, as long as it calls any close() method provided by the iterator. Applications returning a generator or other custom iterator SHOULD NOT assume that the entire iterator will be consumed. (This change makes it explicit that caching middleware or HEAD-processing servers can throw away the response body.) * start_response() SHOULD (MUST in WSGI 2) check for errors in the status or headers at the time it's called, so that an error can be raised as close to the problem as possible * If start_response() raises an error when called normally (i.e. without exc_info), it SHOULD be an error to call it a second time without passing exc_info * The SERVER_PORT variable is of type str, just like any other CGI environ variable. (According to the WSGI wiki, "some implementations" expect it to be an integer, even though there is nothing in the WSGI spec that allows a CGI variable to be anything but a str.) * A server SHOULD (MUST in WSGI 2) support the size hint argument to readline() on its wsgi.input stream. * A server SHOULD (MUST in WSGI 2) return an empty bytestring from read() on wsgi.input to indicate an end-of-file condition. (In WSGI 2, language should be clarified to allow the input stream length and CONTENT_LENGTH to be out of sync, for reasons explained in Graham's blog post.) * A server SHOULD (MUST in WSGI 2) allow read() to be called without an argument, and return the entire remaining contents of the stream * If an application provides a Content-Length header, the server SHOULD NOT (MUST NOT in WSGI 2) send more data to the client than was specified in that header, whether via write(), yielded body bytestrings, or via a wsgi.file_wrapper. (This rule applies to middleware as well.) * wsgi.errors is a text stream accepting "native strings" Rejected Amendments ------------------- * Manlio Perillo's suggestion to allow header specification to be delayed until the response iterator is producing non-empty output. This would've been a possible win for async WSGI, but could require substantial changes to existing servers.

5 6

Re: [Python-Dev] [Web-SIG] Backup plan: WSGI 1 Addenda and wsgiref update for Py3
by P.J. Eby 21 Sep '10

21 Sep '10

At 12:55 PM 9/21/2010 -0400, Ian Bicking wrote: >On Tue, Sep 21, 2010 at 12:47 PM, Chris McDonough ><<mailto:chrism@plope.com>chrism(a)plope.com> wrote: >On Tue, 2010-09-21 at 12:09 -0400, P.J. Eby wrote: > > While the Web-SIG is trying to hash out PEP 444, I thought it would > > be a good idea to have a backup plan that would allow the Python 3 > > stdlib to move forward, without needing a major new spec to settle > > out implementation questions. > >If a WSGI-1-compatible protocol seems more sensible to folks, I'm >personally happy to defer discussion on PEP 444 or any other >backwards-incompatible proposal. > > >I think both make sense, making WSGI 1 sensible for Python 3 (as >well as other small errata like the size hint) doesn't detract from >PEP 444 at all, IMHO. Yep. I agree. I do, however, want to get these amendments settled and make sure they get carried over to whatever spec is the successor to PEP 333. I've had a lot of trouble following exactly what was changed in 444, and I'm a tad worried that several new ambiguities may be being introduced. So, solidifying 333 a bit might be helpful if it gives a good baseline against which to diff 444 (or whatever).

1 0

Re: [Python-Dev] [Python-checkins] r84858 - in python/branches: py3k/Doc/library/logging.rst release27-maint/Doc/library/logging.rst
by Éric Araujo 21 Sep '10

21 Sep '10

Hello + NOTE: If you are thinking of defining your own levels, please see the section + on :ref:`custom-levels`. I think those instances of upper-case-as-markup should either be real reST note/warning/etc. directives or plain English (that is, integrating “NOTE:” into the text flow, for example “Note that...”/“Pay attention to...”). Regards

6 5

Re: [Python-Dev] r84931 - in python/branches/py3k: Include/symtable.h Misc/NEWS Python/ast.c Python/compile.c Python/future.c Python/symtable.c
by Georg Brandl 21 Sep '10

21 Sep '10

Am 21.09.2010 01:02, schrieb benjamin.peterson: > Author: benjamin.peterson > Date: Tue Sep 21 01:02:10 2010 > New Revision: 84931 > > Log: > add column offset to all syntax errors > Modified: python/branches/py3k/Misc/NEWS > ============================================================================== > --- python/branches/py3k/Misc/NEWS (original) > +++ python/branches/py3k/Misc/NEWS Tue Sep 21 01:02:10 2010 > @@ -10,9 +10,8 @@ > Core and Builtins > ----------------- > > -- Issue #9901: Destroying the GIL in Py_Finalize() can fail if some other > - threads are still running. Instead, reinitialize the GIL on a second > - call to Py_Initialize(). > +- All SyntaxErrors now have a column offset and therefore a caret when the error > + is printed. > > - Issue #9252: PyImport_Import no longer uses a fromlist hack to return the > module that was imported, but instead gets the module from sys.modules. > @@ -59,10 +58,6 @@ > Library > ------- > > -- Issue #9877: Expose sysconfig.get_makefile_filename() > - > -- logging: Added hasHandlers() method to Logger and LoggerAdapter. > - > - Issue #1686: Fix string.Template when overriding the pattern attribute. > > - Issue #9854: SocketIO objects now observe the RawIOBase interface in That change looks like and accident. BTW, PyErr_SyntaxLocationEx needs a versionadded. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out.

1 0

Re: [Python-Dev] [Python-checkins] r84906 - python/branches/py3k/Doc/library/ssl.rst
by Éric Araujo 21 Sep '10

21 Sep '10

> Log: > Remove references to read() and write() methods, which are useless > synonyms of recv() and send() Unless I’m mistaken, ssl.SSLSocket.write is still useful for use with print, pprint and maybe other functions, so I think the method should be listed somewhere. Regards

2 1

Re: [Python-Dev] [Python-checkins] r84871 + committing distutils fixes
by Éric Araujo 21 Sep '10

21 Sep '10

>+ @unittest.skipIf(sys.platform.startswith('win'), >+ "This test is only appropriate for POSIX-like systems.") Is win32 the only non-POSIX supported platform now that OS 9 support is gone? I find those values in the docs for sys.platform: win32, cygwin, darwin, os2, os2emx. Is OS/2 a POSIX system? Isn’t it more robust not to ask this question and just use unittest.skipUnless(os.name == 'posix', msg)? On a related note, I’m very grateful that other developers take time to reply to bugs and commit simple fixes that don’t require knowledge of distutils dark corners. Recent changes have tended to turn buildbots red though, so I’d like to respectfully ask/remind you to make sure to run tests before committing in each branch (3.2, 3.1, 2.7), if possible on POSIX and Windows. (Those are Tarek’s rules, not mine; I hope don’t sound bossy or ungrateful for your help.) I apologize in advance if you did run the test and the buildbots just revealed an unforeseen platform-specific quirk; I’m not accusing anyone of anything—I have my share of embarrassing commits, so be sure I’m not judging here, I just want to improve the process. :) It would also be very helpful to always add the Distutils2 component so that synchronizing the fixes is not overlooked. (Maybe I should add a tracker admin to automate that.) mport-unittest-ly yours

1 0

Fwd: Builder: x86 Windows7 3.x OpenSSL compile error
by Hirokazu Yamamoto 21 Sep '10

21 Sep '10

Hello. I've sent following mail to buildbot manager, but I found that buildbot page saids the problem of unsable bot should be sent to python-dev(a)python.org. So I'll do it. -------- Original Message -------- Subject: Builder: x86 Windows7 3.x OpenSSL compile error Date: Mon, 20 Sep 2010 22:53:04 +0900 From: Hirokazu Yamamoto <ocean-city(a)m2.ccsnet.ne.jp> To: db3l.net(a)gmail.com Hello, David Bolen. I'm Hirokazu Yamamoto, Python committer. I've noticed your buildbot x86 Windows7 3.x fails to compile OpenSSL, I'm very sorry because this buildbot is very fast and looks useful. (Running on Windows7 which I don't have) Builder: x86 Windows7 3.x http://www.python.org/dev/buildbot/all/builders/x86%20Windows7%203.x I confirmed this happens when nasmw's version is 2.06. I'm using 2.07 on my machine, so I didn't notice this. I checked Windows buildbot's nasmw versions with Tools/buildbot/external_common.bat hack, the versions were.... x86 Windows7 3.x Buildbot (FAILURE) NASM version 2.06 compiled on Jul 1 2009 x86 XP-5 3.x Buildbot (OK) NASM version 2.07 compiled on Jul 19 2009 x86 XP-4 3.x Buildbot (OK) NASM version 2.02 compiled on Feb 23 2008 I didn't check 2.02 but probably it dosen't have this issue. Could you upgrade nasmw on Windows7 buildbot? Thank you. Regards, Yamamoto.

3 5