Mailman 3 September 2010 - Python-Dev

Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices)
by Nick Coghlan Sept. 23, 2010

Sept. 23, 2010

On Sun, Sep 19, 2010 at 4:18 AM, John Nagle <nagle(a)animats.com> wrote: > On 9/18/2010 2:29 AM, python-dev-request(a)python.org wrote: >> >> Polymorphic best practices [was: (Not) delaying the 3.2 release] > > If you're hung up on this, try writing the user-level documentation > first. Your target audience is a working-level Web programmer, not > someone who knows six programming languages and has a CS degree. > If the explanation is too complex,… [View More] so is the design. > > Coding in this area is quite hard to do right. There are > issues with character set, HTML encoding, URL encoding, and > internationalized domain names. It's often done wrong; > I recently found a Google service which botched it. > Python libraries should strive to deliver textual data to the programmer > in clean Unicode. If someone needs the underlying wire representation > it should be available, but not the default. Even though URL byte sequences are defined as using only an ASCII subset, I'm currently inclined to add raw bytes supports to urlib.parse by providing parallel APIs (i.e. urlib.parse.urlsplitb, etc) rather than doing it implicitly in the normal functions. My rationale is as follows: - while URLs are *meant* to be encoded correctly as an ASCII subset, the real world isn't always quite so tidy (i.e. applications treat as URLs things that technically are not because the encoding is wrong) - separating the APIs forces the programmer to declare that they know they're working with the raw bytes off the wire to avoid the decode/encode overhead that comes with working in the Unicode domain - easier to change our minds later. Adding implicit bytes support to the normal names can be done any time, but removing it would require an extensive deprecation period Essentially, while I can see strong use cases for wanting to manipulate URLs in wire format, I *don't* see strong use cases for manipulating URLs without *knowing* whether they're in wire format (encoded bytes) or display format (Unicode text). For some APIs that work for arbitrary encodings (e.g. os.listdir) switching based on argument type seems like a reasonable idea. For those that may silently produce incorrect output for ASCII-incompatible encodings, the os.environ/os.environb seems like a better approach. I could probably be persuaded to merge the APIs, but the email6 precedent suggests to me that separating the APIs better reflects the mental model we're trying to encourage in programmers manipulating text (i.e. the difference between the raw octet sequence and the text character sequence/parsed data). Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia [View Less]

8 16

Advice sought on memory allocation latency reduction C1X standard proposal
by Niall Douglas Sept. 22, 2010

Sept. 22, 2010

Dear Python Devs, I am hoping to gain feedback on an ISO C1X/C++ standard library proposal I hope to submit. It consists of a rationale (http://mallocv2.wordpress.com/) which shows how growth in RAM capacity is exponentially outgrowing the growth in RAM access speed. The consequences are profound: computer software which has always been written under the assumption of scarcity of RAM capacity will need to be retargeted to assume the scarcity of RAM access speed instead. The C1X … [View More]proposal (http://mallocv2.wordpress.com/the-c-proposal-text/) enables four things of interest to Python: (i) aligned block resizing (ii) speculative in-place block resizing (iii) batch block allocation and (iv) the ability to reserve address space, thus avoiding the need to overallocate array storage. Aligned block resizing is especially useful to numpy. Where one has an array of aligned SSE vector quantities one cannot currently resize that block and guarantee that alignment will not be destroyed. With the new feature of non-relocating realloc() and being able to specify an alignment to realloc() one may avoid memory copying, and therefore reduce memory bandwidth utilisation and therefore overall memory access latencies. The ability to reserve address space and speculative in-place block resizing can be combined to allow Python to reserve an arbitrary amount of address space after the storage for an array object. Should the array then become extended, the speculative in-place block resizing can attempt to expand storage into that reserved space without having to relocate the contents of the storage. This again translates into much reduced memory copying as well as memory consumption, and once again reduces overall memory access latencies. Lastly, the batch allocation mechanism allows a sequence of allocations to be performed at once. I don't know of any attempts to have Python make use of similar functionality in Linux's system allocator, however Perl saw a 18% reduction in startup time (http://groups.google.com/group/perl-compiler/msg/31bca5297764002b). I am not familiar with Python's implementation outside working extensively with Boost.Python, so I was hoping that this list could advise me on what I might be forgetting, what problems there could be for Python with this design and/or any other general concerns and thoughts. I thank the list in advance for your time and consideration. Niall Douglas -- Technology & Consulting Services - ned Productions Limited. http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Company no: 472909. [View Less]

1 0

Re: [Python-Dev] Supporting raw bytes data in urllib.parse.* (was Re: Polymorphic best practices)
by Nick Coghlan Sept. 22, 2010

Sept. 22, 2010

On Tue, Sep 21, 2010 at 3:03 PM, Stephen J. Turnbull <stephen(a)xemacs.org> wrote: > On the other hand, it is dangerous to provide a polymorphic API which > does that more extensive parsing, because a less than paranoid > programmer will have very likely allowed the parsed components to > escape from the context where their encodings can be reliably > determined. Remember, *it is unlikely that they will ever be punished > for their own lack of caution.* The … [View More]

7 9

Backup plan: WSGI 1 Addenda and wsgiref update for Py3
by P.J. Eby Sept. 21, 2010

Sept. 21, 2010

While the Web-SIG is trying to hash out PEP 444, I thought it would be a good idea to have a backup plan that would allow the Python 3 stdlib to move forward, without needing a major new spec to settle out implementation questions. After all, even if PEP 333 is ultimately replaced by PEP 444, it's probably a good idea to have *some* sort of WSGI 1-ish thing available on Python 3, with bytes/unicode and other matters settled. In the past, I was waiting for some consensuses (consensi?) on … [View More]Web-SIG about different approaches to Python 3, looking for some sort of definite, "yes, we all like this" response. However, I can see now that this just means it's my fault we don't have a spec yet. :-( So, unless any last-minute showstopper rebuttals show up this week, I've decided to go ahead officially bless nearly all of what Graham Dumpleton (who's not only the mod_wsgi author, but has put huge amounts of work into shepherding WSGI-on-Python3 proposals, WSGI amendments, etc.) has proposed, with a few minor exceptions. In other words: almost none of the following is my own original work; it's like 90% Graham's. Any praise for this belongs to him; the only thing that belongs to me is the blame for not doing this sooner! (Sorry Graham. You asked me to do this ages ago, and you were right.) Anyway, I'm posting this for comment to both Python-Dev and the Web-SIG. If you are commenting on the technical details of the amendments, please reply to the Web-SIG only. If you are commenting on the development agenda for wsgiref or other Python 3 library issues, please reply to Python-Dev only. That way, neither list will see off-topic discussions. Thanks! The Plan ======== I plan to update the proposal below per comments and feedback during this week, then update PEP 333 itself over the weekend or early next week, followed by a code review of Python 3's wsgiref, and implementation of needed changes (such as recoding os.environ to latin1-captured bytes in the CGI handler). To complete the changes, it is possible that I may need assistance from one or more developers who have more Python 3 experience. If after reading the proposed changes to the spec, you would like to volunteer to help with updating wsgiref to match, please let me know! The Proposal ============ Overview -------- 1. The primary purpose of this update is to provide a uniform porting pattern for moving Python 2 WSGI code to Python 3, meaning a pattern of changes that can be mechanically applied to as little code as practical, while still keeping the WSGI spec easy to programmatically validate (e.g. via ``wsgiref.validate``). The Python 3 specific changes are to use: * ``bytes`` for I/O streams in both directions * ``str`` for environ keys and values * ``bytes`` for arguments to start_response() and write() * text stream for wsgi.errors In other words, "strings in, bytes out" for headers, bytes for bodies. In general, only changes that don't break Python 2 WSGI implementations are allowed. The changes should also not break mod_wsgi on Python 3, but may make some Python 3 wsgi applications non-compliant, despite continuing to function on mod_wsgi. This is because mod_wsgi allows applications to output string headers and bodies, but I am ruling that option out because it forces every piece of middleware to have to be tested with arbitrary combinations of strings and bytes in order to test compliance. If you want your application to output strings rather than bytes, you can always use a decorator to do that. (And a sample one could be provided in wsgiref.) 2. The secondary purpose of the update is to address some long-standing open issues documented here: http://www.wsgi.org/wsgi/Amendments_1.0 As with the Python 3 changes, only changes that don't retroactively invalidate existing implementations are allowed. 3. There is no tertiary purpose. ;-) (By which I mean, all other kinds of changes are out-of-scope for this update.) 4. The section below labeled "A Note On String Types" is proposed for verbatim addition to the "Specification Overview" section in the PEP; the other sections below describe changes to be made inline at the appropriate part of the spec, and changes that were proposed but are rejected for inclusion in this amendment. A Note On String Types ---------------------- In general, HTTP deals with bytes, which means that this specification is mostly about handling bytes. However, the content of those bytes often has some kind of textual interpretation, and in Python, strings are the most convenient way to handle text. But in many Python versions and implementations, strings are Unicode, rather than bytes. This requires a careful balance between a usable API and correct translations between bytes and text in the context of HTTP... especially to support porting code between Python implementations with different ``str`` types. WSGI therefore defines two kinds of "string": * "Native" strings (which are always implemented using the type named ``str``) * "Bytestrings" (which are implemented using the ``bytes`` type in Python 3, and ``str`` elsewhere) So, even though HTTP is in some sense "really just bytes", there are many API conveniences to be had by using whatever Python's default ``str`` type is. Do not be confused however: even if Python's ``str`` is actually Unicode under the hood, the *content* of a native string is still restricted to bytes! See the section on `Unicode Issues`_ later in this document. In short: where you see the word "string" in this document, it refers to a "native" string, i.e., an object of type ``str``, whether it is internally implemented as bytes or unicode. Where you see references to "bytestring", this should be read as "an object of type ``bytes`` under Python 3, or type ``str`` under Python 2". Clarifications (To be made in-line) ----------------------------------- The following amendments are clarifications to parts of the existing spec that proved over the years to be ambiguous or insufficiently specified, as well as some attempts to correct practical errors. (Note: many of these issues cannot be completely fixed in WSGI 1 without breaking existing implementations, and so the text below has notations such as "(MUST in WSGI 2)" to indicate where any replacement spec for WSGI 1 should strengthen them.) * If an application returns a body iterator, a server (or middleware) MAY stop iterating over it and discard the remainder of the output, as long as it calls any close() method provided by the iterator. Applications returning a generator or other custom iterator SHOULD NOT assume that the entire iterator will be consumed. (This change makes it explicit that caching middleware or HEAD-processing servers can throw away the response body.) * start_response() SHOULD (MUST in WSGI 2) check for errors in the status or headers at the time it's called, so that an error can be raised as close to the problem as possible * If start_response() raises an error when called normally (i.e. without exc_info), it SHOULD be an error to call it a second time without passing exc_info * The SERVER_PORT variable is of type str, just like any other CGI environ variable. (According to the WSGI wiki, "some implementations" expect it to be an integer, even though there is nothing in the WSGI spec that allows a CGI variable to be anything but a str.) * A server SHOULD (MUST in WSGI 2) support the size hint argument to readline() on its wsgi.input stream. * A server SHOULD (MUST in WSGI 2) return an empty bytestring from read() on wsgi.input to indicate an end-of-file condition. (In WSGI 2, language should be clarified to allow the input stream length and CONTENT_LENGTH to be out of sync, for reasons explained in Graham's blog post.) * A server SHOULD (MUST in WSGI 2) allow read() to be called without an argument, and return the entire remaining contents of the stream * If an application provides a Content-Length header, the server SHOULD NOT (MUST NOT in WSGI 2) send more data to the client than was specified in that header, whether via write(), yielded body bytestrings, or via a wsgi.file_wrapper. (This rule applies to middleware as well.) * wsgi.errors is a text stream accepting "native strings" Rejected Amendments ------------------- * Manlio Perillo's suggestion to allow header specification to be delayed until the response iterator is producing non-empty output. This would've been a possible win for async WSGI, but could require substantial changes to existing servers. [View Less]

5 6

Re: [Python-Dev] [Web-SIG] Backup plan: WSGI 1 Addenda and wsgiref update for Py3
by P.J. Eby Sept. 21, 2010

Sept. 21, 2010

At 12:55 PM 9/21/2010 -0400, Ian Bicking wrote: >On Tue, Sep 21, 2010 at 12:47 PM, Chris McDonough ><<mailto:chrism@plope.com>chrism(a)plope.com> wrote: >On Tue, 2010-09-21 at 12:09 -0400, P.J. Eby wrote: > > While the Web-SIG is trying to hash out PEP 444, I thought it would > > be a good idea to have a backup plan that would allow the Python 3 > > stdlib to move forward, without needing a major new spec to settle > > out implementation questions. … [View More]

1 0

Re: [Python-Dev] [Python-checkins] r84858 - in python/branches: py3k/Doc/library/logging.rst release27-maint/Doc/library/logging.rst
by Éric Araujo Sept. 21, 2010

Sept. 21, 2010

Hello + NOTE: If you are thinking of defining your own levels, please see the section + on :ref:`custom-levels`. I think those instances of upper-case-as-markup should either be real reST note/warning/etc. directives or plain English (that is, integrating “NOTE:” into the text flow, for example “Note that...”/“Pay attention to...”). Regards

6 5

Re: [Python-Dev] r84931 - in python/branches/py3k: Include/symtable.h Misc/NEWS Python/ast.c Python/compile.c Python/future.c Python/symtable.c
by Georg Brandl Sept. 21, 2010

Sept. 21, 2010

Am 21.09.2010 01:02, schrieb benjamin.peterson: > Author: benjamin.peterson > Date: Tue Sep 21 01:02:10 2010 > New Revision: 84931 > > Log: > add column offset to all syntax errors > Modified: python/branches/py3k/Misc/NEWS > ============================================================================== > --- python/branches/py3k/Misc/NEWS (original) > +++ python/branches/py3k/Misc/NEWS Tue Sep 21 01:02:10 2010 > @@ -10,9 +10,8 @@ > Core and Builtins > -… [View More]

1 0

Re: [Python-Dev] [Python-checkins] r84906 - python/branches/py3k/Doc/library/ssl.rst
by Éric Araujo Sept. 21, 2010

Sept. 21, 2010

> Log: > Remove references to read() and write() methods, which are useless > synonyms of recv() and send() Unless I’m mistaken, ssl.SSLSocket.write is still useful for use with print, pprint and maybe other functions, so I think the method should be listed somewhere. Regards

2 1

Re: [Python-Dev] [Python-checkins] r84871 + committing distutils fixes
by Éric Araujo Sept. 21, 2010

Sept. 21, 2010

>+ @unittest.skipIf(sys.platform.startswith('win'), >+ "This test is only appropriate for POSIX-like systems.") Is win32 the only non-POSIX supported platform now that OS 9 support is gone? I find those values in the docs for sys.platform: win32, cygwin, darwin, os2, os2emx. Is OS/2 a POSIX system? Isn’t it more robust not to ask this question and just use unittest.skipUnless(os.name == 'posix', msg)? On a related note, I’m very grateful that other … [View More]

1 0

Fwd: Builder: x86 Windows7 3.x OpenSSL compile error
by Hirokazu Yamamoto Sept. 21, 2010

Sept. 21, 2010

Hello. I've sent following mail to buildbot manager, but I found that buildbot page saids the problem of unsable bot should be sent to python-dev(a)python.org. So I'll do it. -------- Original Message -------- Subject: Builder: x86 Windows7 3.x OpenSSL compile error Date: Mon, 20 Sep 2010 22:53:04 +0900 From: Hirokazu Yamamoto <ocean-city(a)m2.ccsnet.ne.jp> To: db3l.net(a)gmail.com Hello, David Bolen. I'm Hirokazu Yamamoto, Python committer. I've noticed your buildbot x86 Windows7 3.x … [View More]

3 5