Mailman 3 August 2005 - Python-Dev

Re: [Python-Dev] Revised PEP 349: Allow str() to return unicode strings
by Dieter Maurer 25 Aug '05

25 Aug '05

The following message is a courtesy copy of an article that has been posted to comp.lang.python as well. Neil Schemenauer <nas(a)arctrix.com> writes on Mon, 22 Aug 2005 15:31:42 -0600: > ... > Some code may require that str() returns a str instance. In the > standard library, only one such case has been found so far. The > function email.header_decode() requires a str instance and the > email.Header.decode_header() function tries to ensure this by > calling str() on its argument. The code was fixed by changing > the line "header = str(header)" to: > > if isinstance(header, unicode): > header = header.encode('ascii') Note, that this is not equivalent to the old "str(header)": "str(header)" used Python's "default encoding" while the new code uses 'ascii'. The new code might be more correct than the old one has been. > ... > Alternative Solutions > > A new built-in function could be added instead of changing str(). > Doing so would introduce virtually no backwards compatibility > problems. However, since the compatibility problems are expected to > rare, changing str() seems preferable to adding a new built-in. Can we get a new builtin with the exact same behaviour as the current "str" which can be used when we do require an "str" (and cannot use a "unicode"). Dieter

2 1

New mailbox module
by A.M. Kuchling 25 Aug '05

25 Aug '05

Gregory K. Johnson, who's been working on the mailbox module in nondist/sandbox/mailbox for Google's Summer of Code, thinks his project is essentially complete. He's added the ability to modifying mailboxes by adding and removing messages, adding test cases for the new features, and written the corresponding documentation. So, it's time to start considering it for inclusion in the standard library. This is a big change to a non-obscure module, so don't feel able to make this decision on my own. I believe the code quality is acceptable, but would appreciate comments on any cleanups that need to be made. I still need to read through the docs and make editing suggestions, and check that the code is still backward-compatible with the old version of the module. --amk

2 1

Docs/Pointer to Tools/scripts?
by Reinhold Birkenfeld 24 Aug '05

24 Aug '05

Hi, after adding Oleg Broytmann's findnocoding.py to Tools/scripts, I wonder whether the Tools directory is documented at all. There are many useful scripts there which many people will not find if they are not listed anywhere in the docs. Just a thought. Reinhold -- Mail address is perfectly valid!

4 3

51 Million calls to _PyUnicodeUCS2_IsLinebreak() (???)
by Keir Mierle 24 Aug '05

24 Aug '05

Hi, I'm working on Argon (http://www.third-bit.com/trac/argon) with Greg Wilson this summer We're having a very strange problem with Python's unicode parsing of source files. Basically, our CGI script was running extremely slowly on our production box (a pokey dual-Xeon 3GHz w/ 4GB RAM and 15K SCSI drives). Slow to the tune of 6-10 seconds per request. I eventually tracked this down to imports of our source tree; the actual request was completing in 300ms, the rest of the time was spent in __import__. After doing some gprof profiling, I discovered _PyUnicodeUCS2_IsLinebreak was getting called 51 million times. Our code is 1.2 million characters, so I hardly think it makes sense to call IsLinebreak 50 times for each character; and we're not even importing our entire source tree on every invocation. Our code is a fork of Trac, and originally had these lines at the top: # -*- coding: iso8859-1 -*- This made me suspicious, so I removed all of them. The CGI execution time immediately dropped to ~1 second. gprof revealed that _PyUnicodeUCS2_IsLinebreak is not called at all anymore. Now that our code works fast enough, I don't really care about this, but I thought python-dev might want to know something weird is going on with unicode splitlines. I documented my investigation of this problem; if anyone wants further details, just email me. (I'm not on python-dev) http://www.third-bit.com/trac/argon/ticket/525 Thanks in advance, Keir

6 19

Bare except clauses in PEP 348
by Raymond Hettinger 24 Aug '05

24 Aug '05

The latest version of PEP 348 still proposes that a bare except clause will default to Exception instead of BaseException. Initially, I had thought that might be a good idea but now think it is doomed and needs to be removed from the PEP. A bare except belongs at the end of a try suite, not in the middle. This is obvious when compared to: if a: ... elif b: ... elif c: ... else: ... # The bare else goes at the end # and serves as a catchall or switch c case a: ... case b: ... default: ... # The bare default goes at the end # and serves as a catchall In contrast, Brett's 8/9 note revealed that the following would be allowable and common if the PEP is accepted in its current form: try: ... except: ... # A bare except in the middle. WTF? except (KeyboardInterrupt, SystemExit): ... The right way is, of course: try: ... except (KeyboardInterrupt, SystemExit): ... except: # Implicit or explicit match to BaseException # that serves as a catchall For those not needing a terminating exception handler, the rest of the PEP appropriately allows and encourages a simple and explicit solution that meets most needs: try: ... except Exception: ... The core issue is that the most obvious meaning of a bare except is "catchall", not "catchmost". When the latter is intended, the simple and explicit form shown in the last example is the way to go. If the former is intended, then either a bare except clause or explicit mention of BaseException will do nicely. However, under the PEP proposal, both new and existing code will suffer from having bare except clauses that look like they catch everything, are intended to catch everything, but, in fact, do not. That kind of optical illusion error must be avoided. There is no getting around our mind's propensity to interpret the bare form as defaulting to the top of the tree rather than the middle as proposed by the PEP. Likewise, there is no getting around the mental confusion caused a bare except clause in the middle of a try-suite rather than at the end. We have to avoid code that looks like it does one thing but actually does something else. Raymond

8 11

FW: Bare except clauses in PEP 348
by Raymond Hettinger 24 Aug '05

24 Aug '05

Hey guys, don't give up your bare except clauses so easily. They are useful. And, if given the natural meaning of "catch everything" and put in a natural position at the end of a suite, their meaning is plain and obvious. Remember beauty counts. I don't think there would be similar temptation to eliminate a dangling else clause and replace it with "else Everything". Nor would a final default case in a switch statement benefit from being written as "default Everything". The thought is that it is okay to have useful defaults. My whole issue was that the PEP was choosing the wrong default. If we leave it alone, all is well. An empty except will continue to mean "catch everything", it will always appear at the end, its meaning will be obvious, and existing working code won't break :-) On the occasions where you really intended to catch everything, do you really want to go on an editing binge just to uglify the code to something like: try: ... except SomeException: ... except BaseException: ... It is more beautiful and clear as: try: ... except SomeException: ... except: ... To me, the latter is more attractive and is more obviously a catchall, just like an else-clause or a default-statement. It is a strong visual cue that at least one of the except clauses will always be triggered. In contrast, the first example makes you think twice about whether the final case really does get everything (sometimes implicit IS better than explicit). Raymond

3 2

Re: [Python-Dev] FW: Bare except clauses in PEP 348
by Michael Chermside 24 Aug '05

24 Aug '05

Raymond writes: > Hey guys, don't give up your bare except clauses so easily. [...] Raymond: I agree that when comparing: // Code snippet A try: ... except SomeException: ... except BaseException: ... with // Code snippet B try: ... except SomeException: ... except: ... that B is nicer than A. Slightly nicer. It's a minor esthetic point. But consider these: // Code snippet C try: ... except Exception: ... // Code snippet D try: ... except: ... Esthetically I'd say that D is nicer than A for the same reasons. It's a minor esthetic point. But you see, this case is different. You and I would likely never bother to compare C and D because they do different things! (D is equivalent to catching BaseException, not Exception). But we know that people who are not so careful or not so knowlegable WILL make this mistake... they make it all the time today! Since situation C (catching an exception) is hundreds of times more common than situation A (needing default processing for exceptions that don't get caught, but doing it with try-except instead of try-finally because the nothing-was-thrown case is different), I would FAR rather protect beginners from erroniously confusing C and D than I would provide a marginally more elegent syntax for the experts using A or B. And that elegence is arguable... there's something to be said for simplicity, and having only one kind of "except" clause for try statements is clearly simpler than having both "except <some-exception-type>:" and also bare "except:". -- Michael Chermside

1 0

Re: [Python-Dev] [Python-checkins] python/dist/src/Modules _hashopenssl.c, NONE, 2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE, 2.1 md5module.c, 2.35, 2.36 shamodule.c, 2.22, 2.23
by Raymond Hettinger 24 Aug '05

24 Aug '05

This patch should be reverted or fixed so that the Py2.5 build works again. It contains a disasterous search and replace error that prevents it from compiling. Hence, it couldn't have passed the test suite before being checked in. Also, all of the project and config files need to be updated for the new modules. > -----Original Message----- > From: python-checkins-bounces(a)python.org [mailto:python-checkins- > bounces(a)python.org] On Behalf Of greg(a)users.sourceforge.net > Sent: Sunday, August 21, 2005 2:46 PM > To: python-checkins(a)python.org > Subject: [Python-checkins] python/dist/src/Modules _hashopenssl.c, > NONE,2.1 sha256module.c, NONE, 2.1 sha512module.c, NONE,2.1 md5module.c, > 2.35, 2.36 shamodule.c, 2.22, 2.23 > > Update of /cvsroot/python/python/dist/src/Modules > In directory sc8-pr-cvs1.sourceforge.net:/tmp/cvs-serv32064/Modules > > Modified Files: > md5module.c shamodule.c > Added Files: > _hashopenssl.c sha256module.c sha512module.c > Log Message: > [ sf.net patch # 1121611 ] > > A new hashlib module to replace the md5 and sha modules. It adds > support for additional secure hashes such as SHA-256 and SHA-512. The > hashlib module uses OpenSSL for fast platform optimized > implementations of algorithms when available. The old md5 and sha > modules still exist as wrappers around hashlib to preserve backwards > compatibility.

9 16

Re: [Python-Dev] python/dist/src/Doc/tut tut.tex,1.276,1.277
by Reinhold Birkenfeld 23 Aug '05

23 Aug '05

rhettinger(a)users.sourceforge.net wrote: I'm not a native speaker, but... > @@ -114,7 +114,7 @@ > programs, or to test functions during bottom-up program development. > It is also a handy desk calculator. > > -Python allows writing very compact and readable programs. Programs > +Python enables programs to written compactly and readably. Programs > written in Python are typically much shorter than equivalent C or > \Cpp{} programs, for several reasons: > \begin{itemize} ...shouldn't it be "programs to be written compactly"? > @@ -1753,8 +1753,8 @@ > > \begin{methoddesc}[list]{pop}{\optional{i}} > Remove the item at the given position in the list, and return it. If > -no index is specified, \code{a.pop()} returns the last item in the > -list. The item is also removed from the list. (The square brackets > +no index is specified, \code{a.pop()} removes and returns the last item > +in the list. The item is also removed from the list. (The square brackets > around the \var{i} in the method signature denote that the parameter > is optional, not that you should type square brackets at that > position. You will see this notation frequently in the Thats twice the same the same (removal from list). > @@ -1985,7 +1987,9 @@ > \section{The \keyword{del} statement \label{del}} > > There is a way to remove an item from a list given its index instead > -of its value: the \keyword{del} statement. This can also be used to > +of its value: the \keyword{del} statement. Unlike the \method{pop()}) > +method which returns a value, the \keyword{del} keyword is a statement > +and can also be used to > remove slices from a list (which we did earlier by assignment of an > empty list to the slice). For example: The del keyword is a statement? > @@ -2133,8 +2137,8 @@ > keys. Tuples can be used as keys if they contain only strings, > numbers, or tuples; if a tuple contains any mutable object either > directly or indirectly, it cannot be used as a key. You can't use > -lists as keys, since lists can be modified in place using their > -\method{append()} and \method{extend()} methods, as well as slice and > +lists as keys, since lists can be modified in place using methods like > +\method{append()} and \method{extend()} or modified with slice and > indexed assignments. Is the second "modified" necessary? > @@ -5595,8 +5603,8 @@ > to round it again can't make it better: it was already as good as it > gets. > > -Another consequence is that since 0.1 is not exactly 1/10, adding 0.1 > -to itself 10 times may not yield exactly 1.0, either: > +Another consequence is that since 0.1 is not exactly 1/10, > +summing ten values of 0.1 may not yield exactly 1.0, either: > > \begin{verbatim} > >>> sum = 0.0 Is it clear from context that the "0.1 is not exactly 1/10" refers to floating point only? > @@ -5637,7 +5645,7 @@ > you can perform an exact analysis of cases like this yourself. Basic > familiarity with binary floating-point representation is assumed. > > -\dfn{Representation error} refers to that some (most, actually) > +\dfn{Representation error} refers to fact that some (most, actually) > decimal fractions cannot be represented exactly as binary (base 2) > fractions. This is the chief reason why Python (or Perl, C, \Cpp, > Java, Fortran, and many others) often won't display the exact decimal "...refers to the fact..."? Reinhold -- Mail address is perfectly valid!

2 1

Revised PEP 349: Allow str() to return unicode strings
by Neil Schemenauer 23 Aug '05

23 Aug '05

[Please mail followups to python-dev(a)python.org.] The PEP has been rewritten based on a suggestion by Guido to change str() rather than adding a new built-in function. Based on my testing, I believe the idea is feasible. It would be helpful if people could test the patched Python with their own applications and report any incompatibilities. PEP: 349 Title: Allow str() to return unicode strings Version: $Revision: 1.3 $ Last-Modified: $Date: 2005/08/22 21:12:08 $ Author: Neil Schemenauer <nas(a)arctrix.com> Status: Draft Type: Standards Track Content-Type: text/plain Created: 02-Aug-2005 Post-History: 06-Aug-2005 Python-Version: 2.5 Abstract This PEP proposes to change the str() built-in function so that it can return unicode strings. This change would make it easier to write code that works with either string type and would also make some existing code handle unicode strings. The C function PyObject_Str() would remain unchanged and the function PyString_New() would be added instead. Rationale Python has had a Unicode string type for some time now but use of it is not yet widespread. There is a large amount of Python code that assumes that string data is represented as str instances. The long term plan for Python is to phase out the str type and use unicode for all string data. Clearly, a smooth migration path must be provided. We need to upgrade existing libraries, written for str instances, to be made capable of operating in an all-unicode string world. We can't change to an all-unicode world until all essential libraries are made capable for it. Upgrading the libraries in one shot does not seem feasible. A more realistic strategy is to individually make the libraries capable of operating on unicode strings while preserving their current all-str environment behaviour. First, we need to be able to write code that can accept unicode instances without attempting to coerce them to str instances. Let us label such code as Unicode-safe. Unicode-safe libraries can be used in an all-unicode world. Second, we need to be able to write code that, when provided only str instances, will not create unicode results. Let us label such code as str-stable. Libraries that are str-stable can be used by libraries and applications that are not yet Unicode-safe. Sometimes it is simple to write code that is both str-stable and Unicode-safe. For example, the following function just works: def appendx(s): return s + 'x' That's not too surprising since the unicode type is designed to make the task easier. The principle is that when str and unicode instances meet, the result is a unicode instance. One notable difficulty arises when code requires a string representation of an object; an operation traditionally accomplished by using the str() built-in function. Using the current str() function makes the code not Unicode-safe. Replacing a str() call with a unicode() call makes the code not str-stable. Changing str() so that it could return unicode instances would solve this problem. As a further benefit, some code that is currently not Unicode-safe because it uses str() would become Unicode-safe. Specification A Python implementation of the str() built-in follows: def str(s): """Return a nice string representation of the object. The return value is a str or unicode instance. """ if type(s) is str or type(s) is unicode: return s r = s.__str__() if not isinstance(r, (str, unicode)): raise TypeError('__str__ returned non-string') return r The following function would be added to the C API and would be the equivalent to the str() built-in (ideally it be called PyObject_Str, but changing that function could cause a massive number of compatibility problems): PyObject *PyString_New(PyObject *); A reference implementation is available on Sourceforge [1] as a patch. Backwards Compatibility Some code may require that str() returns a str instance. In the standard library, only one such case has been found so far. The function email.header_decode() requires a str instance and the email.Header.decode_header() function tries to ensure this by calling str() on its argument. The code was fixed by changing the line "header = str(header)" to: if isinstance(header, unicode): header = header.encode('ascii') Whether this is truly a bug is questionable since decode_header() really operates on byte strings, not character strings. Code that passes it a unicode instance could itself be considered buggy. Alternative Solutions A new built-in function could be added instead of changing str(). Doing so would introduce virtually no backwards compatibility problems. However, since the compatibility problems are expected to rare, changing str() seems preferable to adding a new built-in. The basestring type could be changed to have the proposed behaviour, rather than changing str(). However, that would be confusing behaviour for an abstract base type. References [1] http://www.python.org/sf/1266570 Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

6 11