Mailman 3 July 2008 - Python-Dev

Any PEP about 2.6 -> 3000 code transition?
by Jesus Cea Aug. 13, 2008

Aug. 13, 2008

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I can't find any PEP about detailed 2.6 -> 3000 migration guidelines, specially in the module (C code) extension area. Yes, I know about the 2to3 tool, but I'm interested in updating my 2.x code in order to keep (automatic via "2to3") difference between 2.x and 3.x codebase so small as possible. Also, 2to3 doesn't manage migration for C modules. Since I need to port bsddb3 to py3k, what I need to know?. Is any *updated* document out there?. PS:… [View More]

17 31

Re: [Python-Dev] Fuzzing bugs: most bugs are closed
by Victor Stinner Aug. 10, 2008

Aug. 10, 2008

Le Saturday 19 July 2008 21:52:09 A.M. Kuchling, vous avez écrit : > Excellent work! Another fruitful area for fuzzing might be the > miniature virtual machine used by the re module. It's possible to > import _sre and call the compile() function directly (see the end of > Lib/sre_compile.py for how it's invoked); I wonder how the regex VM > copes with random strings of bytecode. Hum... how can I say it? It's trivial to crash _sre :-) So I blacklisted _sre.compile() in my … [View More]

6 9

Re: [Python-Dev] urllib.quote and unquote - Unicode issues
by Guido van Rossum Aug. 7, 2008

Aug. 7, 2008

On Wed, Jul 30, 2008 at 12:49 PM, Bill Janssen <janssen(a)parc.com> wrote: >> > unquote() -- takes string, produces bytes or string >> > >> > If optional "encoding" parameter is specified, decodes bytes with >> > that encoding and returns string. Otherwise, returns bytes. >> >> The default of returning bytes will break almost all uses. Most code >> will uses the unquoted result as a text string, not as bytes -- e.g. a >… [View More]

11 28

Re: [Python-Dev] Matrix product
by Greg Ewing Aug. 1, 2008

Aug. 1, 2008

Sebastien Loisel wrote: > What are the odds of this thing going in? I don't know. Guido has said nothing about it so far this time round, and his is the only opinion that matters in the end. I may write a PEP about this. However, since yesterday I've realised that there's a rather serious problem with part of my proposal. The problem is that being able to multiply matrices isn't much use without also being able to add them and multiply them by numbers, which obviously can't work with the … [View More]

13 19

Memory Error while reading large file
by Sumant Gupta Aug. 1, 2008

Aug. 1, 2008

Hi I have a problem reading very large text file. When I call the len function to get the total lines in python file.i get memory error . I am reading the list of files in a loop ,2 files are read properly but when the third file is read , It gives an memory error . Sumant Gupta Software Engineer Ext:5105 ________________________________ "DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged … [View More]

6 5

critical issues for 2.6 and 3.0
by Benjamin Peterson July 31, 2008

July 31, 2008

I just went through the disturbingly long list of 67 open issues with a "critical" priority pinging and trying to get things moving. There are ~55 now; I was able to close some, but others I promoted to release blocker for beta 3. Shouldn't all criticals be resolved by the final? I've never been through a Python release before, but I find these statistics rather worrying if we want to make the October release date. It doesn't help that we are low on active core developers, presumably because … [View More]

3 2

Re: [Python-Dev] urllib.quote and unquote - Unicode issues
by Matt Giuca July 31, 2008

July 31, 2008

Bill wrote: I'm not sure that's sufficient review, though I agree it's necessary. > The major consumers of quote/unquote are not in the Python standard > library. I figured that Python 3.0 is designed to fix things, with the breaking third-party code being an acceptable side-effect of that. So the most important thing when 3.0 is released is that the stdlib is internally consistent. All other code is "allowed" to be broken. So I've investigated all the code necessary. Having said this,… [View More] my patch breaks almost no code. Your suggestion breaks a hell of a lot. Sure. All I was asking was that we not break the existing usage of > the standard library "unquote" by producing a string by *assuming* a > UTF-8 encoded string is what's in those percent-encoded bytes (instead > of, say, ISO 2022-JP). Let the "new" function produce a string: > "unquote_as_string". You're assuming that a Python 2.x "str" is the same thing as a Python 3.0 "bytes". It isn't. (If it was, this transition would be trivial). A Python 2 "str" is a non-Unicode string. It can be printed, concatenated with Unicode strings, etc etc. It has the semantics of a string. The Python 3.0 "bytes" is not a string at all. What you're saying is "the old behaviour was to output a bytes, so the new behaviour should be consistent". But that isn't true - the old behaviour was to output a string (a non-Unicode one). People, and code, expect it to output something with string semantics. So making unquote output a bytes is just as big a change as making it output a (unicode) str. Python 3.0 doesn't have a type which is like Python 2's "str" type (which is good - that type was very messy). So the argument that "Python 2 unquote outputs a bytes, so we should too" is not legitimate. If you want to keep pushing this, please install my new patch (patch 6). Then rename "unquote" to "unquote_to_string" and rename "unquote_to_bytes" to "unquote", and witness the havoc that ensues. Firstly, you break most Internet-related modules in the standard library. 10 tests failed: > test_SimpleHTTPServer test_cgi test_email test_http_cookiejar > test_httpservers test_robotparser test_urllib test_urllib2 > test_urllib2_localnet test_wsgiref > Fixing these isn't a matter of changing test cases (which all but one of my fixes were). It would require changes to all the modules, to get them to deal with bytes instead of strings (which would generally mean spraying .decode("utf-8") all over the place). My code, on the other hand, "tends to be" compatible with 2.x code. Here I'm seeing: BytesWarning: Comparison between bytes and string. TypeError: expected an object with the buffer interface http.client.BadStatusLine For another example, try this: >>> import http.server >>> s = http.server.HTTPServer(('',8000), http.server.SimpleHTTPRequestHandler) >>> s.serve_forever() The current (unpatched) build works, but links to files with non-ASCII filenames (eg. '漢字') break, because of the URL. This is one example of my patch directly fixing a bug in real code. With my patch applied, the links work fine *because URL quoting and unquoting are consistent, and work on all Unicode characters*. If you change unquote to output a bytes, it breaks completely. You get a "TypeError: expected an object with the buffer interface" as soon as the user visits the page. Matt [View Less]

1 0

Re: [Python-Dev] urllib.quote and unquote - Unicode issues
by Guido van Rossum July 30, 2008

July 30, 2008

On Wed, Jul 30, 2008 at 10:33 AM, Bill Janssen <janssen(a)parc.com> wrote: >> It looks like all other APIs in the Py3k version of >> urllib treat URLs as text. > > The URL is text, a string of ASCII characters. We're just talking > about urllib.quote() and urllib.unquote(), which are there to support > the text-ization of binary values, and the de-text-ization. > >> I think that would break too much code, without a good way to >> automatically fix … [View More]

3 2

Re: [Python-Dev] urllib.quote and unquote - Unicode issues
by Guido van Rossum July 30, 2008

July 30, 2008

On Wed, Jul 30, 2008 at 9:52 AM, Bill Janssen <janssen(a)parc.com> wrote: >> On Wed, Jul 30, 2008 at 8:09 AM, André Malo <nd(a)perlig.de> wrote: >> > I'm actually in favour of encoding bytes only back and forth. A useful >> > extension would be *another* function which wraps quote/unquote and encod= >> es >> > and decodes characters. >> >> I'd reverse this. By all means, add a new pair of functions that is >> bytes in / bytes … [View More]

2 1

Re: [Python-Dev] urllib.quote and unquote - Unicode issues
by Matt Giuca July 30, 2008

July 30, 2008

Hi folks, This issue got some attention a few weeks back but it seems to have fallen quiet, and I haven't had a good chance to sit down and reply again till now. As I've said before this is a serious issue which will affect a great deal of code. However it's obviously not as clear-cut as I originally believed, since there are lots of conflicting opinions. Let us see if we can come to a consensus. (For those who haven't seen the discussion, the thread starts here: http://mail.python.org/… [View More]pipermail/python-dev/2008-July/081013.html continues here for some reason: http://mail.python.org/pipermail/python-dev/2008-July/081066.html and I've got a bug report with a fully tested and documented patch here: http://bugs.python.org/issue3300) Firstly, it looks like most of the people agree we should add an optional "encoding" argument which lets the caller customize which encoding to use. What we tend to disagree about is what the default encoding should be. Here I present the various options as I see it (and I'm trying to be impartial), and the people who've indicated support for that option (apologies if I've misrepresented anybody's opinion, feel free to correct): 1. Leave it as it is. quote is Latin-1 if range(0,256), fallback to UTF-8. unquote is Latin-1. In favour: Anybody who doesn't reply to this thread Pros: Already implemented; some existing code depends upon ord values of string being the same as they were for byte strings; possible to hack around it. Cons: unquote is not inverse of quote; quote behaviour internally-inconsistent; garbage when unquoting UTF-8-encoded URIs. 2. Default to UTF-8. In favour: Matt Giuca, Brett Cannon, Jeroen Ruigrok van der Werven Pros: Fully working and tested solution is implemented; recommended by RFC 3986 for all future schemes; recommended by W3C for use with HTML; UTF-8 used by all major browsers; supports all characters; most existing code compatible by default; unquote is inverse of quote. Cons: By default, URIs may have invalid octet sequences (not possible to reverse). 3. quote default to UTF-8, unquote default to Latin-1. In favour: André Malo Pros: quote able to handle all characters; unquote able to handle all sequences. Cons: unquote is not inverse of quote; totally inconsistent. 4. quote accepts either bytes or str, unquote default to outputting bytes unless given an encoding argument. In favour: Bill Janssen Pros: Technically does what the spec says, which is treat it as an octet encoding. Cons: unquote will break most existing code; almost 100% of the time people will want it as a string. </impartiality> I'll just comment on #4 since I haven't already. Let's talk about quote and unquote separately. For quote, I'm all for letting it accept a bytes as well as a str. That doesn't break anything or surprise anyone. For unquote, I think it will break a lot and surprise everyone. I think that while this may be "purely" the best option, it's pretty silly. I reckon the vast majority of users will be surprised when they see it spitting out a bytes object, and all that most people will do is decode it as UTF-8. Besides, while you're reading the RFCs as "URLs specify a method for encoding octet sequences", I'm reading them as "URLs specify a method for encoding strings, and leave the character encoding unspecified." The second reading supports the idea that unquote outputs a str. I'm also recommending we add unquote_to_bytes to do what you suggest unquote should do. (So either way we'll get both versions of unquote; I'm just suggesting the one called "unquote" do the thing everybody expects). But that's less of a priority so I want to commit these urgent fixes first. I'm basically saying just two things: 1. The standards are undefined; 2. Therefore we should pick the most useful and/or intuitive default. IMHO choosing UTF-8 *is* the most useful AND intuitive, and will be more so in the future when more technologies are hard-coded as UTF-8 (which this RFC recommends they do in the future). I am also quite adamant that unquote be the inverse of quote. Are there any more opinions on this matter? It would be good to reach a consensus. If anyone seriously wants to push a different alternative to mine, please write a working implementation and attach it to issue 3300. On the technical side of things, does anybody have time to review my patch for this issue? http://bugs.python.org/issue3300 Patch 5. It's just a patch for unquote, quote, and small related functions, as well as numerous changes to test cases and documentation. Cheers Matt [View Less]

7 9