Mailman 3 July 2008 - Python-Dev

Any PEP about 2.6 -> 3000 code transition?
by Jesus Cea 13 Aug '08

13 Aug '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I can't find any PEP about detailed 2.6 -> 3000 migration guidelines, specially in the module (C code) extension area. Yes, I know about the 2to3 tool, but I'm interested in updating my 2.x code in order to keep (automatic via "2to3") difference between 2.x and 3.x codebase so small as possible. Also, 2to3 doesn't manage migration for C modules. Since I need to port bsddb3 to py3k, what I need to know?. Is any *updated* document out there?. PS: My plan is keep working in the python side under 2.x, and manage Python 3.0 via "2to3", for a long time. In the C side, I plan keeping the same codebase, with conditional compilation. Ideas?. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea(a)jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea@jabber.org _/_/ _/_/ _/_/_/_/_/ ~ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSDjqJplgi5GaxT1NAQLP3QP/Y2bWmNzHKsIq23dRz9sxd237TSfzbs5X UzEQ6Ze9Kv1lRXYRfiGjk37aOeiO4xqoKPfASc/WjMN/TmRkUNZxMQEQWwFYhmCA aAVqGxNMEX3KCXvRP9MkkZYLPa9RZg8boPTMYCn+uLDW1Ff2Pwj96hFKbWKG0dkJ RamNp8BFhkM= =ngK+ -----END PGP SIGNATURE-----

17 31

Re: [Python-Dev] Fuzzing bugs: most bugs are closed
by Victor Stinner 11 Aug '08

11 Aug '08

Le Saturday 19 July 2008 21:52:09 A.M. Kuchling, vous avez écrit : > Excellent work! Another fruitful area for fuzzing might be the > miniature virtual machine used by the re module. It's possible to > import _sre and call the compile() function directly (see the end of > Lib/sre_compile.py for how it's invoked); I wonder how the regex VM > copes with random strings of bytecode. Hum... how can I say it? It's trivial to crash _sre :-) So I blacklisted _sre.compile() in my fuzzer. For information, it's also very easy to crash CPython with fuzzed .pyc file. It's hard to check bytecode without execute it. It's maybe better to add checks directly in the VM. -- Victor Stinner aka haypo http://www.haypocalc.com/blog/

6 9

Re: [Python-Dev] urllib.quote and unquote - Unicode issues
by Guido van Rossum 08 Aug '08

08 Aug '08

On Wed, Jul 30, 2008 at 12:49 PM, Bill Janssen <janssen(a)parc.com> wrote: >> > unquote() -- takes string, produces bytes or string >> > >> > If optional "encoding" parameter is specified, decodes bytes with >> > that encoding and returns string. Otherwise, returns bytes. >> >> The default of returning bytes will break almost all uses. Most code >> will uses the unquoted result as a text string, not as bytes -- e.g. a >> server has to unquote the values it receives from a form (whether POST >> or GET), but almost always the unquoted values are text, e.g. >> someone's name or address, or a draft email message. > > I actually do know a lot about the uses of this function... > > But: OK, OK, I yield. Though I still think this is a bad idea, I'll > shut up if we can also add "unquote_as_bytes" which returns a byte > sequence instead of a string. I'll just change my code to use that. > >> (Aside: I dislike functions that have a different return type based on >> the value of a parameter.) > > Fair enough. I think this is as close as consensus as we can get on this issue. Can whoever wrote the patch adjust the patch to this outcome? (I think the only change is to remove the encoding arguments and make separate functions for bytes.) -- --Guido van Rossum (home page: http://www.python.org/~guido/)

11 28

Re: [Python-Dev] Matrix product
by Greg Ewing 02 Aug '08

02 Aug '08

Sebastien Loisel wrote: > What are the odds of this thing going in? I don't know. Guido has said nothing about it so far this time round, and his is the only opinion that matters in the end. I may write a PEP about this. However, since yesterday I've realised that there's a rather serious problem with part of my proposal. The problem is that being able to multiply matrices isn't much use without also being able to add them and multiply them by numbers, which obviously can't work with the built-in sequence types, since they already have other meanings for + and *. However, I still think that adding an @ operator for numpy to use is a good idea. So I'm now suggesting that the operator be added, with the intended meaning of matrix multiplication, but that no implementation of it be provided in core Python. There is a precedent for this -- the ellipsis notation was added purely for use by Numeric and its successors, and nothing in core Python attaches any meaning to it. > How do the PEPs work? Someone writes a PEP. People talk about it. Eventually, Guido either accepts it or rejects it (although in some cases it is an infinitely long time before that happens:-). -- Greg

13 19

Memory Error while reading large file
by Sumant Gupta 01 Aug '08

01 Aug '08

Hi I have a problem reading very large text file. When I call the len function to get the total lines in python file.i get memory error . I am reading the list of files in a loop ,2 files are read properly but when the third file is read , It gives an memory error . Sumant Gupta Software Engineer Ext:5105 ________________________________ "DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error,please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility forloss or damage arising from the use of the information transmitted by this email including damage from virus."

6 5

critical issues for 2.6 and 3.0
by Benjamin Peterson 01 Aug '08

01 Aug '08

I just went through the disturbingly long list of 67 open issues with a "critical" priority pinging and trying to get things moving. There are ~55 now; I was able to close some, but others I promoted to release blocker for beta 3. Shouldn't all criticals be resolved by the final? I've never been through a Python release before, but I find these statistics rather worrying if we want to make the October release date. It doesn't help that we are low on active core developers, presumably because they are taking full advantage of their summer vacations. :) (Speaking of which, I'm leaving this Saturday.) Please focus getting fixes reviewed, checked in, and their issue's closed so we can bring beta 3 out on time! -- Cheers, Benjamin Peterson "There's no place like 127.0.0.1."

3 2

Re: [Python-Dev] urllib.quote and unquote - Unicode issues
by Matt Giuca 31 Jul '08

31 Jul '08

Bill wrote: I'm not sure that's sufficient review, though I agree it's necessary. > The major consumers of quote/unquote are not in the Python standard > library. I figured that Python 3.0 is designed to fix things, with the breaking third-party code being an acceptable side-effect of that. So the most important thing when 3.0 is released is that the stdlib is internally consistent. All other code is "allowed" to be broken. So I've investigated all the code necessary. Having said this, my patch breaks almost no code. Your suggestion breaks a hell of a lot. Sure. All I was asking was that we not break the existing usage of > the standard library "unquote" by producing a string by *assuming* a > UTF-8 encoded string is what's in those percent-encoded bytes (instead > of, say, ISO 2022-JP). Let the "new" function produce a string: > "unquote_as_string". You're assuming that a Python 2.x "str" is the same thing as a Python 3.0 "bytes". It isn't. (If it was, this transition would be trivial). A Python 2 "str" is a non-Unicode string. It can be printed, concatenated with Unicode strings, etc etc. It has the semantics of a string. The Python 3.0 "bytes" is not a string at all. What you're saying is "the old behaviour was to output a bytes, so the new behaviour should be consistent". But that isn't true - the old behaviour was to output a string (a non-Unicode one). People, and code, expect it to output something with string semantics. So making unquote output a bytes is just as big a change as making it output a (unicode) str. Python 3.0 doesn't have a type which is like Python 2's "str" type (which is good - that type was very messy). So the argument that "Python 2 unquote outputs a bytes, so we should too" is not legitimate. If you want to keep pushing this, please install my new patch (patch 6). Then rename "unquote" to "unquote_to_string" and rename "unquote_to_bytes" to "unquote", and witness the havoc that ensues. Firstly, you break most Internet-related modules in the standard library. 10 tests failed: > test_SimpleHTTPServer test_cgi test_email test_http_cookiejar > test_httpservers test_robotparser test_urllib test_urllib2 > test_urllib2_localnet test_wsgiref > Fixing these isn't a matter of changing test cases (which all but one of my fixes were). It would require changes to all the modules, to get them to deal with bytes instead of strings (which would generally mean spraying .decode("utf-8") all over the place). My code, on the other hand, "tends to be" compatible with 2.x code. Here I'm seeing: BytesWarning: Comparison between bytes and string. TypeError: expected an object with the buffer interface http.client.BadStatusLine For another example, try this: >>> import http.server >>> s = http.server.HTTPServer(('',8000), http.server.SimpleHTTPRequestHandler) >>> s.serve_forever() The current (unpatched) build works, but links to files with non-ASCII filenames (eg. '漢字') break, because of the URL. This is one example of my patch directly fixing a bug in real code. With my patch applied, the links work fine *because URL quoting and unquoting are consistent, and work on all Unicode characters*. If you change unquote to output a bytes, it breaks completely. You get a "TypeError: expected an object with the buffer interface" as soon as the user visits the page. Matt

1 0

Re: [Python-Dev] urllib.quote and unquote - Unicode issues
by Guido van Rossum 31 Jul '08

31 Jul '08

On Wed, Jul 30, 2008 at 10:33 AM, Bill Janssen <janssen(a)parc.com> wrote: >> It looks like all other APIs in the Py3k version of >> urllib treat URLs as text. > > The URL is text, a string of ASCII characters. We're just talking > about urllib.quote() and urllib.unquote(), which are there to support > the text-ization of binary values, and the de-text-ization. > >> I think that would break too much code, without a good way to >> automatically fix it. > > You'd rather break Python? Somehow I don't think so. Let's stop the rhetoric, or I'll have to beat you over the head with the Zen of Python. :-) urllib is not meant as a reference implementation of any RFC; it is meant as a practical tool for Python users writing web apps (servers and clients). > Here's the signature I'm proposing: > > quote() -- takes string or bytes, and produces string. > > If input is a string, looks to optional "encoding" parameter to > determine character set encoding to use to transform it to byte before > quoting it. If "encoding" is not specified, defaults to UTF-8. No contest here, since it supports the common string->string use case. E.g. quote('a%b') returns 'a%25b'. > unquote() -- takes string, produces bytes or string > > If optional "encoding" parameter is specified, decodes bytes with > that encoding and returns string. Otherwise, returns bytes. The default of returning bytes will break almost all uses. Most code will uses the unquoted result as a text string, not as bytes -- e.g. a server has to unquote the values it receives from a form (whether POST or GET), but almost always the unquoted values are text, e.g. someone's name or address, or a draft email message. (Aside: I dislike functions that have a different return type based on the value of a parameter.) -- --Guido van Rossum (home page: http://www.python.org/~guido/)

3 2

Re: [Python-Dev] urllib.quote and unquote - Unicode issues
by Guido van Rossum 31 Jul '08

31 Jul '08

On Wed, Jul 30, 2008 at 9:52 AM, Bill Janssen <janssen(a)parc.com> wrote: >> On Wed, Jul 30, 2008 at 8:09 AM, André Malo <nd(a)perlig.de> wrote: >> > I'm actually in favour of encoding bytes only back and forth. A useful >> > extension would be *another* function which wraps quote/unquote and encod= >> es >> > and decodes characters. >> >> I'd reverse this. By all means, add a new pair of functions that is >> bytes in / bytes out. But keep the existing functions purely string in >> / string out, hardcoded to UTF-8. People wanting another encoding can >> use the bytes functions and explicit encode / decode calls. > > Actually (as I pointed out before) the existing functions are not > string-in/string-out. They are something-in and bytes-out. just look > like string-in/string-out because of the confusion between byte > strings and Unicode strings in Python 1 and 2. Actually, we'd need to look at the various other APIs in Py3k before we can decide whether these should be considered taking or returning bytes or text. It looks like all other APIs in the Py3k version of urllib treat URLs as text. I don't think switching these to bytes would be a good idea; you might as well claim that filenames should be bytes because that's how the filesystem stores them. > Look, Matt's suggestion is a degradation of the integrity of the > stdlib, because it enthrones a broken understanding, a misreading of > the RFC, in a very prominent place. I'd prefer not to have Python > contribute to that breakage. Keep the functions the way they are now: > bytes-in and bytes-out. I think that would break too much code, without a good way to automatically fix it. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

2 1

Re: [Python-Dev] urllib.quote and unquote - Unicode issues
by Matt Giuca 31 Jul '08

31 Jul '08

Hi folks, This issue got some attention a few weeks back but it seems to have fallen quiet, and I haven't had a good chance to sit down and reply again till now. As I've said before this is a serious issue which will affect a great deal of code. However it's obviously not as clear-cut as I originally believed, since there are lots of conflicting opinions. Let us see if we can come to a consensus. (For those who haven't seen the discussion, the thread starts here: http://mail.python.org/pipermail/python-dev/2008-July/081013.html continues here for some reason: http://mail.python.org/pipermail/python-dev/2008-July/081066.html and I've got a bug report with a fully tested and documented patch here: http://bugs.python.org/issue3300) Firstly, it looks like most of the people agree we should add an optional "encoding" argument which lets the caller customize which encoding to use. What we tend to disagree about is what the default encoding should be. Here I present the various options as I see it (and I'm trying to be impartial), and the people who've indicated support for that option (apologies if I've misrepresented anybody's opinion, feel free to correct): 1. Leave it as it is. quote is Latin-1 if range(0,256), fallback to UTF-8. unquote is Latin-1. In favour: Anybody who doesn't reply to this thread Pros: Already implemented; some existing code depends upon ord values of string being the same as they were for byte strings; possible to hack around it. Cons: unquote is not inverse of quote; quote behaviour internally-inconsistent; garbage when unquoting UTF-8-encoded URIs. 2. Default to UTF-8. In favour: Matt Giuca, Brett Cannon, Jeroen Ruigrok van der Werven Pros: Fully working and tested solution is implemented; recommended by RFC 3986 for all future schemes; recommended by W3C for use with HTML; UTF-8 used by all major browsers; supports all characters; most existing code compatible by default; unquote is inverse of quote. Cons: By default, URIs may have invalid octet sequences (not possible to reverse). 3. quote default to UTF-8, unquote default to Latin-1. In favour: André Malo Pros: quote able to handle all characters; unquote able to handle all sequences. Cons: unquote is not inverse of quote; totally inconsistent. 4. quote accepts either bytes or str, unquote default to outputting bytes unless given an encoding argument. In favour: Bill Janssen Pros: Technically does what the spec says, which is treat it as an octet encoding. Cons: unquote will break most existing code; almost 100% of the time people will want it as a string. </impartiality> I'll just comment on #4 since I haven't already. Let's talk about quote and unquote separately. For quote, I'm all for letting it accept a bytes as well as a str. That doesn't break anything or surprise anyone. For unquote, I think it will break a lot and surprise everyone. I think that while this may be "purely" the best option, it's pretty silly. I reckon the vast majority of users will be surprised when they see it spitting out a bytes object, and all that most people will do is decode it as UTF-8. Besides, while you're reading the RFCs as "URLs specify a method for encoding octet sequences", I'm reading them as "URLs specify a method for encoding strings, and leave the character encoding unspecified." The second reading supports the idea that unquote outputs a str. I'm also recommending we add unquote_to_bytes to do what you suggest unquote should do. (So either way we'll get both versions of unquote; I'm just suggesting the one called "unquote" do the thing everybody expects). But that's less of a priority so I want to commit these urgent fixes first. I'm basically saying just two things: 1. The standards are undefined; 2. Therefore we should pick the most useful and/or intuitive default. IMHO choosing UTF-8 *is* the most useful AND intuitive, and will be more so in the future when more technologies are hard-coded as UTF-8 (which this RFC recommends they do in the future). I am also quite adamant that unquote be the inverse of quote. Are there any more opinions on this matter? It would be good to reach a consensus. If anyone seriously wants to push a different alternative to mine, please write a working implementation and attach it to issue 3300. On the technical side of things, does anybody have time to review my patch for this issue? http://bugs.python.org/issue3300 Patch 5. It's just a patch for unquote, quote, and small related functions, as well as numerous changes to test cases and documentation. Cheers Matt

7 9