On Oct 6, 2008, at 8:52 PM, Benjamin Peterson wrote:

I'm not sure we do. Correct me if I'm wrong, but the "big ticket", issue bytes/unicode filepaths, has been resolved. And looking at the tracker, I only see 18 release blockers.

Well, if you mean that the resolution decided upon is to "simply" allow access to all system APIs using either byte or unicode strings, then it seems to me that there's a rather large amount of work left to do... Here's some I found from a few minutes of futzing around with r66821 of py3k on Linux. - Having os.getcwdb isn't much use when you can't even run python in the first place when the current directory has "bad" bytes in it. Currently Python outputs: Could not find platform independent libraries <prefix> Could not find platform dependent libraries Consider setting $PYTHONHOME to <prefix>[:] Fatal Python error: Py_Initialize: can't initialize sys standard streams ImportError: No module named encodings.utf_8 Aborted - I'd think "find . -type f -print0 | xargs -0 python -c 'pass'" ought to work (with files with "bad" bytes being returned by find), which means that Python shouldn't blow up and refuse to start when there's a non-properly-encoding argv ("Could not convert argument 1 to string" and exiting isn't appropriate behavior). - Of course, just being able to start the interpreter isn't quite enough: you'll want to be able to access that argument list too, somehow (add sys.argvb?). - And then, getopt and optparse modules should work on bytestring vectors, so that you can use sys.argvb without writing your own argument parser. They don't currently. - There's no os.environb for bytewise access to the environment. Seems important. - Isn't it a potential security issue that " 'WHATEVER' in os.environ" can return False if WHATEVER had some "bad" bytes in it, but spawning a subprocess actually will include WHATEVER in the subprocess's environment? Actually, even better: the behavior depends on whether you use subprocess.call('foo') or subprocess.call('foo', os.environ). The first passes through the "bad" environment variables, while the second does not. A bit surprising, perhaps. - Shouldn't this work? subprocess.call(b'/bin/echo') Currently raises an exception: AttributeError: 'int' object has no attribute 'rfind' - I suppose sys.path should handle bytestrings on the path, and should be populated using the bytes-version of os.environ so that PYTHONPATH gets read in properly. Which of course implies that all the importers need to handle byte filenames. - zipfile.ZipFile(b'whatever.zip') doesn't work. - zipfile decodes/encodes the filenames inside the zip file to unicode, so thus can only handle correctly encoded filenames. I'm sure there's even more APIs dealing with pathnames, command line arguments, or environment variables that ought to be able to handle both bytes and strings, that currently don't. James

On Oct 7, 2008, at 3:47 AM, Martin v. Löwis wrote:

...

- Having os.getcwdb isn't much use when you can't even run python in the first place when the current directory has "bad" bytes in it.

That's not true: it *is* of much use. Python will live in /usr/bin, which has a nicely-decodable path.

...

Currently Python outputs: Could not find platform independent libraries <prefix> Could not find platform dependent libraries Consider setting $PYTHONHOME to <prefix>[:] Fatal Python error: Py_Initialize: can't initialize sys standard streams ImportError: No module named encodings.utf_8 Aborted

I can't reproduce that. This happens (for me) when Python lives in a directory that has an undecodable path - not when the current directory is undecodable.

Sorry about that: this test was indeed in error: I ran "../python" from an undecodeable current directory, rather than "/full/path/to/ python", or putting python on the PATH and running it as "python". The first does not work, but the other more common ways to start it do.

...

I'm sure there's even more APIs dealing with pathnames, command line arguments, or environment variables that ought to be able to handle both bytes and strings, that currently don't.

Please, no.

I completely and totally agree with your distate, it's rather gross to allow bytes-or-str for every API that touches anything like filenames/ argv/environ. That's why I was pushing for the reversible conversion to str...But if bytes-or-str is the solution that's been chosen for this issue, it ought to either be fully committed to and implemented, or at least fully recognized and documented as a half-baked solution. Of course, if an reversible encoding into string solution is used instead, none of these things would need special treatment: they would all work already. FWIW: Qt works fine with undecodeable filenames, and it too uses unicode strings everywhere in its API. I looked into what it does, and found that it uses your (Martin)'s original idea for solving this: it stores undecodeable bytes as characters from 0x10fe00 to 0x10feff (which is valid private-use codespace). While that might not be ideally correct, since you lose those 256 PUA characters, even that is IMO better than pushing out bytes to every API, or worse, giving up and just having python unable to access files, as it is now. See lines 3074: QString::toUtf8() and 3408: QString::fromUtf8()) of http://www.google.com/codesearch?q=+show:o7fNK6SzOYs:NO-Bv-AR2rI:toIOngLf1V8&cs_p=http://ie.archive.ubuntu.com/trolltech/pub/qt/snapshots/qt-x11-opensource-src-4.4.0-snapshot-20070402.tar.bz2&cs_f=qt-x11-opensource-src-4.4.0-snapshot-20070402/src/corelib/tools/qstring.cpp James

On Tue, Oct 7, 2008 at 9:51 AM, James Y Knight wrote:

On Oct 7, 2008, at 3:47 AM, Martin v. Löwis wrote:

...

...

- Having os.getcwdb isn't much use when you can't even run python in the first place when the current directory has "bad" bytes in it.

That's not true: it *is* of much use. Python will live in /usr/bin, which has a nicely-decodable path.

...

Currently Python outputs: Could not find platform independent libraries <prefix> Could not find platform dependent libraries Consider setting $PYTHONHOME to <prefix>[:] Fatal Python error: Py_Initialize: can't initialize sys standard streams ImportError: No module named encodings.utf_8 Aborted

I can't reproduce that. This happens (for me) when Python lives in a directory that has an undecodable path - not when the current directory is undecodable.

Sorry about that: this test was indeed in error: I ran "../python" from an undecodeable current directory, rather than "/full/path/to/python", or putting python on the PATH and running it as "python". The first does not work, but the other more common ways to start it do.

...

...

I'm sure there's even more APIs dealing with pathnames, command line arguments, or environment variables that ought to be able to handle both bytes and strings, that currently don't.

Please, no.

I completely and totally agree with your distate, it's rather gross to allow bytes-or-str for every API that touches anything like filenames/argv/environ. That's why I was pushing for the reversible conversion to str...But if bytes-or-str is the solution that's been chosen for this issue, it ought to either be fully committed to and implemented, or at least fully recognized and documented as a half-baked solution.

Of course, if an reversible encoding into string solution is used instead, none of these things would need special treatment: they would all work already.

FWIW: Qt works fine with undecodeable filenames, and it too uses unicode strings everywhere in its API. I looked into what it does, and found that it uses your (Martin)'s original idea for solving this: it stores undecodeable bytes as characters from 0x10fe00 to 0x10feff (which is valid private-use codespace). While that might not be ideally correct, since you lose those 256 PUA characters, even that is IMO better than pushing out bytes to every API, or worse, giving up and just having python unable to access files, as it is now.

See lines 3074: QString::toUtf8() and 3408: QString::fromUtf8()) of

http://www.google.com/codesearch?q=+show:o7fNK6SzOYs:NO-Bv-AR2rI:toIOngLf1V8&cs_p=http://ie.archive.ubuntu.com/trolltech/pub/qt/snapshots/qt-x11-opensource-src-4.4.0-snapshot-20070402.tar.bz2&cs_f=qt-x11-opensource-src-4.4.0-snapshot-20070402/src/corelib/tools/qstring.cpp

So what does Qt do when given a file name already using those PUA? Looks like they get passed through untouched when decoded, but will get translated into invalid names upon encoding. So you still have file names you can't open, and you're incompatible with what other libraries do. The only thing going for Qt is that they seem specifically interested in latin-1, rather than arbitrary bad names. The latin-1 strings that would correspond to the UTF-8 PUA used would include at least one control character, as well as other unusual bits, so it's pretty unlikely to encounter a real latin-1 file name like that. -- Adam Olsen, aka Rhamphoryncus

Hi, First of all, please read my document: http://wiki.python.org/moin/Python3UnicodeDecodeError I moved the document to a public wiki to allow anyone to edit it! Le Tuesday 07 October 2008 05:22:09 James Y Knight, vous avez écrit :

On Oct 6, 2008, at 8:52 PM, Benjamin Peterson wrote:

...

I'm not sure we do. Correct me if I'm wrong, but the "big ticket", issue bytes/unicode filepaths, has been resolved.

Python3 now accepts bytes for os.listdir(), open() (io.open()), os.unlink(), os.path.*(), etc. But it's not enough to say that Python3 can use bytes everywhere. It would take months or *years* to fix all issues related to bytes and unicode. Remember, this task started in 2000 with Python *2.0* (creation of the unicode type).

Well, if you mean that the resolution decided upon is to "simply" allow access to all system APIs using either byte or unicode strings, then it seems to me that there's a rather large amount of work left to do...

If you know a problem, open a ticket and propose a solution. It's not possible to list all new problems since we don't know them yet :-)

- Having os.getcwdb isn't much use when you can't even run python in the first place when the current directory has "bad" bytes in it.

My python3.0 works correctly in a directory with an invalid name. What is your OS / locale / Python version? Please create a ticket if needed.

- I'd think "find . -type f -print0 | xargs -0 python -c 'pass'" ought to work (with files with "bad" bytes being returned by find),

First, fix your home directory :-) There are good tools (convmv?) to fix invalid filenames.

which means that Python shouldn't blow up and refuse to start when there's a non-properly-encoding argv ("Could not convert argument 1 to string" and exiting isn't appropriate behavior)

Why not? It's a good idea to break compatibility to refuse invalid bytes sequences. You can still uses the command line, an input file or a GUI to read raw bytes sequences.

- Of course, just being able to start the interpreter isn't quite enough: you'll want to be able to access that argument list too, somehow (add sys.argvb?).

If we create sys.argvb, what shoul be done if sys.argv creation failed? sys.argv would be empty or unset? Or some values would be removed (and so argv[2] is argv[1])? I think that many (a lot of) programs suppose that sys.argv exists and "is valid". If you introduce a special case (sometimes, sys.argv doesn't exist or is truncated !?), it will introduce new issues.

- There's no os.environb for bytewise access to the environment. Seems important.

It would be strange if you can put a variable in bytes to os.environb whereas os.environ would not get the key. I know two major usages of the environment: (1) read a variable in Python (2) put a variable for a child process (1) can be done with os.getenv() and returns None if the variable (key or value) is an invalid bytes sequence. (2) can be done with subprocess.Popen(). subprocess doesn't support bytes yet but I wrote patches: #4035 and #4036.

- Isn't it a potential security issue that " 'WHATEVER' in os.environ" can return False if WHATEVER had some "bad" bytes in it, but spawning a subprocess actually will include WHATEVER in the subprocess's environment?

Yes. Python should remove the key while creating os.environ.

- Shouldn't this work? subprocess.call(b'/bin/echo')

Yes. Most programs (at least on Linux and Mac) supports bytes and so you should be able use bytes arguments in their command lines, see issues #4035 and #4036.

- I suppose sys.path should handle bytestrings on the path, and should be populated using the bytes-version of os.environ so that PYTHONPATH gets read in properly. Which of course implies that all the importers need to handle byte filenames.

If your file system is broken, rename your directory but don't introduce a special case for sys.path.

- zipfile.ZipFile(b'whatever.zip') doesn't work.

Since zipfile uses bytes in its file structure, zipfile should accept bytes. But the right question is: should this issue block Python3 or can we wait for Python 3.1 (maybe 3.0.1)? -- People wants to try the new Python version! Python3 introduces new amazing features like "keyword only arguments". The bytes/unicode problem is old and only affects broken systems Windows (90% of the computers in the world?) only uses characters for the filenames, environment and command line. Mac and Linux use UTF-8 most of the time, and slowly everything speaks UTF-8! Python3 should not be delayed because of this problem. About the initial barry's question: why Python3 is delayed until december? There are too much open issues? -- Victor Stinner aka haypo http://www.haypocalc.com/blog/

First of all, please read my document: http://wiki.python.org/moin/Python3UnicodeDecodeError

I have problems understanding that document. Is it supposed to be a PEP (i.e. a proposal to enhance Python), or is it a description of the status quo? If it is a PEP, it should clearly separate status quo, specification, and rationale (in any order that you find reasonable). It should also have an "open issues" section, explicitly listing the questions that haven't been resolved, and it should record objections to the proposal. I think I would object to the specification (perhaps to the degree of proposing a counter-PEP), but to do so, I first need a specification to object to. In terms of time-line, I think any such PEP is *clearly* out of scope for Python 3.0. All the remaining issues should deferred to 3.1. That the approach "we can use bytes in the file system API" was so rushed into the code base is already unfortunate, but I can understand the motivation - people want to write backup programs in Python. If I take the text as if it was a specification, here are some of my objections: - Default encoding: a) seems irrelevant for the PEP. The default encoding doesn't nearly have the role anymore that it had in 2.x, and shouldn't have any effect on how file names are treated. b) I would propose that the notion of a default encoding is entirely eliminated from Python, along with sys.(get|set)defaultencoding - argv and environ: are you suggesting that the behavior described in the PEP is desirable? I don't think it is (but I don't think it should change for 3.0, either, only for 3.1) Regards, Martin

On Tue, Oct 7, 2008 at 1:18 PM, Fred Drake wrote:

On Oct 7, 2008, at 4:06 PM, Martin v. Löwis wrote:

...

b) I would propose that the notion of a default encoding is entirely eliminated from Python, along with sys.(get|set)defaultencoding

+1

I expect that the only effect of this change would be that the filesystem encoding would become the de-facto default encoding for other contexts as well. Not that that is necessarily a bad thing... -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Barry Warsaw wrote:

On Oct 6, 2008, at 9:48 PM, Raymond Hettinger wrote:

...

[Barry Warsaw]

...

So, we need to come up with a new release schedule for Python 3.0. My suggestion: 15-Oct-2008 3.0 beta 4 05-Nov-2008 3.0 rc 2 19-Nov-2008 3.0 rc 3 03-Dec-2008 3.0 final Given what still needs to be done, is this a reasonable schedule? Do we need two more betas?

...

Yes to both questions.

I think that's contradictory :). If we need two betas, then 05-Nov becomes beta 5, 19-Nov is rc 2. If we don't need another rc then we can still do a final release on 03-Dec, otherwise we probably go 2 weeks later. I don't want to go much later than that though because then we get into the holiday season.

Do we need the full two weeks between rc's? Or is it too much of a pain to cut releases 3 weeks in a row? E.g. something like: 15-Oct-2008 3.0 beta 4 05-Nov-2008 3.0 beta 5 19-Nov-2008 3.0 rc 2 26-Nov-2008 3.0 rc 3 (if needed) 03-Dec-2008 3.0 final Cheers, Nick. _______________________________________________ Python-3000 mailing list Python-3000@python.org http://mail.python.org/mailman/listinfo/python-3000 Unsubscribe: http://mail.python.org/mailman/options/python-3000/ncoghlan%40gmail.com -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Oct 7, 2008, at 5:47 PM, Nick Coghlan wrote:

Barry Warsaw wrote:

...

On Oct 6, 2008, at 9:48 PM, Raymond Hettinger wrote:

...

[Barry Warsaw]

...

So, we need to come up with a new release schedule for Python 3.0. My suggestion: 15-Oct-2008 3.0 beta 4 05-Nov-2008 3.0 rc 2 19-Nov-2008 3.0 rc 3 03-Dec-2008 3.0 final Given what still needs to be done, is this a reasonable schedule? Do we need two more betas?

...

Yes to both questions.

I think that's contradictory :). If we need two betas, then 05-Nov becomes beta 5, 19-Nov is rc 2. If we don't need another rc then we can still do a final release on 03-Dec, otherwise we probably go 2 weeks later. I don't want to go much later than that though because then we get into the holiday season.

Do we need the full two weeks between rc's? Or is it too much of a pain to cut releases 3 weeks in a row?

E.g. something like:

15-Oct-2008 3.0 beta 4 05-Nov-2008 3.0 beta 5 19-Nov-2008 3.0 rc 2 26-Nov-2008 3.0 rc 3 (if needed) 03-Dec-2008 3.0 final

I won't be able to cut another release between the 15th and 5th, so at least that one should be 2 weeks. If we don't need the additional rc, then we can release early, which would put us just before the US Thanksgiving holiday. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSOvcQ3EjvBPtnXfVAQK5mwP9GQfw3zNvGhJWiSkZ2gQ1LNr0rnmfVmpF WcDePkz3e5nsOjtkwiN0rlYHIQE9ySPfvtqqrInBW8y97y79mTjiM4S32XHLyAsd WEWRb0ClcLuZs+JveAb8KF5pO0RlDgX9Dd6puuPr8kGa5aN/rosfsnXra1GrYpj3 JQghQ89JNkE= =+Ymq -----END PGP SIGNATURE-----

2008/10/6 Raymond Hettinger :

...

15-Oct-2008 3.0 beta 4 05-Nov-2008 3.0 rc 2 19-Nov-2008 3.0 rc 3 03-Dec-2008 3.0 final

Given what still needs to be done, is this a reasonable schedule? Do we need two more betas?

Yes to both questions.

I agree with you here.

I'm seeing that people are just starting to download and play with 3.0. I expect that we'll start getting more feedback on conversion issues, the C API, screwy interactions with operating systems, bytes/text issues, unanticipated interactions with other tools, etc. Each user will stress it in new ways and perhaps reveal a bunch of little integration issues and documentation issues. Those little fixups way go a long way toward establishing a good first impression and reputation for 3.0 from the outset.

And maybe also here, but bounded. I don't want to keep deferring 3.0 months and months, I prefer to have a redesigned schedule now, and stick to it as much as possible, even if the 3.0 version is not as robust as we would want. Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

More specifically, I think 2to3 is shaping up well. pywin32 is taking the approach of "port where possible, but keep in py2x syntax and convert at 'setup.py' time" and this is working out fairly well

I can't say how glad I am that you say that. It supports lib2to3 being a proper library, despite the problems that this may cause in itself.

* Better support for 2to3 in distutils (specifically, the support in build_py is stale, plus 'build_scripts' and 'install_data' should convert .py files to py3k syntax.)

Please do create a bug report for that. It sounds like it's easy to fix.

An 'example' project that uses py2k syntax and "just works" on py3k using this strategy might be useful here.

Perhaps pywin32 :-? I don't think a demo project would do much good, as it doesn't exercise all the issues that may occur.

* A standard 'helper script' that allows people to use py3k to execute a py2x syntax script by auto-converting the code. I've a 10ish-line script that uses lib2to3 plus exec() to achieve that result, but a helper in 2to3 for this would be nice. For a concrete use-case, we want to keep our distutils script in py2x syntax, but execute it via py3k. Its very possible this already exists and I've just missed it...

For the case of setup.py, I was hoping that it could be written in compatible syntax even without needing conversion. That worked fine for my Django port. Is that not the case for pywin32? This specific issue might be out of scope for 3.x, IMO.

Either way, I'm fairly confident a pywin32 build for py3k will be available in the next month or 2 (but as a result, I'm not really in a position to help with the above for that period...)

But please do file bug reports, preferably along with any patches to distutils that you already have. Regards, Martin

at such a script, which I promptly stopped looking at as soon as it worked

Which is quite obvious really given that:

# nuke ourselves from argv del sys.argv[1]

is removing the wrong value! Mark

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Oct 7, 2008, at 4:28 PM, Guido van Rossum wrote:

15-Oct-2008 3.0 rc 2 05-Nov-2008 3.0 rc 3 19-Nov-2008 3.0 rc 4 03-Dec-2008 3.0 final

I've updated PEP 361 and the Google calendar with this schedule, except that the PEP says that rc3 and rc4 are planned "if needed". - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSOvfg3EjvBPtnXfVAQKDfwP/Sz9Ioe1tIrKtvD7JPG2cg2F+wfDJrc+9 vqfh6/eMWiUIOeSKJu6+gye7oXRcHwQXAPivNza3993HesOu0TjudnwXfkAlfsdE m09Rh70AXQQiY7JX46etugRC4BwkuNeBo253cvmfo6hPK0ZhOHZSy3H1LkhvvLA6 Cq56CVqDUgs= =i/Km -----END PGP SIGNATURE-----