Mailman 3 October 2008 - Python-Dev

Python security team
by Victor Stinner Oct. 1, 2008

Oct. 1, 2008

Hi, I would like to know if a Python security team does exist. I sent an email about an imageop issue, and I didn't get any answer. Later I learned that a security ticket was created, I don't have access to it. First, I would like to access to these informations. Not only this issue, but all security related issues. I have some knowledges about security and I can help to resolve issues and/or estimate the criticity of an issue. Second, I would like to help to fix all Python security … [View More]

10 14

Determine minimum required version for a script
by techtonik Oct. 1, 2008

Oct. 1, 2008

Can somebody remind how to check script compatibility with old Python versions? I can remember PHP_CompatInfo class for PHP that parses a script or directory to find out the minimum version and extensions required for them to run, and I wonder if there was anything like this for Python? -- --anatoly t.

2 1

Re: [Python-Dev] Patch for an initial support of bytes filename in Python3
by glyph＠divmod.com Oct. 1, 2008

Oct. 1, 2008

On 12:47 am, victor.stinner(a)haypocalc.com wrote: This is the most sane contribution I've seen so far :). >See attached patch: python3_bytes_filename.patch > >Using the patch, you will get: >- open() support bytes >- listdir(unicode) -> only unicode, *skip* invalid filenames > (as asked by Guido) Forgive me for being a bit dense, but I couldn't find this hunk in the patch. Do I understand properly that (listdir(bytes) -> bytes)? If so, this seems basically sane … [View More]

5 14

Python 2.6 final today
by Barry Warsaw Oct. 1, 2008

Oct. 1, 2008

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I've been out of town since Friday, but I don't yet see anything in the 700 billion email messages I'm now catching up on that leads me to think we need to delay the release. Yay! I will be on irc later today and will be trolling through the tracker and buildbots soon. Don't trust email to get an important issue in front of me today, please use irc or submit a showstopper bug against 2.6 if something /must/ be addressed before today's … [View More]

1 0

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by "Martin v. Löwis" Oct. 1, 2008

Oct. 1, 2008

>> On Windows, we might reject bytes filenames for all file operations: open(), >> unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError) > > Since I've seen no objections to this yet: please no. If we offer a > "lower-level" bytes filename API, it should work for all platforms. Unfortunately, it can't. You cannot represent all possible file names in a byte string in Windows (just as you can't do so in a Unicode string on Unix). So using byte strings on … [View More]

6 8

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Guido van Rossum Oct. 1, 2008

Oct. 1, 2008

On Tue, Sep 30, 2008 at 12:42 PM, Terry Reedy <tjreedy(a)udel.edu> wrote: > Guido van Rossum wrote: >> >> On Tue, Sep 30, 2008 at 11:13 AM, Georg Brandl <g.brandl(a)gmx.net> wrote: >>> >>> Victor Stinner schrieb: >>>> >>>> On Windows, we might reject bytes filenames for all file operations: >>>> open(), >>>> unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError) >>> >>> … [View More]

2 1

Filename as byte string in python 2.6 or 3.0?
by Victor Stinner Oct. 1, 2008

Oct. 1, 2008

Hi, I read that Python 2.6 is planned to Wednesday. One bug is is still open and important for me: Python 2.6/3.0 are unable to use filename as byte strings. http://bugs.python.org/issue3187 The problem =========== On Windows, all filenames are unicode strings (I guess UTF-16-LE), but on UNIX for historical reasons, filenames are byte strings. On Linux, you can expect UTF-8 valid filenames but sometimes (eg. copy from a FAT32 USB key to an ext3 filesystem) you get invalid filename (… [View More]byte string in a different charset than your default filesystem encoding (utf8)). Python functions using filenames ================================ In Python, you have (incomplete list): - filename producer: os.listdir(), os.walk(), glob.glob() - filename manipulation: os.path.*() - access file: open(), os.unlink(), shutil.rmtree() If you give unicode to producer, they return unicode _or_ byte strings (type may change for each filename :-/). Guido proposed to break this behaviour: raise an exception if unicode conversion fails. We may consider an option like "skip invalid". If you give bytes to producer, they only return byte strings. Great. Filename manipulation: in python 2.6/3.0, os.path.*() is not compatible with the type "bytes". So you can not use os.path.join(<your unicode path>, <bytes filename>) *nor* os.path.join(<your bytes path>, <bytes filename>) because os.path.join() (eg. with the posix version) uses path.endswith('/'). Access file: open() rejects the type bytes (it's just a test, open() supports bytes if you remove the test). As I remember, unlink() is compatible with bytes. But rmtree() fails because it uses os.path.join() (even if you give bytes directory, join() fails). Solutions ========= - producer: unicode => *only* unicode // bytes => bytes - manipulation: support both unicode and bytes but avoid (when it's possible) to mix bytes and characters - open(): allow bytes I implemented these solutions as a patch set attached to the issue #3187: * posix_path_bytes.patch: fix posixpath.join() to support bytes * io_byte_filename.patch: open() allows bytes filename * fnmatch_bytes.patch: patch fnmatch.filter() to accept bytes filenames * glob1_bytes.patch: fix glob.glob() to accept invalid directory name Mmmh, there is no patch for stop os.listdir() on invalid filename. Priority ======== I think that the problem is important because it's a regression from 2.5 to 2.6/3.0. Python 2.5 uses bytes filename, so it was possible to open/unlink "invalid" unicode strings (since it's not unicode but bytes). Well, if it's too late for the final versions, this problem should be at least fixed quickly. Test the problem ================ Example to create invalid filenames on Linux: $ mkdir /tmp/test $ cd /tmp/test $ touch $(echo -e "a\xffb") $ mkdir $(echo -e "dir\xffname") $ touch $(echo -e "dir\xffname/file") $ find . ./a?b ./dir?name ./dir?name/file Python 2.5: >>> import os >>> os.listdir('.') ['a\xffb', 'dir\xffname'] >>> open(os.listdir('.')[0]).close() # open file: ok >>> os.unlink(os.listdir('.')[0]) # remove file: ok >>> os.listdir('.') ['dir\xffname'] >>> shutil.rmtree(os.listdir('.')[0]) # remove dir: ok Wrong solutions =============== New type -------- I proposed an ugly type "InvalidFilename" mixing bytes and characters. As everybody using unicode knows, it's a bad idea :-) (and it introduces a new type). Convert bytes to unicode (replace) ---------------------------------- unicode_filename = unicode(bytes_filename, charset, "replace") Ok, you will get valid unicode strings which can be used in os.path.join() & friends, but open() or unlink() will fails because this filename doesn't exist! -- Victor Stinner aka haypo http://www.haypocalc.com/blog/ [View Less]

17 38