Mailman 3 November 2006 - Python-Dev

PEP: Adding data-type objects to Python
by Jim Jewett 03 Nov '06

03 Nov '06

Travis E. Oliphant wrote: > Two packages need to share a chunk of memory (the package authors do not > know each other and only have and Python as a common reference). They > both want to describe that the memory they are sharing has some > underlying binary structure. As a quick sanity check, please tell me where I went off track. it sounds to me like you are assuming that: (1) The memory chunk represents a single object (probably an array of some sort) (2) That subchunks can themselves be described by a (single?) repeating C struct. (3) You can't just use the C header, since you want this at run-time. (4) It would be enough if you could say This is an array of 500 elements that look like struct { int simple; struct nested { char name[30]; char addr[45]; int amount; } (5) But is it not acceptable to use Martin's suggested ctypes equivalent of (building out from the inside): class nested(Structure): _fields_ = [("name", c_char*30), ("addr", c_char*45), ("amount", c_long)] class struct(Structure): _fields_ = [("simple", c_int), ("nested", nested)] struct * 500 If I misunderstood, could you show me where? If I did understand correctly, could you expand on why (5) is unacceptable, given that ctypes is now in the core? (New and unknown, I would understand -- but that is also true of any datatype proposal, for the people who haven't already used it. I suspect that any differences from Numpy would be a source of pain for those who *have* used Numpy, but following Numpy exactly is ... not much simpler than the above.) Or are you just saying that "anything with a buffer interface should also have a datatype object describing the layout in a standard way"? If so, that makes sense, but I'm inclined to prefer the ctypes way, so that most people won't ever have to worry about things like endianness/strides/Fortan layout. -jJ

7 13

Re: [Python-Dev] [Tracker-discuss] Getting Started
by Michael Twomey 02 Nov '06

02 Nov '06

On 11/1/06, Brett Cannon <brett(a)python.org> wrote: > > > > Right. Brett, do we need accounts on python.org for this ? > > Yep. It just requires SSH 2 keys from each of you. You can then email > python-dev with those keys and your first.last name and someone there will > install the keys for you. > I'll need svn access to svn.python.org too for the roundup tracker. My key is over at http://translucentcode.org/mick/ssh_key.txt firstname.lastname: michael.twomey cheers, Michael

1 0

Re: [Python-Dev] Path object design
by Mike Orr 02 Nov '06

02 Nov '06

Argh, it's difficult to respond to one topic that's now spiraling into two conversations on two lists. glyph(a)divmod.com wrote: > On 03:14 am, sluggoster(a)gmail.com wrote: > > >One thing is sure -- we urgently need something better than os.path. > >It functions well but it makes hard-to-read and unpythonic code. > > I'm not so sure. The need is not any more "urgent" today than it was > 5 years ago, when os.path was equally "unpythonic" and unreadable. > The problem is real but there is absolutely no reason to hurry to a > premature solution. Except that people have had to spend five years putting hard-to-read os.path functions in the code, or reinventing the wheel with their own libraries that they're not sure they can trust. I started to use path.py last year when it looked like it was emerging as the basis of a new standard, but yanked it out again when it was clear the API would be different by the time it's accepted. I've gone back to os.path for now until something stable emerges but I really wish I didn't have to. > I've already recommended Twisted's twisted.python.filepath module as a > possible basis for the implementation of this feature.... > *It is already used in a large body of real, working code, and > therefore its limitations are known.* This is an important consideration.However, to me a clean API is more important. Since we haven't agreed on an API there is no widely-used module that implements it... it's a chicken-and-egg problem since it takes significant time to write and test an implementation. So I'd like to start from the standpoint of an ideal API rather than just taking the API of the most widely-used implementation. os.path is clearly the most widely-used implementation, but that doesn't mean that OOizing it as-is would be my favorite choice. I took a quick look at filepath. It looks similar in concept to PEP 355. Four concerns: - unfamiliar method names (createDirectory vs mkdir, child vs join) - basename/dirname/parent are methods rather than properties: leads to () overproliferation in user code. - the "secure" features may not be necessary. If they are, this should be a separate discussion, and perhaps implemented as a subclass. - stylistic objection to verbose camelCase names like createDirectory > Proposals for extending the language are contentious and it is very > difficult to do experimentation with non-trivial projects because > nobody wants to do that and then end up with a bunch of code written > in a language that is no longer supported when the experiment fails. True. > Path representation is a bike shed. Nobody would have proposed > writing an entirely new embedded database engine for Python: python > 2.5 simply included SQLite because its utility was already proven. There's a quantum level of difference between path/file manipulation -- which has long been considered a requirement for any full-featured programming language -- and a database engine which is much more complex. Georg Brandl <g.brandl(a)gmx.net> wrote: > I have been a supporter of the full-blown Path object in the past, but the > recent discussions have convinved me that it is just too big and too confusing, > and that you can't kill too many birds with one stone in this respect. > Most of the ugliness really lies in the path name manipulation functions, which > nicely map to methods on a path name object. Fredrik has convinced me that it's more urgent to OOize the pathname conversions than the filesystem operations. Pathname conversions are the ones that frequently get nested or chained, whereas filesystem operations are usually done at the top level of a program statement, or return a different "kind" of value (stat, true/false, etc). However, it's interesting that all the proposals I've seen in the past three years have been a "monolithic" OO class. Clearly there are a lot of people who prefer this way, or at least have never heard of anything different. Where have all the proponents of non-OO or limited-OO strategies been? The first proposal of that sort I've seen was Nich Cochlan's October 1. Have y'all just been ignoring the monolithic OO efforts without offering any alternatives? Fredrik Lundh <fredrik(a)pythonware.com> wrote: > > This is fully backwards compatible, can go right into 2.6 without > > breaking anything, allows people to update their code as they go, > > and can be incrementally improved in future releases: > > > > 1) Add a pathname wrapper to "os.path", which lets you do basic > > path "algebra". This should probably be a subclass of unicode, > > and should *only* contain operations on names. > > > > 2) Make selected "shutil" operations available via the "os" name- > > space; the old POSIX API vs. POSIX SHELL distinction is pretty > > irrelevant. Also make the os.path predicates available via the > > "os" namespace. > > > > This gives a very simple conceptual model for the user; to manipulate > > path *names*, use "os.path.<op>(string)" functions or the "<path>" > > wrapper. To manipulate *objects* identified by a path, given either as > > a string or a path wrapper, use "os.<op>(path)". This can be taught in > > less than a minute. Making this more concrete, I think Fredrik is suggesting: - Make (os.path) abspath, basename, commonprefix, dirname, expanduser, expandvars, isabs, join, normcase, normpath, split, splitdrive, splitext, splitunc methods of a Path object. - Copy functions into os: (os.path) exists, lexists, get{atime,mtime,ctime,size}, is{file,dir,link,mount}, realpath, samefile, sameopenfile, samestat, (shutil) copy, copy2, copy{file,fileobj,mode,stat,tree}, rmtree, move. - Deprecate the old functions to remove in 3.0. - Abandon os.path.walk because os.walk is better. This is worth considering as a start. It does mean moving a lot of functions that may be moved again at some point in the future. If we do move shutil functions into os, I'd at least like to make some tiny improvements in them. Adding four lines to the beginning of rmtree would make it behave like my purge() function without detracting from its existing use: if not os.exists(p): return if not os.isdir(p): p.remove() Also, do we really need six copy methods? copy2 can be handled by a third argument, etc. -- Mike Orr <sluggoster(a)gmail.com>

1 0

Re: [Python-Dev] [Tracker-discuss] Getting Started
by Stefan Seefeld 02 Nov '06

02 Nov '06

Brett Cannon wrote: > On 11/1/06, Stefan Seefeld <seefeld(a)sympatico.ca> wrote: >> Right. Brett, do we need accounts on python.org for this ? > > > Yep. It just requires SSH 2 keys from each of you. You can then email > python-dev with those keys and your first.last name and someone there will > install the keys for you. My key is at http://www3.sympatico.ca/seefeld/ssh.txt, I'm Stefan Seefeld. Thanks ! Stefan -- ...ich hab' noch einen Koffer in Berlin...

3 3

Path object design
by Jim Jewett 02 Nov '06

02 Nov '06

On 10:06 am, g.brandl at gmx.net wrote: >> What a successor to os.path needs is not security, it's a better (more pythonic, >> if you like) interface to the old functionality. Glyph: > Why? > Rushing ... could exacerbate a very real problem, e.g. > the security and data-integrity implications of idiomatic usage. The proposed Path object (or new path module) is intended to replace os.path. If it can't do the equivalent of "cd ..", then it isn't a replacement; it is just another similar alternative to confuse beginners. If you're saying that a webserver should use a more restricted subclass (or even the existing FilePath alternative), then I agree. I'll even agree that a restricted version would ideally be available out of the box. I don't think it should be the only option. -jJ

1 0

Re: [Python-Dev] Path object design
by glyph＠divmod.com 01 Nov '06

01 Nov '06

On 10:06 am, g.brandl(a)gmx.net wrote: >What a successor to os.path needs is not security, it's a better (more pythonic, >if you like) interface to the old functionality. Why? I assert that it needs a better[1] interface because the current interface can lead to a variety of bugs through idiomatic, apparently correct usage. All the more because many of those bugs are related to critical errors such as security and data integrity. If I felt the current interface did a good job at doing the right thing in the right situation, but was cumbersome to use, I would strenuously object to _any_ work taking place to change it. This is a hard API to get right. [1]: I am rather explicitly avoiding the word "pythonic" here. It seems to have grown into a shibboleth (and its counterpart, "unpythonic", into an expletive). I have the impression it used to mean something a bit more specific, maybe adherence to Tim Peters' "Zen" (although that was certainly vague enough by itself and not always as self-evidently true as some seem to believe). More and more, now, though, I hear it used to mean 'stuff should be more betterer!' and then everyone nods sagely because we know that no filthy *java* programmer wants things to be more betterer; *we* know *they* want everything to be horrible. Words like this are a pet peeve of mine though, so perhaps I am overstating the case. Anyway, moving on... as long as I brought up the Zen, perhaps a particular couplet is appropriate here: Now is better than never. Although never is often better than *right* now. Rushing to a solution to a non-problem, e.g. the "pythonicness" of the interface, could exacerbate a very real problem, e.g. the security and data-integrity implications of idiomatic usage. Granted, it would be hard to do worse than os.path, but it is by no means impossible (just look at any C program!), and I can think of a couple of kinds of API which would initially appear more convenient but actually prove more problematic over time. That brings me back to my original point: the underlying issue here is too important a problem to get wrong *again* on the basis of a superficial "need" for an API that is "better" in some unspecified way. os.path is at least possible to get right if you know what you're doing, which is no mean feat; there are many path-manipulation libraries in many languages which cannot make that claim (especially portably). Its replacement might not be. Getting this wrong outside the standard library might create problems for some people, but making it worse _in_ the standard library could create a total disaster for everyone. I do believe that this wouldn't get past the dev team (least of all the release manager) but it would waste a lot less of everyone's time if we focused the inevitable continuing bike-shed discussion along the lines of discussing the known merits of widely deployed alternative path libraries, or at least an approach to *get* that data on some new code if there is consensus that existing alternatives are in some way inadequate. If for some reason it _is_ deemed necessary to go with an untried approach, I can appreciate the benefits that /F has proposed of trying to base the new interface entirely and explicitly off the old one. At least that way it will still definitely be possible to get right. There are problems with that too, but they are less severe.

3 2

Re: [Python-Dev] Path object design
by glyph＠divmod.com 01 Nov '06

01 Nov '06

On 03:14 am, sluggoster(a)gmail.com wrote: >One thing is sure -- we urgently need something better than os.path. >It functions well but it makes hard-to-read and unpythonic code. I'm not so sure. The need is not any more "urgent" today than it was 5 years ago, when os.path was equally "unpythonic" and unreadable. The problem is real but there is absolutely no reason to hurry to a premature solution. I've already recommended Twisted's twisted.python.filepath module as a possible basis for the implementation of this feature. I'm sorry I don't have the time to pursue that. I'm also sad that nobody else seems to have noticed. Twisted's implemenation has an advantage that it doesn't seem that these new proposals do, an advantage I would really like to see in whatever gets seriously considered for adoption: *It is already used in a large body of real, working code, and therefore its limitations are known.* If I'm wrong about this, and I can't claim to really know about the relative levels of usage of all of these various projects when they're not mentioned, please cite actual experiences using them vs. using os.path. Proposals for extending the language are contentious and it is very difficult to do experimentation with non-trivial projects because nobody wants to do that and then end up with a bunch of code written in a language that is no longer supported when the experiment fails. I understand, therefore, that language-change proposals are therefore going to be very contentious no matter what. However, there is no reason that library changes need to follow this same path. It is perfectly feasible to write a library, develop some substantial applications with it, tweak it based on that experience, and *THEN* propose it for inclusion in the standard library. Users of the library can happily continue using the library, whether it is accepted or not, and users of the language and standard library get a new feature for free. For example, I plan to continue using FilePath regardless of the outcome of this discussion, although perhaps some conversion methods or adapters will be in order if a new path object makes it into the standard library. I specifically say "library" and not "recipie". This is not a useful exercise if every user of the library has a subtly incompatible and manually tweaked version for their particular application. Path representation is a bike shed. Nobody would have proposed writing an entirely new embedded database engine for Python: python 2.5 simply included SQLite because its utility was already proven. I also believe it is important to get this issue right. It might be a bike shed, but it's a *very important* bike shed. Google for "web server url filesystem path vulnerability" and you'll see what I mean. Getting it wrong (or passing strings around everywhere) means potential security gotchas lurking around every corner. Even Twisted, with no C code at all, got its only known arbitrary-code-execution vulnerability from a path manipulation bug. That was even after we'd switched to an OO path-manipulation layer specifically to avoid bugs like this! I am not addressing this message to the py3k list because its general message of extreme conservatism on new features is more applicable to python-dev. However, py3k designers might also take note: if py3k is going to do something in this area and drop support for the "legacy" os.path, it would be good to choose something that is known to work and have few gotchas, rather than just choosing the devil we don't know over the devil we do. The weaknesses of os.path are at least well-understood.

5 5

RELEASED Python 2.3.6, FINAL
by Anthony Baxter 01 Nov '06

01 Nov '06

On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.3.6 (FINAL). Python 2.3.6 is a security bug-fix release. While Python 2.5 is the latest version of Python, we're making this release for people who are still running Python 2.3. Unlike the recently released 2.4.4, this release only contains a small handful of security-related bugfixes. See the website for more. * Python 2.3.6 contains a fix for PSF-2006-001, a buffer overrun * in repr() of unicode strings in wide unicode (UCS-4) builds. * See http://www.python.org/news/security/PSF-2006-001/ for more. This is a **source only** release. The Windows and Mac binaries of 2.3.5 were built with UCS-2 unicode, and are therefore not vulnerable to the problem outlined in PSF-2006-001. The PCRE fix is for a long-deprecated module (you should use the 're' module instead) and the email fix can be obtained by downloading the standalone version of the email package. Most vendors who ship Python should have already released a patched version of 2.3.5 with the above fixes, this release is for people who need or want to build their own release, but don't want to mess around with patch or svn. There have been no changes (apart from the version number) since the release candidate of 2.3.6. Python 2.3.6 will complete python.org's response to PSF-2006-001. If you're still on Python 2.2 for some reason and need to work with UCS-4 unicode strings, please obtain the patch from the PSF-2006-001 security advisory page. Python 2.4.4 and Python 2.5 have both already been released and contain the fix for this security problem. For more information on Python 2.3.6, including download links for source archives, release notes, and known issues, please see: http://www.python.org/2.3.6 Highlights of this new release include: - A fix for PSF-2006-001, a bug in repr() for unicode strings on UCS-4 (wide unicode) builds. - Two other, less critical, security fixes. Enjoy this release, Anthony Anthony Baxter anthony(a)python.org Python Release Manager (on behalf of the entire python-dev team)

1 0

patch 1462525 or similar solution?
by Paul Jimenez 01 Nov '06

01 Nov '06

I submitted patch 1462525 awhile back to solve the problem described even longer ago in http://mail.python.org/pipermail/python-dev/2005-November/058301.html and I'm wondering what my appropriate next steps are. Honestly, I don't care if you take my patch or someone else's proposed solution, but I'd like to see something go into the stdlib so that I can eventually stop having to ship custom code for what is really a standard problem. --pj

3 2

Path object design
by Mike Orr 01 Nov '06

01 Nov '06

I just saw the Path object thread ("PEP 355 status", Sept-Oct), saying that the first object-oriented proposal was rejected. I'm in favor of the "directory tuple" approach which wasn't mentioned in the thread. This was proposed by Noal Raphael several months ago: a Path object that's a sequence of components (a la os.path.split) rather than a string. The beauty of this approach is that slicing and joining are expressed naturally using the [] and + operators, eliminating several methods. Introduction: http://wiki.python.org/moin/AlternativePathClass Feature discussion: http://wiki.python.org/moin/AlternativePathDiscussion Reference implementation: http://wiki.python.org/moin/AlternativePathModule (There's a link to the introduction at the end of PEP 355.) Right now I'm working on a test suite, then I want to add the features marked "Mike" in the discussion -- in a way that people can compare the feature alternatives in real code -- and write a PEP. But it's a big job for one person, and there are unresolved issues on the discussion page, not to mention things brought up in the "PEP 355 status" thread. We had three people working on the discussion page but development seems to have ground to a halt. One thing is sure -- we urgently need something better than os.path. It functions well but it makes hard-to-read and unpythonic code. For instance, I have an application that has to add its libraries to the Python path, relative to the executable's location. /toplevel app1/ bin/ main_progam.py utility1.py init_app.py lib/ app_module.py shared/ lib/ shared_module.py The solution I've found is an init_app module in every application that sets up the paths. Conceptually it needs "../lib" and "../../shared/lib", but I want the absolute paths without hardcoding them, in a platform-neutral way. With os.path, "../lib" is: os.path.join(os.path.dirname(os.path.dirname(__FILE__)), "lib") YUK! Compare to PEP 355: Path(__FILE__).parent.parent.join("lib") Much easier to read and debug. Under Noam's proposal it would be: Path(__FILE__)[:-2] + "lib" I'd also like to see the methods more intelligent: don't raise an error if an operation is already done (e.g., a directory exists or a file is already removed). There's no reason to clutter one's code with extra if's when the methods can easily encapsulate this. This was considered a too radical departure from os.path for some, but I have in mind even more radical convenience methods which I'd put in a third-party subclass if they're not accepted into the standard library, the way 'datetime' has third-party subclasses. In my application I started using Orendorff's path module, expecting the standard path object would be close to it. When PEP 355 started getting more changes and the directory-based alternative took off, I took path.py out and rewrote my code for os.path until an alternative becomes more stable. Now it looks like it will be several months and possibly several third-party packages until one makes it into the standard library. This is unfortunate. Not only does it mean ugly code in applications, but it means packages can't accept or return Path objects and expect them to be compatible with other packages. The reasons PEP 355 was rejected also sound strange. Nick Coghlan wrote (Oct 1): > Things the PEP 355 path object lumps together: > - string manipulation operations > - abstract path manipulation operations (work for non-existent filesystems) > - read-only traversal of a concrete filesystem (dir, stat, glob, etc) > - addition & removal of files/directories/links within a concrete filesystem > Dumping all of these into a single class is certainly practical from a utility > point of view, but it's about as far away from beautiful as you can get, which > creates problems from a learnability point of view, and from a > capability-based security point of view. What about the convenience of the users and the beauty of users' code? That's what matters to me. And I consider one class *easier* to learn. I'm tired of memorizing that 'split' is in os.path while 'remove' and 'stat' are in os. This seems arbitrary: you're statting a path, aren't you? Also, if you have four classes (abstract path, file, directory, symlink), *each* of those will have 3+ platform-specific versions. Then if you want to make an enhancement subclass you'll have to make 12 of them, one for each of the 3*4 combinations of superclasses. Encapsulation can help with this, but it strays from the two-line convenience for the user: from path import Path p = Path("ABC") # Works the same for files/directories on any platform. Nevertheless, I'm open to seeing a multi-class API, though hopefully less verbose than Talin's preliminary one (Oct 26). Is it necessary to support path.parent(), pathobj.parent(), io.dir.listdir(), *and* io.dir.Directory(). That's four different namespaces to memorize which function/method is where, and if a function/method belongs to multiple ones it'll be duplicated, and you'll have to remember that some methods are duplicated and others aren't... Plus, constructors like io.dir.Directory() look too verbose. io.Directory() might be acceptable, with the functions as class methods. I agree that supporting non-filesystem directories (zip files, CSV/Subversion sandboxes, URLs) would be nice, but we already have a big enough project without that. What constraints should a Path object keep in mind in order to be forward-compatible with this? If anyone has design ideas/concerns about a new Path class(es), please post them. If anyone would like to work on a directory-based spec/implementation, please email me. -- Mike Orr <sluggoster(a)gmail.com>

4 3