Mailman 3 February 2006 - Python-Dev

The path module PEP
by BJörn Lindqvist 02 Feb '06

02 Feb '06

The last time this was discussed six months ago it seemed like most of python-dev fancied Jason Orendorff's path module. But Guido wanted a PEP and noone created one. So I decided to claim the fame and write one since I also love the path module. :) Much of it is copy-pasted from Peter Astrand's PEP 324, many thanks to him. ################################################# PEP: XXX Title: path - Object oriented handling of filesystem paths Version: $Revision: XXXX $ Last-Modified: $Date: 2006-01-24 19:24:59 +0100 (Sat, 29 Jan 2005) $ Author: Björn Lindqvist <bjourne(a)gmail.com> Status: Dummo Type: Standards Track (library) Created: 24-Jan-2006 Content-Type: text/plain Python-Version: 2.4 Abstract This PEP describes a new class for handling paths in an object oriented fashion. Motivation Dealing with filesystem paths is a common task in any programming language, and very common in a high-level language like Python. Good support for this task is needed, because: - Almost every program uses paths to access files. It makes sense that a task, that is so often performed, should be as intuitive and as easy to perform as possible. - It makes Python an even better replacement language for over-complicated shell scripts. Currently, Python has a large number of different functions scattered over half a dozen modules for handling paths. This makes it hard for newbies and experienced developers to to choose the right method. The Path class provides the following enhancements over the current common practice: - One "unified" object provides all functionality from previous functions. - Subclassability - the Path object can be extended to support paths other than filesystem paths. The programmer does not need to learn a new API, but can reuse his or her knowledge of Path to deal with the extended class. - With all related functionality in one place, the right approach is easier to learn as one does not have to hunt through many different modules for the right functions. - Python is an object oriented language. Just like files, datetimes and sockets are objects so are paths, they are not merely strings to be passed to functions. Path objects are inherently a pythonic idea. - Path takes advantage of properties. Properties make for more readable code. if imgpath.ext == 'jpg': jpegdecode(imgpath) Is better than: if os.path.splitexit(imgpath)[1] == 'jpg': jpegdecode(imgpath) Rationale The following points summarizes the design: - Path extends from string, therefore all code which expects string pathnames need not be modified and no existing code will break. - A Path object can be created either by using the classmethod Path.cwd, by instantiating the class with a string representing a path or by using the default constructor which is equivalent with Path("."). - The factory functions in popen2 have been removed, because I consider the class constructor equally easy to work with. - Path provides for common pathname manipulation, pattern expansion, pattern matching and other high-level file operations including copying. Basically everything path related except for manipulation of file contents which file objects are better suited for. - Platform incompatibilites are dealt with by not instantiating system specific methods. Specification This class defines the following public methods: # Special Python methods. def __new__(cls, init = os.curdir): ... def __repr__(self): ... def __add__(self, more): ... def __radd__(self, other): ... def __div__(self, rel): ... def __truediv__(self, rel): ... # Alternative constructor. def cwd(cls): ... # Operations on path strings. def abspath(sef): ... def normcase(self): ... def normpath(self): ... def realpath(self): ... def expanduser(self): ... def expandvars(self): ... def dirname(self): ... def basename(self): ... def expand(self): ... def splitpath(self): ... def splitdrive(self): ... def splitext(self): ... def stripext(self): ... def splitunc(self): ... [1] def joinpath(self, *args): ... def splitall(self): ... def relpath(self): ... def relpathto(self, dest): ... # Properties about the path. parent, name, namebase, ext, drive, uncshare[1] # Operations that return lists of paths. def listdir(self, pattern = None): ... def dirs(self, pattern = None): ... def files(self, pattern = None): ... def walk(self, pattern = None): ... def walkdirs(self, pattern = None): ... def walkfiles(self, pattern = None): ... def match(self, pattern): def matchcase(self, pattern): def glob(self, pattern): # Methods for retrieving information about the filesystem # path. def exists(self): ... def isabs(self): ... def isdir(self): ... def isfile(self): ... def islink(self): ... def ismount(self): ... def samefile(self, other): ... [1] def getatime(self): ... def getmtime(self): ... def getctime(self): ... def getsize(self): ... def access(self, mode): ... [1] def stat(self): ... def lstat(self): ... def statvfs(self): ... [1] def pathconf(self, name): ... [1] def utime(self, times): ... def chmod(self, mode): ... def chown(self, uid, gid): ... [1] def rename(self, new): ... def renames(self, new): # Filesystem properties for path. atime, getmtime, getctime, size # Methods for manipulating information about the filesystem # path. def utime(self, times): ... def chmod(self, mode): ... def chown(self, uid, gid): ... [1] def rename(self, new): ... def renames(self, new): ... # Create/delete operations on directories def mkdir(self, mode = 0777): ... def makedirs(self, mode = 0777): ... def rmdir(self): ... def removedirs(self): ... # Modifying operations on files def touch(self): ... def remove(self): ... def unlink(self): ... # Modifying operations on links def link(self, newpath): ... def symlink(self, newlink): ... def readlink(self): ... def readlinkabs(self): ... # High-level functions from shutil def copyfile(self, dst): ... def copymode(self, dst): ... def copystat(self, dst): ... def copy(self, dst): ... def copy2(self, dst): ... def copytree(self, dst, symlinks = True): ... def move(self, dst): ... def rmtree(self, ignore_errors = False, onerror = None): ... # Special stuff from os def chroot(self): ... [1] def startfile(self): ... [1] [1] - Method is not availible on all platforms. Replacing older functions with the Path class In this section, "a ==> b" means that b can be used as a replacement for a. In the following examples, we assume that the Path class is imported with "from path import Path". Replacing os.path.join ---------------------- os.path.join(os.getcwd(), "foobar") ==> Path.cwd() / "foobar" Replacing os.path.splitext -------------------------- os.path.splitext("Python2.4.tar.gz")[1] ==> Path("Python2.4.tar.gz").ext Replacing glob.glob ------------------- glob.glob("/lib/*.so") ==> Path("/lib").glob("*.so") Deprecations Introducing this module to the standard library introduces the need to deprecate a number of existing modules and functions. The table below explains which existing functionality that must be deprecated. PATH METHOD DEPRECATES FUNCTION normcase() os.path.normcase() normpath() os.path.normpath() realpath() os.path.realpath() expanduser() os.path.expanduser() expandvars() os.path.expandvars() dirname() os.path.dirname() basename() os.path.basename() splitpath() os.path.split() splitdrive() os.path.splitdrive() splitext() os.path.splitext() splitunc() os.path.splitunc() joinpath() os.path.join() listdir() os.listdir() [fnmatch.filter()] match() fnmatch.fnmatch() matchcase() fnmatch.fnmatchcase() glob() glob.glob() exists() os.path.exists() isabs() os.path.isabs() isdir() os.path.isdir() isfile() os.path.isfile() islink() os.path.islink() ismount() os.path.ismount() samefile() os.path.samefile() getatime() os.path.getatime() getmtime() os.path.getmtime() getsize() os.path.getsize() cwd() os.getcwd() access() os.access() stat() os.stat() lstat() os.lstat() statvfs() os.statvfs() pathconf() os.pathconf() utime() os.utime() chmod() os.chmod() chown() os.chown() rename() os.rename() renames() os.renames() mkdir() os.mkdir() makedirs() os.makedirs() rmdir() os.rmdir() removedirs() os.removedirs() remove() os.remove() unlink() os.unlink() link() os.link() symlink() os.symlink() readlink() os.readlink() chroot() os.chroot() startfile() os.startfile() copyfile() shutil.copyfile() copymode() shutil.copymode() copystat() shutil.copystat() copy() shutil.copy() copy2() shutil.copy2() copytree() shutil.copytree() move() shutil.move() rmtree() shutil.rmtree() The Path class deprecates the whole of os.path, shutil, fnmatch and glob. A big chunk of os is also deprecated. Open Issues Some functionality of Jason Orendorff's path module have been omitted: * Function for opening a path - better handled by the builtin open(). * Functions for reading and writing a whole file - better handled by file objects read() and write() methods. * A chdir() function may be a worthy inclusion. * A deprecation schedule needs to be setup. How much functionality should Path implement? How much of existing functionality should it deprecate and when? * Where should the class be placed and what should it be called? The functions and modules that this new module is trying to replace (os.path, shutil, fnmatch, glob and parts of os are expected to be available in future Python versions for a long time, to preserve backwards compatibility. Reference Implementation Currently, the Path class is implemented as a thin wrapper around the standard library modules sys, os, fnmatch, glob and shutil. The intention of this PEP is to move functionality from the aforementioned modules to Path while they are being deprecated. For more detail, and diverging implementations see: * http://www.jorendorff.com/articles/python/path/path.py * http://svn.python.org/projects/sandbox/trunk/path/path.py * http://cafepy.com/quixote_extras/rex/path/enhpath.py Examples In this section, "a ==> b" means that b can be used as a replacement for a. 1. Make all python files in the a directory executable: DIR = '/usr/home/guido/bin' for f in os.listdir(DIR): if f.endswith('.py'): path = os.path.join(DIR, f) os.chmod(path, 0755) ==> for f in Path('/usr/home/guido/bin'): f.chmod(0755) 2. Delete emacs backup files: def delete_backups(arg, dirname, names): for name in names: if name.endswith('~'): os.remove(os.path.join(dirname, name)) ==> d = Path(os.environ['HOME']) for f in d.walkfiles('*~'): f.remove() 3. Finding the relative path to a file: b = Path('/users/peter/') a = Path('/users/peter/synergy/tiki.txt') a.relpathto(b) 4. Splitting a path into directory and filename: os.path.split("/path/to/foo/bar.txt") ==> Path("/path/to/foo/bar.txt").splitpath() 5. List all Python scripts in the current directory tree: list(Path().walkfiles("*.py")) 6. Create directory paths: os.path.join("foo", "bar", "baz") ==> Path("foo") / "bar" / "baz" Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: -- mvh Björn

27 84

Compiler warnings
by Thomas Wouters 02 Feb '06

02 Feb '06

I noticed a few compiler warnings, when I compile Python on my amd64 with gcc 4.0.3: Objects/longobject.c: In function ‘PyLong_AsDouble’: Objects/longobject.c:655: warning: ‘e’ may be used uninitialized in this function Objects/longobject.c: In function ‘long_true_divide’: Objects/longobject.c:2263: warning: ‘aexp’ may be used uninitialized in this function Objects/longobject.c:2263: warning: ‘bexp’ may be used uninitialized in this function Modules/linuxaudiodev.c: In function ‘lad_obuffree’: Modules/linuxaudiodev.c:392: warning: ‘ssize’ may be used uninitialized in this function Modules/linuxaudiodev.c: In function ‘lad_bufsize’: Modules/linuxaudiodev.c:348: warning: ‘ssize’ may be used uninitialized in this function Modules/linuxaudiodev.c: In function ‘lad_obufcount’: Modules/linuxaudiodev.c:369: warning: ‘ssize’ may be used uninitialized in this function Those are all fairly harmless, just the compiler can't figure out that they are never actually used when they aren't explicitly initialized, because the initialization happens a few functioncalls deeper into the callstack. This one: Python/marshal.c: In function ‘PyMarshal_ReadShortFromFile’: Python/marshal.c:410: warning: ‘rf.end’ is used uninitialized in this function Python/marshal.c:410: warning: ‘rf.ptr’ is used uninitialized in this function (The linenumber is off, it should be 884) is more of the same, except you can't tell from the function that it is a "can never happen" situation: if PyMarshal_ReadShortFromFile() was called with NULL as fp, it would actually use the uninitialized 'end' and 'ptr' struct members. An assert() is probably in place here. Should these warnings be fixed? I know Tim has always argued to fix them, in the past (and I agree,) and it doesn't look like doing so, by initializing the variables, wouldn't be too big a performance hit. I also noticed test_logging is spuriously failing, and not just on my machine (according to buildbot logs.) Is anyone (Vinay?) looking at that yet? -- Thomas Wouters <thomas(a)xs4all.net> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

12 29

webmaster@python.org failing sender verification.
by David Wilson 01 Feb '06

01 Feb '06

Hi there, Recently, updates from MoinMoin have started getting quarantined due to sender verification failing. On investigating the problem, it seems that an assumption about the webmaster mailbox is incorrect: 220 bag.python.org ESMTP Postfix (Debian/GNU) MAIL FROM: <> 503 Error: send HELO/EHLO first HELO argon.maildefence.co.uk 250 bag.python.org MAIL FROM: <> 250 Ok RCPT TO: webmaster(a)python.org 553 invalid bounce (address does not send mail) The MoinMoin instance on Python.org is sending mail as "webmaster(a)python.org". Can somebody take a look? Or at least tell me who to contact. Thanks, David. PS: Please CC me in replies as I am not currently subscribed. -- It's never too late to have a happy childhood.

1 0

Extension to ConfigParser
by Fuzzyman 01 Feb '06

01 Feb '06

Hello all, In the past there has been some discussion about a new module to replace ConfigParser. Most notably at http://wiki.python.org/moin/ConfigParserShootout Specific issues that could be addressed include : * Simpler API * Nested subsections * List values * Storing/converting datatypes * Config file schema * Keeps track of order of values Plus other issues. I'm the (co-)author of ConfigObj - http://www.voidspace.org.uk/python/configobj.html This is a reasonably mature project (now in it's fourth incarnation), and is being used in projects like `Bazaar <http://www.bazaar-ng.org/>`_ and `PlanetPlus <http://planetplus.python-hosting.com/>`_. It occurs to me that the ConfigObj API and syntax is *almost* fully compatible with ConfigParser. It would be possible to extend to the ConfigObj API to be backwards compatible with ConfigParser. This would bring the added benefits of ConfigObj, without needing to add an extra module to the standard library. Well nearly. ConfigObj supports config file schema with (optional) type conversion, through a companion module called validate. This could be included or left as an added option. Anyway. If this stands a *chance* of acceptance, I'll write the PEP (and if accepted, do the work - which is not inconsiderable). Summary of ConfigObj ================ ConfigObj is a Python 2.2 compatible config file parser. It's major feature is simplicity of use. It reads (and writes) INI file like config files and presents the members using a dictionary interface. The order of keys/sections is preserved, and it allows nested subsections to any level : e.g. :: key = value [section] key = value [[sub-section]] key = value It is fully documented with a barrage of doctests. All comments in the config file are also preserved as attributes of the object, and will be written back out. This can be useful for including comments in programatically generated config files. It is integrated with a powerful validation system. Difficulties & Differences ================= A ConfigObj instance is a sub-class of the dictionary datatpe. This means that the ``get`` method of ConfigParser clashes. ConfigObj allows values in the root section (why not ?). ConfigObj doesn't support line continuations (it does all multi-line values through the use of triple quoted strings). ConfigObj currently only allows '=' as a valid divider. Creating ConfigParser (and related classes) compatibility is a big job. Solution ===== All of these problems (if deemed necessary) can be resolved. Either through options, or just by extending the ConfigObj API. I'm happy to put the work in. Comments ? All the best, Fuzzyman http://www.voidspace.org.uk/python/index.shtml

15 32

Python-Dev February 2006