Mailman 3 February 2002 - Python-Dev

Order that site-packages is added to sys.path
by barry＠zope.com 13 Feb '02

13 Feb '02

I have a bit of a dilemma when it comes to sys.path and the location of the site-packages directory. The problem comes when someone is using Mailman 2.1 with Python 2.2. The latter comes with the email package, which is in the standard library. Through some contributions, my standalone email package now supports multibyte character sets in RFC-compliant ways (e.g. splitting long headers correctly). The question is, how do I get the updated package to Python 2.2 users? The standalone email package is a simple distutils thingie with a directory and a bunch of .py files. distutils sticks this in site-packages. But an "import email" will always get the standard library version instead of the site-packages version because site.py /appends/ site-packages to sys.path instead of prepending it. I can work around this by adding my own path-hacking code before any import of email.* modules. This is a bit ugly because now it means that the proper functioning of the application depends on import order, and that's nasty. So the question is: why does site.py append site-packages instead of prepending it to sys.path? If there's a valid reason, I don't remember it, and I'm currently blind to any valuable use case. If there's no good reason, it would seem to me that the following use case is better served by prepending: - We want to provide an enhanced version, or a fixed version of a module or package. Distribute it w/distutils and do a normal install. As long as you don't start Python w/ -S, you'll always get the improved version. Don't want the improved version? Start Python w/ -S or just don't ever install the new package. I'm mostly looking for rationale right now, before I try to decide whether it's something worth debating and/or changing. Thanks, -Barry

7 18

Accessing globals without dict lookup
by Guido van Rossum 13 Feb '02

13 Feb '02

Inspired by talks by Jeremy and Skip on DevDay, here's a different idea for speeding up access to globals. It retain semantics but (like Jeremy's proposal) changes the type of a module's __dict__. - Let a cell be a really simple PyObject, containing a PyObject pointer and a cell pointer. Both pointers may be NULL. (It may have to be called PyGlobalCell since I believe there's already a PyCell object.) (Maybe it doesn't even have to be an object -- it could just be a tiny struct.) - Let a celldict be a mapping that is implemented using a dict of cells. When you use its getitem method, the PyObject * in the cell is dereferenced, and if a NULL is found, getitem raises KeyError even if the cell exists. Using setitem to add a new value creates a new cell and stores the value there; using setitem to change the value for an existing key stores the value in the existing cell for that key. There's a separate API to access the cells. - We change the module implementation to use a celldict for its __dict__. The module's getattr and setattr operations now map to getitem and setitem on the celldict. I think the type of <module>.__dict__ and globals() is the only backwards incompatibility. - When a module is initialized, it gets its __builtins__ from the __builtin__ module, which is itself a celldict. For each cell in __builtins__, the new module's __dict__ adds a cell with a NULL PyObject pointer, whose cell pointer points to the corresponding cell of __builtins__. - The compiler generates LOAD_GLOBAL_CELL (and STORE_GLOBAL_CELL etc.) opcodes for references to globals, where is a small index with meaning only within one code object like the const index in LOAD_CONST. The code object has a new tuple, co_globals, giving the names of the globals referenced by the code indexed by . I think no new analysis is required to be able to do this. - When a function object is created from a code object and a celldict, the function object creates an array of cell pointers by asking the celldict for cells corresponding to the names in the code object's co_globals. If the celldict doesn't already have a cell for a particular name, it creates and an empty one. This array of cell pointers is stored on the function object as func_cells. When a function object is created from a regular dict instead of a celldict, func_cells is a NULL pointer. - When the VM executes a LOAD_GLOBAL_CELL instruction, it gets cell number from func_cells. It then looks in the cell's PyObject pointer, and if not NULL, that's the global value. If it is NULL, it follows the cell's cell pointer to the next cell, if it is not NULL, and looks in the PyObject pointer in that cell. If that's also NULL, or if there is no second cell, NameError is raised. (It could follow the chain of cell pointers until a NULL cell pointer is found; but I have no use for this.) Similar for STORE_GLOBAL_CELL , except it doesn't follow the cell pointer chain -- it always stores in the first cell. - There are fallbacks in the VM for the case where the function's globals aren't a celldict, and hence func_cells is NULL. In that case, the code object's co_globals is indexed with to find the name of the corresponding global and this name is used to index the function's globals dict. I believe that's it. I think this faithfully implements the current semantics (where a global can shadow a builtin), but without the need for any dict lookups when accessing globals, except in cases where an explicit dict is passed to exec or eval(). Compare this to Jeremy's scheme using dlicts: http://www.zope.org/Members/jeremy/CurrentAndFutureProjects/FastGlobals - My approach doesn't require global agreement on the numbering of the globals; each code object has its own numbering. This avoids the need for more global analysis, and allows adding code to a module using exec that introduces new globals without having to fall back on a less efficient scheme. - Jeremy's approach might be a teensy bit faster because it may have to do less work in the LOAD_GLOBAL; but I'm not convinced. Here's a implementation sketch for cell and celldict. Note in particular that keys() only returns the keys for which the cell's objptr is not NULL. NULL = object() # used as a token class cell(object): def __init__(self): self.objptr = NULL self.cellptr = NULL class celldict(object): def __init__(self): self.__dict = {} # dict of cells def getcell(self, key): c = self.__dict.get(key) if c is None: c = cell() self.__dict[key] = c return c def __getitem__(self, key): c = self.__dict.get(key) if c is None: raise KeyError, key value = c.objptr if value is NULL: raise KeyError, key else: return value def __setitem__(self, key, value): c = self.__dict.get(key) if c is None: c = cell() self.__dict[key] = c c.objptr = value def __delitem__(self, key): c = self.__dict.get(key) if c is None or c.objptr is NULL: raise KeyError, key c.objptr = NULL def keys(self): return [c.objptr for c in self.__dict.keys() if c.objptr is not NULL] def clear(self): for c in self.__dict.values(): c.objptr = NULL # Etc. --Guido van Rossum (home page: http://www.python.org/~guido/)

12 53

SyntaxError tracebacks in 2.2
by Greg Ward 13 Feb '02

13 Feb '02

Has anyone else noticed that SyntaxError tracebacks no longer include the name of the file where the error occurs? Instead, they just say "<file>". Eg. $ python2.1 foo.py File "foo.py", line 1 foo = ^ SyntaxError: invalid syntax $ python2.2 foo.py File "<string>", line 1 foo = ^ SyntaxError: invalid syntax This is annoying enough that I just filed SF bug #516712: http://sourceforge.net/tracker/index.php?func=detail&aid=516712&group_id=54… Greg -- Greg Ward - Linux nerd gward(a)python.net http://starship.python.net/~gward/ There are no stupid questions -- only stupid people.

3 2

syntactic sugar idea for {static,class}methods
by Michael Hudson 12 Feb '02

12 Feb '02

Some time ago, Gareth McCaughan suggested a syntax for staticmethods. You'd write class C(object): def static(arg) [staticmethod]: return 1 + arg C.static(2) => 3 The way this works is that the above becomes syntactic sugar for roughly: class C(object): def $temp(arg): return 1 + arg static = staticmethod($temp) Anyway, I thought this was a reasonably pythonic idea, so I implemented it, and thought I'd mention it here. Patch at: http://starship.python.net/crew/mwh/hacks/meth-syntax-sugar.diff Some other things that become possible: >>> class D(object): ... def x(self) [property]: ... return "42" ... hello! >>> D().x '42' (the hello! is a debugging printf I haven't taken out yet...) >>> def published(func): ... func.publish = 1 ... return func ... >>> def m() [published]: ... print "hiya!" ... hello! >>> m.publish 1 >>> def hairy_constant() [apply]: ... return math.cos(1 + math.log(34)) ... hello! >>> hairy_constant -0.18495734252481616 >>> def memoize(func): ... cache = {} ... def f(*args): ... try: ... return cache[args] ... except: ... return cache.setdefault(args, func(*args)) ... return f ... >>> def fib(a) [memoize]: ... if a < 2: return 1 ... return fib(a-1) + fib(a-2) ... hello! >>> fib(40) 165580141 # fairly quickly I'm not sure all of these are Good Things (esp. the [apply] one...). OTOH, I think the idea is worth discussion (or squashing by Guido :). Cheers, M. -- For every complex problem, there is a solution that is simple, neat, and wrong. -- H. L. Mencken

6 6

re: syntactic sugar idea for {static,class}methods
by James_Althoff＠i2.com 12 Feb '02

12 Feb '02

I would think that specifying a list (as in [staticmethod]) would be very desirable so that you could do a sequence of transformations, not just one. Michael's examples seem to suggest elements of an Aspect-oriented approach to things. If you have several relevant "Aspect wrappers", then you might want to apply each cascaded in sequence. If, using previous examples, I want a "static" method that is also "memoized" and "SOAPed" I could write: def mymethod(arg) [staticmethod,memoize,webmethod]: or some such combination that is presumably well-defined. Jim

1 0

a different approach to argument parsing
by Russ Cox 12 Feb '02

12 Feb '02

[Hi. I'm responsible for the Plan 9 port of Python; I typically just lurk here.] Regarding the argument parsing discussion, it seems like many of the "features" of the various argument parsing packages are aimed at the fact that in C (whence this all originated) the original getopt interface wasn't so great. To use getopt, you end up specifying the argument set twice: once to the parser and then once when processing the list of returned results. Packages like Optik make this a little better by letting you wrap up the actual processing in some form and hand that to the parser too. Still, you have to wrap up your argument parsing into little actions; the getopt style processing loop is usually a bit clearer. Ultimately, I find getopt unsatisfactory because of the duplication; and I find Optik and other similar packages unsatisfactory because of the contortions you have to go through to invoke them. I don't mean to pick on Optik, since many others appear to behave in similar ways, but it seems to be the yardstick. For concreteness, I'd much rather write: if o=='-n' or o=='--num': ncopies = opt.optarg(opt.needtype(int)) than: parser.add_option("-n", "--num", action="store", type="int", dest="ncopies") The second strikes me as clumsy at best. The Plan 9 argument parser (for C) avoids these problems by making the parser itself small enough to be a collection of preprocessor macros. Although the implementation is ugly, the external interface that programmers see is trivial. A modified version of the example at http://optik.sourceforge.net would be rendered: char *usagemessage = "usage: example [-f FILE] [-h] [-q] who where\n" "\n" " -h show this help message\n" " -f FILE write report to FILE\n" " -q don't print status messages to stdout\n"; void usage(void) { write(2, usagemessage, strlen(usagemessage)); exits("usage"); } void main(int argc, char **argv) { ... ARGBEGIN{ case 'f': report = EARGF(usage()); break; case 'q': verbose = 0; break; case 'h': default: usage(); }ARGEND if(argc != 2) usage(); ... [This is documented at http://plan9.bell-labs.com/magic/man2html/2/ARGBEGIN, for anyone who is curious.] Notice that the argument parsing machinery only gets the argument parameters in one place, and is kept so simple because it is driven by what happens in the actions: if I run "example -frsc" and the f option case doesn't call EARGF() to fetch the "rsc", the next iteration through the loop will be for option 'r'; a priori there's no way to tell. Now that Python has generators, it is easy to do a similar sort of thing, so that the argument parsing can be kept very simple. The running example would be written using the attached argument parser as: usagemessage=\ '''usage: example.py [-h] [-f FILE] [-n N] [-q] who where -h, --help show this help message -f FILE, --file=FILE write report to FILE -n N, --num=N print N copies of the report -q, --quiet don't print status messages to stdout ''' def main(): opt = OptionParser(usage=usagemessage) report = 'default.file' ncopies = 1 verbose = 1 for o in opt: if o=='-f' or o=='--file': report = opt.optarg() elif o=='-n' or o=='--num': ncopies = opt.optarg(opt.typecast(int, 'integer')) elif o=='-q' or o=='--quiet': verbose = 0 else: opt.error('unknown option '+o) if len(opt.args()) != 2: opt.error('incorrect argument count') print 'report=%s, ncopies=%s verbose=%s' % (report, ncopies, verbose) print 'arguments: ', opt.args() It's fairly clear what's going on, and the option parser itself is very simple too. While it may not have all the bells and whistles that some packages do, I think it's simplicity makes most of them irrelevant. It or something like it might be the right approach to take to present a simpler interface. The simplicity of the interface has the benefit that users (potentially anyone who writes a Python program) don't have to learn a lot of stuff to parse their command-line arguments. Suppose I want to write a program with an option that takes two arguments instead of one. Given the Optik-style example it's not at all clear how to do this. Given the above example, there's one obvious thing to try: call opt.optarg() twice. That sort of thing. Addressing the benchmark set by Optik: [1] > * it ties short options and long options together, so once you > define your options you never have to worry about the fact that > -f and --file are the same Here the code does that for you, and if you want to use some other convention, you're not tied to anything. (You do have to tie -f and --file in the usage message too, see answer to [3].) [2] > * it's strongly typed: if you say option --foo expects an int, > then Optik makes sure the user supplied a string that can be > int()'ified, and supplies that int to you There are plenty of ways you could consider adding this. The easiest is what I did in the example. The optarg argument fetcher takes a function to transform the argument before returning. Here, our function calls opt.error() if the argument cannot be converted to an int. The added bells and whistles that Optik adds (choice sets, etc.) can be added in this manner as well, as external functions that the parser doesn't care about, or as internally-supplied helper functions that the user can call if he wants. [3] > * it automatically generates full help based on snippets of > help text you supply with each option This is the one shortcoming: you have to write the usage message yourself. I feel that the benefit of having much clearer argument parsing makes it worth bearing this burden. Also, tools like Optik have to work fairly hard to present the usage message in a reasonable manner, and if it doesn't do what you want you either have to write extension code or just write your own usage message anyway. I'd rather give this up and get the rest of the benefits. [4] > * it has a wide range of "actions" -- ie. what to do with the > value supplied with each option. Eg. you can store that value > in a variable, append it to a list, pass it to an arbitrary > callback function, etc. Here the code provides the widest possible range of actions: you run arbitrary code for each option, and it's all in once place rather than scattered. [5] > * you can add new types and actions by subclassing -- how to > do this is documented and tested The need for new actions is obviated by not having actions at all. The need for new types could be addressed by the argument transformer, although I'm not really happy with that and wouldn't mind seeing it go away. In particular, ncopies = opt.optarg(opt.typecast(int, 'integer')) seems a bit more convoluted and slightly ad hoc compared to the straightforward: try: ncopies = int(opt.optarg()) except ValueError: opt.error(opt.curopt+' requires an integer argument') especially when the requirements get complicated, like the integer has to be prime. Perhaps a hybrid is best, using a collection of standard transformers for the common cases and falling back on actual code for the tough ones. [6] > * it's dead easy to implement simple, straightforward, GNU/POSIX- > style command-line options, but using callbacks you can be as > insanely flexible as you like Here, ditto, except you don't have to use callbacks in order to be as insanely flexible as you like. [7] > * provides lots of mechanism and only a tiny bit of policy (namely, > the --help and (optionally) --version options -- and you can > trash that convention if you're determined to be anti-social) In this version there is very little mechanism (no need for lots), and no policy. It would be easy enough to add the --help and --version hacks as a standard subclass. Anyhow, there it is. I've attached the code for the parser, which I just whipped up tonight. If people think this is a promising thing to explore and someone else wants to take over exploring, great. If yes promising but no takers, I'm willing to keep at it. Russ --- opt.py from __future__ import generators import sys, copy class OptionError(Exception): pass class OptionParser: def __init__(self, argv=sys.argv, usage=None): self.argv0 = argv[0] self.argv = argv[1:] self.usage = usage def __iter__(self): # this assumes the " while self.argv: if self.argv[0]=='-' or self.argv[0][0]!='-': break a = self.argv.pop(0) if a=='--': break if a[0:2]=='--': i = a.find('=') if i==-1: self.curopt = a yield self.curopt self.curopt = None else: self.curarg = a[i+1:] self.curopt = a[0:i] yield self.curopt if self.curarg: # wasn't fetched with optarg self.error(self.curopt+' does not take an argument') self.curopt = None continue self.curarg = a[1:] while self.curarg: a = self.curarg[0:1] self.curarg = self.curarg[1:] self.curopt = '-'+a yield self.curopt self.curopt = None def optarg(self, fn=lambda x:x): if self.curarg: ret = self.curarg self.curarg='' else: try: ret = self.argv.pop(0) except IndexError: self.error(self.curopt+' requires argument') return fn(ret) def _typecast(self, t, x, desc=None): try: return t(x) except ValueError: d = desc if d == None: d = str(t) self.error(self.curopt+' requires '+d+' argument') def typecast(self, t, desc=None): return lambda x: self._typecast(t, x, desc) def args(self): return self.argv def error(self, msg): if self.usage != None: sys.stderr.write('option error: '+msg+'\n\n'+self.usage) sys.stderr.flush() sys.exit(0) else: raise OptionError(), msg ######## import sys usagemessage=\ '''usage: example.py [-h] [-f FILE] [-n N] [-q] who where -h, --help show this help message -f FILE, --file=FILE write report to FILE -n N, --num=N print N copies of the report -q, --quiet don't print status messages to stdout ''' def main(): opt = OptionParser(usage=usagemessage) report = 'default.file' ncopies = 1 verbose = 1 for o in opt: if o=='-f' or o=='--file': report = opt.optarg() elif o=='-n' or o=='--num': ncopies = opt.optarg(opt.typecast(int, 'integer')) elif o=='-q' or o=='--quiet': verbose = 0 else: opt.error('unknown option '+o) if len(opt.args()) != 2: opt.error('incorrect argument count') print 'report=%s, ncopies=%s verbose=%s' % (report, ncopies, verbose) print 'arguments: ', opt.args() if __name__=='__main__': main()

6 7

syntactic sugar idea for {static,class}methods
by M.-A. Lemburg 12 Feb '02

12 Feb '02

1 0

Re: RFC: Option Parsing Libraries
by Jason R. Mastaler 12 Feb '02

12 Feb '02

I haven't specifically looked at Optik, but if another option parser is going to be added to the standard lib, there is one thing I'd like it to have which the getopt module currently doesn't: Support for optional arguments. That is, the ability to specify that an option *may* have an argument, and not just that it either must or can't have an argument. I find this limitation in getopt very frustrating. -- http://tmda.sourceforge.net/

1 0

Re: RFC: Option Parsing Libraries
by David Boddie 12 Feb '02

12 Feb '02

Paul Prescod wrote: > Greg Ward has proposed to add his Optik module to the Python library. > > * http://mail.python.org/pipermail/python-dev/2002-February/019934.html > > Optik is a parser for command line options with the features described > here: > > * http://mail.python.org/pipermail/python-dev/2002-February/019937.html > > If you have a competitive library, or suggestions for changes to Optik, > please forward your comments to python-dev mailing list > (python-dev(a)python.org). If a long-running conversation ensues, you may > need to join the list to participate. Discussions in comp.lang.python > will not be considered unless you at least forward a reference to > python-dev. If you propose a competitor to Optik, please describe how it > compares to the feature list described above. I have been working on a library which I hope begins to unify the presentation of arguments to the programmer with the syntax that is presented to the user. In the form of the list in the first reference above: cmdsyntax.py (Feb 2002) Author: David Boddie <david(a)boddie.net> URL: http://www-solar.mcs.st-and.ac.uk/~davidb/Software/Python/cmdsyntax/ * An attempt at OO design, with limited functionality allowing you to: 1. Set up a syntax object using a syntax definition. 2. Supply arguments from sys.argv or a string and retrieve either: 2a. A dictionary containing values corresponding to the required arguments. 2b. A list of possible matches with the definition. * Arguments passed must conform to a familiar looking syntax definition. For example, the string "infile [-o outfile]" indicates that one argument is necessary and that another may be specified using the -o switch. * Short and long options are allowed. The short options cannot accept arguments as in the example "-ffoo", but lists of options such as "-avx" are supported and accept combinations of these options in any order. Long options support both the "--no-value" and "--name=value" variants. * Command arguments may be specified, which must be matched exactly by user input, as in the case "add"|"remove" value which requires that either the command "add" or "remove" be given with a following argument. * Arguments/options may be grouped using brackets. * No type information is specified, so arguments are not typed before being presented to the programmer. * Excessive method used to match arguments with the syntax definition: all possible definitions are generated then arguments are matched against each one. * No ability to catch remaining unspecified arguments. * No license. * Needs more testing. I'm not proposing this as a competitor to Optik, but I'm happy to donate any ideas and code to the effort. David ________________________________________________________________________ This email has been scanned for all viruses by the MessageLabs SkyScan service. For more information on a proactive anti-virus service working around the clock, around the globe, visit http://www.messagelabs.com ________________________________________________________________________

1 0

Zest
by Ka-Ping Yee 12 Feb '02

12 Feb '02

A few people have expressed interest in the mail-archiving project i mentioned on Developer's Day (see http://lfw.org/python/pydev-sample for some sample output). I've registered a SourceForge project named "Zest" under which i'll be doing this work, and created a mailing list for anyone who wants to discuss it. Please direct any thoughts and comments about Zest to zest-devel(a)lists.sf.net. I don't expect to be blabbing a great deal on the list until i've got a better prototype ready, but it's good to have a place for the project to reside, so that ideas and conversations don't get lost. (Thanks for the prod, Barry.) If you're interested, please join the list: http://lists.sf.net/lists/listinfo/zest-devel Sorry for the diversion. -- ?!ng

1 0