Mailman 3 March 2000 - Python-Dev

Re: [Python-checkins] CVS: python/dist/src/Lib ConfigParser.py,1.16,1.17
by Greg Stein March 7, 2000

March 7, 2000

You may as well remove the entire "vi" concept from ConfigParser. Since "vi" can be *only* a '=' or ':', then you aren't truly checking anything in the "if" statement. Further, "vi" is used nowhere else, so that variable and the corresponding regex group can be nuked altogether. IMO, I'm not sure why the ";" comment form was initially restricted to just one option format in the first place. Cheers, -g On Fri, 3 Mar 2000, Jeremy Hylton wrote: > Update of /projects/cvsroot/python/dist/src/Lib > In directory bitdiddle:/home/jhylton/python/src/Lib > > Modified Files: > ConfigParser.py > Log Message: > allow comments beginning with ; in key: value as well as key = value > > > Index: ConfigParser.py > =================================================================== > RCS file: /projects/cvsroot/python/dist/src/Lib/ConfigParser.py,v > retrieving revision 1.16 > retrieving revision 1.17 > diff -C2 -r1.16 -r1.17 > *** ConfigParser.py 2000/02/28 23:23:55 1.16 > --- ConfigParser.py 2000/03/03 20:43:57 1.17 > *************** > *** 359,363 **** > optname, vi, optval = mo.group('option', 'vi', 'value') > optname = string.lower(optname) > ! if vi == '=' and ';' in optval: > # ';' is a comment delimiter only if it follows > # a spacing character > --- 359,363 ---- > optname, vi, optval = mo.group('option', 'vi', 'value') > optname = string.lower(optname) > ! if vi in ('=', ':') and ';' in optval: > # ';' is a comment delimiter only if it follows > # a spacing character > > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://www.python.org/mailman/listinfo/python-checkins > -- Greg Stein, http://www.lyra.org/

6 11

Unicode character property methods
by M.-A. Lemburg March 7, 2000

March 7, 2000

As you may have noticed, the Unicode objects provide new methods .islower(), .isupper() and .istitle(). Finn Bock mentioned that Java also provides .isdigit() and .isspace(). Question: should Unicode also provide these character property methods: .isdigit(), .isnumeric(), .isdecimal() and .isspace() ? Plus maybe .digit(), .numeric() and .decimal() for the corresponding decoding ? Similar APIs are already available through the unicodedata module, but could easily be moved to the Unicode object (they cause the builtin interpreter to grow a bit in size due to the new mapping tables). BTW, string.atoi et al. are currently not mapped to string methods... should they be ? -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

2 4

Re: [Compiler-sig] Re: example checkers based on compiler package
by Michael Hudson March 7, 2000

March 7, 2000

Moshe Zadka <moshez(a)math.huji.ac.il> writes: > On Mon, 6 Mar 2000, Jeremy Hylton wrote: > > > I think these kinds of warnings are useful, and I'd like to see a more > > general framework for them built are Python abstract syntax originally > > from P2C. Ideally, they would be available as command line tools and > > integrated into GUIs like IDLE in some useful way. > > Yes! Guido already suggested we have a standard API to them. One thing > I suggested was that the abstract API include not only the input (one form > or another of an AST), but the output: so IDE's wouldn't have to parse > strings, but get a warning class. That would be seriously cool. > Something like a: > > An output of a warning can be a subclass of GeneralWarning, and should > implemented the following methods: > > 1. line-no() -- returns an integer > 2. columns() -- returns either a pair of integers, or None > 3. message() -- returns a string containing a message > 4. __str__() -- comes for free if inheriting GeneralWarning, > and formats the warning message. Wouldn't it make sense to include function/class name here too? A checker is likely to now, and it would save reparsing to find it out. [little snip] > > I'm hoping to collect a few more examples of checkers and generalize > > from them to develop a framework for checking for errors and reporting > > them. > > Cool! > Brainstorming: what kind of warnings would people find useful? In > selfnanny, I wanted to include checking for assigment to self, and > checking for "possible use before definition of local variables" sounds > good. Another check could be a CP4E "checking that no two identifiers > differ only by case". I might code up a few if I have the time... Is there stuff in the current Compiler code to do control flow analysis? You'd need that to check for use before definition in meaningful cases, and also if you ever want to do any optimisation... > What I'd really want (but it sounds really hard) is a framework for > partial ASTs: warning people as they write code. I agree (on both points). Cheers, M. -- very few people approach me in real life and insist on proving they are drooling idiots. -- Erik Naggum, comp.lang.lisp

1 0

example checkers based on compiler package
by Jeremy Hylton March 6, 2000

March 6, 2000

There was some discussion on python-dev over the weekend about generating warnings, and Moshe Zadke posted a selfnanny that warned about methods that didn't have self as the first argument. I think these kinds of warnings are useful, and I'd like to see a more general framework for them built are Python abstract syntax originally from P2C. Ideally, they would be available as command line tools and integrated into GUIs like IDLE in some useful way. I've included a couple of quick examples I coded up last night based on the compiler package (recently re-factored) that is resident in python/nondist/src/Compiler. The analysis on the one that checks for name errors is a bit of a mess, but the overall structure seems right. I'm hoping to collect a few more examples of checkers and generalize from them to develop a framework for checking for errors and reporting them. Jeremy ------------ checkself.py ------------ """Check for methods that do not have self as the first argument""" from compiler import parseFile, walk, ast, misc class Warning: def __init__(self, filename, klass, method, lineno, msg): self.filename = filename self.klass = klass self.method = method self.lineno = lineno self.msg = msg _template = "%(filename)s:%(lineno)s %(klass)s.%(method)s: %(msg)s" def __str__(self): return self._template % self.__dict__ class NoArgsWarning(Warning): super_init = Warning.__init__ def __init__(self, filename, klass, method, lineno): self.super_init(filename, klass, method, lineno, "no arguments") class NotSelfWarning(Warning): super_init = Warning.__init__ def __init__(self, filename, klass, method, lineno, argname): self.super_init(filename, klass, method, lineno, "self slot is named %s" % argname) class CheckSelf: def __init__(self, filename): self.filename = filename self.warnings = [] self.scope = misc.Stack() def inClass(self): if self.scope: return isinstance(self.scope.top(), ast.Class) return 0 def visitClass(self, klass): self.scope.push(klass) self.visit(klass.code) self.scope.pop() return 1 def visitFunction(self, func): if self.inClass(): classname = self.scope.top().name if len(func.argnames) == 0: w = NoArgsWarning(self.filename, classname, func.name, func.lineno) self.warnings.append(w) elif func.argnames[0] != "self": w = NotSelfWarning(self.filename, classname, func.name, func.lineno, func.argnames[0]) self.warnings.append(w) self.scope.push(func) self.visit(func.code) self.scope.pop() return 1 def check(filename): global p, check p = parseFile(filename) check = CheckSelf(filename) walk(p, check) for w in check.warnings: print w if __name__ == "__main__": import sys # XXX need to do real arg processing check(sys.argv[1]) ------------ badself.py ------------ def foo(): return 12 class Foo: def __init__(): pass def foo(self, foo): pass def bar(this, that): def baz(this=that): return this return baz def bar(): class Quux: def __init__(self): self.sum = 1 def quam(x, y): self.sum = self.sum + (x * y) return Quux() ------------ checknames.py ------------ """Check for NameErrors""" from compiler import parseFile, walk from compiler.misc import Stack, Set import __builtin__ from UserDict import UserDict class Warning: def __init__(self, filename, funcname, lineno): self.filename = filename self.funcname = funcname self.lineno = lineno def __str__(self): return self._template % self.__dict__ class UndefinedLocal(Warning): super_init = Warning.__init__ def __init__(self, filename, funcname, lineno, name): self.super_init(filename, funcname, lineno) self.name = name _template = "%(filename)s:%(lineno)s %(funcname)s undefined local %(name)s" class NameError(UndefinedLocal): _template = "%(filename)s:%(lineno)s %(funcname)s undefined name %(name)s" class NameSet(UserDict): """Track names and the line numbers where they are referenced""" def __init__(self): self.data = self.names = {} def add(self, name, lineno): l = self.names.get(name, []) l.append(lineno) self.names[name] = l class CheckNames: def __init__(self, filename): self.filename = filename self.warnings = [] self.scope = Stack() self.gUse = NameSet() self.gDef = NameSet() # _locals is the stack of local namespaces # locals is the top of the stack self._locals = Stack() self.lUse = None self.lDef = None self.lGlobals = None # var declared global # holds scope,def,use,global triples for later analysis self.todo = [] def enterNamespace(self, node): ## print node.name self.scope.push(node) self.lUse = use = NameSet() self.lDef = _def = NameSet() self.lGlobals = gbl = NameSet() self._locals.push((use, _def, gbl)) def exitNamespace(self): ## print self.todo.append((self.scope.top(), self.lDef, self.lUse, self.lGlobals)) self.scope.pop() self._locals.pop() if self._locals: self.lUse, self.lDef, self.lGlobals = self._locals.top() else: self.lUse = self.lDef = self.lGlobals = None def warn(self, warning, funcname, lineno, *args): args = (self.filename, funcname, lineno) + args self.warnings.append(apply(warning, args)) def defName(self, name, lineno, local=1): ## print "defName(%s, %s, local=%s)" % (name, lineno, local) if self.lUse is None: self.gDef.add(name, lineno) elif local == 0: self.gDef.add(name, lineno) self.lGlobals.add(name, lineno) else: self.lDef.add(name, lineno) def useName(self, name, lineno, local=1): ## print "useName(%s, %s, local=%s)" % (name, lineno, local) if self.lUse is None: self.gUse.add(name, lineno) elif local == 0: self.gUse.add(name, lineno) self.lUse.add(name, lineno) else: self.lUse.add(name, lineno) def check(self): for s, d, u, g in self.todo: self._check(s, d, u, g, self.gDef) # XXX then check the globals def _check(self, scope, _def, use, gbl, globals): # check for NameError # a name is defined iff it is in def.keys() # a name is global iff it is in gdefs.keys() gdefs = UserDict() gdefs.update(globals) gdefs.update(__builtin__.__dict__) defs = UserDict() defs.update(gdefs) defs.update(_def) errors = Set() for name in use.keys(): if not defs.has_key(name): firstuse = use[name][0] self.warn(NameError, scope.name, firstuse, name) errors.add(name) # check for UndefinedLocalNameError # order == use & def sorted by lineno # elements are lineno, flag, name # flag = 0 if use, flag = 1 if def order = [] for name, lines in use.items(): if gdefs.has_key(name) and not _def.has_key(name): # this is a global ref, we can skip it continue for lineno in lines: order.append(lineno, 0, name) for name, lines in _def.items(): for lineno in lines: order.append(lineno, 1, name) order.sort() # ready contains names that have been defined or warned about ready = Set() for lineno, flag, name in order: if flag == 0: # use if not ready.has_elt(name) and not errors.has_elt(name): self.warn(UndefinedLocal, scope.name, lineno, name) ready.add(name) # don't warn again else: ready.add(name) # below are visitor methods def visitFunction(self, node, noname=0): for expr in node.defaults: self.visit(expr) if not noname: self.defName(node.name, node.lineno) self.enterNamespace(node) for name in node.argnames: self.defName(name, node.lineno) self.visit(node.code) self.exitNamespace() return 1 def visitLambda(self, node): return self.visitFunction(node, noname=1) def visitClass(self, node): for expr in node.bases: self.visit(expr) self.defName(node.name, node.lineno) self.enterNamespace(node) self.visit(node.code) self.exitNamespace() return 1 def visitName(self, node): self.useName(node.name, node.lineno) def visitGlobal(self, node): for name in node.names: self.defName(name, node.lineno, local=0) def visitImport(self, node): for name in node.names: self.defName(name, node.lineno) visitFrom = visitImport def visitAssName(self, node): self.defName(node.name, node.lineno) def check(filename): global p, checker p = parseFile(filename) checker = CheckNames(filename) walk(p, checker) checker.check() for w in checker.warnings: print w if __name__ == "__main__": import sys # XXX need to do real arg processing check(sys.argv[1]) ------------ badnames.py ------------ # XXX can we detect race conditions on accesses to global variables? # probably can (conservatively) by noting variables _created_ by # global decls in funcs import string import time def foo(x): return x + y def foo2(x): return x + z a = 4 def foo3(x): a, b = x, a def bar(x): z = x global z def bar2(x): f = string.strip a = f(x) import string return string.lower(a) def baz(x, y): return x + y + z def outer(x): def inner(y): return x + y return inner

2 1

PyBufferProcs
by Fred L. Drake, Jr. March 6, 2000

March 6, 2000

While working on the documentation, I've noticed a naming inconsistency regarding PyBufferProcs; it's peers are all named Py*Methods (PySequenceMethods, PyNumberMethods, etc.). I'd like to propose that a synonym, PyBufferMethods, be made for PyBufferProcs, and use that in the core implementations and the documentation. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives

2 1

Re: [Patches] Reference cycle collection for Python
by Guido van Rossum March 6, 2000

March 6, 2000

[I don't like to cross-post to patches and python-dev, but I think this belongs in patches because it's a followup to Neil's post there and also in -dev because of its longer-term importance.] Thanks for the new patches, Neil! We had a visitor here at CNRI today, Eric Tiedemann <est(a)hyperreal.org>, who had a look at your patches before. Eric knows his way around the Scheme, Lisp and GC literature, and presented a variant on your approach which takes the bite out of the recursive passes. Eric had commented earlier on Neil's previous code, and I had used the morning to make myself familiar with Neil's code. This was relatively easy because Neil's code is very clear. Today, Eric proposed to do away with Neil's hash table altogether -- as long as we're wasting memory, we might as well add 3 fields to each container object rather than allocating the same amount in a separate hash table. Eric expects that this will run faster, although this obviously needs to be tried. Container types are: dict, list, tuple, class, instance; plus potentially user-defined container types such as kjbuckets. I have a feeling that function objects should also be considered container types, because of the cycle involving globals. Eric's algorithm, then, consists of the following parts. Each container object has three new fields: gc_next, gc_prev, and gc_refs. (Eric calls the gc_refs "refcount-zero".) We color objects white (initial), gray (root), black (scanned root). (The terms are explained later; we believe we don't actually need bits in the objects to store the color; see later.) All container objects are chained together in a doubly-linked list -- this is the same as Neil's code except Neil does it only for dicts. (Eric postulates that you need a list header.) When GC is activated, all objects are colored white; we make a pass over the entire list and set gc_refs equal to the refcount for each object. Next, we make another pass over the list to collect the internal references. Internal references are (just like in Neil's version) references from other container types. In Neil's version, this was recursive; in Eric's version, we don't need recursion, since the list already contains all containers. So we simple visit the containers in the list in turn, and for each one we go over all the objects it references and subtract one from *its* gc_refs field. (Eric left out the little detail that we ened to be able to distinguish between container and non-container objects amongst those references; this can be a flag bit in the type field.) Now, similar to Neil's version, all objects for which gc_refs == 0 have only internal references, and are potential garbage; all objects for which gc_refs > 0 are "roots". These have references to them from other places, e.g. from globals or stack frames in the Python virtual machine. We now start a second list, to which we will move all roots. The way to do this is to go over the first list again and to move each object that has gc_refs > 0 to the second list. Objects placed on the second list in this phase are considered colored gray (roots). Of course, some roots will reference some non-roots, which keeps those non-roots alive. We now make a pass over the second list, where for each object on the second list, we look at every object it references. If a referenced object is a container and is still in the first list (colored white) we *append* it to the second list (colored gray). Because we append, objects thus added to the second list will eventually be considered by this same pass; when we stop finding objects that sre still white, we stop appending to the second list, and we will eventually terminate this pass. Conceptually, objects on the second list that have been scanned in this pass are colored black (scanned root); but there is no need to to actually make the distinction. (How do we know whether an object pointed to is white (in the first list) or gray or black (in the second)? We could use an extra bitfield, but that's a waste of space. Better: we could set gc_refs to a magic value (e.g. 0xffffffff) when we move the object to the second list. During the meeting, I proposed to set the back pointer to NULL; that might work too but I think the gc_refs field is more elegant. We could even just test for a non-zero gc_refs field; the roots moved to the second list initially all have a non-zero gc_refs field already, and for the objects with a zero gc_refs field we could indeed set it to something arbitrary.) Once we reach the end of the second list, all objects still left in the first list are garbage. We can destroy them in a similar to the way Neil does this in his code. Neil calls PyDict_Clear on the dictionaries, and ignores the rest. Under Neils assumption that all cycles (that he detects) involve dictionaries, that is sufficient. In our case, we may need a type-specific "clear" function for containers in the type object. We discussed more things, but not as thoroughly. Eric & Eric stressed the importance of making excellent statistics available about the rate of garbage collection -- probably as data structures that Python code can read rather than debugging print statements. Eric T also sketched an incremental version of the algorithm, usable for real-time applications. This involved keeping the gc_refs field ("external" reference counts) up-to-date at all times, which would require two different versions of the INCREF/DECREF macros: one for adding/deleting a reference from a container, and another for adding/deleting a root reference. Also, a 4th color (red) was added, to distinguish between scanned roots and scanned non-roots. We decided not to work this out in more detail because the overhead cost appeared to be much higher than for the previous algorithm; instead, we recommed that for real-time requirements the whole GC is disabled (there should be run-time controls for this, not just compile-time). We also briefly discussed possibilities for generational schemes. The general opinion was that we should first implement and test the algorithm as sketched above, and then changes or extensions could be made. I was pleasantly surprised to find Neil's code in my inbox when we came out of the meeting; I think it would be worthwhile to compare and contrast the two approaches. (Hm, maybe there's a paper in it?) The rest of the afternoon was spent discussing continuations, coroutines and generators, and the fundamental reason why continuations are so hard (the C stack getting in the way everywhere). But that's a topic for another mail, maybe. --Guido van Rossum (home page: http://www.python.org/~guido/)

9 20

return statements in lambda
by Fredrik Lundh March 6, 2000

March 6, 2000

from "Python for Lisp Programmers": http://www.norvig.com/python-lisp.html > Don't forget return. Writing def twice(x): x+x is tempting > and doesn't signal a warning or > ception, but you probably > meant to have a return in there. This is particularly irksome > because in a lambda you are prohibited from writing return, > but the semantics is to do the return. maybe adding an (optional but encouraged) "return" to lambda would be an improvement? lambda x: x + 10 vs. lambda x: return x + 10 or is this just more confusing... opinions? </F>

2 1

Design question: call __del__ only after successful __init__?
by Guido van Rossum March 6, 2000

March 6, 2000

I was looking at the code that invokes __del__, with the intent to implement a feature from Java: in Java, a finalizer is only called once per object, even if calling it makes the object live longer. To implement this, we need a flag in each instance that means "__del__ was called". I opened the creation code for instances, looking for the right place to set the flag. I then realized that it might be smart, now that we have this flag anyway, to set it to "true" during initialization. There are a number of exits from the initialization where the object is created but not fully initialized, where the new object is DECREF'ed and NULL is returned. When such an exit is taken, __del__ is called on an incompletely initialized object! Example: >>> class C: def __del__(self): print "deleting", self >>> x = C(1) !--> deleting <__main__.C instance at 1686d8> Traceback (innermost last): File "<stdin>", line 1, in ? TypeError: this constructor takes no arguments >>> Now I have a choice to make. If the class has an __init__, should I clear the flag only after __init__ succeeds? This means that if __init__ raises an exception, __del__ is never called. This is an incompatibility. It's possible that someone has written code that relies on __del__ being called even when __init__ fails halfway, and then their code would break. But it is just as likely that calling __del__ on a partially uninitialized object is a bad mistake, and I am doing all these cases a favor by not calling __del__ when __init__ failed! Any opinions? If nobody speaks up, I'll make the change. --Guido van Rossum (home page: http://www.python.org/~guido/)

8 15

Re: Python-checkins digest, Vol 1 #370 - 8 msgs
by M.-A. Lemburg March 6, 2000

March 6, 2000

> Message: 7 > Date: Thu, 24 Feb 2000 10:23:05 -0500 (EST) > From: Guido van Rossum <guido(a)cnri.reston.va.us> > To: python-checkins(a)python.org > Subject: [Python-checkins] CVS: python/dist/src/Objects listobject.c,2.63,2.64 > > Update of /projects/cvsroot/python/dist/src/Objects > In directory eric:/projects/python/develop/guido/src/Objects > > Modified Files: > listobject.c > Log Message: > Made all list methods use PyArg_ParseTuple(), for more accurate > diagnostics. > > *** INCOMPATIBLE CHANGE: This changes append(), remove(), index(), and > *** count() to require exactly one argument -- previously, multiple > *** arguments were silently assumed to be a tuple. Not sure about remove(), index() and count(), but the change to .append() will break *lots* of code ! -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

15 60

Unicode mapping tables
by M.-A. Lemburg March 5, 2000

March 5, 2000

I am just coding the translate method for Unicode objects and have come along a design question that may have some importance with resp. to speed and memory allocation size. Currently, mapping tables map characters to Unicode characters and vice-versa. Now the .translate method will use a different kind of table: mapping integer ordinals to integer ordinals. Question: What is more of efficient: having lots of integers in a dictionary or lots of characters ? Another aspect of this question is: the translate method will be able to handle sequences *and* mappings because it looks up integers which can be interpreted as indexes as well as dictionary keys. The character mapping codec uses characters as key and thus only allows dictionaries to be used (the reason is that in some future version it should be possible to map single characters to multiple characters or even combinations to bnew combinations). BTW, I dropped the deletions argument from the translate method: it is not needed, since a mapping to None will have the same effect. Note that not specifying a mapping causes the characters to be copied as-is. This has the nice side-effect of grealty reducing the mapping table's size. Note that there will be no .maketrans() method. The same functionality can easily be coded in Python if needed and doesn't fit into the OO-style nature of string and Unicode objects anymore. -- Something else that changed is the way .capitalize() works. The Unicode version uses the Unicode algorithm for it (see TechRep. 13 on the www.unicode.org site). Here's the new doc string: S.capitalize() -> unicode Return a capitalized version of S, i.e. words start with title case characters, all remaining cased characters have lower case. Note that *all* characters are touched, not just the first one. The change was needed to get it in sync with the .iscapitalized() method which is based on the Unicode algorithm too. Should this change be propogated to the string implementation ? -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

3 8