Mailman 3 February 2001 - Python-Dev

editing FAQ?
by Skip Montanaro 27 Feb '01

27 Feb '01

Seems like maybe the FAQ needs some touchup. Is it still under the control of the FAQ wizard (what's the password)? If not, is it in CVS somewhere? Skip

3 6

The more I think about __all__ ...
by Skip Montanaro 27 Feb '01

27 Feb '01

... the more I think I should just yank out all those definitions. I've already been bitten by an incomplete __all__ list. I think the only people who can realistically create them are the authors of the modules. In addition, maintaining them is going to be as difficult as keeping any other piece of documentation up-to-date. Any other thoughts? BDFL - would you care to pronounce? Skip

4 4

first correct explanation wins a beer...
by Martin v. Loewis 27 Feb '01

27 Feb '01

> My guess: Unicode. Try casting to an 8-bit string and see what happens. Paul is right, so I guess you owe him a beer... To see this in more detail, compare popen2.Popen3("/bin/ls").fromchild.readlines() to popen2.Popen3(u"/bin/ls").fromchild.readlines() Specifically, it seems the problem is def _run_child(self, cmd): if type(cmd) == type(''): cmd = ['/bin/sh', '-c', cmd] in popen2. I still think there should be types.isstring function, and then this fragment should read def _run_child(self, cmd): if types.isstring(cmd): cmd = ['/bin/sh', '-c', cmd] Now, if somebody would put "funny characters" into cmd, it would still give an error, which is then almost silently ignored, due to the try: os.execvp(cmd[0], cmd) finally: os._exit(1) fragment. Perhaps it would be better to put if type(cmd) == types.UnicodeType: cmd = cmd.encode("ascii") into Popen3.__init__, so you'd get an error if you pass those funny characters. Regards, Martin

1 0

PEP 236: Back to the __future__
by Tim Peters 27 Feb '01

27 Feb '01

The text of this PEP can also be found online, at: http://python.sourceforge.net/peps/pep-0236.html PEP: 236 Title: Back to the __future__ Version: $Revision: 1.2 $ Author: Tim Peters <tim(a)digicool.com> Python-Version: 2.1 Status: Active Type: Standards Track Created: 26-Feb-2001 Post-History: 26-Feb-2001 Motivation From time to time, Python makes an incompatible change to the advertised semantics of core language constructs, or changes their accidental (implementation-dependent) behavior in some way. While this is never done capriciously, and is always done with the aim of improving the language over the long term, over the short term it's contentious and disrupting. The "Guidelines for Language Evolution" PEP [1] suggests ways to ease the pain, and this PEP introduces some machinery in support of that. The "Statically Nested Scopes" PEP [2] is the first application, and will be used as an example here. Intent [Note: This is policy, and so should eventually move into PEP 5[1]] When an incompatible change to core language syntax or semantics is being made: 1. The release C that introduces the change does not change the syntax or semantics by default. 2. A future release R is identified in which the new syntax or semantics will be enforced. 3. The mechanisms described in the "Warning Framework" PEP [3] are used to generate warnings, whenever possible, about constructs or operations whose meaning may[4] change in release R. 4. The new future_statement (see below) can be explicitly included in a module M to request that the code in module M use the new syntax or semantics in the current release C. So old code continues to work by default, for at least one release, although it may start to generate new warning messages. Migration to the new syntax or semantics can proceed during that time, using the future_statement to make modules containing it act as if the new syntax or semantics were already being enforced. Note that there is no need to involve the future_statement machinery in new features unless they can break existing code; fully backward- compatible additions can-- and should --be introduced without a corresponding future_statement. Syntax A future_statement is simply a from/import statement using the reserved module name __future__: future_statement: "from" "__future__" "import" feature ["as" name] ("," feature ["as" name])* feature: identifier name: identifier In addition, all future_statments must appear near the top of the module. The only lines that can appear before a future_statement are: + The module docstring (if any). + Comments. + Blank lines. + Other future_statements. Example: """This is a module docstring.""" # This is a comment, preceded by a blank line and followed by # a future_statement. from __future__ import nested_scopes from math import sin from __future__ import alabaster_weenoblobs # compile-time error! # That was an error because preceded by a non-future_statement. Semantics A future_statement is recognized and treated specially at compile time: changes to the semantics of core constructs are often implemented by generating different code. It may even be the case that a new feature introduces new incompatible syntax (such as a new reserved word), in which case the compiler may need to parse the module differently. Such decisions cannot be pushed off until runtime. For any given release, the compiler knows which feature names have been defined, and raises a compile-time error if a future_statement contains a feature not known to it[5]. The direct runtime semantics are the same as for any import statement: there is a standard module __future__.py, described later, and it will be imported in the usual way at the time the future_statement is executed. The *interesting* runtime semantics depend on the specific feature(s) "imported" by the future_statement(s) appearing in the module. Note that there is nothing special about the statement: import __future__ [as name] That is not a future_statement; it's an ordinary import statement, with no special semantics or syntax restrictions. Example Consider this code, in file scope.py: x = 42 def f(): x = 666 def g(): print "x is", x g() f() Under 2.0, it prints: x is 42 Nested scopes[2] are being introduced in 2.1. But under 2.1, it still prints x is 42 and also generates a warning. In 2.2, and also in 2.1 *if* "from __future__ import nested_scopes" is included at the top of scope.py, it prints x is 666 Standard Module __future__.py Lib/__future__.py is a real module, and serves three purposes: 1. To avoid confusing existing tools that analyze import statements and expect to find the modules they're importing. 2. To ensure that future_statements run under releases prior to 2.1 at least yield runtime exceptions (the import of __future__ will fail, because there was no module of that name prior to 2.1). 3. To document when incompatible changes were introduced, and when they will be-- or were --made mandatory. This is a form of executable documentation, and can be inspected programatically via importing __future__ and examining its contents. Each statment in __future__.py is of the form: FeatureName = ReleaseInfo ReleaseInfo is a pair of the form: (OptionalRelease, MandatoryRelease) where, normally, OptionalRelease < MandatoryRelease, and both are 5-tuples of the same form as sys.version_info: (PY_MAJOR_VERSION, # the 2 in 2.1.0a3; an int PY_MINOR_VERSION, # the 1; an int PY_MICRO_VERSION, # the 0; an int PY_RELEASE_LEVEL, # "alpha", "beta", "candidate" or "final"; string PY_RELEASE_SERIAL # the 3; an int ) OptionalRelease records the first release in which from __future__ import FeatureName was accepted. In the case of MandatoryReleases that have not yet occurred, MandatoryRelease predicts the release in which the feature will become part of the language. Else MandatoryRelease records when the feature became part of the language; in releases at or after that, modules no longer need from __future__ import FeatureName to use the feature in question, but may continue to use such imports. MandatoryRelease may also be None, meaning that a planned feature got dropped. No line will ever be deleted from __future__.py. Example line: nested_scopes = (2, 1, 0, "beta", 1), (2, 2, 0, "final", 0) This means that from __future__ import nested_scopes will work in all releases at or after 2.1b1, and that nested_scopes are intended to be enforced starting in release 2.2. Unresolved Problems: Runtime Compilation Several Python features can compile code during a module's runtime: 1. The exec statement. 2. The execfile() function. 3. The compile() function. 4. The eval() function. 5. The input() function. Since a module M containing a future_statement naming feature F explicitly requests that the current release act like a future release with respect to F, any code compiled dynamically from text passed to one of these from within M should probably also use the new syntax or semantics associated with F. This isn't always desired, though. For example, doctest.testmod(M) compiles examples taken from strings in M, and those examples should use M's choices, not necessarily doctest module's choices. It's unclear what to do about this. The initial release (2.1b1) is likely to ignore these issues, saying that each dynamic compilation starts over from scratch (i.e., as if no future_statements had been specified). In any case, a future_statement appearing "near the top" (see Syntax above) of text compiled dynamically by an exec, execfile() or compile() applies to the code block generated, but has no further effect on the module that executes such an exec, execfile() or compile(). This can't be used to affect eval() or input(), however, because they only allow expression input, and a future_statement is not an expression. Unresolved Problems: Interactive Shells An interactive shell can be seen as an extreme case of runtime compilation (see above): in effect, each statement typed at an interactive shell prompt runs a new instance of exec, compile() or execfile(). The initial release (2.1b1) is likely to be such that future_statements typed at an interactive shell have no effect beyond their runtime meaning as ordinary import statements. It would make more sense if a future_statement typed at an interactive shell applied to the rest of the shell session's life, as if the future_statement had appeared at the top of a module. Again, it's unclear what to do about this. Questions and Answers Q: What about a "from __past__" version, to get back *old* behavior? A: Outside the scope of this PEP. Seems unlikely to the author, though. Write a PEP if you want to pursue it. Q: What about incompatibilites due to changes in the Python virtual machine? A: Outside the scope of this PEP, although PEP 5[1] suggests a grace period there too, and the future_statement may also have a role to play there. Q: What about incompatibilites due to changes in Python's C API? A: Outside the scope of this PEP. Q: I want to wrap future_statements in try/except blocks, so I can use different code depending on which version of Python I'm running. Why can't I? A: Sorry! try/except is a runtime feature; future_statements are primarily compile-time gimmicks, and your try/except happens long after the compiler is done. That is, by the time you do try/except, the semantics in effect for the module are already a done deal. Since the try/except wouldn't accomplish what it *looks* like it should accomplish, it's simply not allowed. We also want to keep these special statements very easy to find and to recognize. Note that you *can* import __future__ directly, and use the information in it, along with sys.version_info, to figure out where the release you're running under stands in relation to a given feature's status. Q: Going back to the nested_scopes example, what if release 2.2 comes along and I still haven't changed my code? How can I keep the 2.1 behavior then? A: By continuing to use 2.1, and not moving to 2.2 until you do change your code. The purpose of future_statement is to make life easier for people who keep keep current with the latest release in a timely fashion. We don't hate you if you don't, but your problems are much harder to solve, and somebody with those problems will need to write a PEP addressing them. future_statement is aimed at a different audience. Copyright This document has been placed in the public domain. References and Footnotes [1] http://python.sourceforge.net/peps/pep-0005.html [2] http://python.sourceforge.net/peps/pep-0227.html [3] http://python.sourceforge.net/peps/pep-0230.html [4] Note that this is "may" and not "will": better safe than sorry. Of course spurious warnings won't be generated when avoidable with reasonable cost. [5] This ensures that a future_statement run under a release prior to the first one in which a given feature is known (but >= 2.1) will raise a compile-time error rather than silently do a wrong thing. If transported to a release prior to 2.1, a runtime error will be raised because of the failure to import __future__ (no such module existed in the standard distribution before the 2.1 release, and the double underscores make it a reserved name). Local Variables: mode: indented-text indent-tabs-mode: nil End:

1 0

first correct explanation wins a beer...
by Greg Wilson 26 Feb '01

26 Feb '01

...or the caffeinated beverage of your choice, collectable at IPC9. I'm running on a straightforward Linux box: $ uname -a Linux akbar.nevex.com 2.2.16 #3 Mon Aug 14 14:43:46 EDT 2000 i686 unknown with Python 2.1, built fresh from today's repo: $ python Python 2.1a2 (#2, Feb 26 2001, 15:27:11) [GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2 I have one tiny script called "tryout.py": $ cat tryout.py print "We made it!" and a small HTML file called "printing.html": $ cat printing.html <html><body> <pre prog="tryout.py"> We made it! </pre> </body></html> The idea is that my little SAX handler will look for "pre" elements with "prog" attributes, re-run the appropriate script, and compare the output with what's in the HTML page (it's an example for the class). The problem is that "popen2" doesn't work as expected when called from within a SAX content handler, even though it works just fine when called from a method of another class, or on its own. The whole script is: $ cat repy #!/usr/bin/env python import sys from os import popen2 from xml.sax import parse, ContentHandler class JustAClass: def method(self, progName): shellCmd = "python " + progName print "using just a class, shell command is '" + shellCmd + "'" inp, outp = popen2(shellCmd) inp.close() print "using just a class, result is", outp.readlines() class UsingSax(ContentHandler): def startElement(self, name, attrs): if name == "pre": shellCmd = "python " + attrs["prog"] print "using SAX, shell command is '" + shellCmd + "'" inp, outp = popen2(shellCmd) inp.close() print "using SAX, result is", outp.readlines() if __name__ == "__main__": # Run it directly inp, outp = popen2("python tryout.py") inp.close() print "Running popen2 directly, result is", outp.readlines() # Use a plain old class JustAClass().method("tryout.py") # Using SAX input = open("printing.html", 'r') parse(input, UsingSax()) input.close() The output is: $ python repy Running popen2 directly, result is ['We made it!\n'] using just a class, shell command is 'python tryout.py' using just a class, result is ['We made it!\n'] using SAX, shell command is 'python tryout.py' using SAX, result is [] My system has a stock 1.5.2 in /usr/bin/python, but my path is: $ echo $PATH /home/gvwilson/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/sbin:/ho me/gnats/bin so that I get the 2.1 version: $ which python /home/gvwilson/bin/python My PYTHONPATH is set up properly as well (I think): $ echo $PYTHONPATH /home/gvwilson/lib/python2.1:/home/gvwilson/lib/python2.1/lib-dynload I'm using PyXML-0.6.4, built fresh from the .tar.gz source today. So, like I said --- a beer or coffee to the first person who can explain what's up. I'm attaching the Python scripts, the HTML file, and a verbose strace output from my machine. Thanks, Greg <<printing.html>> <<repy>> <<strace.txt>> <<tryout.py>>

2 1

Re: Draft PEP (RE: Other situations like this (was RE: [Python-Dev] Nested scopes resolution -- you can breathe again!))
by Samuele Pedroni 26 Feb '01

26 Feb '01

Hi. I have understood the point about making future feature inheritance automatic ;) So I imagine that the future features should at least end up being visible as a (writeable?) code attribute: co_futures or co_future_features being a list of feature name strings. or I'm wrong? regards, Samuele Pedroni

2 1

Update to PEP 232
by barry＠digicool.com 26 Feb '01

26 Feb '01

After some internal discussions amongst the Pythonlabbers, we've had to make some updates to PEP 232, Function Attributes. Attached is the complete current PEP draft, also available at http://python.sourceforge.net/peps/pep-0232.html The PEP has been moved back to Draft status, but will be Accepted and Finalized for Python 2.1. It will also be propagated forward for Python 2.2 for the next step in implementation. -Barry -------------------- snip snip -------------------- PEP: 232 Title: Function Attributes Version: $Revision: 1.6 $ Author: barry(a)digicool.com (Barry A. Warsaw) Status: Draft Type: Standards Track Created: 02-Dec-2000 Python-Version: 2.1 / 2.2 Post-History: 20-Feb-2001 Introduction This PEP describes an extension to Python, adding attribute dictionaries to functions and methods. This PEP tracks the status and ownership of this feature. It contains a description of the feature and outlines changes necessary to support the feature. This PEP summarizes discussions held in mailing list forums, and provides URLs for further information, where appropriate. The CVS revision history of this file contains the definitive historical record. Background Functions already have a number of attributes, some of which are writable, e.g. func_doc, a.k.a. func.__doc__. func_doc has the interesting property that there is special syntax in function (and method) definitions for implicitly setting the attribute. This convenience has been exploited over and over again, overloading docstrings with additional semantics. For example, John Aycock has written a system where docstrings are used to define parsing rules[1]. Zope's ZPublisher ORB[2] uses docstrings to signal "publishable" methods, i.e. methods that can be called through the web. And Tim Peters has developed a system called doctest[3], where docstrings actually contain unit tests. The problem with this approach is that the overloaded semantics may conflict with each other. For example, if we wanted to add a doctest unit test to a Zope method that should not be publishable through the web. Proposal This proposal adds a new dictionary to function objects, called func_dict (a.k.a. __dict__). This dictionary can be set and get using ordinary attribute set and get syntax. Methods also gain `getter' syntax, and they currently access the attribute through the dictionary of the underlying function object. It is not possible to set attributes on bound or unbound methods, except by doing so explicitly on the underlying function object. See the `Future Directions' discussion below for approaches in subsequent versions of Python. A function object's __dict__ can also be set, but only to a dictionary object (i.e. setting __dict__ to UserDict raises a TypeError). Examples Here are some examples of what you can do with this feature. def a(): pass a.publish = 1 a.unittest = '''...''' if a.publish: print a() if hasattr(a, 'unittest'): testframework.execute(a.unittest) class C: def a(self): 'just a docstring' a.publish = 1 c = C() if c.a.publish: publish(c.a()) Other Uses Paul Prescod enumerated a bunch of other uses: http://mail.python.org/pipermail/python-dev/2000-April/003364.html Future Directions - A previous version of this PEP (and the accompanying implementation) allowed for both setter and getter of attributes on unbound methods, and only getter on bound methods. A number of problems were discovered with this policy. Because method attributes were stored in the underlying function, this caused several potentially surprising results: class C: def a(self): pass c1 = C() c2 = C() c1.a.publish = 1 # c2.a.publish would now be == 1 also! Because a change to `a' bound c1 also caused a change to `a' bound to c2, setting of attributes on bound methods was disallowed. However, even allowing setting of attributes on unbound methods has its ambiguities: class D(C): pass class E(C): pass D.a.publish = 1 # E.a.publish would now be == 1 also! For this reason, the current PEP disallows setting attributes on either bound or unbound methods, but does allow for getting attributes on either -- both return the attribute value on the underlying function object. The proposal for Python 2.2 is to implement setting (bound or unbound) method attributes by setting attributes on the instance or class, using special naming conventions. I.e. class C: def a(self): pass C.a.publish = 1 C.__a_publish__ == 1 # true c = C() c.a.publish = 2 c.__a_publish__ == 2 # true d = C() d.__a_publish__ == 1 # true Here, a lookup on the instance would look to the instance's dictionary first, followed by a lookup on the class's dictionary, and finally a lookup on the function object's dictionary. - Currently, Python supports function attributes only on Python functions (i.e. those that are written in Python, not those that are built-in). Should it be worthwhile, a separate patch can be crafted that will add function attributes to built-ins. - __doc__ is the only function attribute that currently has syntactic support for conveniently setting. It may be worthwhile to eventually enhance the language for supporting easy function attribute setting. Here are some syntaxes suggested by PEP reviewers: def a { 'publish' : 1, 'unittest': '''...''', } (args): # ... def a(args): """The usual docstring.""" {'publish' : 1, 'unittest': '''...''', # etc. } It isn't currently clear if special syntax is necessary or desirable. Dissenting Opinion When this was discussed on the python-dev mailing list in April 2000, a number of dissenting opinions were voiced. For completeness, the discussion thread starts here: http://mail.python.org/pipermail/python-dev/2000-April/003361.html The dissenting arguments appear to fall under the following categories: - no clear purpose (what does it buy you?) - other ways to do it (e.g. mappings as class attributes) - useless until syntactic support is included Countering some of these arguments is the observation that with vanilla Python 2.0, __doc__ can in fact be set to any type of object, so some semblance of writable function attributes are already feasible. But that approach is yet another corruption of __doc__. And while it is of course possible to add mappings to class objects (or in the case of function attributes, to the function's module), it is more difficult and less obvious how to extract the attribute values for inspection. Finally, it may be desirable to add syntactic support, much the same way that __doc__ syntactic support exists. This can be considered separately from the ability to actually set and get function attributes. Reference Implementation The reference implementation is available on SourceForge as a patch against the Python CVS tree (patch #103123). This patch doesn't include the regrtest module and output file. Those are available upon request. http://sourceforge.net/patch/?func=detailpatch&patch_id=103123&group_id=5470 This patch has been applied and will become part of Python 2.1. References [1] Aycock, "Compiling Little Languages in Python", http://www.foretec.com/python/workshops/1998-11/proceedings/papers/aycock-l… [2] http://classic.zope.org:8080/Documentation/Reference/ORB [3] ftp://ftp.python.org/pub/python/contrib-09-Dec-1999/System/doctest.py Copyright This document has been placed in the Public Domain. Local Variables: mode: indented-text indent-tabs-mode: nil End:

3 4

GC and Vladimir's obmalloc
by nas＠arctrix.com 26 Feb '01

26 Feb '01

Executive Summary: obmalloc will allow more efficient GC and we should try hard to get it into 2.1. I've finally spent some time looking at obmalloc and thinking about how to iterate the GC. The advantage would be that objects managed by obmalloc would no longer have to kept in a linked list. That saves both time and memory. I think the right way to do this is to have obmalloc kept track of two separate heaps. One would be for "atomic" objects, the other for "container" objects. I haven't yet figured out how to implement this yet. A lower level malloc interface that takes a heap structure as an argument is an obvious solution. When the GC runs it needs to find container objects. Since obmalloc only deals with blocks of 256 bytes or less, large containers would still have to be stored in a linked list. The rest can be found by searching the obmalloc container heap. Searching the heap is fairly easy. The heap is an array of pointers to pool lists. The only trick is figuring out which parts of the pools are allocated. I think adding the invariant ob_type = NULL means object not allocated is a good solution. That pointer could be set to NULL when the object is deallocated which would also be good for catching bugs. If we pay attention to pool->ref.count we don't even have to set those pointers for a newly allocated pool. Some type of GC locking will probably have to be added (we don't want the collector running when objects are in inconsistent states). I think the GC state (an int for each object) for obmalloc objects should be stored separately. Each pool header could have a pointer to an array of ints. This array could be allocated lazily when the GC runs. The advantages would be better cache behavior and less memory use if GC is disabled. Crude generational collection could be done by doing something like treating the first partially used pool in each size class as generation 0, other partially used pools and the first used pool as generation 1, and all other non-free pools as generation 2. Is the only issue with obmalloc treading? If so, what do we do to resolve this? Neil

5 8

Re: [Python-Dev] GC and Vladimir's obmalloc
by Michael Hudson 26 Feb '01

26 Feb '01

Guido van Rossum <guido(a)digicool.com> writes: > So what about the optional memory profiler, on Jeremy's plate? > > http://sourceforge.net/tracker/index.php?func=detail&aid=401229&group_id=54… > > I'm sure Jeremy would also love it if someone else took care of this > -- he's busy with the future_statement implementation. In a way, I think this is less important. IMO, only people with a fair amount of wizadry are going to want to use this, and telling them to go and get the patch and apply it isn't too much of a stretch (though it would help if it applied cleanly...). OTOH, obmalloc can improve performance (esp. if Neil can do his cool GC optimizations with it), and so it becomes more important to get it into 2.1 (as a prelude to turning it on by default in 2.2, right?). Just my opinion, M. -- This is the fixed point problem again; since all some implementors do is implement the compiler and libraries for compiler writing, the language becomes good at writing compilers and not much else! -- Brian Rogoff, comp.lang.functional

1 0

Re: [Python-Dev] Backwards Incompatibility
by Michael Hudson 24 Feb '01

24 Feb '01

"Eric S. Raymond" <esr(a)thyrsus.com> writes: > Guido van Rossum <guido(a)digicool.com>: > > > > Language theorists love [exec]. > > > > > > Really? I'd have thought language theorists would be the ones > > > who hate it, given all the problems it causes... > > > > Depends on where they're coming from. Or maybe I should have said > > Lisp folks... > > You are *so* right, Guido! :-) I almost commented about this in reply > to Greg's post earlier. > > Crusty old LISP hackers like me tend to be really attached to being > able to (a) lash up S-expressions that happen to be LISP function calls on > the fly, and then (b) hand them to eval. "No separation between code > and data" is one of the central dogmas of our old-time religion. Really? I thought the "no separation between code and data" thing more referred to macros than anything else. Having the full language around at compile time is one of the things that really separates Common Lisp from anything else. I don't think I've ever used #'eval in CL code - it tends to bugger up efficiency even more than the Python version does, for one thing! (eval-when (:compile-toplevel))-ly y'rs M. -- In many ways, it's a dull language, borrowing solid old concepts from many other languages & styles: boring syntax, unsurprising semantics, few automatic coercions, etc etc. But that's one of the things I like about it. -- Tim Peters, 16 Sep 93

2 1