Mailman 3 August 2004 - Python-Dev

Re: [Python-Dev] Minimal 'stackless' PEP using generators?
by Phillip J. Eby Aug. 23, 2004

Aug. 23, 2004

At 03:41 PM 8/23/04 -0400, Clark C. Evans wrote: >(off list) > >On Mon, Aug 23, 2004 at 03:34:13PM -0400, Phillip J. Eby wrote: >| >Yes. The idea is for Python to maintain a stack of generators, and when >| >ever the top-most generator yields <magic> this yield skips the >| >intermediate generators on the stack, and goes immediately to the >| >calling function. >| >| That's not the part I mean. I'm talking about *after* that happens, when >| you *resume* the generators, and you'll need to reconstruct the stack as >| it stood when the generators were interrupted. > >Yes. The entire generator stack would have to be put back into play. Right, but you haven't offered any proposal for where this stack will come from, how it will actually work, etc. I personally don't see any way to implement your approach in CPython that doesn't essentially require re-implementing a portion of 'peak.events' in C, while adding the overhead of the control value check to *every* generator .next() operation, even in programs that don't use the feature.

1 0

Multiple decorators per line
by Robert Brewer Aug. 23, 2004

Aug. 23, 2004

Guido, Are multiple decorators per line allowed in the current (pie) syntax? Should they be? They seem to be enabled in CVS, but nobody is verbally promoting them, and the wiki actually points out the "single deco per line" as a positive feature of the current syntax (A1). Robert Brewer MIS Amor Ministries fumanchu(a)amor.org

3 2

RE: [Python-Dev] python-dev Summary for 2004-08-01 through 2004-08-15[draft]
by Robert Brewer Aug. 23, 2004

Aug. 23, 2004

Brett C. wrote: > Option two out of all of this is people just say, "summarize what you > want, Brett." Then I just cover what I find interesting and just don't > worry about covering a specific area. +1. Bring back "Skipped Threads" if you can, just for the sake of indexing completeness. Robert Brewer MIS Amor Ministries fumanchu(a)amor.org

2 1

RE: [Python-Dev] Re: PEP 292 - Simpler String Substitutions
by Michael Chermside Aug. 23, 2004

Aug. 23, 2004

Tim Peters writes: > The same applies to string.join(iterable), for that matter. Clark Evans writes: > This code-snippet is littered everwhere in my applications: > > string.join([str(x) for x in iterable]) > > Its tedious and makes code hard to read. Do we need a PEP to fix this? I'm convinced! Of course any code like the following would break: class Item: # ... def __eq__(self, other): "All Items are equal!" return isinstance(other, Item) def process_item(item): """Processes an item, returning a string.""" # ... def process_items(items): """This processes each Item in the list and returns a string containing the entire thing.""" while True: try: return ','.join(items) except TypeError: pos = items.index(Item()) # find first Item items[pos] = process_item( items[pos] ) continue <Big Wink> Seriously, I can't think of a sane use for <String>.join() raising a TypeError, so I'd say run it by Guido and if it doesn't offend his design intuition, let's fix it! -- Michael Chermside

1 0

pre-PEP: Complete, Structured Regular Expression Group Matching
by Mike Coleman Aug. 23, 2004

Aug. 23, 2004

If you've ever been frustrated that there wasn't an easy way to parse a string or file, even though you could easily match it with re.match, then this PEP may be for you. If what's being proposed seems a little ghastly, well, I'll just say that I've been wanting something like this for a long time and this is the best I could come up with. Your comments invited. Mike ############################################################################## PEP: XXX Title: Complete, Structured Regular Expression Group Matching Version: $Revision: 1.3 $ Last-Modified: $Date: 2004/08/02 05:18:31 $ Author: Mike Coleman <mkc(a)mathdogs.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 1-Aug-2004 Python-Version: 2.4 Post-History: 1-Aug-2004 Abstract ======== This proposal is to add a new method ``re.structmatch`` that fully captures matches for groups that match repeatedly. The method returns a structured match tree whose shape is determined by the pattern groups. Ultimately this will allow a string that can be described by a Python regular expressions (e.g., the contents of ``/etc/passwd`` or ``.ini`` files, or the output of ``diff``) to be parsed into the obvious parse tree with a single call to ``re.structmatch``. Motivation ========== A notable limitation of the ``re.match`` method is that it fails to capture all group match information for repeatedly matched groups. For example, in a call like this :: m0 = re.match(r'([A-Z]|[a-z])*', 'XxxxYzz') one would like to see that the group which matched four times matched the strings ``'X'``, ``'xxx'``, ``'Y'`` and ``'zz'``. In the current implementation, only the last match for each group (``'zz'`` in this case) is available, even though the other matches are calculated implicitly during the matching process. For simple cases, the missing strings might be easily recovered by other means, but for more complicated cases this is a significant annoyance. The simplest extension would be to collect all of the matches for each group in a list, so that in the call above for example :: m0.group(1) --> ['X', 'xxx', 'Y', 'zz'] (A mechanism like this is apparently to be added to Perl in version 6 [#PERL6]_, though that did not inspire this PEP.) This immediately suggests the question of what is to be done for nested repeats, like so :: m1 = re.match(r'("([A-Z]|[a-z])*"\s*)*', '"Xx" "yy" "ZzZ"') We could have :: m1.group(2) --> ['X', 'x', 'yy', 'Z', 'z', 'Z' ] but this flattened result would discard useful information about the structure of the match. A more useful result would be :: m1.group(2) --> [['X', 'x'], ['yy'], ['ZzZ']] This is definitely an improvement. Consider the following slightly more complicated case, though :: m1 = re.match(r'("([A-Z]|[a-z])*"(\s*))*', '"Xx" "yy" "ZzZ"') which would give :: m1.group(2) --> [['X', 'x'], ['yy'], ['Z', 'z', 'Z']] m1.group(3) --> [' ', ' ', ''] This is less useful than the putative conceptual structure of the match, which would be something like :: [[['X', 'x'], ' '], [['yy'], ' '], [['Z', 'z', 'Z'], '']] In this simple case, this structure could be recovered using the ``zip`` function, but in the general case this is a non-trivial transformation. This PEP proposes a mechanism to address the general case directly. Proposal ======== The new function ``re.structmatch`` has the same arguments as ``re.match`` and will match using exactly the same algorithm. Instead of returning a MatchObject upon success, ``re.structmatch`` will return a "tree" (or rather, a forest) in the form of a Python list. First we will define some terms. Groups in the input pattern are parenthesized subexpressions that "capture" strings when matched, and are of the form ``'(abc)'`` or ``'(?P<name>abc)'``. Leaf groups are groups that do not contain any subgroups. So, for example, the group ``'(abc)'`` is a leaf group and the group ``'(a(xyz)b)'`` is not a leaf group (but it does contain the leaf group ``'(xyz)'``). We shall call parenthesized expressions that are not groups "nongroups"; these include constructs like ``'(?:abc)'`` as well as lookahead and lookbehind assertions. The structure of the returned tree is determined from the grouping tree present in the pattern in the following manner: * Leaf groups not followed immediately by a repeat operator map to a single string:: re.structmatch(r'(...)', 'abcdef') --> ['abc'] re.structmatch(r'(...).(..)', 'abcdef') --> ['abc', 'ef'] * Leaf groups followed immediately by a repeat operator map to a list of strings:: re.structmatch(r'([^d])*', 'abcdef') --> [['a', 'b', 'c']] re.structmatch(r'([^d])*(.)*', 'abcdef') --> [['a', 'b', 'c'], ['d', 'e', 'f']] re.structmatch(r'(..)*', 'abcdef') --> [['ab', 'cd', 'ef']] re.structmatch(r'(.){2}', 'abcdef') --> [['a', 'b']] * Non-leaf groups not followed immediately by a repeat operator map to a list of the mappings of their subgroups:: re.structmatch(r'(...)', 'abcdef') --> ['abc'] re.structmatch(r'((...))', 'abcdef') --> [['abc']] re.structmatch(r'(((...)))', 'abcdef') --> [[['abc']]] re.structmatch(r'((...)())', 'abcdef') --> [['abc'], []] re.structmatch(r'(.(.)(.(.)).(.))', 'abcdef') --> ['a', ['b'], ['c', ['d']], 'e', ['f']] * Non-leaf groups followed immediately by a repeat operator map to a list of the mappings of their repeated matches:: re.structmatch(r'((.).(.))*', 'abcdef') --> [[['a', 'c'], ['d', 'f']]] re.structmatch(r'(([ab])*(x)*)*', 'baxbxx') --> [[['b', 'a'], ['x']], [['b'], ['x', 'x']]] * In the case of alternation, only the matched groups appear in the output:: re.structmatch(r'([^a])*|([^d])*', 'abcdef') --> [['a', 'b', 'c']] If it's important to know which alternative matched, named groups can be used. * Named groups map to a pair where the first member is the name and the second member is what the unnamed group would have mapped to:: re.structmatch(r'([^d])*(?P<rest>.)*', 'abcdef') --> [['a', 'b', 'c'], ('rest', ['d', 'e', 'f'])] * Nongroups do not affect the output structure. Compared to non-leaf groups, nongroups have the effect of "flattening" the output, like so:: re.structmatch(r'((.).(.))', 'abcdef') --> [['a', 'c']] re.structmatch(r'(.).(.)', 'abcdef') --> ['a', 'c'] re.structmatch(r'(?:(.).(.))', 'abcdef') --> ['a', 'c'] re.structmatch(r'((.).(.))*', 'abcdef') --> [[['a', 'c'], ['d', 'f']]] re.structmatch(r'(?:(.).(.))*', 'abcdef') --> [['a', 'c', 'd', 'f']] (Nongroups that do not contain leaf groups have no effect whatsoever on the output structure.) Additional Features ------------------- In many cases in which ``'re.structmatch'`` fails to match, the cause will be due to an unexpected error in the format of the string being matched. In order to assist the calling program in generating a more meaningful possible error message, ``'re.structmatch'`` will return the endpoint of the largest match against the searched string. So, for example :: re.structmatch('abcd', 'abxxx') --> 2 re.structmatch('abcde|z', 'abxxx') --> 2 re.structmatch('x*?y', 'xxx') --> 3 (This return value should be considered advisory rather than exact, as future improvements in the match algorithm may make it difficult to calculate the exact value.) Examples ======== Parsing ``/etc/group`` ---------------------- If ``/etc/group`` contains the following lines :: root:x:0: daemon:x:1: bin:x:2: sys:x:3: then it can be parsed as follows :: p = r'''(?xm) # VERBOSE, MULTILINE ( ( # one field, preceded by a ':' if # the field is not the line's first (?:^|:) ([^:\n]*) )* \n )* ''' s = open('/etc/group').read() tree = re.structmatch(p, s) which will give ``tree`` the following value:: [['root', 'x', '0', ''], ['daemon', 'x', '1', ''], ['bin', 'x', '2', ''], ['sys', 'x', '3', '']] Note that the above pattern is written to allow any ``/etc/group`` field to be empty. The pattern won't work correctly for versions of Python prior to 2.4 because of the possibility of repeating empty matches. This problem seems to have been fixed in Python 2.4. (A slightly more complicated pattern would work for pre-2.4 versions.) Parsing ``.ini`` files ---------------------- The Microsoft ``.ini`` file format is found in various contexts (there is one in the Python source distribution: ``Tools/freeze/extensions_win32.ini``). Given a file with contents :: [singers] good=Annie Lennox bad=William Shatner [comedians] good=Monty Python the file can be parsed with pattern :: r'''(?xm) # VERBOSE, MULTILINE \s* # leading whitespace ( # begin sections ^\[ ([^]]+) \] \s* # section header ( # begin params ^([^=]+) = # param name (.*) $ # param value \s* # intervening whitespace )* # end params )* # end sections \Z # assert that the entire # input was matched ''' to give :: [['singers', ['good', 'Annie Lennox'], ['bad', 'William Shatner']], ['comedians', ['good', 'Monty Python']]] The pattern given omits support for ``.ini`` file comments for the sake of simplicity, but this could be added. Details ======= The proposal states that ``re.structmatch`` will match using exactly the same algorithm as ``re.match``. This might be a little odd for a pattern like ``r'(x|y|z)*aaa\1'``, where the algorithm will require that the ``\1`` back-reference match (at most) one character. It's not obvious whether there is any better option, though, and there are advantages of implementation and simplicity for keeping the match algorithms of these two methods identical. (A similar argument applies to ``'(?P=NAME)'``.) Discussion ========== Part of the reason the proposed method would be so useful is that ``re.split`` currently does not split on empty matches. If it had this feature, one could split on lookahead and lookbehind assertions, which would significantly ease parsing of strings with a recursive regular structure (e.g., ``.ini`` files). Patch `#988761`_ will correct this ``re.split`` deficiency, if it is accepted. .. _#988761: https://sourceforge.net/tracker/?func=detail&aid=988761&group_id=5470&atid=… For particularly complex patterns, the current 99 group limit might actually become a practical problem. One could imagine a variation in which subtrees of named groups might be captured in dictionaries rather than lists, with the group names used as keys. Rejected Alternatives ===================== Several simpler alternatives are rejected in the `Motivation`_ section above. Although these alternatives would be better than nothing, they would not adequately satisfy the goal of this proposal, which is to allow the programmer to extract the structure of a string in an immediate manner. It would be possible to use tuples for some substructures in the return tree, rather than composing it strictly of lists. This alternative was rejected because it was thought useful to be able to modify the resulting tree in place, perhaps to add decorations, etc., and tuples would make this more difficult. References ========== .. [#PERL6] Multiple regex matches in Perl Apocalypse 5 (http://www.perl.com/pub/a/2002/06/04/apo5.html?page=22#rfc%20360:%20allow%2…) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

7 21

PEP 318: Can't we all just get along?
by Kevin Smith Aug. 23, 2004

Aug. 23, 2004

For what it's worth, I wrote the original PEP 318. I probably wasn't qualified, but I just wanted a nice simple way to declare class methods without having to repeat the function name. After submitting it to BDFL for approval, more work was needed and the discussion of PEP 318 on python-dev increased rapidly. It was evident that I was in over my head, so I asked more someone more experienced to take over. I guess others had bigger plans for my proposal that I had planned. It has turned into the "solution" to many problems: type checking (both arguments and returned values), metaclasses, metadata, interfaces, function attributes, etc.). Unfortunately, in the process, this simple request for syntactical sugar has turned into a monstrosity. In my opinion, none of the proposed syntaxes really seem Pythonic. This PEP just seems to be trying to solve too many problems. Bear with me, but I'd like to propose one more syntax that is simple, easy for newbies to understand, and nowhere near as powerful as the current PEP's syntax. However, it doesn't add incoherent, arbitrary syntax either. def classmethod foo(x, y, z): pass That's it. One "decorator" that is a callable object that takes a method as it's only argument. No expressions, lists, tuples, etc. Just one callable object. Ok, if you absolutely must have more than one. def classmethod synchronized foo(x, y, z): pass Once again, no expressions. I know that this isn't going to solve everyone's type-checking, metadata, and function attribute problems, but let's face it, using this PEP for all of those things just creates ugly syntax. There must be more Pythonic ways to do those things in their own PEPs. Kevin Smith Kevin.Smith(a)themorgue.org

21 32

[Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0020.txt, NONE, 1.1
by Guido van Rossum Aug. 23, 2004

Aug. 23, 2004

A good idea! What made you think of doing this? --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

Re: [Python-checkins] python/dist/src/Modules socketmodule.c, 1.298, 1.299
by Neal Norwitz Aug. 23, 2004

Aug. 23, 2004

> Log Message: > Patch #1003700: Add socketpair function to socket module. > > Index: socketmodule.c > + #ifdef HAVE_SOCKETPAIR > + /* Create a pair of sockets using the socketpair() function. > + Arguments as for socket(). */ > + > + /*ARGSUSED*/ > + static PyObject * > + socket_socketpair(PyObject *self, PyObject *args) > + { > + PySocketSockObject *s0 = NULL, *s1 = NULL; > + SOCKET_T sv[2]; > + int family, type = SOCK_STREAM, proto = 0; > + PyObject *res = NULL; > + > + #if defined(AF_UNIX) > + family = AF_UNIX; > + #else > + family = AF_INET; > + #endif The docstring (below) states the arguments are the same as socket(). However, in sock_initobj() line 2496, the family is initialized to AF_INET. I think the #if defined(AF_UNIX) code above should be removed and family should be initialized to AF_INET. I didn't look to see if the documentation agrees with the docstring. > + if (!PyArg_ParseTuple(args, "|iii:socketpair", > + &family, &type, &proto)) > + return NULL; > + /* Create a pair of socket fds */ > + if (socketpair(family, type, proto, sv) < 0) > + return set_error(); > + #ifdef SIGPIPE > + (void) signal(SIGPIPE, SIG_IGN); > + #endif I don't think the #ifdef SIGPIPE code is correct. If the user installed a signal handler calling signal() will remove it. I think the call to signal() should be removed. Neal > + s0 = new_sockobject(sv[0], family, type, proto); > + if (s0 == NULL) > + goto finally; > + s1 = new_sockobject(sv[1], family, type, proto); > + if (s1 == NULL) > + goto finally; > + res = PyTuple_Pack(2, s0, s1); > + > + finally: > + if (res == NULL) { > + if (s0 == NULL) > + SOCKETCLOSE(sv[0]); > + if (s1 == NULL) > + SOCKETCLOSE(sv[1]); > + } > + Py_XDECREF(s0); > + Py_XDECREF(s1); > + return res; > + } > + > + PyDoc_STRVAR(socketpair_doc, > + "socketpair([family[, type[, proto]]]) -> (socket object, socket object)\n\ > + \n\ > + Create a pair of socket objects from the sockets returned by the platform\n\ > + socketpair() function.\n\ > + The arguments are the same as for socket()."); > + > + #endif /* HAVE_SOCKETPAIR */

4 7

Threading in the Standard Library Tour Part II
by Raymond Hettinger Aug. 22, 2004

Aug. 22, 2004

Aahz had suggested that the threading section of the tutorial's Standard Library Tour Part II be re-written with the idea of making people smarter about what Python threading can and cannot do, and about approaches most likely to assure success. Please comment on the proposed revision listed below. Raymond ------------------------------------------------------------------------ --- --------------- Multi-threading --------------- Threading is a technique for decoupling tasks which are not sequentially dependent and creating the illusion of concurrency. Threads can be used to improve the responsiveness of applications that accept user input while other tasks run in the background. The following code shows how the high level threading module can run tasks in background while the main program continues to run: import threading, zipfile class AsyncZip(threading.Thread): def __init__(self, infile, outfile): threading.Thread.__init__(self) self.infile = infile self.outfile = outfile def run(self): f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED) f.write(self.infile) f.close() print 'Finished background zip of: %s' % self.infile background = AsyncZip('mydata.txt', 'myarchive.zip') background.start() print 'The main program continues to run in foreground.' background.join() # Wait for the background task to finish print 'Main program waited until background was done.' The principal challenge of multi-thread applications is coordinating threads that share data or other resources. To that end, the threading module provides a number of synchronization primitives including locks, events, condition variables, and semaphores. While those tools are powerful, minor design errors can result in problems that are difficult to reproduce. Hence, the preferred approach to task coordination is to concentrate all access to a resource in a single thread and then use the Queue module to feed that thread with requests from other threads. Applications using Queue objects for inter-thread communication and coordination tend to be easier to design, more readable, and more reliable. All that being said, a few cautions are in order. Thread programming is difficult to get right. And, its overhead decreases total application performance. Also, multiple processors cannot boost performance because Python's Global Interpreter Lock (GIL) precludes more than one thread from running in the interpreter at the same time (this was done to simplify re-entrancy issues). Another issue is that threading doesn't work with the event driven model used by most GUIs.

10 19

__metaclass__ and __author__ are already decorators
by Paul Morrow Aug. 22, 2004

Aug. 22, 2004

[I posted this on comp.lang.python.general but I'm not sure how many of the folks here read that newsgroup, so my apologies if you're seeing this twice.] Thinking about decorators, and looking at what we are already doing in our Python code, it seems that __metaclass__, __author__, __version__, etc. are all examples of decorators. So we already have a decorator syntax. What is the compelling reason to invent a new one? And if we do, what's to become of the old one? Here's my take on this. We have two kinds of decorators: those that are informational only (like __author__ and __version__), and those which have side-effects, i.e. those that actually *do* something (like __metaclass__). class Foo: """ This describes the Foo class as normal. """ __metaclass__ = M __author__ = 'Paul Morrow' __version__ = '0.1' __automethods__ = True def baz(self, a, b): """ This describes the baz method. """ __synchronized__ = True __returns__ = None __author__ = 'Neville Shunt' # body of baz goes here... There, that looks pretty clear and pythonic. Now how to define the decorators. def metaclass(decoratedClass, subType): """ This describes the 'metaclass' decorator. """ __decorator__ = True __version__ = '1.9' # perform the metaclass operation on decoratedClass def synchronized(decoratedFunc, trueOrFalse): """ This describes the 'synchronized' decorator. """ __decorator__ = True __author__ = 'Martin Curry' __version__ = '0.5' # perform the synchronized operation on decoratedFunc def returns(decoratedFunc, *args): """Docstring for 'returns' decorator.""" __decorator__ = True # body of decorator goes here Each decorator function receives the class|function|method being decorated as the first parameter of the function (e.g. decoratedClass and decoratedFunc above). The remaining parameters are the values assigned to the __xxx__ variable in the definition of the decorated function. So, in the above example, when the 'returns' decorator function is called to decorate the 'baz' method, decoratedFunc would recieve the baz object and args[0] would be set to None (because of the statement "__returns__ = None" in the definition of baz). For a function to be used as a decorator function, it must be decorated as such, by setting its __decorator__ attribute to True. Does this handle enough of the decorator concerns? Paul

6 28