Mailman 3 April 2006 - Python-Dev

__getslice__ usage in sre_parse
by Sanghyeon Seo 01 May '06

01 May '06

Hello, Python language reference 3.3.6 deprecates __getslice__. I think it's okay that UserList.py has it, but sre_parse shouldn't use it, no? __getslice__ is not implemented in IronPython and this breaks usage of _sre.py, a pure-Python implementation of _sre, on IronPython: http://ubique.ch/code/_sre/ _sre.py is needed for me because IronPython's own regex implementation using underlying .NET implementation is not compatible enough for my applications. I will write a separate bug report for this. It should be a matter of removing __getslice__ and adding isinstance(index, slice) check in __getitem__. I would very much appreciate it if this is fixed before Python 2.5. Seo Sanghyeon

3 2

socket module recvmsg/sendmsg
by Heiko Wundram 01 May '06

01 May '06

Hi all! I've implemented recvmsg and sendmsg for the socket module in my private Python tree for communication between two forked processes, which are essentially wrappers for proper handling of SCM_RIGHTS and SCM_CREDENTIALS Unix-Domain-Socket messages (which are the two types of messages that are defined on Linux). The main reason I need these two primitives is that I require (more or less transparent) file/socket descriptor exchange between two forked processes, where one process accepts a socket, and delegates processing of the socket connection to another process of a set of processes; this is much like a ForkingTCPServer, but with the Handler-process prestarted. As connection to the Unix-Domain-Socket is openly available, the receiving process needs to check the identity of the first process; this is done using a getsockopt(SO_PEERCRED) call, which is also handled more specifically by my socket extension to return a socket._ucred-type structure, which wraps the pid/uid/gid-structure returned by the corresponding getsockopt call, and also the socket message (SCM_CREDENTIALS) which passes or sets this information for the remote process. I'd love to see these two socket message primitives (of which the first, SCM_RIGHTS, is available on pretty much any Unix derivative) included in a Python distribution at some point in time, and as I've not got the time to push for an inclusion in the tree (and even less time to work on other Python patches myself) at the moment, I thought that I might just post here so that someone interested might pick up the work I've done so far and check the implementation for bugs, and at some stage these two functions might actually find their way into the Python core. Anyway, my private Python tree (which has some other patches which aren't of general interest, I'd think) is available at: http://dev.modelnine.org/hg/python and I can, if anyone is interested, post a current diff of socketmodule.* against 2.4.3 to the Python bug tracker at sourceforge. I did that some time ago (about half a year) when socket-passing code wasn't completely functioning yet, but at least at that point there didn't seem much interest in the patch. The patch should apply pretty cleanly against the current HEAD too, at least it did the last time I checked. I'll add a small testsuite for both functions to my tree some time tomorrow. --- Heiko.

2 1

elimination of scope bleeding of iteration variables
by Ben Wing 01 May '06

01 May '06

apologies if this has been brought up on python-dev already. a suggestion i have, perhaps for python 3.0 since it may break some code (but imo it could go into 2.6 or 2.7 because the likely breakage would be very small, see below), is the elimination of the misfeature whereby the iteration variable used in for-loops, list comprehensions, etc. bleeds out into the surrounding scope. [i'm aware that there is a similar proposal for python 3.0 for list comprehensions specifically, but that's not enough.] i've been bitten by this problem more than once. most recently, i had a simple function like this: # Replace property named PROP with NEW in PROPLIST, a list of tuples. def property_name_replace(prop, new, proplist): for i in xrange(len(proplist)): if x[i][0] == prop: x[i] = (new, x[i][1]) the use of `x' in here is an error, as it should be `proplist'; i previously had a loop `for x in proplist', but ran into the problem that tuples can't be modified, and lists can't be modified except by index. despite this obviously bad code, however, it ran without any complaint from python -- which amazed me, when i finally tracked this problem down. turns out i had two offenders, both way down in the "main" code at the bottom of the file -- a for-loop with loop variable x, and a list comprehension [x for x in output_file_map] -- and suddenly `x' is a global variable. yuck. i suggest that the scope of list comprehension iteration variables be only the list comprehension itself (see python 3000), and the scope of for-loop iteration variables be only the for-loop (including any following `else' statement -- this way you can access it and store it in some other variable if you really want it). in practice, i seriously doubt this will break much code and probably could be introduced like the previous scope change: using `from __future__' in python 2.5 or 2.6, and by default in the next version. it should be possible, in most circumstances, to issue a warning about code that relies on the old behavior. ben

5 5

Tkinter lockups.
by Thomas Wouters 01 May '06

01 May '06

For a while now, I've noticed test_tcl locking up when trying to refleaktest it. I was able to reproduce it quite simply: import Tkinter import os if "DISPLAY" in os.environ: del os.environ["DISPLAY"] tcl = Tkinter.Tcl() try: tcl.loadtk() except Exception, e: print e tcl.loadtk() Or, more directly, using _tkinter directly: import _tkinter import os if "DISPLAY" in os.environ: del os.environ["DISPLAY"] tk = _tkinter.create(None, "test", "Tk", 0, 1, 0) try: tk.loadtk() except: pass tk.loadtk() In either case, the second loadtk never finishes. It seems that, on my platform at least, Tk_Init() doesn't like being called twice even when the first call resulted in an error. That's Tcl and Tk 8.4.12. Tkapp_Init() (which is the Tkinter part that calls Tk_Init()) does its best to guard against calling Tk_Init() twice when the first call was succesful, but it doesn't remember failure cases. I don't know enough about Tcl/Tk or Tkinter how this is best handled, but it would be mightily convenient if it were. ;-) I've created a bugreport on it, and I hope someone with Tkinter knowledge can step in and fix it. (It looks like SF auto-assigned it to Martin already, hmm.) http://sourceforge.net/tracker/index.php?func=detail&aid=1475162&group_id=5… -- Thomas Wouters <thomas(a)python.org> Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

3 5

Re: [Python-Dev] More on contextlib - adding back a contextmanager decorator
by Phillip J. Eby 01 May '06

01 May '06

At 09:53 AM 4/30/2006 -0700, Guido van Rossum wrote: >I have a counter-proposal: let's drop __context__. Nearly all use >cases have __context__ return self. In the remaining cases, would it >really be such a big deal to let the user make an explicit call to >some appropriately named method? The only example that I know of where >__context__ doesn't return self is the decimal module. So the decimal >users would have to type > > with mycontext.some_method() as ctx: # ctx is a clone of mycontext > ctx.prec += 2 > <BODY> > >The implementation of some_method() could be exactly what we currently >have as the __context__ method on the decimal.Context object. Its >return value is a decimal.WithStatementContext() instance, whose >__enter__() method returns a clone of the original context object >which is assigned to the variable in the with-statement (here 'ctx'). > >This even has an additional advantage -- some_method() could have >keyword parameters to set the precision and various other context >parameters, so we could write this: > > with mycontext.some_method(prec=mycontext.prec+2): > <BODY> > >Note that we can drop the variable too now (unless we have another >need to reference it). An API tweak for certain attributes that are >often incremented or decremented could reduce writing: > > with mycontext.some_method(prec_incr=2): > <BODY> But what's an appropriate name for some_method? Given that documentation is the sore spot that keeps us circling around this point, doesn't this just push the problem to finding a name to use in place of __context__? And not only for this use case, but for others? After all, for any library that has a notion of "the current X", it seems reasonable to want to be able to say "with some_X" to mean "use some_X as the current X for this block". And it thus seems to me that people will want to have something like: def using(obj): if hasattr(obj,'__context__'): obj = obj.__context__() return obj so they can do "with using(some_X)", because "with some_X.using()" or "with some_X.as_current()" is awkward. If you can solve the naming issue for these use cases (and I notice you punted on that issue by calling it "some_method"), then +1 on removing __context__. Otherwise, I'm -0; we're just fixing one documentation/explanation problem (that only people writing contexts will care about) by creating others (that will affect the people *using* contexts too).

3 2

PEP 3101: Advanced String Formatting
by Talin 01 May '06

01 May '06

PEP: 3101 Title: Advanced String Formatting Version: $Revision$ Last-Modified: $Date$ Author: Talin <talin at acm.org> Status: Draft Type: Standards Content-Type: text/plain Created: 16-Apr-2006 Python-Version: 3.0 Post-History: Abstract This PEP proposes a new system for built-in string formatting operations, intended as a replacement for the existing '%' string formatting operator. Rationale Python currently provides two methods of string interpolation: - The '%' operator for strings. [1] - The string.Template module. [2] The scope of this PEP will be restricted to proposals for built-in string formatting operations (in other words, methods of the built-in string type). The '%' operator is primarily limited by the fact that it is a binary operator, and therefore can take at most two arguments. One of those arguments is already dedicated to the format string, leaving all other variables to be squeezed into the remaining argument. The current practice is to use either a dictionary or a tuple as the second argument, but as many people have commented [3], this lacks flexibility. The "all or nothing" approach (meaning that one must choose between only positional arguments, or only named arguments) is felt to be overly constraining. While there is some overlap between this proposal and string.Template, it is felt that each serves a distinct need, and that one does not obviate the other. In any case, string.Template will not be discussed here. Specification The specification will consist of 4 parts: - Specification of a set of methods to be added to the built-in string class. - Specification of a new syntax for format strings. - Specification of a new set of class methods to control the formatting and conversion of objects. - Specification of an API for user-defined formatting classes. String Methods The build-in string class will gain a new method, 'format', which takes takes an arbitrary number of positional and keyword arguments: "The story of {0}, {1}, and {c}".format(a, b, c=d) Within a format string, each positional argument is identified with a number, starting from zero, so in the above example, 'a' is argument 0 and 'b' is argument 1. Each keyword argument is identified by its keyword name, so in the above example, 'c' is used to refer to the third argument. The result of the format call is an object of the same type (string or unicode) as the format string. Format Strings Brace characters ('curly braces') are used to indicate a replacement field within the string: "My name is {0}".format('Fred') The result of this is the string: "My name is Fred" Braces can be escaped using a backslash: "My name is {0} :-\{\}".format('Fred') Which would produce: "My name is Fred :-{}" The element within the braces is called a 'field'. Fields consist of a name, which can either be simple or compound, and an optional 'conversion specifier'. Simple names are either names or numbers. If numbers, they must be valid decimal numbers; if names, they must be valid Python identifiers. A number is used to identify a positional argument, while a name is used to identify a keyword argument. Compound names are a sequence of simple names seperated by periods: "My name is {0.name} :-\{\}".format(dict(name='Fred')) Compound names can be used to access specific dictionary entries, array elements, or object attributes. In the above example, the '{0.name}' field refers to the dictionary entry 'name' within positional argument 0. Each field can also specify an optional set of 'conversion specifiers'. Conversion specifiers follow the field name, with a colon (':') character separating the two: "My name is {0:8}".format('Fred') The meaning and syntax of the conversion specifiers depends on the type of object that is being formatted, however many of the built-in types will recognize a standard set of conversion specifiers. The conversion specifier consists of a sequence of zero or more characters, each of which can consist of any printable character except for a non-escaped '}'. The format() method does not attempt to intepret the conversion specifiers in any way; it merely passes all of the characters between the first colon ':' and the matching right brace ('}') to the various underlying formatters (described later.) Standard Conversion Specifiers For most built-in types, the conversion specifiers will be the same or similar to the existing conversion specifiers used with the '%' operator. Thus, instead of '%02.2x", you will say '{0:2.2x}'. There are a few differences however: - The trailing letter is optional - you don't need to say '2.2d', you can instead just say '2.2'. If the letter is omitted, the value will be converted into its 'natural' form (that is, the form that it take if str() or unicode() were called on it) subject to the field length and precision specifiers (if supplied). - Variable field width specifiers use a nested version of the {} syntax, allowing the width specifier to be either a positional or keyword argument: "{0:{1}.{2}d}".format(a, b, c) (Note: It might be easier to parse if these used a different type of delimiter, such as parens - avoiding the need to create a regex that handles the recursive case.) - The support for length modifiers (which are ignored by Python anyway) is dropped. For non-built-in types, the conversion specifiers will be specific to that type. An example is the 'datetime' class, whose conversion specifiers are identical to the arguments to the strftime() function: "Today is: {0:%x}".format(datetime.now()) Controlling Formatting A class that wishes to implement a custom interpretation of its conversion specifiers can implement a __format__ method: class AST: def __format__(self, specifiers): ... The 'specifiers' argument will be either a string object or a unicode object, depending on the type of the original format string. The __format__ method should test the type of the specifiers parameter to determine whether to return a string or unicode object. It is the responsibility of the __format__ method to return an object of the proper type. string.format() will format each field using the following steps: 1) See if the value to be formatted has a __format__ method. If it does, then call it. 2) Otherwise, check the internal formatter within string.format that contains knowledge of certain builtin types. 3) Otherwise, call str() or unicode() as appropriate. User-Defined Formatting Classes The code that interprets format strings can be called explicitly from user code. This allows the creation of custom formatter classes that can override the normal formatting rules. The string and unicode classes will have a class method called 'cformat' that does all the actual work of formatting; The format() method is just a wrapper that calls cformat. The parameters to the cformat function are: -- The format string (or unicode; the same function handles both.) -- A field format hook (see below) -- A tuple containing the positional arguments -- A dict containing the keyword arguments The cformat function will parse all of the fields in the format string, and return a new string (or unicode) with all of the fields replaced with their formatted values. For each field, the cformat function will attempt to call the field format hook with the following arguments: field_hook(value, conversion, buffer) The 'value' field corresponds to the value being formatted, which was retrieved from the arguments using the field name. (The field_hook has no control over the selection of values, only how they are formatted.) The 'conversion' argument is the conversion spec part of the field, which will be either a string or unicode object, depending on the type of the original format string. The 'buffer' argument is a Python array object, either a byte array or unicode character array. The buffer object will contain the partially constructed string; the field hook is free to modify the contents of this buffer if needed. The field_hook will be called once per field. The field_hook may take one of two actions: 1) Return False, indicating that the field_hook will not process this field and the default formatting should be used. This decision should be based on the type of the value object, and the contents of the conversion string. 2) Append the formatted field to the buffer, and return True. Alternate Syntax Naturally, one of the most contentious issues is the syntax of the format strings, and in particular the markup conventions used to indicate fields. Rather than attempting to exhaustively list all of the various proposals, I will cover the ones that are most widely used already. - Shell variable syntax: $name and $(name) (or in some variants, ${name}). This is probably the oldest convention out there, and is used by Perl and many others. When used without the braces, the length of the variable is determined by lexically scanning until an invalid character is found. This scheme is generally used in cases where interpolation is implicit - that is, in environments where any string can contain interpolation variables, and no special subsitution function need be invoked. In such cases, it is important to prevent the interpolation behavior from occuring accidentally, so the '$' (which is otherwise a relatively uncommonly-used character) is used to signal when the behavior should occur. It is the author's opinion, however, that in cases where the formatting is explicitly invoked, that less care needs to be taken to prevent accidental interpolation, in which case a lighter and less unwieldy syntax can be used. - Printf and its cousins ('%'), including variations that add a field index, so that fields can be interpolated out of order. - Other bracket-only variations. Various MUDs (Multi-User Dungeons) such as MUSH have used brackets (e.g. [name]) to do string interpolation. The Microsoft .Net libraries uses braces ({}), and a syntax which is very similar to the one in this proposal, although the syntax for conversion specifiers is quite different. [4] - Backquoting. This method has the benefit of minimal syntactical clutter, however it lacks many of the benefits of a function call syntax (such as complex expression arguments, custom formatters, etc.). - Other variations include Ruby's #{}, PHP's {$name}, and so on. Sample Implementation A rought prototype of the underlying 'cformat' function has been coded in Python, however it needs much refinement before being submitted. Backwards Compatibility Backwards compatibility can be maintained by leaving the existing mechanisms in place. The new system does not collide with any of the method names of the existing string formatting techniques, so both systems can co-exist until it comes time to deprecate the older system. References [1] Python Library Reference - String formating operations http://docs.python.org/lib/typesseq-strings.html [2] Python Library References - Template strings http://docs.python.org/lib/node109.html [3] [Python-3000] String formating operations in python 3k http://mail.python.org/pipermail/python-3000/2006-April/000285.html [4] Composite Formatting - [.Net Framework Developer's Guide] http://msdn.microsoft.com/library/en-us/cpguide/html/cpconcompositeformatti… Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

7 7

Problem with inspect and PEP 302
by Georg Brandl 30 Apr '06

30 Apr '06

Recently, the inspect module was updated to conform with PEP 302. Now this is broken: >>> import inspect >>> inspect.stack() The traceback shows clearly what's going on. However, I don't know how to resolve the problem. Georg

2 1

Re: [Python-Dev] [Python-3000] in-out parameters
by Edward Loper 30 Apr '06

30 Apr '06

Rudy Rudolph wrote: > 2) pass-by-reference: > def f(wrappedParam): > wrappedParam[0] += 5 # ugh > return "this is my result" > > # call it > x = 2 > result = f([x]) > # also ugly, but x is now 7 This example is broken; here's what you get when you run it: >>> def f(wrappedParam): ... wrappedParam[0] += 5 ... return "this is my result" ... >>> # call it ... x = 2 >>> result = f([x]) >>> x 2 You probably intended something more like: >>> x = [2] >>> result = f(x) >>> x[0] 7 (As for the actual topic, I'm personally -0 for adding in-out parameters to python.) -Edward

1 0

Re: [Python-Dev] methods on the bytes object
by Phillip J. Eby 30 Apr '06

30 Apr '06

At 08:22 AM 4/30/2006 -0700, Guido van Rossum wrote: >Still, I expect that having a bunch of string-ish methods on bytes >arrays would be convenient for certain types of data handling. Of >course, only those methods that don't care about character types would >be added, but that's a long list: startswith, endswith, index, rindex, >find, rfind, split, rsplit, join, count, replace, translate. I've often wished *lists* had startswith and endswith, and somewhat less often wished they had split or rsplit. Those seem like things that are generally applicable to sequences, not just strings or bytes.

1 0

PyThreadState_SetAsyncExc and native extensions
by Gabriel Becedillas 30 Apr '06

30 Apr '06

Does anyone know if PyThreadState_SetAsyncExc stops a thread while its inside a native extension ? I'm trying to stop a testing script that boils down to this: while True: print "aaa" native_extension_call() print "bbb" Using PyThreadState_SetAsyncExc the module doesn't stop but if I add more print statements to increase the chance that PyThreadState_SetAsyncExc is called when the module is executing Python code, the module stops.

2 1