Mailman 3 May 2004 - Python-Dev

pre-announce: Python 2.4a1 is about a month away
by Anthony Baxter 02 Jun '04

02 Jun '04

This is a pre-announcement that the first alpha of Python 2.4 is about a month away. The purpose of this notice is to give people a heads up - if you have a bug that you want to see fixed for 2.4, start looking at it now. Fixes are welcome through the release cycle, although after the first beta fixes that result in a change to behaviour will be much less likely to be accepted. So, if you have a bug you want to see fixed, what should you do? - If it's not logged on SF, log it. - If it's logged, consider adding a patch that fixes the problem, or at least a simple test case that demonstrates the bug. - If someone else has supplied a fix, see if this fix works for you, and post your results to the bug. - If there's a working fix, feel free to add a note asking for the fix to go into CVS. The SF bug tracker for Python has a _lot_ of bugs in it, and it's easy for bugs to be overlooked. If you're just interested in what's coming up in 2.4, see the current development version of the "What's New in 2.4" document at http://www.python.org/dev/doc/devel/whatsnew/whatsnew24.html On behalf of the python-dev team, thanks! Anthony [This form of pre-announcement will hopefully become a part of the release process for all future releases of Python, both major and bugfix releases, to give people a chance to get their bugfix-of-choice submitted in time. Any feedback on this process, please feel free to email me] -- Anthony Baxter <anthony(a)interlink.com.au> It's never too late to have a happy childhood.

2 1

Why aren't more things weak referencable
by Raymond Hettinger 01 Jun '04

01 Jun '04

Instances of classes inheriting from str, tuple, etc cannot be weakly referenced. Does anyone know the reason for this? >>> from weakref import proxy >>> class Object(object): pass >>> x = Object() >>> y = proxy(x) # no problem here >>> class Str(str): pass >>> x = Str() >>> y = proxy(x) # but why won't this work? Traceback (most recent call last): File "<pyshell#9>", line 1, in -toplevel- y = proxy(x) TypeError: cannot create weak reference to 'Str' object Weak referencing would be much more useful to me if it included strings, tuples, and such. Right now, my awkward workaround is substituting UserString or some other wrapper class replacing inheritance with delegation. Raymond Hettinger

7 12

Uninformative error message about subclassing
by Edward C. Jones 01 Jun '04

01 Jun '04

Here are two small python files: ---- a.py: class A(object): pass ---- b.py: import a class B(a): pass ---- If I run "python b.py" I get the uninformative error message: Traceback (most recent call last): File "./b.py", line 5, in ? class B(a): TypeError: function takes at most 2 arguments (3 given) I think the message should say that "a" is not a class.

3 3

doctest improvements
by Edward Loper 01 Jun '04

01 Jun '04

About a month and a half ago, I filed several patches for doctest, and they haven't gotten any comments; so I was wondering if anyone could look at them. The most interesting is #933238, which addresses the often-noted problem that doctest can't handle blank lines. It adds a special blank-line marker (currently "-"), which must be *dedented* with respect to the doctest block. Here's an example use (the "x"s are to prevent mail readers from interpreting ">>>" as a quotation). x def test_blankline_marker(): x r""" x Use a dedented "-" to signify a blank line: x x >>> print 'First line\n\nThis is after a blank line\n' x First line x - x This is after a blank line x - x """ (Note that both internal and trailing blank lines can be handled.) This change is backwards compatible, since currently any line in a doctest block that's dedented past the prompt (">>>") is an error. If anyone has time to take a look at this, let me know what you think. Thanks. -Edward My doctest patches: [ 933238 ] doctest: add a special (dedented) marker for blank lines http://sourceforge.net/tracker/?func=detail&aid=933238&group_id=5470&atid=3… [ 932930 ] doctest: suggest the use of rawstrings for backslashes http://sourceforge.net/tracker/?func=detail&aid=932930&group_id=5470&atid=3… [ 932935 ] doctest: allow custom matchers for testing if got==want http://sourceforge.net/tracker/?func=detail&aid=932935&group_id=5470&atid=3… [ 932932 ] doctest: Add Tester params to DocTestSuite http://sourceforge.net/tracker/?func=detail&aid=932932&group_id=5470&atid=3…

3 2

cookielib
by John J Lee 31 May '04

31 May '04

I've uploaded a patch for my proposed new 2.4 module cookielib: http://python.org/sf/963318 What next? John

3 6

Need traceback expert to review patch 954922
by Brett C. 31 May '04

31 May '04

Back in February, I tried to fix an issue of threads executed using 'threading' that did not shut themselves down before the interpreter started tearing down globals, the thread triggering an exception, and then a new exception being raised while trying to report the exception raised in the thread by accessing globals (see http://mail.python.org/pipermail/python-dev/2004-February/042457.html for the previous discussion). An initial fix was devised, but since there was no way to trigger the issue every time I was not sure if the fix was thorough enough. Well, someone finally has an app that causes the error consistently and the previous fix did not go far enough. But with the help of the app (mnet, specifically) I was finally able to get rid of all global accesses while reporting exceptions so that the original exception caught in the thread can get printed. The patch is at http://python.org/sf/954922 while the bug report is at http://python.org/sf/754449 . Trouble is that I had to basically fake traceback.format_exc() in the patch to a reasonable level since the 'traceback' module bounces around between its various functions too much to just store a local copy of the function and have it still work after globals are set to None. So I need someone to just look over the code to make sure that the code is not doing something stupid when this code is called (default way is still used when 'sys' is defined and thus the interpreter is not being torn down). And the reason I bring this up now is just in case there is any desire to put this in 2.3.4 . I obviously know that I missed rc1, but I didn't get the details of triggering the problem soon enough and people running 2.3.x are the ones bringing this up since this apparently did not happen under 2.2 for some reason. I doubt Anthony wants to risk mucking with 'threading' this late (although the patch is fairly self-contained but would require applying the previous patch to attempt to fix this as well), but I thought I would at least put it out there. -Brett

5 10

Anyone object to a rewrite of site.py?
by Brett C. 31 May '04

31 May '04

At PyCON I rewrote site.py . The biggest change is that I broke it into functions and moved it over to use sets. Basically the file was just old and I brought it up to modern usage. Performance-wise I can't see a real difference between the current version and my rewrite by executing ``time ./python.exe -c "pass"``; fluctuations in values are standard white noise. Breaking it up into functions also allowed me to write a test suite for site.py . Patch 921793 (http://www.python.org/sf/921793) has the patch of my rewrite. Does anyone object to me applying the patch? Of course I would love to have someone take a look at the code since this gets executed every time. I have been using the rewritten version since PyCON (so about two months) and I haven't had a bug yet. Obviously I would like to get this into 2.4a1 to get as much testing as possible. -Brett

2 1

slots, properties, descriptors, and pydoc
by John Belmonte 31 May '04

31 May '04

Hello, I'd like to make some observations and proposals about pydoc regarding slots, properties, and descriptors in general. Given this program: class CountingDescriptor(object): def __init__(self, doc=None): self.val = 0 self.__doc__ = doc def __get__(self, obj, objtype): self.val += 1 return self.val class Foo(object): __slots__ = {'bar': 'slot doc test'} counter = CountingDescriptor(doc='descriptor doc test') def _set_counter_multiple(self, val): print 'I hear you want to set me to %s?' % val counter_multiple = property( lambda self: self.counter*2, _set_counter_multiple, doc='property doc test' ) print help(Foo) The output is: Help on class Foo in module __main__: class Foo(__builtin__.object) | Properties defined here: | | counter_multiple | property doc test | | lambdaself | | <set> = _set_counter_multiple(self, val) | | ------------------------------------------------------------ | Data and other attributes defined here: | | __slots__ = {'bar': 'slot doc test'} | | bar = <member 'bar' of 'Foo' objects> | | counter = 2 Observations (in order from least to most arguable :-): 1) pydoc's property display doesn't handle lambda well 2) pydoc doesn't show the doc string of general descriptors 3) Using dictionary values as a way to set the doc string of slots, as suggested in Guido's "Unifying types and classes", has not been implemented. Currently there is no way to set a doc string on a descriptor generated by __slots__. Even if you could, pydoc would not display it, according to #2. 4) pydoc is displaying implementation details of properties (set and get method names, etc.). Such details should probably not be displayed, similar to underbar-prefixed method and data attributes. Proposals: 1) remove property-specific handling from pydoc 2) extend pydoc with descriptor-specific handling. In a section called "Descriptors defined here", list each attribute name, with associated doc string if it exists. 3) extend the __slots__ handler to set the descriptor's doc string to dictionary values if the slot's value is a dictionary. This is the only Python change being proposed. 4) have pydoc make a special case for the display of __slots__. Alternatives: A) don't display __slots__ if the value is a dictionary. The display of __slots__ is redundant, because all items are displayed according to #2. B) if for some reason distinguishing slots from other descriptors is essential, then always display __slots__, but normalize it to a list of attribute names (show only dictionary keys, etc.) With these changes, the new output of the example would be: Help on class Foo in module __main__: class Foo(__builtin__.object) | Descriptors defined here: | | bar | slot doc test | | counter | descriptor doc test | | counter_multiple | property doc test I'd be willing to work on a PEP. If these issues are covered in or conflict with an existing PEP, please let me know. Thanks, -John -- http:// if ile.org/

3 13

Standard Library Tour Part II
by Raymond Hettinger 31 May '04

31 May '04

Comments are invited on the attached draft for a second overview of the standard library. The first overview of basic modules at http://docs.python.org/tut/node12.html was well received and seemed to get people off to a running start. The primary goal of this follow-up is to reduce the transition time for a programmer to start writing real applications using Python. A second goal is give information to people who work through tutorials to determine the capabilities of various languages. A third goal is provide a jumping-off point for someone learning brand new concepts such as weak references or threading. The current docs merely describe the cockpit controls and assume that you already know how to fly an aircraft. Raymond ========================================================= Brief Tour of Modules in the Standard Libary (part II) This second tour covers more advanced modules that support professional programming needs. Their use rarely arises for small scripts. Output Formatting ----------------- The repr module provides an version of repr() for abbreviated displays of large or deeply nested containers:: >>> import repr >>> repr.repr(set('supercalafraglisticexpallidoches')) "set(['a', 'c', 'd', 'e', 'f', 'g', ...])" The pprint module offers more sophisticated control over printing both built-in and user defined objects in a way that is readable by the interpreter. When the result is longer than one line, the "pretty printer" adds line breaks and indentation to more clearly reveal data structure:: >>> import pprint >>> t = [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta', ... 'yellow'], 'blue']]] ... >>> pprint.pprint(t, width=30) [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta', 'yellow'], 'blue']]] The textwrap module formats paragraphs of text to fit a given screen width:: >>> import textwrap >>> doc = """The wrap() method is just like fill() except that it returns ... a list of strings instead of one big string with newlines to separate ... the wrapped lines.""" ... >>> print textwrap.fill(doc, width=40) The wrap() method is just like fill() except that it returns a list of strings instead of one big string with newlines to separate the wrapped lines. The locale module accesses a database of cultural specific data formats. The grouping attribute of locale's format function provides a direct way of formatting numbers with group separators. >>> import locale >>> locale.setlocale(locale.LC_ALL, 'English_United States.1252') 'English_United States.1252' >>> conv = locale.localeconv() # get a mapping of conventions >>> x = 1234567.8 >>> locale.format("%d", x, grouping=True) '1,234,567' >>> locale.format("%s%.*f", (conv['currency_symbol'], ... conv['int_frac_digits'], x), grouping=True) '$1,234,567.80' Working with Binary Data Record Layouts --------------------------------------- The struct module provides pack() and unpack() functions for working with variable length binary record formats. The following example shows how to loop through header information in a ZIP file (with pack codes "H" and "L" representing two and four byte numbers respectively):: import struct data = open('myfile.zip', 'rb').read() start = 0 for i in range(3): # show the first 3 file headers start += 14 fields = struct.unpack('LLLHH', data[start:start+16]) crc32, comp_size, uncomp_size, filenamesize, extra_size = fields start += 16 filename = data[start:start+filenamesize] start += filenamesize extra = data[start:start+extra_size] print filename, hex(crc32), comp_size, uncomp_size start += extra_size + comp_size # skip to the next header Multi-threading --------------- Threading is a technique for decoupling tasks which are not sequentially dependent. Python threads are driven by the operating system and run in a single process and share memory space in a single interpreter. Threads can be used to improve the responsiveness of applications that accept user input while other tasks run in the background. The following code shows how the high level threading module can run tasks in background while the main program continues to run:: import threading, zipfile class AsyncZip(threading.Thread): def __init__(self, infile, outfile): Thread.__init__(self) self.infile = infile self.outfile = outfile def run(self): f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED) f.write(self.infile) f.close() print 'Finished background zip of: ', self.infile AsyncZip('mydata.txt', 'myarchive.zip').start() print 'The main program continues to run' The principal challenge of multi-thread applications is coordinating threads that share data or other resources. To that end, the threading module provides a number of synchronization primitives including locks, events, condition variables, and semaphores. While those tools are powerful, minor design errors can result in problems that are difficult to reproduce. A simpler and more robust approach to task coordination is concentrating all access to a resource in a single thread and then using the Queue module to feed that thread with requests from other threads. Applications that use Queue objects for inter-thread communication and coordination tend to be easier to design, more readable, and more reliable. Logging ------- The logging module offers a full featured and flexible logging system. At its simplest, log messages are sent to a file or to sys.stderr. Other options include routing messages through email, datagrams, sockets, or to an HTTP Server. Messages are assigned priorities including DEBUG, INFO, WARNING, ERROR, and CRITICAL. Filtering allows messages to be directed based on their priority (for example: email only critcal messages, ignore debug messages, and write everything else to a file): [XXX add example] Weak References --------------- Python does automatic memory management (reference counting for most objects and garbage collection to eliminate cycles). The memory is freed shortly after the last reference to it has been eliminated. This approach works fine for most applications but occasionally there is a need to track objects only as long as they are being used by something else. Unfortunately, just tracking them creates a reference that makes them permanent. The weakref module provides tools tracking objects without creating a reference. When the object is no longer needed, it is automatically removed from a weakref table and a callback is triggered for weakref objects. Typically applications include caching objects that are expensive to create: >>> import weakref, gc >>> class A: def __init__(self, value): self.value = value def __repr__(self): return str(self.value) >>> a = A(10) # create a reference >>> d = weakref.WeakValueDictionary() >>> d['primary'] = a # does not create a reference >>> d['primary'] # fetch the object if it is still alive 10 >>> del a # remove the one reference >>> gc.collect() # make garbage collection run right away 0 >>> d['primary'] = a # the entry was automatically removed Traceback (most recent call last): File "<pyshell#108>", line 1, in -toplevel- d['primary'] = a # does not create a reference NameError: name 'a' is not defined Tools for working with lists ---------------------------- Many data structure needs can be met with the built-in lists. However, sometimes there is a need for alternative implementations with different performance trades-offs. The array module provides an array() object that is like a list that stores only homongenous data but stores it more compactly. The following example shows an array of numbers stored as two byte unsigned binary numbers (typecode "H") rather than the usual 16 bytes per entry for regular lists of python int objects:: >>> from array import array >>> a = array('H', [4000, 10, 700, 22227]) >>> a[1:3] array('H', [10, 700]) The collections module provides a deque() object that is like a list with faster appends and pops from the left side but slower lookups in the middle. These objects are well suited for implementing queues and breadth first tree searches:: >>> from collections import deque >>> d = deque(["task1", "task2", "task3"]) >>> d.append("task4") >>> print "Handling task,", d.popleft() task1 unsearched = deque([initialnode]) def breadth_first_search(unsearched): node = unsearched.popleft() for m in gen_moves(node): if is_goal(m): return m unsearched.append(m) In addition to alternative list implementations, the library also offers other tools such as the bisect module with functions for manipulating sorted lists:: >>> import bisect >>> scores = [(100, 'perl'), (200, 'tcl'), (400, 'lua'), (500, 'python')] >>> bisect.insort(scores, (300, 'ruby')) >>> scores [(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500, 'python')] The heapq module provides functions for implementing heaps based on regular lists. The lowest valued entry is always kept at position zero. This is useful for applications which repeatedly access the smallest element but do not want to run a full list sort:: >>> from heapq import heapify, heappop, heappush >>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0] >>> heapify(data) # rerrange the list into heap order >>> heappush(data, -5) # add a new entry >>> [heappop(data) for i in range(3)] # fetch the three smallest entries [-5, 0, 1] [??? Should the tempfile module be in the tour ???] ---------------------------------------------------

4 3

RE: [Python-Dev] Why aren't more things weak referencable
by Delaney, Timothy C (Timothy) 30 May '04

30 May '04

Guido van Rossum wrote: >> For strings at least, perhaps it is time to bite the bullet and >> include weak reference support directly. Weak reference support ups >> the per string memory overhead from five words (ob_type, ob_refcnt, >> ob_size, ob_shash, ob_sstate) to six. The whole concept of weak >> dictionaries is much more useful when strings can be used as keys >> and/or values. > > Hmm... it is a high price to pay to add another word (*and* some extra > code at dealloc time!) to every string object when very few apps need > them and strings are about the most common data type. And since > they're immutable, what's the point of having weak refs to strings in > the first place? (Note that the original poster asked about > *subclasses* of strings.) Perhaps there should be a standard weakly-referenceable proxy whose sole purpose in life would be to wrap objects that can't be directly weakly referenced. Kind of the opposite of ProxyTypes - it would have an internal strong reference to the object. This could be done transparently as well - if creating an instance of ReferenceType would raise a TypeError, we take a hit and wrap the object. Tim Delaney

1 0