Comments are invited on the attached draft for a second overview of the standard library. The first overview of basic modules at http://docs.python.org/tut/node12.html was well received and seemed to get people off to a running start. The primary goal of this follow-up is to reduce the transition time for a programmer to start writing real applications using Python. A second goal is give information to people who work through tutorials to determine the capabilities of various languages. A third goal is provide a jumping-off point for someone learning brand new concepts such as weak references or threading. The current docs merely describe the cockpit controls and assume that you already know how to fly an aircraft. Raymond ========================================================= Brief Tour of Modules in the Standard Libary (part II) This second tour covers more advanced modules that support professional programming needs. Their use rarely arises for small scripts. Output Formatting ----------------- The repr module provides an version of repr() for abbreviated displays of large or deeply nested containers:: >>> import repr >>> repr.repr(set('supercalafraglisticexpallidoches')) "set(['a', 'c', 'd', 'e', 'f', 'g', ...])" The pprint module offers more sophisticated control over printing both built-in and user defined objects in a way that is readable by the interpreter. When the result is longer than one line, the "pretty printer" adds line breaks and indentation to more clearly reveal data structure:: >>> import pprint >>> t = [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta', ... 'yellow'], 'blue']]] ... >>> pprint.pprint(t, width=30) [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta', 'yellow'], 'blue']]] The textwrap module formats paragraphs of text to fit a given screen width:: >>> import textwrap >>> doc = """The wrap() method is just like fill() except that it returns ... a list of strings instead of one big string with newlines to separate ... the wrapped lines.""" ... >>> print textwrap.fill(doc, width=40) The wrap() method is just like fill() except that it returns a list of strings instead of one big string with newlines to separate the wrapped lines. The locale module accesses a database of cultural specific data formats. The grouping attribute of locale's format function provides a direct way of formatting numbers with group separators. >>> import locale >>> locale.setlocale(locale.LC_ALL, 'English_United States.1252') 'English_United States.1252' >>> conv = locale.localeconv() # get a mapping of conventions >>> x = 1234567.8 >>> locale.format("%d", x, grouping=True) '1,234,567' >>> locale.format("%s%.*f", (conv['currency_symbol'], ... conv['int_frac_digits'], x), grouping=True) '$1,234,567.80' Working with Binary Data Record Layouts --------------------------------------- The struct module provides pack() and unpack() functions for working with variable length binary record formats. The following example shows how to loop through header information in a ZIP file (with pack codes "H" and "L" representing two and four byte numbers respectively):: import struct data = open('myfile.zip', 'rb').read() start = 0 for i in range(3): # show the first 3 file headers start += 14 fields = struct.unpack('LLLHH', data[start:start+16]) crc32, comp_size, uncomp_size, filenamesize, extra_size = fields start += 16 filename = data[start:start+filenamesize] start += filenamesize extra = data[start:start+extra_size] print filename, hex(crc32), comp_size, uncomp_size start += extra_size + comp_size # skip to the next header Multi-threading --------------- Threading is a technique for decoupling tasks which are not sequentially dependent. Python threads are driven by the operating system and run in a single process and share memory space in a single interpreter. Threads can be used to improve the responsiveness of applications that accept user input while other tasks run in the background. The following code shows how the high level threading module can run tasks in background while the main program continues to run:: import threading, zipfile class AsyncZip(threading.Thread): def __init__(self, infile, outfile): Thread.__init__(self) self.infile = infile self.outfile = outfile def run(self): f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED) f.write(self.infile) f.close() print 'Finished background zip of: ', self.infile AsyncZip('mydata.txt', 'myarchive.zip').start() print 'The main program continues to run' The principal challenge of multi-thread applications is coordinating threads that share data or other resources. To that end, the threading module provides a number of synchronization primitives including locks, events, condition variables, and semaphores. While those tools are powerful, minor design errors can result in problems that are difficult to reproduce. A simpler and more robust approach to task coordination is concentrating all access to a resource in a single thread and then using the Queue module to feed that thread with requests from other threads. Applications that use Queue objects for inter-thread communication and coordination tend to be easier to design, more readable, and more reliable. Logging ------- The logging module offers a full featured and flexible logging system. At its simplest, log messages are sent to a file or to sys.stderr. Other options include routing messages through email, datagrams, sockets, or to an HTTP Server. Messages are assigned priorities including DEBUG, INFO, WARNING, ERROR, and CRITICAL. Filtering allows messages to be directed based on their priority (for example: email only critcal messages, ignore debug messages, and write everything else to a file): [XXX add example] Weak References --------------- Python does automatic memory management (reference counting for most objects and garbage collection to eliminate cycles). The memory is freed shortly after the last reference to it has been eliminated. This approach works fine for most applications but occasionally there is a need to track objects only as long as they are being used by something else. Unfortunately, just tracking them creates a reference that makes them permanent. The weakref module provides tools tracking objects without creating a reference. When the object is no longer needed, it is automatically removed from a weakref table and a callback is triggered for weakref objects. Typically applications include caching objects that are expensive to create: >>> import weakref, gc >>> class A: def __init__(self, value): self.value = value def __repr__(self): return str(self.value) >>> a = A(10) # create a reference >>> d = weakref.WeakValueDictionary() >>> d['primary'] = a # does not create a reference >>> d['primary'] # fetch the object if it is still alive 10 >>> del a # remove the one reference >>> gc.collect() # make garbage collection run right away 0 >>> d['primary'] = a # the entry was automatically removed Traceback (most recent call last): File "", line 1, in -toplevel- d['primary'] = a # does not create a reference NameError: name 'a' is not defined Tools for working with lists ---------------------------- Many data structure needs can be met with the built-in lists. However, sometimes there is a need for alternative implementations with different performance trades-offs. The array module provides an array() object that is like a list that stores only homongenous data but stores it more compactly. The following example shows an array of numbers stored as two byte unsigned binary numbers (typecode "H") rather than the usual 16 bytes per entry for regular lists of python int objects:: >>> from array import array >>> a = array('H', [4000, 10, 700, 22227]) >>> a[1:3] array('H', [10, 700]) The collections module provides a deque() object that is like a list with faster appends and pops from the left side but slower lookups in the middle. These objects are well suited for implementing queues and breadth first tree searches:: >>> from collections import deque >>> d = deque(["task1", "task2", "task3"]) >>> d.append("task4") >>> print "Handling task,", d.popleft() task1 unsearched = deque([initialnode]) def breadth_first_search(unsearched): node = unsearched.popleft() for m in gen_moves(node): if is_goal(m): return m unsearched.append(m) In addition to alternative list implementations, the library also offers other tools such as the bisect module with functions for manipulating sorted lists:: >>> import bisect >>> scores = [(100, 'perl'), (200, 'tcl'), (400, 'lua'), (500, 'python')] >>> bisect.insort(scores, (300, 'ruby')) >>> scores [(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500, 'python')] The heapq module provides functions for implementing heaps based on regular lists. The lowest valued entry is always kept at position zero. This is useful for applications which repeatedly access the smallest element but do not want to run a full list sort:: >>> from heapq import heapify, heappop, heappush >>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0] >>> heapify(data) # rerrange the list into heap order >>> heappush(data, -5) # add a new entry >>> [heappop(data) for i in range(3)] # fetch the three smallest entries [-5, 0, 1] [??? Should the tempfile module be in the tour ???] ---------------------------------------------------