[Python-Dev] Standard Library Tour Part II

Raymond Hettinger raymond.hettinger at verizon.net
Mon May 24 19:13:44 EDT 2004


Comments are invited on the attached draft for a second overview of the
standard library.

The first overview of basic modules at
http://docs.python.org/tut/node12.html was well received and seemed to
get people off to a running start.  

The primary goal of this follow-up is to reduce the transition time for
a programmer to start writing real applications using Python.  

A second goal is give information to people who work through tutorials
to determine the capabilities of various languages.

A third goal is provide a jumping-off point for someone learning brand
new concepts such as weak references or threading.  The current docs
merely describe the cockpit controls and assume that you already know
how to fly an aircraft.


Raymond

=========================================================

Brief Tour of Modules in the Standard Libary (part II)


This second tour covers more advanced modules that support professional
programming needs.  Their use rarely arises for small scripts.


Output Formatting
-----------------

The repr module provides an version of repr() for abbreviated displays
of large or deeply nested containers::

    >>> import repr   
    >>> repr.repr(set('supercalafraglisticexpallidoches'))
    "set(['a', 'c', 'd', 'e', 'f', 'g', ...])"

The pprint module offers more sophisticated control over printing both
built-in and user defined objects in a way that is readable by the
interpreter.  When the result is longer than one line, the "pretty
printer"
adds line breaks and indentation to more clearly reveal data structure::

    >>> import pprint
    >>> t = [[[['black', 'cyan'], 'white', ['green', 'red']],
[['magenta',
    ...     'yellow'], 'blue']]]
    ...
    >>> pprint.pprint(t, width=30)
    [[[['black', 'cyan'],
       'white',
       ['green', 'red']],
      [['magenta', 'yellow'],
       'blue']]]

The textwrap module formats paragraphs of text to fit a given screen
width::

    >>> import textwrap
    >>> doc = """The wrap() method is just like fill() except that it
returns
    ... a list of strings instead of one big string with newlines to
separate
    ... the wrapped lines."""
    ...
    >>> print textwrap.fill(doc, width=40)
    The wrap() method is just like fill()
    except that it returns a list of strings
    instead of one big string with newlines
    to separate the wrapped lines.

The locale module accesses a database of cultural specific data formats.
The grouping attribute of locale's format function provides a direct way
of formatting numbers with group separators.

    >>> import locale
    >>> locale.setlocale(locale.LC_ALL, 'English_United States.1252')
    'English_United States.1252'
    >>> conv = locale.localeconv()     # get a mapping of conventions
    >>> x = 1234567.8
    >>> locale.format("%d", x, grouping=True)
    '1,234,567'
    >>> locale.format("%s%.*f", (conv['currency_symbol'],
    ...	      conv['int_frac_digits'], x), grouping=True)
    '$1,234,567.80'


Working with Binary Data Record Layouts
---------------------------------------

The struct module provides pack() and unpack() functions for working
with variable length binary record formats.  The following example shows
how to loop through header information in a ZIP file (with pack codes
"H" and "L" representing two and four byte numbers respectively)::

    import struct
    data = open('myfile.zip', 'rb').read()
    start = 0
    for i in range(3):                      # show the first 3 file
headers
        start += 14
        fields = struct.unpack('LLLHH', data[start:start+16])
        crc32, comp_size, uncomp_size, filenamesize, extra_size =
fields

        start += 16
        filename = data[start:start+filenamesize]
        start += filenamesize
        extra = data[start:start+extra_size]
        print filename, hex(crc32), comp_size, uncomp_size

        start += extra_size + comp_size     # skip to the next header


Multi-threading
---------------

Threading is a technique for decoupling tasks which are not sequentially
dependent.  Python threads are driven by the operating system and run
in a single process and share memory space in a single interpreter.

Threads can be used to improve the responsiveness of applications
that accept user input while other tasks run in the background.
The following code shows how the high level threading module can run
tasks in background while the main program continues to run::

    import threading, zipfile

    class AsyncZip(threading.Thread):
        def __init__(self, infile, outfile):
            Thread.__init__(self)        
            self.infile = infile
            self.outfile = outfile
        def run(self):
            f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED)
            f.write(self.infile)
            f.close()
            print 'Finished background zip of: ', self.infile

    AsyncZip('mydata.txt', 'myarchive.zip').start()
    print 'The main program continues to run'

The principal challenge of multi-thread applications is coordinating
threads that share data or other resources.  To that end,
the threading module provides a number of synchronization primitives
including locks, events, condition variables, and semaphores.

While those tools are powerful, minor design errors can result in
problems that are difficult to reproduce.  A simpler and more robust
approach to task coordination is concentrating all access to a resource
in a single thread and then using the Queue module to feed that thread
with requests from other threads.  Applications that use Queue objects
for
inter-thread communication and coordination tend to be easier to design,
more readable, and more reliable.


Logging
-------

The logging module offers a full featured and flexible logging system.
At its simplest, log messages are sent to a file or to sys.stderr.
Other options include routing messages through email, datagrams,
sockets, or to an HTTP Server.  Messages are assigned priorities
including DEBUG, INFO, WARNING, ERROR, and CRITICAL.  Filtering allows
messages to be directed based on their priority (for example:
email only critcal messages, ignore debug messages, and write everything
else to a file):

    [XXX add example]
                           

Weak References
---------------

Python does automatic memory management (reference counting for most
objects and garbage collection to eliminate cycles).  The memory is
freed shortly after the last reference to it has been eliminated.

This approach works fine for most applications but occasionally there
is a need to track objects only as long as they are being used by
something else.  Unfortunately, just tracking them creates a reference
that makes them permanent.  The weakref module provides tools tracking
objects without creating a reference.  When the object is no longer
needed, it is automatically removed from a weakref table and a callback
is triggered for weakref objects.  Typically applications include
caching objects that are expensive to create:

    >>> import weakref, gc
    >>> class A:
            def __init__(self, value):
                    self.value = value
            def __repr__(self):
                    return str(self.value)

            
    >>> a = A(10)			# create a reference
    >>> d = weakref.WeakValueDictionary()
    >>> d['primary'] = a      # does not create a reference
    >>> d['primary']		# fetch the object if it is still alive
    10
    >>> del a			# remove the one reference
    >>> gc.collect()          # make garbage collection run right away
    0
    >>> d['primary'] = a      # the entry was automatically removed

    Traceback (most recent call last):
      File "<pyshell#108>", line 1, in -toplevel-
        d['primary'] = a            # does not create a reference
    NameError: name 'a' is not defined


Tools for working with lists
----------------------------

Many data structure needs can be met with the built-in lists.  However,
sometimes there is a need for alternative implementations with different
performance trades-offs.

The array module provides an array() object that is like a list that
stores
only homongenous data but stores it more compactly.  The following
example shows an array of numbers stored as two byte unsigned binary
numbers (typecode "H") rather than the usual 16 bytes per entry
for regular lists of python int objects::

    >>> from array import array
    >>> a = array('H', [4000, 10, 700, 22227])
    >>> a[1:3]
    array('H', [10, 700])

The collections module provides a deque() object that is like a list
with
faster appends and pops from the left side but slower lookups in the
middle.
These objects are well suited for implementing queues and breadth first
tree searches::

    >>> from collections import deque
    >>> d = deque(["task1", "task2", "task3"])
    >>> d.append("task4")
    >>> print "Handling task,", d.popleft()
    task1

    unsearched = deque([initialnode])
    def breadth_first_search(unsearched):
        node = unsearched.popleft()
        for m in gen_moves(node):
            if is_goal(m):
                return m
            unsearched.append(m)

In addition to alternative list implementations, the library also offers
other
tools such as the bisect module with functions for manipulating sorted
lists::

    >>> import bisect
    >>> scores = [(100, 'perl'), (200, 'tcl'), (400, 'lua'), (500,
'python')]
    >>> bisect.insort(scores, (300, 'ruby'))
    >>> scores
    [(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500,
'python')]

The heapq module provides functions for implementing heaps based on
regular
lists.  The lowest valued entry is always kept at position zero.  This
is useful for applications which repeatedly access the smallest element
but do not want to run a full list sort::

    >>> from heapq import heapify, heappop, heappush
    >>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0]
    >>> heapify(data)                      # rerrange the list into heap
order
    >>> heappush(data, -5)                 # add a new entry
    >>> [heappop(data) for i in range(3)]  # fetch the three smallest
entries
    [-5, 0, 1]    


[??? Should the tempfile module be in the tour ???]
--------------------------------------------------- 




More information about the Python-Dev mailing list