[Python-Dev] Tutorial: Brief Introduction to the Standard Libary

[Raymond Hettinger]

Thank you everyone for the ideas on what to include and exclude from
the new tutorial section.

Attached is a revised draft.  Comments, suggestions, nitpicks,
complaints, and accolades are welcome.



Raymond Hettinger



-------------- next part --------------
Brief Tour of some Standard Library Modules



Operating System Interface

The os module provides many functions for interacting with the operating system:

>>> import os
>>> os.system('copy /data/mydata.fil /backup/mydata.fil')
0
>>> os.getcwd() 
'C:\\Python24'
>>> os.chdir('/server/accesslogs')

Be sure to use the "import os" style instead of "from os import *".  This will keep
os.open() from shadowing __builtin__.open()  which operates much differently.


File Wildcards

The glob module provides a function for making file lists from directory wildcard searches:

>>> glob.glob('*.py')
['primes.py', 'random.py', 'quote.py']


Command Line Arguments

Common utility scripts often invoke processing command line arguments.
These arguments are stored in the sys module's argv attribute as a list.
For instance the following output results from running
"python demo.py one two three" at the command line:

>>> import sys
>>> print sys.argv[]
['demo.py', 'one', 'two', 'three']

The getopt module processes sys.argv using the conventions of the Unix
getopt() function:

>>> import getopt
>>> # sys.argv is ['myprog.py', '-a', '-b', '-cfoo', '-d', 'bar', 'a1', 'a2']
>>> optlist, args = getopt.getopt(sys.argv[1:], 'abc:d:')
>>> optlist
[('-a', ''), ('-b', ''), ('-c', 'foo'), ('-d', 'bar')]
>>> args
['a1', 'a2']

More powerful and flexible command line processing is provided by
the optparse module.


Error Output Redirection and Program Termination

The sys module also has attributes for stdin, stdout, and stderr.  The latter
is useful for emitting warnings and error messages to make them visible
even when stdout has been redirected:

>>> sys.stderr.write('Warning, log file not found starting a new one')
Warning, log file not found starting a new one

The most direct way to terminate a script is to use sys.exit().


String Pattern Matching

The re module provides regular expression tools for advanced string processing.
When only simple capabilities are needed, string methods are preferred
because they are easier to read and debug.  For more sophisticated applications,
regular expressions can provide succinct, optimized solutions:

>>> import re
>>> re.findall(r'\bf[a-z]*', 'which foot or hand fell fastest')
['foot', 'fell', 'fastest']
>>> re.sub(r'(\b[a-z]+) \1', r'\1', 'cat in the the hat')
'cat in the hat'


Math Library

The math module gives access to the underlying C library functions for floating point math:

>>> import math
>>> math.cos(math.pi / 4.0)
0.70710678118654757
>>> math.log(1024, 2)
10.0

The random module provides tools for making random selections:

>>> import random
>>> random.choice(['apple', 'pear', 'banana'])
'apple'
>>> random.sample(xrange(100), 10)   # sampling without replacement
[30, 83, 16, 4, 8, 81, 41, 50, 18, 33]
>>> random.random()    # random float
0.17970987693706186
>>> random.randrange(6)    # random integer chosen from range(6)
4   


Internet Access

There are a number of modules for accessing the internet and processing
internet protocols. Two of the simplest are urllib2 for retrieving data
from urls and smtplib for sending mail:

>>> import urllib
>>> for line in urllib2.urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'):
... if 'EST' in line:      # look for Eastern Standard Time
...     print line
    
<BR>Nov. 25, 09:43:32 PM EST

>>> import smtplib
>>> server = smtplib.SMTP('localhost')
>>> server.sendmail('soothsayer at tmp.org', 'jceasar at tmp.org',
"""To: jceasar at tmp.org
From: soothsayer at tmp.org

Beware the Ides of March.
""")
>>> server.quit()


Dates and Times

The datetime module supplies classes for manipulating dates and times in both
simple and complex ways. While date and time arithmetic is supported, the focus
of the implementation is on efficient member extraction for output formatting
and manipulation.  The module also supports objects that are time zone aware.

# dates are easily constructed and formatted
>>> from datetime import date
>>> now = date.today()
>>> now
datetime.date(2003, 12, 2)
>>> now.strftime("%m-%d-%y or %d%b %Y is a %A on the %d day of %B")
'12-02-03 or 02Dec 2003 is a Tuesday on the 02 day of December'

# dates support calendar arithmetic
>>> birthday = date(1964, 7, 31)
>>> age = now - birthday
>>> age.days
14368


Data Compression

Common data archiving and compression formats are directly supported
by modules including : zlib, gzip, bz2, zipfile, and tar.

>>> import zlib
>>> s = 'witch which has which witches wrist watch'
>>> len(s)
41
>>> t = zlib.compress(s)
>>> len(t)
37
>>> zlib.decompress(t)
'witch which has which witches wrist watch'
>>> zlib.crc32(t)
-1438085031


Performance Measurement

Some Python users develop a deep interest in knowing the relative performance
between different approaches to the same problem.   Python provides a measurement
tool that answers those questions immediately.

For example, it may be tempting to use the tuple packing and unpacking feature
instead of the traditional approach to swapping arguments.  The timeit module
quickly demonstrates that the traditional approach is faster:

>>> from timeit import Timer
>>> dir(Timer)
>>> Timer('t=a; a=b; b=t', 'a=1; b=1').timeit()
0.60864915603680925
>>> Timer('a,b = b,a', 'a=1; b=1').timeit()
0.8625194857439773

In contrast to timeit's fine level of granularity, the profile and pstats modules
provide tools for identifying time critical sections of larger blocks of code.


Quality Control

One approach for developing high quality software is to write tests for each
function as it is developed and to run those tests frequently during the
development process.

The doctest module provides a tool for scanning a module and validating
tests embedded in a program's docstrings.  Test construction is as simple as
cutting-and-pasting a typical call along with its results into the docstring.
This improves the documentation by providing the user with an example
and it allows the doctest module to make sure the code remains true to the
documentation:

def average(values):
    """Computes the arithmetic mean of a list of numbers.

    >>> print average([20, 30, 70])
    40.0
    """
    return sum(values, 0.0) / len(values)

import doctest
doctest.testmod()   # automatically validate the embedded tests

The unittest module is not as effortless as the doctest module, but it allows
a more comprehensive set of tests to be maintained in a separate file:

import unittest

class TestStatisticalFunctions(unittest.TestCase):

    def test_average(self):
        self.assertEqual(average([20, 30, 70]), 40.0)
        self.assertEqual(round(average([1, 5, 7]), 1), 4.3)
        self.assertRaises(ZeroDivisionError, average, [])
        self.assertRaises(TypeError, average, 20, 30, 70)

unittest.main() # Calling from the command line invokes all tests


Batteries Included

Python has a "batteries included" philosophy.  The is best seen through the
sophisticated and robust capabilites of its larger packages:

* The xmlrpclib and SimpleXMLRPCServer modules make implementing remote
procedure calls into an almost trivial task.  Despite the names, no direct
knowledge or handling of XML is needed.

* The email package is a library for managing email messages, including MIME
and other RFC 2822-based message documents.  Unlike smtplib and poplib which
actually send and receive messages, the email package has a complete toolset
for building or decoding complex message structures (including attachments)
and for implementing internet encoding and header protocols.

* The xml.dom and xml.sax packages provide robust support for parsing this
popular data interchange format.  Likewise, the csv module supports direct
reads and writes in a common database format.  Together, these modules and
packages greatly simplify data interchange between python applications and
other tools.

* Internationalization is supported by a number of modules including gettext
and the codecs package.