Mailman 3 August 2003 - Python-announce-list

Unbrand America
by bill＠unbrandamerica.org Aug. 24, 2003

Aug. 24, 2003

In the coming months a black spot will pop up everywhere . . . on store windows and newspaper boxes, on gas pumps and supermarket shelves. Open a magazine or newspaper - it's there. It's on TV. It stains the logos and smears the nerve centers of the world's biggest, dirtiest corporations. This is the mark of the people who don't approve of Bush's plan to control the world, who don't want countries "liberated" without UN backing, who can't stand anymore neo-con bravado shoved down their throats. This is the mark of the people who want the Kyoto Protocol for the environment, who want the International Criminal Court for greater justice, who want a world where all nations, including the U.S.A., are free of weapons of mass destruction, and who pledge to take their country back. -- Some hot painters before the clean highway were solving around the open dorm.

1 0

Python Library & Extension FAQ
by amk＠nyman.amk.ca Aug. 24, 2003

Aug. 24, 2003

This is the last of the new FAQ files assembled from the old 240K FAQ. As usual, comments on existing answers and suggestions for new questions are welcome. --amk ==================================== Python Library and Extension FAQ ==================================== :Date: $Date: 2003/08/19 20:42:47 $ :Version: $Revision: 1.9 $ :Web site: http://www.python.org/ .. contents:: .. sectnum:: General Library Questions =============================== How do I find a module or application to perform task X? ------------------------------------------------------------- Check `the Library Reference <http://www.python.org/doc/lib/>`_ to see if there's a relevant standard library module. (Eventually you'll learn what's in the standard library and will able to skip this step.) Search the `Python Package Index <http://www.python.org/pypi>`_. Next, check the `Vaults of Parnassus <http://www.vex.net/parnassus/>`_, an older index of packages. Finally, try `Google <http://www.google.com>`_ or other Web search engine. Searching for "Python" plus a keyword or two for your topic of interest will usually find something helpful. Where is the math.py (socket.py, regex.py, etc.) source file? --------------------------------------------------------------------- If you can't find a source file for a module it may be a builtin or dynamically loaded module implemented in C, C++ or other compiled language. In this case you may not have the source file or it may be something like mathmodule.c, somewhere in a C source directory (not on the Python Path). There are (at least) three kinds of modules in Python: 1) modules written in Python (.py); 2) modules written in C and dynamically loaded (.dll, .pyd, .so, .sl, etc); 3) modules written in C and linked with the interpreter; to get a list of these, type:: import sys print sys.builtin_module_names How do I make a Python script executable on Unix? --------------------------------------------------------- You need to do two things: the script file's mode must be executable and the first line must begin with ``#!`` followed by the path of the Python interpreter. The first is done by executing ``chmod +x scriptfile`` or perhaps ``chmod 755 scriptfile``. The second can be done in a number of ways. The most straightforward way is to write :: #!/usr/local/bin/python as the very first line of your file, using the pathname for where the Python interpreter is installed on your platform. If you would like the script to be independent of where the Python interpreter lives, you can use the "env" program. Almost all Unix variants support the following, assuming the python interpreter is in a directory on the user's $PATH:: #! /usr/bin/env python *Don't* do this for CGI scripts. The $PATH variable for CGI scripts is often very minimal, so you need to use the actual absolute pathname of the interpreter. Occasionally, a user's environment is so full that the /usr/bin/env program fails; or there's no env program at all. In that case, you can try the following hack (due to Alex Rezinsky):: #! /bin/sh """:" exec python $0 ${1+"$@"} """ The minor disadvantage is that this defines the script's __doc__ string. However, you can fix that by adding :: __doc__ = """...Whatever...""" Is there a curses/termcap package for Python? ---------------------------------------------------- For Unix variants: The standard Python source distribution comes with a curses module in the Modules/ subdirectory, though it's not compiled by default (note that this is not available in the Windows distribution -- there is no curses module for Windows). The curses module supports basic curses features as well as many additional functions from ncurses and SYSV curses such as colour, alternative character set support, pads, and mouse support. This means the module isn't compatible with operating systems that only have BSD curses, but there don't seem to be any currently maintained OSes that fall into this category. For Windows: use `the consolelib module <http://effbot.org/zone/console-index.htm>`_. Is there an equivalent to C's onexit() in Python? -------------------------------------------------------- `The atexit module <http://www.python.org/doc/lib/module-atexit.html>`_ provides a register function that is similar to C's onexit. Why don't my signal handlers work? -------------------------------------------- The most common problem is that the signal handler is declared with the wrong argument list. It is called as :: handler(signum, frame) so it should be declared with two arguments:: def handler(signum, frame): ... Common tasks ================= How do I test a Python program or component? ---------------------------------------------------- Python comes with two testing frameworks. The `doctest module <http://www.python.org/doc/lib/module-doctest.html>`_ finds examples in the docstrings for a module and runs them, comparing the output with the expected output given in the docstring. The `unittest module <http://www.python.org/doc/lib/module-unittest.html>`_ is a fancier testing framework modelled on Java and Smalltalk testing frameworks. For testing, it helps to write the program so that it may be easily tested by using good modular design. Your program should have almost all functionality encapsulated in either functions or class methods -- and this sometimes has the surprising and delightful effect of making the program run faster (because local variable accesses are faster than global accesses). Furthermore the program should avoid depending on mutating global variables, since this makes testing much more difficult to do. The "global main logic" of your program may be as simple as :: if __name__=="__main__": main_logic() at the bottom of the main module of your program. Once your program is organized as a tractable collection of functions and class behaviours you should write test functions that exercise the behaviours. A test suite can be associated with each module which automates a sequence of tests. This sounds like a lot of work, but since Python is so terse and flexible it's surprisingly easy. You can make coding much more pleasant and fun by writing your test functions in parallel with the "production code", since this makes it easy to find bugs and even design flaws earlier. "Support modules" that are not intended to be the main module of a program may include a self-test of the module. :: if __name__ == "__main__": self_test() Even programs that interact with complex external interfaces may be tested when the external interfaces are unavailable by using "fake" interfaces implemented in Python. How do I create documentation from doc strings? ------------------------------------------------------- The `pydoc module <http://www.python.org/doc/lib/module-pydoc.html>`_ can create HTML from the doc strings in your Python source code. An alternative is `pythondoc <http://starship.python.net/crew/danilo/pythondoc/>`_. How do I get a single keypress at a time? ----------------------------------------------- For Unix variants:There are several solutions. It's straightforward to do this using curses, but curses is a fairly large module to learn. Here's a solution without curses:: import termios, fcntl, sys, os fd = sys.stdin.fileno() oldterm = termios.tcgetattr(fd) newattr = termios.tcgetattr(fd) newattr[3] = newattr[3] & ~termios.ICANON & ~termios.ECHO termios.tcsetattr(fd, termios.TCSANOW, newattr) oldflags = fcntl.fcntl(fd, fcntl.F_GETFL) fcntl.fcntl(fd, fcntl.F_SETFL, oldflags | os.O_NONBLOCK) try: while 1: try: c = sys.stdin.read(1) print "Got character", `c` except IOError: pass finally: termios.tcsetattr(fd, termios.TCSAFLUSH, oldterm) fcntl.fcntl(fd, fcntl.F_SETFL, oldflags) You need the ``termios`` and the ``fcntl`` module for any of this to work, and I've only tried it on Linux, though it should work elsewhere. In this code, characters are read and printed one at a time. ``termios.tcsetattr()`` turns off stdin's echoing and disables canonical mode. ``fcntl.fnctl()`` is used to obtain stdin's file descriptor flags and modify them for non-blocking mode. Since reading stdin when it is empty results in an ``IOError``, this error is caught and ignored. Threads ============= How do I program using threads? --------------------------------- Be sure to use `the threading module <http://www.python.org/doc/lib/module-threading.html>`_ and not the ``thread`` module. The ``threading`` module builds convenient abstractions on top of the low-level primitives provided by the ``thread`` module. Aahz has a set of slides from his threading tutorial that are helpful; see http://starship.python.net/crew/aahz/OSCON2001/. None of my threads seem to run: why? ------------------------------------------- As soon as the main thread exits, all threads are killed. Your main thread is running too quickly, giving the threads no time to do any work. A simple fix is to add a sleep to the end of the program that's long enough for all the threads to finish:: import threading, time def thread_task(name, n): for i in range(n): print name, i for i in range(10): T = threading.Thread(target=thread_task, args=(str(i), i)) T.run() time.sleep(10) # <----------------------------! But now (on many platforms) the threads don't run in parallel, but appear to run sequentially, one at a time! The reason is that the OS thread scheduler doesn't start a new thread until the previous thread is blocked. A simple fix is to add a tiny sleep to the start of the run function:: def thread_task(name, n): time.sleep(0.001) # <---------------------! for i in range(n): print name, i for i in range(10): T = threading.Thread(target=thread_task, args=(str(i), i)) T.run() time.sleep(10) Instead of trying to guess how long a ``time.sleep()`` delay will be enough, it's better to use some kind of semaphore mechanism. One idea is to use the `Queue module <http://www.python.org/doc/lib/module-Queue.html>`_ to create a queue object, let each thread append a token to the queue when it finishes, and let the main thread read as many tokens from the queue as there are threads. How do I parcel out work among a bunch of worker threads? ---------------------------------------------------------------- Use the `Queue module <http://www.python.org/doc/lib/module-Queue.html>`_ to create a queue containing a list of jobs. The ``Queue`` class maintains a list of objects with ``.put(obj)`` to add an item to the queue and ``.get()`` to return an item. The class will take care of the locking necessary to ensure that each job is handed out exactly once. Here's a trivial example:: import threading, Queue, time # The worker thread gets jobs off the queue. When the queue is empty, it # assumes there will be no more work and exits. # (Realistically workers will run until terminated.) def worker (): print 'Running worker' time.sleep(0.1) while True: try: arg = q.get(block=False) except Queue.Empty: print 'Worker', threading.currentThread(), print 'queue empty' break else: print 'Worker', threading.currentThread(), print 'running with argument', arg time.sleep(0.5) # Create queue q = Queue.Queue() # Start a pool of 5 workers for i in range(5): t = threading.Thread(target=worker, name='worker %i' % (i+1)) t.start() # Begin adding work to the queue for i in range(50): q.put(i) # Give threads time to run print 'Main thread sleeping' time.sleep(5) When run, this will produce the following output: Running worker Running worker Running worker Running worker Running worker Main thread sleeping Worker <Thread(worker 1, started)> running with argument 0 Worker <Thread(worker 2, started)> running with argument 1 Worker <Thread(worker 3, started)> running with argument 2 Worker <Thread(worker 4, started)> running with argument 3 Worker <Thread(worker 5, started)> running with argument 4 Worker <Thread(worker 1, started)> running with argument 5 ... Consult the module's documentation for more details; the ``Queue`` class provides a featureful interface. What kinds of global value mutation are thread-safe? ------------------------------------------------------------ A global interpreter lock (GIL) is used internally to ensure that only one thread runs in the Python VM at a time. In general, Python offers to switch among threads only between bytecode instructions; how frequently it switches can be set via ``sys.setcheckinterval()``. Each bytecode instruction and therefore all the C implementation code reached from each instruction is therefore atomic from the point of view of a Python program. In theory, this means an exact accounting requires an exact understanding of the PVM bytecode implementation. In practice, it means that operations on shared variables of builtin data types (ints, lists, dicts, etc) that "look atomic" really are. For example, the following operations are all atomic (L, L1, L2 are lists, D, D1, D2 are dicts, x, y are objects, i, j are ints):: L.append(x) L1.extend(L2) x = L[i] x = L.pop() L1[i:j] = L2 L.sort() x = y x.field = y D[x] = y D1.update(D2) D.keys() These aren't:: i = i+1 L.append(L[-1]) L[i] = L[j] D[x] = D[x] + 1 Operations that replace other objects may invoke those other objects' ``__del__`` method when their reference count reaches zero, and that can affect things. This is especially true for the mass updates to dictionaries and lists. When in doubt, use a mutex! Can't we get rid of the Global Interpreter Lock? -------------------------------------------------------- The Global Interpreter Lock (GIL) is often seen as a hindrance to Python's deployment on high-end multiprocessor server machines, because a multi-threaded Python program effectively only uses one CPU, due to the insistence that (almost) all Python code can only run while the GIL is held. Back in the days of Python 1.5, Greg Stein actually implemented a comprehensive patch set (the "free threading" patches) that removed the GIL and replaced it with fine-grained locking. Unfortunately, even on Windows (where locks are very efficient) this ran ordinary Python code about twice as slow as the interpreter using the GIL. On Linux the performance loss was even worse because pthread locks aren't as efficient. Since then, the idea of getting rid of the GIL has occasionally come up but nobody has found a way to deal with the expected slowdown, and users who don't use threads would not be happy if their code ran at half at the speed. Greg's free threading patch set has not been kept up-to-date for later Python versions. This doesn't mean that you can't make good use of Python on multi-CPU machines! You just have to be creative with dividing the work up between multiple *processes* rather than multiple *threads*. Judicious use of C extensions will also help; if you use a C extension to perform a time-consuming task, the extension can release the GIL while the thread of execution is in the C code and allow other threads to get some work done. It has been suggested that the GIL should be a per-interpreter-state lock rather than truly global; interpreters then wouldn't be able to share objects. Unfortunately, this isn't likely to happen either. It would be a tremendous amount of work, because many object implementations currently have global state. For example, small integers and short strings are cached; these caches would have to be moved to the interpreter state. Other object types have their own free list; these free lists would have to be moved to the interpreter state. And so on. And I doubt that it can even be done in finite time, because the same problem exists for 3rd party extensions. It is likely that 3rd party extensions are being written at a faster rate than you can convert them to store all their global state in the interpreter state. And finally, once you have multiple interpreters not sharing any state, what have you gained over running each interpreter in a separate process? Input and Output ========================= How do I delete a file? (And other file questions...) --------------------------------------------------------- Use ``os.remove(filename)`` or ``os.unlink(filename)``; for documentation, see `the POSIX module <http://www.python.org/doc/lib/module-posix.html>`_. The two functions are identical; ``unlink()`` is simply the name of the Unix system call for this function. To remove a directory, use ``os.rmdir()``; use ``os.mkdir()`` to create one. ``os.makedirs(path)`` will create any intermediate directories in ``path`` that don't exist. ``os.removedirs(path)`` will remove intermediate directories as long as they're empty; if you want to delete an entire directory tree and its contents, use ``shutil.rmtree()``. To rename a file, use ``os.rename(old_path, new_path)``. To truncate a file, open it using ``f = open(filename, "r+")``, and use ``f.truncate(offset)``; offset defaults to the current seek position. There's also ```os.ftruncate(fd, offset)`` for files opened with ``os.open()``, where ``fd`` is the file descriptor (a small integer). The ``shutil`` module also contains a number of functions to work on files including ``copyfile``, ``copytree``, and ``rmtree``. How do I copy a file? ----------------------------- The ``shutil`` module contains a ``copyfile()`` function. Note that on MacOS 9 it doesn't copy the resource fork and Finder info. How do I read (or write) binary data? --------------------------------------------- or complex data formats, it's best to use `the struct module <http://www.python.org/doc/lib/module-struct.html>`_. It allows you to take a string containing binary data (usually numbers) and convert it to Python objects; and vice versa. For example, the following code reads two 2-byte integers and one 4-byte integer in big-endian format from a file:: import struct f = open(filename, "rb") # Open in binary mode for portability s = f.read(8) x, y, z = struct.unpack(">hhl", s) The '>' in the format string forces big-endian data; the letter 'h' reads one "short integer" (2 bytes), and 'l' reads one "long integer" (4 bytes) from the string. For data that is more regular (e.g. a homogeneous list of ints or thefloats), you can also use `the array module <http://www.python.org/doc/lib/module-array.html>`_. I can't seem to use os.read() on a pipe created with os.popen(); why? ------------------------------------------------------------------------ ``os.read()`` is a low-level function which takes a file descriptor, a small integer representing the opened file. ``os.popen()`` creates a high-level file object, the same type returned by the builtin ``open()`` function. Thus, to read n bytes from a pipe p created with ``os.popen()``, you need to use ``p.read(n)``. How do I run a subprocess with pipes connected to both input and output? -------------------------------------------------------------------------------- Use `the popen2 module <http://www.python.org/doc/lib/module-popen2.html>`_. For example:: import popen2 fromchild, tochild = popen2.popen2("command") tochild.write("input\n") tochild.flush() output = fromchild.readline() Warning: in general it is unwise to do this because you can easily cause a deadlock where your process is blocked waiting for output from the child while the child is blocked waiting for input from you. This can be caused because the parent expects the child to output more text than it does, or it can be caused by data being stuck in stdio buffers due to lack of flushing. The Python parent can of course explicitly flush the data it sends to the child before it reads any output, but if the child is a naive C program it may have been written to never explicitly flush its output, even if it is interactive, since flushing is normally automatic. Note that a deadlock is also possible if you use ``popen3`` to read stdout and stderr. If one of the two is too large for the internal buffer (increasing the buffer size does not help) and you ``read()`` the other one first, there is a deadlock, too. Note on a bug in popen2: unless your program calls ``wait()`` or ``waitpid()``, finished child processes are never removed, and eventually calls to popen2 will fail because of a limit on the number of child processes. Calling ``os.waitpid`` with the ``os.WNOHANG`` option can prevent this; a good place to insert such a call would be before calling ``popen2`` again. In many cases, all you really need is to run some data through a command and get the result back. Unless the amount of data is very large, the easiest way to do this is to write it to a temporary file and run the command with that temporary file as input. The `standard module tempfile <http://www.python.org/doc/lib/module-tempfile.html>`_ exports a ``mktemp()`` function to generate unique temporary file names. :: import tempfile import os class Popen3: """ This is a deadlock-safe version of popen that returns an object with errorlevel, out (a string) and err (a string). (capturestderr may not work under windows.) Example: print Popen3('grep spam','\n\nhere spam\n\n').out """ def __init__(self,command,input=None,capturestderr=None): outfile=tempfile.mktemp() command="( %s ) > %s" % (command,outfile) if input: infile=tempfile.mktemp() open(infile,"w").write(input) command=command+" <"+infile if capturestderr: errfile=tempfile.mktemp() command=command+" 2>"+errfile self.errorlevel=os.system(command) >> 8 self.out=open(outfile,"r").read() os.remove(outfile) if input: os.remove(infile) if capturestderr: self.err=open(errfile,"r").read() os.remove(errfile) Note that many interactive programs (e.g. vi) don't work well with pipes substituted for standard input and output. You will have to use pseudo ttys ("ptys") instead of pipes. Or you can use a Python interface to Don Libes' "expect" library. A Python extension that interfaces to expect is called "expy" and available from http://expectpy.sourceforge.net. A pure Python solution that works like expect is ` pexpect <http://pexpect.sourceforge.net>`_. How do I access the serial (RS232) port? ------------------------------------------------ For Win32, POSIX (Linux, BSD, etc.), Jython: http://pyserial.sourceforge.net For Unix, see a Usenet post by Mitch Chapman: http://groups.google.com/groups?selm=34A04430.CF9@ohioee.com Why doesn't closing sys.stdout (stdin, stderr) really close it? ----------------------------------------------------------------------- Python file objects are a high-level layer of abstraction on top of C streams, which in turn are a medium-level layer of abstraction on top of (among other things) low-level C file descriptors. For most file objects you create in Python via the builtin ``file`` constructor, ``f.close()`` marks the Python file object as being closed from Python's point of view, and also arranges to close the underlying C stream. This also happens automatically in f's destructor, when f becomes garbage. But stdin, stdout and stderr are treated specially by Python, because of the special status also given to them by C. Running ``sys.stdout.close()`` marks the Python-level file object as being closed, but does *not* close the associated C stream. To close the underlying C stream for one of these three, you should first be sure that's what you really want to do (e.g., you may confuse extension modules trying to do I/O). If it is, use os.close:: os.close(0) # close C's stdin stream os.close(1) # close C's stdout stream os.close(2) # close C's stderr stream Network/Internet Programming ======================================= What WWW tools are there for Python? -------------------------------------------- See the chapters titled `"Internet Protocols and Support" <http://www.python.org/doc/lib/internet.html>`_ and `"Internet Data Handling" <http://www.python.org/doc/lib/netdata.html>`_ in the Library Reference Manual. Python has many modules that will help you build server-side and client-side web systems. A summary of available frameworks is maintained by Paul Boddie at http://www.python.org/cgi-bin/moinmoin/WebProgramming . Cameron Laird maintains a useful set of pages about Python web technologies at http://starbase.neosoft.com/~claird/comp.lang.python/web_python.html The `Web Programming topic guide <http://www.python.org/topic/web/>`_ also points to many useful resources. How can I mimic CGI form submission (METHOD=POST)? ---------------------------------------------------------- I would like to retrieve web pages that are the result of POSTing a form. Is there existing code that would let me do this easily? Yes. Here's a simple example that uses httplib:: #!/usr/local/bin/python import httplib, sys, time ### build the query string qs = "First=Josephine&MI=Q&Last=Public" ### connect and send the server a path httpobj = httplib.HTTP('www.some-server.out-there', 80) httpobj.putrequest('POST', '/cgi-bin/some-cgi-script') ### now generate the rest of the HTTP headers... httpobj.putheader('Accept', '*/*') httpobj.putheader('Connection', 'Keep-Alive') httpobj.putheader('Content-type', 'application/x-www-form-urlencoded') httpobj.putheader('Content-length', '%d' % len(qs)) httpobj.endheaders() httpobj.send(qs) ### find out what the server said in response... reply, msg, hdrs = httpobj.getreply() if reply != 200: sys.stdout.write(httpobj.getfile().read()) Note that in general for URL-encoded POST operations, query strings must be quoted by using ``urllib.quote()``. For example to send name="Guy Steele, Jr.":: >>> from urllib import quote >>> x = quote("Guy Steele, Jr.") >>> x 'Guy%20Steele,%20Jr.' >>> query_string = "name="+x >>> query_string 'name=Guy%20Steele,%20Jr.' What module should I use to help with generating HTML? -------------------------------------------------------------- There are many different modules available: * HTMLgen is a class library of objects corresponding to all the HTML 3.2 markup tags. It's used when you are writing in Python and wish to synthesize HTML pages for generating a web or for CGI forms, etc. * DocumentTemplate and Zope Page Templates are two different systems that are part of Zope. * Quixote's PTL uses Python syntax to assemble strings of text. Consult the `Web Programming topic guide <http://www.python.org/topic/web/>`_ for more links. How do I send mail from a Python script? ------------------------------------------------ Use `the standard library module smtplib <http://www.python.org/doc/lib/module-smtplib.html>`_. Here's a very simple interactive mail sender that uses it. This method will work on any host that supports an SMTP listener. :: import sys, smtplib fromaddr = raw_input("From: ") toaddrs = raw_input("To: ").split(',') print "Enter message, end with ^D:" msg = '' while 1: line = sys.stdin.readline() if not line: break msg = msg + line # The actual mail send server = smtplib.SMTP('localhost') server.sendmail(fromaddr, toaddrs, msg) server.quit() A Unix-only alternative uses sendmail. The location of the sendmail program varies between systems; sometimes it is ``/usr/lib/sendmail``, sometime ``/usr/sbin/sendmail``. The sendmail manual page will help you out. Here's some sample code:: SENDMAIL = "/usr/sbin/sendmail" # sendmail location import os p = os.popen("%s -t -i" % SENDMAIL, "w") p.write("To: receiver(a)example.com\n") p.write("Subject: test\n") p.write("\n") # blank line separating headers from body p.write("Some text\n") p.write("some more text\n") sts = p.close() if sts != 0: print "Sendmail exit status", sts How do I avoid blocking in the connect() method of a socket? -------------------------------------------------------------------------- The select module is commonly used to help with asynchronous I/O on sockets. To prevent the TCP connect from blocking, you can set the socket to non-blocking mode. Then when you do the ``connect()``, you will either connect immediately (unlikely) or get an exception that contains the error number as ``.errno``. ``errno.EINPROGRESS`` indicates that the connection is in progress, but hasn't finished yet. Different OSes will return different values, so you're going to have to check what's returned on your system. You can use the ``connect_ex()`` method to avoid creating an exception. It will just return the errno value. To poll, you can call ``connect_ex()`` again later -- 0 or ``errno.EISCONN`` indicate that you're connected -- or you can pass this socket to select to check if it's writeable. Databases ===================== Are there any interfaces to database packages in Python? ---------------------------------------------------------------- Yes. Python 2.3 includes the ``bsddb`` package which provides an interface to the `BerkeleyDB <http://www.python.org/doc/lib/modules-bsddb.html>`_ library. Interfaces to disk-based hashes such as `DBM <http://www.python.org/doc/lib/modules-dbm.html>`_ and `GDBM <http://www.python.org/doc/lib/modules-gdbm.html>`_ are also included with standard Python. Support for most relational databases is available. See the `Database Topic Guide <http://www.python.org/topics/database>`_ for details. How do you implement persistent objects in Python? ------------------------------------------------------------ The `pickle library module <http://www.python.org/doc/lib/modules-pickle.html>`_ solves this in a very general way (though you still can't store things like open files, sockets or windows), and the `shelve library module <http://www.python.org/doc/lib/modules-shelve.html>`_ uses pickle and (g)dbm to create persistent mappings containing arbitrary Python objects. For better performance, you can use `the cPickle module <http://www.python.org/doc/lib/modules-cPickle.html>`_. A more awkward way of doing things is to use pickle's little sister, marshal. `The marshal module <http://www.python.org/doc/lib/modules-marshal.html>`_ provides very fast ways to store noncircular basic Python types to files and strings, and back again. Although marshal does not do fancy things like store instances or handle shared references properly, it does run extremely fast. For example loading a half megabyte of data may take less than a third of a second. This often beats doing something more complex and general such as using gdbm with pickle/shelve. Why is cPickle so slow? -------------------------------- The default format used by the pickle module is a slow one that results in readable pickles. Making it the default, but it would break backward compatibility:: largeString = 'z' * (100 * 1024) myPickle = cPickle.dumps(largeString, protocol=1) If my program crashes with a bsddb (or anydbm) database open, it gets corrupted. How come? -------------------------------------------------------------------------------------------------- Databases opened for write access with the bsddb module (and often by the anydbm module, since it will preferentially use bsddb) must explicitly be closed using the ``.close()`` method of the database. The underlying library caches database contents which need to be converted to on-disk form and written. If you have initialized a new bsddb database but not written anything to it before the program crashes, you will often wind up with a zero-length file and encounter an exception the next time the file is opened. I tried to open Berkeley DB file, but bsddb produces bsddb.error: (22, 'Invalid argument'). Help! How can I restore my data? ------------------------------------------------------------------------------------------------------------------------------------ Don't panic! Your data is probably intact. The most frequent cause for the error is that you tried to open an earlier Berkeley DB file with a later version of the Berkeley DB library. Many Linux systems now have all three versions of Berkeley DB available. If you are migrating from version 1 to a newer version use db_dump185 to dump a plain text version of the database. If you are migrating from version 2 to version 3 use db2_dump to create a plain text version of the database. In either case, use db_load to create a new native database for the latest version installed on your computer. If you have version 3 of Berkeley DB installed, you should be able to use db2_load to create a native version 2 database. You should move away from Berkeley DB version 1 files because the hash file code contains known bugs that can corrupt your data. Mathematics and Numerics ================================ How do I generate random numbers in Python? --------------------------------------------------- The `standard module random <http://www.python.org/doc/lib/modules-random.html>`_ implements a random number generator. Usage is simple:: import random random.random() This returns a random floating point number in the range [0, 1). There are also many other specialized generators in this module, such as: * ``randrange(a, b)`` chooses an integer in the range [a, b). * ``uniform(a, b)`` chooses a floating point number in the range [a, b). * ``normalvariate(mean, sdev)`` samples the normal (Gaussian) distribution. Some higher-level functions operate on sequences directly, such as: * ``choice(S)`` chooses random element from a given sequence * ``shuffle(L)`` shuffles a list in-place, i.e. permutes it randomly There's also a ``Random`` class you can instantiate to create independent multiple random number generators.

1 0

New py2exe binary distribution for Python 2.3
by Thomas Heller Aug. 22, 2003

Aug. 22, 2003

Unfortunately, the py2exe binary distribution, version 0.4.1, contained broken binaries for Python 2.3, for building Windows NT services. I've uploaded a new binary installer for Python 2.3 only, and changed the version number to 0.4.2. <http://prdownloads.sourceforge.net/py2exe/py2exe-0.4.2.win32-py2.3.exe?down…> If you are still using Python 2.2, there's no need to change anything. Enjoy, Thomas

1 0

@python.org and @zope.org email troubles
by Martijn Pieters Aug. 22, 2003

Aug. 22, 2003

Thanks to the enormously proliferant W32.Sobig.F, the mail.python.org email server has been having real trouble dealing with the flood of email pounding on it's ports. We are currently hard at work trying to bring the mailserver back to sanity again, and you should be seeing email emerge again, although it may take some time still before things will be back to normal operation. In the meantime, we apologize for the inconvenience. -- Martijn Pieters | Software Engineer mailto:mj@zope.com | Zope Corporation http://www.zope.com/ | Creators of Zope http://www.zope.org/ ---------------------------------------------

1 0

cx_Oracle 3.1
by Anthony Tuininga Aug. 22, 2003

Aug. 22, 2003

What is cx_Oracle? cx_Oracle is a Python extension module that allows access to Oracle and conforms to the Python database API 2.0 specifications with a few exceptions. Where do I get it? http://starship.python.net/crew/atuining http://www.computronix.com/utilities.shtml (it may be a few days before the second site is updated) What's new? 1) Added support for connecting with SYSDBA and SYSOPER access which is needed for connecting as sys in Oracle 9i. 2) Only check the dictionary size if the variable is not NULL; otherwise, an error takes place which is not caught or cleared; this eliminates a spurious "Objects/dictobject.c:1258: bad argument to internal function" in Python 2.3. 3) Add support for session pooling. This is only support for Oracle 9i but is amazingly fast -- about 100 times faster than connecting. 4) Add support for statement caching when pooling sessions, this reduces the parse time considerably. Unfortunately, the Oracle OCI does not allow this to be easily turned on for normal sessions. 5) Add method trim() on CLOB and BLOB variables for trimming the size. 6) Add support for externally identified users; to use this feature leave the username and password fields empty when connecting. 7) Add method cancel() on connection objects to cancel long running queries. Note that this only works on non-Windows platforms. 8) Add method callfunc() on cursor objects to allow calling a function without using an anonymous PL/SQL block. 9) Added documentation on objects that were not documented. At this point all objects, methods and constants in cx_Oracle have been documented. 10) Added support for timestamp columns in Oracle 9i. 11) Added module level method makedsn() which creates a data source name given the host, port and SID. 12) Added constant "buildtime" which is the time when the module was built as an additional means of identifying the build that is in use. 13) Binding a value that is incompatible to the previous value that was bound (data types do not match or array size is larger) will now result in a new bind taking place. This is more consistent with the DB API although it does imply a performance penalty when used. -- Anthony Tuininga anthony(a)computronix.com Computronix Distinctive Software. Real People. Suite 200, 10216 - 124 Street NW Edmonton, AB, Canada T5N 4A3 Phone: (780) 454-3700 Fax: (780) 454-3838 http://www.computronix.com

1 0

ANNOUNCE: Pmw megawidgets 1.2
by Greg McFarlane Aug. 22, 2003

Aug. 22, 2003

Pmw megawidgets for Tkinter Version 1.2 Greg McFarlane <gregm(a)iname.com> http://pmw.sourceforge.net/ --oOo-- A new release of Pmw is out. This release makes Pmw compatable with recent releases of python and Tcl/Tk. This version of Pmw was tested against python 2.3, Tcl/Tk 8.4.2 and Blt 2.4z. Except for Blt, all tests pass. Blt seems to be more flacky than usual, with core dumps occuring in the Graph code. Since this is a problem in Blt (not Pmw), the Blt test script has been removed from the default Pmw test sequence. Several other bug fixes and improvements were made. See the change log for details. To download Pmw or for more information, see the home page at http://pmw.sourceforge.net/ If you have any comments, enhancements or new contributions, please contact me (gregm(a)iname.com). ===================================================================== What is Pmw? Pmw is a toolkit for building high-level compound widgets, or megawidgets, constructed using other widgets as component parts. It promotes consistent look and feel within and between graphical applications, is highly configurable to your needs and is easy to use. It uses the Tkinter python library. Pmw consists of: - A few base classes, providing a foundation for building megawidgets. - A library of flexible and extensible megawidgets built on the base classes, such as buttonboxes, notebooks, comboboxes, selection widgets, paned widgets, scrolled widgets and dialog windows. - A lazy importer/dynamic loader which is automatically invoked when Pmw is first imported. This gives unified access to all Pmw classes and functions through the Pmw. prefix. It also speeds up module loading time by only importing Pmw sub-modules when needed. - Complete reference documentation, covering all classes and functions including all megawidgets and their options, methods and components. Helpful tutorial material is also available. - A test framework and tests for Pmw megawidgets. - A slick demonstration of the megawidgets. - An interface to the BLT busy, graph and vector commands. The interface to Pmw megawidgets is similar to basic Tk widgets, so it is easy for developers to include both megawidgets and basic Tk widgets in their graphical applications. In addition, Pmw megawidgets may themselves be extended, using either inheritance or composition. The use of the Pmw megawidgets replaces common widget combinations with higher level abstractions. This simplifies code, making it more readable and maintainable. The ability to extend Pmw megawidgets enables developers to create new megawidgets based on previous work. -- Greg McFarlane Really Good Software Pty Ltd Sydney Australia gregm(a)iname.com

1 0

ANNOUNCE: SCons 0.92 fixes a potentially critical Win32 problem
by Steven Knight Aug. 20, 2003

Aug. 20, 2003

SCons is a software construction tool (build tool, or make tool) written in Python. It is based on the design which won the Software Carpentry build tool competition in August 2000. Version 0.92 of SCons has been released and is available for download from the SCons web site: http://www.scons.org/ Or through the download link at the SCons project page at SourceForge: http://sourceforge.net/projects/scons/ RPM and Debian packages and a Win32 installer are all available, in addition to the traditional .tar.gz and .zip files. WHAT'S NEW IN THIS RELEASE? Release 0.92 contains an important fix for a critical bug found shortly after SCons 0.91 was released, plus two other fixes. SCons 0.91 contains a bug that prevents SCons from working on Win32 systems unless either the Microsoft Visual Studio linker (link.exe) or the GCC linker is installed, or a custom-defined linker is used. This bug has been fixed in release 0.92. This bug does not cause incorrect builds and does not damage a source tree or build tree in any way. Anyone with either the Visual Studio or GCC linker installed on their Win32 system, or using a custom-defined linker, will never see this bug and could continue to use 0.91 without any serious repercussions or side effects. Release 0.92 contains two other fixes: - Support for the PharLap linker was not working properly in 0.91, and has been fixed. - When the YACC -o option was used to specify an output file with a different basename than the source file, SCons would mistakenly think that the basename of the generated .h file should match the source file, not the target file. This has been fixed. Additionally, the Debian (.deb) package available from the SCons web site or SourceForge project page has been updated to match the dependencies and other particulars of the (out of date) Debian package available from debian.org. In particular, the package is now dependent on Python 2.2 instead of Python 2.1. ABOUT SCONS Distinctive features of SCons include: - a global view of all dependencies; no multiple passes to get everything built properly - configuration files are Python scripts, allowing the full use of a real scripting language to solve difficult build problems - a modular architecture allows the SCons Build Engine to be embedded in other Python software - the ability to scan files for implicit dependencies (#include files); - improved parallel build (-j) support that provides consistent build speedup regardless of source tree layout - use of MD5 signatures to decide if a file has really changed; no need to "touch" files to fool make that something is up-to-date - easily extensible through user-defined Builder and Scanner objects - build actions can be Python code, as well as external commands An scons-users mailing list is available for those interested in getting started using SCons. You can subscribe at: http://lists.sourceforge.net/lists/listinfo/scons-users Alternatively, we invite you to subscribe to the low-volume scons-announce mailing list to receive notification when new versions of SCons become available: http://lists.sourceforge.net/lists/listinfo/scons-announce ACKNOWLEDGEMENTS Special thanks to Charles Crain, Gary Oberbrunner and Gerard Patel for their contributions to this release. On behalf of the SCons team, --SK

1 0

Toronto-area Pythonistas meeting Tuesday the 26th of August...
by Mike Fletcher Aug. 20, 2003

Aug. 20, 2003

Since no-one seems to have announced this yet... PyGTA meeting this upcoming Tuesday at the regular time (8pm) and place (519 Comm. Centre (on Church St., in downtown Toronto)) See: http://web.engcorp.com/pygta/wiki/NextMeeting for details. Feel free to bring questions or problems to ask other Pythonistas about. We've been having good turnouts for the last few meetings, almost enough to overflow the rooms, so you don't have to just listen to Peter and I blathering on ;) . If you'd like to do a short presentation on your Python work please propose it on the pygta-general mailing list. Enjoy all, Mike _______________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://members.rogers.com/mcfletch/

1 0

Pointrel20030812.1 Data Repository System released
by Paul D. Fernhout Aug. 20, 2003

Aug. 20, 2003

Pointrel20030812.1 for Python 2.3 is now available at SourceForge: http://sourceforge.net/projects/pointrel/ * Highlights for this version? + Uses 64 bit file positions. + Log file format is XML. + Specification document included. + CRC32 now used for string hashes. + Locks no longer deleted automatically when old by default. + Now uses WILD constant instead of "*" for wildcards in searches. + Data strings can be Unicode (UTF-8), binary, or Python objects. * Release notes? This version is pure-Python and has been tested under Python 2.3. It may work with earlier 2.x Python versions or require only minor changes for them. It was tested only under Debian GNU/Linux, but earlier versions ran under Windows. The biggest change from the previous version is support for very large files (>2GB) by using 64 bit file positions. It still needs more testing though, so don't put all your terabytes of vital enterprise data in it just yet without some kind of parallel system going, just in case. ;-) But seriously, the software comes with NO WARRANTY (see the included file license.txt for details). * In what situations is the Pointrel Data Repository System intended to useful? The Pointrel System is intended to provide a flexible framework for storing information that can co-evolve with your data storage needs. The best match is a situation where you have potentially a lot of adhoc data which you or others may want to use in a variety of ways now and in the future, and you want a solution that lets you get started right now and then keep making changes on the fly, while preserving all data contributed permanently as an archive. An example of this would be an extensible peer-to-peer groupware application supporting continual frequent revision of everything in the repository. I intend to use it in this way, as part of the infrastructure of a project related to collecting manufacturing knowledge. Or, you might have well defined data storage needs and just want to support some sort of complex persistent transactional data storage by including only one or two extra files in your application which don't need special installation or configuration, and you are willing to live with potentially slower performance and some other limitations. There are other potential mini-database candidates for such a role as well (including Python's DBM module), so the Pointrel System might only be selected over them in this case with an eye to future expansion in more adhoc ways, such as supporting third party tools that will use the same store. Or, perhaps, you just might want to stretch your mind after too much SQL coding. :-) * What is the underlying design of the Pointrel Data Repository System? The Pointrel Data Repository System is a variant of an Entity-Relationship (E-R) model database. The Pointrel system provides a way to easily handle loosely structured data stored on disk, like for INI files, version control systems, bug tracking systems, or simple AI type applications. It takes an approach to data storage which emphasizes flexibility over speed. It also emphasizes storing new information for the long term over modifying or deleting old information. It hopefully makes it easier to build new layers of abstraction and indexing over old data. The Pointrel Data Repository System bears some resemblance to the ROSE/STAR system described by William Kent in his book "Data & Reality". It also bears some resemblance to RDF. The Pointrel System is to an extent mainly a mindset about how to build extensible applications using persistent E-R data, and this release is one example of a tool to make such applications easier to build. In a nutshell, the Pointrel Data Repository System helps you build associations which define relationships between entities. These associations are essentially triadal links between things indicating one thing is linked to a second thing in a way defined by a third thing. The simplest way to use such links is to make the equivalent of object properties or a dictionary, such as "Fluffy weight 20kg" which if a dictionary would be Fluffy["weight] = "20kg". However, Pointrel differs from a dictionary in that is supports queries like one for all dictionaries which define a weight of 20kg or all relationships between "Fluffy" and "20kg". Triads are all defined within a specific context space that gives meaning to the associations (making triads actually have four fields). The context allows triads to be handled within an archive in a somewhat more modular fashion using them as filters, since you can easily ignore triads not in the context of interest. All fields of a triad are indefinite length strings (binary, Unicode, or a pickled Python object) -- so they could be anything from the test "foo" to the contents of a binary file to a Python dictionary. * What is the Pointrel Data Repository System not? At this point the Pointrel system is not a polished product. This release is best enjoyed by someone who is more experimentally minded who just wants to have some fun while learning about the entity-relationship model of data storage as another tool for their toolbox. This release is also for those with a desperate need to make sense of a deluge of adhoc data and are willing to take risks on something new in practice (but old in theory). If you are looking for tried and proven methods and software for storing data right now, relational databases like MySQL or PostgreSQL are probably safer bets. Obviously, having said that, I hope you will try this system anyway and supply constructive criticism and positive feedback. * What is a simple example of the Pointrel API being used? See "pointrelTestFluffyExample.py" for an example of using the simplified global function interface. Here is an excerpt from that file: from pointrel20030812 import * Pointrel_initialize("archive_fluffyExample") Pointrel_startTransaction() Pointrel_add("examplecontext", "Fluffy", "weight", "20kg") Pointrel_add("examplecontext", "Fluffy", "color", "beige") Pointrel_add("examplecontext", "Fluffy", "teeth", "pointy") Pointrel_add("examplecontext", "Fluffy", "teeth", "nasty") Pointrel_add("examplecontext", "Fluffy", "preferred food", "Knights who say 'Nie!'") Pointrel_finishTransaction() string = Pointrel_lastMatch("examplecontext", WILD, "weight", "20kg") print string # string would be --> "Fluffy" string = Pointrel_lastMatch("examplecontext", "Fluffy", "weight", WILD) print string # string would be --> "20kg" string = Pointrel_lastMatch("examplecontext", "Fluffy", "teeth", WILD) print string # string would be --> "nasty" list = Pointrel_allMatches("examplecontext", "Fluffy", "teeth", WILD) print list # list would be --> ["pointy", "nasty"] * What is a more complex example of using the Pointrel system? See the file "tkPointrelMemex.py". It implements a version of the Memex archiving system proposed by Vannevar Bush it the 1940s. To my knowledge, it is the first software implementation of something this close to his concept (technically, depending on how you read his work, his trails may be more like trees than the linear trails here). Feel free to send me pointers to earlier software implementations of MEMEX -- I will be happy to retract this claim in exchange for learning more of the history of his great idea. Here is a typical API useage for the more complex OO API: repository = PointrelDataRepositorySystem(archiveName) repository.startTransaction() repository.add(context, a, b, c) repository.finishTransaction() string = repository.lastMatch(context, WILD, b, c) string = repository.lastMatch(context, a, b, WILD) list = repository.allMatches(context, a, b, WILD) string = repository.generateUniqueID() * What is the license? BSDish. See license.txt for details. --Paul Fernhout http://www.pointrel.org -----= Posted via Newsfeeds.Com, Uncensored Usenet News =----- http://www.newsfeeds.com - The #1 Newsgroup Service in the World! -----== Over 100,000 Newsgroups - 19 Different Servers! =-----

1 0

ANN: PyScript 0.4
by cochrane Aug. 19, 2003

Aug. 19, 2003

PyScript 0.4 has just been released! Get it from http://pyscript.sourceforge.net Here is the README: =============== pyscript =============== -------------------------------------------------------------------- Summary -------------------------------------------------------------------- Pyscript is a set of modules and scripts for python that facilitate the creation of high-quality postscript diagrams. The diagrams are scripted rather than drawn. See http://pyscript.sourceforge.net -------------------------------------------------------------------- Instalation -------------------------------------------------------------------- Pyscript is just a python module ... treat it as any other module * Global Instalation As root use > python setup.py install This will install the files, in the appropriate place for your python distribution. This will be something like e.g. /usr/lib/python2.2/site-packages/ * Local instalation: You can supply the base directory using > python setup.py install --home=<dir> which will install the files in <dir>/lib/python/ for more help and options use > python setup.py install --help also see http://www.python.org/doc/current/inst/ for more details on using the distutils package * By hand copy all the files in the pyscript directory to somewhere in your python path eg cp -r pyscript ~/lib/python/

1 0