Mailman 3 January 2004 - Python-Dev

PEP 324: popen5 - New POSIX process module
by Peter Åstrand Jan. 6, 2004

Jan. 6, 2004

There's a new PEP available: PEP 324: popen5 - New POSIX process module A copy is included below. Comments are appreciated. ---- PEP: 324 Title: popen5 - New POSIX process module Version: $Revision: 1.4 $ Last-Modified: $Date: 2004/01/03 10:32:53 $ Author: Peter Astrand <astrand(a)lysator.liu.se> Status: Draft Type: Standards Track (library) Created: 19-Nov-2003 Content-Type: text/plain Python-Version: 2.4 Abstract This PEP describes a new module for starting and communicating with processes on POSIX systems. Motivation Starting new processes is a common task in any programming language, and very common in a high-level language like Python. Good support for this task is needed, because: - Inappropriate functions for starting processes could mean a security risk: If the program is started through the shell, and the arguments contain shell meta characters, the result can be disastrous. [1] - It makes Python an even better replacement language for over-complicated shell scripts. Currently, Python has a large number of different functions for process creation. This makes it hard for developers to choose. The popen5 modules provides the following enhancements over previous functions: - One "unified" module provides all functionality from previous functions. - Cross-process exceptions: Exceptions happening in the child before the new process has started to execute are re-raised in the parent. This means that it's easy to handle exec() failures, for example. With popen2, for example, it's impossible to detect if the execution failed. - A hook for executing custom code between fork and exec. This can be used for, for example, changing uid. - No implicit call of /bin/sh. This means that there is no need for escaping dangerous shell meta characters. - All combinations of file descriptor redirection is possible. For example, the "python-dialog" [2] needs to spawn a process and redirect stderr, but not stdout. This is not possible with current functions, without using temporary files. - With popen5, it's possible to control if all open file descriptors should be closed before the new program is executed. - Support for connecting several subprocesses (shell "pipe"). - Universal newline support. - A communicate() method, which makes it easy to send stdin data and read stdout and stderr data, without risking deadlocks. Most people are aware of the flow control issues involved with child process communication, but not all have the patience or skills to write a fully correct and deadlock-free select loop. This means that many Python applications contain race conditions. A communicate() method in the standard library solves this problem. Rationale The following points summarizes the design: - popen5 was based on popen2, which is tried-and-tested. - The factory functions in popen2 have been removed, because I consider the class constructor equally easy to work with. - popen2 contains several factory functions and classes for different combinations of redirection. popen5, however, contains one single class. Since popen5 supports 12 different combinations of redirection, providing a class or function for each of them would be cumbersome and not very intuitive. Even with popen2, this is a readability problem. For example, many people cannot tell the difference between popen2.popen2 and popen2.popen4 without using the documentation. - One small utility function is provided: popen5.run(). It aims to be an enhancement over os.system(), while still very easy to use: - It does not use the Standard C function system(), which has limitations. - It does not call the shell implicitly. - No need for quoting; using a variable argument list. - The return value is easier to work with. - The "preexec" functionality makes it possible to run arbitrary code between fork and exec. One might ask why there are special arguments for setting the environment and current directory, but not for, for example, setting the uid. The answer is: - Changing environment and working directory is considered fairly common. - Old functions like spawn() has support for an "env"-argument. - env and cwd are considered quite cross-platform: They make sense even on Windows. - No MS Windows support is available, currently. To be able to provide more functionality than what is already available from the popen2 module, help from C modules is required. Specification This module defines one class called Popen: class Popen(args, bufsize=0, argv0=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, preexec_args=(), close_fds=0, cwd=None, env=None, universal_newlines=0) Arguments are: - args should be a sequence of program arguments. The program to execute is normally the first item in the args sequence, but can be explicitly set by using the argv0 argument. The Popen class uses os.execvp() to execute the child program. - bufsize, if given, has the same meaning as the corresponding argument to the built-in open() function: 0 means unbuffered, 1 means line buffered, any other positive value means use a buffer of (approximately) that size. A negative bufsize means to use the system default, which usually means fully buffered. The default value for bufsize is 0 (unbuffered). - stdin, stdout and stderr specify the executed programs' standard input, standard output and standard error file handles, respectively. Valid values are PIPE, an existing file descriptor (a positive integer), an existing file object, and None. PIPE indicates that a new pipe to the child should be created. With None, no redirection will occur; the child's file handles will be inherited from the parent. Additionally, stderr can be STDOUT, which indicates that the stderr data from the applications should be captured into the same file handle as for stdout. - If preexec_fn is set to a callable object, this object will be called in the child process just before the child is executed, with arguments preexec_args. - If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. - If cwd is not None, the current directory will be changed to cwd before the child is executed. - If env is not None, it defines the environment variables for the new process. - If universal_newlines is true, the file objects fromchild and childerr are opened as a text files, but lines may be terminated by any of '\n', the Unix end-of-line convention, '\r', the Macintosh convention or '\r\n', the Windows convention. All of these external representations are seen as '\n' by the Python program. Note: This feature is only available if Python is built with universal newline support (the default). Also, the newlines attribute of the file objects fromchild, tochild and childerr are not updated by the communicate() method. The module also defines one shortcut function: run(*args): Run command with arguments. Wait for command to complete, then return the returncode attribute. Example: retcode = popen5.run("stty", "sane") Exceptions ---------- Exceptions raised in the child process, before the new program has started to execute, will be re-raised in the parent. Additionally, the exception object will have one extra attribute called 'child_traceback', which is a string containing traceback information from the child's point of view. The most common exception raised is OSError. This occurs, for example, when trying to execute a non-existent file. Applications should prepare for OSErrors. A PopenException will also be raised if Popen is called with invalid arguments. Security -------- popen5 will never call /bin/sh implicitly. This means that all characters, including shell metacharacters, can safely be passed to child processes. Popen objects ------------- Instances of the Popen class have the following methods: poll() Returns -1 if child process hasn't completed yet, or its exit status otherwise. See below for a description of how the exit status is encoded. wait() Waits for and returns the exit status of the child process. The exit status encodes both the return code of the process and information about whether it exited using the exit() system call or died due to a signal. Functions to help interpret the status code are defined in the os module (the W*() family of functions). communicate(input=None) Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate. The optional stdin argument should be a string to be sent to the child process, or None, if no data should be sent to the child. communicate() returns a tuple (stdout, stderr). Note: The data read is buffered in memory, so do not use this method if the data size is large or unlimited. The following attributes are also available: fromchild A file object that provides output from the child process. tochild A file object that provides input to the child process. childerr A file object that provides error output from the child process. pid The process ID of the child process. returncode The child return code. A None value indicates that the process hasn't terminated yet. A negative value means that the process was terminated by a signal with number -returncode. Open Issues Perhaps the module should be called something like "process", instead of "popen5". Reference Implementation A reference implementation is available from http://www.lysator.liu.se/~astrand/popen5/. References [1] Secure Programming for Linux and Unix HOWTO, section 8.3. http://www.dwheeler.com/secure-programs/ [2] Python Dialog http://pythondialog.sourceforge.net/ Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: -- /Peter Åstrand <astrand(a)lysator.liu.se>

7 13

pybench.py stats
by M.-A. Lemburg Jan. 6, 2004

Jan. 6, 2004

I thought you might find this interesting. For some time I've been running pybench.py as nightly cronjob against fresh CVS checkouts and recorded the findings. Attached is a PNG diagram showing the findings. The results are simple to interpret: there was only a very slight performance improvement over the last year. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 05 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

3 2

PEP 326 now online
by Josiah Carlson Jan. 5, 2004

Jan. 5, 2004

I believe I've addressed the majority of concerns had for the All (previously Some) concept and object (the PEP editor seems to agree), and the PEP has been posted as PEP 326. Give it a look: http://www.python.org/peps/pep-0326.html As always, comments, questions, and suggestions are always appreciated. - Josiah

5 7

Visual Stduio .NET build of python
by Stephen Pheiffer Jan. 5, 2004

Jan. 5, 2004

Does a Visual Studio .NET build of python exist? distutils will not work to build python because it was created using Visual C 6.0, while I have Visual Studio .NET installed on my machine.

2 1

Switch to VC.NET 7.1 completed
by Martin v. Loewis Jan. 5, 2004

Jan. 5, 2004

I now have committed the bits for building Python with VC.NET. Anybody regularly building on Windows should delete all object files, i.e. the PCBuild/temp folders, PCBuild/*.{lib,dll,pyd,exe} (probably more, but they should get generated anew), and object files of prereq libraries you regularly use. In the process, I also updated a few prerequisite libraries to the current versions: Tcl, Sleepycat, openssl (updating Tcl is not strictly needed - you do need to rebuild it, though). I have also incorporated a few more modules into python24.dll. I have moved the VC6 files to PC/VC6. I have no plans of maintaining them, and they are as they were last in PCBuild. I see no problem though with somebody else maintaining the for the time being, if somebody volunteers. Please let me know of any problems you encounter. Regards, Martin

3 3

RE: [Python-Dev] PEP 326 now online
by Robert Brewer Jan. 5, 2004

Jan. 5, 2004

Josiah Carlson wrote: > I've submitted my latest changes that I believe address the concerns > brought up since I last emailed the list. Until they are in > CVS and on > the standard PEP page, it can be found here: > http://josiahcarlson.no-ip.org/pep-0326.html > > The major change of note: it is no longer a builtin. My only comment on the changes: I definitely do not approve of the use of the abbreviations "hi" and "lo" in place of standard English "high" and "low"--human parsing of "hi" and "lo" depends upon a familiarity with English which many people in this world do not possess. In my book, three characters of extra typing are more than offset by the improved accessibility. As far as the whole PEP, I'm +0. I'd never use it. Fighting unnecessary abbreviations wherever they are found, Robert Brewer MIS Amor Ministries fumanchu(a)amor.org

1 0

PyCon Reminder: Proposal deadline 1/15
by Guido van Rossum Jan. 5, 2004

Jan. 5, 2004

This is a reminder that the deadline for sending proposals for presentations at PyCon DC 2004 is January 15, 2004. We are interested in any and all submissions about uses of Python and the development of the language. Since there is expected to be a strong educational community presence for the next PyCon, teaching materials of various kinds are also encouraged. You can submit your proposal at: http://submit.pycon.org/ For more information about proposals, see: http://www.pycon.org/dc2004/cfp/ If you have further questions about the submission web interface or the format of submissions, please write to: pycon-organizers(a)python.org We would like to publish all accepted papers on the web. If your paper is accepted and you prepare an electronic presentation (in PDF, PythonPoint or PowerPoint) we will also happily publish that on the web site once PyCon is over. If you don't want to make a formal presentation, there will be a significant amount of Open Space to allow for informal and spur-of-the-moment presentations for which no formal submission is required. There will also be several Lightning Talk sessions (five minutes or less). About PyCon: PyCon is a community-oriented conference targeting developers (both those using Python and those working on the Python project). It gives you opportunities to learn about significant advances in the Python development community, to participate in a programming sprint with some of the leading minds in the Open Source community, and to meet fellow developers from around the world. The organizers work to make the conference affordable and accessible to all. PyCon DC 2004 will be held March 24-26, 2004 in Washington, D.C. The keynote speaker is Mitch Kapor of the Open Source Applications Foundation (http://www.osafoundation.org/). There will be a four-day development sprint before the conference. We're looking for volunteers to help run PyCon. If you're interested, subscribe to http://mail.python.org/mailman/listinfo/pycon-organizers Don't miss any PyCon announcements! Subscribe to http://mail.python.org/mailman/listinfo/pycon-announce You can discuss PyCon with other interested people by subscribing to http://mail.python.org/mailman/listinfo/pycon-interest The central resource for PyCon DC 2004 is http://www.pycon.org/ Pictures from last year's PyCon: http://www.python.org/cgi-bin/moinmoin/PyConPhotos I'm looking forward to seeing you all in DC in March!!! --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

Weekly Python Bug/Patch Summary
by Skip Montanaro Jan. 5, 2004

Jan. 5, 2004

Bug/Patch Summary ----------------- 618 open / 4468 total bugs (+81) 224 open / 2514 total patches (+36) New Bugs -------- html problem in mailbox docs (2003-12-30) http://python.org/sf/867556 Configure error with 2.3.3 on AIX 5.2 using gcc 3.3.2 (2003-12-31) http://python.org/sf/868529 HTTPResponse.read(amt) fails when response length is UNKNOWN (2003-12-31) http://python.org/sf/868571 Calling builtin function 'eval' from C causes seg fault. (2003-12-31) http://python.org/sf/868706 os.tmpfile seek(0) fails on cygwin (2004-01-01) http://python.org/sf/868743 Multiple problems with GC in 2.3.3 (2004-01-01) http://python.org/sf/868896 setgroups rejects long integer arguments (2004-01-02) http://python.org/sf/869197 Quicktime missing funcitonality (2004-01-02) http://python.org/sf/869649 segmentation fault in test_re (2004-01-03) http://python.org/sf/870120 New Patches ----------- Bad behavior of email.Charset.Charset when locale is tr_TR (2003-12-29) http://python.org/sf/866982 regrtest.py -T for code coverage (2003-12-31) http://python.org/sf/868499 Allow use of embedded Tcl without requiring Tk (2004-01-02) http://python.org/sf/869468 Building on Mac OS X 10.1 (2004-01-02) http://python.org/sf/869555 Add start and end optional args to array.index (2004-01-02) http://python.org/sf/869688 make Demo/scripts/primes.py usable as a module (2004-01-04) http://python.org/sf/870286 Fix docstring for posixpath.getctime (2004-01-04) http://python.org/sf/870287 Closed Bugs ----------- Document sys.path on Windows (2001-09-05) http://python.org/sf/459007 __cdecl / __stdcall unspecified in *.h (2001-10-15) http://python.org/sf/471432 locale.setlocale is broken (2002-01-15) http://python.org/sf/504219 isdir behavior getting odder on UNC path (2002-02-05) http://python.org/sf/513572 long file name support broken in windows (2002-04-10) http://python.org/sf/542314 Windows getpass bug (2002-04-20) http://python.org/sf/546558 os.tmpfile() can fail on win32 (2002-08-05) http://python.org/sf/591104 os.popen() negative error code IOError (2002-08-29) http://python.org/sf/602245 long list in Pythonwin -> weird text (2002-09-03) http://python.org/sf/604387 registry functions don't handle null characters (2003-01-21) http://python.org/sf/672132 PyErr_Warn may cause import deadlock (2003-02-09) http://python.org/sf/683658 print raises exception when no console available (2003-03-19) http://python.org/sf/706263 test_socketserver: Fatal error: Invalid thread state (2003-05-26) http://python.org/sf/743692 ipv6 sockaddr is bad (2003-05-27) http://python.org/sf/744164 Failure in comparing unicode string (2003-06-26) http://python.org/sf/761267 No syntax hilite on new file until saved as *.py (2003-07-21) http://python.org/sf/775012 IDLE: "Save As..." keybind (Ctrl+Shift+S) does not work (2003-07-21) http://python.org/sf/775353 Tooltip-window doesn't vanish if... (2003-07-22) http://python.org/sf/775535 idle.py doesn't accept ( in some cases (2003-07-22) http://python.org/sf/775541 Tab / Space Configuration Does Not Work in IDLE (2003-08-05) http://python.org/sf/783887 IDLE starts with no menus (Cygwin build of python2.3) (2003-08-11) http://python.org/sf/786827 Arguments tooltip wrong if def contains tuple (2003-08-20) http://python.org/sf/791968 IDLE / PyOS_InputHook (2003-08-31) http://python.org/sf/798058 color highlighting not working on EDIT windows (2003-09-04) http://python.org/sf/800432 tarfile violates bufsize (2003-09-25) http://python.org/sf/812325 Fatal Python error: GC object already tracked (2003-10-02) http://python.org/sf/816476 Idle fails on loading .idlerc if Home path changes. (2003-10-22) http://python.org/sf/828049 Inconsitent line numbering in traceback (2003-10-29) http://python.org/sf/832535 Ctrl+key combos stop working in IDLE (2003-10-31) http://python.org/sf/833957 os.chmod/os.utime/shutil do not work with unicode filenames (2003-11-20) http://python.org/sf/846133 mbcs encoding ignores errors (2003-11-28) http://python.org/sf/850997 HTMLParser parsers AT&T to AT (2003-12-08) http://python.org/sf/856617 Hangs when opening 2nd shell with subprocess under windows (2003-12-23) http://python.org/sf/865014 Closed Patches -------------- Fix for bug 494589 (2002-12-25) http://python.org/sf/658599 Faster commonprefix (OS independent) (2003-02-06) http://python.org/sf/681780 expose lowlevel setlocale (2003-07-24) http://python.org/sf/776854 Tk Dialog Upon Subprocess Socket Error (2003-07-26) http://python.org/sf/778323 Highlight builtins (2003-09-13) http://python.org/sf/805830 cmath.log optional base argument, fixes #823209 (2003-10-18) http://python.org/sf/826074 Small error in test_format (2003-11-25) http://python.org/sf/849252 Modify Setup.py to Detect Tcl/Tk on BSD (2003-11-28) http://python.org/sf/850977 heapq: A way to change the compare function (2003-12-28) http://python.org/sf/866594

3 2

Re: [Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0325.txt, NONE, 1.1
by Samuele Pedroni Jan. 5, 2004

Jan. 5, 2004

At 07:15 05.01.2004 -0800, Guido van Rossum wrote: >Should I comment on PEP 325 yet? I haven't seen an official >announcement. I sent that version in October, but David was moving house and because of its ISP fault it never got it (but I didn't know), then I was busy and I resubmitted it just now because I was puzzled, it was the new year and I didn't like having it sitting on my HD even for longer. But to be honest, right now I have no time for lengthy discussions. regards.

1 0

[Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0325.txt, NONE, 1.1
by Guido van Rossum Jan. 5, 2004

Jan. 5, 2004

Should I comment on PEP 325 yet? I haven't seen an official announcement. --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0