Mailman 3 October 2004 - Python-Dev

Reducing core dependencies for Python on OS X / distutils nit
by Bob Ippolito Oct. 16, 2004

Oct. 16, 2004

Remove CoreServices / CoreFoundation dependencies in core http://python.org/sf/1035255 It came to my attention a couple weeks ago that Python unnecessarily links to a few frameworks on OS X, so I generated a patch that removed these dependencies. I would hope that this is uncontroversial. Presumably Jack has been really busy lately so I understand that this patch didn't make it into b1, but I would like to get it rolled into CVS by *somebody* after CVS is reopened so the final of Python 2.4 doesn't link to a bunch of frameworks for no reason, which potentially (not benchmarked) increases memory usage and startup time because the resultant Python.framework and python interpreter no longer have any explicit dependencies at all (just libSystem, which is linked in by default). -- distutils.util.get_platform() should include sys.version[:3] http://python.org/sf/1035703 On a side note, when I was playing with Python 2.4, I noticed that distutils doesn't create build directories that have the major Python version encoded in them, only the operating system information. This doesn't really make much sense, because extensions and bytecode aren't typically compatible between major versions! Nobody commented on this, but I'm willing to write a patch to create build directories with sys.version[:3] in them if it will be accepted... -bob

2 2

Wither docs for the subprocess module?
by Raymond Hettinger Oct. 16, 2004

Oct. 16, 2004

Is anyone working on documenting the module for the library reference? Raymond

2 2

python-dev Summary for 2004-09-16 through 2004-09-30 [draft]
by Brett C Oct. 15, 2004

Oct. 15, 2004

So we were all rather quiet in the last half of September. The whole summary fits on two sheets of 8.5x11 (normally it is over 10 and I have hit over 20 when I was summarizing *everything*). Going to send this out no earlier than Friday night so send in corrections by then. ---------------------------------- ===================== Summary Announcements ===================== Wow. This must have been the easiest summary I have ever done. Why can't they all be like this? I didn't even skip that much! ========= Summaries ========= ------------------------------------------ Assume nothing when mutability is possible ------------------------------------------ Tim Peters discovered a new way to create an infinite list thanks to generator expressions. But what really came out of this whole discussion came about when someone else came up with an example that exposed a bug in list.extend(). The first thing was that "you can't assume anything about a mutable object after potentially calling back into Python." Basically you can't assume the state of any mutable object was not changed if you execute Python code from C. While it might seem handy to store state while in a loop for instance, you can't count on things not change by the time you get control back so you just have to do it the hard way and get state all over again. Second was that you need to be careful when dealing with iterators. If you mutate an iterator while iterating you don't have a guarantee it won't explode in your face. Unless you explicitly support it, document it, and take care to protect against it then just don't assume you can mutate an iterator while using it. Contributing threads: - `A cute new way to get an infinite loop <>`__ - `More data points <>`__ ---------------------------- The less licenses the better ---------------------------- The idea of copying some code from OpenSSH_ for better pty handling was proposed. This was frowned upon since that becomes one more legal issue to keep track of. Minimizing the licenses that Python must keep track of and make sure to comply with, no matter how friendly, is a good thing. .. _OpenSSH: http://www.openssh.com/ Contributing threads: - `using openssh's pty code <>`__ ------------------------------------------------------------------------ Trying to deal with the exception hierarchy and a backwards-friendly way ------------------------------------------------------------------------- Nick Coghlan came up with the idea of having a tuple that contained all of the exceptions you normally would not want to catch in a blanket 'except' statement; KeyboardInterrupt, MemoryError, SystemExit, etc.). This tuple was proposed to live in sys.special_exceptions with the intended usage of:: try: pass # stuff... except sys.special_exceptions: raise # exceptions that you would not want to catch should keep propogating up the call chain except: pass # if you reach here the exception should not be a *huge* deal Obviously the best solution is to just clean up the exception hierarchy, but that breaks backwards-compatibility. But this idea seemed to lose steam. Contributing threads: - `Proposing a sys.special_exceptions tuple <>`__ =============== Skipped Threads =============== - Decimal, copyright and license - Planning to drop gzip compression for future releases. - built on beer? - Noam's open regex requests - Socket/Asyncore bug needs attention - open('/dev/null').read() -> MemoryError - Finding the module from PyTypeObject? - Odd compile errors for bad genexps - Running a module as a script

13 21

About the Wise installer on Windows
by Alexandre Parenteau Oct. 15, 2004

Oct. 15, 2004

Hi, Again forgive me if the answer is somewhere, but I could not find it. I am looking for a way to build and make an installer out of Python on a Win x86_64 platform which is not supported yet by the Python team. My plan is to compile Python for this platform (several developers on the list told me it should be straightforward), and make an installer out of the Python binaries in order to be able to distribute the Python binaries internally. I downloaded the evaluation copy of Wise Installer System 9, but it complains that the installer file (which I assume is PCBuild\Python20.wse) is corrupted. I need to establish exactly the version of the installer to use, if we need to acquire a license for the Wise installer. - Could someone give me some pointer about the exact version of the Wise Installer to use in order to be able to replicate the same installer that the Python team is distributing on Windows? - The ISS files (Inno) do not seem up to date. Is anybody having a ISS version of the installer which works for 2.3.4? Thanks, alex.

6 10

Re: python/dist/src/Objects unicodeobject.c, 2.228, 2.229
by Fredrik Lundh Oct. 15, 2004

Oct. 15, 2004

> Applied patch for [ 1047269 ] Buffer overwrite in PyUnicode_AsWideChar. > > Python 2.3.x candidate. why bother? the unicode object you're copying to holds size+1 characters, so all the code does is copying an extra NULL character... completely harmless. </F>

2 2

Patch wanted
by Scott David Daniels Oct. 15, 2004

Oct. 15, 2004

Ray Hettinger's fix 1.29 to PC/pyconfig.h: Revision 1.29 - (view) (download) (annotate) - [select for diffs] CVS Tags: r24a2, r24a3 Changes since 1.28: +4 -2 lines Restore compilation on MSVC++ 6.0 =================================== /* Atleast VC 7.1 has them. If some compiler does not provide them, #ifdef appropriately .*/ #define HAVE_UINTPTR_T 1 #define HAVE_INTPTR_T 1 =================================== /* VC 7.1 has them and VC 6.0 does not. VC 6.0 has a version number of 1200. If some compiler does not provide them, modify the #if appropriately. */ #if _MSC_VER != 1200 #define HAVE_UINTPTR_T 1 #define HAVE_INTPTR_T 1 #endif =================================== Tests the wrong thing. His test presumes every non VC 7.1 PC compiler has those types defined. For non-MSC compilers, _MSC_VER is undefined, and the expression #if _MSC_VER != 1200 becomes: #if 0 != 1200 (Not what was intended, I presume). Even: #if _MSC_VER >= 1200 would be preferable (better failure, will presumably work with VC 7.2), but I believe the correct test should be: #if HAVE_STD_INT as I inexpertly explained in patch 1020042. This affects any C compilers on Windows that don't have the uintptr_t and intptr_t types (those without the newer include file stdint.h) except VC 6.0. Such compilers cannot compile any file that contains #include "Python.h" I think this needs to be fixed before 2.4 goes out. -- -- Scott David Daniels Scott.Daniels(a)Acm.Org

2 3

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser Oct. 15, 2004

Oct. 15, 2004

Patch / Bug Summary ___________________ Patches : 240 open ( -1) / 2655 closed (+15) / 2895 total (+14) Bugs : 766 open ( +0) / 4514 closed (+22) / 5280 total (+22) RFE : 155 open ( +1) / 131 closed ( +0) / 286 total ( +1) New / Reopened Patches ______________________ True/False instead of 1/0 in libstdtypes.tex (2004-10-06) CLOSED http://python.org/sf/1041364 opened by Gerrit Holl bbetter document popenX 'cmd' argument (2004-10-07) CLOSED http://python.org/sf/1042705 opened by Jeff Epler broken TeX markup (2004-10-08) CLOSED http://python.org/sf/1042969 opened by George Yoshida Decimal.py enhancements (2004-10-08) CLOSED http://python.org/sf/1043218 opened by Facundo Batista Enhancing '-m' to support packages (2004-10-09) http://python.org/sf/1043356 opened by Nick Coghlan External storage protocol for large email messages (2004-10-09) http://python.org/sf/1043706 opened by Barry A. Warsaw tarfile: add extractall() method (2004-10-10) http://python.org/sf/1043890 opened by Lars Gustäbel tarfile.py: patch for bug #1017553 (2004-10-10) http://python.org/sf/1043972 opened by Lars Gustäbel Bug #1011380 alternate fix - locking is noop without threads (2004-10-11) CLOSED http://python.org/sf/1044089 opened by Nick Coghlan Add FreeBSD* to configure.in for --enable-shared (2004-10-11) http://python.org/sf/1044395 opened by James William Pye improved configure.in output (2004-10-12) http://python.org/sf/1045620 opened by Wummel Remove unncessary NLST from ftp transfers (2004-10-12) http://python.org/sf/1045783 opened by Chris Cogdon Consistent retrieval of Python version. (2004-10-14) http://python.org/sf/1046831 opened by uiltje Buffer overwrite in PyUnicode_AsWideChar (2004-10-14) http://python.org/sf/1047269 opened by Thomas Heller Patches Closed ______________ -m option to run a module as a script (2004-09-27) http://python.org/sf/1035498 closed by rhettinger True/False instead of 1/0 in libstdtypes.tex (2004-10-06) http://python.org/sf/1041364 closed by rhettinger pdb enhancements and bug fixes (2004-02-12) http://python.org/sf/896011 closed by jlgijsbers bbetter document popenX 'cmd' argument (2004-10-08) http://python.org/sf/1042705 closed by jlgijsbers popen2.popen2() 'cmd' list syntax docstring (2004-02-01) http://python.org/sf/888380 closed by jlgijsbers broken TeX markup (2004-10-08) http://python.org/sf/1042969 closed by jlgijsbers Decimal.py enhancements (2004-10-08) http://python.org/sf/1043218 closed by rhettinger replacing rfc822.formatdate (2004-08-09) http://python.org/sf/1005857 closed by anthonybaxter email/__init__.py: Parse headersonly (2004-03-20) http://python.org/sf/920037 closed by bwarsaw Bad behavior of email.Charset.Charset when locale is tr_TR (2003-12-29) http://python.org/sf/866982 closed by bwarsaw mailbox / fromline matching (2002-02-22) http://python.org/sf/521478 closed by bwarsaw Bug #1011380 alternate fix - locking is noop without threads (2004-10-10) http://python.org/sf/1044089 closed by tim_one Logger file objects (2004-08-02) http://python.org/sf/1001864 closed by vsajip Signal fixes for BSD systems (2004-06-18) http://python.org/sf/975056 closed by anthonybaxter Allow ldshared to be overridden by environment var (2004-07-01) http://python.org/sf/983206 closed by anthonybaxter New / Reopened Bugs ___________________ copy.deepcopy exceptions.TypeError: 'NoneType' object ... (2004-10-06) CLOSED http://python.org/sf/1041325 opened by Arthur Lutz SimpleXMLRPCServer example is broken (2004-10-06) CLOSED http://python.org/sf/1041501 opened by Kent Johnson Thread management corrupts heap (2004-10-06) CLOSED http://python.org/sf/1041645 opened by benson margulies Patch (2004-10-07) CLOSED http://python.org/sf/1042179 opened by Michiel de Hoon Lib/compiler chokes on certain genexps (2004-10-07) CLOSED http://python.org/sf/1042238 opened by Michael Hudson mimetypes.guess_extension('text/plain') = '.ksh' ??? (2004-10-08) http://python.org/sf/1043134 opened by Robert Kiendl Uninterruptable loop (2004-10-09) http://python.org/sf/1043446 opened by Jp Calderone compile error with stlport (2004-10-10) http://python.org/sf/1044299 opened by Debug Can't raise "C API version mismatch" warning (2004-10-11) http://python.org/sf/1044382 opened by Frank Sonnenburg docs for Py_UNICODE are wrong (2004-10-11) http://python.org/sf/1044479 opened by Thomas Heller Unable to install Docs - Linux (2004-10-11) http://python.org/sf/1044872 opened by Colin J. Williams strptime doesn't work with %U (2004-10-12) http://python.org/sf/1045381 opened by Sebastien JUST Problems with os.system and ulimit -f (2004-10-12) http://python.org/sf/1045509 opened by Tomasz Kowaltowski test_subprocess vs Windows (2004-10-12) CLOSED http://python.org/sf/1045748 opened by Tim Peters warning '"_POSIX_C_SOURCE" redefined' when include Python.h (2004-10-13) http://python.org/sf/1045893 opened by Eugene Sizikov urllib2: better import ftplib and gopherlib etc late (2004-10-13) http://python.org/sf/1046077 opened by Robert Kiendl HTMLParser fix to accept mailformed tag attributes (2004-10-13) http://python.org/sf/1046092 opened by Vsevolod Novikov improving distutils swig support (2004-10-14) CLOSED http://python.org/sf/1046404 opened by Anthony Baxter improving distutils swig support (2) (2004-10-14) CLOSED http://python.org/sf/1046644 opened by Lars Immisch difflib.HtmlDiff doc errors (EASY) (2004-10-13) CLOSED http://python.org/sf/1046690 opened by Dan Gass Error in the given example (2004-10-14) http://python.org/sf/1046800 opened by Vishnu httplib index out of range (2004-10-14) CLOSED http://python.org/sf/1046855 opened by Grzegorz Makarewicz improving distutils swig support - documentation (2004-10-14) http://python.org/sf/1046945 opened by Lars Immisch cgitb failures (2004-10-14) http://python.org/sf/1047397 opened by Robin Becker Bugs Closed ___________ copy.deepcopy exceptions.TypeError: 'NoneType' object ... (2004-10-06) http://python.org/sf/1041325 closed by arthur_lutz SimpleXMLRPCServer example is broken (2004-10-06) http://python.org/sf/1041501 closed by akuchling Thread management corrupts heap (2004-10-06) http://python.org/sf/1041645 closed by tim_one PyOS_InputHook broken (2004-09-19) http://python.org/sf/1030629 closed by mwh Patch (2004-10-07) http://python.org/sf/1042179 closed by mwh Lib/compiler chokes on certain genexps (2004-10-07) http://python.org/sf/1042238 closed by mwh Enhancements to pdb.py when invoked as script (2003-06-09) http://python.org/sf/751124 closed by jlgijsbers shutils.rmtree() uses excessive amounts of memory (2004-09-09) http://python.org/sf/1025127 closed by jlgijsbers Conflicting descriptions of application order of decorators (2004-09-21) http://python.org/sf/1031897 closed by akuchling x, y in curses window object documentation (2004-09-04) http://python.org/sf/1022311 closed by akuchling Confusing description of strict option for email.Parser (2004-09-22) http://python.org/sf/1032960 closed by bwarsaw email package uses \n to rebuild content of a message (2004-06-01) http://python.org/sf/964433 closed by bwarsaw smtpd.py docstring says wrong default class (2004-08-16) http://python.org/sf/1010102 closed by bwarsaw email: mishandles Content-Transfer-Enc. for text/* messages (2004-05-09) http://python.org/sf/950747 closed by bwarsaw Email message croaks the new email pkg parser (2004-09-19) http://python.org/sf/1030941 closed by bwarsaw Make GIL acquire/release functions noops if PyEval_Init (2004-08-18) http://python.org/sf/1011380 closed by tim_one test_subprocess vs Windows (2004-10-12) http://python.org/sf/1045748 closed by tim_one BSD restartable signals not correctly disabled (2004-06-09) http://python.org/sf/969574 closed by anthonybaxter Solaris likes sys/loadavg.h (2004-06-22) http://python.org/sf/977343 closed by anthonybaxter improving distutils swig support (2004-10-14) http://python.org/sf/1046404 closed by anthonybaxter improving distutils swig support (2) (2004-10-14) http://python.org/sf/1046644 closed by anthonybaxter difflib.HtmlDiff doc errors (EASY) (2004-10-13) http://python.org/sf/1046690 closed by tim_one httplib index out of range (2004-10-14) http://python.org/sf/1046855 closed by rhettinger

1 0

PyPy Vilnius Sprint 15-23 nov 2004
by hpk＠trillke.net Oct. 14, 2004

Oct. 14, 2004

Hi Pythonistas and interested developers, PyPy, the python-in-python implementation, is steadily moving on. The next coding sprint will take place in Vilnius, Lithunia, from 15th to 23rd of November, 2004 and is organized by the nice Programmers of Vilnius (POV) company. See http://codespeak.net/pypy/index.cgi?doc for more in-depth information about PyPy. Again, we will be heading towards a first generated C version of our already pretty compliant Python interpreter and types implementation. Last time, before the EuroPython 2004 conference, we actually had a similar goal (a PyPy/C-version) but discovered we had to largely refactor the basic model for attribute accesses. We are now closely mirroring the marvelous "descriptor"-mechanism of CPython. If you are interested to participate in our fun and somewhat mind-altering python sprint event then please subscribe at http://codespeak.net/moin/pypy/moin.cgi/VilniusSprintAttendants and look around for more information. You'll find that most of the core PyPy developers are already determined to come. There are also many areas that need attention so that we should have tasks suited for different levels of expertise. At http://codespeak.net/moin/pypy/moin.cgi/VilniusSprintTasks you'll find our sprint planning task list which will probably grow in the next weeks. Note that our EU funding efforts are at the final stage now. In the next weeks, quite likely before the Vilnius sprint, we _hope_ to get a "go!" from the european commission. One side effect would be that coders - probably restricted to european citizens - may generally apply for getting travel and accomodation costs refunded for PyPy sprints. This would reduce the barrier of entry to the question if you like to spend your time with a pypy sprint. However, we probably need some time to work out the details after when we get more information from the EU. If you have any questions don't hesitate to contact pypy-sprint(a)codespeak.net or one of us personally. cheers & a bientot, Holger Krekel, Armin Rigo

1 0

Cyclic GC issues
by Jason Evans Oct. 14, 2004

Oct. 14, 2004

Since the spring of 2003, I have been developing Crux, which is a computer program for phylogenetic inferencing (bioinformatics) research. In March of 2004, I switched Crux to using Python from having used a different embeddable interpreter. For the most part, I have been very happy with Python, but Python's garbage collector has been a major source of frustration. Below, I describe my trials and tribulations with Python's GC. I also offer some suggestions for changes to Python; if any of the proposed changes receive favorable feedback, I am willing to help develop patches to Python. Naturally, if I am somehow abusing Python, and there are better ways to do things, I'd be happy to hear how to improve Crux. The important aspect of Crux is that it deals with trees. These trees are unrooted (there is no up or down), and multifurcating (nodes can have an arbitrary number of neighboring nodes). Thus, the trees are self-referential, and without the cyclic GC capabilities of Python, there would be little hope of making these trees integrate well with Python. Following is a diagram that illustrates the linkage between various objects for a simple tree. Crux exposes all of the components of the trees as Python objects. All lines in the diagram represent bi-directional references (except for the T-->N line). Every object refers to the tree object; those lines are left out in order to reduce clutter. T: Tree N N N: Node \ / E: Edge R R R: Ring \ / E E \ / R---------R / \ / \ / \ / \ | \ / | | \ / | | T--->N | | | | \ | / \ | / \----R----/ | E | R | N At the C (not Python object) level, the R-E-R construct is actually a set of structures that are allocated/deallocated as a single unit. Edges are *always* connected to two rings, so there's no point in allocating these separately. Also, lone ring objects actually are rings with one member; they refer to themselves (prev and next pointers). That should be enough information to understand the problems I encountered. 1) I originally had lone rings refer to themselves (ob_refcnt started out at 3; 2 self-references and one reference held by the associated edge). This didn't work. It appears that the cyclic GC does not actually calculate the number of live references to objects (references that are reachable by traversing all objects accessible from the root set); instead it assumes that if tp_clear() doesn't drop the reference count to a number that equals the number of references from live objects, there must still be references from live objects. Unfortunately, visit()ing self does not work, so there is no way to convince Python that all references are from unreachable objects. Working around this in Crux requires a lot of extra reference counting complexity, because there are three different cases for reference counts, depending on how many members there are in a ring (1, 2, or 3+ members). 2) This issue is really just a different manifestation of issue (1). At the C (not Python object) level, each node only actually stores a pointer to a single member of the associated ring. Given a single ring member, it is possible to traverse the ring and reach all other ring members. As mentioned in issue (1), the cyclic GC expects tp_traverse() to call visit() once for each reference held. It is not enough for a node to visit() one ring member; it must visit() all ring members, in order for the GC to count how many references are from unreachable objects, versus reachable from the root set. In summary, issues (1) and (2) are due to how the cyclic GC does the "marking" phase of mark/sweep GC. My expectation of mark/sweep GC is that it should be sufficient to assure that all objects reachable from the root set are visit()ed at least once; it should not be important how many times each unreachable object is visit()ed. I don't have a deep enough understanding of the Python interpreter to give a detailed suggestion for improvement. I suspect that determining the root set is not an easy operation; if this is the case, then I think we're stuck with the current design. If determining the root set *is* easy (or even possible), then I would suggest using a standard mark/sweep collector, where unreachable objects are scheduled for destruction. tp_traverse(), tp_clear(), and tp_dealloc() would retain the same structure; the only difference would be the logic that determines which objects can be destroyed. 3) A strange thing can happen when tp_clear() is called. Suppose that an edge object is being cleared, and it drops its references to the associated rings. If ob_refcnt of one of the rings drops to 0 as a result, Python will tp_dealloc() the ring *right* *now*, without ever calling tp_clear() on the ring. That means that I have to keep track of whether tp_clear() has been called on objects, if it is at all important that tp_clear() be called, so that I can manually do so in tp_dealloc(), if necessary. It is in my opinion reasonable to have cleanup code in tp_clear(), with the assumption that it will be called precisely once, but Python makes me do extra work to make sure that this happens. This should be pretty easy to change. A single bit per object is needed to keep track of whether tp_clear() has been called. I think this only needs to be done for object types that support cyclic GC. 4) There is no way to control the order in which objects are tp_dealloc()ed. This is a problem for the R-E-R construct, since at a low level, these objects are always linked together. What I would like to do is defer tp_dealloc() on the edge until after both rings have been deleted. Instead, I am forced to use a reference-counted deletion function. Not calling self->ob_type->tp_free() on the edge in tp_dealloc() until later is not a reasonable option, because this defers deletion of the edge until a later round of garbage collection. This could be addressed in the Python interpreter by paying heed to the return value of tp_dealloc(). If the return value is non-zero, move the object to the end of the list of objects to be destroyed, so that destruction is tried later. This allows the module to impose its own destruction ordering. I look forward to feedback. Thank you, Jason Evans

7 12

[Python-Dev] Re: python/dist/src/Doc/whatsnew whatsnew24.tex, 1.108, 1.109
by Fredrik Lundh Oct. 14, 2004

Oct. 14, 2004

> +\begin{seealso} > +\seepep{324}{subprocess - New process module}{Written and implemented by Peter Astrand, with > assistance from Fredrik Lundh and others.} > +\end{seealso} I don't know how to add ISO-8859-1 characters to a Latex document, but in HTML notation, Peter's last name should be Åstrand. cheers /F

4 3