Mailman 3 April 2000 - Python-Dev

ANNOUNCE: Distutils 0.8 released
by Greg Ward 12 Apr '00

12 Apr '00

Python Distribution Utilities release 0.8 April 11, 2000 The Python Distribution Utilities, or Distutils for short, are a collection of modules that aid in the development, distribution, and installation of Python modules. (It is intended that ultimately the Distutils will grow up into a system for distributing and installing whole Python applications, but for now their scope is limited to module distributions.) The Distutils are a standard part of Python 1.6; if you are running 1.6, you don't need to install the Distutils separately. This release is primarily so that you can add the Distutils to a Python 1.5.2 installation -- you will then be able to install modules that require the Distutils, or use the Distutils to distribute your own modules. More information is available at the Distutils web page: http://www.python.org/sigs/distutils-sig/ and in the README.txt included in the Distutils source distribution. You can download the Distutils from http://www.python.org/sigs/distutils-sig/download.html Trivial patches can be sent to me (Greg Ward) at gward(a)python.net. Larger patches should be discussed on the Distutils mailing list: distutils-sig(a)python.org. Here are the changes in release 0.8, if you're curious: * some incompatible naming changes in the command classes -- both the classes themselves and some key class attributes were renamed (this will break some old setup scripts -- see README.txt) * half-hearted, unfinished moves towards backwards compatibility with Python 1.5.1 (the 0.1.4 and 0.1.5 releases were done independently, and I still have to fold those code changes in to the current code) * added ability to search the Windows registry to find MSVC++ (thanks to Robin Becker and Thomas Heller) * renamed the "dist" command to "sdist" and introduced the "manifest template" file (MANIFEST.in), used to generate the actual manifest * added "build_clib" command to build static C libraries needed by Python extensions * fixed the "install" command -- we now have a sane, usable, flexible, intelligent scheme for doing standard, alternate, and custom installations (and it's even documented!) (thanks to Fred Drake and Guido van Rossum for design help) * straightened out the incompatibilities between the UnixCCompiler and MSVCCompiler classes, and cleaned up the whole mechanism for compiling C code in the process * reorganized the build directories: now build to either "build/lib" or "build/lib.<plat>", with temporary files (eg. compiler turds) in "build/temp.<plat>" * merged the "install_py" and "install_ext" commands into "install_lib" -- no longer any sense in keeping them apart, since pure Python modules and extension modules build to the same place * added --debug (-g) flag to "build_*" commands, and make that carry through to compiler switches, names of extensions on Windows, etc. * fixed many portability bugs on Windows (thanks to many people) * beginnings of support for Mac OS (I'm told that it's enough for the Distutils to install itself) (thanks to Corran Webster) * actually pay attention to the "--rpath" option to "build_ext" (thanks to Joe Van Andel for spotting this lapse) * added "clean" command (thanks to Bastien Kleineidam) * beginnings of support for creating built distributions: changes to the various build and install commands to support it, and added the "bdist" and "bdist_dumb" commands * code reorganization: split core.py up into dist.py and cmd.py, util.py into *_util.py * removed global "--force" option -- it's now up to individual commands to define this if it makes sense for them * better error-handling (fewer extravagant tracebacks for errors that really aren't the Distutils' fault -- Greg Ward - just another Python hacker gward(a)python.net http://starship.python.net/~gward/ All the world's a stage and most of us are desperately unrehearsed.

1 0

How *does* Python determine sys.prefix?
by Greg Ward 12 Apr '00

12 Apr '00

Ooh, here's a yucky problem. Last night, I installed Oliver Andrich's Python 1.5.2 RPM on my Linux box at home, so now I have two Python installations there: * my build, in /usr/local/python and /usr/local/python.i86-linux (I need to test Distutils in the prefix != exec_prefix case) * Oliver's RPM, in /usr I have a symlink /usr/local/bin/python pointing to ../../python.i86-linux/bin/python, and /usr/local/bin is first in my path: $ ls -lF `which python` lrwxrwxrwx 1 root root 30 Aug 28 1999 /usr/local/bin/python -> ../python.i86-linux/bin/python* Since I installed the RPM, /usr/local/bin/python reports an incorrect prefix: $ /usr/local/bin/python Python 1.5.2 (#1, Jun 20 1999, 19:56:42) [GCC 2.7.2.3] on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import sys ; sys.prefix, sys.exec_prefix ('/usr', '/usr/local/bin/../python.i86-linux') Essentially the same thing if I run it directly, not through the symlink: $ /usr/local/python.i86-linux/bin/python Python 1.5.2 (#1, Jun 20 1999, 19:56:42) [GCC 2.7.2.3] on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import sys ; sys.prefix, sys.exec_prefix ('/usr', '/usr/local/python.i86-linux') /usr/bin/python gets it right, though: $ /usr/bin/python Python 1.5.2 (#1, Apr 18 1999, 16:03:16) [GCC pgcc-2.91.60 19981201 (egcs-1.1.1 on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import sys ; sys.prefix, sys.exec_prefix ('/usr', '/usr') This strikes me as a pretty reasonable and straightforward way to have multiple Python installations; if Python is fooled into getting the wrong sys.prefix, then the Distutils are going to have a much tougher job! Don't tell me I have to write my own prefix-finding code now... (And no, I have not tried this under 1.6 yet.) Damn and blast my last-minute pre-release testing... I should have just released the bloody thing and let the bugs fly. Oh hell, I think I will anyways. Greg -- Greg Ward - software developer gward(a)mems-exchange.org MEMS Exchange / CNRI voice: +1-703-262-5376 Reston, Virginia, USA fax: +1-703-262-5367

2 1

ANNOUNCE: Python 1.6 alpha 2
by Guido van Rossum 12 Apr '00

12 Apr '00

I've just released a source tarball and a Windows installer for Python 1.6 alpha 2 to the Python website: http://www.python.org/1.6/ If you missed the announcement for 1.6a1, probably the biggest news is Unicode support. More news is on the above webpage; Unicode is being discussed in the i18n-sig. Most changes since 1.6a1 affect either details of the Unicode support, or details of what the Windows installer installs where. Note: this is an alpha release. Some of the code is very rough! Please give it a try with your favorite Python application, but don't trust it for production use yet. I plan to release several more alpha and beta releases over the next two months, culminating in an 1.6 final release before June first. We need your help to make the final 1.6 release as robust as possible -- please test this alpha release!!! --Guido van Rossum (home page: http://www.python.org/~guido/)

2 1

Re: [Patches] Unicode Patch Set 2000-04-10
by Greg Stein 12 Apr '00

12 Apr '00

On Mon, 10 Apr 2000, M.-A. Lemburg wrote: > The attached patch includes the following fixes and additions: >... > * '...%s...' % u"abc" now coerces to Unicode just like > string methods. Care is taken not to reevaluate already formatted > arguments -- only the first Unicode object appearing in the > argument mapping is looked up twice. Added test cases for > this to test_unicode.py. >... I missed a chance to bring this up on the first round of discussion, but is this really the right thing to do? We never coerce the string on the left based on operands. For example: if the operands are class instances, we call __str__ -- we don't call __coerce__. It seems a bit weird to magically revise the left operand. In many cases, a Unicode used as a string is used as a UTF-8 value. Why is that different in this case? Seems like a wierd special case. Cheers, -g -- Greg Stein, http://www.lyra.org/

5 8

Re: [Patches] add string precisions to PyErr_Format calls
by Greg Stein 11 Apr '00

11 Apr '00

On Tue, 11 Apr 2000, Andrew M. Kuchling wrote: > Greg Stein writes: > >Wouldn't it be best to simply fix PyErr_Format so that we don't have to > >continue to worry about buffer overruns? > > A while ago I suggested using nsprintf() in PyErr_Format, but that > means stealing the implementation from Apache for those platforms > where libc doesn't include nsprintf(). Haven't done it yet... Seems like it would be cake to write one that took *only* the %d and %s (unadorned) modifiers. We wouldn't need anything else, would we? [ ... grep'ing the source ... ] I see the following format codes which would need to change: %.###s -- switch to %s %i -- switch to %d %c -- hrm. probably need to support this (in stringobject.c) %x -- maybe switch to %d? (in stringobject.c) The last two are used once, both in stringobject.c. I could see a case for revising that call use just %s and %d. One pass to count the length, alloc, then one pass to fill in. The second pass could actually be handled by vsprintf() since we know the buffer is large enough. The only tricky part would be determining the max length for %d. For a 32-bit value, it is 10 digits; for 64-bit value, it is 20 digits. I'd say allocate room for 20 digits regardless of platform and be done with it. Maybe support %%, but I didn't see that anywhere. Somebody could add support when the need arises. Last problem: backwards compat for third-party modules using PyErr_Format. IMO, leave PyErr_Format for them (they're already responsible for buffer overruns (or not) since PyErr_Format isn't helping them). The new one would be PyErr_SafeFormat. Recommend the Safe version, deprecate the unsafe one. Cheers, -g -- Greg Stein, http://www.lyra.org/

1 0

Python for small platforms
by Jeff Collins 11 Apr '00

11 Apr '00

I've just had the chance to examine the unicode implementation and was surprised by the size of the code introduced - not just by the size of the database extension module (which I understand Christian Tismer is optimizing and which I assume can be configured away), but in particular by the size of the additional objects (unicodeobject.c, unicodetype.c). These additional objects alone contribute approximately 100K to the resulting executable. On desktop systems, this is not of much concern and suggestions have been made previously to reduce this if necessary (shared extension modules and possibly a shared VM - libpython.so). However, on small embedded systems (eg, PalmIII), this additional code is tremendous. The current size of the python-1.5.2-pre-unicode VM (after removal of float and complex objects with more reductions to come) on the PalmIII is 240K (already huge by Palm standards). (For reference, the size of python-1.5.1 on the PalmIII is 160K, after removal of the compiler, parser, float/long/complex objects.) With the unicode additions, this value jumps to 340K. The upshot of this is that for small platforms on which I am working, unicode support will have to be removed. My immediated concern is that unicode is getting so embedded in python that it will be difficult to extract. The approach I've taken for removing "features" (like float objects): 1) removes the feature with WITHOUT_XXX #ifdef/#endif decorations, where XXX denotes the removable feature (configurable in config.h) 2) preserves the python API: builtin functions, C API, PyArg_Parse, print format specifiers, etc., raise MissingFeatureError if attempts are made to use them. Of course, the API associated with the removed feature is no longer present. 3) protects the reduced VM: all reads (via marshal, compile, etc.) involving source/compiled python code will fail with a MissingFeatureError if the reduced VM doesn't support it. 4) does not yet support a MissingFeatureError in the tokenizer if, say, 2.2 (for removed floats) is entered on the python command line. This instead results in a SyntaxError indicating a problem with the decimal point. It appears that another error token would have to be added to support this error. Of course, I may have missed something, but if the above appears to be a reasonable approach, I can supply patches (at least for floats and complexes) for further discussion. In the longer term, it would be helpful if developers would follow this (or a similar agreed upon approach) when adding new features. This would reduce the burden of maintaining python for small embedded platforms. Thanks, Jeff

2 1

Re: [Python-checkins] CVS: distutils README.txt,1.9,1.10
by Guido van Rossum 11 Apr '00

11 Apr '00

You realize that that README didn't make it into 1.6a2, right? Shouldn't be a problem. --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

Re: [Python-Dev] Crash in new "trashcan" mechanism.
by Christian Tismer 11 Apr '00

11 Apr '00

Mark Hammond wrote: > > > Can you perhaps tell me what the call stack says? > > Is it somewhere, or are we in finalization code of the > > interpreter? > > The crash is in _Py_Dealloc - op is a pointer, but all fields > (ob_type, ob_refcnt, etc) are all 0 - hence the crash. > > Next up is list_dealloc - op is also trashed - ob_item != NULL > (hence we are in the if condition, and calling Py_XDECREF() (which > triggers the _Py_Dealloc) - ob_size ==9, but all other fields are 0. > > Next up is Py_Dealloc() > > Next up is _PyTrash_Destroy() > > Next up is frame_dealloc() > > _Py_Dealloc() > > Next up is eval_code2() - the second last line - Py_DECREF(f) to > cleanup the frame it just finished executing. > > Up the stack are lots more eval_code2() - we are just running the > code - not shutting down or anything. And you do obviously not have any threads, right? And you are in the middle of a simple, heavy computing application. Nothing with GUI events happening? That can only mean there is a bug in the Python core or in the parser module. That happens to be exposed by trashcan, but isn't trashcan's fault. Well. Trashcan might change the order of destruction a little. This *should* not be a problem. But here is a constructed situation where I can think of a problem, if we have buggy code, somewhere: Assume you have something like a tuple that holds other elements. If there is a bug, like someone is dereferencing an argument in an arg tuple, what is always an error. This error can hide for a million of years: a contains (b, c, d) The C function decref's a first, and erroneously then also one of the contained elements. If b is already deallotted by decreffing a, it has refcount zero, but that doesn't hurt, since the dead object is still there, and no mallcos have taken place (unless there is a __del__ trigered of course). This eror would never be detected. With trashcan, it could happen that destruction of a is deferred, but by chance now the delayed erroneous decref of b might happen before a's decref, and there may be mallocs in between, since I have a growing list. If my code is valid (and it appears so), then I guess we have such a situation somewhere in the core code. I-smell-some-long-nightshifts-again - ly y'rs - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaunstr. 26 : *Starship* http://starship.python.net 14163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF where do you want to jump today? http://www.stackless.com

1 0

Re: [Python-checkins] CVS: python/dist/src/Include unicodeobject.h,2.7,2.8
by Fredrik Lundh 11 Apr '00

11 Apr '00

> Changed PyUnicode_Splitlines() maxsplit argument to keepends. shouldn't that be "PyUnicode_SplitLines" ? (and TailMatch, IsLineBreak, etc.) </F>

1 0

1.6a2 prerelease for Windows
by Guido van Rossum 11 Apr '00

11 Apr '00

I've made a prerelease of the Windows installer available through the python.org/1.6 webpage (the link is in the paragraph *below* the a1 downloads). This is mostly to give Mark Hammond an opportunity to prepare win32all build 131, to deal with the changed location of the python16.dll file. Hey, it's still alpha software! --Guido van Rossum (home page: http://www.python.org/~guido/)

2 1