Mailman 3 October 2002 - Python-Dev

Re: [Python-Dev] Memory size overflows
by Gerald S. Williams 22 Oct '02

22 Oct '02

Guido van Rossum wrote: > I.e. a macro callable as > SAFE_MULTIPLY(destination, src1, src2, on_overflow); > meaning roughly > destination = src1 * src2; > if (<overflow detected>) > on_overflow; > There are also additions...These are easier to test: if you add a small > positive constant to a size_t, and the result is smaller than the > original size_t, an overflow occurred. Why not use the same trick for multiplication? For src1,src2 > 0, dest should always be >= MAX(src1,src2). SAFE_MULTIPLY could be implemented something like this: #define HALF_BITS (sizeof(size_t) * 4U) #define LO_MASK ((((size_t)1) << (HALF_BITS))-1) #define HI_MASK (~(LO_MASK)) #define MAX(a,b) (((a) >= (b)) ? (a) : (b)) #define SAFE_MULTIPLY(dest,src1,src2,on_error) \ { \ size_t _x = src1; \ size_t _y = src2; \ size_t _dest = _x * _y; \ \ if (_x && _y && ((_x|_y) & HI_MASK)) \ { \ if (_dest < MAX(_x,_y)) \ { \ on_error; \ } \ } \ \ dest = _dest; \ } -Jerry Williams

4 14

Re: [Python-Dev] __getitem__ in user defined classes
by Michael Hudson 21 Oct '02

21 Oct '02

"Raymond Hettinger" <python(a)rcn.com> writes: > In Py2.3, __getitem__ conveniently supports slices for > builtin sequences: 'abcde'.__getitem__(slice(2,4)) > > For user defined classes to emulate this behavior, they need > to test the index argument to see whether it is a slice and then > loop over the slice indices like this: > > class SquaresToTen: > """Acts like a list of squares but > computes only when needed""" > > def __len__(self): > return 11 > > def __getitem__(self, index): > if isinstance(index, slice): > return [x**2 for x in range(index.start, index.stop, index.step)] You can spell that if isinstance(index, slice): return [x**2 for x in range(*index.indices(11))] and, as a bonus, it'll work more often <wink> (consider "SquaresToTen()[7::-2]", for example). > This could be simplified somewhat by making slices iterable so that > the __getitem__ definition looks more like this: > > def __getitem__(self, index): > if isinstance(index, slice): > return [x**2 for x in index] > else: > return index**2 However to make omitted slice places work, you need to pass in the length of the sequence, so I don't think this can fly. Cheers, M. -- We've had a lot of problems going from glibc 2.0 to glibc 2.1. People claim binary compatibility. Except for functions they don't like. -- Peter Van Eynde, comp.lang.lisp

1 0

Discussion Draft: PyDev Summary for week ending 20-Oct-2002
by Raymond Hettinger 21 Oct '02

21 Oct '02

I'm filling in while Brett is on vacation. To make everyone appreciate his formatting and lucidity, I'll skip the reStructuredText and write in my own terse style. http://mail.python.org/pipermail/python-dev/2002-October/029577.html Python 2.2.2. was released in final form this week. Oh, yeah! http://mail.python.org/pipermail/python-dev/2002-October/029527.html Why does the Window's installer use shortened 8.3 pathnames? Answer: It doesn't have to but it less confusing than dealing with pathnames that can have embedded spaces. http://mail.python.org/pipermail/python-dev/2002-October/029544.html PEP 218 was updated to match the sets.py module in CVS for Python 2.3. The question arose as to whether immutable sets should be implemented in C as a separate type or whether mutability should be a property. Answer: Keep them as different types since they support different methods. Next question: Should there be a separate language syntax for the two different types. Answer: Probably not, leave as an open question in the PEP. http://mail.python.org/pipermail/python-dev/2002-October/029542.html The -U flag was put in to test the Python standard lib for being able to handle Unicode seamlessly. It remained undocumented so that bug reports wouldn't pile up while the issues were being worked out. Progress is being made but it is possible the -U flag will never be fully implemented and released as a documented feature. http://mail.python.org/pipermail/python-dev/2002-October/029535.html Python has more than one place that is vulnerable to 32 bit word overflows unless a bunch of tests are put in place to catch unusual highly contrived cases that never come up in real world examples. Since the fixes would be ugly, wordy, and slow, it was decided to leave them alone. http://mail.python.org/pipermail/python-dev/2002-October/029024.html Tcl 8.0 and 8.1 are ancient history except in the world of Cygwin which is currently stuck with 8.0. Accordingly, Python 2.3's support for the prior versions has been cleaned out and support for UCS-4 in _tkinter has been added. http://mail.python.org/pipermail/python-dev/2002-October/029591.html Getting mtime, atime, and ctime to return floats instead of ints seemed like a simple enough change. Much discussion was held on how to accommodate the change without breaking anything that relied on the type. Though breakage would've been rare (since the only normal use for the data is to compare it or print it), Guido proposed a solution (putting the ints in the tuple) that seemed to accommodate all practical cases. http://mail.python.org/pipermail/python-dev/2002-October/029679.html Brian Quinlan teasesd out the design issue behind proxies and weakref objects.

2 1

Weekly Python Bug/Patch Summary
by Skip Montanaro 20 Oct '02

20 Oct '02

Bug/Patch Summary ----------------- 316 open / 2963 total bugs (+4) 92 open / 1733 total patches (-4) New Bugs -------- textwrap fails on unicode using defaults (2002-10-13) http://python.org/sf/622831 inconsistent results of leading whitespace in textwrap input (2002-10-13) http://python.org/sf/622849 urllib.pathname2url(), .url2pathname() (2002-10-14) http://python.org/sf/623196 tempfile crashes (2002-10-15) http://python.org/sf/623464 __rdiv__ vs new-style classes (2002-10-15) http://python.org/sf/623669 Broken link in local documentation. (2002-10-16) http://python.org/sf/624024 email 2.4.3 pkg mail header error (2002-10-16) http://python.org/sf/624254 sq_concat prevents __radd__ from working (2002-10-17) http://python.org/sf/624807 Creation of struct_seq types (2002-10-17) http://python.org/sf/624827 help(UNICODE) -- reference page missing (2002-10-17) http://python.org/sf/624860 pydoc shows tools/scripts but blows up (2002-10-17) http://python.org/sf/624893 Typo at line 351 of email/Charset.py (2002-10-18) http://python.org/sf/625509 Errors with recursive objects (2002-10-19) http://python.org/sf/625698 Reorganize MacPython resources on OSX (2002-10-19) http://python.org/sf/625725 sys.executable in applets (2002-10-19) http://python.org/sf/625728 IDE starts in / on OSX (2002-10-19) http://python.org/sf/625734 missing names in telnetlib.py (2002-10-19) http://python.org/sf/625823 New Patches ----------- 2.2 patches for BSD/OS 5.0 (2002-03-26) http://python.org/sf/535335 autoconf-2.5X tweaks for configure.in (2002-10-15) http://python.org/sf/623771 HTTP Auth support for xmlrpclib (2002-10-16) http://python.org/sf/624180 attributes for urlsplit, urlparse result (2002-10-16) http://python.org/sf/624325 sets.BaseSet.isdisjointset(other) (2002-10-18) http://python.org/sf/625513 Closed Bugs ----------- Broken \ref link in documentation (2000-10-18) http://python.org/sf/217195 bad links at Ref Guide (2001-11-23) http://python.org/sf/484967 re.finditer (2002-07-24) http://python.org/sf/585882 list(xrange(sys.maxint / 4)) -> swapping (2002-09-28) http://python.org/sf/616019 segmentation fault importing huge source (2002-10-07) http://python.org/sf/619789 __ipow__ broken for 'object's (2002-10-08) http://python.org/sf/620179 core setup.py fails on Windows (2002-10-08) http://python.org/sf/620364 websucker relative-URL errors (2002-10-09) http://python.org/sf/620705 SocketServer/socket allow_reuse_address (2002-10-09) http://python.org/sf/620783 email: ASCII decoding error on 8bit data (2002-10-10) http://python.org/sf/621457 docs do not include spec info from PEPs (2002-10-10) http://python.org/sf/621526 Closed Patches -------------- specifying headers for extensions (2002-03-21) http://python.org/sf/533008 opt. timeouts for Queue.put() and .get() (2002-06-22) http://python.org/sf/572628 makesockaddr, use addrlen with AF_UNIX (2002-06-27) http://python.org/sf/574707 incompatible, but nice strings improveme (2002-07-08) http://python.org/sf/578688 smtplib.py patch for macmail esmtp auth (2002-07-17) http://python.org/sf/583180 db4 include not found (2002-08-02) http://python.org/sf/590377 zipfile.py reads archives with comments (2002-09-19) http://python.org/sf/611760 koi8_u codec (2002-09-23) http://python.org/sf/613173 tkFileDialog multiple option (2002-10-11) http://python.org/sf/621891

1 0

Gone for a week
by Guido van Rossum 19 Oct '02

19 Oct '02

I'm going to disappear for a week; back on the 28th. I'll be on a cruise ship teaching Python and Zope. (Can you say boondoggle? It's spelled www.geekcruises.com. :-) They've threatened to hook us up with internet access to maybe I won't be offline completely, but I expect to be checking for email rarely. The rest of the PL crew is at your service, of course! --Guido van Rossum (home page: http://www.python.org/~guido/)

1 0

To -I(nclude) or not to -I(nclude), that is the question...
by Skip Montanaro 19 Oct '02

19 Oct '02

In installing mxBase 2.0.4 on my MacOS 10.2.1 system I get warnings like the following: gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -Wno-long-double -no-cpp-precomp -Imx/Queue/mxQueue -I/Users/skip/local/include/python2.3 -I/usr/include -I/usr/local/include -c mx/Queue/mxQueue/mxQueue.c -o build/temp.darwin-6.1-Power Macintosh-2.3/mx/Queue/mxQueue/mxQueue/mxQueue.o cc1: warning: changing search order for system directory "/usr/local/include" cc1: warning: as it has already been specified as a non-system directory cc1: warning: changing search order for system directory "/usr/include" cc1: warning: as it has already been specified as a non-system directory This warning bothers me a bit, as it suggests I'm screwing up the compiler's notions about header file search order. Has anyone else seen this and investigated how to get rid of this problem? This is related to bug http://python.org/sf/589427 (which was assigned to me). It's due to the gen_preprocess_options() function in distutils/ccompiler.py. This warning seems to be related to gcc version >= 3.1. I have a quick hack in my local copy of distutils/ccompiler.py. At the bottom of gen_preprocess_options() I replaced for dir in include_dirs: pp_opts.append ("-I%s" % dir) with pp_opts.extend (gen_preprocess_includes(include_dirs)) and added these two functions to the file: def gen_preprocess_includes_macosx_gcc(dirs): """GCC on MacOSX complains if /usr/include or /usr/local/include are mentioned in -I. """ pp_opts = [] for dir in dirs: if dir not in ("/usr/include", "/usr/local/include"): pp_opts.append ("-I%s" % dir) return pp_opts def gen_preprocess_includes(dirs): """Generate the -I flags for a compile command.""" if sys.platform == "darwin": return gen_preprocess_includes_macosx_gcc(dirs) pp_opts = [] for dir in dirs: pp_opts.append ("-I%s" % dir) return pp_opts This is an obscure solution, at best. I'd prefer to at least test the compiler and version. How can I tell what compiler and version will be used to compile files, or can't I can this level? (Seems to me that gen_preprocess_options begs to be a method of the CCompiler class.) Thx, Skip

7 17

Incorporating bsddb3
by Martin v. Loewis 18 Oct '02

18 Oct '02

I have changes in my sandbox to incorporate bsddb3 into Python proper. This essentially is a copy of bsddb3 3.4.0, with the following modifications: - the extension module is called _bsddb (not bsddb3._db) - the package is called bsddb (not bsddb3) Both test_bsddb and test_anydbm continue to work unmodified. Is it ok to commit these changes? Regards, Martin

1 0

float atime/mtime/ctime - a bad idea?
by Guido van Rossum 18 Oct '02

18 Oct '02

I just found a place in my own code that broke because stat now returns floats for mtime etc. I was using a Zope "OIBTree" object, which is a dict-like extension type mapping strings to ints; it very reasonbly doesn't like it if you assign a float to it. I'm not say that because *my* code breaks, this is a bad feature. But I am worried that we'll find more programs using external data types that won't accept floats (I'm thinking of Numpy arrays), and I'm now not so sure if this is acceptable breakage... Apparently the fact that it "works" on the Mac isn't really a great proof. Maybe we can introduce a variant of the stat() function that returns floats, or alternative field names that are only available when using attributes (not when using the tuple-ish API)? --Guido van Rossum (home page: http://www.python.org/~guido/)

8 29

creating struct-seq types from Python
by Fred L. Drake, Jr. 18 Oct '02

18 Oct '02

Martin, I've addressed this to you since you seem to be playing with the relevant code these days, but others may be interested as well. Have you looked at what it would take to create new "structure sequence" types from Python code? This seems like a reasonable thing to do, especially when an existing API returns a long(ish) tuple of fields. I'd like to be able to use it for my proposed urlparse changes (http://www.python.org/sf/624325), but there seems to be no way to get to it. What's needed to do something currently feels like too much code for what little is being done (see the patch if unsure of what I mean). Thanks! -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation

2 2

Ann: Stackless technology merge under the way
by Christian Tismer 18 Oct '02

18 Oct '02

Hi friends, Stackless is changing heavily, once again. After yesterday's move to Python 2.2.2, I am now working on an integration of the old technology (software only) and the new (modifying the hardware stack). This has a number of advantages: - porting of existing applications, which depend on the old changes to Python's interpreter, becomes much easier. They tend to rely on a Python version of 2.0 or lower and use the old structures at the same time. Supporting these, again, makes it possible to do stepwise upgrades. - existing applications like thread pickling (written by TwinSun.com) will be enabled again and merged into Stackless as a feature. While limited to plain frame contexts (no C function or method calls in between), this is very worth doing. - the core Stackless code is using C stack manipulations right now in order to reduce stack size. This is not guaranteed to work with every C extension, since it is hard to check whether a foreign C call is sitting on the stack and relying on variables which must stay in place. The old technology was only able to switch Python stacks, when no recursive C calls were existing. This locks switches of C extensions automagically. - the new technology can be removed from the Python core and be only used for the stackless extension, for tasklet switching. This is not decided yet, but an option. - the amount of C stack space to me moved around is reduced drastically by this hybrid approach, since most of these vanish automatically. Although stack compression has been prepared (and will be supported), this is more time efficient. The reason while I have no longer a problem to make this decision is Python's current state. After generators have been introduced, the core code already has a lot of properties which I need to enable the old technology with ease. Furthermore, this support code will not again try to cover everything, but just the minimum to avoid most recursion. It is neither planned to support stackless map or apply, nor are continuations desired. The target are applications which implemented coroutines on top of the old kernel. This hybrid implementation has started today and will continue over a couple of weeks. The last "plain hardware switching" version is available under the CVS tag "last_hardware_only", which is based on the final 2.2.2 version. I regard this move as the beginning of the re-union of my brain halves :-) continuation-less - ly y'rs chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

2 2