Mailman 3 June 2002 - Python-Dev

On the topic of garbage collection
by Neil Schemenauer 29 Jun '02

29 Jun '02

Seen on the net: http://www.ravenbrook.com/project/mps/ The Memory Pool System is a very general, adaptable, flexible, reliable, and efficient memory management system. It permits the flexible combination of memory management techniques, supporting manual and automatic memory management, in-line allocation, finalization, weakness, and multiple concurrent co-operating incremental generational garbage collections. It also includes a library of memory pool classes implementing specialized memory management policies. The code is offered under an open source license. Neil

1 0

Improved tmpfile module
by Zack Weinberg 29 Jun '02

29 Jun '02

I'm not subscribed to python-dev. Please cc: me directly on replies. I'm going to respond to all the comments at once. Greg Ward wrote: > > Attached please find a rewritten and improved tmpfile.py. The major > > change is to make the temporary file names significantly harder to > > predict. This foils denial-of-service attacks, where a hostile > > program floods /tmp with files named @12345.NNNN to prevent process > > 12345 from creating any temp files. It also makes the race condition > > inherent in tmpfile.mktemp() somewhat harder to exploit. > > Oh, good! I've long wished that there was a tmpfile module written by > someone who understands the security issues involved in generating > temporary filenames and files. I hope you do... ;-) Well, I wrote the analogous code in the GNU C library (using basically the same algorithm). I'm confident it is safe on a Unix-based system. On Windows and others, I am relying on os.open(..., os.O_EXCL) to do what it claims to do; assuming it does, the code should be safe there too. > > (fd, name) = mkstemp(suffix="", binary=1): Creates a temporary file, > > returning both an OS-level file descriptor open on it and its name. > > This is useful in situations where you need to know the name of the > > temporary file, but can't risk the race in mktemp. > > +1 except for the name. What does the "s" stand for? Unfortunately, I > can't think of a more descriptive name offhand. Fredrik Lundh's suggestion that it is for "safer" seems plausible, but I do not actually know. I chose the names mkstemp and mkdtemp to match the functions of the same name in most modern Unix C libraries. Since they don't take the same "template" parameter that those functions do, that was probably a bad idea. [Note to Fredrik: at the C level, mkstemp is not deprecated in favor of tmpfile, as they do very different things - tmpfile(3) is analogous to tmpfile.TemporaryFile(), you don't get the file name back.] I'm open to suggestions for a better routine name; I can't think of a good one myself. > > name = mkdtemp(suffix=""): Creates a temporary directory, without > > race. > > How about calling this one mktempdir() ? Sure. > I've scanned your code and the existing tempfile.py. I don't > understand why you rearranged things. Please explain why your > arrangement of _TemporaryFileWrapper/TemporaryFile/ > NamedTemporaryFile is better than what we have. I was trying to get all the user-accessible interfaces to be at the top of the file. Also, I do not understand the bits in the existing file that delete names out of the module namespace after we're done with them, so I wound up taking all of that out to get it to work. I think the existing file's organization was largely determined by those 'del' statements. I'm happy to organize the file any way y'all like -- I'm kind of new to Python and I don't know the conventions yet. > A few minor comments on the code... > > > if os.name == 'nt': > > _template = '~%s~' > > elif os.name in ('mac', 'riscos'): > > _template = 'Python-Tmp-%s' > > else: > > _template = 'pyt%s' # better ideas? > > Why reveal the implementation language of the application creating these > temporary names? More importantly, why do it certain platforms, but not > others? This is largely as it was in the old file. I happen to know that ~%s~ is conventional for temporary files on Windows. I changed 'tmp%s' to 'pyt%s' for Unix to make it consistent with Mac/RiscOS Ideally one would allow the calling application to control the prefix, but I'm not sure what the right interface is. Maybe tmpfile.mkstemp(prefix="", suffix="") where if one argument is provided it gets treated as the suffix, but if two are provided the prefix comes first, a la range()? Is there a way to express that in the prototype? > > ### Recommended, user-visible interfaces. > > > > _text_openflags = os.O_RDWR | os.O_CREAT | os.O_EXCL > > if os.name == 'posix': > > _bin_openflags = os.O_RDWR | os.O_CREAT | os.O_EXCL > > Why not just "_bin_openflags = _text_openflags" ? That clarifies their > equality on Unix. > > > else: > > _bin_openflags = os.O_RDWR | os.O_CREAT | os.O_EXCL | os.O_BINARY > > Why not "_bin_openflags = _text_openflags | os.O_BINARY" ? *shrug* Okay. > > > def mkstemp(suffix="", binary=1): > > """Function to create a named temporary file, with 'suffix' for > > its suffix. Returns an OS-level handle to the file and the name, > > as a tuple. If 'binary' is 1, the file is opened in binary mode, > > otherwise text mode (if this is a meaningful concept for the > > operating system in use). In any case, the file is readable and > > writable only by the creating user, and executable by no one.""" > > "Function to" is redundant. I didn't change much of this text from the old file. Where are docstring conventions documented? > """Create a named temporary file. > > Create a named temporary file with 'suffix' for its suffix. Return > a tuple (fd, name) where 'fd' is an OS-level handle to the file, and > 'name' is the complete path to the file. If 'binary' is true, the > file is opened in binary mode, otherwise text mode (if this is a > meaningful concept for the operating system in use). In any case, > the file is readable and writable only by the creating user, and > executable by no one (on platforms where that makes sense). > """ Okay. > Hmmm: if suffix == ".bat", the file is executable on some platforms. > That last sentence still needs work. ... In any case, the file is readable and writable only by the creating user. On platforms where the file's permission bits control whether it can be executed as a program, no one can. Other platforms have other ways of controlling this: for instance, under Windows, the suffix determines whether the file can be executed. How's that? > > class _TemporaryFileWrapper: > > """Temporary file wrapper > > > > This class provides a wrapper around files opened for temporary use. > > In particular, it seeks to automatically remove the file when it is > > no longer needed. > > """ > > Here's where I started getting confused. I don't dispute that the > existing code could stand some rearrangement, but I don't understand why > you did it the way you did. Please clarify! See above. What would you consider a sensible arrangement? > > > ### Deprecated, user-visible interfaces. > > > > def mktemp(suffix=""): > > """User-callable function to return a unique temporary file name.""" > > while 1: > > name = _candidate_name(suffix) > > if not os.path.exists(name): > > return name > > The docstring for mktemp() should state *why* it's bad to use this > function -- otherwise people will say, "oh, this looks like it does what > I need" and use it in ignorance. So should the library reference > manual. Good point. """Suggest a name to be used for a temporary file. This function returns a file name, with 'suffix' for its suffix, which did not correspond to any file at some point in the past. By the time you get the return value of this function, a file may have already been created with that name. It is therefore unsafe to use this function for any purpose. It is deprecated and may be removed in a future version of Python.""" and corresponding text in the library manual? Tim Peters wrote: > > -1 on the implementation here, because it didn't start with current CVS, so > is missing important work that went into improving this module on Windows > for 2.3. Whether spawned/forked processes inherit descriptors for "temp > files" is also a security issue that's addressed in current CVS but seemed > to have gotten dropped on the floor here. I'll get my hands on a copy of current CVS and rework my changes against that. > A note on UI: for many programmers, "it's a feature" that temp file names > contain the pid. I don't think we can get away with taking that away no > matter how stridently someone claims it's bad for us <wink>. GNU libc took that away from C programmers about four years ago and no one even noticed. FreeBSD libc, ditto, although I'm not sure when it happened. zw

4 3

list.extend
by David Abrahams 29 Jun '02

29 Jun '02

I just submitted a patch to the list.extend docstring, to reflect the fact that x.extend(xrange(10)) and x.extend((2,3)) both work when x is a list. Then I went to look at the documentation and noticed it says at http://www.python.org/dev/doc/devel/lib/typesseq-mutable.html: s.extend(x) same as s[len(s):len(s)] = x (2) ... (2) Raises an exception when x is not a list object. The extend() method is experimental and not supported by mutable sequence types other than lists. Now I'm wondering what all this means. It is /not/ equivalent to the slice assignment, because list slice assignment requires a list rhs. What does this "experimental" label mean? Is my patch to the docstring wrong, in the sense that it suggests exploiting undefined behavior in the same way that the old append-multiple-items behavior was undefined? Also, I note that the table referenced above seems to be missing some right parentheses, at least on the .pop and .sort method descriptions. -Dave +---------------------------------------------------------------+ David Abrahams C++ Booster (http://www.boost.org) O__ == Pythonista (http://www.python.org) c/ /'_ == resume: http://users.rcn.com/abrahams/resume.html (*) \(*) == email: david.abrahams(a)rcn.com +---------------------------------------------------------------+

7 15

pre.sub's broken under 2.2
by Fredrik Lundh 28 Jun '02

28 Jun '02

just for the record, one of those "let's change a lot of code that we don't understand, just because we can" things broke the "pre" module in 2.2. someone changed: try: repl = pcre_expand(_Dummy, repl) except: m = MatchObject(self, source, 0, end, []) to try: repl = pcre_expand(_Dummy, repl) except error: m = MatchObject(self, source, 0, end, []) but in the most common use case (replacement strings containing group references), the pcre_expand function raises a TypeError exception... </F>

2 3

SF task tracker confusion
by Fredrik Lundh 28 Jun '02

28 Jun '02

on my "my sf.net" page, there are a couple of development tasks listed for python 2.1 (!). however, if I click on one of the links, e.g. https://sourceforge.net/pm/task.php?func=detailtask&project_task_id=25031&g… all I get is a page saying that: Permission Denied This project's administrator will have to grant you permission to view this page. any ideas? maybe the project's administrator could remove the tasks for me? </F>

2 2

Help, Compile / debug Python 2.2.1 64 bit on AIX
by Davide Tacchella 27 Jun '02

27 Jun '02

I'm trying to build Python with 64 bit support on AIX, so far I've encountered 2 problems, dynload_aix.c is not 100% 64 bit compliant (it includes some cast from pointer to (int)); this was causing python to SEGV when building extensions, after changing from int to long, the error is now ILL (SIGILL), there is a pointer to NULL: The call stack from debugger is: 0x000000 initstruct (structmodule.c - line 1508) _PyImport_LoadDynamicModule ( importdl.c - line 53) load_module (import.c - line 1365) import_submodule (import.c - line 1895) load_next (import.c - line 1751) import_module_ex (import.c - line 1602) PyImport_ImportModuleEx (import.c - line 1643) builtin___import__ (bltinmodule.c - line 40) PyCFunction_Call (methodobject.c - line 80) eval_frame (ceval.c - line 2004) .... Any idea ? Any help is always welcome. Can anybody help me out ? Davide

1 0

Improved tmpfile module
by Zack Weinberg 25 Jun '02

25 Jun '02

Attached please find a rewritten and improved tmpfile.py. The major change is to make the temporary file names significantly harder to predict. This foils denial-of-service attacks, where a hostile program floods /tmp with files named @12345.NNNN to prevent process 12345 from creating any temp files. It also makes the race condition inherent in tmpfile.mktemp() somewhat harder to exploit. I also implemented three new interfaces: (fd, name) = mkstemp(suffix="", binary=1): Creates a temporary file, returning both an OS-level file descriptor open on it and its name. This is useful in situations where you need to know the name of the temporary file, but can't risk the race in mktemp. name = mkdtemp(suffix=""): Creates a temporary directory, without race. file = NamedTemporaryFile(mode='w+b', bufsize=-1, suffix=""): This is just the non-POSIX version of tmpfile.TemporaryFile() made available on all platforms, and with the .path attribute documented. It provides a convenient way to get a temporary file with a name, that will be automatically deleted on close, and with a high-level file object associated with it. Finally, I tore out a lot of the posix/not-posix conditionals, relying on the os module to provide open() and O_EXCL -- this should make all the recommended interfaces race-safe on non-posix systems, which they were not before. Comments? I would very much like to see something along these lines in 2.3; I have an application that needs to be reliable in the face of the aforementioned denial of service. Please note that I wound up removing all the top-level 'del foo' statements (cleaning up the namespace) as I could not figure out how to do them properly. I'm not a python guru. zw """Temporary files and filenames.""" import os from errno import EEXIST from random import Random __all__ = [ "TemporaryFile", "NamedTemporaryFile", # recommended (high level) "mkstemp", "mkdtemp", # recommended (low level) "mktemp", "gettempprefix", # deprecated "tempdir", "template" # control ] ### Parameters that the caller may set to override the defaults. tempdir = None # _template contains an appropriate pattern for the name of each # temporary file. if os.name == 'nt': _template = '~%s~' elif os.name in ('mac', 'riscos'): _template = 'Python-Tmp-%s' else: _template = 'pyt%s' # better ideas? ### Recommended, user-visible interfaces. _text_openflags = os.O_RDWR | os.O_CREAT | os.O_EXCL if os.name == 'posix': _bin_openflags = os.O_RDWR | os.O_CREAT | os.O_EXCL else: _bin_openflags = os.O_RDWR | os.O_CREAT | os.O_EXCL | os.O_BINARY def mkstemp(suffix="", binary=1): """Function to create a named temporary file, with 'suffix' for its suffix. Returns an OS-level handle to the file and the name, as a tuple. If 'binary' is 1, the file is opened in binary mode, otherwise text mode (if this is a meaningful concept for the operating system in use). In any case, the file is readable and writable only by the creating user, and executable by no one.""" if binary: flags = _bin_openflags else: flags = _text_openflags while 1: name = _candidate_name(suffix) try: fd = os.open(name, flags, 0600) return (fd, name) except OSError, e: if e.errno == EEXIST: continue # try again raise def mkdtemp(suffix=""): """Function to create a named temporary directory, with 'suffix' for its suffix. Returns the name of the directory. The directory is readable, writable, and searchable only by the creating user.""" while 1: name = _candidate_name(suffix) try: os.mkdir(name, 0700) return name except OSError, e: if e.errno == EEXIST: continue # try again raise class _TemporaryFileWrapper: """Temporary file wrapper This class provides a wrapper around files opened for temporary use. In particular, it seeks to automatically remove the file when it is no longer needed. """ # Cache the unlinker so we don't get spurious errors at shutdown # when the module-level "os" is None'd out. Note that this must # be referenced as self.unlink, because the name TemporaryFileWrapper # may also get None'd out before __del__ is called. unlink = os.unlink def __init__(self, file, path): self.file = file self.path = path self.close_called = 0 def close(self): if not self.close_called: self.close_called = 1 self.file.close() self.unlink(self.path) def __del__(self): self.close() def __getattr__(self, name): file = self.__dict__['file'] a = getattr(file, name) if type(a) != type(0): setattr(self, name, a) return a def NamedTemporaryFile(mode='w+b', bufsize=-1, suffix=""): """Create a named temporary file, with 'suffix' for its suffix. It will automatically be deleted when it is closed. Pass 'mode' and 'bufsize' to fdopen. Returns a file object; the name of the file is accessible as file.path.""" if 'b' in mode: binary = 1 else: binary = 0 (fd, name) = mkstemp(suffix, binary) file = os.fdopen(fd, mode, bufsize) return _TemporaryFileWrapper(file, name) if os.name != 'posix': # A file cannot be unlinked while open, so TemporaryFile # degenerates to NamedTemporaryFile. TemporaryFile = NamedTemporaryFile else: def TemporaryFile(mode='w+b', bufsize=-1, suffix=""): """Create a temporary file. It has no name and will not survive being closed; the 'suffix' argument is ignored. Pass 'mode' and 'bufsize' to fdopen. Returns a file object.""" if 'b' in mode: binary = 1 else: binary = 0 (fd, name) = mkstemp(binary=binary) file = os.fdopen(fd, mode, bufsize) os.unlink(name) return file ### Deprecated, user-visible interfaces. def mktemp(suffix=""): """User-callable function to return a unique temporary file name.""" while 1: name = _candidate_name(suffix) if not os.path.exists(name): return name def gettempprefix(): """Function to calculate a prefix of the filename to use. This incorporates the current process id on systems that support such a notion, so that concurrent processes don't generate the same prefix. """ global _template return (_template % `os.getpid`) + '.' ### Threading gook. try: from thread import allocate_lock except ImportError: class _DummyMutex: def acquire(self): pass release = acquire def allocate_lock(): return _DummyMutex() del _DummyMutex _init_once_lock = allocate_lock() def _init_once(var, constructor): """If 'var' is not None, initialize it to the return value from 'constructor'. Do this exactly once, no matter how many threads call this routine. FIXME: How would I cause 'var' to be passed by reference to this routine, so that the caller can write simply _init_once(foo, make_foo) instead of foo = _init_once(foo, make_foo) ?""" # Check once outside the lock, so we can avoid acquiring it if # the variable has already been initialized. if var is not None: return var try: _init_once_lock.acquire() # Check again inside the lock, in case someone else got # here first. if var is None: var = constructor() finally: _init_once_lock.release() return var ### Internal routines and data. _seq = None def _candidate_name(suffix): """Return a candidate temporary name in 'tempdir' (global) ending with 'suffix'.""" # We have to make sure that _seq and tempdir are initialized only # once, even in the presence of multiple threads of control. global _seq global tempdir _seq = _init_once(_seq, _RandomFilenameSequence) tempdir = _init_once(tempdir, _gettempdir) # Most of the work is done by _RandomFilenameSequence. return os.path.join(tempdir, _seq.get()) + suffix class _RandomFilenameSequence: characters = ( "abcdefghijklmnopqrstuvwxyz" + "ABCDEFGHIJKLMNOPQRSTUVWXYZ" + "0123456789-_") def __init__(self): self.mutex = allocate_lock() self.rng = Random() def get(self): global _template # Only one thread can call into the RNG at a time. self.mutex.acquire() c = self.characters r = self.rng letters = ''.join([r.choice(c), r.choice(c), r.choice(c), r.choice(c), r.choice(c), r.choice(c)]) self.mutex.release() return (_template % letters) # XXX This tries to be not UNIX specific, but I don't know beans about # how to choose a temp directory or filename on MS-DOS or other # systems so it may have to be changed... # _gettempdir deduces whether a candidate temp dir is usable by # trying to create a file in it, and write to it. If that succeeds, # great, it closes the file and unlinks it. There's a race, though: # the *name* of the test file it tries is the same across all threads # under most OSes (Linux is an exception), and letting multiple threads # all try to open, write to, close, and unlink a single file can cause # a variety of bogus errors (e.g., you cannot unlink a file under # Windows if anyone has it open, and two threads cannot create the # same file in O_EXCL mode under Unix). The simplest cure is to serialize # calls to _gettempdir, which is done above in _candidate_name(). def _gettempdir(): """Function to calculate the directory to use.""" try: pwd = os.getcwd() except (AttributeError, os.error): pwd = os.curdir attempdirs = ['/tmp', '/var/tmp', '/usr/tmp', pwd] if os.name == 'nt': attempdirs.insert(0, 'C:\\TEMP') attempdirs.insert(0, '\\TEMP') elif os.name == 'mac': import macfs, MACFS try: refnum, dirid = macfs.FindFolder(MACFS.kOnSystemDisk, MACFS.kTemporaryFolderType, 1) dirname = macfs.FSSpec((refnum, dirid, '')).as_pathname() attempdirs.insert(0, dirname) except macfs.error: pass elif os.name == 'riscos': scrapdir = os.getenv('Wimp$ScrapDir') if scrapdir: attempdirs.insert(0, scrapdir) for envname in 'TMPDIR', 'TEMP', 'TMP': if os.environ.has_key(envname): attempdirs.insert(0, os.environ[envname]) testfile = gettempprefix() + 'test' for dir in attempdirs: try: filename = os.path.join(dir, testfile) fd = os.open(filename, os.O_RDWR | os.O_CREAT | os.O_EXCL, 0700) fp = os.fdopen(fd, 'w') fp.write('blat') fp.close() os.unlink(filename) del fp, fd return dir except IOError: pass msg = "Can't find a usable temporary directory amongst " + `attempdirs` raise IOError, msg

4 3

Re: [Python-Dev] PEP 294: Type Names in the types Module
by Michael Hudson 25 Jun '02

25 Jun '02

Oren Tirosh <oren-py-l(a)hishome.net> writes: > Abstract > > This PEP proposes that symbols matching the type name should be > added to the types module for all basic Python types in the types > module: > > types.IntegerType -> types.int > types.FunctionType -> types.function > types.TracebackType -> types.traceback > ... > > The long capitalized names currently in the types module will be > deprecated. Um, can I be a little confused? If you are writing code that you know will be run in 2.2 and later, you write isinstance(obj, int) If you want to support 2.1 and so on, you write isinstance(obj, types.IntType) What would writing isinstance(obj, types.int) ever gain you except restricting execution to 2.3+? I mean, I don't have any real opinion *against* this pep, I just don't really see why anyone would care... Cheers, M. -- it's not that perl programmers are idiots, it's that the language rewards idiotic behavior in a way that no other language or tool has ever done -- Erik Naggum, comp.lang.lisp

7 10

Please give this patch for building bsddb a try
by Skip Montanaro 25 Jun '02

25 Jun '02

If you build the bsddb module on a Unix-like system (that is, you use configure and setup.py to build the interpreter and it attempts to build the bsddb module), please give the new patch attached to http://python.org/sf/553108 a try. Ignore the subject of the patch. I just tacked my patch onto this item and assigned it to myself. If/when the issue is settled I'll track down and close other patches and bug reports related to building the bsddb module. Briefly, it attempts the following: 1. Makes it inconvenient (though certainly not impossible) to build/link with version 1 of the Berkeley DB library by commenting out the relevant part of the db_try_this dictionary in setup.py. 2. Links the search for a DB library and corresponding include files so you don't find a version 2 include file and a version 3 library (for example). 3. Attempts to do the same for the dbm module when it decides to link with the Berkeley DB library for compatibility (this is stuff under "development" and will almost certainly require further changes). (You can ignore the debug print I forgot to remove before creating the patch. ;-) I asked on c.l.py about where people have the Berkeley DB stuff installed so I could tune the locations listed in db_try_this, but the thread almost immediately went off into the weeds arguing about berkdb license issues. I therefore humbly request your more rational input on this topic. If you have a Unix-ish system and Berkeley DB is installed somewhere not listed in the db_try_this dictionary in setup.py, please let me know. Thx, Skip

11 52

strptime recapped
by Brett Cannon 25 Jun '02

25 Jun '02

I have written the callout to strptime.strptime (strptime is SF patch #474274) as Guido asked. Since that was the current hold-up and the thread has gone dormant, I figured I should summarize the discussion up to this point. 1) what is the need?: The question was raised why this was done. The answer was that since time is just a wrapper around time.h, strptime was not guaranteed since it is not a part of ANSI C. Some ANSI C libraries include it, though (like glibc), because it is so useful. Unfortunately Windows and OS X do not have it. Having it in Python means it is completely portable and no longer reliant on the ANSI C library being kind enough to provide it. 2) strftime dependence: Some people worried about the dependence upon strftime for calculating some info. But since strftime is guaranteed to be there by Python (since it is a part of ANSI C), the dependence is not an issue. 3) locale info for dates: Skip and Guido pointed out that calendar.py now generates the names of the weekdays and months on the fly similar to my solution. So I did go ahead and use it. But Skip pointed out that perhaps we should centralize any code that calculates locale info for dates (calendar.py's names and my code for figuring out format for date/time). I had suggested adding it to the locale module and Guido responded that Martin had to ok that. Martin hasn't responded to that idea. 4) location of strptime: Skip asked why Guido was having me write the callout patch to timemodule.c. He wondered why Lib/time.py wasn't just created holding my code and then renaming timemodule.c to _timemodule.c and importing it at the end of time.py. No response has been given thus far for that. I also suggested a possible time2 where things like strptime, my helper fxns (calculate the Julian date from the Gregorian date, etc.), and things such as naivetime could be kept. That would allow time to stay as a straight wrapper to time.h while all bonus code gets relegated to time2. Guido said it might be a good idea but would have to wait until he got back from vacation. That pretty much sums up everything to this point; hope I got it right and didn't miss anything. -Brett C.

6 13

New subscriber
by Lalo Martins 25 Jun '02

25 Jun '02

On Mon, Jun 24, 2002 at 10:53:15AM -0400, python-dev-request(a)python.org wrote: > If you are a new subscriber, please take the time to introduce yourself > briefly in your first post. Hmm, ok. My name is Fernando Martins, known as Lalo. I'm currently 27 and I live in Brazil. I've been a Python advocate since Bruce Perens introduced me to it in, what, '96. I've been working professionally with Zope - ranging from site building to training, from infrastructure hacking to consulting - since mid-99, when I selected it from a range of options due to the fact that it was in Python. In the course of zope training and consulting, I take every opportunity to give python courses and talks. I never subscribed to python-dev before because I was very involved in the Zope community and preferred to keep my mind out of lower-level stuff, but now I find there are lots of interesting things going on and I'd prefer to be a part of it. ;-) (Also, Zope is very cool but the web marketing can get tiresome - if I can find a way, I'm planning to retire from it at least in part and spend more time doing plain Python.) []s, |alo +---- -- It doesn't bother me that people say things like "you'll never get anywhere with this attitude". In a few decades, it will make a good paragraph in my biography. You know, for a laugh. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ Python Foundry Guide http://www.sf.net/foundry/python-foundry/

1 0

Added SSL through HTTP proxies support to httplib.py
by Mihai Ibanescu 24 Jun '02

24 Jun '02

Hello, Can somebody please verify if the following patch makes enough sense to be accepted? http://sourceforge.net/tracker/index.php?func=detail&aid=515003&group_id=54… Thanks, Misa

1 0

Behavior of matching backreferences
by Gustavo Niemeyer 24 Jun '02

24 Jun '02

Hi everyone! I was studying the sre module, when I came up with the following regular expression: re.compile("^(?P<a>a)?(?P=a)$").match("ebc").groups() The (?P=a) matches with whatever was matched by the "a" group. If "a" is optional and doesn't match, it seems to make sense that (?P=a) becomes optional as well, instead of failing. Otherwise the regular expression above will allways fail if the first group fails, even being optional. One could argue that to make it a valid regular expression, it should become "^(?P<a>a)?(?P=a)?". But that's a different regular expression, since it would match "a", while the regular expression above would match "aa" or "", but not "a". This kind of pattern is useful, for example, to match a string which could be optionally surrounded by quotes, like shell variables. Here's an example of such pattern: r"^(?P<a>')?((?:\\'|[^'])*)(?P=a)$". This pattern matches "'a'", "\'a", "a\'a", "'a\'a'" and all such variants, but not "'a", "a'", or "a'a". I've submitted a patch to make this work to http://python.org/sf/571976 -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ]

4 10

Playstation 2 and GameCube ports
by Jason L. Asbahr 24 Jun '02

24 Jun '02

Pythonistas, During the past year, I did some work that involved porting Python 2.1 to the Sony Playstation 2 (professional developer system) and Nintendo GameCube platforms. Since it involved little code change to Python itself, I was wondering if there is interest in merging these changes into the main trunk? This would first involved bringing those ports up to speed with the current CVS trunk. But before that, a PEP? Cheers, Jason ______________________________________________________________________ Jason Asbahr jason(a)asbahr.com

5 6

Weekly Python Bug/Patch Summary
by Skip Montanaro 23 Jun '02

23 Jun '02

Bug/Patch Summary ----------------- 254 open / 2603 total bugs (-3) 128 open / 1565 total patches (no change) New Bugs -------- ConfigParser code cleanup (2002-04-17) http://python.org/sf/545096 odd index entries (2002-06-17) http://python.org/sf/570003 "python -u" not binary on cygwin (2002-06-17) http://python.org/sf/570044 Broken pre.subn() (and pre.sub()) (2002-06-17) http://python.org/sf/570057 inspect.getmodule symlink-related failur (2002-06-17) http://python.org/sf/570300 .PYO files not imported unless -OO used (2002-06-18) http://python.org/sf/570640 bdist_rpm and the changelog option (2002-06-18) http://python.org/sf/570655 CGIHTTPServer flushes read-only file. (2002-06-18) http://python.org/sf/570678 glob() fails for network drive in cgi (2002-06-19) http://python.org/sf/571167 imaplib fetch is broken (2002-06-19) http://python.org/sf/571334 Mixing framework and static Pythons (2002-06-19) http://python.org/sf/571343 Numeric Literal Anomoly (2002-06-19) http://python.org/sf/571382 test_import crashes/hangs for MacPython (2002-06-20) http://python.org/sf/571845 Segmentation fault in Python 2.3 (2002-06-20) http://python.org/sf/571885 python-mode IM parses code in docstrings (2002-06-21) http://python.org/sf/572341 Memory leak in object comparison (2002-06-22) http://python.org/sf/572567 New Patches ----------- Remove support for Win16 (2002-06-16) http://python.org/sf/569753 Fix bug in encodings.search_function (2002-06-20) http://python.org/sf/571603 Changes (?P=) with optional backref (2002-06-20) http://python.org/sf/571976 AUTH method LOGIN for smtplib (2002-06-21) http://python.org/sf/572031 Remove import string in Tools/ directory (2002-06-21) http://python.org/sf/572113 opt. timeouts for Queue.put() and .get() (2002-06-22) http://python.org/sf/572628 Closed Bugs ----------- Incorporate timeoutsocket.py into core (2001-08-30) http://python.org/sf/457114 ext call doco warts (2001-12-14) http://python.org/sf/493243 It's the future for generators (2001-12-21) http://python.org/sf/495978 PyModule_AddObject doesn't set exception (2002-02-27) http://python.org/sf/523473 Incomplete list of escape sequences (2002-03-06) http://python.org/sf/526390 range() description: rewording suggested (2002-03-11) http://python.org/sf/528748 Popen3 might cause dead lock (2002-03-16) http://python.org/sf/530637 6.9 The raise statement is confusing (2002-03-20) http://python.org/sf/532467 cut-o/paste-o in Marshalling doc: 2.2.1 (2002-03-22) http://python.org/sf/533735 mimify.mime_decode_header only latin1 (2002-05-03) http://python.org/sf/551912 [RefMan] Special status of "as" (2002-05-07) http://python.org/sf/553262 Expat improperly described in setup.py (2002-05-15) http://python.org/sf/556370 \verbatiminput and name duplication (2002-05-20) http://python.org/sf/558279 Missing operator docs (2002-06-02) http://python.org/sf/563530 PyUnicode_Find() returns wrong results (2002-06-09) http://python.org/sf/566631 __slots__ attribute and private variable (2002-06-14) http://python.org/sf/569257 Closed Patches -------------- Pure python version of calendar.weekday (2001-11-20) http://python.org/sf/483864 Janitoring in ConfigParser (2002-04-17) http://python.org/sf/545096 add support for HtmlHelp output (2002-05-06) http://python.org/sf/552835 texi2html.py - add support for HTML Help (2002-05-06) http://python.org/sf/552837 OSX build -- make python.app (2002-05-18) http://python.org/sf/557719 unicode in sys.path (2002-06-10) http://python.org/sf/566999

1 0

Re: [Python-Dev] PEP 292, Simpler String Substitutions
by Steve Holden 23 Jun '02

23 Jun '02

"Barry A. Warsaw" <barry(a)zope.com> wrote ... > > I'm so behind on my email, that the anticipated flamefest will surely > die down before I get around to reading it. Yet still, here is a new > PEP. :) > [flamefest-fodder] Seems to me that the volume of comment might imply that string formatting isn't the one obvious best way to do it. Now the flamefest has died down somewhat, the only thing I can see PEP 292 justifying is better documentation for the string-formatting operator. "But then I'm an $pytpe".sub({"ptype": "old fogey"}) regards Steve ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ -----------------------------------------------------------------------

2 1

*Simpler* string substitutions
by Paul Prescod 23 Jun '02

23 Jun '02

We will never come to a solution unless we agree on what, if any, the problem is. Here is my sense of the "interpolation" problem (based entirely on the code I see): * 95% of all scripts (or modules) need to do string interpolation * 5% of all scripts want to be explicit about the types * 10% of all scripts want to submit a dictionary rather than the current namespace * 5% of all scripts want to do printf-style formatting tricks Which means that if we do the math in a simplistic way, 20% modules/scripts need these complicated features but the other 75% pay for these features that they are not using. They pay through having to use "% locals()" (which uses two advanced features of Python, operator overloading and the local namespace). They pay through counting the lengths of their %-tuples (in my case, usually miscounting). They pay through adding (or forgetting to add) the format specifier after "%(...)". They pay through having harder to read strings where they have to go back and forth to figure out what various positional variables mean. They through having to remember the special case for singletons -- except for singleton tuples! Of course the syntax is flexible: you get to choose HOW you pay (shifting from positional to name) and thus reduce some costs while you incur others, but you can't choose simply NOT to pay, as you can in every other scripting language I know. And remember that Python is a language that *encourages* readability. But this kind of code is common: * exception.append('\n<br>%s%s =\n%s' % (indent, name, value)) whereas it could be just: * exception.append('\n<br>${ident}${name} =\n${value}') Which is shorter, uses fewer concepts, and keeps variables close to where they are used. We could argue that the programmer here made the wrong choice (versus using % locals()) but the point is that Python itself favoured the wrong choice by making the wrong choice shorter and simpler. Usually Python favours the right choice. The tax is small but it is collected on almost every script, almost every beginner and almost every programmer almost every day. So it adds up. If we put this new feature in a module: (whether "text", "re", "string"), then we are just divising another way to make people pay. At that point it becomes a negative feature, because it will clutter up the standard library without getting use.As long as you are agreeing to pay some tax, "%" is a smaller tax (at least at first) because it does not require you to interrupt your workflow to insert an import statement. In my mind, this feature is only worth adding if we agree that it is now the standard string interpolation feature and "%" becomes a quaint historical feature -- a bad experiment in operator overloading gone wrong. "%" could be renamed "text.printf" and would actually become more familiar to its core constituency and less of a syntactic abberation. "interp" could be a built-in and thus similarly simple syntactically. But I am against adding "$" if half of Python programmers are going to use that and half are going to use %. $ needs to be a replacement. There should be one obvious way to solve simple problems like this, not two. I am also against adding it as a useless function buried in a module that nobody will bother to import. Paul Prescod

13 20

Re: [Python-Dev] strptime recapped
by Michael Hudson 23 Jun '02

23 Jun '02

Brett Cannon <bac(a)OCF.Berkeley.EDU> writes: > It's quite fine with me. I want to see this done right just like > everyone else who cares about Python's development. Personally, I > am just ecstatic that I am getting to help out in some way. I feel > more like a giddy little kid who is helping out some grown-ups with > some important project than a recent college graduate. =) Feel like being release manager for 2.2.2? <wink> Cheers, M. -- at any rate, I'm satisfied that not only do they know which end of the pointy thing to hold, but where to poke it for maximum effect. -- Eric The Read, asr, on google.com

3 2

Re: [Python-Dev] *Simpler* string substitutions
by David Abrahams 22 Jun '02

22 Jun '02

----- Original Message ----- From: "Alex Martelli" <aleax(a)aleax.it> > > I was thinking of the use of template specialization to describe the > > relationship of a type to a library, e.g. specialization of > > std::iterator_traits<libA::some_class> by libB, which makes > > libA::some_class available for use as an iterator with the standard library > > (assuming it has some appropriate interface). > > That requires proper design for extensibility in advance -- the standard > library must have had the forethought to define, and use everywhere > appropriate, std::iterator_traits, AND libA must expose classes that > can be plugged into that "slot". Very true. > As I tried indicating, if you're willing to require design-in-advance for > such issues, PEP 246 (together with Python's general mechanisms) > already offer what you need. Super! +1 > Yes, factories and such creational patterns ARE a better long-term > answer, BUT there's no denying that Ruby's ability to patch things > up with duct tape (while having its own costs, of course!-) can > be a short-term lifesaver. "If God had WANTED us to get things > right the first time he wouldn't have created duct tape", after all:-). In Alaska, where my wife grew up, they call it "100-mile-an-hour tape" -- good for any use up to 100 mph. [apparently not for ducts, though, even if they're sitting still :(] > > > Only Haskell's typeclass, AFAIK, has (among widely used languages and > > > objectmodels) a smooth way to allow noninvasive 3rd party post-facto > > > adaptation (and another couple of small gems too), but I guess it has an > > > easier life because it's compile-time rather than runtime. > > > > IIUC the same kind of thing can be implemented in C++ templates, if you > > know where to look. There's been a lot of discussion of how to build > > variant types lately. > > I don't think you can do it without some degree of design forethought, > but admittedly I'm starting to get very slightly rusty (haven't designed a > dazzling new C++ template in almost six months, after all:-). Well, I have to admit that I don't have the time to say anything backed up by any research at this point ...I'm currently stuck in a Microsoft gravity well trying to survive the descent... but thanks as always for your educational and broad perspective! -Dave

1 0

Re: [Python-Dev] *Simpler* string substitutions
by David Abrahams 22 Jun '02

22 Jun '02

From: "Alex Martelli" <aleax(a)aleax.it> > On Friday 21 June 2002 07:52 pm, David Abrahams wrote: > ... > > Such a strong endorsement from you made me go take a cursory look; I think > > I'd be -1 on this in its current form. It seems like an intrusive mechanism > > in that it forces the adapter or the adaptee to know how to do the job. > > That's point (e) in the Requirements of the PEP: > > """ > e) When the context knows about the object and the protocol and > knows how to adapt the object so that the required protocol is > satisfied. This could use an adapter registry or similar > method. > """ Oh, sorry I missed that. > However, do notice that even in its present form it's WAY less invasive > than C++'s dynamic_cast<>, which ONLY allows the _adaptee_ to solve things -- > and in a very inflexible way, too. With dynamic_cast there's no way the > "protocol" can noninvasively "adopt" existing objects, nor can an object > have any say about it (e.g. how to disambiguate between multiple > inheritance cases). QueryInterface does let the adaptee have an explicit > say, but still, the adaptee is the only party consulted. I wasn't trying to spark a comparison with C++ here, nor was I talking about runtime-dispatched stuff in C++. I'm not even sure I would call dynamic_cast<> a candidate for this kind of job, at least, not by iteself. I was thinking of the use of template specialization to describe the relationship of a type to a library, e.g. specialization of std::iterator_traits<libA::some_class> by libB, which makes libA::some_class available for use as an iterator with the standard library (assuming it has some appropriate interface). > Only Haskell's typeclass, AFAIK, has (among widely used languages and > objectmodels) a smooth way to allow noninvasive 3rd party post-facto > adaptation (and another couple of small gems too), but I guess it has an > easier life because it's compile-time rather than runtime. IIUC the same kind of thing can be implemented in C++ templates, if you know where to look. There's been a lot of discussion of how to build variant types lately. -Dave

2 1

Re: [Python-Dev] *Simpler* string substitutions
by Guido van Rossum 22 Jun '02

22 Jun '02

> But his biggest remaining "general peeve" struck me hard the other > day, exactly because that's not something he "heard", but an > observation he came up with all by himself, by reasonably unbiased > examination of "Python as she's spoken". "I wouldn't mind Python so > much" (I'm paraphrasing, but that IS the kind of grudging-compliment > understatement he did use:-) "except that there's always so MANY > deuced ways to do everything -- can't they just pick one and STICK > with it?!". In the widespread subtext of most Python discourse this > might sound like irony, but in his case, it was just an issue of > fact (compared, remember, with SMALL languages such as Limbo -- > bloated ones such as, e.g., C++, are totally *outside* his purvey > and experience) -- a bewildering array of possible variations. > Surely inevitable when viewed diachronically (==as an evolution over > time), but his view, like that of anybody who comes to Python anew > today, is synchronic (==a snapshot at one moment). > > I don't think there's anything we can do to AVOID this phenomenon, > of course, but right now I'm probably over-sensitized to the > "transition costs" of introducing "yet one more way to do it" by > this recent episode. So, it appears to me that REDUCING the > occurrence of such perceptions is important. AFAIK Limbo has a very small user base (and its key designer is much more arrogant than your average BDFL even :-). It's much easier to withstand the pressure to add features in that case. And lately, most new features have been better ways to do things you could already do, but only clumsily. That would add to his impressions. Plus, inevitably, that not everybody at Strakt uses the same coding style. I understand the sentiment, but users are like this: they all want you to stop adding features except the one thing they absolutely need. (Myhrvold) --Guido van Rossum (home page: http://www.python.org/~guido/)

6 11

Re: [Python-Dev] *Simpler* string substitutions
by Matthias Urlichs 22 Jun '02

22 Jun '02

Aahz: > On Thu, Jun 20, 2002, Gustavo Niemeyer wrote: > > > > "Serving HTTP on", sa[0], "port", sa[1], "..." > _("Serving HTTP on ${addr} port ${port}").sub(addr=sa[0], port=sa[1]) _("Serving HTTP on ${sa[0]} port ${sa[1]}").sub() or some equivalent syntax. Note that the second way, above, has a distinct disadvantage for the translating person. He or she would probably know what "addr" stands for, but what is a "sa[0]" ??? > This is where current string handling comes up short. What's the > correct way to internationalize this string? Currently, you do the same thing but use % and a dictionary. As I said, equivalent syntax. > What if the person > handling I18N isn't a Python programmer? > The person gets frustrated and bitches at the programmer until the programmer fixes the code... -- Matthias Urlichs

1 0

Re: [Python-Dev] *Simpler* string substitutions
by Matthias Urlichs 22 Jun '02

22 Jun '02

Guido: > publishers often turn 'foo' into `foo' It gets worse. The opposite of ` isn't ' -- it's ´. Besides, these are apostrophes and not quotes. _Real_ symmetric quotes are " " or « » or " or ' ' or ' or ..., but you can't use any of these with just ASCII. Apple's MPW Shell language played with some of these. Anyway, I agree that real languages use ${} or $WORD and that formatting is best done with ${NAME:format}. Personally, the "${foo}".sub(foo="bar") syntax (using keyword arguments) looks good and works reasonably well for i18n. A possible simplification would be to use the local+global variables if no arguments are given. > def f(x, y): > return e"The sum of $x and $y is $(x+y)" How would that work with i18n? Proposal: The compiler should translate _"foo is ${foo}" to _("foo is ${foo}") and e"foo is ${foo}" to "foo is ${foo}".sub() >That looks OK to me, especially if it can be combined with u and r to > create unicode and raw strings. > Exactly. > PEP 292 is an attempt to do this *without* involving the parser: > > def f(x, y): > return "The sum of $x and $y is $(x+y)".sub() > > Downsides are that it invites using non-literals as formats, with all > the security aspects, and that its parsing happens at run-time (no big > deal IMO). > You can't do it any other way if you want to use i18nalized strings and formats. Note that some sentences cannot be internationalized without rearranging some parameters... -- Matthias Urlichs

1 0

RE: [Python-Dev] Re: PEP 292, Simpler String Substitutions
by Joe Mason 22 Jun '02

22 Jun '02

Tim wrote: > [Guido, quotes Christian] > >> The following statements are ordered by increasing hate. > >> 1 - I do hate the idea of introducing a "$" sign at all. > >> 2 - giving "$" special meaning in strings via a module > >> 3 - doing it as a builtin function > >> 4 - allowing it to address local/global variables > > [and adds] > > Doesn't 4 contradict your +1 on allvars()? > > Since Christian's reply only increased the apparent contradiction, allow me > to channel: they are ordered by increasing hate, but starting at the > bottom. s/increasing/decreasing/ in his original, or s/hate/love/, and you > can continue to read it in the top-down Dutch way <wink>. If you'll allow me to counter-channel: Christian hates giving this special syntax form access to local/global variables, since it's a security risk that's not apparent unless you know what you're looking for. He prefers to use allvars() to achieve the same end, since it's explicit. He's not opposed to variable access in general. Write-only variables don't tend to find much use. Joe

1 0

Behavior of buffer()
by Raymond Hettinger 22 Jun '02

22 Jun '02

I would like to solicit py-dev's thoughts on the best way to resolve a bug, www.python.org/sf/546434 . The root problem is that mybuf[:] returns a buffer type and mybuf[2:4] returns a string type. A similar issue exists for buffer repetition. One way to go is to have the slices always return a string. If code currently relies on the type of a buffer slice, it is more likely to be relying on it being a string as in: print mybuf[:4]. This is an intuitive guess because I can't find empirical evidence. Another reason to choose a string return type is that buffer() appears to have been designed to be as stringlike as possible so that it can be easily substituted in code originally designed for strings. The other way to go is to return a buffer object everytime. Slices usually, but not always (see subclasses of list), return the same type that was being sliced. If we choose this route, another issue remains -- mybuf[:] returns self instead of a new buffer. I think that behavior is also a bug and should be changed to be consistent with the Python idiom where: b = a[:] assert id(a) != id(b) Incidental to the above, GvR had a thought that slice repetition ought to always return an error. Though I don't see any use cases for buffer repetition, bufferobjects do implement all other sequence behaviors and I think it would be weird to nullify the sq_repeat slot. I appreciate your thoughts on the best way to proceed. fixing-bugs-is-easier-than-deciding-appropriate-behavior-ly yours, 'regnitteh dnomyar'[::-1]

1 0