Mailman 3 June 2002 - Python-Dev

RE: [Python-Dev] Those import related syntax errors again...
by Samuele Pedroni 23 Feb '05

23 Feb '05

Hi. [Mark Hammond] > The point isn't about my suffering as such. The point is more that > python-dev owns a tiny amount of the code out there, and I don't believe we > should put Python's users through this. > > Sure - I would be happy to "upgrade" all the win32all code, no problem. I > am also happy to live in the bleeding edge and take some pain that will > cause. > > The issue is simply the user base, and giving Python a reputation of not > being able to painlessly upgrade even dot revisions. I agree with all this. [As I imagined explicit syntax did not catch up and would require lot of discussions.] [GvR] > > Another way is to use special rules > > (similar to those for class defs), e.g. having > > > > <frag> > > y=3 > > def f(): > > exec "y=2" > > def g(): > > return y > > return g() > > > > print f() > > </frag> > > > > # print 3. > > > > Is that confusing for users? maybe they will more naturally expect 2 > > as outcome (given nested scopes). > > This seems the best compromise to me. It will lead to the least > broken code, because this is the behavior that we had before nested > scopes! It is also quite easy to implement given the current > implementation, I believe. > > Maybe we could introduce a warning rather than an error for this > situation though, because even if this behavior is clearly documented, > it will still be confusing to some, so it is better if we outlaw it in > some future version. > Yes this can be easy to implement but more confusing situations can arise: <frag> y=3 def f(): y=9 exec "y=2" def g(): return y return y,g() print f() </frag> What should this print? the situation leads not to a canonical solution as class def scopes. or <frag> def f(): from foo import * def g(): return y return g() print f() </frag> [Mark Hammond] > > This probably won't be a very popular suggestion, but how about pulling > > nested scopes (I assume they are at the root of the problem) > > until this can be solved cleanly? > > Agreed. While I think nested scopes are kinda cool, I have lived without > them, and really without missing them, for years. At the moment the cure > appears worse then the symptoms in at least a few cases. If nothing else, > it compromises the elegant simplicity of Python that drew me here in the > first place! > > Assuming that people really _do_ want this feature, IMO the bar should be > raised so there are _zero_ backward compatibility issues. I don't say anything about pulling nested scopes (I don't think my opinion can change things in this respect) but I should insist that without explicit syntax IMO raising the bar has a too high impl cost (both performance and complexity) or creates confusion. [Andrew Kuchling] > >Assuming that people really _do_ want this feature, IMO the bar should be > >raised so there are _zero_ backward compatibility issues. > > Even at the cost of additional implementation complexity? At the cost > of having to learn "scopes are nested, unless you do these two things > in which case they're not"? > > Let's not waffle. If nested scopes are worth doing, they're worth > breaking code. Either leave exec and from..import illegal, or back > out nested scopes, or think of some better solution, but let's not > introduce complicated backward compatibility hacks. IMO breaking code would be ok if we issue warnings today and implement nested scopes issuing errors tomorrow. But this is simply a statement about principles and raised impression. IMO import * in an inner scope should end up being an error, not sure about 'exec's. We will need a final BDFL statement. regards, Samuele Pedroni.

18 78

PEP 293, Codec Error Handling Callbacks
by Walter Dörwald 14 Aug '02

14 Aug '02

Here's another new PEP. Bye, Walter Dörwald ---------------------------------------------------------------------- PEP: 293 Title: Codec Error Handling Callbacks Version: $Revision: 1.1 $ Last-Modified: $Date: 2002/06/19 03:22:11 $ Author: Walter Dörwald Status: Draft Type: Standards Track Created: 18-Jun-2002 Python-Version: 2.3 Post-History: Abstract This PEP aims at extending Python's fixed codec error handling schemes with a more flexible callback based approach. Python currently uses a fixed error handling for codec error handlers. This PEP describes a mechanism which allows Python to use function callbacks as error handlers. With these more flexible error handlers it is possible to add new functionality to existing codecs by e.g. providing fallback solutions or different encodings for cases where the standard codec mapping does not apply. Specification Currently the set of codec error handling algorithms is fixed to either "strict", "replace" or "ignore" and the semantics of these algorithms is implemented separately for each codec. The proposed patch will make the set of error handling algorithms extensible through a codec error handler registry which maps handler names to handler functions. This registry consists of the following two C functions: int PyCodec_RegisterError(const char *name, PyObject *error) PyObject *PyCodec_LookupError(const char *name) and their Python counterparts codecs.register_error(name, error) codecs.lookup_error(name) PyCodec_LookupError raises a LookupError if no callback function has been registered under this name. Similar to the encoding name registry there is no way of unregistering callback functions or iterating through the available functions. The callback functions will be used in the following way by the codecs: when the codec encounters an encoding/decoding error, the callback function is looked up by name, the information about the error is stored in an exception object and the callback is called with this object. The callback returns information about how to proceed (or raises an exception). For encoding, the exception object will look like this: class UnicodeEncodeError(UnicodeError): def __init__(self, encoding, object, start, end, reason): UnicodeError.__init__(self, "encoding '%s' can't encode characters " + "in positions %d-%d: %s" % (encoding, start, end-1, reason)) self.encoding = encoding self.object = object self.start = start self.end = end self.reason = reason This type will be implemented in C with the appropriate setter and getter methods for the attributes, which have the following meaning: * encoding: The name of the encoding; * object: The original unicode object for which encode() has been called; * start: The position of the first unencodable character; * end: (The position of the last unencodable character)+1 (or the length of object, if all characters from start to the end of object are unencodable); * reason: The reason why object[start:end] couldn't be encoded. If object has consecutive unencodable characters, the encoder should collect those characters for one call to the callback if those characters can't be encoded for the same reason. The encoder is not required to implement this behaviour but may call the callback for every single character, but it is strongly suggested that the collecting method is implemented. The callback must not modify the exception object. If the callback does not raise an exception (either the one passed in, or a different one), it must return a tuple: (replacement, newpos) replacement is a unicode object that the encoder will encode and emit instead of the unencodable object[start:end] part, newpos specifies a new position within object, where (after encoding the replacement) the encoder will continue encoding. If the replacement string itself contains an unencodable character the encoder raises the exception object (but may set a different reason string before raising). Should further encoding errors occur, the encoder is allowed to reuse the exception object for the next call to the callback. Furthermore the encoder is allowed to cache the result of codecs.lookup_error. If the callback does not know how to handle the exception, it must raise a TypeError. Decoding works similar to encoding with the following differences: The exception class is named UnicodeDecodeError and the attribute object is the original 8bit string that the decoder is currently decoding. The decoder will call the callback with those bytes that constitute one undecodable sequence, even if there is more than one undecodable sequence that is undecodable for the same reason directly after the first one. E.g. for the "unicode-escape" encoding, when decoding the illegal string "\\u00\\u01x", the callback will be called twice (once for "\\u00" and once for "\\u01"). This is done to be able to generate the correct number of replacement characters. The replacement returned from the callback is a unicode object that will be emitted by the decoder as-is without further processing instead of the undecodable object[start:end] part. There is a third API that uses the old strict/ignore/replace error handling scheme: PyUnicode_TranslateCharmap/unicode.translate The proposed patch will enhance PyUnicode_TranslateCharmap, so that it also supports the callback registry. This has the additional side effect that PyUnicode_TranslateCharmap will support multi-character replacement strings (see SF feature request #403100 [1]). For PyUnicode_TranslateCharmap the exception class will be named UnicodeTranslateError. PyUnicode_TranslateCharmap will collect all consecutive untranslatable characters (i.e. those that map to None) and call the callback with them. The replacement returned from the callback is a unicode object that will be put in the translated result as-is, without further processing. All encoders and decoders are allowed to implement the callback functionality themselves, if they recognize the callback name (i.e. if it is a system callback like "strict", "replace" and "ignore"). The proposed patch will add two additional system callback names: "backslashreplace" and "xmlcharrefreplace", which can be used for encoding and translating and which will also be implemented in-place for all encoders and PyUnicode_TranslateCharmap. The Python equivalent of these five callbacks will look like this: def strict(exc): raise exc def ignore(exc): if isinstance(exc, UnicodeError): return (u"", exc.end) else: raise TypeError("can't handle %s" % exc.__name__) def replace(exc): if isinstance(exc, UnicodeEncodeError): return ((exc.end-exc.start)*u"?", exc.end) elif isinstance(exc, UnicodeDecodeError): return (u"\\ufffd", exc.end) elif isinstance(exc, UnicodeTranslateError): return ((exc.end-exc.start)*u"\\ufffd", exc.end) else: raise TypeError("can't handle %s" % exc.__name__) def backslashreplace(exc): if isinstance(exc, (UnicodeEncodeError, UnicodeTranslateError)): s = u"" for c in exc.object[exc.start:exc.end]: if ord(c)<=0xff: s += u"\\x%02x" % ord(c) elif ord(c)<=0xffff: s += u"\\u%04x" % ord(c) else: s += u"\\U%08x" % ord(c) return (s, exc.end) else: raise TypeError("can't handle %s" % exc.__name__) def xmlcharrefreplace(exc): if isinstance(exc, (UnicodeEncodeError, UnicodeTranslateError)): s = u"" for c in exc.object[exc.start:exc.end]: s += u"&#%d;" % ord(c) return (s, exc.end) else: raise TypeError("can't handle %s" % exc.__name__) These five callback handlers will also be accessible to Python as codecs.strict_error, codecs.ignore_error, codecs.replace_error, codecs.backslashreplace_error and codecs.xmlcharrefreplace_error. Rationale Most legacy encoding do not support the full range of Unicode characters. For these cases many high level protocols support a way of escaping a Unicode character (e.g. Python itself supports the \x, \u and \U convention, XML supports character references via &#xxx; etc.). When implementing such an encoding algorithm, a problem with the current implementation of the encode method of Unicode objects becomes apparent: For determining which characters are unencodable by a certain encoding, every single character has to be tried, because encode does not provide any information about the location of the error(s), so # (1) us = u"xxx" s = us.encode(encoding) has to be replaced by # (2) us = u"xxx" v = [] for c in us: try: v.append(c.encode(encoding)) except UnicodeError: v.append("&#%d;" % ord(c)) s = "".join(v) This slows down encoding dramatically as now the loop through the string is done in Python code and no longer in C code. Furthermore this solution poses problems with stateful encodings. For example UTF-16 uses a Byte Order Mark at the start of the encoded byte string to specify the byte order. Using (2) with UTF-16, results in an 8 bit string with a BOM between every character. To work around this problem, a stream writer - which keeps state between calls to the encoding function - has to be used: # (3) us = u"xxx" import codecs, cStringIO as StringIO writer = codecs.getwriter(encoding) v = StringIO.StringIO() uv = writer(v) for c in us: try: uv.write(c) except UnicodeError: uv.write(u"&#%d;" % ord(c)) s = v.getvalue() To compare the speed of (1) and (3) the following test script has been used: # (4) import time us = u"äa"*1000000 encoding = "ascii" import codecs, cStringIO as StringIO t1 = time.time() s1 = us.encode(encoding, "replace") t2 = time.time() writer = codecs.getwriter(encoding) v = StringIO.StringIO() uv = writer(v) for c in us: try: uv.write(c) except UnicodeError: uv.write(u"?") s2 = v.getvalue() t3 = time.time() assert(s1==s2) print "1:", t2-t1 print "2:", t3-t2 print "factor:", (t3-t2)/(t2-t1) On Linux this gives the following output (with Python 2.3a0): 1: 0.274321913719 2: 51.1284689903 factor: 186.381278466 i.e. (3) is 180 times slower than (1). Codecs must be stateless, because as soon as a callback is registered it is available globally and can be called by multiple encode() calls. To be able to use stateful callbacks, the errors parameter for encode/decode/translate would have to be changed from char * to PyObject *, so that the callback could be used directly, without the need to register the callback globally. As this requires changes to lots of C prototypes, this approach was rejected. Currently all encoding/decoding functions have arguments const Py_UNICODE *p, int size or const char *p, int size to specify the unicode characters/8bit characters to be encoded/decoded. So in case of an error the codec has to create a new unicode or str object from these parameters and store it in the exception object. The callers of these encoding/decoding functions extract these parameters from str/unicode objects themselves most of the time, so it could speed up error handling if these object were passed directly. As this again requires changes to many C functions, this approach has been rejected. Implementation Notes A sample implementation is available as SourceForge patch #432401 [2]. The current version of this patch differs from the specification in the following way: * The error information is passed from the codec to the callback not as an exception object, but as a tuple, which has an additional entry state, which can be used for additional information the codec might want to pass to the callback. * There are two separate registries (one for encoding/translating and one for decoding) The class codecs.StreamReaderWriter uses the errors parameter for both reading and writing. To be more flexible this should probably be changed to two separate parameters for reading and writing. The errors parameter of PyUnicode_TranslateCharmap is not availably to Python, which makes testing of the new functionality of PyUnicode_TranslateCharmap impossible with Python scripts. The patch should add an optional argument errors to unicode.translate to expose the functionality and make testing possible. Codecs that do something different than encoding/decoding from/to unicode and want to use the new machinery can define their own exception classes and the strict handlers will automatically work with it. The other predefined error handlers are unicode specific and expect to get a Unicode(Encode|Decode|Translate)Error exception object so they won't work. Backwards Compatibility The semantics of unicode.encode with errors="replace" has changed: The old version always stored a ? character in the output string even if no character was mapped to ? in the mapping. With the proposed patch, the replacement string from the callback callback will again be looked up in the mapping dictionary. But as all supported encodings are ASCII based, and thus map ? to ?, this should not be a problem in practice. References [1] SF feature request #403100 "Multicharacter replacements in PyUnicode_TranslateCharmap" http://www.python.org/sf/403100 [2] SF patch #432401 "unicode encoding error callbacks" http://www.python.org/sf/432401 Copyright This document has been placed in the public domain. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End:

7 34

Priority queue (binary heap) python code
by Kevin O'Connor 11 Aug '02

11 Aug '02

I often find myself needing priority queues in python, and I've finally broken down and written a simple implementation. Previously I've used sorted lists (via bisect) to get the job done, but the heap code significantly improves performance. There are C based implementations, but the effort of compiling in an extension often isn't worth the effort. I'm including the code here for everyone's amusement. Any chance something like this could make it into the standard python library? It would save a lot of time for lazy people like myself. :-) Cheers, -Kevin def heappush(heap, item): pos = len(heap) heap.append(None) while pos: parentpos = (pos - 1) / 2 parent = heap[parentpos] if item <= parent: break heap[pos] = parent pos = parentpos heap[pos] = item def heappop(heap): endpos = len(heap) - 1 if endpos <= 0: return heap.pop() returnitem = heap[0] item = heap.pop() pos = 0 while 1: child2pos = (pos + 1) * 2 child1pos = child2pos - 1 if child2pos < endpos: child1 = heap[child1pos] child2 = heap[child2pos] if item >= child1 and item >= child2: break if child1 > child2: heap[pos] = child1 pos = child1pos continue heap[pos] = child2 pos = child2pos continue if child1pos < endpos: child1 = heap[child1pos] if child1 > item: heap[pos] = child1 pos = child1pos break heap[pos] = item return returnitem -- ------------------------------------------------------------------------ | Kevin O'Connor "BTW, IMHO we need a FAQ for | | kevin(a)koconnor.net 'IMHO', 'FAQ', 'BTW', etc. !" | ------------------------------------------------------------------------

16 43

test_socket failure on FreeBSD
by Andrew MacIntyre 19 Jul '02

19 Jul '02

Below is the output of test_socket with the -v option, from a CVS tree of about 1915 UTC June 18. FreeBSD 4.4, gcc 2.95.3 (-g -O3). In speaking up now, I'm making the assumption that the non-blocking socket changes should be complete, modulo bugfixes. If this is not the case, please let me know, and I'll wait for the situation to stabilise. Otherwise, is there any more info I can (attempt to) provide? I tried "print"ing the addr variable when running the test, and just get "ERROR" (sans quotes of course). I've not yet tried to build the OS/2 port with the current CVS code, so I don't yet know what the situation is there. Won't have much time to dig until the weekend... -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac(a)bullseye.apana.org.au | Snail: PO Box 370 andymac(a)pcug.org.au | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia ---------- Forwarded message ---------- Date: Wed, 19 Jun 2002 22:13:45 +1000 (EST) {...} test_socket Testing for mission critical constants. ... ok Testing getservbyname(). ... ok Testing getsockopt(). ... ok Testing hostname resolution mechanisms. ... ok Making sure getnameinfo doesn't crash the interpreter. ... ok Testing for existance of non-crucial constants. ... ok Testing reference count for getnameinfo. ... ok Testing setsockopt(). ... ok Testing getsockname(). ... ok Testing that socket module exceptions. ... ok Testing fromfd(). ... ok Testing receive in chunks over TCP. ... ok Testing recvfrom() in chunks over TCP. ... ERROR Testing large receive over TCP. ... ok Testing large recvfrom() over TCP. ... ERROR Testing sendall() with a 2048 byte string over TCP. ... ok Testing shutdown(). ... ok Testing recvfrom() over UDP. ... ok Testing sendto() and Recv() over UDP. ... ok Testing non-blocking accept. ... FAIL Testing non-blocking connect. ... ok Testing non-blocking recv. ... FAIL Testing whether set blocking works. ... ok Performing file readline test. ... ok Performing small file read test. ... ok Performing unbuffered file read test. ... ok ====================================================================== ERROR: Testing recvfrom() in chunks over TCP. ---------------------------------------------------------------------- Traceback (most recent call last): File "./Lib/test/test_socket.py", line 359, in testOverFlowRecvFrom hostname, port = addr TypeError: unpack non-sequence ====================================================================== ERROR: Testing large recvfrom() over TCP. ---------------------------------------------------------------------- Traceback (most recent call last): File "./Lib/test/test_socket.py", line 347, in testRecvFrom hostname, port = addr TypeError: unpack non-sequence ====================================================================== FAIL: Testing non-blocking accept. ---------------------------------------------------------------------- Traceback (most recent call last): File "./Lib/test/test_socket.py", line 451, in testAccept self.fail("Error trying to do non-blocking accept.") File "/home/andymac/cvs/python/python-cvs/Lib/unittest.py", line 254, in fail raise self.failureException, msg AssertionError: Error trying to do non-blocking accept. ====================================================================== FAIL: Testing non-blocking recv. ---------------------------------------------------------------------- Traceback (most recent call last): File "./Lib/test/test_socket.py", line 478, in testRecv self.fail("Error trying to do non-blocking recv.") File "/home/andymac/cvs/python/python-cvs/Lib/unittest.py", line 254, in fail raise self.failureException, msg AssertionError: Error trying to do non-blocking recv. ---------------------------------------------------------------------- Ran 26 tests in 0.330s FAILED (failures=2, errors=2) test test_socket failed -- errors occurred; run in verbose mode for details 1 test failed: test_socket

2 5

Fw: Behavior of buffer()
by Raymond Hettinger 18 Jul '02

18 Jul '02

GvR thought you guys might have some ideas on this one for me. If I don't get any replies, I may have to rely on my own instincts and judgment and no one knows what follies might ensue ;) Raymond Hettinger ----- Original Message ----- From: "Raymond Hettinger" <python(a)rcn.com> To: <python-dev(a)python.org> Sent: Friday, June 21, 2002 1:16 PM Subject: Behavior of buffer() > I would like to solicit py-dev's thoughts on the best way to resolve a bug, > www.python.org/sf/546434 . > > The root problem is that mybuf[:] returns a buffer type and mybuf[2:4] > returns a string type. A similar issue exists for buffer repetition. > > One way to go is to have the slices always return a string. If code > currently relies on the type of a buffer slice, it is more likely to be > relying on it being a string as in: print mybuf[:4]. This is an intuitive > guess because I can't find empirical evidence. Another reason to choose a > string return type is that buffer() appears to have been designed to be as > stringlike as possible so that it can be easily substituted in code > originally designed for strings. > > The other way to go is to return a buffer object everytime. Slices usually, > but not always (see subclasses of list), return the same type that was being > sliced. If we choose this route, another issue remains -- mybuf[:] returns > self instead of a new buffer. I think that behavior is also a bug and > should be changed to be consistent with the Python idiom where: > b = a[:] > assert id(a) != id(b) > > Incidental to the above, GvR had a thought that slice repetition ought to > always return an error. Though I don't see any use cases for buffer > repetition, bufferobjects do implement all other sequence behaviors and I > think it would be weird to nullify the sq_repeat slot. > > I appreciate your thoughts on the best way to proceed. > > fixing-bugs-is-easier-than-deciding-appropriate-behavior-ly yours, > > > 'regnitteh dnomyar'[::-1] > > > > > > > > > > > > > >

9 33

PEP 292-related: why string substitution is not the same operation as data formatting
by Lalo Martins 18 Jul '02

18 Jul '02

For a moment, please remove for your mind your experience of C and printf. Meditate with me and picture yourself in a happy world of object-orientation and code readability, where everything cryptic and obscured is banished. Just to help you do that, I'll avoid the notation chosen by the PEP. Let's use, for the duration of this post, the hypothetic notation suggested by some other reader: "<<name>> is from <<country>>". Now, this thing we're talking about is replacing parts of the string with other strings. These strings may be the result of running some non-string objects trough str(foo) - but, we are making no assumptions about these objects. Just that str(foo) is somehow meaningful. And, to my knowledge, there are no python objects for which str(foo) doesn't work. So, string substitution is non-intrusive. Also, if you keep your templates (let's call a string containing substitution markup a template, shall we?) outside your source code, as is the case with i18n, pure substitution doesn't require the people who edit them (for example, translators) to know anything about python *or* even programming. String substitution only depends on an identifier ('name' or 'country'), no sick abbreviations like 's' or 'd' or 'f' or 'r' or 'x' that you have to keep a table for. So, string substitution is readable and non-cryptic. Now, data formatting is another animal entirely. It's a way to request one specific representation of a piece of data. But there is a catch. When you do '%8.3d' % foo you are *expecting* that foo a floating-point number and you know you'll get TypeError otherwise. This is, IMO, invasive. In my ideal OO-paradise I would rather have something like foo.format(8, 3) (THIS IS NOT A PEP!). IMO, if you, as I asked in the first paragraph, pretend you don't know C and printf and python's % operator and then pretend you're having your first contact with it, while already having some experience with python's readability, it's hard not to be shocked. And I bet you'd go to great lengths to avoid using the "feature". Conclusion: I think string formatting is a cryptic and obscure misfeature inherited from C that should be deprecated in favour of something less invasive and more readable/explicit. More, I'm completely opposed to "<<name>> is <<age:.0d>> years old" because it's still cryptic and invasive. This should instead read similar to "<<name>> is <<age>> years old".sub({'name': x.name, 'age': x.age.format(None, 0)}) Guido, can you please, for our enlightenment, tell us what are the reasons you feel %(foo)s was a mistake? []s, |alo +---- -- It doesn't bother me that people say things like "you'll never get anywhere with this attitude". In a few decades, it will make a good paragraph in my biography. You know, for a laugh. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Python Foundry Guide http://www.sf.net/foundry/python-foundry/

10 15

Provide a Python wrapper for any new C extension
by Hamish Lawson 15 Jul '02

15 Jul '02

One of the arguments put forward against renaming the existing time module to _time (as part of incorporating a pure-Python strptime function) is that it could break some builds. Therefore I'd suggest that it could be a useful principle for any C extension added in the future to the standard library to have an accompanying pure-Python wrapper that would be the one that client code would usually import. Hamish Lawson

9 35

PEP 294: Type Names in the types Module
by Oren Tirosh 15 Jul '02

15 Jul '02

PEP: 294 Title: Type Names in the types Module Version: $Revision: 1.1 $ Last-Modified: $Date: 2002/06/23 23:52:19 $ Author: oren at hishome.net (Oren Tirosh) Status: Draft Type: Standards track Created: 19-Jun-2002 Python-Version: 2.3 Post-History: Abstract This PEP proposes that symbols matching the type name should be added to the types module for all basic Python types in the types module: types.IntegerType -> types.int types.FunctionType -> types.function types.TracebackType -> types.traceback ... The long capitalized names currently in the types module will be deprecated. With this change the types module can serve as a replacement for the new module. The new module shall be deprecated and listed in PEP 4. Rationale Using two sets of names for the same objects is redundant and confusing. In Python versions prior to 2.2 the symbols matching many type names were taken by the factory functions for those types. Now all basic types have been unified with their factory functions and therefore the type names are available to be consistently used to refer to the type object. Most types are accessible as either builtins or in the new module but some types such as traceback and generator are only accssible through the types module under names which do not match the type name. This PEP provides a uniform way to access all basic types under a single set of names. Specification The types module shall pass the following test: import types for t in vars(types).values(): if type(t) is type: assert getattr(types, t.__name__) is t The types 'class', 'instance method' and 'dict-proxy' have already been renamed to the valid Python identifiers 'classobj', 'instancemethod' and 'dictproxy', making this possible. Backward compatibility Because of their widespread use it is not planned to actually remove the long names from the types module in some future version. However, the long names should be changed in documentation and library sources to discourage their use in new code. Reference Implementation A reference implementation is available in SourceForge patch #569328: http://www.python.org/sf/569328 Copyright This document has been placed in the public domain.

5 7

Minor socket timeout quibble - timeout raises socket.error
by Skip Montanaro 14 Jul '02

14 Jul '02

I just noticed in the development docs that when a timeout on a socket occurs, socket.error is raised. I rather liked the idea that a different exception was raised for timeouts (I used Tim O'Malley's timeout_socket module). Making a TimeoutError exception a subclass of socket.error would be fine so you can catch it with existing code, but I could see recovering differently for a timeout as opposed to other possible errors: sock.settimeout(5.0) try: data = sock.recv(8192) except socket.TimeoutError: # maybe requeue the request ... except socket.error, codes: # some more drastic solution is needed ... Skip

4 7

Dict constructor
by Raymond Hettinger 14 Jul '02

14 Jul '02

Second wild idea of the day: The dict constructor currently accepts sequences where each element has length 2, interpreted as a key-value pair. Let's have it also accept sequences with elements of length 1, interpreted as a key:None pair. The benefit is that it provides a way to rapidly construct sets: lowercase = dict('abcdefghijklmnopqrstuvwxyz') if char in lowercase: ... dict([key1, key2, key3, key1]).keys() # eliminate duplicate keys Raymond Hettinger 'regnitteh dnomyar'[::-1]

13 27

Xrange and Slices
by Raymond Hettinger 12 Jul '02

12 Jul '02

Wild idea of the day: Merge the code for xrange() into slice(). So that old code will work, make the word 'xrange' a synonym for 'slice' >>> x = xrange(0,10,2) >>> s = slice(0,10,2) >>> [m for m in dir(x) if m not in dir(s)] ['__getitem__', '__iter__', '__len__'] >>> [m for m in dir(s) if m not in dir(x)] ['__cmp__', 'start', 'step', 'stop'] Raymond Hettinger 'regnitteh dnomyar'[::-1]

7 15

PEP 292, Simpler String Substitutions
by Damien Morton 12 Jul '02

12 Jul '02

"I'd rather have a notation that's less error-prone than a better way to check for errors. (Not that PyChecker 2 isn't a great idea. :-)" Percent notation "%s" notation already exists for strings. Backquote notation is already in python, though I think, little used. The $ notation reeks of obscure languages such as perl and shell. Why not simply add backquote notation to python strings. I read in a recent email from Timbot, I think, that the backquote notation was originally intended for string interpolation too. name = "guido" country = "the netherlands" height = 1.92 "`name` is from `country`".sub() -> "guido is from the netherlands" "`name.capitalize()` is from `country`" -> "Guido is from the netherlands" "`name` is %`height`4.1f meters tall".sub() -> "guido is 1.9 meters tall" "`name.capitalize()` can jump `height*1.7` meters".sub() -> "guido can jump 3.264 meters" You could probably also compile these interpolation strings as well. Another thought: One of the main problems with the "%(name)4.2f" notation is that the format comes after the variable name. Is easy to forget adding the actual format specifier in after the name. Why not alter the notation to allow the format specifier to come before the name part. "%4.2f(height)" I think would be a whole lot less error prone, and would allow for the format specifier to default to "s" where omitted. "%(height)" is also less error prone, though it is ambiguous in the current scheme. Ive seen a nice class which evaluates the string used in the name part. class itpl: def __getitem__(self, s): return eval(s, globals())

14 29

New Subscriber Introduction
by Brett Cannon 12 Jul '02

12 Jul '02

Uh, I realize this is a little late, but I didn't thoroughly read the intro email for the list and so I didn't realize this was requested into lalo sent his email. Anyway, better late then never. So my name is Brett Cannon. I am a recent graduate of the philosophy program here at UC Berkeley. I am taking a year off from school (and apparently employment =P) while I apply to grad school in hopes of persuing as masters or doctorate in CS. I also hope to discover an area of CS that I love above all else during this year so that I can stop jumping between different areas of programming (I have the slight issue of wnating to be the best that I can be at everything and this jumping around is not helping with that =). I discovered Python in Fall 2000. Was trying to choose a language to use to teach myself OOP before I took my first CS course here at Cal. Have been using Python pretty exclusively since except for CS coursework; Python spoiled me and didn't help when I had to use Scheme, Lisp, or Java in my classes. The only grumblings anyone has heard out of me as yet on this list is over my Python implementation of strptime. I do plan to stay on this list, though, even after this is resolved and be as involved as I can on the list (which is going to be limited until I get off my rear and really dive into the C source). -Brett C.

5 10

Indexing builtin sequences with objects which supply __int__
by Todd Miller 12 Jul '02

12 Jul '02

There has been some recent interest in the Numeric/numarray community for using array objects as indices for builtin sequences. I know this has come up before, but to make myself clear, the basic idea is to make the following work: class C: def __int__(self): return 5 object = C() l = "Another feature..." print l[object] "h" Are there any plans (or interest) for developing Python in this direction? Todd

6 16

deprecating string module?
by Skip Montanaro 11 Jul '02

11 Jul '02

I know this has been discussed before. In light of the recent PEP8 update: Avoid the use of the string module; instead use string methods. These are always much faster and share the same API with unicode strings. I thought I'd raise it again. How about deprecating the string module? In the past, this has presented two major problems. First, its near ubiquity in Python programs before the availability of string methods. Second, the presence of a few useful data objects (digits, uppercase, etc). The first problem can be solved by extending the deprecation time suitably (two years/four releases?) I think the second problem can be solved by adding those data objects to either sys or locale(preferably the latter). string.letters and its subsets are locale-dependent. In relevant locales I presume string.punctuation and string.whitespace would also be different than their common ASCII elements. Finally, I'd amend the above PEP8 admontion something like Avoid the use of the string module unless backward-compatibility with versions earlier than Python 2.0 is important; instead use string methods. These are always much faster and share the same API with unicode strings. Skip

22 118

Re: *Simpler* string substitutions
by Moore, Paul 11 Jul '02

11 Jul '02

Some points on the current thread, in no particular order... 1. While I agree that "$" is better known as an interpolation character than "%", it shouldn't be forgotten that "%" is the interpolation character in DOS/Windows shells. Some recent examples which showed "%" used ("The sum of %x and %y is %(x+y)") looked entirely natural to me (I use Windows more than Unix) - in fact, more so than "$"!! 2. The internationalisation issue is clearly important. However, it has very different characteristics insofar as the template string is (of necessity) handled at runtime, so issues of compilation and security become relevant. I'm no I18N expert, so I can't comment on details, but I *do* think it's worth separating out the I18N issues from the "simple interpolation" issues... 3. I feel that the existing % formatting operator cannot realistically be removed. Tidying up some of its warts may be possible, and even sensible, but there's too much code using it (and as was pointed out, template strings may not event be stored in code files) to make major changes. 4. Access to variables is also problematic. Without compile-time support, access to nested scopes is impossible (AIUI). But on the other hand, a scheme with subtle limitations such as lack of such access may not realistically count as "simple"... 5. (Personal opinion here!) I believe that formatting specifiers so not belong in a "simple" scheme - leave them for the "advanced" verion (the existing % operator). On the other hand, I feel that expression interpolation, within limits, *is* suitable. It's the user's responsibility not to go overboard, though... Sorry for butting into an already long thread. I hope the summary is useful, at least... Paul.

4 3

List comprehensions
by Gerald S. Williams 05 Jul '02

05 Jul '02

Has anyone summarized the list comprehension design discussions? I found references to "lots of discussion" about it but haven't yet found the discussions themselves. I don't want to rehash any old discussions, but I came across a surprise recently while converting constructs like "map(lambda x:x+1,x)" and just wanted to see the rationale behind not creating a local scope for list comprehension variables. Any pointer would be appreciated. Thanks, -Jerry

6 7

Infinie recursion in Pickle
by Oleg Broytmann 02 Jul '02

02 Jul '02

Hello! Nobody noted the message in c.l.py, let me try to ask you before I file a bug report on SF. ----- Forwarded message from Oleg Broytmann <phd(a)phd.pp.ru> ----- On Thu, Jun 27, 2002 at 08:45:16PM +0000, Bengt Richter wrote: > > Recently Python (one program that I am dbugging) started to crash. > >FreeBSD kills it with "Bus error", Linux with "Segmentation fault". > > > > I think the program crashed in the cPickle.dump(file, 1) I replaced cPikle.dump with pikle.dump and got infinite rcursion. The traceback is below. What's that? Are there any limits that an object to be pikled must follow? Could it be a tree with loops? (I am pretty sure it could - I used the program for years, and data structures was not changed much). Could it be "new" Python class? (Recently I changed one of my classes to be derived from builtin list instead of UserList). Well (or not so well), the traceback: Traceback (most recent call last): File "/home/phd/lib/bookmarks_db/check_urls.py", line 158, in ? run() File "/home/phd/lib/bookmarks_db/check_urls.py", line 145, in run storage.store(root_folder) File "bkmk_stpickle.py", line 23, in store File "/usr/local/lib/python2.2/pickle.py", line 973, in dump Pickler(file, bin).dump(object) File "/usr/local/lib/python2.2/pickle.py", line 115, in dump self.save(object) File "/usr/local/lib/python2.2/pickle.py", line 219, in save self.save_reduce(callable, arg_tup, state) File "/usr/local/lib/python2.2/pickle.py", line 245, in save_reduce save(arg_tup) File "/usr/local/lib/python2.2/pickle.py", line 225, in save f(self, object) File "/usr/local/lib/python2.2/pickle.py", line 374, in save_tuple save(element) File "/usr/local/lib/python2.2/pickle.py", line 225, in save f(self, object) [about 1000 lines skipped - they are all the same] File "/usr/local/lib/python2.2/pickle.py", line 498, in save_inst save(stuff) File "/usr/local/lib/python2.2/pickle.py", line 225, in save f(self, object) File "/usr/local/lib/python2.2/pickle.py", line 447, in save_dict save(value) File "/usr/local/lib/python2.2/pickle.py", line 219, in save self.save_reduce(callable, arg_tup, state) File "/usr/local/lib/python2.2/pickle.py", line 245, in save_reduce save(arg_tup) File "/usr/local/lib/python2.2/pickle.py", line 225, in save f(self, object) File "/usr/local/lib/python2.2/pickle.py", line 374, in save_tuple save(element) File "/usr/local/lib/python2.2/pickle.py", line 225, in save f(self, object) File "/usr/local/lib/python2.2/pickle.py", line 414, in save_list save(element) File "/usr/local/lib/python2.2/pickle.py", line 143, in save pid = self.persistent_id(object) RuntimeError: maximum recursion depth exceeded ----- End forwarded message ----- Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd(a)phd.pp.ru Programmers don't die, they just GOSUB without RETURN.

10 19

Silent Deprecation Candidate -- buffer()
by Raymond Hettinger 01 Jul '02

01 Jul '02

As far as I can tell, buffer() is one of the least used or known about Python tools. What do you guys think about this as a candidate for silent deprecation (moving out of the primary documentation)? Raymond Hettinger

6 12

posixmodule.c diffs for working forkpty() and openpty() under Solaris 2.8
by Lance Ellinghaus 30 Jun '02

30 Jun '02

Hello everyone! I had to get forkpty() and openpty() working under Solaris 2.8 for a project I am working on. Here are the diffs to the 2.2.1 source file. Please let me know if anyone has any problems with this! Lance Ellinghaus ===== -- Lance Ellinghaus __________________________________________________ Do You Yahoo!? Yahoo! - Official partner of 2002 FIFA World Cup http://fifaworldcup.yahoo.com *** Python-2.2.1/Modules/posixmodule.c Tue Mar 12 16:38:31 2002 --- Python-2.2.1.new/Modules/posixmodule.c Tue May 21 01:16:29 2002 *************** *** 1904,1910 **** } #endif ! #if defined(HAVE_OPENPTY) || defined(HAVE_FORKPTY) #ifdef HAVE_PTY_H #include <pty.h> #else --- 1904,1913 ---- } #endif ! #if defined(HAVE_OPENPTY) || defined(HAVE_FORKPTY) || defined(sun) ! #ifdef sun ! #include <sys/stropts.h> ! #endif #ifdef HAVE_PTY_H #include <pty.h> #else *************** *** 1914,1920 **** #endif /* HAVE_PTY_H */ #endif /* defined(HAVE_OPENPTY) || defined(HAVE_FORKPTY) */ ! #if defined(HAVE_OPENPTY) || defined(HAVE__GETPTY) static char posix_openpty__doc__[] = "openpty() -> (master_fd, slave_fd)\n\ Open a pseudo-terminal, returning open fd's for both master and slave end.\n"; --- 1917,1923 ---- #endif /* HAVE_PTY_H */ #endif /* defined(HAVE_OPENPTY) || defined(HAVE_FORKPTY) */ ! #if defined(HAVE_OPENPTY) || defined(HAVE__GETPTY) || defined(sun) static char posix_openpty__doc__[] = "openpty() -> (master_fd, slave_fd)\n\ Open a pseudo-terminal, returning open fd's for both master and slave end.\n"; *************** *** 1925,1932 **** int master_fd, slave_fd; #ifndef HAVE_OPENPTY char * slave_name; #endif ! if (!PyArg_ParseTuple(args, ":openpty")) return NULL; --- 1928,1941 ---- int master_fd, slave_fd; #ifndef HAVE_OPENPTY char * slave_name; + #ifdef sun + void *sig_saved; #endif ! #endif ! #if !defined(HAVE_OPENPTY) && !defined(HAVE__GETPTY) && defined(sun) ! extern char *ptsname(); ! #endif ! if (!PyArg_ParseTuple(args, ":openpty")) return NULL; *************** *** 1933,1939 **** #ifdef HAVE_OPENPTY if (openpty(&master_fd, &slave_fd, NULL, NULL, NULL) != 0) return posix_error(); ! #else slave_name = _getpty(&master_fd, O_RDWR, 0666, 0); if (slave_name == NULL) return posix_error(); --- 1942,1948 ---- #ifdef HAVE_OPENPTY if (openpty(&master_fd, &slave_fd, NULL, NULL, NULL) != 0) return posix_error(); ! #elif HAVE__GETPTY slave_name = _getpty(&master_fd, O_RDWR, 0666, 0); if (slave_name == NULL) return posix_error(); *************** *** 1941,1946 **** --- 1950,1966 ---- slave_fd = open(slave_name, O_RDWR); if (slave_fd < 0) return posix_error(); + #else + master_fd = open("/dev/ptmx", O_RDWR|O_NOCTTY); /* open master */ + sig_saved = signal(SIGCHLD, SIG_DFL); + grantpt(master_fd); /* change permission of slave */ + unlockpt(master_fd); /* unlock slave */ + signal(SIGCHLD,sig_saved); + slave_name = ptsname(master_fd); /* get name of slave */ + slave_fd = open(slave_name, O_RDWR); /* open slave */ + ioctl(slave_fd, I_PUSH, "ptem"); /* push ptem */ + ioctl(slave_fd, I_PUSH, "ldterm"); /* push ldterm*/ + ioctl(slave_fd, I_PUSH, "ttcompat"); /* push ttcompat*/ #endif /* HAVE_OPENPTY */ return Py_BuildValue("(ii)", master_fd, slave_fd); *************** *** 1948,1954 **** } #endif /* defined(HAVE_OPENPTY) || defined(HAVE__GETPTY) */ ! #ifdef HAVE_FORKPTY static char posix_forkpty__doc__[] = "forkpty() -> (pid, master_fd)\n\ Fork a new process with a new pseudo-terminal as controlling tty.\n\n\ --- 1968,1974 ---- } #endif /* defined(HAVE_OPENPTY) || defined(HAVE__GETPTY) */ ! #if defined(HAVE_FORKPTY) || defined(sun) static char posix_forkpty__doc__[] = "forkpty() -> (pid, master_fd)\n\ Fork a new process with a new pseudo-terminal as controlling tty.\n\n\ *************** *** 1959,1968 **** --- 1979,2067 ---- posix_forkpty(PyObject *self, PyObject *args) { int master_fd, pid; + #if defined(sun) + int slave; + char * slave_name; + void *sig_saved; + int fd; + #endif if (!PyArg_ParseTuple(args, ":forkpty")) return NULL; + #if defined(sun) + master_fd = open("/dev/ptmx", O_RDWR|O_NOCTTY); /* open master */ + sig_saved = signal(SIGCHLD, SIG_DFL); + grantpt(master_fd); /* change permission of slave */ + unlockpt(master_fd); /* unlock slave */ + signal(SIGCHLD,sig_saved); + slave_name = ptsname(master_fd); /* get name of slave */ + slave = open(slave_name, O_RDWR); /* open slave */ + ioctl(slave, I_PUSH, "ptem"); /* push ptem */ + ioctl(slave, I_PUSH, "ldterm"); /* push ldterm*/ + ioctl(slave, I_PUSH, "ttcompat"); /* push ttcompat*/ + if (master_fd < 0 || slave < 0) + { + return posix_error(); + } + switch (pid = fork()) { + case -1: + return posix_error(); + case 0: + /* First disconnect from the old controlling tty. */ + #ifdef TIOCNOTTY + fd = open("/dev/tty", O_RDWR | O_NOCTTY); + if (fd >= 0) { + (void) ioctl(fd, TIOCNOTTY, NULL); + close(fd); + } + #endif /* TIOCNOTTY */ + if (setsid() < 0) + return posix_error(); + + /* + * Verify that we are successfully disconnected from the controlling + * tty. + */ + fd = open("/dev/tty", O_RDWR | O_NOCTTY); + if (fd >= 0) { + return posix_error(); + close(fd); + } + /* Make it our controlling tty. */ + #ifdef TIOCSCTTY + if (ioctl(slave, TIOCSCTTY, NULL) < 0) + return posix_error(); + #endif /* TIOCSCTTY */ + fd = open(slave_name, O_RDWR); + if (fd < 0) { + return posix_error(); + } else { + close(fd); + } + /* Verify that we now have a controlling tty. */ + fd = open("/dev/tty", O_WRONLY); + if (fd < 0) + return posix_error(); + else { + close(fd); + } + (void) close(master_fd); + (void) dup2(slave, 0); + (void) dup2(slave, 1); + (void) dup2(slave, 2); + if (slave > 2) + (void) close(slave); + pid = 0; + break; + defautlt: + /* + * parent + */ + (void) close(slave); + } + #else pid = forkpty(&master_fd, NULL, NULL, NULL); + #endif if (pid == -1) return posix_error(); if (pid == 0) *************** *** 5607,5616 **** #ifdef HAVE_FORK {"fork", posix_fork, METH_VARARGS, posix_fork__doc__}, #endif /* HAVE_FORK */ ! #if defined(HAVE_OPENPTY) || defined(HAVE__GETPTY) {"openpty", posix_openpty, METH_VARARGS, posix_openpty__doc__}, #endif /* HAVE_OPENPTY || HAVE__GETPTY */ ! #ifdef HAVE_FORKPTY {"forkpty", posix_forkpty, METH_VARARGS, posix_forkpty__doc__}, #endif /* HAVE_FORKPTY */ #ifdef HAVE_GETEGID --- 5706,5715 ---- #ifdef HAVE_FORK {"fork", posix_fork, METH_VARARGS, posix_fork__doc__}, #endif /* HAVE_FORK */ ! #if defined(HAVE_OPENPTY) || defined(HAVE__GETPTY) || defined(sun) {"openpty", posix_openpty, METH_VARARGS, posix_openpty__doc__}, #endif /* HAVE_OPENPTY || HAVE__GETPTY */ ! #if defined(HAVE_FORKPTY) || defined(sun) {"forkpty", posix_forkpty, METH_VARARGS, posix_forkpty__doc__}, #endif /* HAVE_FORKPTY */ #ifdef HAVE_GETEGID

3 4

Building Python cvs w/ gcc 3.1
by barry＠zope.com 29 Jun '02

29 Jun '02

File this under "you just can't win". I'm building Python cvs w/gcc 3.1 and I get warnings for every extension, e.g.: building 'zlib' extension gcc -g -Wall -Wstrict-prototypes -fPIC -I. -I/home/barry/projects/python/./Include -I/usr/local/include -I/home/barry/projects/python/Include -I/home/barry/projects/python -c /home/barry/projects/python/Modules/zlibmodule.c -o build/temp.linux-i686-2.3/zlibmodule.o cc1: warning: changing search order for system directory "/usr/local/include" cc1: warning: as it has already been specified as a non-system directory gcc -shared build/temp.linux-i686-2.3/zlibmodule.o -L/usr/local/lib -lz -o build/lib.linux-i686-2.3/zlib.so The problem is the inclusion of -I/usr/local/include because that's a directory on gcc's system include path. Adding such directories can cause gcc headaches because it likes to treat system include dirs specially, doing helpful things like fix bugs in vendor's header files and sticking them in special locations. -I apparently overrides the special treatment of system include dirs, so the warnings are gcc's way of helpfully reminding us not to do that. Unfortunately, it seems difficult to fix this in a principled way. I can't figure out a way to reliably ask gcc what its system include dirs are. -v doesn't give you the information. There's no switch to turn off these warnings. You could ask cpp ("cpp -v") which does provide output that could be grep'd for the system include dirs, but that just seems way too fragile. Besides, that doesn't play well with distutils because it only wants to invoke the preprocessor using "gcc -E" and /that/ interprets -v as one of its options. You could use "gcc -E -Wp,-v dummyfile.c" but then you'd have to redirect stderr, capture that output, and grep it. Blech, blech, blech. If I comment out the line in setup.py which add /usr/local/include to self.compiler.include_dirs, it takes care of the problem, but that might break other builds, so I'm loathe to do that. The other option is to ignore the warnings since I don't think gcc 3.1 (3.x?) is distributed as the default compiler for very many distros yet. OTOH, it /will/ at some point and then it will be a PITA for support <wink>. OTTH, from some quick googling, I gather that this warning is somewhat controversial inside the gcc community, and other projects have dealt with it in heavyhanded ways (just don't -I/usr/local/include), so if we ignore it long enough, the problem might just go away. Sigh, I'm done with this for now, but wanted to get it into the archives for future reference. -Barry

2 2

Asyncore/asynchat
by Steve Holden 29 Jun '02

29 Jun '02

I thought I might try to add appropriate module documentation for asynchat. This effective code doesn't get enough recognition (IMHO), partly because you are forced to read the code to understand how to use it. I notice that Sam Rushing's code tends to use spaces before the parentheses around argument lists. Should I think about cleaning up the code at the same time, or are we best letting sleeping dogs lie? regards ----------------------------------------------------------------------- Steve Holden http://www.holdenweb.com/ Python Web Programming http://pydish.holdenweb.com/pwp/ -----------------------------------------------------------------------

4 3

Re: PEP 292, Simpler String Substitutions
by pinard＠iro.umontreal.ca 29 Jun '02

29 Jun '02

[Barry A. Warsaw] > This PEP describes a simpler string substitution feature, also > known as string interpolation. This PEP is "simpler" in two > respects: > 1. Python's current string substitution feature (commonly known as > %-substitutions) is complicated and error prone. This PEP is > simpler at the cost of less expressiveness. > 2. PEP 215 proposed an alternative string interpolation feature, > introducing a new `$' string prefix. PEP 292 is simpler than > this because it involves no syntax changes and has much simpler > rules for what substitutions can occur in the string. For one, I do not like seeing `$' as a string prefix in Python, and wonder if we could not merely go with `%' as we always did in Python. At least, it keeps a kind of clear cut distance between Python and Perl. :-) > In addition, the rules for what can follow a % sign are fairly > complex, while the usual application rarely needs such complexity. This premise seems exaggerated to me. `%' as it stands is not that complex to understand. Moreover, many of us use `%' formatting a lot, so it is not so rare that the current `%' specification is useful. > 1. $$ is an escape; it is replaced with a single $ Let's suppose we stick with `%', the above rule reduces to something already known. > 3. ${identifier} [...] We could use %{identifier} as meaning `%(identifier)s'. Clean. Simple. > 2. $identifier [...] This is where the difficulty lies. Since the PEP already suggests that ${identifier} was to be preferred over $identifier, why not just go a bit forward, and drop 2. altogether? Or else, how do you justify that using it really make things more legible? Then, the whole proposal would reduce to adding %{identifier}, and instead of having `.sub()' methods or whatever, just stick with what we already have. This would be a mild change instead of a whole new feature, and keep Python a little more wrapped to itself. Interpolation proposals I've seen always looked a bit awkward and foreign so far. I guess that merely adding %{identifier} would wholly satisfy the given justifications for the PEP (that is, giving a mean for people to avoid the %()s as error prone), with a minimal impact on the current Python definition, and a bit less of a surprise. Python does not have to look like Perl to be useful, you know! :-) > Handling Missing Keys This would be a non-issue, by the fact that %(identifier)s behaviour, for undefined identifier, is already what we want. > The mapping argument is optional; if it is omitted then the mapping is > taken from the locals and globals of the context in which the .sub() > method is executed. This is an interesting idea. However, there are other contexts where the concept of a compound dictionary of all globals and locals would be useful. Maybe we could have some allvars() similar to globals() and locals(), and use `... % allvars()' instead of `.sub()'? So this would serve both string interpolation and other avenues. I hope I succeed to express my feeling that we should try keeping string interpolation rather natural with what Python already is. We should not carelessly multiply paradigms. -- François Pinard http://www.iro.umontreal.ca/~pinard

14 46

Re: [Python-Dev] Silent Deprecation Candidate -- buffer()
by Michael Hudson 29 Jun '02

29 Jun '02

j.her(a)t-online.de (Juergen Hermann) writes: > On Sat, 29 Jun 2002 13:02:13 +0200, Fredrik Lundh wrote: > > >does anyone have any real-life use cases? I've never been > >able to use it for anything, and cannot recall ever seeing it > >being used by anyone else... > > We use it for BLOB support in our Python binding to our C++ binding to > Oracle OCI. Oracle allows loading of limited ranges out of BLOBs, and > the buffer interface perfectly fits into that. But that's from C, right? I don't think anyone's suggested removing the C-level buffer interface. Cheers, M. -- I think perhaps we should have electoral collages and construct our representatives entirely of little bits of cloth and papier mache. -- Owen Dunn, ucam.chat, from his review of the year

1 0

Garbage collector problem
by Kevin Jacobs 29 Jun '02

29 Jun '02

I've found what I consider a major problem with the garbage collector in the Python 2.3 CVS tree. Here is a small kernel that demonstrates the problem: lst = [] for i in range(100000): lst.append( (1,) ) The key ingredients are: 1) A method is called on a container (rather than __setitem__ or __setattr__). 2) A new object is allocated while the method object lives on the Python VM stack, as shown by the disassembled bytecodes: 40 LOAD_FAST 1 (lst) 43 LOAD_ATTR 3 (append) 46 LOAD_CONST 2 (1) 49 BUILD_TUPLE 1 52 CALL_FUNCTION 1 These ingredients combine in the following way to trigger quadratic-time behavior in the Python garbage collector: * First, the LOAD_ATTR on "lst" for "append" is called, and a PyCFunction is returned from this code in descrobject.c:method_get: return PyCFunction_New(descr->d_method, obj); Thus, a _new_ PyCFunction is allocated every time the method is requested. * This new method object is added to generation 0 of the garbage collector, which holds a reference to "lst". * The BUILD_TUPLE call may then trigger a garbage collection cycle. * Since the "append" method is in generation 0, the reference traversal must also follow all objects within "lst", even if "lst" is in generation 1 or 2. This traversal requires time linear in the number of objects in "lst", thus increasing the overall time complexity of the code to quadratic in the number of elements in "lst". Also note that this is a much more general problem than this small example. It can affect many types of objects in addition to methods, including descriptors, iterator objects, and any other object that contains a "back reference". So, what can be done about this.... One simple solution would be to not traverse some "back references" if we are collecting objects in generation 0. This will avoid traversing virtually all of these ephemoral objects that will trigger such expensive behavior. If they live long enough to pass through to generation one or two, then clearly they should be traversed. So, what do all of you GC gurus think? Provided that my analysis is sound, I can rapidly propose a patch to demonstrate this approach if there is sufficient positive sentiment. There is a bug open on sourceforge on this issue, so feel free to reply via python-dev or via the bug -- I read both. As usual sourceforge is buggered, so I have not been able to update the bug with the contents of this e-mail. http://sourceforge.net/tracker/?func=detail&atid=105470&aid=572567&group_id… Regards, -Kevin PS: I have not looked into why this doesn't happen in Python 2.2.x or before. I suspect that it must be related to the recent GC changes in methodobject.py. I'm not motivated to spend much time looking into this, because the current GC behavior is technically correct, though clearly sub-optimal. -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs(a)theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com

6 15