Mailman 3 November 2003 - Python-Dev

XXX undetected error (why=3)
by Jeremy Hylton Nov. 4, 2003

Nov. 4, 2003

I've been seeing these problems sporadically over the last several months. Fred and I tracked one of them down to a bug in pyexpat.c. I noticed several more running the Zope 3 test suite today. I'd like to change the code for the check to call Py_FatalError() instead of printing a message to stderr. The check is only enabled during a debug build. I'd be much happier debugging this from a core dump than trying to figure out what happened to cause the message to be printed. Any objections? Jeremy

2 1

Re: PEP 318 (was Re: [Python-Dev] Re: Guido's Magic Code was: inlinesort option)
by Phillip J. Eby Nov. 4, 2003

Nov. 4, 2003

At 09:24 AM 11/4/03 +0100, Alex Martelli wrote: >On Tuesday 04 November 2003 02:09 am, Phillip J. Eby wrote: > ... > > I'm not all that enthused about the metaclass usage, mainly because there's > > already an okay syntax (__metaclass__) for it. I'd rather that class > >Hmmm -- why is: > >class Foo: > __metaclass__ = MetaFoo > ... > >"ok", compared to e.g.: > >class Foo is MetaFoo: > ... > >while, again for example, > > def foo(): > ... > foo = staticmethod(foo) > >is presumably deemed "not ok" compared to e.g.: > > def foo() is staticmethod: > ... > >??? Isn't it obvious from the above? Note the positioning of the '...' in all but the third example you've shown. :) > > decorators (if added) were decorators in the same way as function > > decorators. Why? Because I think that correct, combinable class > > decorators are probably easier for most people to write than correct, > > combinable metaclasses, and they are more easily combined than metaclasses > > are. > >Combinable metaclasses may not be trivial to write, but with multiple >inheritance it will often be feasible (except, presumably, when implied >layout or __new__ have conflicting requirements). I guess my point is that it's harder to *learn* how to write a co-operative metaclass, than it is to simply *write* a co-operative decorator. A metaclass must explicitly invoke its collaborators, but a decorator is just a simple function; the chaining is external. Now, certainly you and I both know how to write our metaclasses co-operatively, but I believe both of us have also been told (repeatedly) that we're not typical Python programmers. :) > Of course, not having use >cases of either custom metaclasses or class decorators in production use, the >iscussion does risk being a bit abstract. Did you have any specific use case >in mind? PyProtocols has an API call that "wants" to be a class decorator, to declare interface information about the class, e.g: class MyClass is protocols.instancesProvide(IFoo): .... But, since there are no such things as class decorators, it actually uses a sys._getframe() hack to replace the metaclass and simulate decoratorness. Steve Alexander originally proposed the idea as an implementation technique for interface declarations in Zope 3, and I worked up the actual implementation, that's now shared by PyProtocols and Zope 3. So the above is actually rendered now as: class MyClass: protocols.advise(instancesProvide=[IFoo]) (Note that any explicit __metaclass__ declaration has to come *before* the advise() call.) The principal limitation of this technique is that writing co-operative decorators of this sort is just as difficult as writing co-operative metaclasses. So, PyProtocols and Zope 3 include a library function, 'addClassAdvisor(decorator_callable)' which adds a decorator function (in a PEP 218-style execution order) to those that will be called on the resulting class. IOW, we created a decorator mechanism for classes that is almost identical to the PEP 218 mechanism for functions, to make it easy to call functions on a created class, using declarations that occur near the class statement. This was specifically to make it easier to do simple decorator-like things, without writing metaclasses, and thus not interfering with user-supplied metaclasses. Note, by the way, that since you can only have one explicit metaclass, and Python does not automatically generate new metaclasses, users must explicitly mix metaclasses in order to use them. That's all well and good for gurus such as ourselves, but if you're creating a framework that wants to play nicely with other frameworks, and is for non-guru users, then metaclasses are right out unless they're the *only* way to achieve the desired effect. For supplying framework metadata, decorators are an adequate mechanism that's simpler to implement, and are therefore preferable. Since the 'addClassAdvisor()' mechanism has been available, I've used it for other framework metadata annotations, such as security restrictions, and to perform miscellaneous other "postprocessing" operations on classes. Now, in the time since Steve Alexander first proposed the idea, I've actually grown to like the in-body declaration style for classes, and it's possible that PEP 218-style declaration for classes would be more unwieldy. So I'm only +0 on having a class decorator syntax at all. But I do think that if there *is* a class decorator syntax, its semantics should exactly match function decorator syntax, and am therefore -1 on it being metaclass syntax. In my experience, non-guru usage of metaclasses is usually by inheriting the metaclass from a framework base class, and this is the "right way to do it" because the user shouldn't need to know about metaclasses unless they are mixing them. (And if Python mixed them for you, there'd be no need for non-gurus to know about metaclasses at all.)

1 1

RE: PEP 318 (was Re: [Python-Dev] Re: Guido's Magic Code was: inlinesort option)
by Phillip J. Eby Nov. 4, 2003

Nov. 4, 2003

At 11:02 AM 11/4/03 +1100, Delaney, Timothy C (Timothy) wrote: > > From: Alex Martelli [mailto:aleaxit@yahoo.com] > > > > BTW, when we do come around to PEP 318, I would suggest the 'as' > > clause on a class statement as the best way to specify a metaclass. > >I just realised what has been bugging me about the idea of > > def foop() as staticmethod: > >and it applies equally well to > > class Newstyle as type: > >Basically, it completely changes the semantics associated with 'as' in >Python - which are to give something a different name (technically, to >rebind the object to a different name). > >OTOH, the first case above means 'create this (function) object, call this >decorator, and bind the name to the new object'. So instead of taking an >existing object (with an existing name) and rebinding it to a new name, it >is creating an object, doing something to it and binding it to a name. A >definite deviation from the current 'as' semantics, but understandable. > >However, the second case above is doing something completely different. It >is creating a new object (a class) and binding it to a name. As a side >effect, it is changing the metaclass of the object. The 'as' in this case >has nothing whatsoever to do with binding the object name, but a name in >the object's namespace. > >I suppose you could make the argument that the metaclass has to act as a >decorator (like in the function def above) and set the __metaclass__ >attribute, but that would mean that existing metaclasses couldn't work. It >would also mean you were defining the semantics at an implementation level. > >I'm worried that I'm being too picky here, because I *like* the way the >above reads. I'm just worried about overloading 'as' with too many >essentially unrelated meanings. Well, there's always 'is'... def foop() is staticmethod: class Newstyle is type: Interestingly, this usage is rather similar to Eiffel, which IIRC introduces code suites with 'is', although I think without the modifier. I'm not all that enthused about the metaclass usage, mainly because there's already an okay syntax (__metaclass__) for it. I'd rather that class decorators (if added) were decorators in the same way as function decorators. Why? Because I think that correct, combinable class decorators are probably easier for most people to write than correct, combinable metaclasses, and they are more easily combined than metaclasses are.

4 3

RE: [Python-Dev] PEP 322: Reverse Iteration
by Moore, Paul Nov. 4, 2003

Nov. 4, 2003

From: Guido van Rossum [mailto:guido@python.org] > Such a thing would face zero opposition if it was part of itertools: > itertools.revrange([start, ] stop[, step]) makes total sense to me... I also like Alex's suggestion of itertools.irange([start,] stop[, step][,reverse=False]) I'd rather this than a revrange - that feels over-specialised, whereas an irange with a reverse keyword parameter seems natural. I'd still support this addition to itertools whether or not the reversed() builtin was implemented (although with irange, reversed() loses a lot of its use cases...) Paul.

2 1

reflections on basestring -- and other abstract basetypes
by Alex Martelli Nov. 4, 2003

Nov. 4, 2003

1. Shouldn't class UserString.UserString inherit from basestring? After all, basestring exists specifically in order to encourage typetests of the form isinstance(x, basestring) -- wouldn't it be better if such tests could also catch "user-tweaked strings" derived from UserString ... ? 2. If we do want to encourage such typetest idioms, it might be a good idea to provide some other such abstract basetypes for the purpose. For example, I see quite a few cases of isinstance(x, (int,long,gmpy.mpz)) in my code -- and that, despite the fact that I'm not enamoured of typetesting as a general idea and that I'm quite aware that this kind of check could miss some other kind of user-coded "integeroid number". If there was an abstract basetype, say "baseinteger", from which int and long derived, I'd be happy to tweak gmpy to make mpz subclass it (in 2.4 and later versions of Python only, of course) and allow such typetests to happen more smoothly, faster and with more generality too. 3. And perhaps baseinteger (and float and complex) should all subclass yet another basetype, say "basenumber"? Why not? I admit that right now I have no use cases where I _do_ want to accept complex numbers as well as int, long, float, and gmpy thingies (so, maybe there should be a more specific "basereal" keeping complex out...?), but apart from this detail such an abstract basetype would be similarly useful (in practice I would use it since I do not expect complex in my apps anyway). 4. Furthermore, providing "basenumber" would let user-coded classes "flag" in a simple and direct way "I'm emulating numbers". This might well be useful _to Python itself_... Right now, I'm stuck for an answer to the bug that a user-coded class which exposes __mul__ but not __rmul__ happens to support its instances being multiplied by an integer on the right -- quite surprising to users! The problem is that this behavior is apparently expected, though not documented, when the user-coded class is trying to simulate a _sequence_ rather than a number. So, I can't just take the peculiar "accidental commutativity with integers only" away. IF a user class could flag itself as "numeroid" by inheriting basenumber, THEN the "accidental commutativity" COULD be easily removed at least for such classes. 5. in fact, now that we fill in type descriptor slots bases on user-coded classes' special methods, I suspect this isn't the only such issue. While "flagging" (inheriting one of the abstract basetypes) would be entirely optional for user-coded classes, it would at least provide a way to _explicitly disambiguate_ what it is that the user-coded class IS trying to emulate, if the user wants to. 6. of course, for that to be any use, the various basetypes should not be "ambiguously" multiply inheritable from. Right now, is isnt so...: >>> class x(basestring, int): pass ... >>> isinstance(x(), int) True >>> isinstance(x(), basestring) True ...does anybody see any problem if, in 2.4, we take away the ability to multiply inherit from basestring AND also from another builtin type which does not in turn inherit from basestring...? I have the impression that right now this is working "sort of accidentally", rather than by design. 7. one might of course think of other perhaps-useful abstract basetypes, such as e.g. basesequence or basemapping -- right now the new forthcoming built-in 'reverse' is trying to avoid "accidentally working" on mappings by featuretesting for (e.g.) has_key, but if the user could optionally subclass either of these abstract basetypes (but not both at once, see [6]:-), that might ease reverse's task in some cases. Why, such abstract basetypes might even make operator.isMappingType useful again -- right now, of course: >>> operator.isMappingType([]) True and therefore there isn't much point in that function:-). But I think that points 1-6 may be enough to discuss for the moment (and I brace myself for the flames of the antitypetesters -- why, if I hadn't matured this idea myself I might well be one of the flamers:-) so I have no concrete proposals sub [7] -- yet. <donning suit="asbestos"> ...just a sec... </donning> Ok, ready -- fire away! Alex

4 7

RE: [Python-Dev] reflections on basestring -- and other abstractbasetypes
by Michael Chermside Nov. 4, 2003

Nov. 4, 2003

I (Michael Chermside) wrote: > Great idea... I think there should be single type from which all built-in > integer-like types inherit, and which user-designed types can inherit > if they want to behave like integers. I think that type should be called > "int". Alex replies: > Unfortunately, unless int is made an abstract type, that doesn't help at > all to "type-flag" user-coded types (be they C-coded or Python-coded): > they want to tell "whoever it may concern" that they're intended to be > usable as integers, but not uselessly carry around an instance of int for > the purpose (and need to contort their own layout, if C-coded, for that). Valid point. Of course, we've reduced the use cases to those which want to emulate integers and ALSO don't want the layout of ints. It seems like a small number of situations, but there ARE some, and it IS a valid point. > Abstract basetypes such as basestring are useful only to "flag" types as > (intending to conform to) some concept: they don't carry implementation. Well, yes, but Python strives very hard to not NEED to know what type an object is before operating on it. As long as it supports the operations that are used, it's "good enough". It's an ideal, not a universal rule, and there are pleanty of small exceptions, but to introduce a system of basetypes seems inappropriate. On the other hand, string and unicode need a common base class because they are a special case. Really, there are two things going on... the need to process arbitrary collections of bytes, and the need to process arbitrary collections of characters. The whole thing is thrown into confusion because "string" is used for storing characters, particularly when the characters are expected to be ascii. This is for historical reasons, performance reasons, out of ignorance, because "string" is easier to type and u"" is more annoying... lots of reasons both good and bad. But since lots of string objects contain character data just like unicode objects, we need a type lable for dealing with "character data", and that can't be either "unicode" or "string". I don't see any such issue in numbers (although the int/long flaw is somewhat similar, but that's being healed). > Surely you're not claiming that > Numeric is "abusing operator overloading" by allowing users to code > a+b, a*b, a-b etc where a and b are multi-dimensional arrays? The > ability to use such notation, which is fully natural in the application areas > those users come from, is important to many users. Um... no, I didn't mean to claim that. When I wrote it, I was thinking "okay, you'd only use these operations (sensibly) on something which had an algebra... ie, a number." But that was wrong... matrices have an algebra, but they're NOT numbers. I wrote: > What use cases do you have for "basenumber" (I don't mean > examples of classes that would inherit from basenumber, I mean examples > where that inheritance would make a difference)? Alex responded with actual examples, and I'll have to take the time to read them properly before I can respond meaningfully. (But THANKS for giving specific examples... it always helps me reason about abstract ideas (like "are baseclasses wise for numbers") when I have a few concrete examples to check myself against as I go.) Let this be a warning to me... be careful of getting in an argument with Alex, since he'll swamp me with far more well-reasoned arguments and examples than I have time to _read_, much less respond to. <wink> -- Michael Chermside

2 1

RE: PEP 318 (was Re: [Python-Dev] Re: Guido's Magic Code was: inlinesort option)
by Delaney, Timothy C (Timothy) Nov. 4, 2003

Nov. 4, 2003

> From: Alex Martelli [mailto:aleaxit@yahoo.com] > > BTW, when we do come around to PEP 318, I would suggest the 'as' > clause on a class statement as the best way to specify a metaclass. I just realised what has been bugging me about the idea of def foop() as staticmethod: and it applies equally well to class Newstyle as type: Basically, it completely changes the semantics associated with 'as' in Python - which are to give something a different name (technically, to rebind the object to a different name). OTOH, the first case above means 'create this (function) object, call this decorator, and bind the name to the new object'. So instead of taking an existing object (with an existing name) and rebinding it to a new name, it is creating an object, doing something to it and binding it to a name. A definite deviation from the current 'as' semantics, but understandable. However, the second case above is doing something completely different. It is creating a new object (a class) and binding it to a name. As a side effect, it is changing the metaclass of the object. The 'as' in this case has nothing whatsoever to do with binding the object name, but a name in the object's namespace. I suppose you could make the argument that the metaclass has to act as a decorator (like in the function def above) and set the __metaclass__ attribute, but that would mean that existing metaclasses couldn't work. It would also mean you were defining the semantics at an implementation level. I'm worried that I'm being too picky here, because I *like* the way the above reads. I'm just worried about overloading 'as' with too many essentially unrelated meanings. Tim Delaney

2 1

FW: [Python-Dev] PEP 322: Reverse Iteration
by Raymond Hettinger Nov. 3, 2003

Nov. 3, 2003

> > The pep has been through several rounds of public comment on > > comp.lang.python. As a result, the proposal has evolved away from > > several methods called iter_backwards() and into a simple builtin > > function called reversed(). Other simplifications emerged as well. The > > improved pep is at: > > > > www.python.org/sf/pep-0322.html [Aahz] > Oops? That URL don't work. Drat! http://www.python.org/peps/pep-0322.html Raymond Hettinger

1 0

RE: [Python-Dev] reflections on basestring -- and other abstractbasetypes
by Michael Chermside Nov. 3, 2003

Nov. 3, 2003

Alex muses on basestring: > 2. If we do want to encourage such typetest idioms, it might be a good idea > to provide some other such abstract basetypes for the purpose. [...] > If there was an abstract basetype, say "baseinteger", from which int and > long derived, Great idea... I think there should be single type from which all built-in integer-like types inherit, and which user-designed types can inherit if they want to behave like integers. I think that type should be called "int". Once the int/long distinction is completely gone, this will be quite clean, the only confusion now is that the int/long distinction isn't yet completely hidden. > 4. Furthermore, providing "basenumber" would let user-coded classes "flag" > in a simple and direct way "I'm emulating numbers". Okay, that sounds like it might be useful, at least to those people who work with wierd varieties of numbers. But I can't think how. Normally, I figure that if you overload addition, multiplication, subtraction, and perhaps a few other such operators, then you're trying to emulate numbers (that or you're abusing operator overloading, and I have no real sympathy for you). What use cases do you have for "basenumber" (I don't mean examples of classes that would inherit from basenumber, I mean examples where that inheritance would make a difference)? > IF a user class could flag itself as "numeroid" by inheriting basenumber, > THEN the "accidental commutativity" COULD be easily removed at least > for such classes. Okay, that's one use case. Any others? 'cause I'm coming up blank. > ...does anybody see any problem if, in 2.4, we take away the ability to > multiply inherit from basestring AND also from another builtin type which > does not in turn inherit from basestring...? I do! I personally wouldn't try to create the class "perlnum" which inherits from basestring and also basenumber and which tries to magicaly know which is desired and convert back and forth on demand. But I'm sure *someone* out there is just dying to write such a class. Why prevent them? Not that I'd every USE such a monstrocity, but just don't see the ADVANTAGE in providing the programmer with a straightjacket by typechecking them (at the language level) to prevent uses outside of those envisioned by the language implementers. It sounds decidedly non-pythonic to me. -- Michael Chermside

3 2

Looking for master thesis ideas involving Python
by Chris Ryland Nov. 3, 2003

Nov. 3, 2003

Michael Hudson wrote: > Remove the ob_type field from all PyObjects. Make pymalloc mandatory, > make it use type specific pools and store a pointer to the type object > at the start of each pool. > > So instead of > p->ob_type > it's > *(p&MASK) > > I think having each type in its own pools would also let you lose the > gc_next & gc_prev fields. > > Combined with a non-refcount GC, you could hammer sizeof(PyIntObject) > down to sizeof(long)! Yes, this is a variant of an implementation technique used in early Lisp and Lisp-like language systems with types (e.g., Harvard's EL-1) back in the early 70's (at least--that's when I first encountered it). In those systems, you'd use the "page #" (higher-order bits) of a pointer to reference a type table. Good idea, but perhaps less effective these days where memory isn't quite so dear. (Back then, a large system was a PDP-10 with 256K 36-bit words, or around 1MB.) Cheers! --Chris Ryland / Em Software, Inc. / www.emsoftware.com

2 1