Mailman 3 February 2002 - Python-Dev

RE: [Python-Dev] Meta-reflections
by Moore, Paul 20 Feb '02

20 Feb '02

From: Kevin Jacobs [mailto:jacobs@penguin.theopalgroup.com] > > FWIW, I disagree with this completely. I would expect > > slots with a particular name in derived classes to hide > > the same-named slots in base classes. Whether or not the > > base class slot is available via some sort of "super" > > shenannigans is less relevant. But hiding semantics > > is critical. How do you expect to reliably make it > > possible to derive from a class with slots otherwise? > > Perl gets into this sort of mess with its implementation > > of objects. > > Attributes currently have a flat namespace, and the > construct that I feel is most natural would maintain that > characteristic. e.g.: Oops. Sorry, I got that completely wrong. OK, I side with silent re-use. That's what attributes do. [Meta-meta comment: the rule should be to work "just like" attributes.] So why did you feel the need to separate this point from your point (3)? It gives the impression that this point differs from attributes, in contrast to the things mentioned in (3). > > Agreed - up to a point. I don't see a need for a way > > to distinguish between slots and "normal" attributes, > > personally. But I don't do anything fancy here, so my > > experience isn't very indicative. > > Without a more formal reflection API, the traditional > way to get all normal dictionary attributes is by using > instance.__dict__.keys(). The official, and supported, way is to use dir(). This was hashed out on python-dev at the time. As I understand it, dir() always "worked", and was extended to support slots when they were added. __dict__ clearly only handles dict-based attributes, and so cannot be extended to include slots. The official advice on reflection was therefore modified to point out that dir() and __dict__.keys() were no longer equivalent, and dir() was "the way" to get the full set. (Whether this advice was included into the formal documentation, I couldn't confirm, but it was written down - arguing "if it's not in the documentation, it's not official", is a little naive, given the new and relatively experimental status of the whole area...) > All I'm proposing is that instance.__slotattrs__ (or > possibly instance.__class__.__slotattrs__) returns a > list of objects that reveal the name of all slots in the > instance (including those declared in base classes). Do you have any reason why you would need to get a list of only slots, or only dict-based attributes? Given that I'm arguing that the two should work exactly the same (apart from storage and efficiency details), it seems unreasonable to want to make the distinction (unless you're doing something incestuous and low-level, when you're on your own...) Remember, instance.__dict__['attrname'] is now regarded as incomplete in the face of slots. Again, I point you to the descrintro document, just below the discussion of slots, in the paragraphs starting from "The correct way to get any attribute from self inside __getattribute__ is to call the base class's __getattribute__ method". > The only issue that concerns me is that I am not sure if the > slot to slot name mapping should be fixed. I haven't melted my brain enough to understand the PEPs, but I believe that there are ways of doing all sorts of low-level hacking with descriptors, if you really want to. I don't believe that making this easy for "normal" users is a good thing. [BTW, this reminds me of your point on what is documented. I believe that the PEPs and descrintro count as the canonical documentation of these features. If they haven't been fully migrated into the Python documentation set yet, that's a secondary issue. The PEPs *are* the definition of what was agreed - people had time to comment on the PEPs at the time. And the descrintro document is the current draft of the user-level documentation. You can assume it will end up in the manuals. That's my view...] > EXACTLY! I want to use slots (or slotattrs, or whatever > you call them) to be solely an allocation declaration. > For small objects that you expect to create many, many > instance of, they make a huge difference. In which case, you are giving the impression of wanting large changes, when you really want things to stay as they are (modulo some relatively non-controversial (IMHO) bugs). If you read the descrintro document on slots, you will see that it presents an identical viewpoint. OK, there are some technical restrictions, which will always apply because of the nature of the optimisation, but the intention is clear. And it matches yours... Paul.

4 4

Parser-SIG created
by Andrew Kuchling 20 Feb '02

20 Feb '02

Parser SIG: Selection of a parser for the standard library Description: This SIG is for discussing and comparing several different parser generators in order to assess which one would be worth including to the Python standard library. Deliverables (in roughly this order): 1) A list of requirements for a parser generator suitable for inclusion. 2) If no parser meets those requirements, the SIG might work to enhance one or more parsers until the requirements are met. (It would be nice if this step became a null operation; otherwise we might fall prey to creeping scope.) 3) A recommendation for a parser to include, along with a patch against the Python CVS tree. The BDFL can then ignore or follow the recommendation and patch as he sees fit. Martin von Loewis presented a paper at Python10 comparing several different parser generators in order to assess which one would be worth adding to the standard library; it will likely serve as the starting point for discussion. Jonathan Riehl suggested creating a Parser SIG, and I offered to champion it. The SIG will aim to complete its task in time for Python 2.3. No schedule for 2.3 has been officially announced yet, but probably the SIG will have to complete its mission by May or June 2002. To join the SIG mailing list, use Mailman at: http://mail.python.org/mailman/listinfo/parser-sig/ --amk (www.amk.ca) I can see you've been doing the TARDIS up a bit. I don't like it. -- The second Doctor, in "The Three Doctors"

1 0

Global variable access schemes
by Ka-Ping Yee 20 Feb '02

20 Feb '02

I've added diagrams for Guido's more recent proposal, and summarized everything on a web page: http://lfw.org/python/globals.html Check out http://lfw.org/python/guido2a.gif and http://lfw.org/python/guido2b.gif. About Guido2: - I renamed some things -- globals_vector is a structure, not a vector, so i put it in md_cache and used the prefix mc_ for its fields. - When you del a module variable, do you just go through all of mc_names to find the entry to invalidate? (I suppose if you sort mc_names you can binary search.) - It should be possible to add entries in the cache for attributes in other modules, too, right? If we assume that varibles don't get deleted often, it should pay off. Haven't heard anything from anybody about this topic in a while. Has anyone been thinking about it? -- ?!ng

1 0

Meta-reflections
by Kevin Jacobs 20 Feb '02

20 Feb '02

Hello all, I've been meta-reflecting a lot lately: reflecting on reflection. My recent post on __slots__ not being picklable (and the resounding lack of response to it) inspired me to try my hand at channeling Guido and reverse- engineer some of the design decisions that went into the new-style class system. Unfortunately, the more I dug into the code, the more philosophical my questions became. So, I've written up some questions that help lay bare some of basic design questions that I've been asking myself and that you should be aware of. While there are several subtle issues I could raise, I do want some feedback on some simple and fundamental ones first. Please don't disqualify yourself from commenting because you haven't read the code or used the new features yet. I've written my examples assuming only a basic and cursor understanding of the new Python 2.2 features. [In this discussion I am only going to talk about native Python classes, not C-extension or native Python types (e.g., ints, lists, tuples, strings, cStringIO, etc.)] 1) Should class instances explicitly/directly know all of their attributes? Before Python 2.2, all object instances contained a __dict__ attribute that mapped attribute names to their values. This made pickling and some other reflection tasks fairly easy. e.g.: class Foo: def __init__(self): self.a = 1 self.b = 2 class Bar(Foo): def __init__(self): Foo.__init__(self) self.c = 3 bar = Bar() print bar.__dict__ > {'a': 1, 'c': 3, 'b': 2} I am aware that there are situations where this simple case does not hold (e.g., when implementing __setattr__ or __getattr__), but let's ignore those for now. Rather, I will concentrate on how this classical Python idiom interacts with the new slots mechanism. Here is the above example using slots: e.g.: class Foo(object): __slots__ = ['a','b'] def __init__(self): self.a = 1 self.b = 2 class Bar(Foo): __slots__ = ['c'] def __init__(self): Foo.__init__(self) self.c = 3 bar = Bar() print bar.__dict__ > AttributeError: 'Bar' object has no attribute '__dict__' We can see that the class instance 'bar' has no __dict__ attribute. This is because the slots mechanism allocates space for attribute storage directly inside the object, and thus does not use (or need) a per-object instance dictionary to store attributes. Of course, it is possible to request that a per-instance dictionary by inheriting from a new-style class that does not list any slots. e.g. continuing from above: class Baz(Bar): def __init__(self): Bar.__init__(self) self.d = 4 self.e = 5 baz = Baz() print baz.__dict__ > {'e': 5, 'd': 4} We have now created a class that has __dict__, but it only contains the attributes not stored in slots! So, should class instances explicitly know their attributes? Or more precisely, should class instances always have a __dict__ attribute that contains their attributes? Don't worry, this does not mean that we cannot also have slots, though it does have some other implications. Keep reading... 2) Should attribute access follow the same resolution order rules as methods? class Foo(object): __slots__ = ['a'] self.a def __init__(self): self.a = 1 class Bar(Foo): __slots__ = ('a',) def __init__(self): Foo.__init__(self) self.a = 2 bar = Bar() print bar.a > 2 print super(Bar,bar).a # this doesn't actually work > 2 or 1? Don't worry -- this isn't a proposal and no, this doesn't actually work. However, the current implementation only narrowly escapes this trap: print bar.__class__.a.__get__(bar) > 2 print bar.__class__.__base__.a.__get__(bar) > AttributeError: a Ok, let me explain what just happened. Slots are implemented via the new descriptor interface. In short, descriptor objects are properties and support __get__ and __set__ methods. The slot descriptors are told the offset within an object instance the PyObject* lives and proxy operations for them. So getting and setting slots involves: # print bar.a a_descr = bar.__class__.a print a_descr.__set__(bar) # bar.a = 1 a_descr = bar.__class__.a a_descr.__set__(bar, 1) So, above we get an attribute error when trying to access the 'a' slot from Bar since it was never initialized. However, with a little ugliness you can do the following: # Get the descriptors for Foo.a and Bar.a a_foo_descr = bar.__class__.__base__.a a_bar_descr = bar.__class__.a a_foo_descr.__set__(bar,1) a_bar_descr.__set__(bar,2) print bar.a > 2 print a_foo_descr.__get__(bar) > 1 print a_bar_descr.__get__(bar) > 2 In other words, the namespace for slots is not really flat, although there is no simple way to access these hidden attributes since method resolution order rules are not invoked by default. 3) Should __slots__ be immutable? The __slots__ attribute of a new-style class lists all of the slots defined by that class. It is represented as whatever sequence type what given when the object was declared: print Foo.__slots__ > ['a'] print Bar.__slots__ > ('a',) This allows us to do things like: Foo.__slots__.append('b') foo = Foo() foo.b = 42 > AttributeError: 'Foo' object has no attribute 'b' So modifying the slots does not do what one may expect. This is because slot descriptors and the space for slots are only allocated when the classes are created (i.e., when they are inherited from 'object', or from an object that descends from 'object'). 4) Should __slots__ be flat? bar.__slots__ only lists the slots specifically requested in bar, even though it inherits from 'foo', which has its own slots. Which would be the preferable behavior? class Foo(object): __slots__ = ('a','b') class Bar(object): __slots__ = ('c','d') print Bar.__slots__ > ('c','d') # current behavior or > ('a','b','c','d') # alternate behavior Clearly, this issue goes back to the ideas addressed in question 1. If slot descriptors are not stored in a per-instance dictionary, then the assumptions on how to do object reflection must change. However, which version of the following code do you prefer to print all attributes of a given object: Old style or if descriptors are stored in obj.__dict__: if hasattr(obj,'__dict__'): print ''.join([ '%s=%s' % nameval for nameval in obj.__dict__ ]) Currently in Python 2.2 (and still not quite correct): def print_slot_attrs(obj,cls=None): if not cls: cls = obj.__class__ for name,obj in cls.__dict__.items() if str(type(obj)) == "<type 'member_descriptor'>": if hasattr(obj, name): print "%s=%s" % (name,getattr(obj, name)) for base in cls.__bases__: print_slot_attrs(obj,base) if hasattr(obj,'__dict__'): print [ '%s=%s' % nameval for nameval in obj.__dict__ ] print_slot_attrs(obj) Flat and immutable slot namespace: a = [ '%s=%s' % nameval for nameval in obj.__dict__ ] a += [ '%s=%s' % (name,val) for name,val in obj.__slots__ \ if hasattr(obj, name) ] print ''.join(a) So, which one of these do you want to support or explain to a new user? Thanks, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs(a)theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com

6 13

2.2.1 issues
by Michael Hudson 19 Feb '02

19 Feb '02

Well, we have the first 2.2 bugfix that isn't a no-brainer to port to 2.2.1. This is to do with the [ #495401 ] Build troubles: --with-pymalloc bug. As far as understand it, there were two problems. 1) with wide unicode characters, some function in unicodeobject.c to do with interpreting escape codes could write into memory it didn't own. 2) something to do with the handling of "unpaired high surrogates" in the utf-8 codec. Were these problems related? I think they got fixed at the same time, but I may have gotten confused. 1) shouldn't be too much of an issue to get into 2.2.1 (there was some contention about which fix performed better, but for 2.2.1 I don't care too much). 2) is more troublesome, because to fix it properly breaks .pycs, in turn because marshal uses the utf-8 codec to store unicode string constants, and this is a no-no according to PEP 6. Is it possible to worm around 2) by reconstructing valid strings from the bad marshal data, or has information been lost? How severe is the bug? Maybe it would be best to leave it unfixed in 2.2.1. Basically, I guess I'm saying I'm too much of a unicode dunce to understand all the issues involved in fixing this problems in 2.2, so as unofficial bugfix-porter, I'd like someone else (Marc? Martin?) to port these particular fixes. If the mechanics of fiddling with the branch is too much, sending me patches is fine. Cheers, M. -- This is the fixed point problem again; since all some implementors do is implement the compiler and libraries for compiler writing, the language becomes good at writing compilers and not much else! -- Brian Rogoff, comp.lang.functional

5 8

Re: [Python-Dev] Rattlesnake progress
by Michael Hudson 19 Feb '02

19 Feb '02

Daniel Berlin <dan(a)dberlin.org> writes: > On Tuesday, February 19, 2002, at 11:01 AM, Kevin Jacobs wrote: > > > On Tue, 19 Feb 2002, Daniel Berlin wrote: > >> On Tuesday, February 19, 2002, at 09:51 AM, Neil Schemenauer wrote: > >> > >>> Daniel Berlin wrote: > >>>> When you get to optimizations, you want Advanced Compiler Design and > >>>> Implementation by Muchnick. > >>> > >>> Right now I'm not planning to do any optimizations (except perhaps > >>> limiting the number of registers used). > >>> > >> This is, of course, a tricky optimization to do. > >> Limiting registers used involves splitting live ranges at the right > >> places, etc. > > > > Why limit the number of registers at all? So long as they fit in L1 > > cache > > you are golden. > > Err, what makes you think this? > The largest problem on architectures like x86 is the number of registers. > You end up with about 4 usable registers. (hardware register renaming > only helps eliminate instruction dependencies, before someone mentions > it). > Performance quickly drops when you start spilling registers to the stack. I think you misunderstand what Rattlesnake is; AIUI it is (or will/intends to be) a register based VM for Python replacing the current stack based VM -- I think gcc still gets to decide which x86 registers to use... Cheers, M. -- ARTHUR: The ravenours bugblatter beast of Traal ... is it safe? FORD: Oh yes, it's perfectly safe ... it's just us who are in trouble. -- The Hitch-Hikers Guide to the Galaxy, Episode 6

1 0

asyncore.poll behaviour
by Kjetil Jacobsen 19 Feb '02

19 Feb '02

hello, in python2.2 the semantics for asyncore.poll (which uses the select() system call) is different than for asyncore.poll3 (which uses the poll() system call) when an EINTR exception occurs. in asyncore.poll3, the pollset is correctly reset to an empty list, but in asyncore.poll this is not done, which in turn causes a lot of strange things to happen when an EINTR occurs (spurious handler invocations and so on). i've tried to upload the patch to sourceforge, but the patch manager has not responded for me the last couple of days so i'm sending it here instead. the fix is a simple one-liner which makes the semantics of asyncore.poll and asyncore.poll3 similar: *** /usr/local/lib/python2.2/asyncore.py Wed Jan 30 15:51:00 2002 --- asyncore.py Wed Jan 30 16:19:28 2002 *************** *** 80,85 **** --- 80,86 ---- except select.error, err: if err[0] != EINTR: raise + r, w, e = [], [], [] if DEBUG: print r,w,e btw, the asyncore.poll2 function does not seems to have either the behaviour of asyncore.poll or asyncore.poll3 with respect to handling of EINTR. perhaps asyncore.poll2 should be removed altogether or just remapped to asyncore.poll3? regards, - kjetil

1 0

Re: [Python-Dev] PEP needed? Introducing Tcl objects
by Michael Hudson 19 Feb '02

19 Feb '02

martin(a)v.loewis.de (Martin v. Loewis) writes: [schniiiip] > ought to continously increase the counter in the label (once a > second), but doesn't, atleast not on Linux, using Tcl 8.3.3. In the > strace output, it appears that it first does a select call with a > timeout, but that is followed by one without time limit before > Tcl_DoOneEvent returns. > > Jeff, any ideas as to why this is happening? Well, at least this one is easy. From the link Jeff posted: Information provided to Tcl_SetMaxBlockTime is only used for the next call to Tcl_WaitForEvent; it is discarded after Tcl_WaitForEvent returns. The next time an event wait is done each of the event sources' setup procedures will be called again, and they can specify new information for that event wait. so you need to move the Tcl_SetMaxBlockTime inside the while loop. It certainly looks to this novice as if 8.3 provides enough hooks to do what we want, but... Cheers, M. -- Like most people, I don't always agree with the BDFL (especially when he wants to change things I've just written about in very large books), ... -- Mark Lutz, http://python.oreilly.com/news/python_0501.html

1 0

We would like to possibly get some info on your company
by josh.winters＠webstream.net 18 Feb '02

18 Feb '02

Hello, We would like to possibly get some info on your company in an effort to explore the ways that we might be able to work together. We may be able to save you money and offer you the benefits of our reseller program. We have been developing and hosting web sites since 1997. We offer design, programming, hosting and webcasting and videoconferencing. We support Linux, NT and AS400. Please forward this to the proper party, or respond to the address below. You can also visit our web site at http://webstream.net for more information on our services. If by e-mail: josh.winters(a)webstream.net If by mail: WebStream Internet Solutions Outsourcing/Purchasing 2200 W.Commercial Blvd. Suite 204 Ft. Lauderdale, FL 33309 USA Thank you very much. Sincerely, Josh Winters josh.winters(a)webstream.net http://webstream.net Design * Programming * Hosting * WebCasting * Since 1997

1 0

Re: comp.lang.python in English? (was Re: Why Python is like BASIC)
by john coppola 18 Feb '02

18 Feb '02

--- Fran�ois Pinard <pinard(a)iro.umontreal.ca> wrote: > > Exactement. Bravo! Bien dit! :-) I agree with Fran�ois and I'm english speaking. -john __________________________________________________ Do You Yahoo!? Yahoo! Sports - Coverage of the 2002 Olympic Games http://sports.yahoo.com

2 2