Tim Peters wrote:

[Ping]

...

dict[key] = 1 if key in dict: ... for key in dict: ...

[Guido]

...

No chance of a time-machine escape, but I *can* say that I agree that Ping's proposal makes a lot of sense. This is a reversal of my previous opinion on this matter. (Take note -- those don't happen very often! :-)

First to submit a working patch gets a free copy of 2.1a2 and subsequent releases,

Thomas since submitted a patch to do the "if key in dict" part (which I reviewed and accepted, pending resolution of doc issues).

It does not do the "for key in dict" part. It's not entirely clear whether you intended to approve that part too (I've simplified away many layers of quoting in the above <wink>). In any case, nobody is working on that part.

WRT that part, Ping produced some stats in:

http://mail.python.org/pipermail/python-dev/2001-January/012106.html

...

How often do you write 'dict.has_key(x)'? (std lib says: 206) How often do you write 'for x in dict.keys()'? (std lib says: 49)

How often do you write 'x in dict.values()'? (std lib says: 0) How often do you write 'for x in dict.values()'? (std lib says: 3)

However, he did not report on occurrences of

for k, v in dict.items()

I'm not clear exactly which files he examined in the above, or how the counts were obtained. So I don't know how this compares: I counted 188 instances of the string ".items(" in 122 .py files, under the dist/ portion of current CVS. A number of those were assignment and return stmts, others were dict.items() in an arglist, and at least one was in a comment. After weeding those out, I was left with 153 legit "for" loops iterating over x.items(). In all:

153 iterating over x.items() 118 " over x.keys() 17 " over x.values()

So I conclude that iterating over .values() is significantly more common than iterating over .keys().

On c.l.py about an hour ago, Thomas complained that two (out of two) of his coworkers guessed wrong about what

for x in dict:

would do, but didn't say what they *did* think it would do. Since Thomas doesn't work with idiots, I'm guessing they *didn't* guess it would iterate over either values or the lines of a freshly-opened file named "dict" <wink>.

So if you did intend to approve "for x in dict" iterating over dict.keys(), maybe you want to call me out on that "approval post" I forged under your name.

Dictionaries are not sequences. I wonder what order a user of for k,v in dict: (or whatever other of this proposal you choose) will expect... Please also take into account that dictionaries are *mutable* and their internal state is not defined to e.g. not change due to lookups (take the string optimization for example...), so exposing PyDict_Next() in any to Python will cause trouble. In the end, you will need to create a list or tuple to iterate over one way or another, so why bother overloading for-loops w/r to dictionaries ? -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

On Mon, Jan 29, 2001 at 09:48:22AM -0500, Guido van Rossum wrote:

How about:

for key in dict: ... # ... over keys

for key:value in dict: ... # ... over items

This is syntactically unambiguous (a colon is currently illegal in that position).

I won't comment on the syntax right now, I need to look at it for a while first :-) However, what about MAL's point about dict ordering, internally ? Wouldn't FOR_LOOP be forced to generate a list of keys anyway, to avoid skipping keys ? I know currently the dict implementation doesn't do any reordering except during adds/deletes, but there is nothing in the language ref that supports that -- it's an implementation detail. Would we make a future enhancement where (some form of) gc would 'clean up' large dictionaries impossible ? -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

Guido van Rossum wrote:

...

Dictionaries are not sequences. I wonder what order a user of for k,v in dict: (or whatever other of this proposal you choose) will expect...

The same order that for k,v in dict.items() will yield, of course.

And then people find out that the order has some sorting properties and start to use it... "how to sort a dictionary?" comes up again, every now and then.

...

Please also take into account that dictionaries are *mutable* and their internal state is not defined to e.g. not change due to lookups (take the string optimization for example...), so exposing PyDict_Next() in any to Python will cause trouble. In the end, you will need to create a list or tuple to iterate over one way or another, so why bother overloading for-loops w/r to dictionaries ?

Actually, I was going to propose to play dangerously here: the

for k:v in dict: ...

syntax I proposed in my previous message should indeed expose PyDict_Next(). It should be a big speed-up, and I'm expecting (though don't have much proof) that most loops over dicts don't mutate the dict.

Maybe we could add a flag to the dict that issues an error when a new key is inserted during such a for loop? (I don't think the key order can be affected when a key is *deleted*.)

You mean: mark it read-only ? That would be a "nice to have" property for a lot of mutable types indeed -- sort of like low-level locks. This would be another candidate for an object flag (much like the one Fred wants to introduce for weak referenced objects). -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

...

...

Dictionaries are not sequences. I wonder what order a user of for k,v in dict: (or whatever other of this proposal you choose) will expect...

The same order that for k,v in dict.items() will yield, of course.

And then people find out that the order has some sorting properties and start to use it... "how to sort a dictionary?" comes up again, every now and then.

I don't understand why you bring this up. We're not revealing anything new here, the random order of dict items has always been part of the language. The answer to "how to sort a dict" should be "copy it into a list and sort that." Or am I missing something?

...

...

Please also take into account that dictionaries are *mutable* and their internal state is not defined to e.g. not change due to lookups (take the string optimization for example...), so exposing PyDict_Next() in any to Python will cause trouble. In the end, you will need to create a list or tuple to iterate over one way or another, so why bother overloading for-loops w/r to dictionaries ?

Actually, I was going to propose to play dangerously here: the

for k:v in dict: ...

syntax I proposed in my previous message should indeed expose PyDict_Next(). It should be a big speed-up, and I'm expecting (though don't have much proof) that most loops over dicts don't mutate the dict.

Maybe we could add a flag to the dict that issues an error when a new key is inserted during such a for loop? (I don't think the key order can be affected when a key is *deleted*.)

You mean: mark it read-only ? That would be a "nice to have" property for a lot of mutable types indeed -- sort of like low-level locks. This would be another candidate for an object flag (much like the one Fred wants to introduce for weak referenced objects).

Yes. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum :

...

...

Maybe we could add a flag to the dict that issues an error when a new key is inserted during such a for loop? (I don't think the key order can be affected when a key is *deleted*.)

You mean: mark it read-only ? That would be a "nice to have" property for a lot of mutable types indeed -- sort of like low-level locks. This would be another candidate for an object flag (much like the one Fred wants to introduce for weak referenced objects).

Yes.

For different reasons, I'd like to be able to set a constant flag on a object instance. Simple semantics: if you try to assign to a member or method, it throws an exception. Application? I have a large Python program that goes to a lot of effort to build elaborate context structures in core. It would be nice to know they can't be even inadvertently trashed without throwing an exception I can watch for. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> No one is bound to obey an unconstitutional law and no courts are bound to enforce it. -- 16 Am. Jur. Sec. 177 late 2d, Sec 256

[Guido]

I did a less sophisticated count but come to the same conclusion: iterations over items() are (somewhat) more common than over keys(), and values() are 1-2 orders of magnitude less common. My numbers:

$ cd python/src/Lib $ grep 'for .*items():' *.py | wc -l 47 $ grep 'for .*keys():' *.py | wc -l 43 $ grep 'for .*values():' *.py | wc -l 2

I like my larger sample and anal methodology better <wink>. A closer look showed that it may have been unduly biased by the mass of files in Lib/encodings/, where encoding_map = {} for k,v in decoding_map.items(): encoding_map[v] = k is at the end of most files (btw, MAL, that's the answer to your question: people would expect "the same" ordering you expected there, i.e. none in particular).

... I don't much value to the readability argument: typically, one will write "for key in dict" or "for name in dict" and then it's obvious what is meant.

Well, "fiddlesticks" comes to mind <0.9 wink>. If I've got a dict mapping phone numbers to names, "for name in dict" is dead backwards. for vevent in keydefs.keys(): for x in self.subdirs.keys(): for name in lsumdict.keys(): for locale in self.descriptions.keys(): for name in attrs.keys(): for func in other.top_level.keys(): for func in target.keys(): for i in u2.keys(): for s in d.keys(): for url in self.bad.keys(): are other cases in the CVS tree where I don't think the name makes it obvious in the absence of ".keys()". But I don't personally give any weight to whether people can guess what something does at first glance. My rule is that it doesn't matter, provided it's (a) easy to learn; and (especially), (b) hard to *forget* once you've learned it. A classic example is Python's "points between elements" treatment of slice indices: few people guess right what that does at first glance, but once they "get it" they're delighted and rarely mess up again. And I think this is "like that".

... But here's my dilemma. "if (k, v) in dict" is clearly useless (nobody has even asked me for a has_item() method).

Yup.

I can live with "x in list" checking the values and "x in dict" checking the keys. But I can *not* live with "x in dict" equivalent to "dict.has_key(x)" if "for x in dict" would mean "for x in dict.items()".

That's why I brought it up -- it's not entirely clear what's to be done here.

I also think that defining "x in dict" but not "for x in dict" will be confusing.

So we need to think more.

The hoped-for next step indeed.

How about:

for key in dict: ... # ... over keys

for key:value in dict: ... # ... over items

This is syntactically unambiguous (a colon is currently illegal in that position).

Cool! Can we resist adding if key:value in dict for "parallelism"? (I know I can ...) 2/3rd of these are marginally more attractive: for key: in dict: # over dict.keys() for :value in dict: # over dict.values() for : in dict: # a delay loop

This also suggests:

for index:value in list: ... # ... over zip(range(len(list), list)

while doesn't strike me as bad or ugly, and would fulfill my brother's dearest wish.

You mean besides the one that you fry in hell for not adding "for ... indexing"? Ya, probably.

(And why didn't we think of this before?)

Best guess: we were focused exclusively on sequences, and a colon just didn't suggest itself in that context. Second-best guess: having finally approved one of these gimmicks, you finally got desperate enough to make it work <wink>. ponderingly y'rs - tim

This is all PEP material now. Tim, do you want to own the PEP? It seems just up your alley!

Cool! Can we resist adding

if key:value in dict

for "parallelism"? (I know I can ...)

That's easy to resist because, unlike ``for key:value in dict'', it's not unambiguous: ``if key:value in dict'' is already legal syntax currently, with 'key' as the condition and 'value in dict' as the (not particularly useful) body of the if statement.

...

(And why didn't we think of this before?)

Best guess: we were focused exclusively on sequences, and a colon just didn't suggest itself in that context. Second-best guess: having finally approved one of these gimmicks, you finally got desperate enough to make it work <wink>.

I'm certainly more comfortable with just ``for key in dict'' than with the whole slow of extensions using colons. But, again, that's for the PEP to fight over. --Guido van Rossum (home page: http://www.python.org/~guido/)

[ESR]

There's not much I miss from C these days, but one thing I wish Python had is a more general for-loop. The C semantics that let you have any initialization, any termination test, and any iteration you like are rather cool.

Yes, I realize that

for (<init>; <test>; <step>) {<body>}

can be simulated with:

<init> while 1: if <test>: break <body>

Still, having them spatially grouped the way a C for does it is nice. Makes it easier to see invariants, I think.

Hm, I've seen too many ugly C for loops to have much appreciation for it. I can recognize and appreciate the few common forms that clearly iterate over an array; most other forms look rather contorted to me. Check out the Python C sources; if you find anything more complicated than ``for (i = n; i > 0; i--)'' I probably didn't write it. :-) Common abominations include: - writing a while loop as for(;<test>;) - putting arbitrary initialization code in <init> - having an empty condition, so the <step> becomes an arbitraty extension of the body that's written out-of-sequence --Guido van Rossum (home page: http://www.python.org/~guido/)

[MAL]

I just wanted to hint at a problem which iterating over items in an unordered set can cause. Especially new Python users will find it confusing that the order of the items in an iteration can change from one run to the next.

Do they find "for k, v in dict.items()" confusing now? Would be the same.

... What we really want is iterators for dictionaries, so why not implement these instead of tweaking for-loops.

Seems an unrelated topic: would "iterators for dictionaries" solve the supposed problem with iteration order?

If you are looking for speedups w/r to for-loops, applying a different indexing technique in for-loops would go a lot further and provide better performance not only to dictionary loops, but also to other sequences.

I have made some good experience with a special counter object (sort of like a mutable integer) which is used instead of the iteration index integer in the current implementation.

Please quantify, if possible. My belief (based on past experiments) is that in loops fancier than for i in range(n): pass the loop overhead quickly falls into the noise even now.

Using an iterator object instead of the integer + __getitem__ call machinery would allow more flexibility for all kinds of sequences or containers. ...

This is yet another abrupt change of topic, yes <0.9 wink>? I agree a new iteration *protocol* could have major attractions.

Guido van Rossum :

Common abominations include:

- writing a while loop as for(;<test>;)

Agreed. Bletch.

- putting arbitrary initialization code in <init>

Not sure what's "arbitrary", unless you mean unrelated to the iteration variable.

- having an empty condition, so the <step> becomes an arbitraty extension of the body that's written out-of-sequence

Again agreed. Double bletch. I guess my archetype of the cute C for-loop is the idiom for pointer-list traversal: struct foo {int data; struct foo *next;} *ptr, *head; for (ptr = head; *ptr; ptr = ptr->next) do_something_with(ptr->data) This is elegant. It separates the logic for list traversal from the operation on the list element. Not the highest on my list of wants -- I'd sooner have ?: back. I submitted a patch for that once, and the discussion sort of died. Were you dead det against it, or should I revive this proposal? -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a> "The bearing of arms is the essential medium through which the individual asserts both his social power and his participation in politics as a responsible moral being..." -- J.G.A. Pocock, describing the beliefs of the founders of the U.S.

...

- putting arbitrary initialization code in <init>

Not sure what's "arbitrary", unless you mean unrelated to the iteration variable.

Yes, that.

I guess my archetype of the cute C for-loop is the idiom for pointer-list traversal:

struct foo {int data; struct foo *next;} *ptr, *head;

for (ptr = head; *ptr; ptr = ptr->next) do_something_with(ptr->data)

This is elegant. It separates the logic for list traversal from the operation on the list element.

And it rarely happens in Python, because sequences are rarely represented as linked lists.

Not the highest on my list of wants -- I'd sooner have ?: back. I submitted a patch for that once, and the discussion sort of died. Were you dead det against it, or should I revive this proposal?

Not dead set against something like it, but dead set against the ?: syntax because then : becomes too overloaded for the human reader, e.g.: if foo ? bar : bletch : spam = eggs If you want to revive this, I strongly suggest writing a PEP first before posting here. --Guido van Rossum (home page: http://www.python.org/~guido/)

...

- How to spell it? x.freeze()? x.readonly()?

I like "freeze", it'a a clear imperative where "readonly()" sounds like a test (e.g. "is this readonly()?")

Agreed.

...

- Should we support something like this for instances too? Sometimes it might be cool to be able to freeze changing attribute values...

Moshe Zadka sent me a hack that handles instances: [...]

OK, so no special support needed there.

...

- Should this reversible? I.e. should there be an x.unfreeze()?

I gave this some thought earlier today. There are advantages to either way. Making freeze a one-way operation would make it possible to use freezing to get certain kinds of security and integrity guarantees that you can't have if freezing is reversible.

Fortunately, there's a semantics that captures both. If we allow freeze to take an optional key argument, and require that an unfreeze call must supply the same key or fail, we get both worlds. We can even one-way-hash the keys so they don't have to be stored in the bytecode.

Want to lock a structure permanently? Pick a random long key. Freeze with it. Then throw that key away...

Way too cute. My suggestion freeze(0) freezes forever, freeze(1) can be unfrozen. --Guido van Rossum (home page: http://www.python.org/~guido/)

Note that even adding a "frozen" flag would add 4 bytes to every freezable object on most machines. That's why I'd rather .freeze() replace the type pointer and .unfreeze() restore it. No time or space overhead; no cluttering up the normal-case (i.e., unfrozen) type implementations with new tests.

...

...

...

...

"ESR" == Eric S Raymond writes:

ESR> Fortunately, there's a semantics that captures both. If we ESR> allow freeze to take an optional key argument, and require ESR> that an unfreeze call must supply the same key or fail, we ESR> get both worlds. We can even one-way-hash the keys so they ESR> don't have to be stored in the bytecode. ESR> Want to lock a structure permanently? Pick a random long ESR> key. Freeze with it. Then throw that key away... Clever!

On Mon, Jan 29, 2001 at 09:34:01PM -0500, Eric S. Raymond wrote:

I guess my archetype of the cute C for-loop is the idiom for pointer-list traversal:

struct foo {int data; struct foo *next;} *ptr, *head;

for (ptr = head; *ptr; ptr = ptr->next) do_something_with(ptr->data)

Note two things: in Python, you would use a list, so 'for x i list' does exactly what you want here ;) And if you really need it, you could use iterators for exactly this (once we have them, of course): you are inventing a new storage type. Quite common in C, since the only one it has is useless for anything other than strings<wink>, but not so common in Python.

Not the highest on my list of wants -- I'd sooner have ?: back. I submitted a patch for that once, and the discussion sort of died. Were you dead det against it, or should I revive this proposal?

Triple blech. Guido will never go for it! (There, increased your chance of getting it approved! :) Seriously though, I wouldn't like it much, it's too cryptic a syntax. I notice I use it less and less in C, too. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

...

for key: in dict: # over dict.keys() for :value in dict: # over dict.values() for : in dict: # a delay loop

[Thomas Wouters]

Wot's the last one supposed to do ? 'for unused_var in range(len(dict)):' ?

Well, as the preceding line said in the original:

...

2/3rd of these are marginally more attractive [than "if key:value in dict"]:

I think you've guessed which 2/3 those are <wink>. I don't see that the last line has any visible semantics whatsoever, so Python can do whatever it likes, provided it doesn't do anything visible. You still hang out on c.l.py! So you gotta know that if something of the form x:y is suggested, people will line up to suggest meanings for the 3 obvious variations, along with x::y and x:-:y and x lambda y too <0.9 wink>.

Guido van Rossum wrote:

...

Yes, this is a good thing. Easy to do on lists and dicts. Questions:

- How to spell it? x.freeze()? x.readonly()?

Ping:

I'm not so sure. There seem to be many issues here. More questions:

What's the difference between a frozen list and a tuple?

A frozen list can be unfrozen (maybe)?

Is a frozen list hashable?

Yes -- that's what started this thread (using dicts as dict keys, actually).

...

- Should this reversible? I.e. should there be an x.unfreeze()?

What if two threads lock and then unlock the same structure?

That's up to the threads -- it's no different that other concurrent access.

...

- Should we support something like this for instances too? Sometimes it might be cool to be able to freeze changing attribute values...

If you do this, i bet people will immediately want to freeze individual attributes. Some might be confused by

a.x = [1, 2, 3] lock(a.x) # intend to lock the attribute, not the list a.x = 3 # hey, why is this allowed?

That's a matter of API. I wouldn't make this a built-in, but rather a method on freezable objects (please don't call it lock()!).

What does locking an extension object do?

What does adding 1 to an extension object do?

What happens when you lock an object that implements list or dict semantics? Do we care that locking a UserList accomplishes nothing?

Who says it doesn't?

Should unfreeze/unlock() be disallowed in restricted mode?

I don't see why not. --Guido van Rossum (home page: http://www.python.org/~guido/)

[Tim Peters on adding yet more syntatic sugar]

Available time is finite, and this isn't at the top of the list of things I'd like to see (resuming the discussion of generators + coroutines + iteration protocol comes to mind first).

What's the chances of getting generators into 2.2? The implementation should not be hard. Didn't Steven Majewski have something years ago? Why do we always get sidetracked on trying to figure out how to do coroutines and continuations? Generators would add real power to the language and are simple enough that most users could benefit from them. Also, it should be possible to design an interface that does not preclude the addition of coroutines or continuations later. I'm not volunteering to champion the cause just yet. I just want to know if there is some issue I'm missing. Neil

On Tue, Jan 30, 2001 at 05:34:10PM +0000, Finn Bock wrote:

...

Recall that a similar trick is played during list.sort(), replacing the list's type pointer for the duration (to point to an internal "immutable list" type, same as the list type except the "dangerous" slots point to a function that raises an "immutable list" TypeError). Then no runtime expense is incurred for regular lists to keep checking flags. I thought of this as an elegant use for switching types at runtime; you may still be appalled by it, though!

Changing the type of a type? Yuck!

No, the typeobject itself isn't changed -- that would freeze *all* dicts/lists/whatever, not just the one we want. We'd be changing the type of an object (or 'type instance', if you want, but not "type 'instance'"), not the type of a type.

I might very likely be reading the CPython sources wrongly, but it seems this trick will cause an BadInternalCall if some other C extension are trying to modify a list while it is freezed by the type switching trick. I imagine this would happen if the extension called:

PyList_SetItem(myList, 0, aValue);

Only if PyList_SetItem refuses to handle 'frozen' lists. In my eyes, 'frozen' lists should still pass PyList_Check(), but also PyList_Frozen() (or whatever), and methods/operations that modify the listobject would have to check if the list is frozen, and raise an appropriate error if so. This might throw 'unexpected' errors, but only in situations that can't happen right now! -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

[Guido]

This is all PEP material now.

Yup.

Tim, do you want to own the PEP?

Not really. Available time is finite, and this isn't at the top of the list of things I'd like to see (resuming the discussion of generators + coroutines + iteration protocol comes to mind first).

...

Cool! Can we resist adding

if key:value in dict

for "parallelism"? (I know I can ...)

That's easy to resist because, unlike ``for key:value in dict'', it's not unambiguous:

But if (key:value) in dict is. Just trying to help whoever *does* want the PEP <wink>.

... I'm certainly more comfortable with just ``for key in dict'' than with the whole slow of extensions using colons.

What about just the for key:value in dict for index:value in sequence extensions? The degenerate forms (omitting x or y or both in x:y) are mechanical variations so are likely to get raised.

But, again, that's for the PEP to fight over.

PEPs are easier if you Pronounce on things you hate early so that those can get recorded in the "BDFL Pronouncements" section without further ado. whatever-this-may-look-like-it's-not-a-pep-discussion<wink>-ly y'rs - tim

Not really. Available time is finite, and this isn't at the top of the list of things I'd like to see (resuming the discussion of generators + coroutines + iteration protocol comes to mind first).

OK, get going on that one then!

...

...

Cool! Can we resist adding

if key:value in dict

for "parallelism"? (I know I can ...)

...

That's easy to resist because, unlike ``for key:value in dict'', it's not unambiguous:

But

if (key:value) in dict

is. Just trying to help whoever *does* want the PEP <wink>.

OK, I'll pronounce -1 on this one. It looks ugly to me -- too reminiscent of C's if (...) required parentheses. Also it suggests that (key:value) is a new tuple notation that might be useful in other contexts -- which it's not.

...

... I'm certainly more comfortable with just ``for key in dict'' than with the whole slow of extensions using colons.

What about just the

for key:value in dict for index:value in sequence

extensions?

I'm not against these -- I'd say +0.5.

The degenerate forms (omitting x or y or both in x:y) are mechanical variations so are likely to get raised.

For those, +0.2.

...

But, again, that's for the PEP to fight over.

PEPs are easier if you Pronounce on things you hate early so that those can get recorded in the "BDFL Pronouncements" section without further ado.

At your service -- see above. --Guido van Rossum (home page: http://www.python.org/~guido/)

[Finn Bock]

Changing the type of a type? Yuck!

No, it temporarily changes the type of the single list being sorted, like so, where "self" is a pointer to a PyListObject (which is a list, not a list *type* object): self->ob_type = &immutable_list_type; err = samplesortslice(self->ob_item, self->ob_item + self->ob_size, compare); self->ob_type = &PyList_Type; immutable_list_type is "just like" PyList_Type, except that the slots for mutating methods point to a function that raises a TypeError. Before this drastic step came years of increasingly ugly hacks trying to stop core dumps when people mutated a list during the sort. Python's sort is very complex, and lots of pointers are tucked away -- having the size of the array, or its position in memory, or the set of objects it contains, change as a side effect of doing a compare, would be difficult and expensive to recover from -- and by "difficult" read "nobody ever managed to get it right before this" <0.5 wink>.

I might very likely be reading the CPython sources wrongly, but it seems this trick will cause an BadInternalCall if some other C extension are trying to modify a list while it is freezed by the type switching trick. I imagine this would happen if the extension called:

PyList_SetItem(myList, 0, aValue);

Well, in CPython it's not "legal" for any other thread to use the C API while the sort is in progress, because the thread doing the sort holds the global interpreter lock for the duration. So this could happen "legally" only if a comparison function called by the sort called out to a C extension attempting to mutate the list. In that case, fine, it *is* a bad call: mutation is not allowed during list sorting, so they deserve whatever they get -- and far better a "bad internal call" than a core dump. If the immutable_list_type were used more generally, it would require more general support (but I see Thomas already talked about that -- thanks).

On Tue, Jan 30, 2001 at 07:28:44PM -0500, Guido van Rossum wrote:

...

What about just the

...

for key:value in dict for index:value in sequence

...

extensions?

I'm not against these -- I'd say +0.5.

What, fractions ? Isn't that against the whole idea of (+|-)(0|1) ? :) But since we are voting, I'm -0 on this right now, and might end up -1 or +0, depending on the implementation; I still can't *see* this, though I wouldn't be myself if I hadn't tried to implement it anyway :) And I ran into some fairly mind-boggling issues. The worst bit is 'how the f*ck does FOR_LOOP know if something's a dict or a list'. And the almost-as-bad bit is 'WTF to do for user classes, extension types and almost-list/almost-dict practically-builtin types (arrays, the *dbm's, etc.)'. After some sleep-deprived consideration I gave up and decided we need an iteration/generator protocol first. However, my life's been busy (or rather, my work has been) with all kinds of small and not so small details, and I haven't been getting much sleep in the last week or so, so I might be overlooking something very simple. That's why I can go either way based on implementation -- it might prove me wrong :) Until my boss is back and I stop being 'responsible' (end of this week, start of next week) and I get a chance to get rid of about 2 months of work backlog (the time he was away) I won't have time to champion or even contribute to such a PEP. Then again, by that time I might be preparing for IPC9 (_if_ my boss sends me there) or even my ApacheCon US presentation (which got accepted today, yay!) So, if that other message was an attempt to drop the PEP on me, Guido, the answer is the same as I tend to give to suits that show up next to my desk wanting to discuss something important (to them) right away: "b'gg'r 'ff" :) I'll-save-my-answer-to-PR-officers-doing-the-same-for-when-you-do-something- -*really*-offensive-ly <wink> y'rs -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!

On Tue, 30 Jan 2001 19:49:24 -0500, Guido van Rossum wrote:

There are different ways to do interators.

Here is a very "tame" proposal (and definitely in the realm of 2.2), that doesn't require any coroutine-like tricks. Let's propose that

for var in expr: ...do something with var...

will henceforth be translated into

__iter = iterator(expr) while __iter.more(): var = __iter.next() ...do something with var...

I'm +1 on that...but Tim's "try to use that to write something that will return the nodes of a binary tree" still haunts me. Personally, though, I'd thin down the interface to while 1: try: var = __iter.next() except NoMoreError: break # pseudo-break? With the usual caveat that this is a lie as far as "else" is concerned (IOW, pseudo-break gets into the else)

Then a new built-in function iterator() is needed that creates an iterator object. It should try two things:

(1) If the object implements __iterator__() (or a C API equivalent), call that and be done; this way arbitrary iterators can be created.

(2) If the object smells like a sequence (how to test???), use an iterator sort of like this:

Why not, "if the object doesn't have __iterator__, try this. If it won't work, we'll find out by the exception that will be thrown in our face". class Iterator: def __init__(self, seq): self.seq = seq self.index = 0 def next(self): try: try: return self.seq[self.index] # <- smells like except IndexError: raise NoMoreError(self.index) finally: self.index += 1

(I don't necessarily mean that all those instance variables should be publicly available.)

But what about your poor brother? <wink> Er....I mean, this would make implementing "indexing" really about just getting the index from the iterator.

If the argument to iterator() is itself an iterator (how to test???),

No idea, and this looks problematic. I see your point -- but it's still problematic. -- Moshe Zadka This is a signature anti-virus. Please stop the spread of signature viruses! Fingerprint: 4BD1 7705 EEC0 260A 7F21 4817 C7FC A636 46D0 1BD6

Guido van Rossum wrote:

[ESR]

...

For different reasons, I'd like to be able to set a constant flag on a object instance. Simple semantics: if you try to assign to a member or method, it throws an exception.

Application? I have a large Python program that goes to a lot of effort to build elaborate context structures in core. It would be nice to know they can't be even inadvertently trashed without throwing an exception I can watch for.

Yes, this is a good thing. Easy to do on lists and dicts. Questions:

- How to spell it? x.freeze()? x.readonly()?

How about .lock() and .unlock() ?

- Should this reversible? I.e. should there be an x.unfreeze()?

Yes. These low-level locks could be used in thread programming since the above calls are C level functions and thus thread safe w/r to the global interpreter lock.

- Should we support something like this for instances too? Sometimes it might be cool to be able to freeze changing attribute values...

Sure :) Eric, could you write a PEP for this ? -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Tim Peters wrote:

Note that even adding a "frozen" flag would add 4 bytes to every freezable object on most machines. That's why I'd rather .freeze() replace the type pointer and .unfreeze() restore it. No time or space overhead; no cluttering up the normal-case (i.e., unfrozen) type implementations with new tests.

Note that Fred's weak ref implementation also need a flag on every weak referencable object (at least last time I looked at his patches). Why not add a flag byte or word to these objects -- then we'd have 8 or 16 choices of what to do with them ;-) -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Mark Hammond wrote:

MAL writes:

...

...

- How to spell it? x.freeze()? x.readonly()?

How about .lock() and .unlock() ?

I'm with Greg here - lock() and unlock() imply an operation similar to threading.Lock() - ie, exclusivity rather than immutability.

I don't have a strong opinion on the other names, but definately prefer any of the others over lock() for this operation.

Funny, I though that .lock() and .unlock() could be used to implement exactly what threading.Lock() does... Anyway, names really don't matter much, so how about: .mutable([flag]) -> integer If called without argument, returns 1/0 depending on whether the object is mutable or not. When called with a flag argument, sets the mutable state of the object to the value indicated by flag and returns the previous flag state. The semantics of this interface would be in sync with many other state APIs in Python and C (e.g. setlocale()). The advantage of making this a method should be clear: it allows writing polymorphic code. -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

What stimulated this thread about making mutable objects (temporarily) immutable? Can someone give me an example where this is actually useful and can't be handled through some existing mechanism? I'm definitely with Fredrik on this one. Sounds like madness to me. I'm just guessing here, but since the most common need for immutable objects is a dictionary keys, I can envision having to test the lock state of a list or dict that someone wants to use as a key everywhere you would normally call has_key: if l.islocked() and d.has_key(l): ... If you want immutable dicts or lists in order to use them as dictionary keys, just serialize them first: survey_says = {"spam": 14, "eggs": 42} sl = marshal.dumps(survey_says) dict[sl] = "spam" Here's another pitfall I can envision. survey_says = {"spam": 14, "eggs": 42} survey_says.lock() dict[survey_says] = "Richard Dawson" survey_says.unlock() At this point can I safely iterate over the keys in the dictionary or not? Skip

Skip Montanaro wrote:

What stimulated this thread about making mutable objects (temporarily) immutable? Can someone give me an example where this is actually useful and can't be handled through some existing mechanism? I'm definitely with Fredrik on this one. Sounds like madness to me.

This thread is an offspring of the "for something in dict:" thread. The problem we face when iterating over mutable objects is that the underlying objects can change. By marking them read-only we can safely iterate over their contents. Another advantage of being able to mark mutable as read-only is that they may become usable as dictionary keys. Optimizations such as self-reorganizing read-only dictionaries would also become possible (e.g. attribute dictionaries which are read-only could calculate a second hash value to make the hashing perfect).

I'm just guessing here, but since the most common need for immutable objects is a dictionary keys, I can envision having to test the lock state of a list or dict that someone wants to use as a key everywhere you would normally call has_key:

if l.islocked() and d.has_key(l): ...

If you want immutable dicts or lists in order to use them as dictionary keys, just serialize them first:

survey_says = {"spam": 14, "eggs": 42} sl = marshal.dumps(survey_says) dict[sl] = "spam"

Sure and that's what .items(), .keys() and .values() do. The idea was to avoid the extra step of creating lists or tuples first.

Here's another pitfall I can envision.

survey_says = {"spam": 14, "eggs": 42} survey_says.lock() dict[survey_says] = "Richard Dawson" survey_says.unlock()

At this point can I safely iterate over the keys in the dictionary or not?

Tim already pointed out that we will need two different read-only states: a) temporary b) permanent For dictionaries to become usable as keys in another dictionary, they'd have to marked permanently read-only. -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

M.-A. Lemburg :

Anyway, names really don't matter much, so how about:

.mutable([flag]) -> integer

If called without argument, returns 1/0 depending on whether the object is mutable or not. When called with a flag argument, sets the mutable state of the object to the value indicated by flag and returns the previous flag state.

I'll bear this in mind if things progress to the point where a PEP is indicated. -- <a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Skip Montanaro wrote:

MAL> This thread is an offspring of the "for something in dict:" thread. MAL> The problem we face when iterating over mutable objects is that the MAL> underlying objects can change. By marking them read-only we can MAL> safely iterate over their contents.

I suspect you'll find it difficult to mark dbm/bsddb/gdbm files read-only. (And what about Andy Dustman's cool sqldict stuff?) If you can't extend this concept in a reasonable fashion to cover (most of) the other objects that smell like dictionaries, I think you'll just be adding needless complications for a feature than can't be used where it's really needed.

We are currently only talking about Python dictionaries here, even though other objects could also benefit from this.

I see no problem asking for the items() of an in-memory dictionary in order to get a predictable list to iterate over, but doing that for disk-based mappings would be next to impossible. So, I'm stuck iterating over something can can change out from under me. In the end, the programmer will still have to handle border cases specially. Besides, even if you *could* lock your disk-based mapping, are you really going to do that in situations where its sharable (that's what databases they are there for, after all)? I suspect you're going to keep the database mutable and work around any resulting problems.

If you want to implement "for key in dict:", why not just have the VM call keys() under the covers and use that list? It would be no worse than the situation today where you call "for key in dict.keys():", and with the same caveats. If you're dumb enough to do that for an on-disk mapping object, well, you get what you asked for.

That's why iterators do a much better task here. In DB design these are usually called cursors which the allow moving inside large result sets. But this really is a different topic... Readonlyness could be put to some good use in optimizing data structure for which you know that they won't change anymore. Temporary readonlyness has the nice sideeffect of allowing low-level lock implementations and makes writing thread safe code easier to handle, because you can make assertions w/r to the immutability of an object during a certain period of time explicit in your code. -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

[Ping]

x is sequence-like if it provides __getitem__() but not keys()

[Skip]

So why does this barf?

>>> [].__getitem__ Traceback (most recent call last): File "<stdin>", line 1, in ? AttributeError: __getitem__

(Obviously, lists *do* understand __getitem__ at some level. Why isn't it exposed in the method table?)

The old type/class split: list is a type, and types spell their "method tables" in ways that have little in common with how classes do it. See PyObject_GetItem in abstract.c for gory details (e.g., dicts spell their version of getitem via ->tp_as_mapping->mp_subscript(...), while lists spell it ->tp_as_sequence->sq_item(...); neither has any binding to the attr "__getitem__"; instance objects fill in both the tp_as_mapping and tp_as_sequence slots, then map both the mp_subscript and sq_item slots to classobject.c's instance_item, which in turn looks up "__getitem__"). bet-you're-sorry-you-asked<wink>-ly y'rs - tim

Tim Peters wrote:

[Tim]

...

...

Seems an unrelated topic: would "iterators for dictionaries" solve the supposed problem with iteration order?

[MAL]

...

No, but it would solve the problem in a more elegant and generalized way.

I'm lost. "Would [it] solve the ... problem?" "No [it wouldn't solve the problem], but it would solve the problem ...". Can only assume we're switching topics within single sentences now <wink>.

Sorry, not my brightest day today... what I wanted to say is that iterators would solve the problem of defining "something" in "for something in dict" nicely. Since iterators can define the order in which a data structure is traversed, this would also do away with the second (supposed) problem.

...

Besides, it also allows writing code which is thread safe, since the iterator can take special actions to assure that the dictionary doesn't change during the iteration phase (see the other thread about "making mutable objects readonly").

Sorry, but immutability has nothing to do with thread safety (the latter has to do with "doing a right thing" in the presence of multiple threads, to keep data structures internally consistent; raising an exception is never "a right thing" unless the user is violating the advertised semantics, and if mutation during iteration is such a violation, the presence or absence of multiple threads has nothing to do with that). IOW, perhaps, a critical section is an area of non-exceptional serialization, not a landmine that makes other threads *blow up* if they touch it.

Who said that an exception is raised ? The method I posted on the mutability thread allows querying the current state just like you would query the availability of a resource.

...

... I don't remember the figures, but these micor optimizations

That's plural, but I thought you were talking specifically about the mutable counter object. I don't know which, but the two statements don't jibe.

The counter object patch is a micro-optimization and as such will only give you a gain of a few percent. What makes the difference is the sum of these micro optimizations. Here's the patch for Python 1.5 which includes the optimizations: http://www.lemburg.com/python/mxPython-1.5.patch.gz

...

do speedup loops by a noticable amount. Just compare the performance of stock Python 1.5 against my patched version.

No time now, but after 2.1 is out, sure, wrt it (not 1.5).

-- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

[Neil Schemenauer]

What's the chances of getting generators into 2.2?

Unknown. IMO it has more to do with generalizing the iteration protocol than with generators per se (a generator object that doesn't play nice with "for" is unpleasant to use; otoh, a generator object that can't be used divorced from "for" is frustrating too (like when comparing the fringes of two trees efficiently, which requires interleaving two distinct traversals, each naturally recursive on its own)).

The implementation should not be hard. Didn't Steven Majewski have something years ago?

Yes, but Guido also sketched out a nearly complete implementation within the last year or so.

Why do we always get sidetracked on trying to figure out how to do coroutines and continuations?

Sorry, I've been failing to find a good answer to that question for a decade <0.4 wink>. I should note, though, that Guido's current notion of "generator" is stronger than Icon/CLU/Sather's (which are "strictly stack-like"), and requires machinery more elaborate than StevenM (or Guido) sketched before.

Generators would add real power to the language and are simple enough that most users could benefit from them. Also, it should be possible to design an interface that does not preclude the addition of coroutines or continuations later.

Agreed.

I'm not volunteering to champion the cause just yet. I just want to know if there is some issue I'm missing.

microthreads have an enthusiastic and possibly growing audience. That gets into (C) stacklessness, though, as do coroutines. I'm afraid that once you go beyond "simple" (Icon) generators, a whole world of other stuff gets pulled in. The key trick to implementing simple generators in current Python is simply to decline decrementing the frame's refcount upon a "suspend" (of course the full details are more involved than *just* that, but they mostly follow *from* just that). everything-is-the-enemy-of-something-ly y'rs - tim

Skip Montanaro :

Can someone give me an example where this is actually useful and can't be handled through some existing mechanism?

I can envisage cases where you want to build a data structure incrementally, and then treat it as immutable so you can use it as a dict key, etc. There's currently no way to do that to a list without copying it. So, it could be handy to have a way of turning a list into a tuple in-place. It would have to be a one-way transformation, otherwise you could start using it as a dict key, make it mutable again, and cause havoc. Suggested implementation: When you allocate the space for the values of a list, leave enough room for the PyObject_HEAD of a tuple at the beginning. Then you can turn that memory block into a real tuple later, and flag the original list object as immutable so you can't change it later via that route. Hmmm, would waste a bit of space for each list object. Maybe this should be a special list-about-to-become-tuple type. (Tist? Luple?) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

Guido van Rossum :

But it *is* true that coroutines are a very attractice piece of land "just nextdoor".

Unfortunately there's a big high fence in between topped with barbed wire and patrolled by vicious guard dogs. :-( Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

>> What would break if we decided to simply add __getitem__ (and other >> sequence methods) to list object's method table? Ping> That would work for lists, but not for any extension types that Ping> use the sq_* protocol to behave like sequences. Could extension writers add those methods to their modules? I know I'm really getting off-topic here, but the whole visible interface idea crops up from time-to-time. I guess I'm just nibbling around the edges a bit to try and understand the problem better. Skip

Tim Peters :

The old type/class split: list is a type, and types spell their "method tables" in ways that have little in common with how classes do it.

Maybe as a first step towards type/class unification one day, we could add __xxx__ attributes to all the builtin types, and start to think of the method table as the definitive source of all methods, with the tp_xxx slots being a sort of cache for the most commonly used ones. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+

[MAL]

... Since iterators can define the order in which a data structure is traversed, this would also do away with the second (supposed) problem.

Except we don't need iterators to do that. If anyone thought it was important, they could change the existing PyDict_Next to force an ordering, and then everything building on that would inherit it. So while I'm in favor of better iteration schemes, I'm not in favor of overselling them (on grounds that aren't unique to them).

...

Sorry, but immutability has nothing to do with thread safety ...

Who said that an exception is raised ?

I did <wink>.

The method I posted on the mutability thread allows querying the current state just like you would query the availability of a resource.

This? .mutable([flag]) -> integer If called without argument, returns 1/0 depending on whether the object is mutable or not. When called with a flag argument, sets the mutable state of the object to the value indicated by flag and returns the previous flag state. If I do: if object.mutable(): object.mutate() in a threaded world, the certain (but erratic) outcome is that sometimes it blows up: there's no guarantee that another thread doesn't sneak in and *change* the mutability between the time object.mutable() returns 1 and object.mutate() acts on a bad assumption. Same thing for: if resources.num_printers_available() > 0: action_that_blows_up_if_no_printers_are_available in a threaded world. It's possible to build a thread-safe resource acquisition protocol in either case, but that's really got nothing to do with immutability or iterators (marking a thing immutable doesn't do any good there unless you *also* build a protocol on top of it for communicating state changes, blocking until one occurs, notifications with optional timeouts, etc -- just doing object.mutable(1) is a threaded disaster in the absence of a higher-level protocol guaranteeing that this won't go changing the mutability state in the middle of some other thread's belief that it's got the thing frozen; likewise for object.mutable(0) not stepping on some other thread's belief that it's got permission to mutate). .mutable(flag) is *fine* for what it does, it's simply got nothing to do with threads. Thread safety could *build* on it via coordinated use of a threading.Sempahore (or moral equivalent), though.

Tim Peters wrote:

[MAL]

...

... Since iterators can define the order in which a data structure is traversed, this would also do away with the second (supposed) problem.

Except we don't need iterators to do that. If anyone thought it was important, they could change the existing PyDict_Next to force an ordering, and then everything building on that would inherit it. So while I'm in favor of better iteration schemes, I'm not in favor of overselling them (on grounds that aren't unique to them).

I'm just trying to sell iterators to bare us the pain of overloading the for-loop syntax just to get faster iteration over dictionaries. The idea is simple: put all the lookup, order and item building code into the iterator, have many of them, one for each flavour of values, keys, items and honeyloops, and then optimize the for-loop/iterator interaction to get the best performance out of them. There's really not much use in adding *one* special case to for-loops when there are a gazillion different needs to iterate over data structures, files, socket, ports, coffee cups, etc.

...

...

Sorry, but immutability has nothing to do with thread safety ...

...

Who said that an exception is raised ?

I did <wink>.

...

The method I posted on the mutability thread allows querying the current state just like you would query the availability of a resource.

This?

.mutable([flag]) -> integer

If called without argument, returns 1/0 depending on whether the object is mutable or not. When called with a flag argument, sets the mutable state of the object to the value indicated by flag and returns the previous flag state.

If I do:

if object.mutable(): object.mutate()

in a threaded world, the certain (but erratic) outcome is that sometimes it blows up: there's no guarantee that another thread doesn't sneak in and *change* the mutability between the time object.mutable() returns 1 and object.mutate() acts on a bad assumption.

I know. That's why you would do this: lock = [] # we use the mutable state as lock indicator; initial state is mutable # try to acquire lock: while 1: prevstate = lock.mutable(0) if prevstate == 0: # was already locked continue elif prevstate == 1: # we acquired the lock break # release lock lock.mutable(1)

Same thing for:

if resources.num_printers_available() > 0: action_that_blows_up_if_no_printers_are_available

in a threaded world. It's possible to build a thread-safe resource acquisition protocol in either case, but that's really got nothing to do with immutability or iterators (marking a thing immutable doesn't do any good there unless you *also* build a protocol on top of it for communicating state changes, blocking until one occurs, notifications with optional timeouts, etc -- just doing object.mutable(1) is a threaded disaster in the absence of a higher-level protocol guaranteeing that this won't go changing the mutability state in the middle of some other thread's belief that it's got the thing frozen; likewise for object.mutable(0) not stepping on some other thread's belief that it's got permission to mutate).

.mutable(flag) is *fine* for what it does, it's simply got nothing to do with threads. Thread safety could *build* on it via coordinated use of a threading.Sempahore (or moral equivalent), though.

Ok... :) -- Marc-Andre Lemburg ______________________________________________________________________ Company: http://www.egenix.com/ Consulting: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

On Sat, 3 Feb 2001, Tim Peters wrote:

They're simply distinct issues to me. Whether people want special syntax for iterating over dicts is (to me) independent of how the iteration protocol works. Dislike of the former should probably be stabbed into Ping's heart <wink>.

Ow! Hey. :) We have shorthand like x[k] for spelling x.__getitem__[k]; why not shorthand like 'for k:v in x: ...' for spelling 'iter = x.__iteritems__(); while 1: k, v = iter() ...'? Hmm. What is the issue really with? - the key:value syntax suggested by Guido (i like it quite a lot) - the existence of special __iter*__ methods (seems natural to me; this is how we customize many operators on instances already) - the fact that 'for k:v' checks __iteritems__, __iter__, items, and __getitem__ (it *has* to check all of these things if it's going to play nice with existing mappings and sequences) - or something else? I'm not actually that clear on what the general feeling is about this PEP. Moshe seems to be happy with the first part but not the rest; Tim, do you have a similar position? Eric and Greg both disagreed with Moshe's counter-proposal; does that mean you like the original, or that you would rather do something different altogether? Moshe Zadka wrote:

dict.iteritems() could return not an iterator, but a magical object whose iterator is the requested iterator. Ditto itervalues(), iterkeys()

Seems like too much work to me. I'd rather just have the object produce a straight iterator. (By 'iterator' i mean an ordinary callable object, nothing too magical.) If there are unusual cases where you want to iterate over an object in several different ways i suppose they can create pseudo-sequences in the manner you described, but i think we want to make the most common case (iterating over the object itself) very easy. That is, just implement __iter__ and have it produce a callable. Marc A. Lemburg wrote:

The idea is simple: put all the lookup, order and item building code into the iterator, have many of them, one for each flavour of values, keys, items and honeyloops, and then optimize the for-loop/iterator interaction to get the best performance out of them.

There's really not much use in adding *one* special case to for-loops when there are a gazillion different needs to iterate over data structures, files, socket, ports, coffee cups, etc.

I couldn't tell which way you were trying to argue here. Are you in favour of the general flavour of PEP 234 or did you have in mind something different? Your first paragraph above seems to describe what 234 does. -- ?!ng "There's no point in being grown up if you can't be childish sometimes." -- Dr. Who

...

The old type/class split: list is a type, and types spell their "method tables" in ways that have little in common with how classes do it.

Maybe as a first step towards type/class unification one day, we could add __xxx__ attributes to all the builtin types, and start to think of the method table as the definitive source of all methods, with the tp_xxx slots being a sort of cache for the most commonly used ones.

Yes, I've often thought that we should be able to heal the split for 95% by using a few well-aimed tricks like this. Later... --Guido van Rossum (home page: http://www.python.org/~guido/)

On Sun, Feb 04, 2001 at 11:47:26PM -0500, Guido van Rossum wrote:

...

Yes, I've often thought that we should be able to heal the split for 95% by using a few well-aimed tricks like this. Later...

I was playing around this weekend with the class/type problem. Without too much effort I had an interpreter that could to things like this:

>>> class MyInt(type(1)): ... pass ... >>> i = MyInt(10) >>> i 10 >>> i + 1 11

Now, can you do things like this: >>> from types import * >>> class MyInt(IntType): # add a method def add1(self): return self+1 >>> i = MyInt(10) >>> i.add1() 11 >>> and like this: >>> class MyInt(IntType): # override division def __div__(self, other): return float(self) / other def __rdiv__(self, other): return other / float(self) >>> i = MyInt(10) >>> i/3 0.33333333333333331 >>> I'm not asking for adding new instance variables (slots), but that of course would be the next step of difficulty up.

The major changes were allowing PyClassObject to subclass types (ie. changing PyClass_Check(op) to (PyClass_Check(op) || PyType_Check(op))), writing a _PyType_Lookup function, and making class_lookup use it.

Yeah, but that's still nasty. We should strive for unifying PyClass and PyType instead of having both around.

The experiment has convinced me that we can allow subclasses of types quite easily without major changes. It has also given me some ideas on "the right way" to solve this problem. The rough scheme I can up yesterday goes like this:

p> PyObject {

int ob_refcnt; PyClass ob_class;

(plus type-specific fields I suppose)

}

PyClass { PyObject_HEAD char *cl_name; getattrfunc cl_getattr; PyMethodTable *cl_methods; }

PyMethodTable { binaryfunc nb_add; binaryfunc nb_sub; ... }

When calling a method on a object the interpreter would first check for a direct method and if that does not exist then call cl_getattr. Obviously there are still a few details to be worked out. :-)

Yeah... Like you should be able to ask for ListType.append and get an unbound built-in method back, which can be applied to a list: ListType.append([], 1) === [].append(1) And ditto for operators: IntType.__add__(1, 2) === 1+2 And a C API like PyNumber_Add(x, y) should be equivalent to using x.__add__(y), too. --Guido van Rossum (home page: http://www.python.org/~guido/)

On Mon, Feb 05, 2001 at 01:37:39PM -0500, Guido van Rossum wrote:

...

Now, can you do things like this: [example cut]

No, it would have to be written like this:

>>> from types import * >>> class MyInt(IntType): # add a method def add1(self): return self.value+1

>>> i = MyInt(10) >>> i.add1() 11 >>>

Note the value attribute. The IntType.__init__ method is basicly:

def __init__(self, value): self.value = value

So, "class MyInt(IntType)" acts as a sort-of automagical "UserInt" class creation? (Analogous to UserList etc.) I'm not sure I like that. Why do we have to have this? --Guido van Rossum (home page: http://www.python.org/~guido/)

On Mon, Feb 05, 2001 at 11:04:22AM -0800, Neil Schemenauer wrote:

On Mon, Feb 05, 2001 at 01:37:39PM -0500, Guido van Rossum wrote:

...

Now, can you do things like this: [example cut]

No, it would have to be written like this:

>>> from types import * >>> class MyInt(IntType): # add a method def add1(self): return self.value+1

Why ? Couldn't IntType do with an __add__[*] method that does this ugly magic for you ? Same for __sub__, __int__ and so on. *] Yes, yes, I know, it's a type, not a class, but you know what I mean :) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread!