[Guido, on Andrew's idea for automagically updating classes]

There might be another solution. When you reload a module, the module object and its dictionary are reused.

Perhaps class and function objects could similarly be reused? It would mean that a class or def statement looks for an existing object with the same name and type, and overwrites that. Voila, all references are automatically updated.

Too dangerous, I think. While uncommon in general, I've certainly seen (even written) functions that e.g. return a contained def or class. The intent in such cases is very much to create distinct defs or classes (despite having the same names). In this case I assume "the same name" wouldn't *usually* be found, since the "contained def or class"'s name is local to the containing function. But if there ever happened to be a module-level function or class of the same name, brrrr. Modules differ because their namespace "search path" consists solely of the more-global-than-global <wink> sys.modules.

This is more work (e.g. for classes, a new bytecode may have to be invented because the class creation process must be done differently) but it's much less of a hack, and I think it would be more reliable. (Even though it alters borderline semantics a bit.)

How about an explicit function in the "new" module, new.update(class_or_def_old, class_or_def_new) which overwrites old's guts with new's guts (in analogy with dict.update)? Then no semantics change and you don't need new bytecodes. In return, a user who wants to e.g. replace an existing class C would need to do oldC = C do whatever they do to get the new C new.update(oldC, C) Building on that, a short Python loop could do the magic for every class and function in a module; and building on *that*, a short "updating import" function could be written in Python. View it as providing mechanism instead of policy <0.9 wink>.

(Your extra indirection also slows things down, although I don't know by how much -- not just the extra memory reference but also less locality of reference so more cache hits.)

Across the universe of all Python programs on all platforms, weighted by importance, it was a slowdown of nearly 4.317%. if-i-had-used-only-one-digit-everyone-would-have- known-i-was-making-it-up<wink>-ly y'rs - tim

[Greg Stein]

... In other words, I definitely would support a new class object behavior that allows us to update a class' set of bases and dictionary on the fly. This could then be used to support my solution for the recursive type scenario (which, in turn, means that we don't have to introduce Yet Another Namespace into Python to hold type names).

Parenthetically, I never grasped the appeal of the parenthetical comment. Yet Another Namespace for Yet Another Entirely New Purpose seems highly *desirable* to me! Trying to overload the current namespace set makes it so much harder to see that these are compile-time gimmicks, and users need to be acutely aware of that if they're to use it effectively. Note that I understand (& wholly agree with) the need for runtime introspection. different-things-different-rules-ly y'rs - tim

Greg Stein wrote:

Tim Peters:

...

Trying to overload the current namespace set makes it so much harder to see that these are compile-time gimmicks, and users need to be acutely aware of that if they're to use it effectively. Note that I understand (& wholly agree with) the need for runtime introspection.

And that is the crux of the issue: I think the names that are assigned to these classes, interfaces, typedefs, or whatever, can follow the standard Python semantics and be plopped into the appropriate namespace. There is no overloading.

This is indeed the crux of the issue. For those that missed it last time, it became very clear to me that we are working with radically different design aesthetics when we discussed the idea of having an optional keyword that said: "this thing is usually handled at compile time but I want to handle it at runtime. I know what I am doing." Greg complained that that would require the programmer to understand too much what was being done at compile time and what at runtime and what.

From my point of view this is *exactly* what a programmer *needs* to know and if we make it too hard for them to know it then we have failed.

There is no "overloading" of namespaces. We are using Python namespaces just like they should be, and the type-checker doesn't even have to be all the smart to track this stuff.

There is an overloading of namespaces because we will separately specify the *compile time semantics* of these names. We need to separately specify these semantics because we need all compile time type checkers to behave identically. Yes, it seems elegant to make type objects seem as if they are "just like" Python objects. Unfortunately they aren't. Type objects are evaluated -- and accepted or rejected -- at compile time. Every programmer needs to understand that and it should be blatantly obvious in the syntax, just as everything else in Python syntax is blatantly obvious. -- Paul Prescod - ISOGEN Consulting Engineer speaking for himself Earth will soon support only survivor species -- dandelions, roaches, lizards, thistles, crows, rats. Not to mention 10 billion humans. - Planet of the Weeds, Harper's Magazine, October 1998

From: "Tim Peters"

[Guido, on Andrew's idea for automagically updating classes]

...

There might be another solution. When you reload a module, the module object and its dictionary are reused.

Perhaps class and function objects could similarly be reused? It would mean that a class or def statement looks for an existing object with the same name and type, and overwrites that. Voila, all references are automatically updated.

Too dangerous, I think. While uncommon in general, I've certainly seen (even written) functions that e.g. return a contained def or class. The intent in such cases is very much to create distinct defs or classes (despite having the same names). In this case I assume "the same name" wouldn't *usually* be found, since the "contained def or class"'s name is local to the containing function. But if there ever happened to be a module-level function or class of the same name, brrrr.

Agreed that that would be bad. But I wouldn't search outer scopes -- I would only look for a class/def that I was about to stomp on.

Modules differ because their namespace "search path" consists solely of the more-global-than-global <wink> sys.modules.

"The search path doesn't enter into it."

...

This is more work (e.g. for classes, a new bytecode may have to be invented because the class creation process must be done differently) but it's much less of a hack, and I think it would be more reliable. (Even though it alters borderline semantics a bit.)

How about an explicit function in the "new" module,

new.update(class_or_def_old, class_or_def_new)

which overwrites old's guts with new's guts (in analogy with dict.update)? Then no semantics change and you don't need new bytecodes.

Only a slight semantics change (which my full proposal would require too): function objects would become mutable -- their func_code, func_defaults, func_doc and func_globals fields (and, why not, func_name too) should be changeable. If you make all these assignable, it doesn't even have to be a privileged function.

In return, a user who wants to e.g. replace an existing class C would need to do

oldC = C do whatever they do to get the new C new.update(oldC, C)

Building on that, a short Python loop could do the magic for every class and function in a module; and building on *that*, a short "updating import" function could be written in Python. View it as providing mechanism instead of policy <0.9 wink>.

That's certainly a reasonable compromise. Note that the update on a class should imply an update on its methods, right? --Guido van Rossum (home page: http://www.python.org/~guido/)

[Tim worries about stomping on unintended classes/defs]

[Guido]

...

Agreed that that would be bad. But I wouldn't search outer scopes -- I would only look for a class/def that I was about to stomp on.

Maybe I just don't grasp what that means, exactly. Fair enough, since I'm not expressing myself clearly either!

Suppose someone does

from Tkinter import *

in my.py, and later in my.py just *happens* to define, at module level,

class Misc: blah blah blah

Now Misc was already in my.py's global namespace because Tkinter.py just happens to export a class of that name too (more by accident than design -- but accidents are what I'm most worried about here).

For a second I thought you got me there!

At the time my.py defines Misc, does Misc count as a class we're "about to stomp on"? If so-- & I've assumed so --it would wreak havoc.

But if not, I don't see how this case can be reliably distinguished "by magic" from the cases where update is desired (if people are doing dynamic updates to a long-running program, a new version of a class can come from anywhere, so nothing like original file name or line number can distinguish correctly either).

Fortunately, there's magic available: recently, all classes have a __module__ attribute that is set to the full name of the module that defined it (its key in __sys__.modules). For functions, we would have to invent something similar.

...

...

Modules differ because their namespace "search path" consists solely of the more-global-than-global <wink> sys.modules.

...

"The search path doesn't enter into it."

I agree, but am at a loss to describe what's happening in the case above using other terminology <wink>. In a sense, you need a system-wide "unique handle" to support bulletproof updating, and while sys.modules has supplied that all along for module objects (in the form of the module name), I don't believe there's anything analogous to key off of for function or class objects.

...

...

[suggesting] new.update(class_or_def_old, class_or_def_new)

...

Only a slight semantics change (which my full proposal would require too): function objects would become mutable -- their func_code, func_defaults, func_doc and func_globals fields (and, why not, func_name too) should be changeable.

Of course I meant "no new semantics" in the sense of "won't cause current exception-free code to alter behavior in any way".

...

If you make all these assignable, it doesn't even have to be a privileged function.

I'm all for that!

...

[sketching a Python approach to "updating import/reload" building on the hypothetical new.update]

...

That's certainly a reasonable compromise. Note that the update on a class should imply an update on its methods, right?

Hadn't considered that! Of course you're right. So make it a pair of nested loops <wink>.

so-long-as-it-can-be-written-in-python-it's-easy-ly y'rs - tim

--Guido van Rossum (home page: http://www.python.org/~guido/)

...

...

...

...

"TP" == Tim Peters writes:

TP> Under the explicit new.update scheme, such a service needn't TP> bother clients with communicating the full name of the TP> original module; heck, in a *truly* long-running app, over TP> time the source tree will change, and classes and functions TP> will migrate across modules. That will be a problem for the TP> explicit scheme too (how does it know *which* "class Misc" to TP> update) -- but at least it's an explicit problem then, and not TP> a "mysterous failure" of hidden magic. I completely agree. I think in general, such long running apps are rare, and in those cases you probably want to be explicit about when and how the updates occur anyway. The one place where automatic updates would be convenient would be at the interactive prompt, so it might be nice to add a module that could be imported by PYTHONSTARTUP, and play hook games to enable automatic updates. -Barry

On Thu, 20 Jan 2000, Guido van Rossum wrote:

Tim Peters: ...

...

At the time my.py defines Misc, does Misc count as a class we're "about to stomp on"? If so-- & I've assumed so --it would wreak havoc.

But if not, I don't see how this case can be reliably distinguished "by magic" from the cases where update is desired (if people are doing dynamic updates to a long-running program, a new version of a class can come from anywhere, so nothing like original file name or line number can distinguish correctly either).

Fortunately, there's magic available: recently, all classes have a __module__ attribute that is set to the full name of the module that defined it (its key in __sys__.modules).

For functions, we would have to invent something similar.

func.func_globals __module__ and func_globals can prevent *other* modules from redefining something accidentally, but it doesn't prevent Badness from within the module. [ Tim just posted an example of this: his "def adder()" example... ] Cheers, -g -- Greg Stein, http://www.lyra.org/

Jim Fulton wrote:

Reloading a module redefines the global variables in a module. It doesn't update any references to those global references from other places, such as instances or *other* modules.

For example, imports like:

from foo import spam

are not updated when foo is reloaded.

A change to the way that namespaces are handled could make this work and have a number of other benefits, like global name usage without namespace lookups. I've suggested this to Guido in the past. His reasonable response is that this would be too big a change for Python 1. Maybe this is something to consider for Python 2? The basic idea (borrowed from Smalltalk) is to have a kind of dictionary that is a collection of "association" objects. An association object is simply a pairing of a name with a value. Association objects can be shared among multiple namespaces. An import like: from foo import spam would copy the association between the name 'foo' and a value from module 'spam' into the current module. If foo is reloaded or if the name is reassigned in spam, the association is modified and the change is seen in any namespaces that imported foo. Similarly if a function uses a global variable: spam=1 def bar(): global spam return spam*2 the compiled function contains the association between spam and it's value. This means that: - When spam is used in the function, it doesn't have to be looked up, - The function object no longer needs to keep a reference to it's globals. This eliminates an annoying circular reference. (I would not replace existing dictionaries with this new kind. I'd have both kinds available.) I think that this would be a really nice change for Python 2. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Guido van Rossum wrote:

...

I've suggested this to Guido in the past. His reasonable response is that this would be too big a change for Python 1. Maybe this is something to consider for Python 2?

Note: from now on the new name for Python 2 is Python 3000. :-)

I like it.

...

The basic idea (borrowed from Smalltalk) is to have a kind of dictionary that is a collection of "association" objects. An association object is simply a pairing of a name with a value. Association objects can be shared among multiple namespaces.

I've never liked this very much, mostly because it breaks simplicity: the idea that a namespace is a mapping from names to values (e.g. {"limit": 100, "doit": , ...}) is beautifully simple, while the idea of inserting an extra level of indirection, no matter how powerful, is much murkier.

How so? It doesn't change the mapping semantics.

There's also the huge change in semantics, as you point out; currently,

from foo import bar

has the same effect (on bar anyway) as

import foo bar = foo.bar # i.e. copying an object reference del foo

while under your proposal it would be more akin to changing all references to bar to become references to foo.bar.

Of course that's what the moral equivalent of "from ... import ..." does in most other languages anyway, so we might consider this for Python 3000;

Cool. Again, it would also make function global variable access faster and cleaner in some ways.

however it would break a considerable amount of old code, I think.

Really? I wonder. I bet it would break alot less old code that other recent changes.

(Not to mention brain

It makes my brain feel much better. :)

and book breakage. :-)

Hey, all of the books will have to be rewritten for Python 3000. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

<meta-comment> Eek, I didn't realized this thread had continued until I happened to notice Christian's post today. <waaa>I get too much email</waaa> </meta-comment> Guido van Rossum wrote:

[me]

...

...

I've never liked this very much, mostly because it breaks simplicity: the idea that a namespace is a mapping from names to values (e.g. {"limit": 100, "doit": , ...}) is beautifully simple, while the idea of inserting an extra level of indirection, no matter how powerful, is much murkier.

[Jim F]

...

How so? It doesn't change the mapping semantics.

My assumption is that in your version, the dictionary would contain special <object binding> objects which then would contain the referenced objects. E.g. {"limit": , "doit": >}. Thus, d["limit"] would be that <binding> object, while previously it would return 100.

No. The idea is to have "association" objects. We can create these directly if we want: a=Association('limit',100) print a.key, a.value # whatever The association value is mutable, but the key is not. A namespace object is a collection of association objects such that no two items have the same key. Internally, this would be very much like the current dictionary except that instead of an array of dictentries, you'd have an array of association object pointers. Effectively, associations are exposed dictentries. Externally, a namspace acts more or less like any mapping object. For example, when someone does a getitem, the namespace object will find the association with the desired key and return it's value. In addition, a namspace object would provide methods along the lines of: associations() Return a sequence of the associations in the namespace addAssociation(assoc) Add the given association to the namsspace. This creates another reference to the association. Changing the association's value also changes the value in the namespace. getAssociation(key) Get the association associated with the key. A setitem on a namespace modifies an existing association if there is already an association for the given key. For example: n1=namespace() n1['limit']=100 n2=namespace() n2.addAssociation(n1.getAssociation('limit')) print n2['limit'] # prints 100 n1['limit']=200 print n2['limit'] # prints 200 When a function is compiled that refers to a global variable, we get the association from the global namespace and store it. The function doesn't need to store the global namespace itself, so we don't create a circular reference. Note that circular references are bad even if we have a more powerful gc. For example, by not storing the global namespace in a function, we don't have to worry about the global namespace being blown away before a destructor is run during process exit. When we use the global variable in the function, we simply get the current value from the association. We don't have to look it up. Namespaces would have other benefits: - improve the semantics of: from spam import foo in that you'd be importing a name binding, not a value - Be useful in any application where it's desireable to share a name binding.

...

Again, it would also make function global variable access faster and cleaner in some ways.

But I have other plans for that (if the optional static typing stuff ever gets implemented).

Well, OK, but I argue that the namespace idea is much simpler and more foolproof.

...

...

however it would break a considerable amount of old code, I think.

Really? I wonder. I bet it would break alot less old code that other recent changes.

Oh? Name some changes that broke a lot of code?

The move to class-based exceptions broke alot of our code. Maybe we can drop this point. Do you still think that the namespace idea would break alot of code? Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Guido van Rossum wrote:

...

No. The idea is to have "association" objects. We can create these directly if we want:

a=Association('limit',100) print a.key, a.value # whatever

The association value is mutable, but the key is not.

A namespace object is a collection of association objects such that no two items have the same key. Internally, this would be very much like the current dictionary except that instead of an array of dictentries, you'd have an array of association object pointers. Effectively, associations are exposed dictentries.

Externally, a namspace acts more or less like any mapping object. For example, when someone does a getitem, the namespace object will find the association with the desired key and return it's value. In addition, a namspace object would provide methods along the lines of:

associations()

Return a sequence of the associations in the namespace

addAssociation(assoc)

Add the given association to the namsspace. This creates another reference to the association. Changing the association's value also changes the value in the namespace.

getAssociation(key)

Get the association associated with the key.

A setitem on a namespace modifies an existing association if there is already an association for the given key.

I presume __setitem__() creates a new association if there isn't one.

Yes.

I also presume that if an association's value is NULL, it doesn't show up in keys(), values() and items() and it doesn't exist for has_key() or __getitem__().

Right.

What does a delitem do? Delete the association or set the value to NULL? I suppose the latter.

Good question. I'm inclined to think the former. That is, deleting an item from the namespace would delete the name association. I can see arguments both ways.

...

For example:

n1=namespace() n1['limit']=100 n2=namespace() n2.addAssociation(n1.getAssociation('limit')) print n2['limit'] # prints 100 n1['limit']=200 print n2['limit'] # prints 200

When a function is compiled that refers to a global variable, we get the association from the global namespace and store it. The function doesn't need to store the global namespace itself, so we don't create a circular reference.

For this to work we would have to have to change the division of labor between the function object and the code object. The code object is immutable and contains no references to mutable objects; this means that it can easily be marshalled and unmarshalled. (Also, when a code object is compiled or unmarshalled, the globals in which its function will be defined may not exist yet.) The function object currently contains a pointer to the code object and a pointer to the dictionary with the globals. (It also contains the default arg values.)

It seems that for associations to work, they need to be placed in the function object, and the code object somehow needs to reference them through the function object. To make this concrete: if a function references globals a, b, and c, these need to be numbered, and the bytecodes should look like this:

LOAD_GLOBAL 0 # a STORE_GLOBAL 1 # b DEL_GLOBAL 2 # c

(This could be compiled from ``b = a; del c''.)

The code object should also contains a list of global names, ordered by their ordinals, e.g. ("a", "b", "c").

Then when the function object is created, it looks in that list and creates a corresponding list of associations, e.g.:

L = [] for name in code.co_global_names: L.append(globals.getAssociation(name))

The VM then sticks a pointer to this list into the frame, whenever the function is called (instead of the globals dict which it sticks there now), and the LOAD/STORE/DEL_GLOBAL opcodes reference the associations through this list.

Looks good to me. :)

Some complications left as exercises:

- The built-in functions (and exceptions, etc.) should also be referenced via associations; the loop above would become a bit trickier since it needs to look in two dicts. (We're assuming that the code generator doesn't know which names are globals and which are built-ins.)

- If the association for a name doesn't yet exist, it should be created.

Yup.

Note that the semantics are slightly different than currently: the decision whether a name refers to a global or to a built-in is made when the function is defined rather than each time when the name is referenced. This is a bit cleaner -- in the type-sig we're making similar assumptions but the decision is made even earlier.

But, overall the necessary changes to the implementation and to the semantics (e.g. of the 'for' statement) seem prohibitive to me.

Really? Even for Py3K?

I also think that the namespace implementation will be quite a bit less efficient than a regular dictionary:

Spacewise yes. They'd me much faster in use. This is a space/speed tradeoff.

currently, a dictionary entry is a struct of 12 bytes, and the dictionary has an array of these tightly packed. Your association objects will be "real" objects, which means they have a reference count, a type pointer, a key, and a value, i.e. 16 bytes, without counting the malloc overhead; this probably comes in addition to the 12 bytes in the dict entry.

Why not replace the key and value pointers with the association pointer. Then you'd get back a little of the space.

(If you want to have the association objects directly in the hash table, they can't be shared between namespaces, and a namespace couldn't grow -- when a dict grows its hash table is reallocated.)

...

Note that circular references are bad even if we have a more powerful gc.

I don't understand or believe this statement.

This was discussed at length a year or two ago. You added code to print to stderr when an error occured in a destructor. People noticed that they were getting errors when Python exited. The problem occured when a destructor was called after it's globals had been deallocated. You subsequently added alot of extra rules on shutdown to make this much less likely. I don't think you made the problem go away completely. I find circular references to be bad in other ways. For example, they are a pain with deep copy. You can make deep copy do something in the presense of circular references, but the things it does can be quite surprising.

...

For example, by not storing the global namespace in a function, we don't have to worry about the global namespace being blown away before a destructor is run during process exit.

If we had more powerful gc the global namespace wouldn't have to be blown away at all (it would gently dissolve when __main__ was deleted from the interpreter).

Uh, OK, then we wouldn't have to worry about the global namespace being gently dissolved before a destructor is run during process exit.

...

When we use the global variable in the function, we simply get the current value from the association. We don't have to look it up.

Namespaces would have other benefits:

- improve the semantics of:

from spam import foo

in that you'd be importing a name binding, not a value

But its semantics will be harder to explain, because they will no longer be equivalent to

import spam # assume there's no spam already foo = spam.foo del spam

Will they really be harder to explain? Why not explain them a different way? "The statement: from spam import foo copies a name binding for foo from module spam to the current module." Eh, I guess I can see why someone would find this harder....

Also, we currently *explain* that only objects are shared and name bindings are unique per namespace; this would no longer be true so we would have to explain a much harder rule. ("If you got your foo through an import from another module, assigning to it will affect foo in that other module too; but if you got it through a local assignment, the effect will be local.")

Good point. Perhaps assinging in the client module should break the connection to the other module. This would require some extra magic.

All in all, I think these semantics are messy and unacceptable. True, object sharing is hard to explain too (see diagram on Larning Python page 60), but you'll still have to explain that anyway because it still exists within a namespace; but now in addition we'd have to explain that there is an exception to object sharing... Messy, messy.

Well, I don't have a problem with object sharing, so the notion of sharing namespaces doesn't bother me. I undertand that some folks have a problem with object sharing and I agree that they'd have problems with name sharing. OTOH, I don't think you'd consider the fact that some people have difficulty with object sharing to be sufficient justification for removing the feature from the language.

...

- Be useful in any application where it's desireable to share a name binding.

I think it's better to explicitly share the namespace -- "foo.bar = 1" makes it clear that whoever else has a reference to foo will see bar similarly changed.

...

...

...

Again, it would also make function global variable access faster and cleaner in some ways.

But I have other plans for that (if the optional static typing stuff ever gets implemented).

Well, OK, but I argue that the namespace idea is much simpler and more foolproof.

I claim that it's not foolproof at all -- on the contrary, it creates something that hides in the dark and will bite us in the behind by surprise,

How so?

long after we thought we knew there were no monsters under the bed. (Yes, I've been re-reading Calvin and Hobbes. :-)

...

...

...

...

however it would break a considerable amount of old code, I think.

Really? I wonder. I bet it would break alot less old code that other recent changes.

Oh? Name some changes that broke a lot of code?

The move to class-based exceptions broke alot of our code.

It must have been very traumatic that you're still sore over that; it was introduced in 1.5, over two years ago.

I'm not sore. But it was a bigger (IMO) backward incompatibility. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Guido van Rossum wrote:

...

...

But, overall the necessary changes to the implementation and to the semantics (e.g. of the 'for' statement) seem prohibitive to me. ^^^ (I meant 'from' of course)

...

Really? Even for Py3K?

The implementation wouldn't be a problem for Py3K; I was under the impression that you thought this could be put in earlier.

Jim proposed adding namespaces to Python 2000. This will be, for my understanding, a complete rewrite and redesign that is allowed to break existing code. It would even be run in parallel to Python 1.6++ for a while, right?

But the change in semantics is very hard to swallow to me. It definitely seems to be come murkier.

...

...

I also think that the namespace implementation will be quite a bit less efficient than a regular dictionary:

Spacewise yes. They'd me much faster in use. This is a space/speed tradeoff.

Agreed; though the speedup comes from circumventing the dictionary altogether.

I do not even believe in the space ineffectiveness. The namespace concept works fine without a dictionary and hashes at all. We can implement this as a linear list of pointers to namespace objects, since they are looked up only once, usually. But even if we would keep a dictionary-like structure as well, it is possible to implement it as an array of pointers, and you get things smaller than now, not bigger. In your analysis, you forgot to take into account that the average dictionary slot overhead gives a factor of about two. 1 dict slot = 3 words n dict entries = average 2n dict slots = 6n words versus 1 asso object = = 4 words n asso dict entries = average 2n words + n asso objects This gives 6n words for the proposed solution, actually as effective as today's 6n solution with dicts. Ahem :-) It can of course be that we also need a hash filed, which can be stored in the asso object. This is another word per element, so we'd have a cost increase of 1/6. As said, the dictionary is not necessary and could be created on demand (that is, if globals are really used like a dict). Without it, I count just n + 4n = 5n, actually a saving. This idea bears a lot of potential to also speed up classes and instances. Further analysis is needed, please let us not drop this idea too early. ciao - chris -- Christian Tismer :^) mailto:tismer@appliedbiometrics.com Applied Biometrics GmbH : Have a break! Take a ride on Python's Düppelstr. 31 : *Starship* http://starship.python.net 12163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home

Guido van Rossum wrote:

(Math about sapce savings gracefully accepted.)

...

This idea bears a lot of potential to also speed up classes and instances. Further analysis is needed, please let us not drop this idea too early.

I will gladly use it as an implementation strategy "under the hood" if it makes sense -- I think that with some code analysis (e.g. "what are the globals here") it could be made to work well.

Pleased to hear this. Thank you for keeping it alive.

But I don't think that changing the "from M import v" semantics so that local assignment to v changes the binding of M.v as well is defensible.

Of course this may be the weakest point yet, while it was the reason to reasoning at all in the first place, it is less important now. :-) ciao - chris -- Christian Tismer :^) mailto:tismer@appliedbiometrics.com Applied Biometrics GmbH : Have a break! Take a ride on Python's Düppelstr. 31 : *Starship* http://starship.python.net 12163 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home

Guido van Rossum wrote:

(snip)

But I don't think that changing the "from M import v" semantics so that local assignment to v changes the binding of M.v as well is defensible.

I agree, however, I think that having: from M import v causing a name binding that is broken by local assigment to v *is* defensible and reasonably implementable. Changes to 'v' in M (including by reload of M) would be reflected locally unless someone did: v=something locally. Local assignment would negate an import, as it does now. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Guido van Rossum wrote:

[me]

...

...

But I don't think that changing the "from M import v" semantics so that local assignment to v changes the binding of M.v as well is defensible.

I agree, however, I think that having:

from M import v

causing a name binding that is broken by local assigment to v *is* defensible and reasonably implementable.

Changes to 'v' in M (including by reload of M) would be reflected locally unless someone did:

v=something

locally. Local assignment would negate an import, as it does now.

Hm, but it still wouldn't have the same semantics as currently,

Agreed. Think Python 3000. I think that the semantics differ in boundary cases though.

and that's still a monster hiding under the bed until you're nearly asleep. Consider this example:

# in M: verbose = 1

# in __main__: from M import verbose

# somewhere else: M.verbose = 0

Under the current semantics, that would have no effect on verbose in __main__; but with your semantics it would.

Yup.

I think that is very hard to explain; even more so if you say that assigning a different value to __main__.verbose does not change M.verbose and furthermore breaks the connection. This means that if I add

verbose = verbose

to the __main__ code the semantics are different!

I'm suggesting a model where from "M import x" has a different meaning than it does now. I think the notion of sharing a name is useful. I'll admit that using "M.x" achieves the same thing, although at a higher performance cost (and, OK, typing cost ;).

I don't understand why you wanted these semantics in the first place.

First, let me say that this isn't super important to me. It does solve a problem with reload, which is the context in which I brought it up. Now, consider: from M import x ..... use(x) Many people would (wrongly) consider this to be equivalent to: import M ..... use(M.x) In fact, I'd *prefer* these to be equivalent even in the face of changes to M (e.g. reload). I'd prefer different semantics. Note that if I had: from M import x ..... x=y ..... use(x) I'd no longer exprect x to have any connection to M. Of course: x=x or x=M.x would be a bit more puzzling, but then they're meant to be. ;) They are addressed by a simple rule, which is that assignment in a module overrides imported name definition. Hm...ooh ooh A better solution would be to disallow assignments to imported names, as they are very likely to be errors. This could be detected without any fancy type inferencing. In fact, we could also decide to disallow an import-from to override an existing name binding. Ahhhhhh. :) In any case, I'd feel comfortable explaining a system in which from M import x # reference semantics wrt name had a different meaning from: import M x=M.x # copy semantics since I expect an attribute access to give me a value, not a name, whereas: from M import x seems more to me like it's talking about names. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

[Jeremy Hylton, on JimF's association objects wrt "from M import x" semantics]

This seems to contradict the 2nd Pythonic principle: Explicit is better than implicit.

I don't literally mean to argue that "The Python Way" should be used to make design decisions, but it captures exactly what makes me uncomfortable with the proposed change.

Go ahead & argue it: they were *intended* to be used to guide design decisions! They were my best shot at summarizing what I've learned in a decade of (mostly successful) Guido-channeling. But note that I only listed 19 of the 20 Pythonic Theses: the 20th was left blank, for Guido to fill in however he likes whenever the other 19 suggest a direction he dislikes <wink>. Other relevant theses here include the ones about whether the implementation is, or isn't, easy to explain. I'm suffering an email backlog and haven't yet studied the latest batch on this topic, but a quick skim sure suggests that a concrete implementation isn't self-evident, and its implications perhaps downright subtle regardless. not-a-conclusion-just-a-concern-ly y'rs - tim

Howdy, lemme try... Vladimir Marangozov wrote:

[Jim, Chris & Guido discussing a namespace idea]

Guys, I'm lost.

Please help me understanding this idea from the start.

After rereading this whole thread, I have only a vague intuition of what Jim has proposed, but I fail to understand it; and believe me, I'm very interested in being in sync with you on the subject.

Please filter the concept from the consequences and resubmit it once again (in english, through examples, ascii art, whatever).

Naming it different than before, I think this formulation hits the nail on its top: Jim proposes a construction that yields early binding of names, while late binding of values. Sample layout of an association object: with the same key semantics as for dicts. The key object is assigned when the association object is created, that is when the name is seen the first time. The value is still NULL until assigned. Now assume a namespace object as a collection of pointers to asso objects, and assume that it is only extended, nothing deleted. Then, a code object can refer to such a namespace object by giving the index of the asso object. Since the construction of these objects will occour in the same order after marshal/pickling, the offsets in a code object will always be correct. This means: There is a construction that allows to settle a name as soon as it is seen, without necessarily assigning a value. When a function at compile time sees a global, it tries to resolve it by finding an association object in the module's global scope that contains this name. If not found, it is created.

[Jim Fulton]

... A change to the way that namespaces are handled could make this work and have a number of other benefits, like global name usage without namespace lookups.

I've suggested this to Guido in the past. His reasonable response is that this would be too big a change for Python 1. Maybe this is something to consider for Python 2?

The basic idea (borrowed from Smalltalk) is to have a kind of dictionary that is a collection of "association" objects. An association object is simply a pairing of a name with a value. Association objects can be shared among multiple namespaces.

Jim, I've been intrigued by this idea for all the years you've been suggesting it <wink>, but I've never understood what it is you're proposing! This is the Python-Dev list, so feel encouraged to present it in concrete implementation terms instead of ambiguous English. Or maybe an interface? interface a_kind_of_dictionary_that_is_a_collection_of_\ association_objects: # ??? beats me ... Or maybe as a C struct? For example, is "an association object" a (char*, PyObject*) pair? Does this kind of dictionary have keys? If so, of what type? What type are the values? Best I can make sense of the above, the values are "association objects", each of which contains a name and a value, and a key is maybe a duplicate of the name in the association object to which it maps. "A name" may or may not be a string -- I can't tell. Or maybe by "dictionary" you didn't intend Python's current meaning for that word at all. I assume "a value" is a PyObject*. The whole thrust *appears* to be to get names to map to a PyObject** instead of PyObject*, but if that's the ticket I don't know what association objeects have to do with it.

An import like:

from foo import spam

would copy the association between the name 'foo' and a value from module 'spam' into the current module.

Where does the idea that 'spam' is a *module* here come from? It doesn't make sense to me, and I'm so lost I'll spare everyone my further confusions <wink>. suspecting-the-last-actually-doesn't-make-any-sense<wink>-ly y'rs - tim

Christian Tismer wrote:

Tim Peters wrote:

...

[Jim Fulton]

[association objects]

...

Jim, I've been intrigued by this idea for all the years you've been suggesting it <wink>, but I've never understood what it is you're proposing! This is the Python-Dev list, so feel encouraged to present it in concrete implementation terms instead of ambiguous English. Or maybe an interface?

My guess is:

Ugh. Sorry to make you guess....

An association object adds another level of indirection to namespaces and makes global variables be more like true variables, i.e. changing them in one place changes them everywhere.

Right.

...

Does this kind of dictionary have keys? If so, of what type? What type are the values? Best I can make sense of the above, the values are "association objects", each of which contains a name and a value, and a key is maybe a duplicate of the name in the association object to which it maps. "A name" may or may not be a string -- I can't tell. Or maybe by "dictionary" you didn't intend Python's current meaning for that word at all. I assume "a value" is a PyObject*. The whole thrust *appears* to be to get names to map to a PyObject** instead of PyObject*, but if that's the ticket I don't know what association objeects have to do with it.

I don't believe that the actual implementation matters too much and is still open to be choosen. Here my approach:

Let an association object be a pair of a key and a value. The restrictions for keys may be the same as for dict keys.

We can now either use dicts as they are, inserting asso-objects as values and sharing the key field, or invent new dictionaries which have no key/value pairs at all, but just references to asso-objects.

Right, Replace dictentries with association object pointers.

In either case, we have the advantage that further references by global use from a function or by imports will always add

to the refcount of

to the asso-object, not to its value. This keeps the value changeable, like a list with one element, kind of boxed object. Since the asso-objects stay alive as long as they are referenced, they are never moved, and it is ok to refer to their address.

Yup.

For a function, this means that it can resolve a global at compile time. If the asso-object exists already, it has a fixed memory address and can be placed into the code object. If it does not exist, it can be created in the global dictionary or special asso-dictionary, whatever we'll use. The value will be NULL in this case, and this is perfect. If we do right, a value will have been inserted before the function is called, or we will raise a name error.

exactly. You are a great guesser! :)

The idea is simply to generate fixed slots for global names which never move. By mentioning the name, we create such a slot. The slot is alive as long it is seen, i.e. refcount > 0.

There must be a general way to look these things up, either by the per-module dictionary, or by a specialized one. Finally I'd tend to do the latter, since those unitialized key/value asso-objects would give ambiguity what dict.keys() should be then. For consistency, I would hide all asso-objects in a special asso-collection per module. They could be placed into the modules dict, when their value becomes assigned first time. Alternatively, they are not created at compile time but at runtime, when a value is assigned. I'm not sure yet.

Now, moving on from globals to all name spaces: If they are all handled by the asso-approach, can we use it to speed up attribute access for classes and instances? I guess we can! But I need more thought.

Yes, it needs more thought.

...

...

An import like:

from foo import spam

sentence == """

...

would copy the association between the name 'foo' and a value from module 'spam' into the current module.

Where does the idea that 'spam' is a *module* here come from? It doesn't make sense to me, and I'm so lost I'll spare everyone my further confusions <wink>. """

def swap_words(str, one, two): pieces = string.split(str, one) for i in range(len(pieces)): pieces[i] = string.replace(pieces[i], two, one) return string.join(pieces, two)

sentence = swap_words(sentence, "'foo'", "'spam'")

Ooh ooh, you've invented a 'Jim translator bot'! Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.