Re: [Python-Dev] bug or a feature?
At 03:37 AM 6/11/2008 +0200, Maciej Fijalkowski wrote:
On Wed, Jun 11, 2008 at 3:36 AM, Scott Dial <scott+python-dev@scottdial.com> wrote:
Maciej Fijalkowski wrote:
What do you think about this code:
class A: locals()[42] = 98
Seems people rely on it working.
I apologize for my ignorance, but who? Could you please cite something reputable that relies on this detail?
It's in tests of sqlalchemy. My question is among the lines "should I bug sqlalchemy guys to remove this, or should I change pypy to accept this".
That test is there to ensure that it interoperates with code using the AddOns library from the Cheeseshop; SQLAlchemy is not the source of the usage.
Phillip J. Eby wrote:
That test is there to ensure that it interoperates with code using the AddOns library from the Cheeseshop; SQLAlchemy is not the source of the usage.
Now that's interesting. The AddOns library uses class objects as keys in the __dict__, but that doesn't says anything about the usage of locals(). At no point in the AddOns library is locals() abused like this, so even if one asserts that assignment to the dict returned by locals() is a bug, the underlying behavior of interest is whether __dict__ is allowed to have non-string keys.
from peak.util.addons import AddOn class C: pass class A(AddOn): pass spam = C() print spam.__dict__ {} A(spam) print spam.__dict__ {<class 'A'>: <A object at ...>}
If non-string keys are not allowed in __dict__, then the AddOns library should be changed to add another dict to the object of interest to track these AddOn instances. -Scott -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu
"Scott Dial" <scott+python-dev@scottdial.com> wrote in message news:4850263A.3040805@scottdial.com... || If non-string keys are not allowed in __dict__, then the AddOns library | should be changed to add another dict to the object of interest to track | these AddOn instances. There are three possibilities with respect to __dict__ and non-string keys. 1. All implementations must reject such. 2. Each implementation decides for itself. 3. All implementations must allow such. Current, CPython does not reject, eliminating 1). Since, as I understand it, at least 1 other implementation does reject, 3) is also eliminated until Guido decrees otherwise and such implementation(s) change. This leaves 2) as the de facto situation, but this could be made clearer in the docs: "The result of trying to add non-string keys to any __dict__ attribute is implementation defined." This means that portable Python code must act as if 1) were the case. The Data Model chapter of the Reference Manual lists .__dict__ as a special attribute of callables, modules, classes, and instances. It describes __dict__ as a "namespace dictionary" or "implementation of the namespace" thereof. Since namespaces map names (or possibly non-name strings) to objects, this to me implies that an implementation is and should not be required to allow non-strings in __dict__. The same chapter has more than one sentence saying something like "o.x is equivalent to o.__dict__['x']". While one could read this as prohibiting o.__dict__[non_string], one could also read it as being silent, neither allowing nor prohibiting. The builtin interface functions setattr, hasattr, getattr all require strings for accessing the underlying __dict__. Since they could have been defined otherwise, to accept any object as an attribute 'name' (key), this again implies (to me) that __dict__s are only intended and only required to have string keys. Hence, I was initially surprised that 1) above was not true. So I might add something stronger to the docs, something like "The special __dict__ attributes are only intended to hold string keys. If an implementation allows other keys, that is only an current accidental side-effect of considerations of parsimony and efficiency." Contrariwise, if 3) were mandated, then I would think that the xxxattr functions should be changed. Terry Jan Reedy
On Thu, Jun 12, 2008 at 2:32 AM, Terry Reedy <tjreedy@udel.edu> wrote:
"Scott Dial" <scott+python-dev@scottdial.com> wrote in message news:4850263A.3040805@scottdial.com... || If non-string keys are not allowed in __dict__, then the AddOns library | should be changed to add another dict to the object of interest to track | these AddOn instances.
There are three possibilities with respect to __dict__ and non-string keys. 1. All implementations must reject such. 2. Each implementation decides for itself. 3. All implementations must allow such.
Current, CPython does not reject, eliminating 1). Since, as I understand it, at least 1 other implementation does reject, 3) is also eliminated until Guido decrees otherwise and such implementation(s) change. This leaves 2) as the de facto situation, but this could be made clearer in the docs: "The result of trying to add non-string keys to any __dict__ attribute is implementation defined." This means that portable Python code must act as if 1) were the case.
The Data Model chapter of the Reference Manual lists .__dict__ as a special attribute of callables, modules, classes, and instances. It describes __dict__ as a "namespace dictionary" or "implementation of the namespace" thereof. Since namespaces map names (or possibly non-name strings) to objects, this to me implies that an implementation is and should not be required to allow non-strings in __dict__.
The same chapter has more than one sentence saying something like "o.x is equivalent to o.__dict__['x']". While one could read this as prohibiting o.__dict__[non_string], one could also read it as being silent, neither allowing nor prohibiting.
The builtin interface functions setattr, hasattr, getattr all require strings for accessing the underlying __dict__. Since they could have been defined otherwise, to accept any object as an attribute 'name' (key), this again implies (to me) that __dict__s are only intended and only required to have string keys. Hence, I was initially surprised that 1) above was not true. So I might add something stronger to the docs, something like "The special __dict__ attributes are only intended to hold string keys. If an implementation allows other keys, that is only an current accidental side-effect of considerations of parsimony and efficiency."
Contrariwise, if 3) were mandated, then I would think that the xxxattr functions should be changed.
Terry Jan Reedy
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
This is completely irrelevant. This post is not about assigning non-string stuff to __dict__ of class which works completely fine. It's about abusing locals, which are not even given that they'll modify this dict.
"Maciej Fijalkowski" <fijall@gmail.com> wrote in message news:693bc9ab0806111759k14f7e672r1ebd20a85d4d5af5@mail.gmail.com... | On Thu, Jun 12, 2008 at 2:32 AM, Terry Reedy <tjreedy@udel.edu> wrote: | > | > "Scott Dial" <scott+python-dev@scottdial.com> wrote in message | > news:4850263A.3040805@scottdial.com... | > || If non-string keys are not allowed in __dict__, then the AddOns library | > | should be changed to add another dict to the object of interest to track | > | these AddOn instances. | > | > There are three possibilities with respect to __dict__ and non-string keys. | > 1. All implementations must reject such. | > 2. Each implementation decides for itself. | > 3. All implementations must allow such. | > | > Current, CPython does not reject, eliminating 1). Since, as I understand | > it, at least 1 other implementation does reject, 3) is also eliminated | > until Guido decrees otherwise and such implementation(s) change. This | > leaves 2) as the de facto situation, but this could be made clearer in the | > docs: "The result of trying to add non-string keys to any __dict__ | > attribute is implementation defined." This means that portable Python code | > must act as if 1) were the case. | > | > The Data Model chapter of the Reference Manual lists .__dict__ as a special | > attribute of callables, modules, classes, and instances. It describes | > __dict__ as a "namespace dictionary" or "implementation of the namespace" | > thereof. Since namespaces map names (or possibly non-name strings) to | > objects, this to me implies that an implementation is and should not be | > required to allow non-strings in __dict__. | > | > The same chapter has more than one sentence saying something like "o.x is | > equivalent to o.__dict__['x']". While one could read this as prohibiting | > o.__dict__[non_string], one could also read it as being silent, neither | > allowing nor prohibiting. | > | > The builtin interface functions setattr, hasattr, getattr all require | > strings for accessing the underlying __dict__. Since they could have been | > defined otherwise, to accept any object as an attribute 'name' (key), this | > again implies (to me) that __dict__s are only intended and only required to | > have string keys. Hence, I was initially surprised that 1) above was not | > true. So I might add something stronger to the docs, something like "The | > special __dict__ attributes are only intended to hold string keys. If an | > implementation allows other keys, that is only an current accidental | > side-effect of considerations of parsimony and efficiency." | > | > Contrariwise, if 3) were mandated, then I would think that the xxxattr | > functions should be changed. | This is completely irrelevant. This post is not about assigning | non-string stuff to __dict__ of class which works completely fine. My apologies for clipping too much of Scott's post. Here is the rest that came before what I quoted, which makes clear, from first sentence to last line, that *he* *is* talking about assigning non-string stuff to __dict__ of a class. "The AddOns library uses class objects as keys in the __dict__, but that doesn't says anything about the usage of locals(). At no point in the AddOns library is locals() abused like this, so even if one asserts that assignment to the dict returned by locals() is a bug, the underlying behavior of interest is whether __dict__ is allowed to have non-string keys.
from peak.util.addons import AddOn class C: pass class A(AddOn): pass spam = C() print spam.__dict__ {} A(spam) print spam.__dict__ {<class 'A'>: <A object at ...>} "
Whether non-strings keys in o.__dict__ 'works fine' depends on one's definition of 'works fine'. In any case, as of 3.0a5, I thinks the docs could better clarify the status of *this* 'feature'. Whatever pronouncement Guido has made has not, that I could find, made it in. | It's about abusing locals, which are not even given that they'll | modify this dict. I thought it settled that that is a bad thing to do. Here the doc is pretty clear. tjr
Maciej Fijalkowski wrote:
On Thu, Jun 12, 2008 at 2:32 AM, Terry Reedy <tjreedy@udel.edu> wrote:
"Scott Dial" <scott+python-dev@scottdial.com> wrote in message news:4850263A.3040805@scottdial.com... || If non-string keys are not allowed in __dict__, then the AddOns library | should be changed to add another dict to the object of interest to track | these AddOn instances.
There are three possibilities with respect to __dict__ and non-string keys. 1. All implementations must reject such. 2. Each implementation decides for itself. 3. All implementations must allow such.
Current, CPython does not reject, eliminating 1). Since, as I understand it, at least 1 other implementation does reject, 3) is also eliminated until Guido decrees otherwise and such implementation(s) change. This leaves 2) as the de facto situation, but this could be made clearer in the docs: "The result of trying to add non-string keys to any __dict__ attribute is implementation defined." This means that portable Python code must act as if 1) were the case.
This is completely irrelevant. This post is not about assigning non-string stuff to __dict__ of class which works completely fine. It's about abusing locals, which are not even given that they'll modify this dict.
Not withstanding Terry's respond, this is not as off-topic as you make it out to be. The test case you cited is in fact test this exact 'feature'. And as Terry expounded on it, it was unclear to me whether that was even of allowed. The only reason the test used locals() was because it was the only way to insert a non-string key into the class namespace.
class A: ... locals()[42] = 98 A.__dict__ {'__module__': '__main__', 43: 1, '__doc__': None}
locals() has to be used because __dict__ is unavailable at definition.
class A: ... __dict__[42] = 98 NameError: name '__dict__' is not defined
So, while one can write:
class A: pass a.__dict__[42] = 98
But that's not quite the same. Nevertheless, it was still unclear whether there was any pronouncements on non-string keys. Sorry for wasting your time. -Scott -- Scott Dial scott@scottdial.com scodial@cs.indiana.edu
Hi, On Wed, Jun 11, 2008 at 10:44:17PM -0400, Scott Dial wrote:
The only reason the test used locals() was because it was the only way to insert a non-string key into the class namespace.
This discussion is mistakenly focused on locals(). There is a direct way to have arbitrary keys in the dict of a type:
MyClass = type('MyClass', (Base,), {42: 64}) MyClass.__dict__[42] 64
There is, however, no way to modify or add non-string keys in the type after its creation. So the question is whether the type() constructor is allowed to fail with a TypeError when the initial dict contains non-string keys (this is PyPy's current behavior). A bientot, Armin.
On Thu, Jun 12, 2008 at 8:24 AM, Armin Rigo <arigo@tunes.org> wrote:
This discussion is mistakenly focused on locals(). There is a direct way to have arbitrary keys in the dict of a type:
MyClass = type('MyClass', (Base,), {42: 64}) MyClass.__dict__[42] 64
There is, however, no way to modify or add non-string keys in the type after its creation. So the question is whether the type() constructor is allowed to fail with a TypeError when the initial dict contains non-string keys (this is PyPy's current behavior).
The intention was for these dicts to be used as namespaces. I think of it as follows: (a) Using non-string keys is a no-no, but the implementation isn't required to go out of its way to forbid it. (b) Using non-empty string keys that aren't well-formed identifiers should be allowed. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
On Thu, Jun 12, 2008 at 9:46 PM, Guido van Rossum <guido@python.org> wrote:
The intention was for these dicts to be used as namespaces. I think of it as follows:
(a) Using non-string keys is a no-no, but the implementation isn't required to go out of its way to forbid it.
That will allow easier and more efficient implementation, good!
(b) Using non-empty string keys that aren't well-formed identifiers should be allowed.
ok. Is it allowed to "normalize" subclasses of strings to regular string, e.g. after: class mystring(str): pass class C: pass x = C() setattr(x, mystring('foo'), 42) is it allowed that the dict of x contains a regular string 'foo' instead of the mystring instance? - Willem
On Thu, Jun 12, 2008 at 1:01 PM, Willem Broekema <metawilm@gmail.com> wrote:
On Thu, Jun 12, 2008 at 9:46 PM, Guido van Rossum <guido@python.org> wrote:
The intention was for these dicts to be used as namespaces. I think of it as follows:
(a) Using non-string keys is a no-no, but the implementation isn't required to go out of its way to forbid it.
That will allow easier and more efficient implementation, good!
(b) Using non-empty string keys that aren't well-formed identifiers should be allowed.
ok.
Is it allowed to "normalize" subclasses of strings to regular string, e.g. after:
class mystring(str): pass class C: pass
x = C() setattr(x, mystring('foo'), 42)
is it allowed that the dict of x contains a regular string 'foo' instead of the mystring instance?
I think yes, as this would allow for a more efficient implementation of the custom dict class. -- --Guido van Rossum (home page: http://www.python.org/~guido/)
At 08:32 PM 6/11/2008 -0400, Terry Reedy wrote:
The Data Model chapter of the Reference Manual lists .__dict__ as a special attribute of callables, modules, classes, and instances. It describes __dict__ as a "namespace dictionary" or "implementation of the namespace" thereof. Since namespaces map names (or possibly non-name strings) to objects, this to me implies that an implementation is and should not be required to allow non-strings in __dict__.
The same chapter has more than one sentence saying something like "o.x is equivalent to o.__dict__['x']". While one could read this as prohibiting o.__dict__[non_string], one could also read it as being silent, neither allowing nor prohibiting.
As it happens, most objects' __dict__ slots are settable by default, and *require* that you set it to a dict or subclass thereof. This seems (to me) to imply that a standard dictionary (i.e. one supporting keys of any type) is required. (In the sense that a dict subclass which rejects non-strings would be violating the Liskov principle.)
Phillip J. Eby wrote:
At 08:32 PM 6/11/2008 -0400, Terry Reedy wrote:
The Data Model chapter of the Reference Manual lists .__dict__ as a special attribute of callables, modules, classes, and instances. It describes __dict__ as a "namespace dictionary" or "implementation of the namespace" thereof. Since namespaces map names (or possibly non-name strings) to objects, this to me implies that an implementation is and should not be required to allow non-strings in __dict__.
The same chapter has more than one sentence saying something like "o.x is equivalent to o.__dict__['x']". While one could read this as prohibiting o.__dict__[non_string], one could also read it as being silent, neither allowing nor prohibiting.
As it happens, most objects' __dict__ slots are settable by default, and *require* that you set it to a dict or subclass thereof.
This is wrong for types: $ python Python 2.5.2 (r252:60911, Apr 21 2008, 11:12:42) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Type "help", "copyright", "credits" or "license" for more information.
class A(object): pass ... A.__dict__ = {} Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: attribute '__dict__' of 'type' objects is not writable
In fact it seems to me that there is no way to set non-string keys into a type dict after class creation: Cls.__dict__ supports no setitem, setattr checks for a string argument. I think there are good arguments for not allowing strings keys in type dicts, or at least leaving it up to the implementation. Using non-string keys in type dicts is relatively awkward and allowing them makes many interesting optimizations (like method caches) a _lot_ harder to get right. Cheers, Carl Friedrich Bolz
At 01:34 PM 6/12/2008 +0200, Carl Friedrich Bolz wrote:
Phillip J. Eby wrote:
As it happens, most objects' __dict__ slots are settable by default, and *require* that you set it to a dict or subclass thereof.
This is wrong for types:
Which is why I said "most" - to exclude types, and objects that don't have a __dict__ slot to begin with.
I think there are good arguments for not allowing strings keys in type dicts, or at least leaving it up to the implementation.
That may well be, but there is nothing in Python's spec that I'm aware of that *forbids* it. For example the type() constructor doc doesn't say anything about using string-only keys in the class dictionary.
Using non-string keys in type dicts is relatively awkward and allowing them makes many interesting optimizations (like method caches) a _lot_ harder to get right.
Really? Why? Having non-string dict keys is NOT the same thing as having non-string attribute names, so attribute name lookups should be unaffected. (Attribute names *are* required to be strings, and -- as far as I know -- always have been.)
Phillip J. Eby wrote:
At 01:34 PM 6/12/2008 +0200, Carl Friedrich Bolz wrote:
Phillip J. Eby wrote:
As it happens, most objects' __dict__ slots are settable by default, and *require* that you set it to a dict or subclass thereof.
This is wrong for types:
Which is why I said "most" - to exclude types, and objects that don't have a __dict__ slot to begin with.
Sorry, I thought we were mostly discussing type dictionaries at the moment.
I think there are good arguments for not allowing strings keys in type dicts, or at least leaving it up to the implementation.
That may well be, but there is nothing in Python's spec that I'm aware of that *forbids* it. For example the type() constructor doc doesn't say anything about using string-only keys in the class dictionary.
Fine, but as far as I can see there is also nothing that *mandates* it.
Using non-string keys in type dicts is relatively awkward and allowing them makes many interesting optimizations (like method caches) a _lot_ harder to get right.
Really? Why? Having non-string dict keys is NOT the same thing as having non-string attribute names, so attribute name lookups should be unaffected. (Attribute names *are* required to be strings, and -- as far as I know -- always have been.)
Of course attribute name lookups are affected, because you can have a non-string key that has a __hash__ and __eq__ method to make it look sufficiently like a string to the dict. Then the attribute lookup needs to take these into account. This makes a method cache implementation much more complex, because suddenly arbitrary user code can run during the lookup and your classes (and thus your method caches) can change under your feed during the lookup. In this situation getting the corner cases right is very hard. Cheers, Carl Friedrich
On Thu, Jun 12, 2008 at 2:56 PM, Carl Friedrich Bolz <cfbolz@gmx.de> wrote:
Of course attribute name lookups are affected, because you can have a non-string key that has a __hash__ and __eq__ method to make it look sufficiently like a string to the dict. Then the attribute lookup needs to take these into account. This makes a method cache implementation much more complex, because suddenly arbitrary user code can run during the lookup and your classes (and thus your method caches) can change under your feed during the lookup. In this situation getting the corner cases right is very hard.
I fully agree with this assessment: the theoretical possibilities make attribute handling very hairy. Note that besides non-strings, also user-defined subclasses of string can have their own hash/eq logic. If namespace keys and attributes could be locked down to only be regular strings, not of another type and not a string subclass instance, then implementing attributes becomes a lot less hairy, and optimizations much more straightforward. - Willem (CLPython)
"Phillip J. Eby" <pje@telecommunity.com> wrote in message news:20080612100048.7EEEA3A405F@sparrow.telecommunity.com... | At 08:32 PM 6/11/2008 -0400, Terry Reedy wrote: | >The Data Model chapter of the Reference Manual lists .__dict__ as a special | >attribute of callables, modules, classes, and instances. It describes | >__dict__ as a "namespace dictionary" or "implementation of the namespace" | >thereof. Since namespaces map names (or possibly non-name strings) to | >objects, this to me implies that an implementation is and should not be | >required to allow non-strings in __dict__. | > | >The same chapter has more than one sentence saying something like "o.x is | >equivalent to o.__dict__['x']". While one could read this as prohibiting | >o.__dict__[non_string], one could also read it as being silent, neither | >allowing nor prohibiting. | | As it happens, most objects' __dict__ slots are settable by default, | and *require* that you set it to a dict or subclass thereof. This | seems (to me) to imply that a standard dictionary (i.e. one | supporting keys of any type) is required. (In the sense that a dict | subclass which rejects non-strings would be violating the Liskov principle.) Is this requirement a documented Python requirement (not that I found), an undocumented Python requirement, or a CPython-specific requirement (that other implementations may freely ignore)? I don't have much opinion on what the rules for __dict__ attributes should be, just that whatever they are should be documented. This include implementation-defined aspects. This will help both implementors and users who wish to write Python code rather that CPython code. If there is a consensus and BDFL pronouncement, I will make or help review suggested doc changes. tjr
participants (8)
-
Armin Rigo -
Carl Friedrich Bolz -
Guido van Rossum -
Maciej Fijalkowski -
Phillip J. Eby -
Scott Dial -
Terry Reedy -
Willem Broekema