Support other dict types for type.__dict__
Hi, I'm trying to create read-only objects using a "frozendict" class. frozendict is a read-only dict. I would like to use frozendict for the class dict using a metaclass, but type.__new__() expects a dict and creates a copy of the input dict. I would be nice to support custom dict type: OrderedDict and frozendict for example. It looks possible to patch CPython to implement this feature, but first I would like first to know your opinion about this idea :-) Victor
And you can't use __slots__ because...?
Hum, here is an example: --- def Enum(**kw): class _Enum(object): __slots__ = list(kw.keys()) def __new__(cls, **kw): inst = object.__new__(cls) for key, value in kw.items(): setattr(inst, key, value) return inst return _Enum(**kw) components = Enum(red=0, green=1, blue=2) print(components.red) components.red=2 print(components.red) components.unknown=10 --- components.unknown=10 raises an error, but not components.red=2. __slots__ denies to add new attributes, but not to modify existing attributes. The idea of using a frozendict is to deny the modification of an attribute value after the creation of the object. I don't see how to use __slots__ to implement such constraints. Victor
On 02/24/2012 01:27 AM, Victor Stinner wrote:
class Enum(object): __slots__ = ("_data",) _data = WriteOnceDescr('_data') # left as exercise def __init__(self, **kw): self._data = frozendict(kw) def __getattr__(self, key): try: return self._data[key] except KeyError: raise AttributeError(key)
On Fri, Feb 24, 2012 at 9:34 AM, Victor Stinner <victor.stinner@haypocalc.com> wrote:
Do you have a particular reason for doing it that way rather than just overriding __setattr__ and __delattr__ to raise TypeError? Or overriding the __dict__ descriptor to return a read-only proxy? There are a *lot* of direct calls to the PyDict APIs in the object machinery. Without benchmark results clearly showing a negligible speed impact, I'd be -1 on increasing the complexity of all that code (and making it slower) to support niche use cases that can already be handled a couple of other ways. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 24/02/2012 06:33, David Townshend a écrit :
Can't this also be done using metaclasses?
Hi, Are you thinking of __prepare__? I did to but I read the details of this: http://docs.python.org/py3k/reference/datamodel.html#customizing-class-creat... The class body can be executed "in" any mapping. Then I’m not sure but it looks like type.__new__ only takes a real dict. You have to do something in your overridden __new__ to eg. keep the OrderedDict’s order. Regards, -- Simon Sapin
Can't this also be done using metaclasses?
Yes, my current proof-of-concept (PoC) uses a metadata with __prepare__.
type.__new__ accepts any class inheriting from dict. My frozendict PoC inherits from dict, so it just works. But the point is that type.__new__ makes a copy of the dict and later it is no more possible to replace the dict. I would like to be able to choose the type of the __dict__ of my class. Victor
On Fri, Feb 24, 2012 at 4:08 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I also think about reverse process of removing things that were proved to be underused. That probably requires AST spider that crawls existing Python project to see how various constructs are used. -- anatoly t.
Le 24/02/2012 00:34, Victor Stinner a écrit :
Hi, Combining ideas from other messages in this thread: would this work? 1. Inherit from frozendict 2. Define a __getattr__ that defers to frozendict.__getitem__ 3. Use an empty __slots__ so that there is no "normal" instance attribute. Thinking about it a bit more, it’s probably the same as having a normal __dict__ and raising in __setattr__ and __delattr__. Isn’t this how you implement frozendict? (Raise in __setitem__, __delitem__, update, etc.) Regards, -- Simon Sapin
On Feb 26, 2012 10:27 AM, "Simon Sapin" <simon.sapin@kozea.fr> wrote:
Using frozendict, and especially inheriting from it sounds unnecessarily complicated to me. A simple class which doesn't allow changes to instance attributes could be implemented something like this: class Three_inst: @property def value(self): return 3 def __setattr__(self, attr, value): raise AttributeError def __delattr__(self, attr): raise AttributeError Or, if you're worried about changes to the class attributes, you could do basically the same thing using a metaclass (python 2.7 syntax): class FinalMeta(type): def __setattr__(cls, attr, value): if attr in cls.__dict__ or '__done__' in cls.__dict__: raise AttributeError else: type.__setattr__(cls, attr, value) def __delattr__(cls, attr): raise AttributeError class Three: __metaclass__ = FinalMeta value = 3 __done__ = True # There may be a neater way to do this... Each of the following examples will fail:
Actually, I think this is quite a nice illustration of what can be done with metaclasses! David
Ah, I think I misunderstood exactly what you were trying to achieve. To me, that is essentially immutable - if I ever found myself using type.__setattr__ to change a variable I'd have to seriously question what I was doing! But that would be a way around it, and I don't think it would be possible to implement it fully in python. On the other hand, the same argument could be made for the introduction of private variables; Class.__var is not private because it can be changed through Class._Class__var. I'd also consider having to do this to be indicative of a design flaw in my code. On Sun, Feb 26, 2012 at 3:56 PM, Victor Stinner < victor.stinner@haypocalc.com> wrote:
Le 26/02/2012 14:56, Victor Stinner a écrit :
type.__setattr__(Three, 'value', 4) changes the value.
Then there is the question of how much craziness you want to protect from. Nothing is ever truly private or immutable in CPython, given enough motivation and ctypes. See for example Armin Ronacher’s "Bad Ideas" presentation, especially the "Interpreter Warfare" part near the end: https://ep2012.europython.eu/media/conference/slides/5-years-of-bad-ideas.pd... I think that the code patching tracebacks is in production in Jinja2. I’m sure frozensets could be modified in a similar way. The point is: immutable data types protect against mistakes more than someone truly determined to break the rules. With that in mind, I think that having to go through __setattr__ is good enough to make sure it’s not accidental. Regards, -- Simon Sapin
My pysandbox project uses various hacks to secure Python. The attacker doesn't care of writing pythonic code, (s)he just want to break the sandbox :-) See my pysandbox project for more information: https://github.com/haypo/pysandbox/ See sandbox/test/ if you like weird code :-) Tests ensure that the sandbox is safe. Constant types would also help optimization, especially PyPy JIT. Victor
And you can't use __slots__ because...?
Hum, here is an example: --- def Enum(**kw): class _Enum(object): __slots__ = list(kw.keys()) def __new__(cls, **kw): inst = object.__new__(cls) for key, value in kw.items(): setattr(inst, key, value) return inst return _Enum(**kw) components = Enum(red=0, green=1, blue=2) print(components.red) components.red=2 print(components.red) components.unknown=10 --- components.unknown=10 raises an error, but not components.red=2. __slots__ denies to add new attributes, but not to modify existing attributes. The idea of using a frozendict is to deny the modification of an attribute value after the creation of the object. I don't see how to use __slots__ to implement such constraints. Victor
On 02/24/2012 01:27 AM, Victor Stinner wrote:
class Enum(object): __slots__ = ("_data",) _data = WriteOnceDescr('_data') # left as exercise def __init__(self, **kw): self._data = frozendict(kw) def __getattr__(self, key): try: return self._data[key] except KeyError: raise AttributeError(key)
On Fri, Feb 24, 2012 at 9:34 AM, Victor Stinner <victor.stinner@haypocalc.com> wrote:
Do you have a particular reason for doing it that way rather than just overriding __setattr__ and __delattr__ to raise TypeError? Or overriding the __dict__ descriptor to return a read-only proxy? There are a *lot* of direct calls to the PyDict APIs in the object machinery. Without benchmark results clearly showing a negligible speed impact, I'd be -1 on increasing the complexity of all that code (and making it slower) to support niche use cases that can already be handled a couple of other ways. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 24/02/2012 06:33, David Townshend a écrit :
Can't this also be done using metaclasses?
Hi, Are you thinking of __prepare__? I did to but I read the details of this: http://docs.python.org/py3k/reference/datamodel.html#customizing-class-creat... The class body can be executed "in" any mapping. Then I’m not sure but it looks like type.__new__ only takes a real dict. You have to do something in your overridden __new__ to eg. keep the OrderedDict’s order. Regards, -- Simon Sapin
Can't this also be done using metaclasses?
Yes, my current proof-of-concept (PoC) uses a metadata with __prepare__.
type.__new__ accepts any class inheriting from dict. My frozendict PoC inherits from dict, so it just works. But the point is that type.__new__ makes a copy of the dict and later it is no more possible to replace the dict. I would like to be able to choose the type of the __dict__ of my class. Victor
On Fri, Feb 24, 2012 at 4:08 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
I also think about reverse process of removing things that were proved to be underused. That probably requires AST spider that crawls existing Python project to see how various constructs are used. -- anatoly t.
Le 24/02/2012 00:34, Victor Stinner a écrit :
Hi, Combining ideas from other messages in this thread: would this work? 1. Inherit from frozendict 2. Define a __getattr__ that defers to frozendict.__getitem__ 3. Use an empty __slots__ so that there is no "normal" instance attribute. Thinking about it a bit more, it’s probably the same as having a normal __dict__ and raising in __setattr__ and __delattr__. Isn’t this how you implement frozendict? (Raise in __setitem__, __delitem__, update, etc.) Regards, -- Simon Sapin
On Feb 26, 2012 10:27 AM, "Simon Sapin" <simon.sapin@kozea.fr> wrote:
Using frozendict, and especially inheriting from it sounds unnecessarily complicated to me. A simple class which doesn't allow changes to instance attributes could be implemented something like this: class Three_inst: @property def value(self): return 3 def __setattr__(self, attr, value): raise AttributeError def __delattr__(self, attr): raise AttributeError Or, if you're worried about changes to the class attributes, you could do basically the same thing using a metaclass (python 2.7 syntax): class FinalMeta(type): def __setattr__(cls, attr, value): if attr in cls.__dict__ or '__done__' in cls.__dict__: raise AttributeError else: type.__setattr__(cls, attr, value) def __delattr__(cls, attr): raise AttributeError class Three: __metaclass__ = FinalMeta value = 3 __done__ = True # There may be a neater way to do this... Each of the following examples will fail:
Actually, I think this is quite a nice illustration of what can be done with metaclasses! David
Ah, I think I misunderstood exactly what you were trying to achieve. To me, that is essentially immutable - if I ever found myself using type.__setattr__ to change a variable I'd have to seriously question what I was doing! But that would be a way around it, and I don't think it would be possible to implement it fully in python. On the other hand, the same argument could be made for the introduction of private variables; Class.__var is not private because it can be changed through Class._Class__var. I'd also consider having to do this to be indicative of a design flaw in my code. On Sun, Feb 26, 2012 at 3:56 PM, Victor Stinner < victor.stinner@haypocalc.com> wrote:
Le 26/02/2012 14:56, Victor Stinner a écrit :
type.__setattr__(Three, 'value', 4) changes the value.
Then there is the question of how much craziness you want to protect from. Nothing is ever truly private or immutable in CPython, given enough motivation and ctypes. See for example Armin Ronacher’s "Bad Ideas" presentation, especially the "Interpreter Warfare" part near the end: https://ep2012.europython.eu/media/conference/slides/5-years-of-bad-ideas.pd... I think that the code patching tracebacks is in production in Jinja2. I’m sure frozensets could be modified in a similar way. The point is: immutable data types protect against mistakes more than someone truly determined to break the rules. With that in mind, I think that having to go through __setattr__ is good enough to make sure it’s not accidental. Regards, -- Simon Sapin
My pysandbox project uses various hacks to secure Python. The attacker doesn't care of writing pythonic code, (s)he just want to break the sandbox :-) See my pysandbox project for more information: https://github.com/haypo/pysandbox/ See sandbox/test/ if you like weird code :-) Tests ensure that the sandbox is safe. Constant types would also help optimization, especially PyPy JIT. Victor
participants (7)
-
anatoly techtonik -
Chris Rebert -
David Townshend -
Nick Coghlan -
Ronny Pfannschmidt -
Simon Sapin -
Victor Stinner