[issue35933] python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict)
New submission from Pierre Glaser <pierreglaser@msn.com>: Hello all, This 16-year old commit (*) allows an object's state to be updated using its slots instead of its __dict__ at unpickling time. To use this functionality, the state keyword-argument of Pickler.save_reduce (which maps to the third item of the tuple returned by __reduce__) should be a length-2 tuple. As far as I can tell, this is not mentioned in the documentation (**). I suggest having the docs updated. What do you think? (*) https://github.com/python/cpython/commit/ac5b5d2e8b849c499d323b0263ace22e56b... (**) https://docs.python.org/3.8/library/pickle.html#object.__reduce__ ---------- assignee: docs@python components: Documentation messages: 335031 nosy: alexandre.vassalotti, docs@python, pierreglaser, pitrou, serhiy.storchaka priority: normal severity: normal status: open title: python doc does not say that the state kwarg in Pickler.save_reduce can be a tuple (and not only a dict) versions: Python 3.8 _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Antoine Pitrou <pitrou@free.fr> added the comment: Does it still work? With both the C and Python pickler? Can you post an example? ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Pierre Glaser <pierreglaser@msn.com> added the comment: It turns out that both pickle and _pickle implement this feature, but the behavior is inconsistent. - As a reminder, instances of slotted classes do not have a dict attribute (1) - On the other side, when pickling slotted class instances, __getstate__ can return a tuple of 2 dicts. The first dict represents the __dict__ attribute. Because of (1), this first dict should simply be a sentinel value. In pickle, the condition is that it evaluates to False, but in _pickle, it should be explicitly None. (- Finally, The second dict in state contains the slotted attribute. ) Here are the lines in the two files causing the inconsistent behavior: https://github.com/python/cpython/blob/master/Modules/_pickle.c#L6236 https://github.com/python/cpython/blob/master/Lib/pickle.py#L1549 I included an example that illustrates it. ---------- Added file: https://bugs.python.org/file48111/test_slots.py _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Change by Pierre Glaser <pierreglaser@msn.com>: Removed file: https://bugs.python.org/file48111/test_slots.py _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Change by Pierre Glaser <pierreglaser@msn.com>: Added file: https://bugs.python.org/file48112/test_slots.py _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Change by Pierre Glaser <pierreglaser@msn.com>: Removed file: https://bugs.python.org/file48112/test_slots.py _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Change by Pierre Glaser <pierreglaser@msn.com>: Added file: https://bugs.python.org/file48113/test_slots.py _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Antoine Pitrou <pitrou@free.fr> added the comment: You can have both a dict and slots by subclassing:
class A: ...: __slots__ = ('x',) ...: class B(A): pass
b = B() b.x = 5 b.y = 6 b.__dict__ {'y': 6} A.x <member 'x' of 'A' objects> A.x.__get__(b) 5
---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Raymond Hettinger <raymond.hettinger@gmail.com> added the comment: Interestingly, you can also put an instance dict in slots:
class A: __slots__ = ['x', '__dict__']
a = A() a.x = 5 a.y = 6 a.__dict__ {'y': 6} a.x 5
---------- nosy: +rhettinger _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Pierre Glaser <pierreglaser@msn.com> added the comment: Thanks Antoine and Raymond for the feedback. Indeed, a subclass of a slotted class can have a dict: I enriched the script, pickling_depickling instances of such subclasses, with the length-2 tuple __getstate__ method, and made sure their attributes were properly retrieved. Apart from the different checks on state carried out in the c load_build and the python load_build, AFAICT, it seems like this feature works :) ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Change by Pierre Glaser <pierreglaser@msn.com>: Removed file: https://bugs.python.org/file48113/test_slots.py _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Change by Pierre Glaser <pierreglaser@msn.com>: Added file: https://bugs.python.org/file48114/test_slots.py _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Change by Pierre Glaser <pierreglaser@msn.com>: ---------- keywords: +patch pull_requests: +11981 stage: -> patch review _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Serhiy Storchaka <storchaka+cpython@gmail.com> added the comment: A slotted class will have a dict also when it inherits it from a non-slotted class. This is why the base class of slotted class should have slots if you do not want an instance dict. __getstate__ and __setstate__ for slotted classes are described in PEP 307. Unfortunately this was not copied to the module documentation. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Pierre Glaser <pierreglaser@msn.com> added the comment: I added a PR with a small patch to document this behavior and reconcile _pickle.c and pickle.py Some explanations on why I am pushing this forward: Pickling instances of classes/subclasses with slots is done natively for pickle protocol >= 2. Mentioning this behavior in the docs should *not* make the user worry about implementing custom __getstate__ methods just to preserve slots. Here is the reason why I think this functionality (allowing state and slotstate) is worth documenting: pickle gives us a lot of flexibility for reducing thanks to the dispatch_table. But unpickling is rather rigid: if no __setstate__ exists, we have to deal with the default state updating procedure present in load_build. Might as well document all of it ;) ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Serhiy Storchaka <storchaka+cpython@gmail.com> added the comment: See also issue26579 which propose to add a function or method for standard implementation of __getstate__ and use it consistently in custom __getstate__ implementations. I am not sure about API yet. ---------- _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
Change by Pierre Glaser <pierreglaser@msn.com>: ---------- stage: patch review -> resolved status: open -> closed _______________________________________ Python tracker <report@bugs.python.org> <https://bugs.python.org/issue35933> _______________________________________
participants (4)
-
Antoine Pitrou -
Pierre Glaser
-
Raymond Hettinger
-
Serhiy Storchaka