slots, properties, descriptors, and pydoc
Hello, I'd like to make some observations and proposals about pydoc regarding slots, properties, and descriptors in general. Given this program: class CountingDescriptor(object): def __init__(self, doc=None): self.val = 0 self.__doc__ = doc def __get__(self, obj, objtype): self.val += 1 return self.val class Foo(object): __slots__ = {'bar': 'slot doc test'} counter = CountingDescriptor(doc='descriptor doc test') def _set_counter_multiple(self, val): print 'I hear you want to set me to %s?' % val counter_multiple = property( lambda self: self.counter*2, _set_counter_multiple, doc='property doc test' ) print help(Foo) The output is: Help on class Foo in module __main__: class Foo(__builtin__.object) | Properties defined here: | | counter_multiple | property doc test | | lambdaself | | <set> = _set_counter_multiple(self, val) | | ------------------------------------------------------------ | Data and other attributes defined here: | | __slots__ = {'bar': 'slot doc test'} | | bar = <member 'bar' of 'Foo' objects> | | counter = 2 Observations (in order from least to most arguable :-): 1) pydoc's property display doesn't handle lambda well 2) pydoc doesn't show the doc string of general descriptors 3) Using dictionary values as a way to set the doc string of slots, as suggested in Guido's "Unifying types and classes", has not been implemented. Currently there is no way to set a doc string on a descriptor generated by __slots__. Even if you could, pydoc would not display it, according to #2. 4) pydoc is displaying implementation details of properties (set and get method names, etc.). Such details should probably not be displayed, similar to underbar-prefixed method and data attributes. Proposals: 1) remove property-specific handling from pydoc 2) extend pydoc with descriptor-specific handling. In a section called "Descriptors defined here", list each attribute name, with associated doc string if it exists. 3) extend the __slots__ handler to set the descriptor's doc string to dictionary values if the slot's value is a dictionary. This is the only Python change being proposed. 4) have pydoc make a special case for the display of __slots__. Alternatives: A) don't display __slots__ if the value is a dictionary. The display of __slots__ is redundant, because all items are displayed according to #2. B) if for some reason distinguishing slots from other descriptors is essential, then always display __slots__, but normalize it to a list of attribute names (show only dictionary keys, etc.) With these changes, the new output of the example would be: Help on class Foo in module __main__: class Foo(__builtin__.object) | Descriptors defined here: | | bar | slot doc test | | counter | descriptor doc test | | counter_multiple | property doc test I'd be willing to work on a PEP. If these issues are covered in or conflict with an existing PEP, please let me know. Thanks, -John -- http:// if ile.org/
I've implemented the proposal, as far as the pydoc modifications. See <http://sourceforge.net/tracker/index.php?func=detail&aid=936774&group_id=5470&atid=305470>.
3) extend the __slots__ handler to set the descriptor's doc string to dictionary values if the slot's value is a dictionary.
I haven't implemented this, because I don't know a thing about Python's C interface. If someone more able would knock off a prototype, I would be grateful. -John -- http:// if ile.org/
3) extend the __slots__ handler to set the descriptor's doc string to dictionary values if the slot's value is a dictionary.
I haven't implemented this, because I don't know a thing about Python's C interface. If someone more able would knock off a prototype, I would be grateful.
If I understand this proposal correctly (the dict values would provide the docstrings for the slots) I'd rather not make this default behavior; different metaclasses may want to use the dict value for different purposes, and subsuming it by default for the doc string would make that harder to do. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
3) extend the __slots__ handler to set the descriptor's doc string to dictionary values if the slot's value is a dictionary.
I haven't implemented this, because I don't know a thing about Python's C interface. If someone more able would knock off a prototype, I would be grateful.
If I understand this proposal correctly (the dict values would provide the docstrings for the slots) I'd rather not make this default behavior; different metaclasses may want to use the dict value for different purposes, and subsuming it by default for the doc string would make that harder to do.
Ok. Can anyone suggest an unobtrusive way to set docstrings on slots? Perhaps this (but how could it be implemented?): class Foo(object): __slots__ = docslots( ('slot1': 'description'), ('slot2': """description ...continued """), 'slot3', ) -John -- http:// if ile.org/
On Sunday 18 April 2004 08:45 am, John Belmonte wrote:
Perhaps this (but how could it be implemented?):
class Foo(object): __slots__ = docslots( ('slot1': 'description'), ('slot2':
How about: class Foo(object): docslots(slot1='description', slot2=("Some really long\n" "multi-line description."), ...) The implementation for docslots() would either provide or update __slots__, and defining properities for slots with munged names. This wouldn't actually be difficult to implement, though is would require a sys._getframe() hack. In fact, you could crib from the attached sample. ;-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
Fred L. Drake, Jr. wrote:
"""Sample implementation of the docslots idea."""
import sys
class slotproperty(object): def __init__(self, name, docstring): self.__name__ = name self.__doc__ = docstring self._slotname = "_slotproperty_" + name
def __get__(self, inst, cls=None): return getattr(inst, self._slotname)
def __set__(self, inst, value): setattr(inst, self._slotname, value)
def docslots(**kw): namespace = sys._getframe(1).f_locals __slots__ = namespace.get("__slots__", ()) __slots__ = list(__slots__) for name, docstring in kw.iteritems(): prop = slotproperty(name, docstring) if name in __slots__: __slots__.remove(name) __slots__.append(prop._slotname) namespace[name] = prop namespace["__slots__"] = tuple(__slots__)
Thanks, that at least gives me a framework to try out some ideas. I'm concerned about something though. Doesn't this implementation impose an overhead on access of slots with doc strings, or can a C implementation be made just as efficient as normal slots? I'm also wondering about Guido's comment. Even if the __slots__ handler were extended to handle docstrings directly via dict values, wouldn't metaclasses still be free to intercept the dict for other uses? -John -- http:// if ile.org/
On Sunday 18 April 2004 11:35 am, John Belmonte wrote:
I'm concerned about something though. Doesn't this implementation impose an overhead on access of slots with doc strings, or can a C implementation be made just as efficient as normal slots?
This implementation does, but a C implementation in the core could avoid that easily enough. It may even be that this could be avoided using a Python implementation by generating an __slots__ that isn't munged so heavily and then playing metaclass tricks to cause the docstrings to be inserted directly on the descriptors generated by the current __slots__ machinery. I wasn't too worried about that for the example code, but you're certainly free to construct something that suits you. I think it would be more interesting to create slot-based properties without the renaming currently needed, but I'm not sure how to do that off-hand. If the slot property type were available for subclassing it would be pretty easy, I'd hope.
I'm also wondering about Guido's comment. Even if the __slots__ handler were extended to handle docstrings directly via dict values, wouldn't metaclasses still be free to intercept the dict for other uses?
Metaclasses can do what they want to affect the new type. That's what they're for. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
Fred L. Drake, Jr. wrote:
How about:
class Foo(object): docslots(slot1='description', slot2=("Some really long\n" "multi-line description."), ...)
My intention was to make something that can completely replace the current __slots__ pattern. Your proposal doesn't support slots that don't have a docstring very naturally (you'd use slot=''). Attached is another strategy that uses optional tuples. Combined with my pydoc patch, class Foo(object): slots( ('slot1', 'description'), ('slot2', """description ...continued"""), 'slot3', ('_slot4', 'hidden'), ) yields this help: class Foo(__builtin__.object) | Data descriptors defined here: | | slot1 | description | slot2 | description | ...continued | slot3 -John -- http:// if ile.org/
On Sunday 18 April 2004 11:16 pm, John Belmonte wrote:
My intention was to make something that can completely replace the current __slots__ pattern. Your proposal doesn't support slots that don't have a docstring very naturally (you'd use slot='').
Please don't misunderstand; I wasn't making a proposal, but was just responding to your question about what can be done. I've no doubt something better can be done, and if anything is done as part of stock Python, it would need to be. That said, docless slots would not be handled any differently with my sample code than they are now; just name them in __slots__. My docslots() function was very careful not to blindly overwrite an existing __slots__. So this: class Sample(object): __slots__ = "slot1", docslots(slot2="description") would work just fine. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> PythonLabs at Zope Corporation
Ok. Can anyone suggest an unobtrusive way to set docstrings on slots?
Perhaps this (but how could it be implemented?):
class Foo(object): __slots__ = docslots( ('slot1': 'description'), ('slot2': """description ...continued """), 'slot3', )
Anything can be done using metaclasses. __slots__ is not special once the class exists -- it is a set of instructions for the default metaclass to create a specific set of descriptors (and associated storage). Another metaclass could use a different convention (although it may have to set __slots__ to let the base metaclass create the associated storage slots). --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
Anything can be done using metaclasses. __slots__ is not special once the class exists -- it is a set of instructions for the default metaclass to create a specific set of descriptors (and associated storage). Another metaclass could use a different convention (although it may have to set __slots__ to let the base metaclass create the associated storage slots).
My original proposal was to use __slots__ dict values for docstrings in the default metaclass. You said you'd rather not do that because different metaclasses may want to use the dict value for different purposes. But from what you've explained, metaclasses are free to interpret the value of __slots__ in any way they choose. Metaclasses built on top of the default metaclass could translate their __slots__ value to the __slots__ I proposed. Are optional tuples any better? This wouldn't preclude use of dict values for something else. class Foo(object): __slots__ = [ 'slot1', ('slot2', 'description'), ('slot3', """description ...continued"""), ] -John -- http:// if ile.org/
Guido van Rossum wrote:
Anything can be done using metaclasses. __slots__ is not special once the class exists -- it is a set of instructions for the default metaclass to create a specific set of descriptors (and associated storage). Another metaclass could use a different convention (although it may have to set __slots__ to let the base metaclass create the associated storage slots).
My original proposal was to use __slots__ dict values for docstrings in the default metaclass. You said you'd rather not do that because different metaclasses may want to use the dict value for different purposes. But from what you've explained, metaclasses are free to interpret the value of __slots__ in any way they choose. Metaclasses built on top of the default metaclass could translate their __slots__ value to the __slots__ I proposed.
Yes, but *if* the default metaclass assumed a dict contained only docstrings, this would be the standard usage, and it would be confusing (and sometimes incompatible) if a custom metaclass had a different interpretation. As long as the default metaclass doesn't have any particular interpretation for the values in the dict, custom metaclasses can do what they like.
Are optional tuples any better? This wouldn't preclude use of dict values for something else.
class Foo(object): __slots__ = [ 'slot1', ('slot2', 'description'), ('slot3', """description ...continued"""), ]
But that currently doesn't work. Tbe most future-proof solution would be to put some kind of object in the dict values whose attributes can guide various metaclasses. Perhaps: class slotprop(object): def __init__(self, **kwds): self.__dict__.update(kwds) class C(object): __slots__ = {'slot1': slotprop(doc="this is the docstring"), 'slit2': slotprop('doc="another one")} --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum wrote:
But that currently doesn't work. Tbe most future-proof solution would be to put some kind of object in the dict values whose attributes can guide various metaclasses. Perhaps:
class slotprop(object): def __init__(self, **kwds): self.__dict__.update(kwds)
class C(object): __slots__ = {'slot1': slotprop(doc="this is the docstring"), 'slit2': slotprop('doc="another one")}
Wouldn't it be just as future-proof to make __slots__ a list of bare slot names or objects? One advantage is that slot names don't have to be carried around externally to the objects. Moreover, using standard attribute names like __name__ and __doc__ will make the solution more future-proof. The default metaclass will handle __slots__ along these lines: for item in obj.__slots__: if isinstance(item, StringTypes): slot_name, slot_doc = item, None else: slot_name = item.__name__ slot_doc = getattr(item, '__doc__', None) In that way, the default metaclass does not impose any restrictions on what the slot objects are. Example usage: class MySlot(object): def __init__(self, name, doc=None): self.__name__ = name self.__doc__ = doc class C(object): __slots__ = ['slot1', MySlot('slot2', 'this is the docstring')] -John -- http:// ift ile.org/
I wrote:
Wouldn't it be just as future-proof to make __slots__ a list of bare slot names or objects? One advantage is that slot names don't have to be carried around externally to the objects. Moreover, using standard attribute names like __name__ and __doc__ will make the solution more future-proof.
The default metaclass will handle __slots__ along these lines:
for item in obj.__slots__: if isinstance(item, StringTypes): slot_name, slot_doc = item, None else: slot_name = item.__name__ slot_doc = getattr(item, '__doc__', None)
In that way, the default metaclass does not impose any restrictions on what the slot objects are.
Example usage:
class MySlot(object): def __init__(self, name, doc=None): self.__name__ = name self.__doc__ = doc
class C(object): __slots__ = ['slot1', MySlot('slot2', 'this is the docstring')]
I am still very interested in seeing this happen. I'm looking for a Python C API expert who would be kind enough to get me started with the necessary mods to the default Python metaclass. I already have a prototype of the pydoc improvements which go hand-in-hand with this change (http://sourceforge.net/tracker/index.php?func=detail&aid=936774&group_id=5470&atid=305470). -John -- http://giftfile.org/ :: giftfile project
participants (3)
-
Fred L. Drake, Jr. -
Guido van Rossum -
John Belmonte