Restore the __members__ behavior to python3 for C extension writers
I have been trying to get dir(c_ext_obj) to work for PyCXX as the method used with python2 was removed in python3, namely use the list of names returned from the __members__ attribute. I have failed to find a simple replacement for the python3 API. It seems that I have implement __dir__. But to do that I need to know what dir() will return and add the member variables to the answer. I have been able to figure out what is necessary to write such code. No one on python users or python dev responded to an earlier query on this subject. Would it be possible to simply put back the support for the __members__ attribute in python3? Or provide a API call to get the list that dir() would produce for my object. Barry
On 6/13/2017 2:42 PM, Barry Scott wrote:
__members__ was deprecated about 15 years ago and gone from the stdlib before 2.7. The 2.7 doc says "object.__members__ Deprecated since version 2.2: Use the built-in function dir() to get a list of an object’s attributes. This attribute is no longer available." Ditto for __methods__. These were pre-2.2, pre type-class unification hacks. Only a few builtin types has a .__members__ for non-function data attributes.
dir includes __methods__ + __members__.
I have been able to figure out what is necessary to write such code. No one on python users or python dev responded to an earlier query on this subject.
Framed in terms of something so ancient, the question makes no sense to most people. If you ask again, don't refer to __members__.
Would it be possible to simply put back the support for the __members__ attribute in python3?
No. Perhaps you are looking for __dir__, called by dir(). " dir([object]) Without arguments, return the list of names in the current local scope. With an argument, attempt to return a list of valid attributes for that object. If the object has a method named __dir__(), this method will be called and must return the list of attributes." -- Terry Jan Reedy
On Wed, Jun 14, 2017 at 8:09 AM, Terry Reedy <tjreedy@udel.edu> wrote:
AIUI the OP is looking to implement __dir__, but make use of *what dir() would have returned* in that function. Something like: class Magic: def __getattr__(self, attr): if attr in self.generatables: return self.generated_value(attr) raise AttributeError def __dir__(self): return default_dir(self) + self.generatables For that purpose, is it possible to use super().__dir__()? Are there any considerations where that would fail? ChrisA
Yes.
Remember that I need to do this in the C API and I want default_dir of self in C not python. super().__dir__ looks at the class above me that is typically object() and so is not useful as it does not list the member function from my class or __mro__ or other stuff I may not be aware of that is important to return. Today I solve the problem in 2.7 C extension code by providing a value for __members__. In python3 I have no idea how to do this in C. I can find no example code that addresses this problem. How am I supposed to code this without the __members__ trick? Did I miss the C API that implements default_dir(self)? Barry
On 06/14/2017 01:54 PM, Barry Scott wrote:
__dir__ should return whatever you think is interesting about your object. It does not have to return everything, and in fact makes no guarantees that it will return everything. Enum's __dir__, for example, only returns a handful of entries. -- ~Ethan~
On Wed, Jun 14, 2017 at 1:54 PM, Barry Scott <barry@barrys-emacs.org> wrote:
object.__dir__(your_class_instance) should generally return everything you would get if you didn't override __dir__ at all. Remember, that code doesn't mean "return the methods and attributes defined on the object class", it's "run the object class's __dir__ method with self=your_class_instance". I don't know off-hand if there's a nicer way to do this from C than to manually look up the "__dir__" attribute on PyBaseObject_Type. (And of course if you wanted to get really fancy and handle cases where your object inherits from some type other than 'object', or where some user sticks your type into a multiple-inheritance hierarchy, you might potentially want to find "__dir__" using super lookup instead of going directly to PyBaseObject_Type. From a quick google it looks like this page gives an approach for doing that: https://pythonextensionpatterns.readthedocs.io/en/latest/super_call.html) -n -- Nathaniel J. Smith -- https://vorpus.org
On 15 June 2017 at 11:06, Nathaniel Smith <njs@pobox.com> wrote:
This is the kind of case where https://docs.python.org/3/c-api/object.html#c.PyObject_CallMethod is useful: dir_result = PyObject_CallMethod(base_type, "__dir__", "O", self); /* Add any additional attributes to the dir_result list */ return dir_result; Fully supporting multiple inheritance is more work (as your link shows), and often not needed. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Maybe it would make sense to implement a C-API function to perform a super() lookup, without all the contortions needed currently. It seems wasteful to have to make a super object, use it then immediately discard all the time. However the current logic is entangled into the type a fair bit, so that might take some work. Spencer Brown
But I need the result of __dir__ for my object not its base. Then I need to add in the list of member attributes that are missing because python itself has no knowledge of them they are accessed via getattr(). Now I cannot call __dir__ for my object in the implementation of __dir__ for the obvious reason. Today all classes defined using PyCXX for python3 cannot return the list of member variables via dir(obj). Where as in python2 I can make dir() work.
Fully supporting multiple inheritance is more work (as your link shows), and often not needed.
On Thu, Jun 15, 2017 at 2:44 PM, Barry Scott <barry@barrys-emacs.org> wrote:
Yes, that's what that code should give you. Try it :-) -n -- Nathaniel J. Smith -- https://vorpus.org
On 16 June 2017 at 07:44, Barry Scott <barry@barrys-emacs.org> wrote:
The C code: dir_result = PyObject_CallMethod(base_type, "__dir__", "O", self); is roughly equivalent to the Python code: dir_result = BaseType.__dir__(self) That is, it's calling the base type's __dir__ method, but it's still using the subclass *instance*. It's the same pattern people use to call a base type's __getattr__ or __getattribute__ for the subclass implementation of those methods, just without multiple inheritance support (since calling super() from C is painful). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Let me show you problem with an example. Here is an example run of the PyCXX Demo/Python3/simple.cxx code. : 19:01:15 ~/wc/svn/PyCXX : [1] barry@Expanse $ PYTHONPATH=obj python3.6 Python 3.6.0 (v3.6.0:41df79263a11, Dec 22 2016, 17:23:13) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information.
Notice that 'value' is not in the list of string returned from dir(n). That omission is because python does not know about 'value'. The code does this: Py::Object getattro( const Py::String &name_ ) { std::string name( name_.as_std_string( "utf-8" ) ); if( name == "value" ) { return m_value; } else { return genericGetAttro( name_ ); } } Where getattro is called (indirectly) from tp_getattro. In the python 2 I can tell python that 'value' exists because I provide a value of __members__. What is the way to tell python about 'value' in the python3 world? Barry
On 2017-06-18 20:10, Barry Scott wrote:
What is the way to tell python about 'value' in the python3 world?
Implement a __dir__ method. This should call the superclass' (e.g. object's) __dir__ and add whatever it is you want to add to the list. In general I'd recommend using properties for this kind of thing rather than messing with __getattr__, __setattr__ and __dir__, especially in pure Python. I'm no expert on the C API, but I think this should be possible by setting PyTypeObject.tp_getset https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_getset -- Thomas
On 19 June 2017 at 04:10, Barry Scott <barry@barrys-emacs.org> wrote:
OK, I think I may understand the confusion now. As Thomas noted, the preferred way of informing Python of data attributes for types implemented in C is to ask the interpreter to automatically create the appropriate descriptor objects by setting the `tp_members` slot on the C level *type*, rather than setting `__members__` on the instance: https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_members That approach also works for new-style classes in Python 2: https://docs.python.org/2/c-api/typeobj.html#c.PyTypeObject.tp_members I believe this is actually an old-/new-style class difference, so the relevant Python 3 change is the fact that the old-style approach simply isn't available any more. So if you use tp_members and tp_getset to request the creation of suitable descriptors, then the interpreter will automatically take care of populating the results of `dir()` correctly. However, if you're genuinely dynamically adding attributes in `__getattr__`, then you're going to need to add code to report them from `__dir__` as well. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
This is not useful as the values in my use cases are typically taken from a C or C++ objects that holds the master copy. The python class is a facade that is forwarding to a embedded object typically. And the values return might switch type. Being a PyString or None for example.
tp_getset might work. I'll have to a largest block of time experiment with it and think about C++ API for it.
On 6/13/2017 2:42 PM, Barry Scott wrote:
__members__ was deprecated about 15 years ago and gone from the stdlib before 2.7. The 2.7 doc says "object.__members__ Deprecated since version 2.2: Use the built-in function dir() to get a list of an object’s attributes. This attribute is no longer available." Ditto for __methods__. These were pre-2.2, pre type-class unification hacks. Only a few builtin types has a .__members__ for non-function data attributes.
dir includes __methods__ + __members__.
I have been able to figure out what is necessary to write such code. No one on python users or python dev responded to an earlier query on this subject.
Framed in terms of something so ancient, the question makes no sense to most people. If you ask again, don't refer to __members__.
Would it be possible to simply put back the support for the __members__ attribute in python3?
No. Perhaps you are looking for __dir__, called by dir(). " dir([object]) Without arguments, return the list of names in the current local scope. With an argument, attempt to return a list of valid attributes for that object. If the object has a method named __dir__(), this method will be called and must return the list of attributes." -- Terry Jan Reedy
On Wed, Jun 14, 2017 at 8:09 AM, Terry Reedy <tjreedy@udel.edu> wrote:
AIUI the OP is looking to implement __dir__, but make use of *what dir() would have returned* in that function. Something like: class Magic: def __getattr__(self, attr): if attr in self.generatables: return self.generated_value(attr) raise AttributeError def __dir__(self): return default_dir(self) + self.generatables For that purpose, is it possible to use super().__dir__()? Are there any considerations where that would fail? ChrisA
Yes.
Remember that I need to do this in the C API and I want default_dir of self in C not python. super().__dir__ looks at the class above me that is typically object() and so is not useful as it does not list the member function from my class or __mro__ or other stuff I may not be aware of that is important to return. Today I solve the problem in 2.7 C extension code by providing a value for __members__. In python3 I have no idea how to do this in C. I can find no example code that addresses this problem. How am I supposed to code this without the __members__ trick? Did I miss the C API that implements default_dir(self)? Barry
On 06/14/2017 01:54 PM, Barry Scott wrote:
__dir__ should return whatever you think is interesting about your object. It does not have to return everything, and in fact makes no guarantees that it will return everything. Enum's __dir__, for example, only returns a handful of entries. -- ~Ethan~
On Wed, Jun 14, 2017 at 1:54 PM, Barry Scott <barry@barrys-emacs.org> wrote:
object.__dir__(your_class_instance) should generally return everything you would get if you didn't override __dir__ at all. Remember, that code doesn't mean "return the methods and attributes defined on the object class", it's "run the object class's __dir__ method with self=your_class_instance". I don't know off-hand if there's a nicer way to do this from C than to manually look up the "__dir__" attribute on PyBaseObject_Type. (And of course if you wanted to get really fancy and handle cases where your object inherits from some type other than 'object', or where some user sticks your type into a multiple-inheritance hierarchy, you might potentially want to find "__dir__" using super lookup instead of going directly to PyBaseObject_Type. From a quick google it looks like this page gives an approach for doing that: https://pythonextensionpatterns.readthedocs.io/en/latest/super_call.html) -n -- Nathaniel J. Smith -- https://vorpus.org
On 15 June 2017 at 11:06, Nathaniel Smith <njs@pobox.com> wrote:
This is the kind of case where https://docs.python.org/3/c-api/object.html#c.PyObject_CallMethod is useful: dir_result = PyObject_CallMethod(base_type, "__dir__", "O", self); /* Add any additional attributes to the dir_result list */ return dir_result; Fully supporting multiple inheritance is more work (as your link shows), and often not needed. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Maybe it would make sense to implement a C-API function to perform a super() lookup, without all the contortions needed currently. It seems wasteful to have to make a super object, use it then immediately discard all the time. However the current logic is entangled into the type a fair bit, so that might take some work. Spencer Brown
But I need the result of __dir__ for my object not its base. Then I need to add in the list of member attributes that are missing because python itself has no knowledge of them they are accessed via getattr(). Now I cannot call __dir__ for my object in the implementation of __dir__ for the obvious reason. Today all classes defined using PyCXX for python3 cannot return the list of member variables via dir(obj). Where as in python2 I can make dir() work.
Fully supporting multiple inheritance is more work (as your link shows), and often not needed.
On Thu, Jun 15, 2017 at 2:44 PM, Barry Scott <barry@barrys-emacs.org> wrote:
Yes, that's what that code should give you. Try it :-) -n -- Nathaniel J. Smith -- https://vorpus.org
On 16 June 2017 at 07:44, Barry Scott <barry@barrys-emacs.org> wrote:
The C code: dir_result = PyObject_CallMethod(base_type, "__dir__", "O", self); is roughly equivalent to the Python code: dir_result = BaseType.__dir__(self) That is, it's calling the base type's __dir__ method, but it's still using the subclass *instance*. It's the same pattern people use to call a base type's __getattr__ or __getattribute__ for the subclass implementation of those methods, just without multiple inheritance support (since calling super() from C is painful). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Let me show you problem with an example. Here is an example run of the PyCXX Demo/Python3/simple.cxx code. : 19:01:15 ~/wc/svn/PyCXX : [1] barry@Expanse $ PYTHONPATH=obj python3.6 Python 3.6.0 (v3.6.0:41df79263a11, Dec 22 2016, 17:23:13) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information.
Notice that 'value' is not in the list of string returned from dir(n). That omission is because python does not know about 'value'. The code does this: Py::Object getattro( const Py::String &name_ ) { std::string name( name_.as_std_string( "utf-8" ) ); if( name == "value" ) { return m_value; } else { return genericGetAttro( name_ ); } } Where getattro is called (indirectly) from tp_getattro. In the python 2 I can tell python that 'value' exists because I provide a value of __members__. What is the way to tell python about 'value' in the python3 world? Barry
On 2017-06-18 20:10, Barry Scott wrote:
What is the way to tell python about 'value' in the python3 world?
Implement a __dir__ method. This should call the superclass' (e.g. object's) __dir__ and add whatever it is you want to add to the list. In general I'd recommend using properties for this kind of thing rather than messing with __getattr__, __setattr__ and __dir__, especially in pure Python. I'm no expert on the C API, but I think this should be possible by setting PyTypeObject.tp_getset https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_getset -- Thomas
On 19 June 2017 at 04:10, Barry Scott <barry@barrys-emacs.org> wrote:
OK, I think I may understand the confusion now. As Thomas noted, the preferred way of informing Python of data attributes for types implemented in C is to ask the interpreter to automatically create the appropriate descriptor objects by setting the `tp_members` slot on the C level *type*, rather than setting `__members__` on the instance: https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_members That approach also works for new-style classes in Python 2: https://docs.python.org/2/c-api/typeobj.html#c.PyTypeObject.tp_members I believe this is actually an old-/new-style class difference, so the relevant Python 3 change is the fact that the old-style approach simply isn't available any more. So if you use tp_members and tp_getset to request the creation of suitable descriptors, then the interpreter will automatically take care of populating the results of `dir()` correctly. However, if you're genuinely dynamically adding attributes in `__getattr__`, then you're going to need to add code to report them from `__dir__` as well. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
This is not useful as the values in my use cases are typically taken from a C or C++ objects that holds the master copy. The python class is a facade that is forwarding to a embedded object typically. And the values return might switch type. Being a PyString or None for example.
tp_getset might work. I'll have to a largest block of time experiment with it and think about C++ API for it.
participants (8)
-
Barry Scott -
Chris Angelico -
Ethan Furman -
Nathaniel Smith -
Nick Coghlan -
Spencer Brown -
Terry Reedy -
Thomas Jollans