On Wed, Jun 14, 2017 at 8:09 AM, Terry Reedy wrote:

Perhaps you are looking for __dir__, called by dir().

" dir([object])

Without arguments, return the list of names in the current local scope. With an argument, attempt to return a list of valid attributes for that object.

If the object has a method named __dir__(), this method will be called and must return the list of attributes."

AIUI the OP is looking to implement __dir__, but make use of *what dir() would have returned* in that function. Something like: class Magic: def __getattr__(self, attr): if attr in self.generatables: return self.generated_value(attr) raise AttributeError def __dir__(self): return default_dir(self) + self.generatables For that purpose, is it possible to use super().__dir__()? Are there any considerations where that would fail? ChrisA

On Thu, Jun 15, 2017 at 6:54 AM, Barry Scott wrote:

...

class Magic: def __getattr__(self, attr): if attr in self.generatables: return self.generated_value(attr) raise AttributeError def __dir__(self): return default_dir(self) + self.generatables

For that purpose, is it possible to use super().__dir__()? Are there any considerations where that would fail?

Remember that I need to do this in the C API and I want default_dir of self in C not python.

super().__dir__ looks at the class above me that is typically object() and so is not useful as it does not list the member function from my class or __mro__ or other stuff I may not be aware of that is important to return.

Right, thank you. That's the bit I wasn't aware of - the reason that __dir__ can't be used. So in terms of defining the problem, this is a solid piece of info. ChrisA

On Wed, Jun 14, 2017 at 1:54 PM, Barry Scott wrote:

...

On 13 Jun 2017, at 23:49, Chris Angelico wrote: For that purpose, is it possible to use super().__dir__()? Are there any considerations where that would fail?

Remember that I need to do this in the C API and I want default_dir of self in C not python.

super().__dir__ looks at the class above me that is typically object() and so is not useful as it does not list the member function from my class or __mro__ or other stuff I may not be aware of that is important to return.

object.__dir__(your_class_instance) should generally return everything you would get if you didn't override __dir__ at all. Remember, that code doesn't mean "return the methods and attributes defined on the object class", it's "run the object class's __dir__ method with self=your_class_instance". I don't know off-hand if there's a nicer way to do this from C than to manually look up the "__dir__" attribute on PyBaseObject_Type. (And of course if you wanted to get really fancy and handle cases where your object inherits from some type other than 'object', or where some user sticks your type into a multiple-inheritance hierarchy, you might potentially want to find "__dir__" using super lookup instead of going directly to PyBaseObject_Type. From a quick google it looks like this page gives an approach for doing that: https://pythonextensionpatterns.readthedocs.io/en/latest/super_call.html) -n -- Nathaniel J. Smith -- https://vorpus.org

Maybe it would make sense to implement a C-API function to perform a super() lookup, without all the contortions needed currently. It seems wasteful to have to make a super object, use it then immediately discard all the time. However the current logic is entangled into the type a fair bit, so that might take some work. Spencer Brown

On 15 Jun 2017, at 1:46 pm, Nick Coghlan wrote:

...

On 15 June 2017 at 11:06, Nathaniel Smith wrote: On Wed, Jun 14, 2017 at 1:54 PM, Barry Scott wrote:

...

...

On 13 Jun 2017, at 23:49, Chris Angelico wrote: For that purpose, is it possible to use super().__dir__()? Are there any considerations where that would fail?

Remember that I need to do this in the C API and I want default_dir of self in C not python.

super().__dir__ looks at the class above me that is typically object() and so is not useful as it does not list the member function from my class or __mro__ or other stuff I may not be aware of that is important to return.

object.__dir__(your_class_instance) should generally return everything you would get if you didn't override __dir__ at all. Remember, that code doesn't mean "return the methods and attributes defined on the object class", it's "run the object class's __dir__ method with self=your_class_instance".

I don't know off-hand if there's a nicer way to do this from C than to manually look up the "__dir__" attribute on PyBaseObject_Type.

This is the kind of case where https://docs.python.org/3/c-api/object.html#c.PyObject_CallMethod is useful:

dir_result = PyObject_CallMethod(base_type, "__dir__", "O", self); /* Add any additional attributes to the dir_result list */ return dir_result;

Fully supporting multiple inheritance is more work (as your link shows), and often not needed.

Cheers, Nick.

On Thu, Jun 15, 2017 at 2:44 PM, Barry Scott wrote:

On 15 Jun 2017, at 04:45, Nick Coghlan wrote:

...

dir_result = PyObject_CallMethod(base_type, "__dir__", "O", self); /* Add any additional attributes to the dir_result list */ return dir_result;

But I need the result of __dir__ for my object not its base.

Yes, that's what that code should give you. Try it :-) -n -- Nathaniel J. Smith -- https://vorpus.org

On 16 Jun 2017, at 09:46, Nick Coghlan wrote:

On 16 June 2017 at 07:44, Barry Scott wrote:

...

But I need the result of __dir__ for my object not its base. Then I need to add in the list of member attributes that are missing because python itself has no knowledge of them they are accessed via getattr().

The C code:

dir_result = PyObject_CallMethod(base_type, "__dir__", "O", self);

is roughly equivalent to the Python code:

dir_result = BaseType.__dir__(self)

That is, it's calling the base type's __dir__ method, but it's still using the subclass *instance*.

It's the same pattern people use to call a base type's __getattr__ or __getattribute__ for the subclass implementation of those methods, just without multiple inheritance support (since calling super() from C is painful).

Let me show you problem with an example. Here is an example run of the PyCXX Demo/Python3/simple.cxx code. : 19:01:15 ~/wc/svn/PyCXX : [1] barry@Expanse $ PYTHONPATH=obj python3.6 Python 3.6.0 (v3.6.0:41df79263a11, Dec 22 2016, 17:23:13) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information.

...

...

import simple sizeof(int) 4 sizeof(long) 8 sizeof(Py_hash_t) 8 sizeof(Py_ssize_t) 8 dir(simple) ['SimpleError', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'decode_test', 'derived_class_test', 'encode_test', 'func', 'func_with_callback', 'func_with_callback_catch_simple_error', 'make_instance', 'new_style_class', 'old_style_class', 'var'] n=simple.new_style_class() new_style_class c'tor Called with 0 normal arguments. and with 0 keyword arguments: dir(n) ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'func_keyword', 'func_noargs', 'func_noargs_raise_exception', 'func_varargs', 'func_varargs_call_member'] n.value 'default value'

Notice that 'value' is not in the list of string returned from dir(n). That omission is because python does not know about 'value'. The code does this: Py::Object getattro( const Py::String &name_ ) { std::string name( name_.as_std_string( "utf-8" ) ); if( name == "value" ) { return m_value; } else { return genericGetAttro( name_ ); } } Where getattro is called (indirectly) from tp_getattro. In the python 2 I can tell python that 'value' exists because I provide a value of __members__. What is the way to tell python about 'value' in the python3 world? Barry

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 19 June 2017 at 04:10, Barry Scott wrote:

The code does this:

Py::Object getattro( const Py::String &name_ ) { std::string name( name_.as_std_string( "utf-8" ) );

if( name == "value" ) { return m_value; } else { return genericGetAttro( name_ ); } }

Where getattro is called (indirectly) from tp_getattro.

In the python 2 I can tell python that 'value' exists because I provide a value of __members__.

What is the way to tell python about 'value' in the python3 world?

OK, I think I may understand the confusion now. As Thomas noted, the preferred way of informing Python of data attributes for types implemented in C is to ask the interpreter to automatically create the appropriate descriptor objects by setting the `tp_members` slot on the C level *type*, rather than setting `__members__` on the instance: https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_members That approach also works for new-style classes in Python 2: https://docs.python.org/2/c-api/typeobj.html#c.PyTypeObject.tp_members I believe this is actually an old-/new-style class difference, so the relevant Python 3 change is the fact that the old-style approach simply isn't available any more. So if you use tp_members and tp_getset to request the creation of suitable descriptors, then the interpreter will automatically take care of populating the results of `dir()` correctly. However, if you're genuinely dynamically adding attributes in `__getattr__`, then you're going to need to add code to report them from `__dir__` as well. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 19 Jun 2017, at 13:56, Nick Coghlan wrote:

On 19 June 2017 at 04:10, Barry Scott wrote:

...

The code does this:

Py::Object getattro( const Py::String &name_ ) { std::string name( name_.as_std_string( "utf-8" ) );

if( name == "value" ) { return m_value; } else { return genericGetAttro( name_ ); } }

Where getattro is called (indirectly) from tp_getattro.

In the python 2 I can tell python that 'value' exists because I provide a value of __members__.

What is the way to tell python about 'value' in the python3 world?

OK, I think I may understand the confusion now.

As Thomas noted, the preferred way of informing Python of data attributes for types implemented in C is to ask the interpreter to automatically create the appropriate descriptor objects by setting the `tp_members` slot on the C level *type*, rather than setting `__members__` on the instance: https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_members https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_members That approach also works for new-style classes in Python 2: https://docs.python.org/2/c-api/typeobj.html#c.PyTypeObject.tp_members https://docs.python.org/2/c-api/typeobj.html#c.PyTypeObject.tp_members

This is not useful as the values in my use cases are typically taken from a C or C++ objects that holds the master copy. The python class is a facade that is forwarding to a embedded object typically. And the values return might switch type. Being a PyString or None for example.

I believe this is actually an old-/new-style class difference, so the relevant Python 3 change is the fact that the old-style approach simply isn't available any more.

So if you use tp_members and tp_getset to request the creation of suitable descriptors, then the interpreter will automatically take care of populating the results of `dir()` correctly. However, if you're genuinely dynamically adding attributes in `__getattr__`, then you're going to need to add code to report them from `__dir__` as well.

tp_getset might work. I'll have to a largest block of time experiment with it and think about C++ API for it.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia