What objection do you have to creating a subclass? Adding methods to e.g. list could give you problems if you import 2 modules that each modify list and the modifications are incompatible.  It could conceivably add bugs to code that uses list in a standard way. Best wishes Rob Cliffe On 20/06/2021 13:18, Johan Vergeer wrote:

After working with Kotlin for a while I really started to like the idea of extension methods. I delivers a great way to extend the language without having to add features to the language itself.

As a practical example I would like to take a first item from a list

my_list = [] first = my_list[0] # This will obviously throw an error

In this case it would be great to create an extension method like

def list.first_or_none(self): if len(self): return self[0] return None

Or ...

def list.first_or_else(self, fallback_value): if len(self): return self[0] return fallback_value

Then, we could retrieve a value from the list:

from my_list_extensions import first_or_none, first_or_else

first = my_list.first_or_none() Or ... first = my_list.first_or_else(0)

Some of the main advantages (not an exhaustive list): - It also provides a way to add extension methods before adding methods to the standard library, like in PEP 616 - Classes can be extended without using inheritance - Easier for IDE's to know what methods are available for a type, therefore delivering auto-completion. - It would also become easier to extend classes from other libraries next to the standard library - It will be much easier to create methods on a class that can be chained, which leads to less intermediate variables and (in my opinion) better readability. Sometimes these are called fluent builders or fluent interfaces.

my_list.filter(lambda x: x >= 0).map(lambda x: str(x)) instead of [str(x) for x in my_list if x >= 0]. (ok, in hindsight this looks better with the list comprehension, but there are much more complicated examples that would benefit)

A more comprehensive example would be an html builder. Let's say I created a library that builds html from python code, with a nice builder interface:

class Html: def __init__(self): self.__items = []

def add_p(self, text): self.__items.append(text) return self

def add_h1(self, text): self.__items.append(text) return self

def add_div(self, items): self.__items.append(items) return self

To add methods (which is very likely in a class like this), I would have to create a subclass, or create a pull request with the author (if the author approves).

In all languages that I know of that have extension methods the most given argument is better code readability. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/QXS73E... Code of Conduct: http://python.org/psf/codeofconduct/

The technique you are calling "extension methods" is known as "monkey-patching" in Python and Ruby. With respect to a fine language, Kotlin, it doesn't have the user-base of either Python or Ruby. Python does not allow monkey-patching builtin classes, but Ruby does: https://avdi.codes/why-monkeypatching-is-destroying-ruby/ A cautionary tale. So what does Kotlin do to prevent that sort of thing? Can you use a regular function that takes a list as argument, instead of monkey-patching the list class to add a method? The beauty of a function is that every module is independent, so they can add their own list extensions (as functions) without stomping over each other. Whereas if they add them directly on to the list class itself, they can clash. As for code readability, I don't think that this: mylist.first_or_none() is particularly more readable than: first_or_none(mylist) Not everything has to be wrapped in a class. https://steve-yegge.blogspot.com/2006/03/execution-in-kingdom-of-nouns.html -- Steve

On 2021-06-20 7:48 p.m., Steven D'Aprano wrote:

The technique you are calling "extension methods" is known as "monkey-patching" in Python and Ruby.

With respect to a fine language, Kotlin, it doesn't have the user-base of either Python or Ruby. Python does not allow monkey-patching builtin classes, but Ruby does:

https://avdi.codes/why-monkeypatching-is-destroying-ruby/

A cautionary tale. So what does Kotlin do to prevent that sort of thing?

Can you use a regular function that takes a list as argument, instead of monkey-patching the list class to add a method?

The beauty of a function is that every module is independent, so they can add their own list extensions (as functions) without stomping over each other. Whereas if they add them directly on to the list class itself, they can clash.

The trick to extension methods is that they're only available when you explicitly use them. In other words, a module/package can't force them on you. The problem with "monkey-patching" is that it *does* get forced on you. With the way Python works, your only option to implement extension methods is to monkey-patch __getattr__. An alternative would be to have a module-level __getattr__, which by default resolves to the object's own attributes, but can be replaced by the module to provide proper (module-local) extension methods functionality. Thus, any module-local extension methods would also take priority over subclasses' attributes, without conflicting with them. (It would also slow everything down af, but that's beside the point.)

Your assumption about requiring some form of registration system for Python to implement extensions is correct as Roslyn (The C# compiler) resolves them at compile time (as long as the namespace is imported/the class is in the "global" namespace) When I said static function, it's basically me saying static method, and they're the same as a method in python with "@staticmethod" applied to it. here's a sample "prototype" of a System.Linq extension method (a very useful one imo): "namespace System.Linq { public static class Enumerable { public static IEnumerable<TResult> Select(this IEnumerable<TSource> enumerable, Func selector); } }" As you can see, the param "enumerable" is prefixed by the "this" keyword, this tells Roslyn to treat "enumerable" as if it was the special implicit "this" operator in instance methods: "using System.Linq; namespace MyNS { public sealed class MyCLS { public static readonly List<int> MyInts = new List<int>() { 0, 5, 12, 56, 9 }; public int RNG = 42; public IEnumerable<int> ExpandInts() { return MyInts.Select(@int => @int * this.RNG); } } }" As you can see in the above to quoted blocks (SOMEONE TELL ME HOW TO DO CODE BLOCKS!), I'm using the extension method "Select" defined in "Enumerable" as if it was defined on the interface "IEnumerable<T>" when it's actually not. (List<T> implements IList<T> which inherits ICollection<T> which inherits IEnumerable<T>) The only requirement is that both the static class AND the method is visible to your code (with public being visible to everyone) Here's the official docs: https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and...

On 2021-06-21 12:26 p.m., Stephen J. Turnbull wrote:

Soni L. writes:

...

The trick to extension methods is that they're only available when you explicitly use them.

What does "explicitly use them" mean? How does this help avoid the kinds of problems we know that monkey-patching causes?

Monkey-patching: ```py mod1.py import foo foo.Bar.monkeymethod = ... ``` ```py mod2.py import foo foo.Bar.monkeymethod = ... ``` "Extension methods": ```py mod1.py import foo def __getattr__(o, attr):   if isinstance(o, foo.Bar) and attr == "monkeymethod":     return ...   return getattr(o, attr) ``` ```py mod2.py import foo def __getattr__(o, attr):   if isinstance(o, foo.Bar) and attr == "monkeymethod":     return ...   return getattr(o, attr) ``` Note how the former changes foo.Bar, whereas the latter only changes the module's own __getattr__. You can't have conflicts with the latter. (Also note that this "module's own __getattr__" doesn't provide extension methods by itself, but can be used as a mechanism to implement extension methods.)

On Tue, Jun 22, 2021 at 01:49:56AM +1000, Chris Angelico wrote:

So what you're saying is that, in effect, every attribute lookup has to first ask the object itself, and then ask the module? Which module? The one that the code was compiled in? The one that is currently running? Both?

Mu. https://en.wikipedia.org/wiki/Mu_(negative)#%22Unasking%22_the_question We don't have an implementation yet, so it is too early to worry about precisely where you look up the extension methods, except that it is opt-in (so by default, there's no additional cost involved) and there must be *some* sort of registry *somewhere* that handles the mapping of extension methods to classes. We certainly don't want this to slow down *every method call*, but if it only slowed down method lookups a little bit when you actually used the feature, that might be acceptable. We already make use of lots of features which are slow as continental drift compared to C, because they add power to the language and are *fast enough*. E.g. name lookups are resolved at runtime, not compile-time; dynamic attribute lookups using gettattribute and getattr dunders; virtual subclasses; generic functions (functools.singledispatch); descriptors.

And how is this better than just using a plain ordinary function? Not everything has to be a method.

You get method syntax, obj.method, which is nice but not essential. When a method is called from an instance, you know that the first parameter `self` has got to be the correct type, no type-checking is required. That's good. And the same would apply to extension methods. You get bound methods as first class values, which is useful. You get inheritance, which is powerful. And you get encapsulation, which is important. I think this is a Blub moment. We don't think it's useful because we have functions, and we're not Java, so "not everything needs to be a method". Sure, but methods are useful, and they do bring benefits that top-level functions don't have. (And vice versa of course.) We have staticmethod that allows us to write a "function" (-ish) but get the benefits of inheritance, encapsulation, and method syntax. This would be similar. We acknowledge that there are benefits to monkey-patching. But we can't monkey-patch builtins and we are (rightly) suspicious of those who use monkey-patching in production. And this is good. But this would give us the benefits of monkey-patching without the disadvantages. *If* we can agree on semantics and come up with a reasonable efficient implementation that doesn't slow down every method call. -- Steve

On Tue, Jun 22, 2021 at 3:43 AM Steven D'Aprano wrote:

On Tue, Jun 22, 2021 at 01:49:56AM +1000, Chris Angelico wrote:

...

So what you're saying is that, in effect, every attribute lookup has to first ask the object itself, and then ask the module? Which module? The one that the code was compiled in? The one that is currently running? Both?

Mu.

https://en.wikipedia.org/wiki/Mu_(negative)#%22Unasking%22_the_question

We don't have an implementation yet, so it is too early to worry about precisely where you look up the extension methods, except that it is opt-in (so by default, there's no additional cost involved) and there must be *some* sort of registry *somewhere* that handles the mapping of extension methods to classes.

We certainly don't want this to slow down *every method call*, but if it only slowed down method lookups a little bit when you actually used the feature, that might be acceptable. We already make use of lots of features which are slow as continental drift compared to C, because they add power to the language and are *fast enough*.

E.g. name lookups are resolved at runtime, not compile-time; dynamic attribute lookups using gettattribute and getattr dunders; virtual subclasses; generic functions (functools.singledispatch); descriptors.

I'm actually not concerned so much with the performance as the confusion. What exactly does the registration apply to? And suppose you have a series of extension methods that you want to make use of in several modules in your project, how can you refactor a bunch of method registration calls so you can apply them equally in multiple modules? We don't need an implementation yet - but we need clear semantics.

...

And how is this better than just using a plain ordinary function? Not everything has to be a method.

You get method syntax, obj.method, which is nice but not essential.

When a method is called from an instance, you know that the first parameter `self` has got to be the correct type, no type-checking is required. That's good. And the same would apply to extension methods.

You get bound methods as first class values, which is useful.

You get inheritance, which is powerful.

And you get encapsulation, which is important.

True, all true, but considering that this is *not* actually part of the class, some of that doesn't really apply. For instance, is it really encapsulation? What does that word even mean when you're injecting methods in from the outside?

I think this is a Blub moment. We don't think it's useful because we have functions, and we're not Java, so "not everything needs to be a method". Sure, but methods are useful, and they do bring benefits that top-level functions don't have. (And vice versa of course.)

We have staticmethod that allows us to write a "function" (-ish) but get the benefits of inheritance, encapsulation, and method syntax. This would be similar.

We acknowledge that there are benefits to monkey-patching. But we can't monkey-patch builtins and we are (rightly) suspicious of those who use monkey-patching in production. And this is good. But this would give us the benefits of monkey-patching without the disadvantages.

*If* we can agree on semantics and come up with a reasonable efficient implementation that doesn't slow down every method call.

And that's a very very big "if". Monkey-patching can be used for unittest mocking, but that won't work here. Monkey-patching can be used to fix bugs in someone else's code, but that only works here if *your* code is in a single module, or you reapply the monkey-patch in every module. I'm really not seeing a lot of value in the proposal. Let's completely ignore the performance cost for the moment and just try to figure out semantics, with it being actually useful and not unwieldy. ChrisA

On 2021-06-21 3:01 p.m., Chris Angelico wrote:

Thanks for clarifying. This doesn't change the problem though - it just changes where the issue shows up. (BTW, what you're describing is closer to __getattribute__ than it is to __getattr__, so if you're proposing this as the semantics, I strongly recommend going with that name.)

Oh, sorry, thought __getattribute__ was the fallback and __getattr__ the one always called, what with getattr -> __getattr__. But yeah, __getattribute__ then.

So, here's the question - a clarification of what I asked vaguely up above. Suppose you have a bunch of these extension methods, and a large project. How are you going to register the right extension methods in the right modules within your project? You're binding the functionality to the module in which the code was compiled, which will make exec/eval basically unable to use them, and that means you'll need some way to set them in each module, or to import the setting from somewhere else. How do you propose doing this?

For exec/eval you just pass in the locals: exec(foo, globals(), locals()) because this __getattribute__ is just a local like any other. As for each module, you'd import them. But not quite with "import": import extension_methods # magic module, probably provides an @extend(class_) e.g. @extend(list) import shallow_flatten import deep_flatten __getattribute__ = extension_methods.getattribute(   shallow_flatten.flatten, # uses __name__   deepflatten=deep_flatten.flatten, # name override   __getattribute__=__getattribute__, # optional, defaults to builtins.getattr ) This would have to be done for each .py that wants to use the extension methods.

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/3DL46S... Code of Conduct: http://python.org/psf/codeofconduct/

On Mon, Jun 21, 2021 at 02:54:52PM -0300, Soni L. wrote:

Quite the opposite. You ask the local module (the one that the code was compiled in), and the module decides whether/when to ask the object itself.

In other words, every

foo.bar

would be sugar for

__getattr__(foo, "bar")

(where __getattr__ defaults to builtins.getattr) instead of being sugar for

(foo, "bar")

All you've done here is push the problem further along -- how does `__getattr__` (`__getattribute__`?) decide what to do? * Why is this extension-aware version per module, instead of a builtin? * Does that mean the caller has to write it in every module they want to make use of extensions? * Why do we need a second attribute lookup mechanism instead of having the existing mechanism do the work? * And most problematic, if we have an extension method on a type, the builtin getattr ought to pick it up. By the way, per-module `__getattr__` already has a meaning, so this name won't fly. https://www.python.org/dev/peps/pep-0562/ -- Steve

It seems odd that it would be per module and not per scope? On Tue, 22 Jun 2021, 00:55 Soni L., wrote:

On 2021-06-21 8:42 p.m., Steven D'Aprano wrote:

...

On Mon, Jun 21, 2021 at 02:54:52PM -0300, Soni L. wrote:

...

Quite the opposite. You ask the local module (the one that the code was compiled in), and the module decides whether/when to ask the object itself.

In other words, every

foo.bar

would be sugar for

__getattr__(foo, "bar")

(where __getattr__ defaults to builtins.getattr) instead of being sugar for

(foo, "bar")

All you've done here is push the problem further along -- how does `__getattr__` (`__getattribute__`?) decide what to do?

* Why is this extension-aware version per module, instead of a builtin?

* Does that mean the caller has to write it in every module they want to make use of extensions?

* Why do we need a second attribute lookup mechanism instead of having the existing mechanism do the work?

* And most problematic, if we have an extension method on a type, the builtin getattr ought to pick it up.

By the way, per-module `__getattr__` already has a meaning, so this name won't fly.

https://www.python.org/dev/peps/pep-0562/

No, you got it wrong. Extension methods don't go *on* the type being extended. Indeed, that's how they differ from monkeypatching.

The whole point of extension methods *is* to be per-module. You could shove it in the existing attribute lookup mechanism (aka the builtins.getattr) but that involves runtime reflection, whereas making a new, per-module attribute lookup mechanism specifically designed to support a per-module feature would be a lot better.

Extension methods *do not go on the type*.

And sure, let's call it __opcode_load_attr_impl__ instead. Sounds good? _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/MTOP22... Code of Conduct: http://python.org/psf/codeofconduct/

On 2021-06-21 8:57 p.m., Thomas Grainger wrote:

It seems odd that it would be per module and not per scope?

It's unusual to import things at the scope level. Usually things get imported at the module level, so, using module language doesn't seem that bad. But yes, it's per scope, but in practice it's per module because nobody would actually use this per scope even tho they could. :p

On Tue, 22 Jun 2021, 00:55 Soni L., mailto:fakedme%2Bpy@gmail.com> wrote:

On 2021-06-21 8:42 p.m., Steven D'Aprano wrote: > On Mon, Jun 21, 2021 at 02:54:52PM -0300, Soni L. wrote: > > > Quite the opposite. You ask the local module (the one that the code was > > compiled in), and the module decides whether/when to ask the object itself. > > > > In other words, every > > > > foo.bar > > > > would be sugar for > > > > __getattr__(foo, "bar") > > > > (where __getattr__ defaults to builtins.getattr) instead of being sugar for > > > > (foo, "bar") > > All you've done here is push the problem further along -- how does > `__getattr__` (`__getattribute__`?) decide what to do? > > * Why is this extension-aware version per module, instead of a builtin? > > * Does that mean the caller has to write it in every module they want to >   make use of extensions? > > * Why do we need a second attribute lookup mechanism instead of having >   the existing mechanism do the work? > > * And most problematic, if we have an extension method on a type, the >   builtin getattr ought to pick it up. > > > By the way, per-module `__getattr__` already has a meaning, so this name > won't fly. > > https://www.python.org/dev/peps/pep-0562/ https://www.python.org/dev/peps/pep-0562/ > >

No, you got it wrong. Extension methods don't go *on* the type being extended. Indeed, that's how they differ from monkeypatching.

The whole point of extension methods *is* to be per-module. You could shove it in the existing attribute lookup mechanism (aka the builtins.getattr) but that involves runtime reflection, whereas making a new, per-module attribute lookup mechanism specifically designed to support a per-module feature would be a lot better.

Extension methods *do not go on the type*.

And sure, let's call it __opcode_load_attr_impl__ instead. Sounds good? _______________________________________________ Python-ideas mailing list -- python-ideas@python.org mailto:python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org mailto:python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/MTOP22... https://mail.python.org/archives/list/python-ideas@python.org/message/MTOP22... Code of Conduct: http://python.org/psf/codeofconduct/ http://python.org/psf/codeofconduct/

On Tue, Jun 22, 2021 at 10:40 AM Steven D'Aprano wrote:

...

True, all true, but considering that this is *not* actually part of the class, some of that doesn't really apply. For instance, is it really encapsulation? What does that word even mean when you're injecting methods in from the outside?

Sure it's encapsulation. We can already do this with non-builtin classes:

class SpammySpam: def spam(self, arg): ...

from another_module import eggy_method

def aardvarks(self, foo, bar): ...

SpammySpam.aardvarks = aardvarks

The fact that two of those methods have source code that wasn't indented under the class statement is neither here nor there. Even the fact that eggy_method was defined in another module is irrelevant. What matters is that once I've put the class together, all three methods are fully encapsulated into the SpammySpam class, and other classes can define different methods with the same name.

Encapsulation is less about where you write the source code, and more about the fact that I can have

SpammySpam().spam

and

Advertising().spam

without the two spam methods stomping on each other.

Hmm, that's not what I'd usually understand "encapsulation" to mean. That's what would normally be called "namespacing". "... encapsulation refers to the bundling of data with the methods that operate on that data, or the restricting of direct access to some of an object's components." https://en.wikipedia.org/wiki/Encapsulation_(computer_programming)

Extension methods let us extend classes without the downsides of monkey-patching. Extension methods are completely opt-in while monkey-patching is mandatory for everyone. If we could only have one, extension methods would clearly be the safer choice.

I don't think it's safer necessarily. With this proposal, we have the notion that obj.method() can mean two completely different things *at the same time* and *on the same object* depending on how you refactor the code. # file1.py from file2 import func # and apply some extension methods def spamify(obj): print(obj.method()) print(func(obj)) # file2.py def func(obj): return obj.method() Is that really beneficial? All I'm seeing is myriad ways for things to get confusing - just like in the very worst examples of monkey-patching. And yes, I have some experience of monkey-patching in Python, including a situation where I couldn't just "import A; import B", I had to first import a helper for A, then import B, and finally import A, because there were conflicting monkey-patches. But here's the thing: extension methods (by this pattern) would not have solved it, because the entire *point* of the monkey-patch was to fix an incompatibility. So it HAD to apply to a completely different module. That's why, despite its problems, I still think that monkey-patching is the cleaner option. It prevents objects from becoming context-dependent.

We don't make heavy use of monkey-patching, not because it isn't a useful technique, but because:

- unlike Ruby, we can't extend builtins without subclassing;

- we're very aware that monkey-patching is a massively powerful technique with huge foot-gun potential;

- and most of all, the Python community is a hell of a lot more conservative than Ruby.

And the Ruby community is starting to see the risks of monkey-patching. (There's a quiz floating around the internet - "Ruby or Rails?" - that brings into sharp relief the incredibly far-reaching effects of using Rails. It includes quite a few methods on builtin objects.) So I am absolutely fine with being conservative. We have import hooks and MacroPy. Does anyone use them in production? I certainly don't - not because I can't, but because I won't without a VERY good reason.

Even basic techniques intentionally added to the language (like being able to attach attributes onto function objects) are often looked at as if they were the worst kind of obfuscated self-modifying code. Even when those same techniques are used in the stdlib people are still reluctant to use it. As a community, we're like cats: anything new and different scares us, even if its actually been used for 30 years.

We're a risk-adverse community.

I'm not sure why attaching attributes to functions is frowned upon; I'd personally make very good use of this for static variables, if only I could dependably refer to "this_function". But risk-averse is definitely preferable to the alternative. It means that Python is a language that can be learned as a whole, rather than being fragmented into "the NumPy flavour of Python" and "the Flask flavour of Python" and so on, with their own changes to the fabric of the language. So far, I'm not seeing anything in extension methods to make me want to change that stance. ChrisA

On Tue, Jun 22, 2021 at 8:01 PM Steven D'Aprano wrote:

The whole point of extension methods is that once the caller opts in to use them, method look ups (and that includes hasattr and getattr) must work as if the extension methods **are in place**.

The must be no semantic difference between:

obj.method(arg)

and

getattr(obj, 'method')(arg)

regardless of whether `method` is a regular method or an extension method.

And this is a problem. How is getattr defined? Is it counted as being in the current module? If it is, then has getattr magically become part of the module it's called from? Or do ALL lookups depend on where the function was called, rather than where it's defined? If 'method' is an extension method, where exactly is it visible? ChrisA

Oh this is a long one. Hypothetically, let's say you have a proxy object: class Foo:   def __getattribute__(self, thing):     return getattr(super().__getattribute__(self, "proxied"), thing) Should this really include extension methods into it by default? This is clearly wrong. The local override for the LOAD_ATTR opcode should NOT apply to proxy methods except where explicitly requested. Also sometimes you are supposed to call the dunder directly, like in the above example. It's not *bad* to do it if you know what you're doing. The point is that the caller using your proxy object should opt-in to the extension methods, rather than break with no way to opt-out of them. Your extension methods shouldn't propagate to proxy objects. To go even further, should all your class definitions that happen to extend a class with in-scope extension methods automatically gain those extension methods? Because with actual extension methods, that doesn't happen. You can have class MyList(list): pass and other callers would not get MyList.flatten even with you being able to use MyList.flatten locally. Extension methods are more like Rust traits than inheritance-based OOP. Also note that they use instance method syntax, but no other. That is they apply to LOAD_ATTR opcodes but should not apply to getattr! (Indeed, reflection in C#/Kotlin doesn't see the extension methods!) On 2021-06-22 6:57 a.m., Steven D'Aprano wrote:

I'm sorry Soni, I don't understand what you are arguing here. See below.

On Mon, Jun 21, 2021 at 10:09:17PM -0300, Soni L. wrote:

...

On 2021-06-21 9:39 p.m., Steven D'Aprano wrote:

...

Fourth step is that you go ahead and use lists as normal. Whether you use getattr or dot syntax, any extension methods defined in spam.py will show up, as if they were actual list methods.

hasattr([], 'head') # returns True list.tail # returns the spam.tail function object (unbound method)

They're not monkey-patched: other modules don't see that.

Python is a dynamic language. Maybe you're using hasattr/getattr to forward something from A to B. If "other modules don't see that" then this must work as if there were no extension methods in place.

What's "forward something from A to B" mean? What are A and B?

If "this" (method lookups) "must work as if there were no extension methods in place" then extension methods are a no-op and are pointless. You write an extension method, register it as applying to a type, the caller opts-in to use it, and then... nothing happens, because it "must work as if there were no extension methods in place".

Surely that isn't what you actually want to happen. But if not, I have no idea what you mean.

The whole point of extension methods is that once the caller opts in to use them, method look ups (and that includes hasattr and getattr) must work as if the extension methods **are in place**.

The must be no semantic difference between:

obj.method(arg)

and

getattr(obj, 'method')(arg)

regardless of whether `method` is a regular method or an extension method.

...

So you actually wouldn't want the local load_attr override to apply to those. If you did... well, just call the override directly.

I have no idea what that means. What is "the local load_attr override"?

...

If the override was called __opcode_load_attr_impl__ you'd just call __opcode_load_attr_impl__ directly instead of going through getattr.

As a general rule, you should not be calling dunders directly.

You seem to have missed the point that extension methods are intended as a mechanism to **extend a type** by giving it new methods on an opt-in basis. I want to call them "virtual methods" except that would add confusion regarding virtual subclasses and ABCs etc.

Maybe you need to read the Kotlin docs:

https://kotlinlang.org/docs/extensions.html

and the C# docs:

https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and...

Wikipedia also has a broad overview from a language-agnostic perspective:

https://en.wikipedia.org/wiki/Extension_method

Here's an example in TypeScript and Javascript:

https://putridparrot.com/blog/extension-methods-in-typescript/

In particular note these comments:

# Kotlin "Such functions are available for calling in the usual way as if they were methods of the original class."

# C# "Extension methods are only in scope when you explicitly import the namespace into your source code with a using directive."

Both C# and Kotlin are statically typed languages, and Python is not, but we ought to aim to minimise the differences in semantics. Aside from extension methods being resolved at runtime instead of at compile time, the behaviour ought to be as close as possible.

Just as single dispatch in Python is resolved dynamically, but aims to behave as close as possible to single dispatch in statically typed languages.

Another important quote:

"Because extension methods are called by using instance method syntax, no special knowledge is required to use them from client code. To enable extension methods for a particular type, just add a `using` directive for the namespace in which the methods are defined."

"No special knowledge is required" implies that, aside from the opt-in step itself, extension methods must behave precisely the same as regular methods. That means they will be accessible as bound methods on the instance:

obj.method

and unbound methods (functions) on the type:

type(obj).method

and using dynamic lookup:

getattr(obj, 'method')

and they will fully participate in inheritance heirarchies if you have opted in to use them.

...

There needs to be an escape hatch for this.

The escape hatch is to *not* opt-in to the extension method. If the caller doesn't opt-in, they don't get the extension methods.

That is the critical difference between extension methods and monkey- patching the type. Monkey-patching effects everyone. Extension methods have to be opt-in.

...

Or you *could* have getattr be special (called by load_attr) and overridable, and builtins.getattr be the escape hatch, but nobody would like that.

Huh? Unless you have shadowed getattr with a module-level function, getattr *is* builtins.getattr.

On Tue, Jun 22, 2021 at 9:23 PM Steven D'Aprano wrote:

On Tue, Jun 22, 2021 at 05:50:48PM +1000, Chris Angelico wrote:

...

Hmm, that's not what I'd usually understand "encapsulation" to mean. That's what would normally be called "namespacing".

Pfft, who you going to believe, me or some random folx on the internet editing Wikipedia? *wink*

Okay, using the Wikipedia/OOP definition still applies. Presumably most extension methods are going to be regular instance methods or class methods, rather than staticmethod. So they will take a `self` (or `cls`) parameter, and presumably most such extension methods will actually act on that self parameter in some way.

There is your "bundling of data with the methods that operate on that data", as required :-)

Okay, that's fair. Granted. It's not ALL of encapsulation, but it is, to an extent encapsulation. (It is also namespacing, and your justification of it was actually a justification of namespacing; but this is Python, and I think we all agree that namespacing is good!)

Regarding your example, you're only confused because you haven't take on board the fact that extension methods aren't interpreter-global, just module-global. Because it's new and unfamiliar. But we can do exactly the same thing, right now, with functions instead of methods, and you will find it trivially easy to diagnose the fault:

# file1.py from file2 import func # instead of "applying an extension method" from elsewhere, # import a function from elsewhere from extra_functions import g def spamify(obj): print(g(obj)) # works fine print(func(obj)) # fails

# file2.py def func(obj): return g(obj) # NameError

This example looks easy and not the least bit scary to you because you've been using Python for a while and it has become second nature to you. But you might remember back when you were a n00b, it probably confused you: why doesn't `g(obj)` work when you imported it? How weird and confusing! What do you mean, if I want to use g, I have to import it in each and every module where I want to use it? That's just dumb. Importing g once should make it available EVERYWHERE, right?

Been there, done that.

Fair point. However, I've worked with a good number of languages that have some notion of object methods, and generally, an object has or doesn't have a method based on what the object *is*, not on who's asking. It's going to make for some extremely confusing results. Is getattr() going to act as part of the builtins module or the module that's calling it? What about hasattr? What about an ABC's instance check, or anything else? How do other languages deal with this? How do they have a getattr-like function? Does it have to be a compiler construct?

Chris, here you are defending monkey-patching, not just as a necessary evil under some circumstances, but as a "cleaner" option, and then in your very next sentence:

...

And the Ruby community is starting to see the risks of monkey-patching.

Yes. That is exactly right. I am claiming that monkey-patching is, in many MANY cases, a cleaner option than extension methods. And then I am saying that monkey-patching is usually a bad thing. This is not incompatible, and it forms a strong view of my opinion of extension methods. ChrisA

On 2021-06-22 9:25 a.m., Chris Angelico wrote:

(Oh, and another wrinkle, although a small one: Code objects would need to keep track of their modules. Currently functions do, but code objects don't. But that seems unlikely to introduce further complications.)

What? No you just stop emitting LOAD_ATTR and instead emit LOAD_LOCAL '__opcode_load_attr_impl__' and call it with the relevant values.

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/H5QSWN... Code of Conduct: http://python.org/psf/codeofconduct/

On 2021-06-22 3:43 p.m., Brendan Barnwell wrote:

On 2021-06-22 05:14, Chris Angelico wrote:

...

Fair point. However, I've worked with a good number of languages that have some notion of object methods, and generally, an object has or doesn't have a method based on what the object*is*, not on who's asking.

    I agree, and this is the aspect of the proposal that most confuses me. I still can't understand concretely what is being proposed, though, so I'm not sure I even understand it.  Can someone clarify?  Suppose I have this

***** ### file1.py @extend(list) def len2(self):     return len(self)**2

### file2.py # or whatever I do to say "I want to use extensions to list defined in file1" from file1 extend list

def coolness(some_list):     return some_list.len2() + 1

my_list = [1, 2, 3] print("My list len2:", my_list.len2()) print("My list coolness:", coolness(my_list))

### file3.py import file2

other_list = [1, 2, 3, 4] print("Other list len2:", other_list.len2()) print("other list coolness:", file2.coolness(other_list)) print("My list len2 from outside:", file2.my_list.len2()) print("My list coolness from outside:", file2.coolness(file2.my_list)) *****

    What exactly is supposed to happen here if I run file3?  file2 declares use of file1's extensions.  file2 does not.  But file3 uses a function in file2 that makes use of such extensions.  Who sees the extension?

NameError, value, NameError, value, respectively.

    The list object my_list in file2 is the same object accessed as file2.my_list in file3.  Likewise coolness and file2.coolness.  It is going to be super confusing if calling the same function object with the same list object argument gives different results depending on which file you're in.  Likewise it's going to be confusing if the same list object sometimes has a .len2 method and sometimes doesn't.

It isn't the list object that has the extension method.

    But if it doesn't work that way, then it would seem to mean either every module sees the extensions (even if they didn't opt in), or else my_list in file2 is not the same object as file2.my_list in file3.  And that would be even worse.  (In this example it may seem okay because you can ask why I would call len2 from file3 if I didn't want to use it. But what if the extension is an override of an existing method?  Is that not allowed?)

    In addition, if there is a difference between my_list and other_list, then that apparently means that the syntax for lists now does something different in the two files.  This is maybe the most reasonable approach, since it's at least remotely reminiscent of a __future__ import, which changes syntactic behavior.  But what exactly is the difference between the two objects here?  Are both objects lists?  If they are, then how can they have different methods?  If they're not, then what are they?

    Most __future__ imports don't work like this.  Maybe the closest thing is the generator_stop one, but at least that places a flag on the code object to indicate the difference.  Would "extended lists" have some kind of magic attribute indicating which extensions they're using?  That may have been marginally acceptable in the case of PEP 479, which was essentially a bugfix, and set the attribute on code objects which are an obscure internal data structure.  But allowing this kind of thing for "user-facing" objects like lists would create a profusion of different list objects with different behavior depending on some combination of attributes indicating "what extends me" --- or, even worse, create different behavior without any such overt indication of which extensions are in use for a given object.

    The idea that the file in which code is written would somehow determine this type of runtime behavior seems to me to break my assumption that by knowing an object's identity I should have all the information I need to know about how to use it.  Some of the posts earlier in this thread seem to suggest that somehow the module where something was defined (something --- not sure what --- maybe the object with the extended method?  maybe the extended method itself?) would somehow get a hook to override attribute access on some objects (again, not sure which objects).

    That to me is the exact opposite of encapsulation.  Encapsulation means the object itself contains all its behavior.  If there is some getattr-like hook in some other module somewhere that is lying in wait to override attribute access on a given object "only sometimes" then that's not encapsulation at all.  It's almost as bad as the infamous COME FROM statement!

    Existing mechanisms like __getattribute__ are not parallel at all. When you know an object's identity, you know its MRO, which tells you all you need to know about what __getattribute__ calls might happen. You don't need to know anything about where the object "came from" or what file you're using it in.  But it seems with this proposal you would need to know, and that's kind of creepy to me.

Think about it like this, extension methods give you the ability to make imported functions that look like this: foo(bar, baz) look like this instead: bar.foo(baz) That's all there is to them. They're just a lie to change how you read/write the code. Some languages have an whole operator that has a similar function, where something like bar->foo(baz) is sugar for foo(bar, baz). The OP doesn't specify any particular mechanism for extension methods, so e.g. making the dot operator be implemented by a local function in the module, which delegates to the current attribute lookup mechanism by default, would be perfectly acceptable. It's like deprecating the existing dot operator and introducing a completely different one that has nothing to do with attribute lookup!

On 2021-06-22 13:09, Soni L. wrote:

Think about it like this, extension methods give you the ability to make imported functions that look like this:

foo(bar, baz)

look like this instead:

bar.foo(baz)

That's all there is to them. They're just a lie to change how you read/write the code. Some languages have an whole operator that has a similar function, where something like bar->foo(baz) is sugar for foo(bar, baz). The OP doesn't specify any particular mechanism for extension methods, so e.g. making the dot operator be implemented by a local function in the module, which delegates to the current attribute lookup mechanism by default, would be perfectly acceptable. It's like deprecating the existing dot operator and introducing a completely different one that has nothing to do with attribute lookup!

Okay, if that's the case, then I just think it's a bad idea. :-) We already have a definition for what bar.foo does, and it's totally under the control of the bar object (via the __getattr__/__getattribute__ mechanism). The idea that other things would be able to hook in there does not appeal to me at all. I don't really understand why you would want such a thing, to be honest. I feel it would make code way more difficult to reason about, as it would break locality constraints every which way. Now every time you see`bar.foo` you would have to think about all kinds of other modules that may be hooking in and adding their own complications. What's the point? Mostly the whole benefit of the dot notation is that it specifies a locally constrained relationship between the object and the attribute: you know that bar.foo does what bar decides, and no one else gets any say (unless bar asks for their opinion, e.g. by consulting global variables or whatever). If we want to write foo(bar, baz). . . well, we can just do that! What you're describing would just make existing attribute usages harder to understand while only "adding" something we can already do quite straightforwardly. Imagine a similar proposal for other syntax. Suppose that in any module I could define a function called operator_add and then other modules could "import" this "extension" so that every use of the + operator would somehow hook into this operator_add function. So now every time you do 2 + 2 you might be invoking some extension behavior. In my view that is unambiguously a road to madness, and as far as I can tell the extension mechanism you're proposing is equally ill-advised. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On Wed, Jun 23, 2021 at 6:41 AM Soni L. wrote:

On 2021-06-22 5:23 p.m., Chris Angelico wrote:

...

On Wed, Jun 23, 2021 at 6:13 AM Soni L. wrote:

...

Think about it like this, extension methods give you the ability to make imported functions that look like this:

foo(bar, baz)

look like this instead:

bar.foo(baz)

That's all there is to them. They're just a lie to change how you read/write the code. Some languages have an whole operator that has a similar function, where something like bar->foo(baz) is sugar for foo(bar, baz). The OP doesn't specify any particular mechanism for extension methods, so e.g. making the dot operator be implemented by a local function in the module, which delegates to the current attribute lookup mechanism by default, would be perfectly acceptable. It's like deprecating the existing dot operator and introducing a completely different one that has nothing to do with attribute lookup!

uh... I'm lost. Are you saying that that's a good thing? You *want* to replace the existing dot operator with one that has nothing to do with attribute lookup?? I don't get it.

Sure! As long as the new one can call getattr!

Let's say the new dot operator looks like this:

# file1.py def __dot__(left, right): print(left) print(right) ... foo = [] foo.bar

Now, this would actually print the list [] and the string "bar".

Then you can just use getattr to get attribute lookup behaviour out of it!

def __dot__(left, right): return getattr(left, right) foo = [] foo.bar

It would have local scope, similar to uh... locals. Y'know how locals are just sugar for locals()['foo'] and stuff? Yeah.

Not really, no, they're not. :) The dictionary returned by locals() isn't actually an implementation detail of local name lookups. Have you put any thought into how you would deal with the problem of recursive __dot__ calls? ChrisA

On 2021-06-22 5:34 p.m., Brendan Barnwell wrote:

On 2021-06-22 13:09, Soni L. wrote:

...

Think about it like this, extension methods give you the ability to make imported functions that look like this:

foo(bar, baz)

look like this instead:

bar.foo(baz)

That's all there is to them. They're just a lie to change how you read/write the code. Some languages have an whole operator that has a similar function, where something like bar->foo(baz) is sugar for foo(bar, baz). The OP doesn't specify any particular mechanism for extension methods, so e.g. making the dot operator be implemented by a local function in the module, which delegates to the current attribute lookup mechanism by default, would be perfectly acceptable. It's like deprecating the existing dot operator and introducing a completely different one that has nothing to do with attribute lookup!

    Okay, if that's the case, then I just think it's a bad idea.  :-)

    We already have a definition for what bar.foo does, and it's totally under the control of the bar object (via the __getattr__/__getattribute__ mechanism).  The idea that other things would be able to hook in there does not appeal to me at all.

    I don't really understand why you would want such a thing, to be honest.  I feel it would make code way more difficult to reason about, as it would break locality constraints every which way.  Now every time you see`bar.foo` you would have to think about all kinds of other modules that may be hooking in and adding their own complications. What's the point?

    Mostly the whole benefit of the dot notation is that it specifies a locally constrained relationship between the object and the attribute: you know that bar.foo does what bar decides, and no one else gets any say (unless bar asks for their opinion, e.g. by consulting global variables or whatever).  If we want to write foo(bar, baz). . . well, we can just do that!  What you're describing would just make existing attribute usages harder to understand while only "adding" something we can already do quite straightforwardly.

    Imagine a similar proposal for other syntax.  Suppose that in any module I could define a function called operator_add and then other modules could "import" this "extension" so that every use of the + operator would somehow hook into this operator_add function.  So now every time you do 2 + 2 you might be invoking some extension behavior. In my view that is unambiguously a road to madness, and as far as I can tell the extension mechanism you're proposing is equally ill-advised.

Imagine if Python didn't have an + operator, but instead an + *infix function*. Thus, every module would automatically include the global def infix +(left, right):   ... And indeed, you could say we already have this. Except currently you can't define your own local infix +. But what if you *could*? What if you could just, # file1.py def infix +(left, right):   return left << right x = 4 + 4 # file2.py def infix +(left, right):   return left ** right x = 4 + 4 # file3.py import file1 import file2 print(file1.x) # 64 print(file2.x) # 256 print(4 + 4) # 8 How does this break locality? Same idea with the dot operator, really. (Some languages don't have operators, but only functions. They let you do just this.)

On 2021-06-22 7:38 p.m., Chris Angelico wrote:

On Wed, Jun 23, 2021 at 8:30 AM Soni L. wrote:

...

On 2021-06-22 5:54 p.m., Chris Angelico wrote:

...

On Wed, Jun 23, 2021 at 6:41 AM Soni L. wrote:

...

It would have local scope, similar to uh... locals. Y'know how locals are just sugar for locals()['foo'] and stuff? Yeah.

Not really, no, they're not. :) The dictionary returned by locals() isn't actually an implementation detail of local name lookups.

It's... part of the language. Not an implementation detail. The dictionary returned by locals() is an inherent part of local name lookups, isn't it?

No, it's not. Most definitely not.

https://docs.python.org/3/library/functions.html#locals

Ohh. Fair enough, sorry.

...

...

Have you put any thought into how you would deal with the problem of recursive __dot__ calls?

Let it recurse!

Globals and locals don't go through __dot__, so you can just... use them. In particular, you can always use getattr(), and probably should. Or even set __dot__ to getattr inside it, like so:

def __dot__(left, right): __dot__ = getattr foo.bar # same as getattr(foo, "bar") because we set (local) __dot__ to getattr above

I can't actually pin down what I'm averse to here, but it gives me a really REALLY bad feeling. You're expecting every attribute lookup to now look for a local or global name __dot__ (or, presumably, a nonlocal, class, or builtin), and do whatever that does. That seems like a really effective foot-gun.

Have you actually tried designing this into a larger project to see what problems you run into, or is this something you've only considered at this trivial level?

1. It's opt-in. 2. It's designed to be used by a hypothetical extension methods module, but without imposing any design constraints on such module. It could return a named function every time a given name is looked up (a la "bind the first argument" operator), or do dynamic dispatch based on types or ABCs (a la proper extension methods). In practice, you don't def your own __dot__, but rather use someone else's "__dot__ builder". If you don't wanna deal with it, just don't use __dot__. It's also useful for the occasional domain-specific language.

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/YOPTCI... Code of Conduct: http://python.org/psf/codeofconduct/

On 2021-06-22 8:11 p.m., Chris Angelico wrote:

On Wed, Jun 23, 2021 at 9:06 AM Soni L. wrote:

...

On 2021-06-22 7:38 p.m., Chris Angelico wrote:

...

Have you actually tried designing this into a larger project to see what problems you run into, or is this something you've only considered at this trivial level?

1. It's opt-in. 2. It's designed to be used by a hypothetical extension methods module, but without imposing any design constraints on such module. It could return a named function every time a given name is looked up (a la "bind the first argument" operator), or do dynamic dispatch based on types or ABCs (a la proper extension methods).

In practice, you don't def your own __dot__, but rather use someone else's "__dot__ builder". If you don't wanna deal with it, just don't use __dot__.

It's also useful for the occasional domain-specific language.

What I'm hearing from you is: "No". Lots of words to say it, but, the answer to my question seems to be no.

We have never felt a need for extension methods in python, no.

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/BZ7BR5... Code of Conduct: http://python.org/psf/codeofconduct/

On 2021-06-22 15:35, Soni L. wrote:

Imagine if Python didn't have an + operator, but instead an + *infix function*.

Thus, every module would automatically include the global

def infix +(left, right): ...

And indeed, you could say we already have this. Except currently you can't define your own local infix +. But what if you*could*?

Then that would be bad. All the examples you give seem bad to me. They just make the code more confusing. Python combines various paradigms, but I think one way in which it very smoothly leverage object orientation is by making objects the locus of so much behavior. Operator overloads are defined at the object level, as are "quasi-operator" overloads for things like attribute lookup, iteration, context managers, etc. This means that the semantics of an expression are, by and large, determined by the types of the objects in that expression. What you're describing is basically moving a lot of that out to the module level. Now instead of operator overloads being governed by objects, they'd be governed by the module in which the code appears. The semantics of an expression would be determined not (only) by the types of the objects involved, but also by the module in which the expression textually occurs. There aren't many things in Python that work this way. Future imports are the main one, but those are rare (and rightly so). The import machinery itself provides some possibility for this (as used by stuff like macropy) but is mostly not used for such things (and again rightly so). Beyond that, I just think this kind of thing is a bad idea. Objects naturally cross module boundaries, in that an object may be created in one module and used in many other modules. It is good for an object's behavior to be consistent across modules, so that someone using an object (or readng code that uses an object) can look at the documentation for that object's type and understand how it will work in any context. It is good for code to be understandable in a "bottom up" way in which you understand the parts (the objects, expressions, syntactic structures, etc.) and can combine your understanding of those parts to understand the whole. It is bad for an object to shapeshift and do different things in different contexts. It is bad for code to heavily depend on "top down" information that requires you to know "where you are" (in one module or another) to understand how things work. That increases cognitive burden and makes code more difficult to understand. Personally I'm opposed to anything that moves in that direction. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

On Tue, Jun 22, 2021 at 08:44:56AM -0300, Soni L. wrote:

Oh this is a long one.

Hypothetically, let's say you have a proxy object:

class Foo:   def __getattribute__(self, thing):     return getattr(super().__getattribute__(self, "proxied"), thing)

Should this really include extension methods into it by default?

By default? Absolutely not. Extension methods are opt-in.

This is clearly wrong.

What is clearly wrong? Your question? A "yes" answer? A "no" answer? Your proxy object? Soni, and Chris, you seem to be responding as if extension methods are clearly, obviously and self-evidently a stupid idea. Let me remind you that at least ten languages (C#, Java, Typescript, Oxygene, Ruby, Smalltalk, Kotlin, Dart, VB.NET and Swift) support it. Whatever the pros and cons of the technique, it is not self-evidently wrong or stupid.

The local override for the LOAD_ATTR opcode should NOT apply to proxy methods except where explicitly requested.

Why not? Let me ask you this: - should proxy objects really include `__slots__` by default? - should they really include dynamic attributes generated by `__getattr__`? - should they include attributes in the inheritence hierarchy? - why should extension methods be any different? Let's step back from extension methods and consider a similar technique, the dreaded monkey-patch. If I extend a class by monkey-patching it with a new method: import library library.Klass.method = patch_method would you expect that (by default) the patched method are invisible to proxies of Klass? Would you expect there to be a way to "opt-out" of proxying that method? I hope that your answers are "No, and no", because if either answer is "yes", you will be very disappointed in Python. Why should extension methods be different from any other method? Let's go through the list of methods which are all treated the same: - methods defined on the class; - methods defined on a superclass or mixin; - methods added onto the instance; - methods created dynamically by `__getattr__`. (Did I miss any?) And the list of those which are handled differently, with ways to opt-out of seeing them: - ... um... er... Have I missed any?

The point is that the caller using your proxy object should opt-in to the extension methods, rather than break with no way to opt-out of them.

You opt-out by not opting in.

Your extension methods shouldn't propagate to proxy objects.

Fundamentally, your proxy object is just doing attribute lookups on another object. If you have a proxy to an instance `obj`, there should be no difference in behaviour between extension methods and regular methods. If `obj.method` succeeds, so should `proxy.method`, because that's what proxies do. The origin of obj.method should not make any difference. I'm sorry to have to keep harping on this, but it doesn't matter to the proxy whether the method exists in the instance `__dict__`, or the class `__dict__`, or `__slots__`, or a superclass, or is dynamically generated by `__getattr__`. A method is a method. Extension methods are methods.

To go even further, should all your class definitions that happen to extend a class with in-scope extension methods automatically gain those extension methods? Because with actual extension methods, that doesn't happen.

We might want to follow the state of the art here, assuming there was consensus in other languages about inheriting extension methods. But I would expect this behaviour: # --- extensions.py library --- @extends(list) def flatten(self): ... # --- module A.py --- uses extensions # opt-in to use the extension method class MyListA(list): pass MyListA.flatten # inherits from list # --- module B.py --- class MyListB(list): pass MyListB.flatten # raises AttributeError However, there may be factors I haven't considered.

You can have class MyList(list): pass and other callers would not get MyList.flatten even with you being able to use MyList.flatten locally.

Extension methods are more like Rust traits than inheritance-based OOP.

I understand that Rust doesn't support inheritance at all, and that Rust traits are more like what everyone else calls "interfaces".

Also note that they use instance method syntax, but no other. That is they apply to LOAD_ATTR opcodes but should not apply to getattr! (Indeed, reflection in C#/Kotlin doesn't see the extension methods!)

Okay, that's a good data point. The question is, why doesn't reflection see the extension methods? That will help us decide whether that's a limitation of reflection in those languages, or a deliberate design feature we should follow. A brief search suggests that people using C# do want to access extension methods via reflection, and that there are ways to do so: https://duckduckgo.com/?q=c%23+invoke+extension+method+via+reflection -- Steve

On Wed, Jun 23, 2021 at 6:49 PM Brendan Barnwell wrote:

There aren't many things in Python that work this way. Future imports are the main one, but those are rare (and rightly so). The import machinery itself provides some possibility for this (as used by stuff like macropy) but is mostly not used for such things (and again rightly so).

Future imports are generally about syntax. They only very seldom affect execution, and when they do, there has to be a mechanic like a flag on the code object (as generator_stop does). In order to maintain compatibility between modules with and without the directive, the execution has to be independent of that. (For example, Py2's "from __future__ import division" changes the bytecode created from the division operator, and barry_as_FLUFL generates the same code for "<>" that normal mode generates for "!=".) There's no fundamental problem with having things defined per-module. You can override the print function for the current module, and other modules aren't affected. I think that per-module operator overloading would be extremely confusing, but mainly because type-free operator overloading is an inherently confusing thing to do. Python allows an object to override operators, and C++ allows you to write a function called "operator+" that takes two arguments, but the types of those arguments is what makes everything make sense. That usually removes the problem of recursion because the implementation of operator+(my_obj, my_obj) is usually going to require addition of simpler types like integers. ChrisA

I've just thought of a great use-case for extension methods. Hands up who has to write code that runs under multiple versions of Python? *raises my hand* I'm sure I'm not the only one. You probably have written compatibility functions like this: def bit_length(num): try: return num.bit_length() except AttributeError: # fallback implementation goes here ... and then everywhere you want to write `n.bit_length()`, you write `bit_length(n)` instead. Extension methods would let us do this: # compatibility.py @extends(int): def bit_length(self): # fallback implementation goes here ... # mylibrary.py using compatibility num = 42 num.bit_length() Now obviously that isn't going to help with versions too old to support extension methods, but eventually extension methods will be available in the oldest version of Python you care about: # supports Python 3.14 and above Once we reach that point, then backporting new methods to classes becomes a simple matter of using an extension method. No mess, no fuss. As someone who has written a lot of code like that first bit_length compatibility function in my time, I think I've just gone from "Yeah, extension methods seem useful..." to "OMG I WANT THEM TEN YEARS AGO SO I CAN USE THEM RIGHT NOW!!!". Backporting might not be your killer-app for extension methods, but I really do think they might be mine. -- Steve

On Wed, Jun 23, 2021 at 03:47:05PM +1000, Chris Angelico wrote:

Okay. Lemme give it to you *even more clearly* since the previous example didn't satisfy.

# file1.py

@extend(list) def in_order(self): return sorted(self)

def frob(stuff): return stuff.in_order()

# file2.py

from file1 import frob thing = [1, 5, 2] frob(thing) # == [1, 2, 5] def otherfrob(stuff): return stuff.in_order() otherfrob(thing) # AttributeError

Am I correct so far? The function imported from file1 has the extension method, the code in file2 does not. That's the entire point here, right?

Correct so far.

Okay. Now, what if getattr is brought into the mix?

To a first approximation (ignoring shadowing) every dot lookup can be replaced with getattr and vice versa: obj.name <--> getattr(obj, 'name') A simple source code transformation could handle that, and the behaviour of the code should be the same. Extension methods shouldn't change that.

# file3.py @extend(list) def in_order(self): return sorted(self)

def fetch1(stuff, attr): if attr == "in_order": return stuff.in_order if attr == "unordered": return stuff.unordered return getattr(stuff, attr)

def fetch2(stuff, attr): return getattr(stuff, attr)

In file3's scope, there is no list.unordered method, so any call like some_list.unordered getattr(some_list, 'unordered') will fail, regardless of which list some_list is, or where it was created. That implies that: fetch1(some_list, 'unordered') fetch2(some_list, 'unordered') will also fail. It doesn't matter who is calling the functions, or what module they are called from. What matters is the context where the attribute lookup occurs, which in fetch1 and fetch2 is the file3 scope.

# file4.py from file3 import fetch1, fetch2

Doesn't matter that fetch1 and fetch2 are imported into file4. They are still executed in the global scope of file3. If they called `globals()`, they would see file3's globals, not file4's. Same thing for extension methods.

import random

@extend(list) def unordered(self): return random.shuffle(self[:])

I think that's going to always return None :-)

def fetch3(stuff, attr): if attr == "in_order": return stuff.in_order if attr == "unordered": return stuff.unordered return getattr(stuff, attr)

def fetch4(stuff, attr): return getattr(stuff, attr)

In the scope of file4, there is no list method "in_order", but there is a list method "unordered". So some_list.in_order getattr(some_list, 'in_order') will fail. That implies that: fetch3(some_list, 'unordered') fetch4(some_list, 'unordered') will also fail. It doesn't matter who is calling the functions, or what module they are called from. What matters is the context where the attribute lookup occurs, which in fetch3 and fetch4 is the file4 scope. (By the way, I think that your example here is about ten times more obfuscated than it need be, because of the use of generic, uninformative names with numbers.)

thing = [1, 5, 2] fetch1(thing, "in_order")() fetch2(thing, "in_order")() fetch3(thing, "in_order")() fetch4(thing, "in_order")() fetch1(thing, "unordered")() fetch2(thing, "unordered")() fetch3(thing, "unordered")() fetch4(thing, "unordered")()

Okay. *NOW* which ones raise AttributeError, and which ones give the extension method?

Look at the execution context. fetch1(thing, "in_order") and fetch2(thing, "in_order") execute in the scope of file3, where lists have an in_order extension method. It doesn't matter that they are called from file4: the body of the fetchN functions, where the attribute access takes place, executes where the global scope is file3 and hence the extension method "in_order" is found and returned. For the same reason, both fetch1(thing, "unordered") and fetch2(thing, "unordered") will fail. It doesn't matter that they are called from file4: their execution context is their global scope, file3, and just as they see file3's globals, not the callers, they will see file3's extension methods. (I say "the module's extension methods", not necessarily to imply that the extension methods are somehow attached to the module, but only that there is some sort of registry that says, in effect, "if your execution context is module X, then these extension methods are in use".) Similarly, the body of fetch3 and fetch4 execute in the execution context of file4, where list has been extended with an unordered method. So fetch3(thing, "unordered") and fetch4(thing, "unordered") both return that unordered method. For the same reason (the execution context), fetch3(thing, "in_order") and fetch4(thing, "in_order") both fail.

What exactly are the semantics of getattr?

Oh gods, I don't know the exact semantics of attribute look ups now! Something like this, I think: obj.attr (same as getattr(obj, 'attr'): if type(obj).__dict__['attr'] exists and is a data descriptor: # data descriptors are the highest priority return type(obj).__dict__['attr'].__get__() elif obj.__dict__ exists and obj.__dict__['attr'] exists: # followed by instance attributes in the instance dict return obj.__dict__['attr'] elif type(obj) defines __slots__ and there is an 'attr' slot: # then instance attributes in slots if the slot is filled: return contents of slot 'attr' else: raise AttributeError elif type(obj).__dict__['attr'] exists: if it is a non-data descriptor: return type(obj).__dict__['attr'].__get__() else: return type(obj).__dict__['attr'] elif type(obj) defines a __getattr__ method: return type(obj).__getattr__(obj) else: # search the superclass hierarchy ... # if we get all the way to the end raise AttributeError I've left out `__getattribute__`, I *think* that gets called right at the beginning. Also the result of calling `__getattr__` is checked for descriptor protocol too. And the look ups on classes are slightly different. Also when looking up on classes, metaclasses may get involved. And super() defines its own `__getattribute__` to customize the lookups. (As other objects may do too.) And some of the fine details may be wrong. But, overall, the "big picture" should be more or less correct: 1. check for data descriptors; 2. check for instance attributes (dict or slot); 3. check for non-data descriptors and class attributes; 4. call __getattr__ if it exists; 5. search the inheritance hierarchy; 6. raise AttributeError if none of the earlier steps matched. If we follow C# semantics, extension methods would be checked after step 4 and before step 5: if the execution context is using extensions for this class: and 'attr' is an extension method, return that method

Please explain exactly what the semantics of getattr are, and exactly which modules it is supposed to be able to see. Remember, it is not a compiler construct or an operator. It is a function, and it lives in its own module (the builtins).

You seem to think that getattr being a function makes a difference. Why? Aside from the possibility that it might be shadowed or deleted from builtins, can you give me any examples where `obj.attr` and `getattr(obj. 'attr')` behave differently? Even *one* example? Okay, this is Python. You could write a class with a `__getattr__` or `__getattribute__` method that inspected the call chain and did something different if it spotted a function called "getattr". Congratulations, you are very smart and Python is very dynamic. You might even write a __getattr__ that, oh, I don't know, returned a method if the execution context had opted in to a system that provided extra methods to your class. But I digress. But apart from custom-made classes that deliberately play silly buggers if they see that getattr is involved, can you give an example of where it behaves differently to dot syntax?

...

Not a rhetorical question: is that how it works in something like Swift, or Kotlin?

I have no idea. I'm just asking how you intend it to work in Python. If you want to cite other languages, go ahead, but I'm not assuming that they already have the solution, because they are different languages. Also not a rhetorical question: Is their getattr equivalent actually an operator or compiler construct, rather than being a regular function? Because if it is, then the entire problem doesn't exist.

I really don't know why you think getattr being a function makes any difference here. It's a builtin function, written in C, and can and does call the same internal C routines used by dot notation.

...

...

And what about this?

f = functools.partial(getattr, stuff) f("in_order")

NOW which extension methods should apply? Those registered here? Those registered in the builtins? Those registered in functools?

partial is just a wrapper around its function argument, so that should behave *exactly* the same as `getattr(stuff, 'in_order')`.

So if it behaves exactly the same way that getattr would, then is it exactly the same as fetch2 and fetch4? If not, how is it different?

Okay, let's look at the partial object: >>> import functools >>> f = functools.partial(getattr, [10, 20]) >>> f('index')(20) 1 Partial objects like f don't seem to have anything like a __globals__ attribute that allow me to tell what the execution context would be. I *think* that for Python functions (def or lambda) they just inherit the execution context from the function. For builtins, I'm not sure. I presume their execution context will be the current scope. Right now, I've already spent multiple hours on these posts, and I have more important things to do now than argue about the minutia of partial's behaviour. But if you wanted to do an experiment, you could do something like comparing the behaviour of: # module A.py f = lambda: globals() g = partial(globals) # module B.py from A import f, g f() g() and see whether f and g behave identically. I expect that f would return A's globals regardless of where it was called from, but I'm not sure what g would do. It might very well return the globals of the calling site. In any case, with respect to getattr, the principle would be the same: the execution context defines whether the partial object sees the extension methods or not. If the execution context is A, and A has opted in to use extension methods, then it will see extension methods. If the context is B, and B hasn't opted in, then it won't.

What about other functions implemented in C? If I write a C module that calls PyObject_GetAttr, does it behave as if dot notation were used in the module that called me, or does it use my module's extension methods?

That depends. If you write a C module that calls PyObject_GetAttr right now, is that *exactly* the same as dot notation in pure-Python code? The documentation is terse: https://docs.python.org/3.8/c-api/object.html#c.PyObject_GetAttr but if it is correct that it is precisely equivalent to dot syntax, then the same rules will apply. Has the current module opted in? If so, then does the class have an extension method of the requested name? Same applies to code objects evaluated without a function, or whatever other exotic corner cases you think of. Whatever you think of, the answer will always be the same: - if the execution context is a module that has opted to use extension methods, then attribute access will see extension methods; - if not, then it won't. If you think of a scenario where you are executing code where there is no module scope at all, and all global lookups fail, then "no module" cannot opt in to use extension methods and so the code won't see them. If you can think of a scenario where you are executing code where there are multiple module scopes that fight for supremacy using their two weapons of fear, surprise and a fanatical devotion to the Pope, then the winner will determine the result. *wink* -- Steve

On Wed, Jun 23, 2021 at 11:25 PM Steven D'Aprano wrote:

On Wed, Jun 23, 2021 at 03:47:05PM +1000, Chris Angelico wrote:

...

Okay. Lemme give it to you *even more clearly* since the previous example didn't satisfy.

# file1.py

@extend(list) def in_order(self): return sorted(self)

def frob(stuff): return stuff.in_order()

# file2.py

from file1 import frob thing = [1, 5, 2] frob(thing) # == [1, 2, 5] def otherfrob(stuff): return stuff.in_order() otherfrob(thing) # AttributeError

Am I correct so far? The function imported from file1 has the extension method, the code in file2 does not. That's the entire point here, right?

Correct so far.

...

Okay. Now, what if getattr is brought into the mix?

To a first approximation (ignoring shadowing) every dot lookup can be replaced with getattr and vice versa:

obj.name <--> getattr(obj, 'name')

A simple source code transformation could handle that, and the behaviour of the code should be the same. Extension methods shouldn't change that.

Alright. In that case, getattr() has stopped being a function, and is now a magical construct of the compiler. What happens if I do this? if random.randrange(2): def getattr(obj, attr): return lambda: "Hello, world" def foo(thing): return getattr(thing, "in_order")() Does it respect extension methods or not? If getattr is a perfectly ordinary function, as it now is, then it should be perfectly acceptable to shadow it. It should also be perfectly acceptable to use any other way of accessing attributes - for instance, the PyObject_GetAttr() function in C. Why should getattr() become magical?

...

import random

@extend(list) def unordered(self): return random.shuffle(self[:])

I think that's going to always return None :-)

Oops, my bad :) Not that it changes anything, given that we care more about whether they trigger AttributeError than what they actually do. (Chomp the rest of the discussion, since that was all based on the assumption that getattr was a normal function.)

...

What exactly are the semantics of getattr?

Oh gods, I don't know the exact semantics of attribute look ups now! Something like this, I think:

obj.attr (same as getattr(obj, 'attr'):

That is exactly what's weird about it. Instead of looking up the name getattr and then calling a perfectly ordinary function, now it has to be a magical construct of the compiler, handled right there. It is, in fact, impossible to craft equivalent semantics in a third-party function. Currently, getattr() can be defined in C on top of the C API function PyObject_GetAttr, which looks solely at the object and not the execution context. By your proposal, getattr() can only be compiler magic.

...

Please explain exactly what the semantics of getattr are, and exactly which modules it is supposed to be able to see. Remember, it is not a compiler construct or an operator. It is a function, and it lives in its own module (the builtins).

You seem to think that getattr being a function makes a difference. Why?

Aside from the possibility that it might be shadowed or deleted from builtins, can you give me any examples where `obj.attr` and `getattr(obj. 'attr')` behave differently? Even *one* example?

That is *precisely* the possibility. That is exactly why it is magical by your definition, and nonmagical by the current definition. At the moment, getattr() is just a function, hasattr() is just a function. I can do things like this: def ga(obj, attr): return getattr(obj, attr) Or this: ga = getattr Or this: PyObject *my_getattr(PyObject *obj, PyObject *attr) {return PyObject_GetAttr(obj, attr);} But if it's possible to do a source code transformation from getattr(obj, "attr") to obj.attr, then it is no longer possible to do *ANY* of this. You can't have an alias for getattr, you can't have a wrapper around it, you can't write your own version of it. In some languages, this is acceptable and unsurprising, because the getattr-like feature is actually an operator. (For instance, JavaScript fundamentally defines obj.attr as being equivalent to obj["attr"], so if you want dynamic lookups, you just use square brackets.) In Python, that is simply not the case. Are you proposing to break backward compatibility and all consistency just for the sake of this?

...

...

Not a rhetorical question: is that how it works in something like Swift, or Kotlin?

I have no idea. I'm just asking how you intend it to work in Python. If you want to cite other languages, go ahead, but I'm not assuming that they already have the solution, because they are different languages. Also not a rhetorical question: Is their getattr equivalent actually an operator or compiler construct, rather than being a regular function? Because if it is, then the entire problem doesn't exist.

I really don't know why you think getattr being a function makes any difference here. It's a builtin function, written in C, and can and does call the same internal C routines used by dot notation.

But if it's just a builtin function, then how is it going to know the execution context it's supposed to look up attributes in? If getattr(obj, "attr") can be implemented by a third party, show me how you would write the function such that it knows which extension methods to look for. You keep coming back to this assumption that it has to be fundamentally equivalent to obj.attr, but that's the exact problem - there is no way to define a function that can know the caller's context (barring shenanigans with sys._getframe), so it has to be compiler magic instead, which means it is *not a function any more*.

...

...

...

And what about this?

f = functools.partial(getattr, stuff) f("in_order")

NOW which extension methods should apply? Those registered here? Those registered in the builtins? Those registered in functools?

partial is just a wrapper around its function argument, so that should behave *exactly* the same as `getattr(stuff, 'in_order')`.

So if it behaves exactly the same way that getattr would, then is it exactly the same as fetch2 and fetch4? If not, how is it different?

Okay, let's look at the partial object:

>>> import functools >>> f = functools.partial(getattr, [10, 20]) >>> f('index')(20) 1

Partial objects like f don't seem to have anything like a __globals__ attribute that allow me to tell what the execution context would be. I *think* that for Python functions (def or lambda) they just inherit the execution context from the function. For builtins, I'm not sure. I presume their execution context will be the current scope.

Do you see the problem, then? The partial object has to somehow pass along the execution context. Otherwise, functools.partial(getattr, obj)("attr") won't behave identically to obj.attr.

Right now, I've already spent multiple hours on these posts, and I have more important things to do now than argue about the minutia of partial's behaviour. But if you wanted to do an experiment, you could do something like comparing the behaviour of:

# module A.py f = lambda: globals() g = partial(globals)

# module B.py from A import f, g f() g()

and see whether f and g behave identically. I expect that f would return A's globals regardless of where it was called from, but I'm not sure what g would do. It might very well return the globals of the calling site.

Correct on both counts - f() naturally has to return the globals from where it is, and g() uses the calling site. Whichever way you do it, somewhere, you're going to have a disconnect between obj.attr and the various dynamic ways of looking it up. It is going to happen. So why are you fighting so hard for getattr() to become magical in this way?

...

What about other functions implemented in C? If I write a C module that calls PyObject_GetAttr, does it behave as if dot notation were used in the module that called me, or does it use my module's extension methods?

That depends. If you write a C module that calls PyObject_GetAttr right now, is that *exactly* the same as dot notation in pure-Python code?

The documentation is terse:

https://docs.python.org/3.8/c-api/object.html#c.PyObject_GetAttr

but if it is correct that it is precisely equivalent to dot syntax, then the same rules will apply. Has the current module opted in? If so, then does the class have an extension method of the requested name?

The "current module", logically, would be the extension module.

Same applies to code objects evaluated without a function, or whatever other exotic corner cases you think of. Whatever you think of, the answer will always be the same:

- if the execution context is a module that has opted to use extension methods, then attribute access will see extension methods;

- if not, then it won't.

If you think of a scenario where you are executing code where there is no module scope at all, and all global lookups fail, then "no module" cannot opt in to use extension methods and so the code won't see them.

If you can think of a scenario where you are executing code where there are multiple module scopes that fight for supremacy using their two weapons of fear, surprise and a fanatical devotion to the Pope, then the winner will determine the result.

*wink*

Multiple scopes can definitely be plausible, but there'll just have to be some definition for which one wins. My guess would be that the function object wins, but if not, the code object should know its own context, and if not that, then there is a null context with no extension methods. But that's nothing more than a guess, and there's no rush on pinning that part down precisely. I am wholeheartedly against this proposal if it means that getattr has to become magical. If, however, getattr simply ignores extension methods, and the ONLY thing changed is the way that dot lookup is done, I would be more able to see its value. (Though not so much that I'd actually be using this myself. I don't think it'd benefit any of my current projects. But I can see its potential value for the language.) ChrisA

On 2021-06-23 03:02, Steven D'Aprano wrote:

Attribute lookups are just another form of name lookup. Name lookups depend on the current execution scope, not the caller's scope. With extension methods, so do attribute lookups.

But that's the thing, they aren't. You gave a bunch of examples of lexical scope semantics with imports and function locals vs globals. But attribute lookups do not work that way. Attribute lookups are defined to work via a FUNCTION CALL to the __getattribute__ (and thence often the __getattr__) of the OBJECT whose attribute is being looked up. They do not in any way depend on the name via which that object is accessed. Now of course you can say that you want to make a new rule that throws the old rules out the window. We can do that for anything. We can define a new rule that says now when you do attribute lookups it will call a global function called attribute_lookups_on_tuesdays if it's a Tuesday in your timezone. But what I'm saying is that the way attribute lookups currently work is not the same as the way bare-name lookups work, because attribute lookups are localized to the object (not the name!) and bare-name lookups are not. I consider this difference fundamental to Python. It's why locals() isn't really how local name lookups work (which came up elsewhere in this thread). It's why you can't magically hook into "x = my_obj" and create some magical behavior that depends on my_obj. Attribute lookups are under the control of the object; they come after the scope-based name resolution is all over with and they don't use the scope-based rules. As for other languages, you keep referencing them as if the fact that something known as "extension methods" exists in those other languages makes it self-evident that it would be useful in Python. Python isn't those other languages. I'm not familiar with all of the other languages you mentioned, but I'll bet that at least some of them do not have the same name/attribute lookup rules and dunder-governed object-customization setup as Python. So that's the difference. The fact that extension methods happen to exist and be useful in those languages is really neither here nor there. The attribute lookup procedure you are proposing is deeply inconsistent with the way Python currently does attribute lookup and currently does other things (like operator overloading), and doesn't fit into Python's overall mechanism of object-based hooks. A spatula attachment may be useful on a kitchen mixer; that doesn't mean it's a good idea to add one to your car's dashboard. Apart from that, I will say that I also don't generally assume that because other languages have a feature it's good or worth considering. Some languages are better designed than others. I think Python is a very well designed language. Certainly we can learn from other languages, but even apart from the issues of "fit" that I describe above, the mere fact that some feature is available or even considered useful in other languages doesn't by itself even convince me it's a good idea at all. It could just be a mistake. We need to specifically show that this will make writing and/or reading code easier and better in Python, and I think this proposal would do the opposite, making code harder to read. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown

Oops, sorry, I neglected to trim my response to João. Please ignore my previous response, with the untrimmed quoting at the start, and give any replies to this. On Wed, Jun 23, 2021 at 07:23:19PM +0200, João Santos wrote:

Of course that means the the standard library might also introduce something new that will be shadowed by one of your custom methods,

Extension methods have lower priority than actual methods on the class. So that won't happen. The actual method on the class will shadow the extension method. I'm not sure if you completely understand the use-case I was describing, so let me clarify for you with a concrete example. Ints have a "bit_length" method, starting from Python 2.7. I needed to use that method going all the way back to version 2.4. I have an implementation that works, so I could backport that method to 2.4 through 2.6, except that you can't monkey-patch builtins in Python. So monkey-patching is out. (And besides, I wouldn't want to monkey-patch it: I only need that method in one module. I want to localise the change to only where it is needed.) Subclassing int wouldn't help. I need it to work on actual ints, and any third-party subclasses of int, not just my own custom subclass. (And besides, have you tried to subclass int? It's a real PITA. It's easy enough to write a subclass, but every operation on it returns an actual int instead of the subclass. So you have to write a ton of boilerplate to make int subclasses workable. But I digress.) So a subclass is not a good solution either. That leaves only a function. But that hurts code readability and maintainance. In 2.7 and above, bit_length is a method, not a function. All the documentation for bit_length assumes it is a method. Every tutorial that uses it has it as a method. Other code that uses it treats it as a method. Except my code, where it is a function. Using a function is not a *terrible* solution to the problem of backporting a new feature to older versions of Python. I've done it dozens of times and it's not awful. **But it could be better.** Why can't the backport be a method, just like in 2.7 and above? With extension methods, it can be. Obviously not for Python 2.x code. But plan for the future: if we have extension methods in the language, eventually every version of Python we care about will support it. And then writing compatibility layers will be much simpler.

and then you'll wish you had just used functions or a wrapper class.

Believe me, I won't. I've written dozens of compatibility functions over the last decade or more, going back to Python 2.3. I've written hybrid 2/3 code. Extension methods would not always be useful, but for cases like int.bit_length, it would be a far superior solution.

If you can import extension methods wholesale, you might even be monkeypatching something without realising it

Extension methods is not monkey-patching. It is like adding a global name to one module. If I write: def func(arg): ... in module A.py, that does not introduce func to any other module unless those other modules explicitly import it. Extension methods are exactly analogous: they are only visible in the module where you opt-in to use them. They don't monkey-patch the entire interpreter-wide environment. And because extension methods have a lower priority than actual methods, you cannot override an existing method on a class. You can only extend the class with a new method. -- Steve

On Wed, Jun 23, 2021 at 11:22:26AM -0700, Brendan Barnwell wrote:

On 2021-06-23 03:02, Steven D'Aprano wrote:

...

Attribute lookups are just another form of name lookup. Name lookups depend on the current execution scope, not the caller's scope. With extension methods, so do attribute lookups.

But that's the thing, they aren't.

Of course attribute lookups are another form of name lookup. One hint that this is the case is that some languages, such as Java, call attributes *variables*, just like local and global variables. Both name and attribute lookups are looking up some named variable in some namespace. This shouldn't be controversial. When you look up bare name: x the interpreter follows the LEGB rule and looks for `x` in the local scope, the enclosing scope, the global scope and then the builtin scope. There are some complications to do with locals and class scopes, and comprehensions, etc, but fundamentally you are searching namespaces for names. Think of the LEGB rule as analogous to an MRO. When you look up a dotted name: obj.x the interpreter looks for `x` in the instance scope, the class scope, and any superclass scopes. The detailed search rules are different, what with decorators, inheritance, etc, the MRO could be indefinitely long, and there are dynamic attributes (`__getattr__`) too. But fundamentally, although the details are different, attribute access and name lookup are both looking up names in namespaces.

You gave a bunch of examples of lexical scope semantics with imports and function locals vs globals. But attribute lookups do not work that way. Attribute lookups are defined to work via a FUNCTION CALL to the __getattribute__ (and thence often the __getattr__) of the OBJECT whose attribute is being looked up.

There's that weird obsession with "function call" again. Why do you think that makes a difference?

They do not in any way depend on the name via which that object is accessed.

Ummmm... yes? Why is that relevant?

Now of course you can say that you want to make a new rule that throws the old rules out the window.

That's pure FUD. Extension methods don't require throwing the old rules out the window. The old rules continue to apply.

We can do that for anything. We can define a new rule that says now when you do attribute lookups it will call a global function called attribute_lookups_on_tuesdays if it's a Tuesday in your timezone.

If you want to do that then go ahead and propose it in another thread, but I don't want anything like that strawman.

But what I'm saying is that the way attribute lookups currently work is not the same as the way bare-name lookups work, because attribute lookups are localized to the object

You might have heard of something called "inheritance". And dynamic attributes. Maybe even mixins or traits. Attribute lookups are not localised to the object.

(not the name!) and bare-name lookups are not. I consider this difference fundamental to Python.

There are plenty of differences between name lookups and attribute lookups, but locality is not one of them.

It's why locals() isn't really how local name lookups work (which came up elsewhere in this thread).

I don't see the relevance to extension methods.

It's why you can't magically hook into "x = my_obj" and create some magical behavior that depends on my_obj.

I don't see the relevance to extension methods. You seem to be just listing random facts as if they were objections to the extension method proposal. You're right, we can't hook into assignment to a bare name. So what? We *can* hook into attribute lookups.

As for other languages, you keep referencing them as if the fact that something known as "extension methods" exists in those other languages makes it self-evident that it would be useful in Python. Python isn't those other languages. I'm not familiar with all of the other languages you mentioned, but I'll bet that at least some of them do not have the same name/attribute lookup rules and dunder-governed object-customization setup as Python. So that's the difference.

I think that's a difference that makes no difference. What if I told you that it is likely that Python's name/attribute lookup rules and dunder-governed object-customization are the key features that would make extension methods possible with little or no interpreter support? As I described in a previous post, adding extension methods would be a very small change to the existing attribute lookup rules. The tricky part is to come up with a fast, efficient registration system for determining when to use them.

The fact that extension methods happen to exist and be useful in those languages is really neither here nor there.

Extension methods don't just "happen to exist", they were designed to solve a real problem. That's a problem that can apply to Python just as much as other languages. Your argument here sounds like Not Invented Here. "Sure they're useful, in *other* (lesser) languages, not in Python! We never have any need to extend classes with new methods -- apart from all the times we do, but they don't count."

The attribute lookup procedure you are proposing is deeply inconsistent with the way Python currently does attribute lookup

"Deeply inconsistent" in what way? `__getattr__` can already do **literally anything** on attribute lookups? Anything that you can do in Python, you can do in a `__getattr__` method. Including adding arbitrary new methods.

and currently does other things (like operator overloading), and doesn't fit into Python's overall mechanism of object-based hooks. A spatula attachment may be useful on a kitchen mixer; that doesn't mean it's a good idea to add one to your car's dashboard.

Fortunately this is not a proposal to add spatulas to car dashboards. It is a proposal to add a mechanism to extend classes with extra functionality -- an extremely common activity in OOP. [...]

I think this proposal would do the opposite, making code harder to read.

Okay. Here's a method call with a regular method: obj.method(arg) # easy to read, understandable Here's a method call with a extension method: obj.method(arg) # unreadable, incomprehensible garbage Yes, you're absolutely right, the second is *so much harder to read*. Seriously, there's a time to realise when arguments against a feature devolve down to utterly spurious claims that Python programmers are idiots who will be confused by: from extensions use flatten mylist.flatten() but can instantly understand: from extensions import flatten flatten(mylist) If you can understand importing functions, you can understand using extension methods. If anything, extension methods are simpler: there's not likely to be all the complications of: - circular imports, which are tricky even for seasoned Pythonistas; - differences between standard import and import from, which often trips beginners up; - absolute and relative imports; - regular and namespace packages; and other quirks of importing. Compared to all of those, extension methods are likely to be simple and straightforward. -- Steve

On 6/24/21 7:09 AM, Simão Afonso wrote:

On 2021-06-24 20:59:31, Steven D'Aprano wrote:

...

Seriously, there's a time to realise when arguments against a feature devolve down to utterly spurious claims that Python programmers are idiots who will be confused by:

from extensions use flatten mylist.flatten()

but can instantly understand:

from extensions import flatten flatten(mylist) Does this mean importing a module can modify other objects, including builtins? Should this spooky-action-at-a-distance be encouraged?

OTOH, this already happens in the stdlib with rlcompleter, I assume using monkey-patching. This is a special case for interactive use, though.

https://docs.python.org/3/library/rlcompleter.html

Yes, importing a module runs the global code in that module, and that code can not only define the various things in that module but can also manipulate the contents of other modules. This doesn't mean that spooky-action-at-a-distance is always good, but sometimes it is what is needed. You need to be aware of the power that you wield. -- Richard Damon

Steven, you're making a pretty good case here, but a couple questions: 1) The case of newer versions of python adding methods to builtins, like .bit_length is really compelling. But I do wonder how frequently that comes up. It seems to me on this list, that people are very reluctant to add methods to anything builtin (other than dunders) -- particularly ABCs, or classes that might be subclassed, as that might break anything that currently uses that same name. Anyway, is this a one-off? or something that is likely to come up semi-frequently in the future? Note also that the py2 to py3 transition was (Hopefully) an anomaly -- more subtle changes between versions make it less compelling to support old versions for very long. 2) Someone asked a question about the term "Extension Methods" -- I assume it's "Extension attributes", where the new attribute could be anything, yes? 2) Comprehensibility: Seriously, there's a time to realise when arguments against a feature

devolve down to utterly spurious claims that Python programmers are idiots who will be confused by:

from extensions use flatten mylist.flatten()

but can instantly understand:

from extensions import flatten flatten(mylist)

No -- we're not assuming Python users are idiots -- there is an important difference here: from extensions import flatten flatten(mylist) very clearly adds the name `flatten` to the current module namespace. That itself can be confusing to total newbies, but yes, you can't get anywhere with Python without knowing that. Granted, you still need need to know what `flatten` is, and what it does some other way in any case. Whereas: from extensions use flatten mylist.flatten() does NOT import the name `flatten` into the local namespace -- which I suppose will be "clear" because it's using "use" rather than a regular import, but that might be subtle. But importantly, what it has done is add a name to some particular type -- what type? who knows? In this example, you used the name "mylist", so I can assume it's an extension to list. But if that variable were called "stuff", I"d have absolutely no idea. And as above, you'd need to go find the documentation for the flatten extension method, just as you would for any name in a module, but somehow functions feel more obvious to me. Thinking about this I've found what I think is a key issue for why this may be far less useful for Python that it is for other languages. Using teh "flatten" example, which I imagine you did due to the recent discussion on this list about the such a function as a potential new builtin: Python is dynamically Polymorphic (I may have just made that term up -- I guess it's the same as duck typed) -- but what that means in that context is that I don't usually care exactly what type an object is -- only that it supports particular functionality, so, for instance: from my_utilities import flatten def func_that_works_with_nested_objects(the_things): all_the_things_in_one = flatten(the_things) ... Presumably, that could work with any iterable with iterables in it. but from my_extensions use flatten def func_that_works_with_nested_objects(the_things): all_the_things_in_one = the_things.flatten OOPS! that's only going to work with actual lists. Are you thinking that you could extend an ABC? Or if not that, then at least a superclass and get all subclasses? I'm a bit confused about how the MRO might work there. Anyway, in my mind THAT is the big difference between Python and at least mony of the languages that support extension methods. A "solution" would be to do what we do with numpy -- it has an "asarray()" function that is a no-op if the argument is a numpy array, and creates an array if it's not. WE often put that at the top of a function, so that we can then use all the nifty array stuff inside the function, but not requires the caller to create an array firat. But that buys ALL the numpy functionality, it would be serious overkill for a method or two. It's not a reason it couldn't work, or be useful, but certainly a lot less useful than it might be. In fact, the example for int.bit_length may b e the only compelling use case -- not that method per se, but a built-in type that is rarely duck-typed. That would be integers, floats and strings, at least those are the most common. even ints and floats are two types that are frequently used interchangeably. Side note: I don't see the relevance to extension methods. You seem to be just

listing random facts as if they were objections to the extension method proposal.

Let's keep this civil, and assume good intentions -- if something is irrelevant, it's irrelevant, but please don't assume that the argument was not made in good faith. For my part I've been following this thread, but only recently understood the scope of the proposal well enough to know that e.g. the above issue was not relevant. -Chris B -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

On Thu, Jun 24, 2021 at 12:17:17AM +1000, Chris Angelico wrote:

On Wed, Jun 23, 2021 at 11:25 PM Steven D'Aprano wrote:

...

To a first approximation (ignoring shadowing) every dot lookup can be replaced with getattr and vice versa:

obj.name <--> getattr(obj, 'name')

A simple source code transformation could handle that, and the behaviour of the code should be the same. Extension methods shouldn't change that.

Alright. In that case, getattr() has stopped being a function, and is now a magical construct of the compiler.

How do you come to that conclusion? Did you miss the part where I said "To a first approximation (ignoring shadowing)"? What I'm describing is not some proposed change, it is the status quo. getattr is equivalent to dot notation, as it always has been, all the way back to 1.5 or older. There's no change there. getattr is a function. A regular, plain, ordinary builtin function. You can shadow it, or bind it to another name, or reach into builtins and delete it, just like every other builtin function. But if you don't do any of those things, then it is functionally equivalent to dot lookup.

What happens if I do this?

if random.randrange(2): def getattr(obj, attr): return lambda: "Hello, world"

def foo(thing): return getattr(thing, "in_order")()

According to the result of the random number generator, either the lambda will be returned by the shadowed getattr, or the attribute "in_order" will be looked up on obj. Just like today.

If getattr is a perfectly ordinary function, as it now is, then it should be perfectly acceptable to shadow it.

Correct.

It should also be perfectly acceptable to use any other way of accessing attributes - for instance, the PyObject_GetAttr() function in C.

Again, correct.

Why should getattr() become magical?

It doesn't.

...

...

What exactly are the semantics of getattr?

Oh gods, I don't know the exact semantics of attribute look ups now! Something like this, I think:

obj.attr (same as getattr(obj, 'attr'):

That is exactly what's weird about it. Instead of looking up the name getattr and then calling a perfectly ordinary function, now it has to be a magical construct of the compiler, handled right there. It is, in fact, impossible to craft equivalent semantics in a third-party function.

I don't think that it is impossible to emulate attribute lookup in pure Python code. It's complicated, to be sure, but I'm confident it can be done. Check out the Descriptor How To Guide, which is old but as far as I can tell still pretty accurate in its description of how attributes are looked up.

Currently, getattr() can be defined in C on top of the C API function PyObject_GetAttr, which looks solely at the object and not the execution context. By your proposal, getattr() can only be compiler magic.

The only "magic" that is needed is the ability to inspect the call stack to find out the module being called from. CPython provides functions to do that in the inspect library: `inspect.stack` and `inspect.getmodule`. Strictly speaking, they are not portable Python, but any interpreter ought to be able to provide analogous abilities. Do you think that the functions in the gc library are "compiler magic"? It would be next to impossible to emulate them from pure Python in an interpeter-independent fashion. Some interpreters don't even have reference counts. How about locals()? That too has a privileged implementation, capable of doing things likely impossible from pure, implementation-independent Python code. Its still a plain old regular builtin function that can be shadowed, renamed and deleted.

...

Aside from the possibility that it might be shadowed or deleted from builtins, can you give me any examples where `obj.attr` and `getattr(obj. 'attr')` behave differently? Even *one* example?

That is *precisely* the possibility.

And it will remain the possibility.

That is exactly why it is magical by your definition

It really won't.

But if it's possible to do a source code transformation from getattr(obj, "attr") to obj.attr, then it is no longer possible to do *ANY* of this. You can't have an alias for getattr, you can't have a wrapper around it, you can't write your own version of it.

In the absence of any shadowing or monkey-patching of builtins. -- Steve

Here's a quick and dirty proof of concept I knocked up in about 20 minutes, demonstrating that no deep compiler magic is needed. It's just a small change to the way `object.__getattribute__` works. I've emulated it with my own base class, since `object` can't be monkey-patched. The proof of concept is probably buggy and incomplete. It isn't intended to be a final, polished production-ready implementation. It's not implementation-agnostic: it requires the ability to inspect the call stack. If you're using IronPython, this may not work. You will notice I didn't need to touch getattr to have it work, let alone hack the interpreter to make it some sort of magical construct. It all works through `__getattribute__`. The registration system is just the easiest thing that I could throw together. There are surely better designs. Run A.py to see it in action. -- Steve

Would this feature allow me to declare str objects as not iterable in some contexts? If so, +1. On Thu, Jun 24, 2021, 3:22 PM Chris Angelico wrote:

On Fri, Jun 25, 2021 at 3:31 AM Steven D'Aprano wrote:

...

Here's a quick and dirty proof of concept I knocked up in about 20 minutes, demonstrating that no deep compiler magic is needed. It's just a small change to the way `object.__getattribute__` works.

I've emulated it with my own base class, since `object` can't be monkey-patched.

The proof of concept is probably buggy and incomplete. It isn't intended to be a final, polished production-ready implementation. It's not implementation-agnostic: it requires the ability to inspect the call stack. If you're using IronPython, this may not work.

You will notice I didn't need to touch getattr to have it work, let alone hack the interpreter to make it some sort of magical construct. It all works through `__getattribute__`.

The registration system is just the easiest thing that I could throw together. There are surely better designs.

Run A.py to see it in action.

Okay, so you've hidden the magic away a bit, but you have to choose the number [2] for your stack inspection. That means you have to be sure that that's the correct module, in some way. If you do *anything* to disrupt the exact depth of the call stack, that breaks.

_hasattr = hasattr def hasattr(obj, attr): return _hasattr(obj, attr)

Or any of the other higher level constructs. What if there's a C-level function in there?

This is still magic. It's just that the magic has been buried slightly.

ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RPBIHJ... Code of Conduct: http://python.org/psf/codeofconduct/

On Fri, Jun 25, 2021 at 1:54 PM Ricky Teachey wrote:

Would this feature allow me to declare str objects as not iterable in some contexts?

If so, +1.

That depends. If the proposal is to intercept every attribute lookup (parallel to a class's __getattribute__ method), then yes, but if it's a fallback after default behaviour fails (like __getattr__), then no. I suspect that the latter is more likely; it's much easier to avoid recursion problems if real attributes are tried first, plus it's likely to impact performance a lot less. ChrisA

Chris Angelico writes:

if it's a fallback after default behaviour fails (like __getattr__) [...] it's likely to impact performance a lot less.

I guess the effect on performance is that it takes one more check to get to AttributeError?

On Fri, Jun 25, 2021 at 7:43 PM Stephen J. Turnbull wrote:

Chris Angelico writes:

...

if it's a fallback after default behaviour fails (like __getattr__) [...] it's likely to impact performance a lot less.

I guess the effect on performance is that it takes one more check to get to AttributeError?

It also means that this code applies only when other things have failed, so high performance lookups will still be high performance. But there've been many variants of this proposal in this thread, all subtly different. ChrisA