Module __getattr__ [Was: breaking out of module execution]

On Wed, Apr 25, 2012 at 2:35 PM, Matt Joiner <anacrolix@gmail.com> wrote:
If this is to be done I'd like to see all special methods supported. One of particular interest to modules is __getattr__...
For What It's Worth, supporting __setattr__ and __getattr__ is one of the few reasons that I have considered subclassing modules. The workarounds of either offering public set_varX and get_varX functions, or moving configuration to a separate singleton, just feel kludgy. Since those module methods would be defined at the far left, I don't think it would mess up understanding any more than they already do on regular classes. (There is always *some* surprise, just because they are unusual.) That said, I personally tend to view modules as a special case of classes, so I wouldn't be shocked if others found it more confusing than I would -- particularly as to whether or not the module's __getattr__ would somehow affect the lookup chain for classes defined within the module. -jJ

On Fri, Apr 27, 2012 at 6:31 AM, Jim Jewett <jimjjewett@gmail.com> wrote:
Making modules "simply" be a class that could be subclasses rather than their own thing _would_ be nice for one particular project I've worked on where the project including APIs and basic implementations were open source but which allowed for site specific code to override many/most of those base implementations as a way of customizing it for your own specific (non open source) environment. Any APIs that were unfortunately defined as a module with a bunch of functions in it was a real pain to make site specific overrides for. Anything APIs that were thankfully defined as a class within a module even when there wasn't a real need for a class was much easier to make site specific. But this is not an easy thing to do. I wouldn't want functions in a module to need to be declared as class methods with a cls parameter nor would I want an implicit named equivalent of cls; or does that already exist through an existing __name__ style variable today that I've been ignoring? This could leads to basically treating a module globals() dict as the class dict which at first glance seems surprising but I'd have to ponder this more. (and yes, note that I am thinking of a module as a class, not an instance) -gps

Gregory P. Smith wrote:
This makes no sense to me. What does the *licence* of a project have to do with the library API? I mean, yes, you could do such a thing, but surely you shouldn't. That would be like saying that the accelerator pedal should be on the right in cars you buy outright, but on the left for cars you get on hire-purchase. Nevertheless, I think your focus here is on the wrong thing. It seems to me that you are jumping to an implementation, namely that modules should stop being instances of a type and become classes, without having a clear idea of your functional requirements. The functional requirements *might* be: "There ought to be an easy way to customize the behaviour of attribute access in modules." Or perhaps: "There ought to be an easy way for one module to shadow another module with the same name, but still inherit behaviour from the shadowed module." neither of which *require* modules to become classes. Or perhaps it is something else... it is unclear to me exactly what problems you and Jim wish to solve, or whether they're the same kind of problem, which is why I say the functional requirements are unclear. Changing modules from an instance of ModuleType to "a class that could be a subclass" is surely going to break code. Somewhere, someone is relying on the fact that modules are not types and you're going to break their application.
It shouldn't be. Just ensure the site-specific override module comes first in the path, and "import module" will pick up the override module instead of the standard one. This is a simple exercise in shadowing modules. Of course, this implies that the override module has to override *everything*. There's currently no simple way for the shadowing module to inherit functionality from the shadowed module. You can probably hack something together, but it would be a PITA. -- Steven

On Mon, Apr 30, 2012 at 5:47 PM, Steven D'Aprano <steve@pearwood.info> wrote:
That's an irrelevant, surprising and unfair criticism of Greg's message. He just tried to give a specific example without being too specific.
If there is a bunch of functions and you want to replace a few of those, you can probably get the desired effect quite easily: from base_module import * # Or the specific set of functions that comprise the API. def funct1(<args>): <my version> def funct2(<args>): <my version> Not that I would recommend this -- it's easy to get confused if there are more than a very small number of functions. Also if base_module.funct3 were to call func2, it wouldn't call the overridden version. But all attempts to view modules as classes or instances have lead to negative results. (I'm sure I've thought about it at various times in the past.) I think the reason is that a module at best acts as a class where every method is a *static* method, but implicitly so. Ad we all know how limited static methods are. (They're basically an accident -- back in the Python 2.2 days when I was inventing new-style classes and descriptors, I meant to implement class methods but at first I didn't understand them and accidentally implemented static methods first. Then it was too late to remove them and only provide class methods.) There is actually a hack that is occasionally used and recommended: a module can define a class with the desired functionality, and then at the end, replace itself in sys.modules with an instance of that class (or with the class, if you insist, but that's generally less useful). E.g.: # module foo.py import sys class Foo: def funct1(self, <args>): <code> def funct2(self, <args>): <code> sys.modules[__name__] = Foo() This works because the import machinery is actively enabling this hack, and as its final step pulls the actual module out of sys.modules, after loading it. (This is no accident. The hack was proposed long ago and we decided we liked enough to support it in the import machinery.) You can easily override __getattr__ / __getattribute__ / __setattr__ this way. It also makes "subclassing" the module a little easier (although accessing the class to be used as a base class is a little tricky: you'd have to use foo.__class__). But of course the kind of API that Greg was griping about would never be implemented this way, so that's fairly useless. And if you were designing a module as an inheritable class right from the start you're much better off just using a class instead of the above hack. But all in all I don't think there's a great future in stock for the idea of allowing modules to be "subclassed". In the vast, vast majority of cases it's better to clearly have a separation between modules, which provide no inheritance and no instantiation, and classes, which provide both. I think Python is better off this way than Java, where all you have is classes (its packages cannot contain anything except class definitions). -- --Guido van Rossum (python.org/~guido)

On Tue, May 1, 2012 at 12:33 PM, Guido van Rossum <guido@python.org> wrote:
FWIW, in 3.3 the full import machinery will be exposed in sys.meta_path (and sys.path_hooks), so third parties will be free to experiment with whatever crazy things they want without having to work around the implicit import behaviour :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Apr 30, 2012 at 7:33 PM, Guido van Rossum <guido@python.org> wrote:
heh. right. I didn't want to drag people into a project that shall not be named because I dislike its design to begin with... I was just suggesting a case where we _would_ have found it useful to treat a module as a class that could be subclassed. A better overall design pattern using classes in the code's public APIs to start with would have prevented all such need. I am not strongly in favor of modules being classes but it is an interesting thing to ponder from time to time.
This "oops, I implemented static methods" is a wonderful bit of history! :)

On Fri, Apr 27, 2012 at 6:31 AM, Jim Jewett <jimjjewett@gmail.com> wrote:
Making modules "simply" be a class that could be subclasses rather than their own thing _would_ be nice for one particular project I've worked on where the project including APIs and basic implementations were open source but which allowed for site specific code to override many/most of those base implementations as a way of customizing it for your own specific (non open source) environment. Any APIs that were unfortunately defined as a module with a bunch of functions in it was a real pain to make site specific overrides for. Anything APIs that were thankfully defined as a class within a module even when there wasn't a real need for a class was much easier to make site specific. But this is not an easy thing to do. I wouldn't want functions in a module to need to be declared as class methods with a cls parameter nor would I want an implicit named equivalent of cls; or does that already exist through an existing __name__ style variable today that I've been ignoring? This could leads to basically treating a module globals() dict as the class dict which at first glance seems surprising but I'd have to ponder this more. (and yes, note that I am thinking of a module as a class, not an instance) -gps

Gregory P. Smith wrote:
This makes no sense to me. What does the *licence* of a project have to do with the library API? I mean, yes, you could do such a thing, but surely you shouldn't. That would be like saying that the accelerator pedal should be on the right in cars you buy outright, but on the left for cars you get on hire-purchase. Nevertheless, I think your focus here is on the wrong thing. It seems to me that you are jumping to an implementation, namely that modules should stop being instances of a type and become classes, without having a clear idea of your functional requirements. The functional requirements *might* be: "There ought to be an easy way to customize the behaviour of attribute access in modules." Or perhaps: "There ought to be an easy way for one module to shadow another module with the same name, but still inherit behaviour from the shadowed module." neither of which *require* modules to become classes. Or perhaps it is something else... it is unclear to me exactly what problems you and Jim wish to solve, or whether they're the same kind of problem, which is why I say the functional requirements are unclear. Changing modules from an instance of ModuleType to "a class that could be a subclass" is surely going to break code. Somewhere, someone is relying on the fact that modules are not types and you're going to break their application.
It shouldn't be. Just ensure the site-specific override module comes first in the path, and "import module" will pick up the override module instead of the standard one. This is a simple exercise in shadowing modules. Of course, this implies that the override module has to override *everything*. There's currently no simple way for the shadowing module to inherit functionality from the shadowed module. You can probably hack something together, but it would be a PITA. -- Steven

On Mon, Apr 30, 2012 at 5:47 PM, Steven D'Aprano <steve@pearwood.info> wrote:
That's an irrelevant, surprising and unfair criticism of Greg's message. He just tried to give a specific example without being too specific.
If there is a bunch of functions and you want to replace a few of those, you can probably get the desired effect quite easily: from base_module import * # Or the specific set of functions that comprise the API. def funct1(<args>): <my version> def funct2(<args>): <my version> Not that I would recommend this -- it's easy to get confused if there are more than a very small number of functions. Also if base_module.funct3 were to call func2, it wouldn't call the overridden version. But all attempts to view modules as classes or instances have lead to negative results. (I'm sure I've thought about it at various times in the past.) I think the reason is that a module at best acts as a class where every method is a *static* method, but implicitly so. Ad we all know how limited static methods are. (They're basically an accident -- back in the Python 2.2 days when I was inventing new-style classes and descriptors, I meant to implement class methods but at first I didn't understand them and accidentally implemented static methods first. Then it was too late to remove them and only provide class methods.) There is actually a hack that is occasionally used and recommended: a module can define a class with the desired functionality, and then at the end, replace itself in sys.modules with an instance of that class (or with the class, if you insist, but that's generally less useful). E.g.: # module foo.py import sys class Foo: def funct1(self, <args>): <code> def funct2(self, <args>): <code> sys.modules[__name__] = Foo() This works because the import machinery is actively enabling this hack, and as its final step pulls the actual module out of sys.modules, after loading it. (This is no accident. The hack was proposed long ago and we decided we liked enough to support it in the import machinery.) You can easily override __getattr__ / __getattribute__ / __setattr__ this way. It also makes "subclassing" the module a little easier (although accessing the class to be used as a base class is a little tricky: you'd have to use foo.__class__). But of course the kind of API that Greg was griping about would never be implemented this way, so that's fairly useless. And if you were designing a module as an inheritable class right from the start you're much better off just using a class instead of the above hack. But all in all I don't think there's a great future in stock for the idea of allowing modules to be "subclassed". In the vast, vast majority of cases it's better to clearly have a separation between modules, which provide no inheritance and no instantiation, and classes, which provide both. I think Python is better off this way than Java, where all you have is classes (its packages cannot contain anything except class definitions). -- --Guido van Rossum (python.org/~guido)

On Tue, May 1, 2012 at 12:33 PM, Guido van Rossum <guido@python.org> wrote:
FWIW, in 3.3 the full import machinery will be exposed in sys.meta_path (and sys.path_hooks), so third parties will be free to experiment with whatever crazy things they want without having to work around the implicit import behaviour :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Apr 30, 2012 at 7:33 PM, Guido van Rossum <guido@python.org> wrote:
heh. right. I didn't want to drag people into a project that shall not be named because I dislike its design to begin with... I was just suggesting a case where we _would_ have found it useful to treat a module as a class that could be subclassed. A better overall design pattern using classes in the code's public APIs to start with would have prevented all such need. I am not strongly in favor of modules being classes but it is an interesting thing to ponder from time to time.
This "oops, I implemented static methods" is a wonderful bit of history! :)
participants (5)
-
Gregory P. Smith
-
Guido van Rossum
-
Jim Jewett
-
Nick Coghlan
-
Steven D'Aprano