[Python-ideas] Module __getattr__ [Was: breaking out of module execution]

[Guido van Rossum]

On Mon, Apr 30, 2012 at 5:47 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> Gregory P. Smith wrote:
>
>> Making modules "simply" be a class that could be subclasses rather than
>> their own thing _would_ be nice for one particular project I've worked on
>> where the project including APIs and basic implementations were open
>> source
>> but which allowed for site specific code to override many/most of those
>> base implementations as a way of customizing it for your own specific (non
>> open source) environment.

> This makes no sense to me. What does the *licence* of a project have to do
> with the library API? I mean, yes, you could do such a thing, but surely you
> shouldn't. That would be like saying that the accelerator pedal should be on
> the right in cars you buy outright, but on the left for cars you get on
> hire-purchase.

That's an irrelevant, surprising and unfair criticism of Greg's
message. He just tried to give a specific example without being too
specific.

> Nevertheless, I think your focus here is on the wrong thing. It seems to me
> that you are jumping to an implementation, namely that modules should stop
> being instances of a type and become classes, without having a clear idea of
> your functional requirements.
>
> The functional requirements *might* be:
>
> "There ought to be an easy way to customize the behaviour of attribute
> access in modules."
>
> Or perhaps:
>
> "There ought to be an easy way for one module to shadow another module with
> the same name, but still inherit behaviour from the shadowed module."
>
> neither of which *require* modules to become classes.
>
> Or perhaps it is something else... it is unclear to me exactly what problems
> you and Jim wish to solve, or whether they're the same kind of problem,
> which is why I say the functional requirements are unclear.
>
> Changing modules from an instance of ModuleType to "a class that could be a
> subclass" is surely going to break code. Somewhere, someone is relying on
> the fact that modules are not types and you're going to break their
> application.
>
>
>
>> Any APIs that were unfortunately defined as a
>> module with a bunch of functions in it was a real pain to make site
>> specific overrides for.
>
>
> It shouldn't be. Just ensure the site-specific override module comes first
> in the path, and "import module" will pick up the override module instead of
> the standard one. This is a simple exercise in shadowing modules.
>
> Of course, this implies that the override module has to override
> *everything*. There's currently no simple way for the shadowing module to
> inherit functionality from the shadowed module. You can probably hack
> something together, but it would be a PITA.

If there is a bunch of functions and you want to replace a few of
those, you can probably get the desired effect quite easily:

  from base_module import *  # Or the specific set of functions that
comprise the API.

  def funct1(<args>): <my version>
  def funct2(<args>): <my version>

Not that I would recommend this -- it's easy to get confused if there
are more than a very small number of functions. Also if
base_module.funct3 were to call func2, it wouldn't call the overridden
version.

But all attempts to view modules as classes or instances have lead to
negative results. (I'm sure I've thought about it at various times in
the past.)

I think the reason is that a module at best acts as a class where
every method is a *static* method, but implicitly so. Ad we all know
how limited static methods are. (They're basically an accident -- back
in the Python 2.2 days when I was inventing new-style classes and
descriptors, I meant to implement class methods but at first I didn't
understand them and accidentally implemented static methods first.
Then it was too late to remove them and only provide class methods.)

There is actually a hack that is occasionally used and recommended: a
module can define a class with the desired functionality, and then at
the end, replace itself in sys.modules with an instance of that class
(or with the class, if you insist, but that's generally less useful).
E.g.:

  # module foo.py

  import sys

  class Foo:
    def funct1(self, <args>): <code>
    def funct2(self, <args>): <code>

  sys.modules[__name__] = Foo()

This works because the import machinery is actively enabling this
hack, and as its final step pulls the actual module out of
sys.modules, after loading it. (This is no accident. The hack was
proposed long ago and we decided we liked enough to support it in the
import machinery.)

You can easily override __getattr__ / __getattribute__ / __setattr__
this way. It also makes "subclassing" the module a little easier
(although accessing the class to be used as a base class is a little
tricky: you'd have to use foo.__class__). But of course the kind of
API that Greg was griping about would never be implemented this way,
so that's fairly useless. And if you were designing a module as an
inheritable class right from the start you're much better off just
using a class instead of the above hack.

But all in all I don't think there's a great future in stock for the
idea of allowing modules to be "subclassed". In the vast, vast
majority of cases it's better to clearly have a separation between
modules, which provide no inheritance and no instantiation, and
classes, which provide both. I think Python is better off this way
than Java, where all you have is classes (its packages cannot contain
anything except class definitions).

-- 
--Guido van Rossum (python.org/~guido)