metaclass confusions

[Alex Martelli]

Manuel M. Garcia wrote:
   ...
> (Alex, does your Nutshell book have a section explaining the details
> of this stuff?  If it does, I promise to pre-order 3 copies from
> Amazon: one for work, one for home, and one to put under my pillow
> when I sleep!  ;-)

Yes, I do cover metaclasses in the Nutshell, concisely but,
I hope, in enough details to be of help.  Apologies for any
error in the following -- I'm being a bit hasty as I have
to get back to proofreading that very Nutshell, but I thought
it more helpful to post now rather than wait a week...:-)

> class meta0(type):
>     
>     def __init__(cls, classname, bases, classdict):
>         cls.all_dict = {}
> 
>     # this also worked, but is this overkill?
>     ##def __new__(cls, classname, bases, classdict):
>     ##    classdict['all_dict'] = {}
>     ##    return type.__new__(cls, classname, bases, classdict)

Why should it be overkill?  You override method __new__ of
your superclass (which is the built-in "type") and in your
override you delegate to the superclass implementation after
some tweaking.  Very typical, I'd call this.

>     # neither of these two worked...
>     ##def __init__(cls, classname, bases, classdict):
>     ##    classdict['all_dict'] = {}

Too late -- __init__ runs after __new__ and it's __new__
that has already used classdict -- it's not used any
more now.

>     ##def __init__(cls, classname, bases, classdict):
>     ##    cls.__dict__['all_dict'] = {}

Just a weird way to spell cls.all_dict = {}  but
otherwise equivalent.  What do you mean it didn't work?


> class klass0(object):
>     
>     __metaclass__ = meta0
>     
>     def __new__(cls, a):
>         
>         # can use this instead of metaclass
>         ##try:
>         ##    cls.all_dict
>         ##except AttributeError:
>         ##    cls.all_dict = {}

Yes, you can, it's just more overhead.  Very few
things can be done ONLY with metaclasses -- more
often, the metaclasses are just way faster/neater.

>         if cls.all_dict.has_key(a):
>             return cls.all_dict[a]
>         else:
>             n = object.__new__(cls)
>             n.count = len(cls.all_dict)
>             cls.all_dict[a] = n
>             return n

Now THIS is something I'd much rather code as a
try/except:
    try: return cls.all_dict[a]
    except KeyError: pass
    [etc]
because I hate to duplicate work (has_key and
the access are duplicating each other's work).
But this has little to do with metaclasses.

Note that when your klass0.__new__ returns an
instance n of klass0, even an en existing one,
n.__init__ is run next -- tiny duplication of
work here, but worse in other cases, see later,
as this is your question [3].


> It works, and this is the code I will go with, but I had some
> questions.
> 
> 1) In the metaclass, I now think I understand the difference between
> __new__ and __init__.  Would a metaclass even have both __new__ and
> __init__ defined, and if so why?  Would a metaclass ever have any
> other methods defined?  I guess not, because a metaclass only comes
> into play during the creation of a subclass, so how could those other
> methods ever get run?

For example, you could perfectly well define the metaclass's
__call__ to determine what it MEANS to call your class -- it
doesn't HAVE to be __new__ then conditionally __init__, it's
entirely up to you.


> 2) I think this is the correct way to use metaclass: to run code just
> once at the creation of a subclass, as opposed to code that has to be
> run with every instance creation.  Is this more or less the only
> reason to have a metaclass?  In general, it is hard for me to get my
> head around exactly what is the difference between a base class and a
> metaclass.

Say you have a bunch of classes that you'd like to have
the same behavior pattern at instance creation -- e.g. you
might want to ensure "__init__ is only called on NEW
instances" as a part of a pattern very close to what
you're doing now, and you might want to ensure it across
several classes.  The right way would then be to define
the metaclass's __call__ so it doesn't even call __new__ 
when it can fetch an existing instance.  Bases can't do that.


> 3) klass0 has __new__ and __init__ defined.  When __new__ recognizes a
> instance creation argument, it returns an instance from before.
> __init__ gets run against this instance.  Nothing bad happens, because
> __init__ is cheap to run, and does nothing to destroy data already in
> the instance.  Am I right in thinking that there is no way for __new__
> to tell __init__ it doesn't need to run, except with some ad-hoc
> techniques with special purpose attributes of the instance?

__init__ will run iff __new__ returns an instance of the class.
If you can identify "equivalent instances" by the arguments
passed when calling the class, you've got it made -- e.g assume
all arguments will always be of hashable types...:

class meta1(type):
    
    def __init__(cls, classname, bases, classdict):
        cls.all_dict = {}

    def __call__(cls, *args):
        try: return cls.all_dict[args]
        except KeyError: pass
        n = cls.__new__(cls, *args)
        if isinstance(n, cls): n.__init__(*args)
        return n

Here the class need no awareness of the "equivalent
instances" issue -- meta1 handles that.  More generally,
the class might be tasked to find an equivalent
instance if any by a method whose name YOU decide:


    class NoEquivalent(Exception): pass

    def equivalent_to(cls, *args, **kwds):
        """ you can code a default implementation here,
            e.g. based on dict lookup w. pickle, but I won't... """
        raise cls.NoEquivalent

    def __call__(cls, *args, **kwds):
        try: return cls.equivalent_to(*args, **kwds)
        except cls.NoEquivalent: pass
        n = cls.__new__(cls, *args, **kwds)
        if isinstance(n, cls): n.__init__(*args, **kwds)
        return n


Does this help...?


Alex