[Web-SIG] Standardized template API

[Phillip J. Eby]

At 05:07 PM 2/3/2006 -0800, Ben Bangert wrote:
>On Feb 3, 2006, at 3:49 PM, Phillip J. Eby wrote:
>>I don't understand what you mean by "outside of Myghty" in this
>>context.  My suggestion was simply that would gain pluggability in
>>one main area: the implementation of compiling and serialization of
>>templates.  (i.e., the syntax/semantics of the templates used)
>
>I meant decoupled from the request to render the template. Since
>rendering the template may or may not require reloading/re-caching
>it, it seems like a process that the framework shouldn't have to
>consider.

My thoughts on caching were in the context of assuming the "manager" was 
part of the framework.  Frameworks that implement skins/layers and 
localizable resources have to do this, because template systems generally 
don't have any concept of skins or layers.  But it's seeming like we'll 
have to punt on this to some extent for now.


>>I'm still not understanding.  Since Myghty will be in control of
>>all the streams, it will have full ability to monitor changes etc.
>>The only thing that's different under the embedding scheme is that
>>1: it would support multiple formats for the actual templates, and
>>2: it would return WSGI apps that frontend access to an individual
>>template.
>
>Yes, that is useful functionality. Would the call to the WSGI app
>that renders the template also trigger something that ensures its re- 
>compiled if it needs to be?

Sure.  You can just do something like:

def resource_returned_by_manager(environ, start_response):
     if needs_recompiling(actual_resource):
         recompile(actual_resource)
     return actual_resource(environ, start_response)

That is, the manager doesn't have to return the resource returned by the 
compiler; it can wrap a trivial reloader middleware around it and return that.

Of course, the more I look at this, the more I realize that your idea of 
just putting the entire "manager" interface in one WSGI call might make 
good sense too.  Because unless the "publisher" is going to hang on to the 
returned app, you might as well just integrate this stuff at the manager level.

The problem for me is, I was hoping we would end up defining a 
manager->compiler interface and a publisher->resource interface rather than 
a publisher->manager one.  Then again, maybe we should define both, and let 
people plug things in where they can.  It would provide a path for 
gradually evolving to the place where everything's pluggable.


>It usually is, which is part of the reason it might be nice to have
>it decoupled so the most robust compiler/loader can be used, if thats
>possible. Such template agnosticism might come at a very hefty
>workload cost, as Mike mentioned about how much time it'd take to
>splice this functionality into separate parts.

I'm leaning more now towards defining a publisher->manager and 
publisher->resource interface pair based on WSGI as making the most sense 
for the present.  A manager->compiler interface would be nice too, but 
perhaps not strictly necessary for the original use cases that prompted 
this discussion.


>>I'm seeing now that the key problem with my view of the situation
>>is that I've been assuming *frameworks* have control over where and
>>how templates are stored, and considering the template systems to
>>be just that - implementation of the template itself.
>
>Ah, yes. That explains a lot. In my experience with various template
>systems, the directory structure of the template root can also
>influence the rendering of the template. In Myghty this structure is
>used for automatic inheritance, Cheetah, Kid, Django templates, and
>others all include the ability to "extend" existing templates within
>a template. This means they all need to resolve the included
>template, and cache it. That reason is why all of them already have
>their own caching/compiling stuff integrated.

In my view, this is a bit suboptimal in the long haul, because there's all 
this duplicated code for managing files, and probably little -- if any -- 
of it works with eggs, or templates in a database (e.g. ZODB), etc.  This 
is part of why my mental model didn't really consider much about template 
systems having their own "manager" parts - the idea seems manifestly 
unworkable if you need to do things like what Zope 3 supports.

Of course, my long-term ambition here would be that we get to a place where 
you can do skinning and localization with templates and other resources, 
all using a Python-wide standard for how you identify and deploy these 
resources.  But that's out of scope for the current interface discussions, 
alas.

It might be nice, though, to put together a session at PyCon to do some 
brainstorming on templates, embedding, and deployment to see if there are 
currently any easily-reached opportunities.


>>So, on a given request, the publisher asks the manager for a
>>resource by saying what template it wants.  The manager does
>>whatever it needs to do, possibly invoking the compiler if it needs
>>to.  The publisher then makes a WSGI call on the resource to render
>>it.
>
>Where along these steps, are the variables passed into the template?

When the publisher makes the WSGI call, it would add a 'params' key and/or 
a 'published_object' key, depending on whether the publisher is an object 
publisher or does parameters or both.  I can imagine that Zope, for 
example, might do both, drawing on its request parsing to do parameters.  A 
TurboGears or other controller-type framework that uses templates for 
postprocessing would put the variables in 'params'.  These names are 
hypothetical and subject to getting a firm spec, of course.


>Once the template is compiled/cached, in addition to calling it, I
>need to pass it variables that it will be using. Are all templates
>going to be expected to pull variables they should see from the environ?

Yeah, from the 'params' key or 'published_object' or both.


>>I was at first assuming that "framework" meant the publisher and
>>manager, and "engine" was the compiler and resource.  Then to
>>explain better I split compiler and resource, but now I see that I
>>should've split (conceptually) the publisher and manager.
>
>Ok, so if I'm using this in my web application. Somehow, I'm going to
>need to check to see if the template has been compiled/cached, then I
>have to go and call the WSGI app? With all these compiled/cached
>templates that've been used, what's holding onto all those many, many
>WSGI apps that've been created? Some of my web applications have
>upwards of thousands of templates, is there going to be thousands of
>WSGI apps in memory waiting to be called? I'm guessing/hoping I
>missed something and this isn't the case.

Nah, the issue here was that I envisioned the division differently than 
Ian; my concern about being able to cache was because I envisioned the 
"manager" being on the framework side of the line; ergo, the framework 
needed caching.  If the interface is strictly between the publisher and the 
manager, then you could potentially just do it all with a single WSGI call, 
in which the manager pulls the template identifier from the environ.

But it seems a little ugly to me to cram another key into environ to tell 
the manager what template to use, when you could just have a 
'get(template_id)' call that gets you the WSGI app to use.  It also makes 
the objects' responsibilities a little more clearly defined, since a WSGI 
app in the general case doesn't take a template_id parameter, if you see 
what I mean.


>Since templates in a bunch of these template languages will
>frequently include other templates, all the 'resource' instances will
>need access to call back to the 'publisher' so that they can do what

To the manager actually, since the manager is responsible for id->template 
lookups.

>they need to for the template to be rendered and included. Some
>template systems go farther still than basic including of another
>template and start extending other templates. Kid will have one
>template override parts of templates it "inherits" from. Myghty will
>have parent templates that will insert their "child" templates in
>various fashions. Cheetah and Django also do things like this.

ZPT has macros, peak.web has DOMlets and layouts, etc., etc.  Lots of 
variety to choose from.  :)


>All these systems will need ways to tie back into the manager so that
>they're all loaded, compiled, cached when needed, and that individual
>parts of the whole get reloaded when necessary without reloading
>everything.
>
>If there's a way to separate out this very convoluted process (that
>isn't necessarily the same for each system), it'd be very compelling
>I think. Especially since solid code to handle many of the problems
>that can occur under load when doing all this reloading/caching/etc
>is not easy stuff.

I'm beginning to think that this actually ties in pretty closely with the 
deployment issues, in the sense that you may need basically multiple 
"sources" of template/resource implementations, especially if you're 
supporting skins or localization.  It's almost like you need a registry of 
template ids to ways to get them, that have been registered via plugins.

Of course, in the use cases you guys are talking about, you presume that 
there's just a "template root", which is way too simple for the kinds of 
stuff Zope 3 and peak.web want to do.  Both have the notion of "skins", 
which are composed of "layers".  A layer can provide alternate versions of 
any given template, and layers can be combined to form skins.  There's no 
particular requirement that these layers have any filesystem relationships, 
and in the Zope case they may be in ZODB in any case, not the 
filesystem.  In PEAK's case, the layers can live in different eggs, and I 
suspect Zope will want to allow the same as they increase their support for 
eggs.

Anyway, that's why I think that if we can come up with a "bottom up" way to 
provide template sources, then we can have a manager that's just a thin 
layer over these providers.  Stuff that's supplied by eggs, for example, 
will probably be precompiled and won't need to be refreshed.  Stuff in ZODB 
has its own caching.  So, to a certain extent, I think the ultimate future 
will be that our "manager" will just be something that figures out what 
backend is supplying the stuff and hand off the handling to it.  We'll have 
some "file" managers, "egg" managers, "ZODB" managers, etc. plugged in via 
entry points or some such.

Of course, this is all "future" stuff.  I had hoped we were closer to being 
able to do it, though, as I'd been meaning to build some of it for 
peak.web, and didn't want to end up in the position of appearing to say, 
"hey, everybody should do this stuff I did for peak.web".  :)  Instead, I'd 
be saying, "peak.web now supports the new WSGI embedding and resource 
deployment specs".  :)


>I think many of us are actually talking about two different
>proposals. The proposal for standardized templating you're talking to
>sounds like WSGI for template languages. That is, it pushes common
>parts many template languages have, into a re-usable layer. I'm not
>sure such a thing is possible, its definitely quite ambitious, and I
>look forward to the more clearly defined spec you put out.

Well, I'm realizing now that the more ambitious bits are harder because 
there's more that's actually been done out there in the template engines 
than I thought.  If there were less, more standardization would've been an 
obvious move forward and more doable now.


>The proposal many of us were looking for (I think), was merely a more
>unified way of calling a template. That is, we want a standard
>'publisher' interface, so that we can call someengine.render('some/ 
>template.myt', myvars) and its rendered. The TurboGears template plug- in 
>thing describes such a thing, but it could use some more work. The
>way I interpreted your proposal originally also set things up in such
>a way that there was a base system for configuring a template engine,
>then you just call the app and put your vars in environ.
>
>As the more ambitious one is likely to take significantly more effort
>from everyone involved (assuming it'll work), can we unify a setup/ call 
>structure first? I see no reason it won't be compatible in the
>end with standardizing the "internals" of the template languages as
>well.

I concur.  What I propose right now is that we have a "manager" interface 
that gets asked for a template by a name or ID, and returns a WSGI app 
object.  Calls to that object can then include 'params' and/or 
'published_object'.  (Exact spelling TBD.)

I don't think this interface should suggest/recommend an addressing scheme, 
nor a plugin scheme.  That is, I don't think it should include specifiers 
for entry points or anything like that.  Instead, to do the TG/B style of 
plugins, you should just have a manager that figures out what template 
engine/plugin to use.  The identifier can and should be opaque.

By that I mean, you could have a manager object that looks for a prefix 
like "kid:" at the beginning of the ID and then looks up an entry point to 
forward to another manager.  But that would just be one way of implementing 
the spec, not something required by the spec.  The reason is that this then 
leaves open both the possibility of developing generic and/or polymorphic 
"manager" objects.

The big open question is how to create and configure a manager object, 
since different managers will have different needs.  For example, peak.web 
uses module-relative paths in the default layer of the default skin; it has 
no "root" directory.  Each layer can also have its own directory, and I 
believe Zope 3 does the same.