[Web-SIG] Standardized template API

[Ben Bangert]

On Feb 3, 2006, at 3:49 PM, Phillip J. Eby wrote:

> Well, somebody's got to do it.  :)  Not every template facility out  
> there currently has any kind of location ability or caching.  And  
> different frameworks have their own mechanisms for locating,  
> caching, etc.

True, and caching can be a tough problem, so it'd be nice if caching  
code could be re-used at least.

> I don't understand.  Are you saying there's nothing that  
> corresponds to reading source, compiling, and writing out a  
> compiled version, that would just change to simply call those  
> methods on a different object than the ones they're called on now?

No, you're right, there is mechanisms that do exactly what you describe.

> I don't understand what you mean by "outside of Myghty" in this  
> context.  My suggestion was simply that would gain pluggability in  
> one main area: the implementation of compiling and serialization of  
> templates.  (i.e., the syntax/semantics of the templates used)

I meant decoupled from the request to render the template. Since  
rendering the template may or may not require reloading/re-caching  
it, it seems like a process that the framework shouldn't have to  
consider.

> I'm still not understanding.  Since Myghty will be in control of  
> all the streams, it will have full ability to monitor changes etc.   
> The only thing that's different under the embedding scheme is that  
> 1: it would support multiple formats for the actual templates, and  
> 2: it would return WSGI apps that frontend access to an individual  
> template.

Yes, that is useful functionality. Would the call to the WSGI app  
that renders the template also trigger something that ensures its re- 
compiled if it needs to be?

> It's not as different as you think.  In your idea, Myghty took each  
> WSGI call.  In the actual idea, Myghty would take a call that  
> returns a callable.  That callable gets called for each WSGI call.   
> You still have total control over what happens in each WSGI call;  
> it's just that *which* template is to be used gets factored out by  
> the first call.  It doesn't decouple any of the things you're  
> talking about, as best I understand it.

Got it, that makes sense.

> Ah.  I didn't realize this was so common.  I'm accustomed to  
> template systems that don't control so much stuff, but which are  
> more at the compile/save level.  That may be more of a mismatch,  
> making my idea much less useful for the template use cases that  
> people currently have in mind.  :(

It usually is, which is part of the reason it might be nice to have  
it decoupled so the most robust compiler/loader can be used, if thats  
possible. Such template agnosticism might come at a very hefty  
workload cost, as Mike mentioned about how much time it'd take to  
splice this functionality into separate parts.

> I'm seeing now that the key problem with my view of the situation  
> is that I've been assuming *frameworks* have control over where and  
> how templates are stored, and considering the template systems to  
> be just that - implementation of the template itself.

Ah, yes. That explains a lot. In my experience with various template  
systems, the directory structure of the template root can also  
influence the rendering of the template. In Myghty this structure is  
used for automatic inheritance, Cheetah, Kid, Django templates, and  
others all include the ability to "extend" existing templates within  
a template. This means they all need to resolve the included  
template, and cache it. That reason is why all of them already have  
their own caching/compiling stuff integrated.

> One reason why I also wanted us to focus on template/resource  
> deployment, was because that would help unify caching/compilation  
> concerns and reach a common way to address and access such  
> resources across e.g. different eggs.

Indeed, since the resolver scheme is likely different for different  
template systems, it'd be more likely that purely caching can be done  
separately. Some templates aren't compiled at all for example (I  
think Django's are always rendered on the fly, not 100% sure though).  
There's variations in how the template is handled though.

> I think the best thing to do at this point is for me to try to  
> actually write up a spec that's clear enough to be understood.  I  
> think that probably the issue here is that there are actually  
> *four* roles, not three or two, and I've been lumping some of them  
> together to make three, and most everybody else has been lumping  
> them into two, and so none of our divisions match each other or  
> reality.  :)

Quite likely.

> So, on a given request, the publisher asks the manager for a  
> resource by saying what template it wants.  The manager does  
> whatever it needs to do, possibly invoking the compiler if it needs  
> to.  The publisher then makes a WSGI call on the resource to render  
> it.

Where along these steps, are the variables passed into the template?  
Once the template is compiled/cached, in addition to calling it, I  
need to pass it variables that it will be using. Are all templates  
going to be expected to pull variables they should see from the environ?

> I was at first assuming that "framework" meant the publisher and  
> manager, and "engine" was the compiler and resource.  Then to  
> explain better I split compiler and resource, but now I see that I  
> should've split (conceptually) the publisher and manager.

Ok, so if I'm using this in my web application. Somehow, I'm going to  
need to check to see if the template has been compiled/cached, then I  
have to go and call the WSGI app? With all these compiled/cached  
templates that've been used, what's holding onto all those many, many  
WSGI apps that've been created? Some of my web applications have  
upwards of thousands of templates, is there going to be thousands of  
WSGI apps in memory waiting to be called? I'm guessing/hoping I  
missed something and this isn't the case.

> Now, if I'm understanding correctly, Myghty is manager+compiler 
> +resources, so from the publisher point of view there's no real  
> change from your interpretation; a name goes in and a WSGI callable  
> comes out.  And, from a WSGI embedding point of view, that's really  
> enough.  The only change from your previous idea is that you return  
> some object that the publisher makes a WSGI call on, instead of  
> doing it as a single WSGI call.  However, Myghty is still  
> integrated in its control over the handling.

That's correct. Typically, in a lot of the MVC frameworks, like  
Aquarium, CherryPy, Pylons, Django, etc. the web developer will be  
doing some logic in the controller. They will then decide to render a  
template. The controller is the 'publisher', and the template to be  
rendered is always provided by name, perhaps based on what variables  
the controller sees.

I can see from a Zope oriented view that this is rather different  
from where you'd have a Publisher that determines the template to be  
used for an object. In the frameworks I mentioned, it works a bit  
differently so all thats of most concern to us is that we can render  
a template by name, against a configured template directory root path.

> Does that make sense now, or am I missing something again?

I think it makes sense, we're definitely coming at this from  
different perspectives on what's going on, and where.

Since templates in a bunch of these template languages will  
frequently include other templates, all the 'resource' instances will  
need access to call back to the 'publisher' so that they can do what  
they need to for the template to be rendered and included. Some  
template systems go farther still than basic including of another  
template and start extending other templates. Kid will have one  
template override parts of templates it "inherits" from. Myghty will  
have parent templates that will insert their "child" templates in  
various fashions. Cheetah and Django also do things like this.

All these systems will need ways to tie back into the manager so that  
they're all loaded, compiled, cached when needed, and that individual  
parts of the whole get reloaded when necessary without reloading  
everything.

If there's a way to separate out this very convoluted process (that  
isn't necessarily the same for each system), it'd be very compelling  
I think. Especially since solid code to handle many of the problems  
that can occur under load when doing all this reloading/caching/etc  
is not easy stuff.

I think many of us are actually talking about two different  
proposals. The proposal for standardized templating you're talking to  
sounds like WSGI for template languages. That is, it pushes common  
parts many template languages have, into a re-usable layer. I'm not  
sure such a thing is possible, its definitely quite ambitious, and I  
look forward to the more clearly defined spec you put out.

The proposal many of us were looking for (I think), was merely a more  
unified way of calling a template. That is, we want a standard  
'publisher' interface, so that we can call someengine.render('some/ 
template.myt', myvars) and its rendered. The TurboGears template plug- 
in thing describes such a thing, but it could use some more work. The  
way I interpreted your proposal originally also set things up in such  
a way that there was a base system for configuring a template engine,  
then you just call the app and put your vars in environ.

As the more ambitious one is likely to take significantly more effort  
from everyone involved (assuming it'll work), can we unify a setup/ 
call structure first? I see no reason it won't be compatible in the  
end with standardizing the "internals" of the template languages as  
well.

Cheers,
Ben