[Web-SIG] Standardized template API

[Ben Bangert]

On Feb 3, 2006, at 2:24 PM, Phillip J. Eby wrote:

>> Wait, then how does the template engine know what template to render?
>
> The framework gives the engine the source of the template, and asks  
> for a WSGI app object in return.  The framework is allowed to hold  
> on to this application object and reuse it as often as desired.

I'm really not sure why the framework developer should be further  
burdened with trying to cache templates. Putting additional demands  
on the framework developer will create such a burden that I'd highly  
prefer to use TurboGears plug-in system and ignore any spec requiring  
such additional effort.

> Well, the idea I was trying to do here was to allow the embedder to  
> control how the data is stored.  I think of this as a bit like OLE  
> embedding, where the enclosing application just hands off a stream  
> to the class that implements the embedded object or document.

I don't understand that at all, but probably because I don't do OLE  
embedding and all that jazz.

> What I'd suggest is that this is actually an opportunity for Myghty  
> to put its lookup and caching on one side of the interface, and an  
> actual template rendering facility on the other side.  That is,  
> treat Myghty as an embedd*er* rather than the embedd*ee*, so that  
> the engine can be used with arbitrary apps or templates on the  
> embedded side.

Myghty's lookup and caching is rather complex and requires  
significant synchronization efforts to ensure that under heavy load,  
if the template has changed, its only re-compiled once while the rest  
of the requests continue to use the prior compiled version until the  
updated one is ready. It's complex, and putting it on completely  
separate sides is beyond a nightmare of effort.

The lookup mechanism it uses is also very complex, and it doesn't  
belong outside of Myghty.

> I'm not really familiar with Myghty, though, so I might not be  
> saying something sensible here.  I'm just saying that if you have a  
> mechanism for locating, compiling, and caching templates, then it  
> should live on the framework side of the "engine" interface.  An  
> embedding engine offers the "compile_string, compile_stream,  
> write_compiled, read_compiled" methods, and your locator/compiler/ 
> cache system would just call down to those instead of hardwiring  
> the implementation of compiling and serializing.  You would of  
> course have to add in a way to tell what embedding engine to select  
> in that case, but now Myghty itself (if I understand you correctly)  
> becomes a WSGI embedding host that can run any kind of template  
> using its lookup/cache/etc. facilities.  Does that make sense?

Yes, but its 100% incompatible with Myghty. For the flexibility and  
reloading functionality that Myghty provides, splitting such  
functionality up will only cripple features of Myghty that I really  
enjoy.

It also seems to go far beyond the scope of merely having a  
standardized way of calling a template, as its now somehow going to  
handle caching it, and knowing when its ok to use a cached copy, vs a  
new one, etc. This might work for other template languages that  
merely render a single template and give you its output, but Myghty  
has things like automatic inheritance with its own internal resolver  
for assembling the fully rendered template in the context of its  
location.

Moving this stuff onto the framework side reduces the power of the  
template language. I'm not sure about the other template systems,  
some of them do some inheritance stuff that might be affected as well.

Also, Myghty has per-component (template) caching systems built-in  
which allow individual parts to be cached on separate criteria (maybe  
your side-bar refreshes every 10 seconds, and the page header every 2  
minutes). Caching from outside would remove this ability since these  
fine-grained caching is only available when Myghty is handling its  
caching.

While the thought that you can split off the lookup/cache  
functionality is nice in theory, the reality is that imposing such a  
system also reduces the flexibility and capabilities of the template  
language. For example, in Myghty (and I believe Kid, Cheetah, etc.),  
you can include and call other components (templates) from within one  
template. How does the template lookup the other template you're  
including? It has its own resolver.

When it loads that other template inside a template, what if the  
other template should be cached? This kind of stuff won't work with  
what you're proposing as far as I can tell. If it can, the amount of  
work it will push onto every framework developer will likely doom  
such a proposal for the start, especially with something much simpler  
available (TG's template-plugin approach)

> No; what happens is that the framework locates the serialized form  
> of the template it wants to use, and requests that a specific  
> engine deserialize it to get a WSGI app (callable, not iterable).   
> The framework is allowed to cache the returned app object and reuse  
> it for multiple requests.  The engine doesn't get any data about  
> the current request.

This kind of work sounds best suited for the template language to  
deal with. Especially since template languages differ on how they do  
it substantially, and some of them can't decouple the process from  
the actual render without reducing powerful functionality it affords.

> The app object is used to "render" the template.  It's just called  
> like a normal WSGI app object.

Since Myghty allows template's and template calls within templates to  
be cached on varying keys and expiration times, this cannot be  
decoupled from the compile step.

> I think you'll find that the actual design is even better than the  
> one you interpreted it as, so I think we're doing good.  :)

Not really, the one I interpreted kept vital steps together. Without  
those steps coupled closely, Myghty cannot be compatible without very  
significant changes to underlying structures. I'm not sure what work  
will be necessary for other template languages that compile and  
manage their own caches, but since it can be a very integrated  
process I'd imagine it'd be equally horrible.

> These divisions may cut things at slightly different points than  
> what people were originally thinking.  For example, it sounds from  
> your description like Myghty might have parts in all three areas.   
> So, if it were to implement WSGI embedding, it would become  
> componentized along those lines, making each part potentially  
> interchangeable or reusable.

I thought Cheetah also compiled its templates and would reload them  
if it detected changes. If a template changes on disk, thats the  
great thing about template systems that handle stuff like this. You  
ask for the rendered template, and if it changes, it'll recompile/re- 
cache it, and render it for you.

Decoupling this process removes a very powerful and convenient  
feature. One I'm not ready to part with.

> I appreciate your feedback and questions a lot, as they're helping  
> me improve how I communicate the ideas.  Hopefully, I'll soon be  
> able to explain it with examples and metaphors so that it doesn't  
> take this much back-and-forth.  Despite Ian's fears, I'm not really  
> worried that it's too complex; it's hugely simpler than WSGI  
> itself, and is probably less complex than even the idea of WSGI  
> middleware.  I am just not that good at *communicating* it simply yet.

Perhaps, but if this spec is extremely difficult to implement (as is  
Myghty's case, and maybe Kid/Cheetah as well), or removes  
functionality from existing solutions, I don't see it going anywhere.

It might not be complex for whatever template system you use, but its  
a drastically different situation for others. A pure WSGI call to a  
template engine like I thought you were referring to was more  
agnostic on how the template language deals with it, the key appeal  
and flexibility to me.

Cheers,
Ben