Why isn't Python king of the hill?

[Martijn Faassen]

Geoffrey  Gerrietts <geoff at homegain.com> wrote:
> A further episode in the discussion between Martijn and myself:

>> Or you can do evil and cool stuff with AccessRules (the whole
>> SiteRoot stuff).
>> [ ... quotes from me about controller-centered models ... ]
>> What exactly do you mean by embedding controller elements in the
>> view? I'm not entirely sure what you'd refer to as the 
>> controller here.

> I'm not sure how to manipulate AccessRules; I think this might be
> one of the Zope features that we currently hafta do without.

AccessRules are voodoo. I've only played with them a little. Here is
an example of what you can do; unfortunately written in DTML, but
you can also write it in Python:

http://www.zope.org/Members/4am/SiteAccess2/otheruse

So, it can be used to embed variables in the URL path itself instead of as the
argument; sometimes this can be clearer.

> I think maybe I'm running along here under the assumption that
> everyone more or less understands Model/View/Controller, and
> understands it in the same way that I do. Let me try to define my
> terms.

I more or less understand it, but not the same way you do. :)

[snip view description]
Right, this was clear to me. Presentation, reporting.

> The "Controller" element responds to user input and interfaces it
> with the business objects of the model. The controller is in
> some ways like glue code, but this is (or can be) where you might
> have form validation logic, or where you translate your form
> fields into the attributes of a data object. The controller
> element is typically the center of your application; it might be
> compared to the main() function in a C program. It presents
> various views when required, retrieves model objects, and glues
> them all together.

I can't think of a good translation of this to Zope, as you
said, in Zope control and view tend to be rather intertwined. This is 
not to say you can't use a pattern that does much like this, but it's
less cleanly separated.

[snip model]

This part was clear to me too.

I'm not sure why 'controller' wasn't clear to me, but it's always been
the vaguest part of MVC to me...

[snip]
>> Hm, yes, I think this makes sense (though it could lead to horrible
>> URLs). Of course you don't give the details here so I'm not
>> entirely sure I get the entire picture right.

> The URLs need not be any more "terrible" than in any other web
> application, including any Zope application. If your main servlet
> responds to all URLs that begin with '/myapp', any sub-path is
> something that the servlet can switch on. The difference here is
> that /myapp doesn't map to a folder object either on the
> filesystem or in some content repository.

Nor does it map to a method on some object, as what happens in Zope,
really. The content repository is just something behind that, but not
essential (just useful) to object publishing.
 
> It's mapped in the
> application server to mean "invoke this servlet"; additional path
> information is passed on to the servlet.

[snip]
> Meanwhile, your controller has the request before anything else.
> That means the controller can take action based on form values or
> other request parameters. As far as I know, Zope provides a
> solution for the most common application of this (user
> authentication), but does not provide much in the way of a
> toolkit for other similar problems.

> An example might be a look-and-feel switch -- if you have a
> cobranded application, you might serve from a different batch of
> pages, or use decorations and stylesheets from a different
> directory hierarchy. 

> You /can/ do these things in Zope, it just requires hacking
> around the way Zope is set up to serve pages, not invoke
> processing logic.

For many of those things (look and feel switch for instance) you
can use Acquisition. The alternative is to intercept the traversal
at some point to do something special. Access Rules are one way to do
that. Finally, if your URL end up at a Python script at some point 
in the path, the rest of the path coming after that can be processed by
it and you can do whatever you like, I believe.

[snip]

[speed of view templates]
> Or you could take your Zope objects and turn them into actual
> python code objects that generate HTML and perform some actions
> when called. It's not going to suddenly challenge C for
> near-hardware speeds, but it's a significant step ahead of
> doing the lexical parsing at runtime -- at least the Python
> interpreter is written in C.

Sure. In short, there are a ton of optimization opportunities here, if
necessary.

> I don't think it's a huge priority, but it's one of the things
> the Java stuff does to reduce runtime load, which in turn
> promotes better scaling and better overall peformance.

True, though page generation could in general probably be easily
distributed, so throwing more hardware at the problem is relatively
doable. It's harder to scale controllers and models, I expect.

>> > Your JSP pages have access to at least four levels of state
>> > (there might be another I'm forgetting): application, session,
>> > request, and page. The first three correspond to the same state
>> > repositories managed by the servlet. The 'page' context is for
>> > local objects.
>> 
>> Right, in Zope you'd have 'acquired things', session object (if you
>> have Core Session Tracking or something similar installed), REQUEST
>> object and folder context, for these. I'm just trying to translate
>> the concepts.

> Almost.

> I think the acquired things don't map onto application state
> quite like you suggest. There's one repository, shared by all the
> sessions running inside a single process. This isn't initialized
> with web-content properties. It's a place to stick objects at
> runtime.

Hm.. I'm not sure about an exact equivalent this in Zope.
Generally of course objects in Zope can always refer to runtime
objects, and this frequently happens (I initialize a bunch of 
widget and validator singleton objects in my Formulator component
for instance and any instance of a Formulator Form in the object
database can use these).

> I don't know what the session object gets you, or how it gets it
> for you, but I assume that's correct.

It's quite similar to REQUEST in that you can attach almost
arbitrary Python objects to it.

> The REQUEST object does map
> to the request context. The page context maps more closely to the
> namespace of a DTML document.

>> > You can forward control to other servlets or JSP pages, and you
>> > can also include pages.
>> 
>> What do you mean by 'forwarding control'? Control of what? 
>> 
>> [ snip my explanations ]
>> 
>> Ah, I think I more or less understand. I'm not sure what the use of 
>> forward is compared to include, though. Why'd you ever want to forward
>> control and never come back? Generate half a page with one JSP page,
>> and the other with another? 

> <dtml-return> and RESPONSE.redirect() provide more-or-less
> similar functionality in DTML, as compared with the use of
> <dtml-var> to include a method.

Ah, okay, I see. Though RESPONSE.redirect() does a client-side forward.

> My typical page processing paradigm is to submit form data to the
> page that renders the form. If all the data are acceptable and no
> processing errors arise, I forward on to another page. If all the
> data are not acceptable, I redisplay the current page with error
> messages. Others do it other ways.

Plug: this type of thing becomes pretty easy to build with Formulator. :)
 
>> > You can associate a bean with an object, and map form properties
>> > onto the bean's attributes. I think that this is most useful when
>> > you're programming pure JSP, but it's pretty useful when you're
>> > not.
>> 
>> Right, Zope's REQUEST object does this automatically (the 
>> 'form' subobject).

> Except that your "form" subobject is one standard object, not an
> object customized for your particular form. The servlet API
> allows you to pick the attributes off the REQUEST object, too;
> it's just somewhat more convenient to load up a bean.

Formulator allows you to customize whatever ends up in your REQUEST, or
can return a result dictionary. Of course that's my own component.
I think form handling is something that can be done better in Zope,
so I'm writing a component to do that.
 
[snip]
> The current solution is a reasonable solution, but it doesn't
> scale well. Your front-end load balancing hasta be session-aware,
> so it can redirect to the right instance of Zope.

Does it? It just needs to transmit the changes to the session object
database, right? That's similar to what it is already doing. Or do
you mean such a setup without ZEO?

> Using ZEO and Core Session Tracking could be The Right Thing. It
> certainly seems like the two technologies could be combined
> reasonably. Until it's arrived, though, you're still rolling your
> own sessions if you deploy on more than one box.

I think people already have setups running like that, though I'm not
sure.

>> > Servlets get a request and a response object passed to them.
>> 
>> Like Zope's DTML pages, and of course you can pass those along
>> to Python Scripts or other python code.

> Yes, but the servlets ARE Java code, which means that you don't
> NEED to stick a block of code at the top of every page, designed
> to invoke them.

True, it would be nice to have some way to do that more easily in
Zope. It shouldn't be that hard to whip up, but someone has to come
up with it and make it standard practice.

>> Neat. With Zope exporting subfolders or simply copying 
>> Data.fs is relatively
>> painless, but you need to copy over any external methods and special
>> Python products manually, which isn't hard but an extra step.

> Yeah, Zope isn't all that hard to roll out, as long as you do
> everything in Zope. You can still get sync problems with the
> external dependencies, though. If you use a lot of those (and I
> suspect that you will in a logic-intensive app), then you're
> likely going to hafta come up with some other deployment
> mechanism.

A benefit of Java's "We are the world" approach, I guess, which 
I usually dislike.

[snip]
> Well, except that we're not talking about just ZODB or ZEO here,
> we're talking about distributing dynamic (by which I mean
> run-time) state. That includes things like concurrent access
> issues and synchronization of access, RMI techniques, and
> whatever else it takes to implement these things.

You mean that the whole system can switch to another server in the
middle of handling a single request?

> It's also not just Session stuff. If you could publish arbitrary
> objects to the ZEO store and retrieve them by key -- something
> I'm sure is possible, but not sure how difficult -- you'd be
> close.

That should be pretty trivial from Python. I mean, just use a persistent
dictionary, or, to be more efficient, a BTree, stored in the ZODB.

>> Sort of like ZEO with Core Session Tracking. :) Of course this
>> is still being heavily developed, but it seems to be getting
>> near offering equivalent features.

> I can't speak to that, because I haven't used (and can't really,
> at the moment) Core Session Tracking, let alone using it with
> ZEO.

I haven't worked with ZEO yet either, so I can't make too many
claims about it. Core Session Tracking I've only used in its most
simple form. Eventually I hope to get to using both, though. Let's
call this discussion part of preparing for it. I learn about the
issues this way.

>> > The J2EE stuff makes deployment still one more step simpler, too.
>> > It pushes parameters that might change during runtime out one
>> > notch further, so that deployment configuration can be separated
>> > from the details of servlet instantiation, URI mappings, and
>> > other miscellany concerning the interaction of the app and the
>> > app server.
>> 
>> Hm, I'm not sure what this part means, exactly. I can't
>> translate it to any Zope terms, which is how I tend to think
>> about things, myself.

> I might argue that trying to put everything into Zope terms is
> part of what's limiting your comprehension. :) Zope is really a
> very different animal.

But does that mean this ability is important? I mean, if I can't
think of a Zope equivalent (x exists in Zope *or* x is needed in Zope
but we don't have it (yet))...but I can't think of the need here.

> Still, if you were to take your Zope application, and all its
> external methods, and all the little mini-servers that helped
> distribute your backend processing job, and stuck them all in a
> big tar file, then had one configuration file that detailled ALL
> the configuration details that would vary from machine to machine
> within an installation, and all the details that would vary from
> installation to installation -- if you assembled that tarball and
> that config file, you'd have the deployment mechanism used by the
> Enterprise stuff.

Okay.. there are vague sounds coming from Digital Creations that
they want to move the source code into a ZEO database as well, 
later on (offering tools to temporarily export them to the filesystem).
This sounds like what you're talking about.

>> Yes, I'm not trotting them out so much to do 'see, we have that
>> too!', though there is a component in that, but to learn more
>> about both by comparing the two, and to see if there are
>> interesting directions which Zope is missing. I think Java's
>> system works better for model/view/controller type situations
>> right now; Zope's framework for that is currently still in
>> design phases. Java also has clear (standardized) interfaces
>> for it, something which Zope needs a lot more of. 

> All of which I'll agree with. And I'm not trying to suggest that
> Zope is a poor product, either.

I realize that. It's just that Zope is my main experience with
web application programming, which is why I talk about it so
much here. :)

> I just don't feel like it's a
> good choice for a web-based application -- a website with some
> dynamic elements, maybe, and I could even see it being used for
> an online store.

> And I confess that the more we talk about these things, the more
> optomistic I am about the future. Java's solutions and toolkits
> are great (or at least pretty good), but I would like a good,
> solid alternative to all that wordy Java.

I agree Zope still needs to grow. The interfaces are too vague right now,
and that needs to become more solid. In my opinion that is the most important
thing that Zope needs to do and Java has already done. All the rest is
can be done by both with some creativity, perhaps excluding some
of the enterprise style features.

[snip]

[object/relational database mapping]
>> > Like I said once before,
>> > maybe we can beat Java to the punch here.
>> 
>> Python certainly has a lot of flexibility to work in its favor here,
>> though I don't know if we have the development resources in the
>> Python/Zope world to go fast on this. These discussions often seem to
>> devolve into feature wishlists and pie-in-the-sky designing.

> That's maybe true, but like I said, I started work on it at one
> point, and probably put a few solid days of effort into it. What
> I've arrived at might be less than perfect -- in fact, it IS less
> than perfect -- but it was a running start at something that
> would marshal a database table into an object.

> Even at that level of utility, it's a very, very useful tool.

Any chance you'll open source it? By the way, any useful documentation
on this entire issue that I could browse through?
  
>> [ snip lots, including a description of EOF's qualifiers ]
>> Would this get translated into a query into a relational database?

> Yes. EOF provides a couple different ways to do things -- you can
> build up these qualifiers piecewise, or you can write out the
> query string. But then EOF goes and traces it through its
> internal mapping of entities to database elements, and produces a
> SQL query that retrieves the appropriate object.

Very interesting.

[snip more very interesting stuff about this object to database mapping]  

> This is especially handy when you're building sequences of web
> pages several pages deep before the final commit. You can nest
> your editing contexts, and when the user hops back four pages,
> you can either make the hop with him and toss the deeper changes,
> or even just hop around with him and wait for him to commit the
> whole lot of them.

Is the whole set of changes somehow referred to by a session object?

Extremely interesting. Definitely need to look a lot deeper into this.
See if Python has something along these lines, and if not, see which
people are interested in building such a beastie.

[explicit interfaces]
> I think I agree with you here. I've worked around the interface
> issue in the past (subclass empty classes, then test to see if
> my instance is a subclass of a particular empty class), but it
> seems like being able to unambiguously promise to implement a
> certain interface is much /clearer/.

[zope UI can help beginners]
> I think Zope's web UI is a mixed blessing. It cuts down on the
> learning curve to get in and start making changes, but in the
> end, it becomes an impediment to being able to work smoothly -- a
> simple CVS archive of the DTML would be more practical later in
> the game.

Agreed that there are definitive negative tradeoff. Besides the
great need for explicit interfaces of some kind in Zope (well documented),
the secondmost important thing I see that Zope needs to add is the ability to
smoothly transition from filesystem representation of object database contents
such as DTML and Python code to ZODB object representation and back.

Explicit interfaces would make learning and extending Zope a lot
easier, and filesystem representation would suddenly enable a host
of filesystem based development tools, from editors to version
control systems; it would make the *process* of developing for Zope
much easier.

[snip]

Thanks for your very interesting post. I'm learning quite a bit here.

Regards,

Martijn
-- 
History of the 20th Century: WW1, WW2, WW3?
No, WWW -- Could we be going in the right direction?