Why isn't Python king of the hill?

[Martijn Faassen]

Geoffrey  Gerrietts <geoff at homegain.com> wrote:
>> Ah, a powerful query system, then. Difficult, too. Could you give
>> a real life example of an actual arbitrary retrieval system? It's
>> not full-text-search search, right? (the ZCatalog I believe already
>> can do something like that, on multiple attributes). 

> Arbitrary and full-text are more or less identical. I can't speak
> competently about ZCatalog. If it's got it, that's great -- but
> I'm not sure what it is exactly, or how effective or efficient it
> is to use that on a collection of large text. :)

I'm not entirely sure if it can do full-text on all attributes at
once  (though that's relatively easy to fix, though the order of
the result set may make not much sense), but it can do full-text
on a single attribute. I haven't got much experience with it either,
so I can't tell you much more. :)

>> > I can't speak to how easy it is to move from one app server to
>> > another. I /can/ speak for how much of the "hard work" is done
>> > for you by some of these enterprise containers -- a whole lot!
>> 
>> Could you go into some more detail on this? I've heard this 
>> more often,
>> but I'm still not entirely clear on *what* work is done. :)

> I think the meat of the response lies in answering this question.

Good, as it's my main question. :)

> I'm going to try to answer it first in regards to JSP and
> servlets, and I'm going to mix in some of the J2EE stuff near the
> end. I'm nowhere near as proficient with J2EE as I am with the
> other two.

Just a random note; I heard that WebWare for Python looks quite a bit
like "that Java stuff", at least in inspiration, so it may be interesting
for you to take a look at it.
 
> Meanwhile, others will I'm sure jump in and correct me where I'm
> wrong. Or I /hope/ they will, because I'm sure to be wrong. :)

> First, one framing quote:

>> I'm not interested in promoting a standard, I'm just interested in 
>> stealing ideas for Python-based APIs. I'm not interested in 
>> standardizing these APIs. :)

> That frames my first issue.

> I think that's a big part of what's great about these API's.
> They're standard!

[standardization can be good and helps you find people]

Agreed; makes sense. It's just that I personally am more interested in
the other stuff, not that fact that it's standard. I can't do much to
make something a standard anyway, so I'll just look at stealing the ideas.
The better the Python stuff gets, the more likely it is there are more
people that know about it, too, of course.

> -----
> REQUEST-CENTERED FRAMEWORK
> -----

> When I'm developing under Zope, I start with a page and work
> back from that. It's possible to determine from the request which
> bit of content to serve up -- you either make a conditional page,
> or you stick a bit of code at the top of the page that does
> redirection based on the contents of the request object.

Or you can do evil and cool stuff with AccessRules (the whole SiteRoot
stuff).

[snip]
> I don't want a page-centered application; in fact, I would go so
> far as to say that page centering breaks the MVC (model / view /
> controller) paradigm that I aspire toward in my software designs.
> The way Zope is set up, it seems to me that it encourages a
> design that embeds controller elements in the view.

What exactly do you mean by embedding controller elements in the
view? I'm not entirely sure what you'd refer to as the controller here.
 
[snip]
> When you add servlets to the mix, though, you start eking your
> way into deeper potential. Instead of requests being aimed at
> your view components, requests are aimed at a servlet. The
> servlet is invoked via routines that are designed to handle a
> request. The objects associated with the request are a little
> more "primitive" than the objects you pull out of Zope, but they
> are handled in a fairly elegant fashion.

> Having the servlet in charge of processing the request puts the
> control into the controller, if you will. It makes it easy for
> the app to center around the controller logic, without relying on
> the view logic to hand over the reins.

Hm, yes, I think this makes sense (though it could lead to horrible
URLs). Of course you don't give the details here so I'm not
entirely sure I get the entire picture right.

[Model/View/Controller as JSP/Servlets/Beans]

> -----
> JSP FEATURES
> -----

> Like I said, JSP is much like DTML or PHP in appearance. JSP has
> a few "directives" that you can use to apply general processing
> rules to your page. Of particular interest:

[JSP pages get compiled, not interpreted]

We're not going to accomplish this in Python, though of course you
could still write (possibly with C extensions) a fast interpreter for
web pages in Python. Anyway, we're generally not using Python because
insane speeds are the issue, so while this isn't a major bottleneck for
me I'm not going to bother too much with this.

> Your JSP pages have access to at least four levels of state
> (there might be another I'm forgetting): application, session,
> request, and page. The first three correspond to the same state
> repositories managed by the servlet. The 'page' context is for
> local objects.

Right, in Zope you'd have 'acquired things', session object (if you
have Core Session Tracking or something similar installed), REQUEST
object and folder context, for these. I'm just trying to translate
the concepts.

> You can forward control to other servlets or JSP pages, and you
> can also include pages.

What do you mean by 'forwarding control'? Control of what? 

> I believe (though I'm not sure I've
> tested this) that a page you forward to gets a fresh "page"
> context, while an include uses the current page context. Also,
> forward is terminal, while an include will return control after
> processing the child. Not too revolutionary here, but the
> features are present.

Ah, I think I more or less understand. I'm not sure what the use of 
forward is compared to include, though. Why'd you ever want to forward
control and never come back? Generate half a page with one JSP page,
and the other with another? 

> You can embed Java. In fact, that's the way you do your
> conditional display, and almost everything else interesting. This
> can look kinda goofy in practice, but it does increase ease of
> use.

> You can associate a bean with an object, and map form properties
> onto the bean's attributes. I think that this is most useful when
> you're programming pure JSP, but it's pretty useful when you're
> not.

Right, Zope's REQUEST object does this automatically (the 'form' subobject).

> You can create "custom tag libraries" for a more abstract and
> less "page-centered" approach to reusable page content. Custom
> tag libraries are more Java-centric than markup-centric.

> -----
> SERVLET FEATURES
> -----

> Most of what you get out of servlets is also available indirectly
> in JSP if you jump through the right hoops. For pieces of the
> application that don't need to display, though, there's not much
> sense in hopping through the hoops -- it can be downright
> annoying to need to. That's what makes servlets "natural" for the
> controller component.

Right, in Zope, you'd use Python Scripts (through the web),
External Methods (not through the web) for this. You may want to
bundle them into objects, and use ZClasses or program Python
Zope products. A little bit of Python tends to go a long way, of course,
and it helps to have high-level reusable components like ZSQL methods.
 
> I've discussed session management before. Sounds like they're
> doing some of that in newer Zope? That's good. It's a very hard
> thing to do well. I would be concerned about ZEO and ZODB being
> used in their unadulterated form to handle session management for
> a high traffic site. It just seems like there'd be too much I/O
> for ZODB/ZEO to handle it efficiently.

They're working on various improvements to that too, of course.
ZODB without undo for sessions for instance. ZODB layered on top of
various stuff, like the berkely database. By default Core Session Tracking
just keeps the sessions in RAM and they expire after a customizable time.

> Servlet containers implement session management for you. The
> servlet containers are presently not obliged to make these
> sessions either persistent or distributable, but some of them do,
> including at least one of the freely-available containers.
> Persistent, distributable sessions provide good load balancing
> and good failover.

Right, the intention in Zope is to do this with ZEO and Core Session
Tracking.

> Servlets get a request and a response object passed to them.

Like Zope's DTML pages, and of course you can pass those along
to Python Scripts or other python code.

> Generally you manipulate both. The request object has a list of
> parameters and also a list of attributes. The former are
> generally the form values while the latter is where you might
> store state to pass it between servlets or between a servlet and
> a JSP page.

Something similar is done with REQUEST. REQUEST.set() can be used
to store state for similar purposes. You can of course also directly
call methods and pass parameters that way. ZPublisher makes some of
that stuff work automatically.

> Servlets can dispatch a request using either a forward or an
> include, just like in JSP.

> The container's handling of a servlet (or entire web app, which
> is generally a bunch of servlets and lots of supporting pages) is
> controlled by the "deployment descriptor", a file called web.xml.
> This file has a standard, mandated format. It gives the developer
> a great deal of control over how his or her servlet will be
> instantiated, and how it will be managed once in memory.

> Once written, a web application can be packed up into a .war file
> (analogous to a .jar file) for easy deployment. The first time
> the app is accessed, the .war file is unpacked. That makes
> deployment remarkably (almost disgustingly) simple.

Neat. With Zope exporting subfolders or simply copying Data.fs is relatively
painless, but you need to copy over any external methods and special
Python products manually, which isn't hard but an extra step.

> -----
> ENTERPRISE JAVA BEANS
> -----

[discussion of what 'enterprise' could mean]

Enterprise is a typical buzzword. :) 'scalable' is a much better word
to use if you want it to use in large evironments, and 'portable' works
fine for diverse environments, I'd say. :)

'Has a lot of features for business use' fits the rest of the buzzword,
I guess.

> Adding EJB entails a few sacrifices and a few big gains. Sticking
> your servlets into the J2EE "framework" means that you hafta put
> your "application" level state into an EJB or a database (or at
> least knowing that anything you stick into the application
> storage can't be counted on to persist any longer than the
> request). You really need to make anything you put into the
> session area serializable (though there are a couple exceptions
> to that rule). There are a couple other rules that go along with
> that, I can't put fingers on them off the top of my head. All the
> sacrifices are really pretty minimal.

Okay, parts of this is a small problem with ZODB as well, of course.
I suspect ZODB makes this even easier, as it exploits Python's
flexibility.

Integration with relational databases is another issue; it'd
be nice if there was a nicer mapping to objects. I believe WebWare has
something like that. (MiddleKit?)

> The big gains: you get the distribution and persistence of
> sessions. That's a hard thing to do well. Your servlets load
> balance across all the servers that are running them. That's a
> pretty big win, too -- that's also pretty hard to do in a
> reliable fashion. And you get the EJB stuff, which means your
> backend model objects pretty much distribute themselves with no
> real additional design effort -- the binding mechanism is all
> there, the publication mechanism, the lookup mechanism. It's all
> there.

Sort of like ZEO with Core Session Tracking. :)
Of course this is still being heavily developed, but it seems to
be getting near offering equivalent features.

> The J2EE stuff makes deployment still one more step simpler, too.
> It pushes parameters that might change during runtime out one
> notch further, so that deployment configuration can be separated
> from the details of servlet instantiation, URI mappings, and
> other miscellany concerning the interaction of the app and the
> app server.

Hm, I'm not sure what this part means, exactly. I can't translate it to
any Zope terms, which is how I tend to think about things, myself.

[scaling from a small server to a large distributed system]

This is definitely neat. One of the things that makes ZEO great too;
it makes multiple distributed object database act like it's just one.
I haven't had to use it either, yet, though.

> -----
> JAVA DATA OBJECTS
> -----
> JDO is still in development, so it's maybe not fair of me to talk
> about it. But since a good proportion of the Python/Zope
> technologies that we're trotting out for comparison are still in
> development, I figure I might as well.

Yes, I'm not trotting them out so much to do 'see, we have that too!',
though there is a component in that, but to learn more about both
by comparing the two, and to see if there are interesting directions
which Zope is missing. I think Java's system works better for
model/view/controller type situations right now; Zope's framework for that
is currently still in design phases. Java also has clear (standardized)
interfaces for it, something which Zope needs a lot more of. 

I don't know how to compare ZEO with the scalability features you sketched
out. I would guess it's a question of tradeoffs; which is better probably
depends on your requirements and development style.

> Like I said once before,
> maybe we can beat Java to the punch here.

Python certainly has a lot of flexibility to work in its favor here,
though I don't know if we have the development resources in the
Python/Zope world to go fast on this. These discussions often seem to
devolve into feature wishlists and pie-in-the-sky designing.

That said, I think Digital Creations has got the message that integration
with relational databases is deemed important by Zope users, and
people are of course thinking about this. (in general, we don't seem to be
missing many desired features, though we may not have most of them implemented
yet. That is a good thing to know)

[snip JDO is like EOF]

> The EOF I've used, and the EOF I can comment on with a fair
> amount of accuracy <wink>. The way EOF works, in its most potent
> form, is thus: you develop a class with all its accessors and
> mutators, plus all the business logic you might want to include.
> You map some or all of its data fields to columns in your
> database in what's called an entity description. You include
> things like relationships to other entities -- does my Employee
> entity have a one-to-one relationship with a Boss entity? Model it.
> Does my Boss entity have a one-to-many relationship with Employee
> entities? Model it.

> Fetches can be performed by key, or by selector. Selectors can
> pick out individual objects based on the properties of the
> objects, or through elaborate path constructions that might trace
> through a long series of relationships. An example: you might
> fetch all Bosses who have an employee who has a spouse who has a
> birthday this month. The selector might look something like:
> employees.spouse.birthday.month == now.month

Would this get translated into a query into a relational database?
 
> When you fetch these objects from the database, the EOF puts
> "Faults" in the place of relationships. When you first try to
> read from a fault object, it runs off and fetches the object from
> the database.

A proxy pattern.

> But that's not all! EOF also provides sophisticated caching and
> data source controls. EOF also provides multiple 'editing
> contexts' with undo capabilities and the ability to nest editing
> contexts so that changes can be made within the context of other
> changes, and committed incrementally, or abandoned en masse.

Does this use relational database transaction features?
(with subtransactions?)

Note that the ZODB can already do transactions with subtransactions;
the interesting bits here seem to be in the relationship modelling and
querying, and the relational database integration. I believe ZPatterns
may be able to do some of this, but I've never looked into them in
detail.

> EOF is a powerful tool for management of object graphs and
> mapping the object graph onto the database. If even a fraction of
> the power EOF offers developers comes through in JDO, that will
> be a huge boon to anyone who works with a sophisticated dynamic
> data model.

Yes, sounds very interesting. I'll keep this in mind when I think about
relational database integration in Zope. It's certainly been a fairly hot
topic recently. Perhaps I should point some Zope folks to your post. :)

[reiteration that standard APIs are important]

*part* of the good effects you attribute to standard APIs can be
dealt with by good API *documentation* instead. Another part deals with
user interface; I do think Zope's web UI helps people who have little
programming experience tremendously; I've seen this happen several time.
A hot topic in the Zope world has to do with explicit interfaces,
and clear documentation for them. This inspired PEP 245, currently
under consideration for 2.2. While I don't agree with some of
the details of that PEP, I do think the ability to have a form
of explicit interfaces (run time checked if checked for at all!
ability to lie about implementing an interface! this is Python :)
is very important to the future health of Zope and similar large
Python systems.

http://python.sourceforge.net/peps/pep-0245.html

> I will note, in closing, that recent days have been something of
> a reawakening of faith in the darkened halls of my soul. Sunlight
> has begun to break through the clouds as I consider Python all
> over again. I have enjoyed this discussion a great deal, and
> enjoyed most of all that it has stirred my humility some -- I
> haven't seen all there is to see in the Python world, and much of
> what I haven't seen is quite promising.

> Thanks! :)

And thank you for your extensive description! It must have been a lot of
work writing it, but I certainly enjoyed reading about it. I have a much
clearer idea of what Java's web application architecture is all about
now, and how it maps to systems like Zope. For some time I've been
very interested in making Zope's component model work better by
having explicit interfaces and model/controller/view type abstractions. 
It's certainly been very interesting to hear how other systems have
approached these issues.

Regards,

Martijn
-- 
History of the 20th Century: WW1, WW2, WW3?
No, WWW -- Could we be going in the right direction?