[Web-SIG] more comments on Paste Deploy

[Jeff Shell]

On 3/7/07, Jim Fulton <jim at zope.com> wrote:
>
> On Mar 5, 2007, at 4:38 PM, Phillip J. Eby wrote:
> ...
> > Personally, I don't care for the Paste Deploy syntax -- frankly
> > it's almost barbaric.  :)
>
> I don't mean to pick on you, but I really *hate* comments like this.
> I don't like softer forms like "complicated" or even "makes me
> uneasy".  It would be far more helpful if you provides specific
> criticism.  I'd appreciate it if we would all just ignore statements
> like this and, preferably, stop making them.

I agree.

A problem I have is that I see these files with their syntax and I
balk. I don't think it's the syntax that's at issue as much as it is
that there's now a new set of terms that I don't understand. 'Entry
Point' is one that that shorted out my brain for a long time whenever
I'd try to look at the Paste docs to figure out what Paste was.

I think I hold Python to a different standard as I want to know what
something is doing. I don't think about this when I configure Apache.
I just know that very few of my Zope 3 terms map to Paste terms, and
all of this talk of 'filters' and 'entry points' and the like... I
look at it and go "huh, interesting." And then it's back to work on my
own thing.

...
> > A couple years back, I started writing a library to parse a more
> > sophisticated, Python-like syntax to do the same sorts of things,
> > but only got as far as the parser.
>
> A few years back, we created a library to parse more sophisticated
> apache-like syntax and I wish we hadn't.  The ini/config format is
> pretty standard and, IMO, really quite adequate.  I'm convinced that
> we don't really need another configuration format, at least not at
> this level.

While we're all talking about what we did or did not make, I found
that I wanted a lot more direct control than zc.buildout gave me.
After growing frustrated with writing Recipes and having to mentally
manage the glue between a config file that was like a make file (it
makes a lot of things) but not like a Rake file (no ability to include
my own programming logic within the buildout spec, only in recipes), I
took inspiration from Rake (a Ruby tool) and wrote a tool that looks
for `Rockfile`, which is basically a Python file (no .py extension so
as to avoid accidental imports).

I still don't *really* understand Eggs, nor how to get them to work
easily within individual Zope 3 instances. None of our existing Zope 3
libraries / apps are written as eggs or even as distutils-installable
packages. We just check our packages directly out of CVS, and
typically just check out other libraries from their repositories as
well. We dump them right in $INSTANCE_HOME/lib/python (a layout that I
actually like) and can then rest assured that a newly deployed app's
need/use of SQLAlchemy 0.3.4 doesn't interfere with an already running
app's need/use of SQLAlchemy 0.2.8.

A further benefit of having the Rockfile system is that they can be
used for other tasks done during development, such as updating
MochiKit, generating a special 'NoExport.js' file, and then packing a
few different combinations of MochiKit together.

    from rocketbuild.api import *
    from string import Template

    ns = namespace('mochikit')
    ROCKFILEPATH = globals().get('ROCKFILEPATH', path('.'))
    MOCHIKIT_LIB = ROCKFILEPATH/'libs'/'mochikit'
    MOCHIKIT_DL = ROCKFILEPATH/'mochikit_dl'
    MOCHIKIT_SRC = MOCHIKIT_DL/'MochiKit'
    SCRATCH = MOCHIKIT_LIB/'_scratch.js'
    CLEANUP = [MOCHIKIT_DL]

    NOEXPORT = Template("""\
    /*
     * Built for MochiKit SVN Checkout ${revision}
     */
    var MochiKit = { __export__: false };
    """)

    @ns.task('get')
    def getmochikit():
        if MOCHIKIT_DL.exists() and bool(MOCHIKIT_DL.listdir()):
            return
        svn = Subversion('http://svn.mochikit.com/mochikit')
        svn.co('trunk', target=MOCHIKIT_DL)

    @ns.task('clearmochilib')
    def clearmochilib():
        for jscript in MOCHIKIT_LIB.files('*.js'):
            jscript.remove()

    @ns.task('make-noexport')
    def makenoexport():
        info = Subversion().info(MOCHIKIT_DL)
        src = NOEXPORT.safe_substitute(**info)
        file(MOCHIKIT_LIB/'NoExport.js','w').write(src)

    @ns.task('build', ['get', 'clearmochilib', 'make-noexport'])
    def mochi_install():
        for source in MOCHIKIT_SRC.files('*.js'):
            log.info('copy %s -> %s' % (source, MOCHIKIT_LIB))
            source.copy(MOCHIKIT_LIB)

    @task('clear')
    def clear():
        for p in filter(path.exists, paths(*CLEANUP)):
            log.info('rmtree: %s', p.name)
            p.rmtree()
        if SCRATCH.exists():
            SCRATCH.remove()

I guess I'm just a control freak. It was too hard to control Buildout
to build out something that matches the way we've worked for years; it
was easier to write a tool from scratch. Which I think is the Python
way, for better or worse.

Anyways, this is the tool that we're starting to use at Bottlerocket
to automate our deployments as they grow more complex.

> ...
>
> > Anyway, all that aside, I think it would be fantastic if we could
> > come up with some "universal file format" for single-file
> > configuration and deployment of applications (including auto-
> > install of all needed eggs),

Configuration and deployment?

I'm trying to understand the scope of these terms (or this combined
term) better. I take it 'configuration' means just how an 'app' might
publish itself to a WSGI server. Is that right?

For us, deployment now is:

1. Make a Zope 3 instance home ('appserv1')
2. `cd appserv1/lib/python; cvs checkout customerapp`
3. `rockout -vv customerapp/Rockfile install` (installs dependencies, mostly
   by CVS / Subversion checkout, usually directly into `appserv1/lib/python`)
4. `cd ../../etc` (back to 'appserv1/etc')
5. choose a port number in zope.conf (the zope/twisted server config)
6. add two lines to Zope 3's `site.zcml` to set up our app:

    <include package="customerapp" file="site.zcml"/>
    <include package="customerapp" file="etc/deployment.zcml"/>

    The first line is a single file that sets up all of the dependencies and
    includes them in the proper order (probably only of interest, maybe, to
    other Zope 3 people). Basically this is my startup for my application
    within the Zope 3 application.

    The second line refers to configuration settings for machine local
    resources (database connections, cached resource directories, and so on).
    This may be written at deployment time. We keep it within the app so that
    it stays under source control, and also lets us know the names of services
    on which we may depend. This is also Zope 3 specific. I don't know of
    any way in which a configuration tool could be generic enough to handle
    any of this - even something as generic as a dburi string - unless it
    was restricted to handling ONLY basic values.

7. add site info to apache (rewriterule(s) /  proxy).

Is this analogous to the deployment and configuration being discussed?
Or is the desired outcome really one where I hand someone a tarball
and/or config file/script which would bring in (or have) ALL of the
Zope 3 framework along with my application and its dependents, ALL in
a way that doesn't trample on anyone/anything else (completely self
contained), and that someone can then add a line or two to the web
server's config file (if even that) and it all just runs? I guess Jim
may be the only one with the Zope 3 knowledge to answer this.

...
> > that could get stdlib support and ultimately hosting company
> > support.  This would actually give us a leg up on even PHP for ease-
> > of-deployment.
>
> Aside from the universal configuration file issue, I think this would
> be a terrific thing for us to focus on.  Something I hear a lot is
> how much easier PHP applications are to deploy to hosting providers.
> I would *love* it is Python had a similar story, even if only for
> smaller applications.
>
> I'd love to get some input who know a lot about what makes deploying
> PHP apps so easy.

I believe it's been said already that many PHP apps can just be
un-tarred/gzipped. Plus, PHP has the benefit of being basically built
in to Apache. Most hosting providers can enable PHP for individual
accounts in a snap. So in many cases, deploying a PHP app is seldom
any harder than deploying a static web site.

Granted, there are more advanced applications, and I don't know how
they get packaged or installed.

Perhaps PHP is an unfair case to look at: it's built in, and isn't
terribly complex. It's an easy processor directive. A Pylons,
Turbogears, or Zope 3 'app' isn't a bunch of .psp files that are
executed automatically by Apache. A more fair case to look at is Java
application deployment - maybe. I have no experience (yay!) with this.

I'm still a bit confused by the "write with any framework, deploy on
any server" line I've heard from the Servlet/J2EE world. I think I've
always considered it all to be one and the same, coming from my long
history with Zope, I've thought "if I program against Zope, I serve
from Zope."

But in theory, since Zope 3 has `zope.app.wsgi`, I could serve from...
anything? I guess that since I don't think about serving via Twisted
any more than I thought about serving via ZServer, I could put
CherryPy, mod_wsgi, whatever else underneath, right?

Sorry if that's a lot of questions. I'm still trying to grasp everything.

-- 
Jeff Shell