[Web-SIG] Standardized configuration

[Chris McDonough]

I tried to think of this today in terms of creating a "deployment spec"
but boy, it gets complicated if you want a lot of useful features out of
it.  I have about four or five pages of a straw man "deployment
configuration" proposal, but it makes way too many assumptions.

So I tried to boil the problem down into its parts.  There seem to be
three distinct categories of configuration:

- Server/gateway/application instance configuration.  This is the
  kind of configuration that may be exposed to deployers by
  application authors.  Creating an instance configuration results
  in an instance of an application or gateway or maybe even
  a server.

- "Wiring" configuration which allows you to string together a
  "stack" out of instances.   I like calling it a "pipeline" better,
  but when in Rome... This is the kind of configuration that
  would be useful if you already have a bunch of instance configurations
  from the step above laying around and you want to create a stack
  out of them for deployment purposes.

- "Service" configuration which allows you create bits of 
  context that can be used by applications in the stack, but which
  aren't inserted into the stack itself.

I suspect we should stick to the first category of configuration first,
but I'll note that the desire for the other two categories might impose
some design constraints on the first.  The last kind of configuration
definitely ventures far out into framework land and though it'd be
terribly useful and seems to be where a lot of people think the value of
WSGI is, it might be something other than WSGI entirely.

So, anyway, towards the first category, I'll throw something out to the
wolves.  Note that below when I say "component" I mean a WSGI server,
gateway, or application:

  Each Python package which includes one or more WSGI components may
  optionally include descriptions of these components'
  "meta-configuration".  This meta-configuration would take the form
  of one or more "schemas".  Each schema would enumerate the
  configurable elements of a single WSGI component implementation.
  A schema for a component defines *the minimal number* of typed,
  component-specific keys and values that may be used to create
  instances of this component.

  >>> # load the schemas
  >>> server_schema  = loadSchema('components/server/server.schema')
  >>> gateway_schema = loadSchema('components/gateway/gateway.schema')
  >>> app_schema     = loadSchema('components/app/app.schema')

  >>> # create the instances; any one of these steps would fail
  >>> # if the config file violated its schema.
  >>> server_factory  = loadConfig('instances/server/server.conf',
                                  schema = server_schema)
  >>> gateway_factory = loadConfig('instances/gateway/gateway.conf',
                                  schema = gateway_schema)
  >>> app_factory     = loadConfig('instances/app/app.conf',
                                    schema = app_schema)

  >>> # create instances from the factories
  >>> server = server_factory.create()
  >>> gateway = gateway_factory.create()
  >>> app = app_factory.create()

  # configure the instances into a pipeline
  >>> pipeline = server(gateway(app))

  # serve up the pipeline (notionally)
  >>> server.serve()

Of course this is just a more declarative way to do what is already
possible in code except for the schema-checking part, which presumably
would supply the deployer with clues if he had screwed up a config file.

I purposely didn't attempt to describe the syntax of the configuration
or schema files, but I suspect it would be best to make them both
ConfigParser files.  FWIW, ZConfig already does this exact thing, and
it's already written, but introducing dependencies on non-stdlib things
seems problematic.

Is this more or less what people have in mind for deployment
configuration or am I out in left field?

On Sun, 2005-07-17 at 13:56 -0400, Phillip J. Eby wrote:
> At 07:29 AM 7/17/2005 -0400, Chris McDonough wrote:
> >I'm a bit confused because one of the canonical examples of
> >how WSGI middleware is useful seems to be the example of implementing a
> >framework-agnostic sessioning service.  And for that sessioning service
> >to be useful, your application has to be able to depend on its
> >availability so it can't be "oblivious".
> 
> Exactly.  As soon as you start trying to have configured services, you are 
> creating Yet Another Framework.  Which isn't a bad thing per se, except 
> that it falls outside the scope of  PEP 333.  It deserves a separate PEP, I 
> think, and a separate implementation mechanism than being crammed into the 
> request environment.  These things should be allowed to be static, so that 
> an application can do some reasonable setup, and so that you don't have 
> per-request overhead to shove ninety services into the environment.
> 
> Also, because we are dealing not with basic plumbing but with making a nice 
> kitchen, it seems to me we can afford to make the fixtures nice.  That is, 
> for an add-on specification to WSGI we don't need to adhere to the "let it 
> be ugly for apps if it makes the server easier" principle that guided PEP 
> 333.  The assumption there was that people would mostly port existing 
> wrappers over HTTP/CGI to be wrappers over WSGI.  But for services, we are 
> talking about an actual framework to be used by application developers 
> directly, so more user-friendliness is definitely in order.
> 
> For WSGI itself, the server-side implementation has to be very server 
> specific.  But the bulk of a service stack could be implemented once (e.g. 
> as part of wsgiref), and then just used by servers.  So, we don't have to 
> worry as much about making it easy for server people to implement, except 
> for any server-specific choices about how configuration might be 
> stacked.  (For example, in a filesystem-oriented server like Apache, you 
> might want subdirectories to inherit services defined in parent directories.)
> 
> 
> >OTOH, the primary benefit -- to me, at least -- of modeling services as
> >WSGI middleware is the fact that someone else might be able to use my
> >service outside the scope of my projects (and thus help maintain it and
> >find bugs, etc).  So if I've got the wrong concept of what kinds of
> >middleware that I can expect "normal" people to use, I don't want to go
> >very far down that road without listening carefully to Phillip.  Perhaps
> >I'll have a shot at influencing the direction of WSGI to make it more
> >appropriate for this sort of thing or maybe we'll come up with a better
> >way of doing it.
> >
> >Zope 3 is a component system much like what I'm after, and I may just
> >end up using it wholesale.  But my immediate problem with Zope 3 is that
> >like Zope 2, it's a collection of libraries that have dependencies on
> >other libraries that are only included within its own checkout and don't
> >yet have much of a life of their own.  It's not really a technical
> >problem, it's a social one... I'd rather have a somewhat messy framework
> >with a lot of diversity composed of wildly differing component
> >implementations that have a life of their own than to be be trapped in a
> >clean, pure world where all the components are used only within that
> >world.
> >
> >I suspect there's a middle ground here somewhere.
> 
> Right; I'm suggesting that we grow a "WSGI Deployment" or "WSGI Stack" 
> specification that includes a simple way to obtain services (using the Zope 
> 3 definition of "service" as simply a named component).  This would form 
> the basis for various "WSGI Service" specifications.  And, for existing 
> frameworks there's at least some potential possibility of integrating with 
> this stack, since PEAK and Zope 3 both already have ways to define and 
> acquire named services, so it might be possible to define the spec in such 
> a way that their implementations could be reused by wrapping them in a thin 
> "WSGI Stack" adapter.  Similarly, if there are any other frameworks out 
> there that offer similar functionality, then they ought to be able to play 
> too, at least in principle.
>