[Web-SIG] WSGI deployment config

[Phillip J. Eby]

At 07:33 PM 8/7/2005 +0100, James Gardner wrote:
>Phillip J. Eby wrote:
>
>>This approach is capable of doing everything you've proposed, except that 
>>it doesn't provide access to the private configuration data of individual 
>>services.  It would be possible, however, to load the service chain from 
>>a deployment file without instantiating applications or middleware, in 
>>order to e.g. run utility programs.
>>You can still include arbitrary configuration if you want, just by 
>>creating a service whose job is to provide such information.
>
>OK, fair point and I'm perfectly happy with this.
>
>>The point isn't to have a standardized format or globally accessible 
>>configuration, it's to *hide* configuration so that other objects don't 
>>have to know about it.
>
>What exactly are you defining as a service then?

A component that needs to be available to one or more other components, 
based on some lookup key (like a name or an interface).


>  A service would have to have some way of providing its useful code to 
> utilities etc as well as deploying middleware.

I think maybe you're confusing something here.  I'm suggesting that there 
be a chain of service providers, and that the WSGI API to load a pipeline 
should return both a top-down middleware-to-app chain, and a bottom-up 
service-to-service chain.  Thus, a utility program could load a WSGI file 
and gain access to the service chain, ignoring the middleware.

But, I'm not saying that services are *part* of the middleware chain; 
middleware components get created with access to the middleware chain, but 
the services themselves are not middleware.


>  In the original model each WSGI middleware component might rely on other 
> ones, both the middleware component and middleware it relies on might 
> need configuration. We can describe the whole middleware chain in a 
> config file so that it can all be configured at once. I might be missing 
> the point but is your idea of services that by passing them the service 
> chain they have an opportunity to decide what services to load based on 
> services they rely on and thereby bypass some of the configuration?

I don't understand you.  They just get what services they need from the 
chain.  They don't "bypass" configuration they never cared about in the 
first place.


>  Surely almost all middleware would need at least some configuration so 
> you are unlikely to make the config file much shorter?

But their configuration is in the parameters that get passed to their 
factory, e.g.

      [fooware from blah]
      something1 = "feh"
      # etc.


>>The only other piece I think we're missing is a way to handle branching, 
>>because our pipeline configuration is quite linear.
>>There's no obvious way to branch at the moment, except by having a way to 
>>configure a middleware component to refer to other pipelines.
>
>I don't think I've quite caught your full vision here. Using the services 
>idea my understanding is just that an application needs certain services 
>to function and also certain configuration for those services before it 
>can run, since many applications on the same site may need the same 
>services configured in the same way it is useful to be able to share 
>configuration and to do that

In which case, there should be a mechanism for configuring things based on 
other service lookups, e.g.

     [spazware from spiz]
     fidgety = lookup("fizzit.ping")

If we allowed 'lookup()' to mean, "search the service chain above me for a 
configuration service and return the value of 'fizzit.ping'.

My point here isn't to propose that this be the API, I'm just presenting a 
general concept.  "Wiring" of configuration by simply acquiring values from 
a global namespace doesn't work well even for applications developed 
entirely by a single person; it definitely doesn't scale to plug-and-play 
of components developed by an entire community.


>it is helpful for a local application to inherit configuration from 
>another source of components, possibly in the way I suggested. I don't 
>think branching really fits into that model so how are you envisaging 
>deployments?

The branching was for saying things like "/foo goes to pipeline A, and /bar 
goes to pipeline B".

It's becoming clear to me, though, that we need to *ban* the word 
"configuration" from this discussion, because it's way too overloaded, and 
everybody brings unique baggage to it.  If we don't use that word, we'll 
have to actually explain what we really mean.  :)

So, in that spirit, I will now rephrase my proposal so as not use the word 
"configuration".

A "pipeline spec" describes how to deploy a WSGI application, optionally 
with middleware filters and services, by providing parameters to designated 
factories.  There are three kinds of factories: application, middleware, 
and service.  All three kinds are invoked with the parameters defined in 
the spec and the most-recently specified service object.  Middleware 
factories also receive the *next* middleware or application component 
defined below them in the spec.

An example middleware factory signature:

      def make_middleware(last_service, application_to_wrap, **params):

Example application and service factory signatures:

      def make_app(last_service, **params):

      def make_service(last_service, **params):

Just as the middleware-to-application links form a "downward" chain of 
responsibility for handling WSGI requests, the service-to-service links 
form an "upward" chain of responsibility for acquiring service 
components.  There needs to be a specification for how to search the chain; 
for example we could have a 'get_service(key)' method required on service 
components, and if the service doesn't recognize the key it just calls 
'last_service.get_service()'.

In some circumstances, the same value or object is needed as a factory 
parameter for more than one component.  In these cases, it would be useful 
to be able to have a way to specify shared parameters in the 
specification.  Ordinarily, these shared parameters will be defined at some 
"high level" of the overall system, such as in a server-wide pipeline spec, 
and then acquired in "low level" pipeline specs for specific areas of the 
server or individual application components.  We can thus envision a 
"shared parameter service" interface for publishing values that need to be 
used often, and an API in the pipeline spec to indicate that a parameter 
should be retrieved from the nearest shared parameter service that offers a 
value for a given name.

This approach is superior to using a common namespace for parameters, 
because the level of abstraction at which shared parameters are defined is 
more likely to be concepts like "system administrator e-mail", but that 
value might then be used for more specific component parameters like "email 
errors to" and "administrator login ID".  So, being able to say that the 
"email_errors_to" parameter for a given component should be looked up from 
"sysadmin_email" in the shared parameter service allows for parameters to 
be cleanly shared between components.