[Web-SIG] Web Site Process Bus

[Graham Dumpleton]

On 26/06/07, Robert Brewer <fumanchu at amor.org> wrote:
> I'd like to continue talking about standardization on site-wide process
> state and services.
>
> As described more fully on my blog [1], I'm proposing we create a new
> spec for a simple publish-subscribe Bus to manage site-wide state
> transitions (start, stop, etc.) and connect application components with
> site-wide services (daemonize, autoreload, site logging, etc).

I'm a bit too busy to fully absorb all that is being said on this
topic, but have a few comments or observations to make based on my
experiences. If they relate to anything, it is possibly just to expand
on some of the issues around why a bus might be a reasonable
requirement to have so hopefully it is of use in that respect.

First comment is about WSGI applications somehow themselves using
SIGTERM etc as triggers for things they want to do. For Apache at
least, allowing any part of a hosted Python application to register
its own signal handlers is a big no no. This is because Apache itself
uses a whole range of signals to manage such tasks as shutting down
sub processes or signaling worker and/or listener threads within a
process that its time to wakeup or shutdown. If a WSGI application
starts registering signal handlers it can as a result stop Apache from
even being able to process requests. In mod_wsgi I have had to
specifically take steps to prevent applications breaking things in
this way by replacing signal.signal() on creation of an interpreter.
Instead I log a warning that the signal registration has been ignored
and otherwise do nothing. This was simply the safest thing to do.

Thus I believe a clear statement should be made that UNIX signals are
off limits to WSGI applications or components. I have no problem with
the underlying server upon which the WSGI adapter sits using them for
its own purposes, however, knowledge of signals should stay in that
layer. Now it may be the case that a higher level application signal
may be generated as a result of a SIGTERM in a specific server
implementation, for example shutdown, but the name or type of signal
does have to be distinct, ie., "shutdown" and have no linkage to the
UNIX signal.

In Apache for example, SIGTERM isn't used in child processes to
indicate process shutdown, that is instead signalled  via what is
called the pipe of death. The trigger for process shutdown in this
case is actually the destruction of the child process memory pool.

I know that Robert's diagram shows the translation of UNIX signal at a
lower level to higher level concept, but then elsewhere there was
mention of SIGUSR signals. Apache even uses these in some
configurations so a hosted user application wouldn't even be able to
use them.

Anyway, just wanted to make it absolutely clear that I don't believe a
hosted WSGI application and associated framework has any business
taking direct interest in low level UNIX signals.

Next issue was SystemExit exception, raised explicitly or via
sys.exit(), and the ability of a WSGI application to itself shutdown
the server. Again, shutting down the server using SystemExit is
something I see as being off limits to a WSGI application. In mod_wsgi
I explicitly ignore SystemExit raised by a WSGI application when
handling a request, more or less treating it like any other unexpected
exception. This is necessary because within Apache one can't  tolerate
one application making a decision to shutdown a process because of the
impacts it may have on other application code, possibly non Python
code, running in the same process.

For a Python based web server, also believe that SystemExit in a
request handler should be ignored. Obviously though the underlying
server itself is free to use its for its own purpose. I actually don't
know what other WSGI adapters do about SystemExit, so would be
interested to learn what others do.

One thing I have sometimes thought about though is whether there is
any valid case where a WSGI application should itself be allowed to
request the application be shutdown. I think I'd probably have to say
no for the general case. But then you have the case of trying to
reload an application when the code changes. In this case it may be
reasonable because no in process module reloading scheme is ever going
to be good enough. The closest one can get doing it in process is to
destroy the whole sub interpreter. This though is only a practical
thing for something like mod_wsgi and even then doesn't always work
because of C extension modules, eg PyProtocols, that don't take kindly
to sub interpreters being destroyed and the module then being reloaded
into a new sub interpreter.

Thus, purely to make administration easier for the WSGI application to
request an application reload, perhaps triggered by user through a web
page, one might want to signal process shutdown. Even then though it
isn't that simple as that, as there has to be a compatible supervisor
system in place which will restart the process. In mod_wsgi if using
it in embedded mode whereby application runs in Apache child
processes, an application requesting a reload like this is not viable
as not possible to tell the Apache parent process to restart
everything.

Even when using daemon mode of mod_wsgi where application runs in
separate processes, only the simple case where there is one daemon
process for that application is it trivial to initiate a reload. When
there are multiple processes in the group, need to somehow signal the
other processes in the group to also reload.

So that is my brain dump. I guess I'll see where you guys take the
discussion. At the moment I am guarded as to whether some of the ideas
of how this bus could be used would in practice be transferable.
Certainly with Apache it may be quite challenging because of the
multiprocess nature of the server and the fact that child processes
don't really have any way of controlling the supervisor (parent Apache
process).

Graham