[Web-SIG] A 'shutdown' function in WSGI

[Graham Dumpleton]

On 21 February 2012 21:41, Tarek Ziadé <ziade.tarek at gmail.com> wrote:
>
>
> On Tue, Feb 21, 2012 at 10:24 AM, Graham Dumpleton
> <graham.dumpleton at gmail.com> wrote:
>>
>> ...
>>
>> > But I don't think you can guarantee that everything is still up in
>> > memory by
>> > the time atexit gets called,
>> > so you can't really call cleanup code there.
>>
>> The only thing which is done prior to atexit callbacks being called is
>> waiting on threads which weren't marked as daemonised.
>
> which can lead to completely lock the shutdown if a lib or the program has a
> thread with a loop that waits for a condition.

In mod_wsgi at least there are fail safes such that background C
threads will force kill the process if such a lockup occurs on
shutdown.

> which it is not the case with signals, since you get a chance to properly
> stop everything beforehand.

Yes and no. For a signal handler to even be able to be triggered,
there must be Python code executing in the main thread that originally
created the main interpreter.

In an embedded system such as mod_wsgi, the main thread is never used
to handle requests and actually runs in C code blocked waiting for an
internal notification that process is being shutdown.

>> what do you mean by bypassing its destruction ?
>
>> Non catchable signal from within process or from a distinct monitoring
>> process.
>> One of this things I pointed out is being missed.
>> That is, a WSGI adapter may be running on top of another layer of
>> abstraction, such as FASTCGI for example, where the lower layer isn't
>> going to have any callback mechanism of its own to even notify the
>> WSGI layer to trigger registered cleanup callbacks.
>> This is why the only mechanism one can universally rely on is the
>> Python interpreters own atexit mechanism.
>
> I see.. but what I don't understand is the following: when the whole stack
> is shut down, the python process is being killed by *someone*.
>
> And that someone, as far as I understand, is also able to send requests to
> the WSGI application.
>
> So what makes it impossible to send a shutdown signal prior to killing the
> process ?

Is not impossible and in mod_wsgi at least a signal is used to
initiate shutdown, this coming either from itself in some cases, or
from Apache parent process in others. The signal handler then uses a
socketpair pipe to wake up the blocked main thread to begin shutdown
steps. Either way it is handled at C code level because can't rely on
Python level signal handlers to actually run.

To further complicate things, in a process with multiple sub
interpreters where would the Python signal handler even run. There is
no main thread running waiting to exit. It also can't just cause the
main Python interpreter to be exited. Simply exiting a main thread
even if it did exist wouldn't allow you to cleanup sub interpreters.

In short, embedded systems are going to be quite different to what you
are used to with pure WSGI servers. It is because it is doing all this
to ensure that reliable shutdown can occur that mod_wsgi ignores all
Python signal handler registrations by default.

That all said, technically mod_wsgi could on reception of its signal,
and if there was a registry of known WSGI applications, it could tell
them the process is being shutdown. Presence of sub interpreters makes
that a lot of fun, but would be doable.

Right now without such a registry of applications with enter/exit
methods as being discussed in this thread, the only way in mod_wsgi is
to rely on atexit.

Graham