Running unmodified CGI scripts persistently under mod_wsgi.

[Graham Dumpleton]

On Nov 23, 8:49 am, Graham Dumpleton <Graham.Dumple... at gmail.com>
wrote:
> On Nov 23, 4:00 am, Istvan Albert <istvan.alb... at gmail.com> wrote:
>
>
>
> > On Nov 21, 12:15 am, Graham Dumpleton <Graham.Dumple... at gmail.com>
> > wrote:
>
> > > I would say that that is now debatable. Overall mod_wsgi is probably a
> > > better package in terms of what it has to offer. Only thing against
> > > mod_wsgi at this point is peoples willingness to accept something that
> > > is new in conjunction with Linux distributions and web hosting
> > > companies being slow to adopt new packages.
>
> > Yes that is to be expected, many people want someone else to pay the
> > early adopter's costs. Nonetheless mod_wsgi seems like the right
> > direction to move the python world.
>
> > One confounding factor that may slow its adoption could be the need of
> > running plain old CGI in an efficient way. I'm not sure how that fits
> > into the WSGI picture.
>
> Do note that using mod_wsgi doesn't preclude you still running plain
> old CGI scripts using mod_cgi or mod_cgid. As to making it more
> efficient, one can go two paths on this.
>
> The preferred path would be to put in the effort to convert the Python
> CGI application to WSGI. If one still needs to run it as CGI with
> other hosting solutions, then use a CGI-WSGI adapter.
>
> Second, albeit a bit of a kludge, just like mod_python.cgihandler is a
> kludge, is to emulate the CGI environment on top of WSGI. This would
> work if using single threaded Apache prefork MPM, or using mod_wsgi
> daemon mode with multiple processes but where each is single threaded.
> It wouldn't be practical though to do it for multithread Apache worker
> MPM, or multithreaded daemon processes with mod_wsgi daemon mode.
> Because of how Python leaks environment variables between sub
> interpreters, you would also only want to be running one sub
> interpreter within a process. This would be fine if using mod_wsgi
> daemon mode as different CGI scripts could be delegated to run in
> different daemon processes as necessary to keep them separate, but may
> not be practical if using embedded mode if hosting a range of other
> WSGI applications at the same time in embedded mode.
>
> So, it is doable, but effort would be better off expended at least
> converting the Python CGI script to WSGI instead. It would save a lot
> of trouble in the long run, especially with CGI scripts which weren't
> designed to be run multiple times in same memory context, ie., where
> they don't clean up after themselves.
>
> If someone really wanted to host an existing CGI script under WSGI
> where the same process was used over and over, and had good reasons
> for doing so, it wouldn't be hard for me to come up with a workable
> adapter that allowed it.

I had a play with this and got an initial solution working. It allows
mod_wsgi to be pointed at a directory of existing unmodified Python
CGI scripts and it will run them, maintaining the code for the CGI
script in memory between requests. One doesn't even have to rename
scripts to have a .py extension or follow Python module naming
conventions like mod_python required with its CGI handler.

Now, where as CGI script would run say at 10 requests per second for a
simple hello world program, using mod_wsgi to manage the CGI scripts
one could achieve 450+ requests per second with embedded mode of
mod_wsgi, and 250+ requests per second with daemon mode. Not as quick
as what can be achieved for an equivalent WSGI specific version of the
script, which runs at 900+ requests per second for embedded mode and
500+ requests for daemon mode, but still a lot better than a plain old
CGI script which requires a new process for each request.

One would still need to see how it works for more complicated Python
CGI scripts. Also may need to be tweaked to allow one to configure its
behaviour to suit the CGI script. For example, should additional
modules imported by the script be deleted out of sys.modules when done
or preserved for next request. Similarly, should the globals and/or
locals of the script be preserved between requests. Whether one would
need to adjust such behaviour would depend on the nature of the
specific CGI script.

The other question is whether there is even a demand for this. Do
people want to be able to take unmodified Python CGI scripts and try
to run them persistently in this way, or would they be better off
converting them to proper WSGI applications.

Graham