Threads CGI & database connections...

[Ian Bicking]

On Tuesday, October 7, 2003, at 10:23 AM, Chris Lyon wrote:
> We run a sound effects library site.
>
> I use a cgi script written in python to copy files from our file
> repository to a publically viewable directory for clients to pick up
> with locations sent to them in E-mails and on a completion web page.
> Unfortunately since some of the files are fairly large (20-30 Mbytes
> is not untypical) it can take a considerable length of time to copy
> the files across.
> So we would like to implement something that would allow us to quickly
> report back that the process has been started and then copy the files
> across whilst a small piece of javascript will refresh the viewing
> page to indicate the progress.
>
> The status page extracts it's information from the database where a
> progress column is populated ( 1 to indicate copy started and 2 to
> indicate copy completed).
>
> Now this seems to mean threads( it's all hosted on windows2000 ) an
> area I have little ( read none) experience of. I believe I will have
> to maintain a database cursor within the active thread to indicate the
> progress to the database.

CGI scripts are run in a single process, so threads won't work for you. 
  The response hasn't ended until the process quits.  So since you want 
to send multiple responses (each time the page is reloaded, to give the 
progress as another response) you will have multiple processes for each 
copy.

Which is to say, when you first run the CGI script that starts the 
copying process, that CGI script has to terminate before the copying is 
finished.  You will have to somehow spawn another process to actually 
do the copying.  But you have that Windows problem, since Windows 
doesn't have fork.  There may be some equivalent way to do this, I 
don't know -- someone who knows Windows better may be able to answer.

Another alternative to spawning a process is to have some long-running 
process waiting around to copy files for people.  If you are using a 
database you could, for instance, have field that means 
please-copy-this-file.  This long-running process polls the database, 
and when it finds a row with that field set it starts copying (setting 
that field to a new value, copying-in-progress).  This is what I'd 
recommend, as it is pretty straight-forward and translates to different 
environments fairly easily.  This polling process could use threads or 
not, depending on whether you want to copy files in parallel or 
serially.  (The polling process won't be hooked into the web server in 
any way, it'll be a separate program you run on the server.)

--
Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org