Best way to spawn process on back end computer

[Michael Sparks]

sophie_newbie wrote:

> Hi,
> 
> I'm running a python cgi script on a frontend web server and I want it
> to spawn another script (that takes a long time to run) on a backend
> number crunching server thats connected to the same network. What do
> you think is the best way to do this? I have a few ideas but I'm sure
> there is a "best" way to go about this.

Best is always subjective - what sort of interaction are you after?

Web request to trigger an activity (fire and forget)

Web request to trigger an activity, which produces output suitable for
something else?

Web request to trigger an activity, which produces output after a while that
you want to send back over a later web request?

Web request to trigger an activity, which produces output after a while that
you want to send back over a later web request, and you want to easily tie
the two things together?

Web request to trigger an activity, which produces output after a while that
you want to send back over a later web request, and you want those two
events to look like part of one operation?

Picking a concrete example...

Or something a bit more like:
   * User uploads an image
   * Gets converted to a bunch of standard sizes, and placed into a queue
     for moderation and then later use?

Or perhaps:
   * User uploads a video
   * That gets transcoded to a different video format ? (eg video
     contribution transcoded to flash video?)

In the latter two cases, this is something I've needed to do, so I created a
simple WSGI filter that dumps the uploaded images/videos into a standard
file location, and then had a script that watches that standard location
for new images & videos, and does those conversions and transcodes.

Strictly speaking to be "clean" about it, you need to dump the file to a
temporary location whilst you're grabbing it from the network and when it's
been written to disk rename it.

I've no idea if this maps to your problem at all. (since I can think of a
few scenarios that match your description).

If it does match, then this code may be handy:
http://code.google.com/p/kamaelia/source/browse/branches/private_MPS_Participate/Apps/FileProcessor/App/BatchFileProcessor.py

If you want to use that code, let me know and I'll package it up. There's an
older version here:
   * http://www.kamaelia.org/release/Kamaelia-FileProcessor-0.1.0.tar.gz

Regards,


Michael.
--
http://www.kamaelia.org/Home