[Baypiggies] I need some help architecting the big picture

[Shannon -jj Behrens]

On Mon, Apr 28, 2008 at 10:49 PM, Eric Walstad <eric at ericwalstad.com> wrote:
> Hey JJ,
>
>  On Mon, Apr 28, 2008 at 9:57 PM, Shannon -jj Behrens <jjinux at gmail.com> wrote:
>  ...
>
> >  > Where in the process does this operator sit? Does he/she receive
>  >  > the batches from the customers and then feed them to your toolchain and
>  >  > verify that the batches made it to the database, or something else entirely?
>  >
>  >  That's the problem.  I don't know.  I've never been in a situation
>  >  where my user wasn't on the other side of a Web interface.  For the
>  >  foreseeable future, the operator will be me or some sysadmin.  My
>  >  guess is that he'll get the data via scp either manually or by cron
>  >  job.  Now, I have to figure out how to feed the data to the system.
>  >  Do I simply put it in some place and say "go!"?  I was guessing
>  >  someone else had been in a similar situation and had some best
>  >  practices to recommend.
>  We wrote a python 'sentinel', started periodically by cron, that
>  watches user's incoming scp directories for new files.  We considered
>  writing it as a daemon but the first iteration (running it from cron)
>  turned out to suit our needs.  When a new file arrives in the
>  directory, the sentinel does something that it is configured to do:
>  call a python callable, execute a system executable, etc.  The
>  sentinel only operates on a file that isn't already being operated on
>  (in case the callable's runtime exceeds the sentinel's nap time).  The
>  sentinel can be configured to do a post-process task which usually
>  includes moving the uploaded file to a 'processed' directory.  The
>  sentinel operates in a generic way by reading config files that define
>  how it is supposed to behave.  Each of our customers has a Sentinel
>  config file describing how to tell when that customer's files arrive,
>  how to process their file and what to do when the processing is done.
>  I don't know about best practices here, but our system is pretty
>  generic, flexible and works well for us.

Yep, that sounds a lot like my situation.  Thanks!

-jj

-- 
I, for one, welcome our new Facebook overlords!
http://jjinux.blogspot.com/