[Baypiggies] I need some help architecting the big picture
Shannon -jj Behrens
jjinux at gmail.com
Tue Apr 29 07:53:59 CEST 2008
On Mon, Apr 28, 2008 at 10:49 PM, Eric Walstad <eric at ericwalstad.com> wrote:
> Hey JJ,
> On Mon, Apr 28, 2008 at 9:57 PM, Shannon -jj Behrens <jjinux at gmail.com> wrote:
> > > Where in the process does this operator sit? Does he/she receive
> > > the batches from the customers and then feed them to your toolchain and
> > > verify that the batches made it to the database, or something else entirely?
> > That's the problem. I don't know. I've never been in a situation
> > where my user wasn't on the other side of a Web interface. For the
> > foreseeable future, the operator will be me or some sysadmin. My
> > guess is that he'll get the data via scp either manually or by cron
> > job. Now, I have to figure out how to feed the data to the system.
> > Do I simply put it in some place and say "go!"? I was guessing
> > someone else had been in a similar situation and had some best
> > practices to recommend.
> We wrote a python 'sentinel', started periodically by cron, that
> watches user's incoming scp directories for new files. We considered
> writing it as a daemon but the first iteration (running it from cron)
> turned out to suit our needs. When a new file arrives in the
> directory, the sentinel does something that it is configured to do:
> call a python callable, execute a system executable, etc. The
> sentinel only operates on a file that isn't already being operated on
> (in case the callable's runtime exceeds the sentinel's nap time). The
> sentinel can be configured to do a post-process task which usually
> includes moving the uploaded file to a 'processed' directory. The
> sentinel operates in a generic way by reading config files that define
> how it is supposed to behave. Each of our customers has a Sentinel
> config file describing how to tell when that customer's files arrive,
> how to process their file and what to do when the processing is done.
> I don't know about best practices here, but our system is pretty
> generic, flexible and works well for us.
Yep, that sounds a lot like my situation. Thanks!
I, for one, welcome our new Facebook overlords!
More information about the Baypiggies