[Baypiggies] I need some help architecting the big picture
Shannon -jj Behrens
jjinux at gmail.com
Tue Apr 29 07:04:12 CEST 2008
On Mon, Apr 28, 2008 at 3:02 PM, Michael Pittaro <mikeyp at snaplogic.org> wrote:
> Shannon -jj Behrens wrote:
> > Hi,
> > I need some help architecting the big picture on my current project.
> > I'm usually a Web guy, which I understand very well. However, my
> > current project is more batch oriented. Here are the details:
> > Anyway, so I have all this customer-specifc logic, and all these data
> > pipelines. How do I pull it together into something an operator would
> > want to use? Is the idea of an operator appropriate? I'm pretty sure
> > this is an "operations" problem.
> This sounds like a typical data integration project, which is what
> SnapLogic is designed for.
> Depending on what sort of operations you're doing on the data, we might
> already have the right components or you might need to create a new one, but
> from your description, it sounds like we have most of the database and file
> IO you need.
> You're right about the operations aspect - you probably want an 'operator'
> interface that that will just run everything when the right files are in
> place, either scheduled or triggered by some event.
> Drop me a line if you need more details.
Ah, I haven't forgotten about SnapLogic. I'm not sure I completely
know the definition of a data integration project. I do know I have
data going in. The key thing that I'm doing is apply clever
algorithms to learn stuff from the data. Then I can use the stuff I
learned to answer queries via a Web interface.
I, for one, welcome our new Facebook overlords!
More information about the Baypiggies