[IPython-dev] DAG Dependencies
benjaminrk at gmail.com
Thu Oct 28 15:44:04 EDT 2010
On Thu, Oct 28, 2010 at 12:34, Brian Granger <ellisonbg at gmail.com> wrote:
> On Thu, Oct 28, 2010 at 12:00 PM, Fernando Perez <fperez.net at gmail.com>
> > On Thu, Oct 28, 2010 at 11:55 AM, MinRK <benjaminrk at gmail.com> wrote:
> >> My interactive session starts running into memory issues with 100M ids,
> >> that's something to consider.
> > Well, let's just hope that anyone with 100M ids in real life (not a
> > synthetic benchmark) has the common sense of working on something with
> > 100G of RAM or so :)
> But, with the performance we are getting, some people might want to
> simply run this 24/7 (instead of using a batch system). Then, over a
> long period of time, having that many tasks is not too crazy. We
> definitely need to think about this aspect of things.
Until I put in the DB backend, the memory footprint of the Controller after
100M tasks is going to be huge, so we can worry about that a little later.
Possibly add a sense of msg_ids going stale? Only keep the most recent 10M
> > Thanks a lot for looking into this, Min. Fantastic example to have,
> > and one that is very relevant to many use cases.
> > Cheers,
> > f
> Brian E. Granger, Ph.D.
> Assistant Professor of Physics
> Cal Poly State University, San Luis Obispo
> bgranger at calpoly.edu
> ellisonbg at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the IPython-dev