[IPython-dev] DAG Dependencies

MinRK benjaminrk at gmail.com
Thu Oct 28 15:44:04 EDT 2010


On Thu, Oct 28, 2010 at 12:34, Brian Granger <ellisonbg at gmail.com> wrote:

> On Thu, Oct 28, 2010 at 12:00 PM, Fernando Perez <fperez.net at gmail.com>
> wrote:
> > On Thu, Oct 28, 2010 at 11:55 AM, MinRK <benjaminrk at gmail.com> wrote:
> >> My interactive session starts running into memory issues with 100M ids,
> so
> >> that's something to consider.
> >
> > Well, let's just hope that anyone with 100M ids in real life (not a
> > synthetic benchmark) has the common sense of working on something with
> > 100G of RAM or so :)
>
> But, with the performance we are getting, some people might want to
> simply run this 24/7 (instead of using a batch system).  Then, over a
> long period of time, having that many tasks is not too crazy.  We
> definitely need to think about this aspect of things.
>

Until I put in the DB backend, the memory footprint of the Controller after
100M tasks is going to be huge, so we can worry about that a little later.
 Possibly add a sense of msg_ids going stale?  Only keep the most recent 10M
for checking?

-MinRK


>
> > Thanks a lot for looking into this, Min.  Fantastic example to have,
> > and one that is very relevant to many use cases.
>
> Definitely,
>
> Cheers,
>
> Brian
>
> > Cheers,
> >
> > f
> >
>
>
>
> --
> Brian E. Granger, Ph.D.
> Assistant Professor of Physics
> Cal Poly State University, San Luis Obispo
> bgranger at calpoly.edu
> ellisonbg at gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20101028/33b74df6/attachment.html>


More information about the IPython-dev mailing list