[IPython-dev] DAG Dependencies

Brian Granger ellisonbg at gmail.com
Thu Oct 28 15:34:45 EDT 2010

On Thu, Oct 28, 2010 at 12:00 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Thu, Oct 28, 2010 at 11:55 AM, MinRK <benjaminrk at gmail.com> wrote:
>> My interactive session starts running into memory issues with 100M ids, so
>> that's something to consider.
> Well, let's just hope that anyone with 100M ids in real life (not a
> synthetic benchmark) has the common sense of working on something with
> 100G of RAM or so :)

But, with the performance we are getting, some people might want to
simply run this 24/7 (instead of using a batch system).  Then, over a
long period of time, having that many tasks is not too crazy.  We
definitely need to think about this aspect of things.

> Thanks a lot for looking into this, Min.  Fantastic example to have,
> and one that is very relevant to many use cases.




> Cheers,
> f

Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com

More information about the IPython-dev mailing list