[IPython-dev] DAG Dependencies

[Brian Granger]

On Thu, Oct 28, 2010 at 12:00 PM, Fernando Perez <fperez.net at gmail.com> wrote:
> On Thu, Oct 28, 2010 at 11:55 AM, MinRK <benjaminrk at gmail.com> wrote:
>> My interactive session starts running into memory issues with 100M ids, so
>> that's something to consider.
>
> Well, let's just hope that anyone with 100M ids in real life (not a
> synthetic benchmark) has the common sense of working on something with
> 100G of RAM or so :)

But, with the performance we are getting, some people might want to
simply run this 24/7 (instead of using a batch system).  Then, over a
long period of time, having that many tasks is not too crazy.  We
definitely need to think about this aspect of things.

> Thanks a lot for looking into this, Min.  Fantastic example to have,
> and one that is very relevant to many use cases.

Definitely,

Cheers,

Brian

> Cheers,
>
> f
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com