[IPython-dev] should we make nodb the default hub backend?

MinRK benjaminrk at gmail.com
Tue Jun 12 01:30:17 EDT 2012


On Mon, Jun 11, 2012 at 10:01 PM, Fernando Perez <fperez.net at gmail.com>wrote:

> On Mon, Jun 11, 2012 at 9:54 PM, MinRK <benjaminrk at gmail.com> wrote:
> > I'm 50-50.  I generally think of NoDB as an optimization, which would
> > suggest that *it* should be the optional case for people in the know,
> rather
> > than the other way around.
>
>
> I guess what we're finding is that it can cause a pretty nasty failure
> mode with fairly naive and natural use patterns.


Yes, there is cost to storing large amounts of data.


> Alternatively, would
> it be possible for the dictdb to (at least by default) drop every
> result that gets retrieved?

 That would keep the ability to do delayed
> retrieval while perhaps lowering the memory pressure under common use
> patterns.
>

What do you mean by retrieved?  Under the normal circumstances you
described, exactly zero results are retrieved from the Hub, so this would
have no effect other than breaking delayed retrievals after the first.

Remember, the Hub is not involved *in any way* in the results of basic
execution/reply, and it can be removed entirely with no effect whatsoever
on simple executions.


>
> > I don't view delayed result retrieval as "advanced", but I think has
> turned
> > out to be rare, so I would be okay with this.  The one thing it changes
> is
> > adding big warnings to the docs, because a huge swath of features would
> be
> > totally unavailable by default.
>
> I'm not saying I'm too happy with the idea, but my experience today
> has made me lean towards it more, thinking of 'regular' users...
>

I know, it makes sense.

Another option could be for DictDB to have a finite history, and just dump
old results if it has too many.


> It's also widely varied - If each task sends/receives 10MB arrays, it only
taks a few hundred to be a problem.  If, on the other hand, it's mostly
simple commands without large data or side effects, it takes millions
before there is an issue.  This makes the above suggestion difficult,
because for some users a history of 100 is huge, while for others it's tiny.


> f
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20120611/72280bd7/attachment.html>


More information about the IPython-dev mailing list