[IPython-dev] should we make nodb the default hub backend?

[Min RK]

On Jun 11, 2012, at 22:47, Fernando Perez <fperez.net at gmail.com> wrote:

> On Mon, Jun 11, 2012 at 10:30 PM, MinRK <benjaminrk at gmail.com> wrote:
>> What do you mean by retrieved?  Under the normal circumstances you
>> described, exactly zero results are retrieved from the Hub, so this would
>> have no effect other than breaking delayed retrievals after the first.
>> 
>> Remember, the Hub is not involved *in any way* in the results of basic
>> execution/reply, and it can be removed entirely with no effect whatsoever on
>> simple executions.
> 
> Ah, I was making a mistaken assumption here; never mind then.
> 
>> Another option could be for DictDB to have a finite history, and just dump
>> old results if it has too many.
>> 
>> 
>> It's also widely varied - If each task sends/receives 10MB arrays, it only
>> taks a few hundred to be a problem.  If, on the other hand, it's mostly
>> simple commands without large data or side effects, it takes millions before
>> there is an issue.  This makes the above suggestion difficult, because for
>> some users a history of 100 is huge, while for others it's tiny.
> 
> Yes, that's really the problem: it's memory size, not length of the
> history.  And detecting memory pressure in Python isn't too easy, it
> would require separate codepaths for pickled results and numpy arrays,
> which appear small to most normal python introspection tools (which
> don't see their buffers).
> 
> I don't know if we want to fiddle with code of that nature this close
> to a release, or just go to nodb for now, and think about this issue
> with the calm of the 0.14 period.  I'm leaning towards that...

I agree, switching to nodb for 0.13 is fine, but I wouldn't try to get a clever self-limiting dictdb into this release.

The Parallel code, and Hub in particular, are due for a pretty big pass (two parts of which are already PRs), which I want to get into 0.14, but not this month.


> 
> f
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev