[IPython-dev] should we make nodb the default hub backend?
benjaminrk at gmail.com
Tue Jun 12 02:01:12 EDT 2012
On Jun 11, 2012, at 22:47, Fernando Perez <fperez.net at gmail.com> wrote:
> On Mon, Jun 11, 2012 at 10:30 PM, MinRK <benjaminrk at gmail.com> wrote:
>> What do you mean by retrieved? Under the normal circumstances you
>> described, exactly zero results are retrieved from the Hub, so this would
>> have no effect other than breaking delayed retrievals after the first.
>> Remember, the Hub is not involved *in any way* in the results of basic
>> execution/reply, and it can be removed entirely with no effect whatsoever on
>> simple executions.
> Ah, I was making a mistaken assumption here; never mind then.
>> Another option could be for DictDB to have a finite history, and just dump
>> old results if it has too many.
>> It's also widely varied - If each task sends/receives 10MB arrays, it only
>> taks a few hundred to be a problem. If, on the other hand, it's mostly
>> simple commands without large data or side effects, it takes millions before
>> there is an issue. This makes the above suggestion difficult, because for
>> some users a history of 100 is huge, while for others it's tiny.
> Yes, that's really the problem: it's memory size, not length of the
> history. And detecting memory pressure in Python isn't too easy, it
> would require separate codepaths for pickled results and numpy arrays,
> which appear small to most normal python introspection tools (which
> don't see their buffers).
> I don't know if we want to fiddle with code of that nature this close
> to a release, or just go to nodb for now, and think about this issue
> with the calm of the 0.14 period. I'm leaning towards that...
I agree, switching to nodb for 0.13 is fine, but I wouldn't try to get a clever self-limiting dictdb into this release.
The Parallel code, and Hub in particular, are due for a pretty big pass (two parts of which are already PRs), which I want to get into 0.14, but not this month.
> IPython-dev mailing list
> IPython-dev at scipy.org
More information about the IPython-dev