[IPython-dev] On sqlite history performance improvements...

[Thomas Kluyver]

On 9 April 2011 21:51, Fernando Perez <fperez.net at gmail.com> wrote:

> Hey Thomas,
>
> the history fixes of swapping the context manager and write loop do
> make a huge difference, so I've pushed your fixes already (it's the
> right algorithmic order, so no question on this change).  I rebased it
> to avoid a merge handle on a single commit.  Thanks a lot for this
> fix.
>

OK, glad it helped.


>
> But I think we still should look into further improving the
> performance/usability of the sqlite history support.  A few notes:
>
> - it's worth trying what happens with the writeout method in a thread.
>  If the python sqlite module releases the gil for the disk i/o
> operation (which I dearly hope it does), then it will be a net win and
> will completely solve the slight but annoying pause the system
> currently has on every prompt when using a laptop with a slow disk on
> battery.
>

I'll make a branch for you to test.


> - we also need to expose the cache parameter as a configurable, though
> this may need to wait for the config work to be completed, not sure
> right now if the history object is already accessible to the config
> file mechanisms.
>

It should already be exposed:
db_cache_size = Int(0, config=True)


> - and we certainly want to have a magic to set it at runtime without
> the get_ipython()....whatever=10 dance.  Name? %history_buffer,
> %history_cache, %history_delay, ...?
>

Perhaps there's scope for a more general magic to change those config
options which can be changed while we're running. Something like:

%ipconf history_db_cache 10
%ipconf history_db_log_output true

Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20110409/b923c799/attachment.html>