[IPython-dev] Fwd: [GitHub] SQLite history [ipython/ipython GH-282]

[Fernando Perez]

Hi folks,

Thomas just created this pull request for the sqlite-based history.
I'll try to have a look soon, but this is large and important enough
that more sets of eyes would be very, very welcome.

And many thanks to Thomas for implementing all this!

Best,

f


---------- Forwarded message ----------
From: takluyver <noreply at github.com>
Date: Sat, Feb 26, 2011 at 3:11 PM
Subject: [GitHub] SQLite history [ipython/ipython GH-282]
To: fperez.net at gmail.com


This was discussed previously on the mailing list. The code will
probably still need some refining, but I think it's working well
enough to put it forward.

There are two major changes involved:

History is now stored on disk as an SQLite database. This replaces
both the JSON history and the "shadow history" database. By default,
the input is stored as each line is entered, but (as Fernando
suggested), there is a configurable attribute, db_cache_size, which
lets you determine how often (in number of commands) it will write to
disk. Very large values will cause all input to be written when
IPython quits. A separate configurable attribute, db_log_output,
causes the plaintext repr of output to be written to a separate table
in the file.

Secondly, whereas history was previously treated as one long stream of
commands, it is now stored by "sessions", i.e. runs of IPython. So, if
you are particularly fond of the fifth command you gave IPython in one
session, the next time you start IPython, you can refer to the fifth
command of the last session (the syntax for magic commands is `~1/5`).
Sessions are consecutively numbered from when they start, so
interleaved commands from two shells open at the same time will be
kept separate.

This requires somewhat more complicated interfaces. There are now four
main interfaces to retrieve history from HistoryManager:

* `get_history`: Retrieves specified ranges of history, either from
the database, or the current session.
* `get_hist_from_rangestr`: Is a convenience method for magic
functions, which accepts the "~1/3-6  12" style range specifications.
* `get_hist_tail`: Gets the last n items from the database, ordered by
session and line. Used to populate readline (and the equivalent in the
Qt console).
* `get_hist_search`: Searches the database using glob style matching
(i.e. wildcards, not regexen).

All of these return iterators over 3-tuples, `(session, lineno,
item)`. If asked for output, `item` will be a 2-tuple of `(input,
output)`.

I've also given `%hist` the same range specification parsing as
`%save`, `%macro` and `%edit` (it would previously only accept one or
two numbers separated by a space). A single numeric argument, such as
`%hist 10` previously fetched the last 10 lines; now it gets the 10th
line of the current session. So I've added a new option, `-l` (for
last), which works with a single numeric argument. This is marginally
less convenient if that's what you want, but I think the consistency
among magic functions retrieving history is a good thing.

https://github.com/ipython/ipython/pull/282