[IPython-dev] History
Thomas Kluyver
takowl at gmail.com
Wed Feb 16 17:10:27 EST 2011
I've been having a think about how we store history:
At present: Commands entered are stored in two lists (raw and "translated" -
i.e. turning magic commands into function calls). These are persisted to
disk at the next input every time 60 seconds elapse, with storage in a JSON
file, which is reloaded into the same lists (and into readline history) when
starting IPython. Each command entered is also immediately persisted in the
"shadow history", a collection of files in .ipython managed by the
pickleshare DB. The output objects are also stored in a dictionary by prompt
number, but I'm less concerned with that here.
Uses:
- Readline history (getting previous commands via up arrow)
- Various magic commands (save, macro, hist) can access ranges of input,
using the prompt numbers from the current session.
- %hist -g allows searching shadow and current history with glob syntax.
- %rep can access ranges of this session's history, or single lines from
shadow history.
(These are all I've found so far - please let me know if there are others)
I'd like to propose simplifying this somewhat. Specifically, I think it
makes sense to store history on disk just once. I suggest an SQLite
database. SQLite has been in the standard library since 2.5, is lightweight,
simple and widely used (e.g. Firefox's history). More specifically, I
envisage:
- We keep the session lists of raw and translated history as they are.
- Each command is stored instantly, so we do away with the need for an
autosave timer thread. A crash at any stage should leave your entire history
intact up to the last command completed.
- We store only raw history on disk (I think raw history is what we're
looking for 90% of the time). If we want translated history, we redo the
translation on the fly (this should require minimal computation, unless I've
missed something).
- History is indexed by session number and prompt number. This provides a
sensible behaviour if we have two IPython shells open together - the second
one to be opened will be the latter session (and will be able to access
commands entered in the other session as soon as they are completed).
- For magic commands, accessing a line from a previous session could look
like "-1#9" (9th line of immediately previous session).
- On starting IPython, we load the last (~40 lines/~2 sessions) from the
database into readline history.
Thoughts? Have I overlooked some key reason we use the existing system? Is
there a better alternative to SQLite? Would you design it differently? I've
not written any code for this yet, so I'm open to ideas. But if people think
that makes sense, I'm volunteering to make it happen.
Thanks,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20110216/71895949/attachment.html>
More information about the IPython-dev
mailing list