[IPython-dev] Curses Frontend

[Brian Granger]

Wendell,

> I have decided to implement a curses frontend to ipython.

Fantastic, this would be a great addition to IPython!  This comes at a
great time.
As you know we are finishing up 0.11, which has clean up the codebase
quite a bit.
Also, Fernando has been working with some folks in Coumbia on a Qt based IPython
notebook.  I am not sure of the status (Fernando is in the Amazon
jungle this week), but
once he returns we can coordinate your work with this effort.

> My main goals:
>  - complete functionality of standard ipython - magics, history, etc.
>  - automatic display of completion, not requiring the tab key - tab key
> fills it in
>  - automatic display of docstrings, not requiring the '?' key
>  - zero pollution of output by docstrings/help (easier to read through
> history)
>  - syntax highlighting of input lines as well as output (using
> ipython's color scheme)

That would be absolutely great to have all of these things.

> The first three will require serious work with curses, as the standard
> library curses module is quite limited, but I believe it can be managed
> in the end with no further dependencies. However, I am wondering if
> perhaps it makes more sense to depend on something like 'urwid'.

I am not familiar with urwid, and I don't know much about curses.  We
are trying to
keep dependencies to a minimum, but having curses for a dep for your
frontend is perfectly reasonable.

> As for the fourth, I believe the tokenize module can not handle
> incomplete input, and so I am thinking that it will use 'pygments' if
> its available, and otherwise not highlight. Eventually, perhaps, it
> would have a configuration option for input syntax highlighting; if the
> option were enabled and pygments was not installed, then ipython would
> issue a warning.

We would very much like to replace our current tokenizer with Pygments.
Pygments is so easy to install and so common these days, I think that *all*
of IPython should use it.  Obviously, it should switch to no coloring mode if
pygments is not installed.

> I have, of course, seen and used bpython, and this is partially inspired
> from that. However, I cannot see any reasonable way to integrate the two
> projects; while it will be easy to work with ipython, the code for
> bpython does not look... well... easy to apply to ipython, shall we say.

Yes, IPython's codebase, While cleaner now, is still a bit of a beast.
 We hope to
continue to clean it up, but that will take time.  Eventually, we would like it
if a project like bpython could use IPython as its underlying
interpreter.  But that
will take time.  For now, I think that working with IPython is a good choice.

> I have a number of questions:
> - First and foremost, should it be implemented as an extension, or as a
> frontend + subclass of Shell? I believe implementation as an extension
> is possible, and not terribly complicated or confusing, but it does seem
> a little heavy for the extension system.

Currently, IPython trunk has done away with the old Shell class.  Our core class
is not iplib.InteractiveShell.  Thus, I would start by looking there.

> - Secondly, how should I handle dependencies? I have not looked into
> anything beyond the main curses library, but using a third-party library
> could quite possibly substantially simplify things, but it would add a
> dependency. Also, I would like input syntax highlighting as noted above,
> and if anyone has comments on that I would welcome suggestions.

I would keep them minimal, but if you want to use something
additional, we can definitely
talk about it.  If it is small, we can ship it in IPython.externals.
We do recognize that
various IPython GUIs/frontends will have additional dependencies.

> - Thirdly, I would like code hosting advice. Should this go in the
> ipython bazaar database? should I get my own something or other? This is
> something I would very much appreciate - let me know how I should handle
> this.

I would handle it this way for now:

* Some of your work will likely be on the IPython core itself.  While
we have done a lot of work on it
you will quickly find areas that need to be updated in order for you
to do what you want.  For this part
of it, I would simply create a branch on launchpad of the IPython
trunk.  This will allow us to merge
your work on the IPython core quickly into trunk.

* The part of your work that is focused on the curses frontend, could
be handled in two ways: it could be a separate project on lp for now
and eventually moved into IPython when it matures.  Or you could just
branch your IPython core branch and do the work there.  My intuition
says that having a separate project for now will be a better and more
flexible option.  But this is also up to you.

> - Lastly, I would like comments on the overall design. My current plan
> is as follows:
>
> ------------------------
> | completion/help box  |
> |    6 lines or so     |
> |----------------------|
> | ipython output       |
> | fills the main space |
> | includes completed   |
> | input                |
> | In [1]: 1            |
> | Out[1]: 1             |
> |In [2]: 2             |
> |Out[2]: 2             |
> |----------------------|
> |In[3]: | Input box    |
> |       | 4 lines or so|
> ------------------------

I think having as clean of a design as possible is best, so I would
try to minimize the stuff at the top.  But, the challenge is that some
completions and help strings are quite long.  I am not sure how to
handle that in a clean way.

> My vision is that as one typed, completions  or help would pop-up above.
> If the user pressed tab, completion would fill in as much as possible,
> and also expand to show all the completions available. If one pressed
> tab again, it would page through completions; if one continued typing,
> the completions would narrow; if one finished the word, completions
> would go back to their old 6-10 lines.
> If one started a function, or hit ',' within, the function docstring
> would instead show in the upper box, with the same sort of tab-paging
> capability. If one instead used the old '?' and '??' syntax, everything
> would be replaced by a pager, giving one more space to read it all.
>
> I have many more ideas for smaller things - options to output
> python-interpreter-like code on exit, for example - and there will be
> many obstacles to overcome (such as integrating the built-in editor
> capability), I have spent the past week or so going through ipython
> code, considering possibilities, and experimenting, and I think I have a
> fairly good idea of the work ahead of me.
>
> So that is my idea. I have the time and energy to carry it through - I
> know it will take a long time. It will, of course, require at least
> ipython 0.11 - it will almost certainly not be done before that is
> released in any case, and it makes the most sense to use the wonderfully
> reorganized code of 0.11.

This is fantastic news.  And yes, definitely start from trunk.

> Please reply with any comments or suggestions! Also, if anyone is
> already working on something like this, or would like to help me, I
> would love to hear that!

I am going to send this now as I have to run.  I will reply later with
more thoughts
on the design as it relates to the IPython core.

Cheers,

Brian

> -Wendell
> _______________________________________________
> IPython-dev mailing list
> IPython-dev at scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com