[IPython-dev] Full input syntax support finished and ready for review/merge

[Fernando Perez]

Hi all (esp. frontend authors),

In this branch
http://github.com/fperez/ipython/tree/blockbreaker

I've now completed a functional first pass on complete IPython input
support, so that frontends can convert statically all 'ipython syntax'
that can be determined statically.  This includes all escapes (%, ?,
!, !!, etc) plus things like 'a =! ls' and removal of python/ipython
prompts from pasted input.

Rather than being a mess of multiple little functions scattered all
over ipython and interleaved with execution, logging, etc, everything
now is in a single file:

http://github.com/fperez/ipython/blob/blockbreaker/IPython/core/inputsplitter.py

and it's got a solid set of tests:

http://github.com/fperez/ipython/blob/blockbreaker/IPython/core/tests/test_inputsplitter.py

which give 100% test coverage:

(blockbreaker)amirbar[tests]> nosetests --with-coverage --cover-erase
--with-doctest --cover-package=IPython.core.inputsplitter
IPython.core.inputsplitter test_inputsplitter.py
Name                         Stmts   Exec  Cover   Missing
----------------------------------------------------------
IPython.core.inputsplitter     240    240   100%
----------------------------------------------------------------------
Ran 57 tests in 0.132s

OK


So a few things:

- from anyone: code review/feedback is welcome before I proceed to
merge this.  The only significant feature missing is probably the
creation of a couple of objects to provide extensibility for user
input filters, but I want to delay that until we land this in real
use, so we see better what the right interface should be.  For now,
frontends have a tool they can use and their part of the API should be
pretty stable (modulo any fixes that may be pointed in review).

- from frontend authors: let me know if using this gives you any
troubles, or if you see any missing feature that could make your life
easier.


This took a lot of work, but it's a major, critical chunk of ipython
that is now well isolated, specified and tested.  Since so much of
what we do is provide extended syntax for daily use, rationalizing
this was long overdue and I'm glad we took the time to do it right.
This will let us shed tons of tricky, untestable codepaths.

I should note that I didn't write this completely from scratch: while
the code architecture is new, I reused all the existing little
functions that did the low-level work (especially many tricky
regexps).  But as I integrated those, I added tests for each and every
line.  This gives us the benefit of a clean rethinking, without having
gotten trapped into a 'second system syndrome' madness.

On to the kernel :)

Cheers,

f