[IPython-dev] ipython sphinx directive

[John Hunter]

On Sat, Nov 7, 2009 at 9:21 AM, John Hunter <jdh2358 at gmail.com> wrote:
> On Sat, Nov 7, 2009 at 3:51 AM, Gael Varoquaux
> <gael.varoquaux at normalesup.org> wrote:
>
>> Very, very useful. I can't give much feedback, because I have no
>> available mental bandwidth
>
> That's OK, I don't either <wink>
>
> I did some thinking overnight and have codified my ideas in a proposal
> (appropriately in sphinx) included below -- also in html at
> http://matplotlib.sourceforge.net/ipymode/_build/html/proposal.html

OK, I've implemented 95% of the proposal;  pylab/pyplot is supported,
as are multi-line inputs and outputs, and the verbatim, doctest and
suppress directives.

A demo document is included below, which is rendered at
http://matplotlib.sourceforge.net/ipymode/_build/html/ipy_demo.html,
and the ipython_directive is attached.  Stuff that still needs work:

  * A bit of cleaning needs to be done, since I am currently mixing in
ipython and matplotlib functionality and I suspect some will want a
clean segregation, but that will not be too hard.

  *  I have yet to implement the ifre's from the proposal, but will
take a look at that shortly.

  * Another big piece that I don't know how to handle yet are the
magics like 'cd', 'pwd', 'ls'.

  * on the doctest I am currently raising an exception on failure
which breaks the build.  I should be inserting some red ERROR code in
the rendered docs, probably.

  * there are several limitations -- I hard code the prompt fmts, I
assume two blank new lines breaks an input/output block (some string
output might have multiple new lines in it), I have some internal
hacks arising from my limited understanding of ipy API,

Very exciting stuff!


=================
Ipython Directive
=================

The ipython directive is a stateful ipython shell for embedding in
sphinx documents.  It knows about standard ipython prompts, and
extracts the input and output lines.  These prompts will be renumbered
starting at ``1``.  The inputs will be fed to an embedded ipython
interpreter and the outputs from that interpreter will be inserted as
well.

.. ipython::

   In [136]: x = 2

   In [137]: x**3
   Out[137]: 8

The state from previous sessions is stored, and standard error is
trapped.  At doc build time, ipython's output and std err will be
inserted, and prompts will be renumbered.  So the prompt below should
be ``In [3]:`` in the rendered docs.

.. ipython::

  In [138]: z = x*3   # x is recalled from previous block

  In [139]: z
  Out[139]: 6

  In [140]: print z
  --------> print(z)
  6

  In [141]: q = z[)   # this is a syntax error -- we trap ipy exceptions
  ------------------------------------------------------------
     File "<ipython console>", line 1
       q = z[)   # this is a syntax error -- we trap ipy exceptions
	     ^
  SyntaxError: invalid syntax


The embedded interpeter supports some limited markup.  For example,
you can put comments in your ipython sessions, which are reported
verbatim.  There are some handy "pseudo-decorators" that let you
doctest the output.  The inputs are fed to an embbedded ipython
session and the outputs from the ipython session are inserted into
your doc.  If the output in your doc and in the ipython session don't
match on a doctest assertion, an error will be


.. ipython::

   In [1]: x = 'hello world'

   # this will raise an error if the ipython output is different
   @doctest
   In [2]: x.upper()
   Out[2]: 'HELLO WORLD'

   # some readline features cannot be supported, so we allow
   # "verbatim" blocks, which are dumped in verbatim except prompts
   # are continuously numbered
   @verbatim
   In [3]: x.st<TAB>
   x.startswith  x.strip


Multi-line input is supported.

.. ipython::

   In [130]: url = 'http://ichart.finance.yahoo.com/table.csv?s=CROX\
      .....: &d=9&e=22&f=2009&g=d&a=1&br=8&c=2006&ignore=.csv'

   In [131]: print url.split('&')
   --------> print(url.split('&'))
   ['http://ichart.finance.yahoo.com/table.csv?s=CROX', 'd=9', 'e=22',
'f=2009', 'g=d', 'a=1', 'b=8', 'c=2006', 'ignore=.csv']

   In [60]: import urllib


You can do doctesting on multi-line output as well.  Just be careful
when using non-deterministic inputs like random numbers in the ipython
directive, because your inputs are ruin through a live interpreter, so
if you are doctesting random output you will get an error.  Here we
"seed" the random number generator for deterministic output, and we
suppress the seed line so it doesn't show up in the rendered output

.. ipython::

   In [133]: import numpy.random

   @suppress
   In [134]: numpy.random.seed(2358)

   @doctest
   In [135]: np.random.rand(10,2)
   Out[135]:
   array([[ 0.64524308,  0.59943846],
	  [ 0.47102322,  0.8715456 ],
	  [ 0.29370834,  0.74776844],
	  [ 0.99539577,  0.1313423 ],
	  [ 0.16250302,  0.21103583],
	  [ 0.81626524,  0.1312433 ],
	  [ 0.67338089,  0.72302393],
	  [ 0.7566368 ,  0.07033696],
	  [ 0.22591016,  0.77731835],
	  [ 0.0072729 ,  0.34273127]])


Another demonstration of multi-line input and output

.. ipython::

   In [106]: print x
   --------> print(x)
   jdh

   In [109]: for i in range(10):
      .....:     print i
      .....:
      .....:
   0
   1
   2
   3
   4
   5
   6
   7
   8
   9


Most of the "pseudo-decorators" can be used an options to ipython
mode.  For example, to setup maptlotlib pylab but suppress the output,
you can do.  When using the matplotlib ``use`` directive, it should
occur before any import of pylab.  This will not show up in the
rendered docs, but the commands will be executed in the embedded
interpeter and subsequent line numbers will be incremented to reflect
the inputs::


  .. ipython::
     :suppress:

     In [144]: from pylab import *

     In [145]: ion()

.. ipython::
   :suppress:

   In [144]: from pylab import *

   In [145]: ion()

Likewise, you can set ``:doctest:`` or ``:verbatim:`` to apply these
settings to the entire block.


You can create one or more pyplot plots and insert them with the
``@savefig`` decorator.

.. ipython::

   @savefig plot_simple.png width=4in
   In [151]: plot([1,2,3]);

   # use a semicolon to suppress the output
   @savefig hist_simple.png width=4in
   In [151]: hist(np.random.randn(10000), 100);

In a subsequent session, we can update the current figure with some
text, and then resave

.. ipython::


   In [151]: ylabel('number')

   In [152]: title('normal distribution')

   @savefig hist_with_text.png
   In [153]: grid(True)




That's all folks!