[IPython-dev] cell magics

[Fernando Perez]

Hi Jason,

On Fri, Feb 15, 2013 at 7:21 PM, Jason Grout
<jason-sage at creativetrax.com> wrote:
> William Stein and I have been discussing cell magics lately as he is
> implementing them in his Salvus project.  I think the result, which is a
> little different than the current IPython system, is more consistent,
> easier to read, and more elegant.  What are your thoughts?

Mmh, for now, big -1 from me, for what I think are good reasons within
the design and philosophy of IPython.  I happen to think that our
design is actually more consistent, easier to read, and more elegant,
but you're obviously entitled to your opinion on the matter ;)

> PROPOSAL:
>
> Line and cell magics are normal python functions that take a string as
> their first input.  The string is either the rest of the line, or if
> there is no discernible string on the rest of the line, the string is
> taken as the rest of the cell.  As an example:
>
> %timeit(runs=10) 2+3
>
> which gets translated to time("2+3", runs=10)
>
> A cell magic is exactly the same function, only it is invoked by putting
> the string on the following lines, rather than on the same line, so the
> string is taken as the remainder of the cell:
>
> %timeit(runs=10)
> 2+3
> 5+6
>
> which gets translated to timeit("2+3\n5+6", runs=10)
>
> Notice there is no distinction between line and cell magic
> functions---the distinction entirely happens in how they are invoked
> (whether the string is on the same line or on following lines).  Also,
> magic options are passed in exactly the same way that arguments are
> always passed to python functions, instead of having a non-pythonic,
> bash-like syntax for options.

The whole point of magics is precisely to have a non-pythonic,
no-parentheses, bash-like syntax that emphasizes ease of typing (the
space bar is much, much easier to hit than the parens keys).  Magics
were put in from day one (not even by me, as the rudiments of the
system and the 'magics' name were already in place in Janko Hauser's
IPP) explicitly with the intention of having a syntax that would be
very 'command-like'.

When I inherited IPP and put it into IPython, I had used extensively
IDL which had a similar notion of 'commands', and I later discovered
that matlab has something similar (though I didn't know it at the
time).  These commands are separate from the main programming language
(though for IDL and matlab, they are part of the same specification
since the whole thing is produced by a single group).

The point of such a system is that it meets a number of criteria:

- the namespace where these exist is completely orthogonal to the
Python one.  In IPython, aside from the small transgression we make
with %pylab that does a few global imports, *you* manage your Python
namespaces explicitly as much as you would in normal Python code.
Sage takes a different approach and imports on the order of thousands
of names by default:
----------------------------------------------------------------------
| Sage Version 4.7.1, Release Date: 2011-08-11                       |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
sage: len(dir())
1878

I personally don't like this. For comparison, in IPython you get this
(you can choose to load more stuff, obviously, but you do so
explicitly and knowingly):

In [1]: len(dir())
Out[1]: 45

I can keep a namespace of 45 things in my head, I can't really manage
one with 2000 things.

This orthogonality is important because it lets us easily provide the
convenience of preloading all magics relevant to IPython control
without polluting the user's namespace.  This also makes it possible
to do things like doing a full reset of the user ns without impacting
the control system.

The orthogonality of the namespaces also means that extensions can be
loaded at runtime that add special functionality and new magics with
zero risk of breaking normal user code, as the namespaces will never
collide, by design.


- by being really separate objects, it's clear that they play a role
that's a bit different from that of normal code: they control IPython
itself, and they do things that go beyond the Python syntax.

Sage chose to create a hybrid of the python language and other things
(ints with different semantics, extended list/range syntax, different
power operator, etc).  That's perfectly OK for you guys, but we drew
the design boundaries differently.  We leave the language alone, and
introduce special cases that are always demarcated by special,
orthogonal syntax that's invalid in Python itself (modulo a few
convenience automatic transforms that can be turned off if desired and
are there just to save time for the diehards).  I am personally happy
to have chosen that route over a decade ago, and it seems that over
the year the rest of the project team has also agreed with this
approach.


- the syntax is deliberately different from a python one: not only is
it quicker and more convenient to type 'func -k args' than
'func("arg", kw=1)' when working interactively, I also very much
prefer to know that "magics are like bash, Python is like python" than
thinking that the magic system is some kind of half-way hybrid between
the two.  I find it eminently more readable as it uses two
conceptually distinct categories for which my brain already has
buckets ready, instead of creating a new special-case, bizarre blend
of the two that I wouldn't use anywhere else.


- our design makes it just as easy to unify cell/line magics if you
want.  As you can see in the docs
(http://ipython.org/ipython-doc/dev/interactive/reference.html#defining-your-own-magics),
you can have a single function (not even a class is needed) that does
line/cell magics in one shot.  I don't find that any more complex than
your proposal.  Furthermore, we do have some very clean base classes
that let you handle the more complex cases with zero boilerplate, as
you can focus only on your own functionality and state management
(when classes are warranted) and we take care of the basics).


In summary, while you're free to prefer the design in Sage, I very
much disagree on all points and see very little chance that we'll
change ours.  If anything, the exercise of writing this down has
convinced me further that what we have is a solid piece of
architecture we shouldn't mess with.

All the best,

f