[IPython-dev] How to handle extensions

[Robert Kern]

On Fri, Oct 9, 2009 at 15:56, Brian Granger <ellisonbg.net at gmail.com> wrote:
>
>
> On Fri, Oct 9, 2009 at 11:55 AM, Robert Kern <robert.kern at gmail.com> wrote:
>>
>> On 2009-10-09 13:00 PM, Brian Granger wrote:
>>
>> > * To be in IPython.extensions or IPython.config.profile a module has to
>> > satisfy a few things:
>> >
>> > - There has to be willingness amongst the core IPython developers to
>> > maintain the code.
>> > - The code needs to be reviewed (tests, docs, etc.)
>> > - If the code should be in IPython.core, it shouldn't be in extensions.
>> > - If an extension/profile can be distributed as a third party package,
>> > it should be.  Thus,
>> > the custom completer for enthough.traits should ship with
>> > enthought.traits.  Things related
>> > to numpy should ship with numpy.
>>
>
> This is an important point and I want to understand the issues and fix
> things now...
>
>>
>> There is one issue with this. You want to be able to configure some things
>> for a
>> particular package in your configuration files at startup, but not
>> actually
>> import the package, which may be expensive.
>
> With the changes I have in my local branch this is possible.  You simply
> enter the config
> information in the config files, but don't list the extensions in the config
> field of the extensions that should be loaded at startup.  Then you have a
> number of
> choices for loading the extension:
>
> get_ipython().load_extension('package.myextension')
>
> or
>
> %load_ext package.myextension

That certainly helps.

>> With care, many of the
>> package-specific extensions can be written in such a way as to be enabled
>> at
>> startup but not import the package until it absolutely must (in which case
>> the
>> user has probably already imported it himself). For example, I submitted a
>> patch
>> to ipy_traits_completer that replace isinstance(obj, HasTraits) checks for
>> one
>> that looked for superclasses named "HasTraits" before import HasTraits and
>> doing
>> the real isinstance() check.
>
> The question of a lazy loading of an extension is a good point.  For now,
> let's pretend
> that an extension can be imported/loaded without any side effects (it is not
> in a package
> that has an __init__ that imports everything).  Then lazy importing is easy
> right?
> You can just make sure that the package (traits/numpy/etc) isn't imported
> when the
> extension is imported, but later, when the extension is first used.

It depends strongly on what the extension does. If the extension adds
some %magics, then yes; you just implement the %magic methods to do
the imports inside instead of at the module level. However, for things
like completers, you need an entirely different strategy. You can't do
isinstance() checks.

> But don't you think that if someone has explicitly enabled a numpy based
> extension,
> it is because they are using numpy (so importing it at startup would happen
> anyways).
>
> If a user didn't want to *always* have the numpy extension loaded, they
> could
> simple maintain a numpy profile that loaded numpy.

Not at all. I have many IPython sessions that go on for hundreds and
hundreds of lines, and I change what I am doing frequently inside of
them. I may start a session to do one thing and then switch to do
something else. Or I realize that I need a capability after I've
already started IPython. For example, without the new GUI hooks, I
would frequently start up a plain IPython shell, do a bunch of
computations, then curse myself when I suddenly realize that I really
need to plot something. Ctrl-D, $ ipython -wthread, Up-Up-Up <grumble>
<grumble> <grumble>. Anything that lets me avoid having to know
exactly what I am going to do when I start up the shell is really,
really helpful. Profiles have always been basically useless to my
personal workflow because of this.

Having a quick way to enable an extension by name via %load_ext is
really helpful, but for a lot of things, you just want them to be
always available. If I need to do "%load_ext ipy_traits_completer"
after every time I do "from enthought.traits.api import *", I'm just
going to get annoyed by the needlessly repetitive typing.

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco