[IPython-dev] Configuration system thoughts

[Fernando Perez]

On Mon, Jul 27, 2009 at 12:40 PM, Brian Granger<ellisonbg.net at gmail.com> wrote:
> Hi,
>
> I am currently refactoring the IPython configuration system.  I thought this
> would be a fairly straightforward task to move IPython to a single unified
> config system with a new config file format.  But, it is turning out to be
> quite massive undertaking.  I wanted to share some of my notes at this point
> to generate discussion about this...
>
> * Currently, we read assemble the main config objects in ipmaker.py and save
> them as IP.rc which is a Struct.  From then on, IPython code can reference
> the config variables using a notation like IP.rc.autocall, etc.  The logic
> in ipmaker is super-messy and is one of the main config-y-things that needs
> refactoring.

Sorry you ended up having to see that :)  Indeed, that code is
absolutely heinous,  and the reasons are purely historical artifact.
IPython started life as my personal $PYTHONSTARTUP file a few days
after I'd begun to learn to use python, so initially it was just one
long script to load and configure a bunch of tweaks to
sys.displayhook.  When it became 'a program' I just copied that mess
over, indented it 4 spaces and gave it a name.  Thus was ipmaker born
unto the world :)

But eventually we had to do something about this, and now is the time.
 I'm more than happy to help shoulder a bit of the load.

> * But, this is not merely a configuration system, it is also a runtime
> attribute system.  By this, I mean that the values in IP.rc are changed at
> runtime by various parts of the code to change the behavior of the running
> IPython.  Some attributes (like autocall) can be changed by modifying IP.rc
> at runtime and other's (the prompts) can't.  What I mean by "can't" is that
> you can change IP.rc, but IPython's behavior won't change.  An example of
> this is that to get the prompts to change at runtime you need to  hack into
> IP.outputcache.prompt1.  In other cases (like autocall), the relevant
> IPython code always looks at IP.rc.autocall, so if it is changed, IPython's
> behavior changes.
>
> Anyone who is paying attention at this point will think... "ahh, observers,
> notification, python", this sounds like enthought.traits!!!  Yes, traits
> would definitely be one solution for dealing with this issue.  But, we
> *really* want IPython to run on IronPython and Jython, so having traits as a
> new dependency (it has a ~5000 line C-module) won't work.  The only way we
> could use traits is if someone implemented a pure Python version of traits
> (or a suitable subset).

For the record (you know this already) we're in full agreement here.
For example, as of a few days ago apparently ipython runs in
IronPython:

http://lists.ironpython.com/pipermail/users-ironpython.com/2009-June/010590.html

Obviously this means just the interactive interpreter, not all the
kernel code, but this is still great news.  It would be crazy to kill
this in our next version by introducing a dependency like Traits, as
much as it can have other benefits.

One thing we should strive for, is to remove the need (as much as
possible) for any special behavior with options, like it exists today.
 Basically if something lives in the runtime config holders, then it
should be OK to set its value at runtime and things should behave
normally and honor this change.  If something can't be simply changed
like that (say because it really was an argument to some function that
has already been called and is now gone), then it should NOT be
visible as a runtime option.  We can then put in specific calls to set
more complex behavior at runtime when needed, but such a design
decision would at least allow us to decouple the mess you've
identified of having a values holder (the rc struct) where  some
things can be touched at runtime and others not.

> * We need to move away from the model of "IP holds the config for
> everything" to each object holding and loading its own config data.
>
> * We need to de-couple configuration (the behavior of objects upon creation)
> from notification/observation (how objects work together at runtime).

Broadly speaking I agree, with some minor adjustments.   See below.

> With these things in mind, here is what I am thinking...
>
> All IPython classes will inherit from a number of base classes that provide
> the basic features we need also classes to have:
>
> * Configurable = an object that can be configured from the config files,
> command line, and by passing in options.  Basically, this is an object that
> has a super fancy __init__ logic (possibly not in __init__).  Each class
> will declare where its default config file lives, what its command line
> options are, etc.
>
> * NotificationSource = an object that can notify other objects when its
> attributes change.  This is basically like a HasTraits.
>
> Having these 2 different subclasses should de-couple the question of
> configuration from the question of notification/runtime attributes.
>
> Thoughts?  Ideas? Feedback?

I'm not 100% sure that we want a base class that is at the root of the
object hierarchy to provide management of configuration and
notification.  But I  think both ideas are going in the right
direction, so here's a possible twist on them (I'm tired so this may
be well off-base, we can and should iterate on it):

- Configurable would be used to define configuration objects, but
these would be held as attributes of the various 'user-facing'
objects.  Basically I'm thinking of a 'has a' relationship rather than
an 'is a' one, to reduce couplings.

- Objects that need it will use Configurable, but in practice the main
application 'entry points' will probably use more than one object with
a configuration instance. For these, it should be possible to
compose/aggregate config instances easily.  Something like:

s = IPythonShell()

s has s.kernel, s.frontend components.  Each of these has a config
instance above, but the user should only need to use s.config
directly, which will hold the configuration of all the subcomponents.

I need to crash now though, I'll try to think a little more about this
tomorrow and we'll  continue to iterate...

Thanks for tackling this, I'm really happy about it (long needed)!

f