[IPython-dev] Re: ipython improvements

Fernando Perez fperez at colorado.edu
Tue Sep 30 16:52:02 EDT 2003

Jeffery D. Collins wrote:
> Hey Fernando,
> I've been thinking of a few improvements for IPython and thought I would 
> share a few with you.

I'mm cc-ing the ipyhton-dev list here so the discussion gets archived, and 
also b/c others may also have useful contributions to make.

> 1.  match on the contents of __all__ if present in a module
> The problem I'm attempting to address is the large number of module 
> attributes when that module imports lots of other objects.  One example 
> is: from Numeric import *.  When completing on names in the given 
> module, the names the module defines are lost among the imported names. 
>   We need some way to complete only those names that the module actually 
> defines.  Ideally, there would exist a parameter that would contain this 
> information, but (IIRC) there is none.  However, we could use the 
> user-definable __all__ attribute to determine the attribute matches. 
> The disadvantage is that it must be created by the user.
> I have implemented this feature.  If the __all__ attribute is present, 
> those names are used for completion; otherwise, it defaults to the 
> original behavior.

Great!  Is this implemented in the patches you sent me?  This is something 
which has irked me always, but I never really looked into fixing it.  I think 
your  approach of using __all__ as a marker of what to complete on is the 
right one.  If it was not in your original patches, perhaps you can unify it 
and send it my way.  I still haven't looked at the others, so I could work on 
the whole thin in one pass.

> 2.  configuration
> What I need at the moment (if it doesn't exist) is a 'from x import y as 
> z' mechanism.  As an alternative, I could just import x.y and assign 
> that to z.  This should be possible, correct?

Yes.  But you can also simply 'execute from x import y as z' in the config 
file, or even 'execfile my_extra_configthings.py'.  The 'execute' option is 
for single lines of plain python code you want to run at startup, and the 
'execfile' option gives you full freedom to have an arbitrarily complex 
startup file which is regular python code.

> We have discussed configuration briefly in the past.  You noted at the 
> time that it would have been better to use python itself instead of 
> specialized configuration files.  How difficult would it be to make such 
> changes?

Difficult, not really.  But time-consuming, quite.  The big problem is the 
startup ipmaker routine, which is a single, very long function which carefully 
tiptoes through bringing ipython up with zero modularity.  This stuff should 
probably all go into the constructor, or perhaps partly into __init__ and 
partly into an explicitly callable initialize() method, I'm not sure.  I keep 
the 2nd option open, because it might be useful to have the constructor be 
reasonably simple, with an initializer which must be called before running 
providing fine-tuning, for the purposes of embedders.

In cleaning up ipmaker, one of the first things to go would be the baroque 
config system currently in place.  The only slightly delicate problem is how 
to handle recursive inclusions, so that one config file can include another, 
and in the end have it all work transparently for the user.  That requirement 
is key, because it is what makes profiles in ipython a powerful customization 
tool.  Initially I thought it couldn't be solved if the config files were real 
python code, but that was just a sign of my inexperience with the language. 
With a bit of care that's not a problem, and removing the current config 
manager would vastly simplify a lot of code.

> I guess that leads to the bigger question of how to reformulate IPython 
> given the lessons-learned.  I am pretty satisfied with the current 
> behavior, use it all of the time and in fact am now unable to use the 
> default interactive environment!  Just looking quickly among the 
> modules, has python been sufficiently patched to allow the removal of 
> FlexCompleter?  Is DPyGetOpt really needed?  Python already has a fairly 
> complete version of getopt, now that optparse has made it into the 
> distribution.  For these, I realize these changes may introduce be 
> backwards compatibility problems that would need to be resolved.  In all 
> fairness, these are pretty minor issues.  I'll get back with you when 
> I've had a chance to look through the code.  BTW, which is the 
> appropriate forum for discussing IPython development?  scipy-dev/user? 
> Is there a thread on this; if not, should one be started? :)

Well, I just started it on ipython-dev with this reply to your mail :)  If 
you're not subscribed to the list, don't be afraid, it's very low traffic.

Your other comments are on the nose, and basically mimic what I've been 
thinking lately.  I'm leaning strongly towards making 2.3 a requirement for 
ipython when the internal cleanup starts.  The reasons:

- DPyGetOpt: replace it with getoptparse.
- Logger: replace it with the standard logging module (I think).
- FlexCompleter: that's the new rlcompleter.py (it was my patch :)
- Itpl: remove altogether, we can do without.

These changes would mean yanking a fair bit of code out of ipython, which is 
always a good thing in my book.  Since this rewrite will likely take some 
time, probably by the moment it's production-ready, most people won't see the 
2.3 requirement as a big burden.  And I'd still keep the 0.5x branch around 
for critical fixes for users of py2.1.

If the 2.3 requirement is a big problem for many people, one could always 
package getoptparse and logger from 2.3 along with ipython, I suppose.  But 
I'll worry about that when the time comes.
Basically the cleanup would entail:

- Cleaning ipmaker, with removing the current config handler as part of this.

- Making Magic.py NOT be a mixin, but rather an attribute of the ipython shell 
itself (self.MagicSystem = Magic() or somesuch).

- The 4 module removals mentioned above (requiring 2.3)

Please also take a look at the new_design document, which is where I try to 
keep notes on this whole problem so everyone has access to them.  I may even 
port this thread back to that document later.

If you consider doing any of this for real, as a timetable I'd consider:

1. Incorporate your recent patches for completion, as a 0.5.1 bugfix release.

2. I need to finish another 0.5.x with Gary Bishop's patches for Windows color 
handling.  Those require some discussion and design (under Windows), which is 
why I've fallen way behind on them.  But they are important for Windows users, 
which is a lot of people out there.

3. Start with the cleanup, with no more work going into the 0.5.x branch 
except for bugfixes.

Many thanks for your input, and let's see where this goes,


ps.  Since we're both in Boulder, if one day you want to meet over coffee to 
talk about this, drop me a call.  If you are considering any of this for real, 
it may be a lot more efficient to meet in person, with a laptop going over code.

More information about the IPython-dev mailing list