[IPython-dev] Re: ipython improvements

[Jeffery D. Collins]

Fernando Perez wrote:

> Jeffery D. Collins wrote:
> 
>> Hey Fernando,
>>
>> I've been thinking of a few improvements for IPython and thought I 
>> would share a few with you.
> 
> 
> I'mm cc-ing the ipyhton-dev list here so the discussion gets archived, 
> and also b/c others may also have useful contributions to make.
> 
>> 1.  match on the contents of __all__ if present in a module
>>
>> The problem I'm attempting to address is the large number of module 
>> attributes when that module imports lots of other objects.  One 
>> example is: from Numeric import *.  When completing on names in the 
>> given module, the names the module defines are lost among the imported 
>> names.   We need some way to complete only those names that the module 
>> actually defines.  Ideally, there would exist a parameter that would 
>> contain this information, but (IIRC) there is none.  However, we could 
>> use the user-definable __all__ attribute to determine the attribute 
>> matches. The disadvantage is that it must be created by the user.
>>
>> I have implemented this feature.  If the __all__ attribute is present, 
>> those names are used for completion; otherwise, it defaults to the 
>> original behavior.
> 
> 
> Great!  Is this implemented in the patches you sent me?  This is 
> something which has irked me always, but I never really looked into 
> fixing it.  I think your  approach of using __all__ as a marker of what 
> to complete on is the right one.  If it was not in your original 
> patches, perhaps you can unify it and send it my way.  I still haven't 
> looked at the others, so I could work on the whole thin in one pass.

The patch is on its way...

> 
>> 2.  configuration
>> What I need at the moment (if it doesn't exist) is a 'from x import y 
>> as z' mechanism.  As an alternative, I could just import x.y and 
>> assign that to z.  This should be possible, correct?
> 
> 
> Yes.  But you can also simply 'execute from x import y as z' in the 
> config file, or even 'execfile my_extra_configthings.py'.  The 'execute' 
> option is for single lines of plain python code you want to run at 
> startup, and the 'execfile' option gives you full freedom to have an 
> arbitrarily complex startup file which is regular python code.

Thanks.  I really should read the manual more carefully about this:)

>> We have discussed configuration briefly in the past.  You noted at the 
>> time that it would have been better to use python itself instead of 
>> specialized configuration files.  How difficult would it be to make 
>> such changes?
> 
> 
> Difficult, not really.  But time-consuming, quite.  The big problem is 
> the startup ipmaker routine, which is a single, very long function which 
> carefully tiptoes through bringing ipython up with zero modularity.  
> This stuff should probably all go into the constructor, or perhaps 
> partly into __init__ and partly into an explicitly callable initialize() 
> method, I'm not sure.  I keep the 2nd option open, because it might be 
> useful to have the constructor be reasonably simple, with an initializer 
> which must be called before running providing fine-tuning, for the 
> purposes of embedders.
> 
> In cleaning up ipmaker, one of the first things to go would be the 
> baroque config system currently in place.  The only slightly delicate 
> problem is how to handle recursive inclusions, so that one config file 
> can include another, and in the end have it all work transparently for 
> the user.  That requirement is key, because it is what makes profiles in 
> ipython a powerful customization tool.  Initially I thought it couldn't 
> be solved if the config files were real python code, but that was just a 
> sign of my inexperience with the language. With a bit of care that's not 
> a problem, and removing the current config manager would vastly simplify 
> a lot of code.

I agree that this is quite feasible with python.  There may already 
exist an implementation somewhere on which to model it.

> 
>> I guess that leads to the bigger question of how to reformulate 
>> IPython given the lessons-learned.  I am pretty satisfied with the 
>> current behavior, use it all of the time and in fact am now unable to 
>> use the default interactive environment!  Just looking quickly among 
>> the modules, has python been sufficiently patched to allow the removal 
>> of FlexCompleter?  Is DPyGetOpt really needed?  Python already has a 
>> fairly complete version of getopt, now that optparse has made it into 
>> the distribution.  For these, I realize these changes may introduce be 
>> backwards compatibility problems that would need to be resolved.  In 
>> all fairness, these are pretty minor issues.  I'll get back with you 
>> when I've had a chance to look through the code.  BTW, which is the 
>> appropriate forum for discussing IPython development?  scipy-dev/user? 
>> Is there a thread on this; if not, should one be started? :)
> 
> 
> Well, I just started it on ipython-dev with this reply to your mail :)  
> If you're not subscribed to the list, don't be afraid, it's very low 
> traffic.
> 
> Your other comments are on the nose, and basically mimic what I've been 
> thinking lately.  I'm leaning strongly towards making 2.3 a requirement 
> for ipython when the internal cleanup starts.  The reasons:
> 
> - DPyGetOpt: replace it with getoptparse.
> - Logger: replace it with the standard logging module (I think).
> - FlexCompleter: that's the new rlcompleter.py (it was my patch :)
> - Itpl: remove altogether, we can do without.
> 
> These changes would mean yanking a fair bit of code out of ipython, 
> which is always a good thing in my book.  Since this rewrite will likely 
> take some time, probably by the moment it's production-ready, most 
> people won't see the 2.3 requirement as a big burden.  And I'd still 
> keep the 0.5x branch around for critical fixes for users of py2.1.
> 
> If the 2.3 requirement is a big problem for many people, one could 
> always package getoptparse and logger from 2.3 along with ipython, I 
> suppose.  But I'll worry about that when the time comes.

I agree with the 2.3 requirement here.  I only occasionally refer to 2.2 
for debugging purposes when some odd or unexpected behavior occurs.  If 
there is a need to backport standard packages, a subdirectory could 
contain these modules.

> Basically the cleanup would entail:
> 
> - Cleaning ipmaker, with removing the current config handler as part of 
> this.
> 
> - Making Magic.py NOT be a mixin, but rather an attribute of the ipython 
> shell itself (self.MagicSystem = Magic() or somesuch).
> 
> - The 4 module removals mentioned above (requiring 2.3)
> 
> Please also take a look at the new_design document, which is where I try 
> to keep notes on this whole problem so everyone has access to them.  I 
> may even port this thread back to that document later.

Where can I find this document?  SF (seems to be down at the moment)?




-- 
Jeffery Collins (http://www.boulder.net/~jcollins)