From ralf at brainbot.com  Wed Oct  1 14:21:57 2003
From: ralf at brainbot.com (Ralf Schmitt)
Date: Wed, 01 Oct 2003 20:21:57 +0200
Subject: [IPython-dev] replacement for deep_reload.py
Message-ID: <3F7B1B45.5040907@brainbot.com>

Hi,
I've just hacked my own version of deep_reload.py.
It works by replacing sys.modules with an empty hash and installing
it's __builtin__.__import__, which just calls __builtin__.reload and the old
__builtin__.__import__.
The only problem I've discovered so far, is that dreload(os)
doesn't work (dreload(some module which imports os) however does work).
I've tested it with python 2.3.1 and IPython from cvs.
The code is very simple, so maybe you can save yourself some headache :)
I'll use that version in the next few days, and will report on other 
problems I found..
- Ralf





import sys
import __builtin__


# Save the original hooks
builtin_import = None
old_modules = {}
reloaded = {}

def my_import_hook(name, globals=None, locals=None, fromlist=None):
    global reloaded
   
##     if fromlist:
##         print 'Importing', fromlist, 'from module', name
##     else:
##         print 'Importing module', name

    if old_modules.has_key(name) and not reloaded.has_key(name):
        reloaded[name]=1
        print "Reloading", name
        __builtin__.reload(old_modules[name])
       
    return builtin_import(name, globals, locals, fromlist)


# Replacement for reload()
def reload(module, exclude=['sys', '__builtin__', '__main__']):
    """Recursively reload all modules used in the given module.  Optionally
    takes a list of modules to exclude from reloading.  The default exclude
    list contains sys, __main__, and __builtin__, to prevent, e.g., 
resetting
    display, exception, and io hooks.
    """
    global builtin_import
    global old_modules
    global reloaded

    reloaded = {}
    old_modules = sys.modules
    sys.modules = {}
    for ex in exclude+list(sys.builtin_module_names):
        if old_modules.has_key(ex):
            sys.modules[ex] = old_modules[ex]
            reloaded[ex]=1

    builtin_import = __builtin__.__import__
    __builtin__.__import__ = my_import_hook
    try:
        return __builtin__.reload(module)
    finally:
        __builtin__.__import__ = builtin_import
        for m in old_modules:
            if not sys.modules.has_key(m):
                #print "RESTORING module", m
                sys.modules[m] = old_modules[m]
            else:
                #print "NOT RESTORING", m
                pass



From fperez at colorado.edu  Wed Oct  1 15:44:31 2003
From: fperez at colorado.edu (Fernando Perez)
Date: Wed, 01 Oct 2003 13:44:31 -0600
Subject: [IPython-dev] replacement for deep_reload.py
In-Reply-To: <3F7B1B45.5040907@brainbot.com>
References: <3F7B1B45.5040907@brainbot.com>
Message-ID: <3F7B2E9F.1050701@colorado.edu>

Ralf Schmitt wrote:
> Hi,
> I've just hacked my own version of deep_reload.py.
> It works by replacing sys.modules with an empty hash and installing
> it's __builtin__.__import__, which just calls __builtin__.reload and the old
> __builtin__.__import__.
> The only problem I've discovered so far, is that dreload(os)
> doesn't work (dreload(some module which imports os) however does work).
> I've tested it with python 2.3.1 and IPython from cvs.
> The code is very simple, so maybe you can save yourself some headache :)
> I'll use that version in the next few days, and will report on other 
> problems I found..
> - Ralf

Great!  Many thanks for this.  Please give it a bit more pounding, and I'd 
encourage other users of dreload to also try it out.  The code is definitely 
far simpler than the original dreload, but since I don't understand that code 
too well, I'd like to tiptoe a bit on this issue.  If it survives a bit of 
pounding and discussion, I'll definitely be glad to put it in.

Obviously, I'll also test it myself and think about it a bit.

Best,

f.


From jcollins at boulder.net  Wed Oct  1 17:53:22 2003
From: jcollins at boulder.net (Jeffery D. Collins)
Date: Wed, 01 Oct 2003 15:53:22 -0600
Subject: [IPython-dev] Re: ipython improvements
In-Reply-To: <3F79ECF2.8060901@colorado.edu>
References: <3F79CCCD.5070303@boulder.net> <3F79ECF2.8060901@colorado.edu>
Message-ID: <3F7B4CD2.1090204@boulder.net>

Fernando Perez wrote:

> Jeffery D. Collins wrote:
> 
>> Hey Fernando,
>>
>> I've been thinking of a few improvements for IPython and thought I 
>> would share a few with you.
> 
> 
> I'mm cc-ing the ipyhton-dev list here so the discussion gets archived, 
> and also b/c others may also have useful contributions to make.
> 
>> 1.  match on the contents of __all__ if present in a module
>>
>> The problem I'm attempting to address is the large number of module 
>> attributes when that module imports lots of other objects.  One 
>> example is: from Numeric import *.  When completing on names in the 
>> given module, the names the module defines are lost among the imported 
>> names.   We need some way to complete only those names that the module 
>> actually defines.  Ideally, there would exist a parameter that would 
>> contain this information, but (IIRC) there is none.  However, we could 
>> use the user-definable __all__ attribute to determine the attribute 
>> matches. The disadvantage is that it must be created by the user.
>>
>> I have implemented this feature.  If the __all__ attribute is present, 
>> those names are used for completion; otherwise, it defaults to the 
>> original behavior.
> 
> 
> Great!  Is this implemented in the patches you sent me?  This is 
> something which has irked me always, but I never really looked into 
> fixing it.  I think your  approach of using __all__ as a marker of what 
> to complete on is the right one.  If it was not in your original 
> patches, perhaps you can unify it and send it my way.  I still haven't 
> looked at the others, so I could work on the whole thin in one pass.

The patch is on its way...

> 
>> 2.  configuration
>> What I need at the moment (if it doesn't exist) is a 'from x import y 
>> as z' mechanism.  As an alternative, I could just import x.y and 
>> assign that to z.  This should be possible, correct?
> 
> 
> Yes.  But you can also simply 'execute from x import y as z' in the 
> config file, or even 'execfile my_extra_configthings.py'.  The 'execute' 
> option is for single lines of plain python code you want to run at 
> startup, and the 'execfile' option gives you full freedom to have an 
> arbitrarily complex startup file which is regular python code.

Thanks.  I really should read the manual more carefully about this:)

>> We have discussed configuration briefly in the past.  You noted at the 
>> time that it would have been better to use python itself instead of 
>> specialized configuration files.  How difficult would it be to make 
>> such changes?
> 
> 
> Difficult, not really.  But time-consuming, quite.  The big problem is 
> the startup ipmaker routine, which is a single, very long function which 
> carefully tiptoes through bringing ipython up with zero modularity.  
> This stuff should probably all go into the constructor, or perhaps 
> partly into __init__ and partly into an explicitly callable initialize() 
> method, I'm not sure.  I keep the 2nd option open, because it might be 
> useful to have the constructor be reasonably simple, with an initializer 
> which must be called before running providing fine-tuning, for the 
> purposes of embedders.
> 
> In cleaning up ipmaker, one of the first things to go would be the 
> baroque config system currently in place.  The only slightly delicate 
> problem is how to handle recursive inclusions, so that one config file 
> can include another, and in the end have it all work transparently for 
> the user.  That requirement is key, because it is what makes profiles in 
> ipython a powerful customization tool.  Initially I thought it couldn't 
> be solved if the config files were real python code, but that was just a 
> sign of my inexperience with the language. With a bit of care that's not 
> a problem, and removing the current config manager would vastly simplify 
> a lot of code.

I agree that this is quite feasible with python.  There may already 
exist an implementation somewhere on which to model it.

> 
>> I guess that leads to the bigger question of how to reformulate 
>> IPython given the lessons-learned.  I am pretty satisfied with the 
>> current behavior, use it all of the time and in fact am now unable to 
>> use the default interactive environment!  Just looking quickly among 
>> the modules, has python been sufficiently patched to allow the removal 
>> of FlexCompleter?  Is DPyGetOpt really needed?  Python already has a 
>> fairly complete version of getopt, now that optparse has made it into 
>> the distribution.  For these, I realize these changes may introduce be 
>> backwards compatibility problems that would need to be resolved.  In 
>> all fairness, these are pretty minor issues.  I'll get back with you 
>> when I've had a chance to look through the code.  BTW, which is the 
>> appropriate forum for discussing IPython development?  scipy-dev/user? 
>> Is there a thread on this; if not, should one be started? :)
> 
> 
> Well, I just started it on ipython-dev with this reply to your mail :)  
> If you're not subscribed to the list, don't be afraid, it's very low 
> traffic.
> 
> Your other comments are on the nose, and basically mimic what I've been 
> thinking lately.  I'm leaning strongly towards making 2.3 a requirement 
> for ipython when the internal cleanup starts.  The reasons:
> 
> - DPyGetOpt: replace it with getoptparse.
> - Logger: replace it with the standard logging module (I think).
> - FlexCompleter: that's the new rlcompleter.py (it was my patch :)
> - Itpl: remove altogether, we can do without.
> 
> These changes would mean yanking a fair bit of code out of ipython, 
> which is always a good thing in my book.  Since this rewrite will likely 
> take some time, probably by the moment it's production-ready, most 
> people won't see the 2.3 requirement as a big burden.  And I'd still 
> keep the 0.5x branch around for critical fixes for users of py2.1.
> 
> If the 2.3 requirement is a big problem for many people, one could 
> always package getoptparse and logger from 2.3 along with ipython, I 
> suppose.  But I'll worry about that when the time comes.

I agree with the 2.3 requirement here.  I only occasionally refer to 2.2 
for debugging purposes when some odd or unexpected behavior occurs.  If 
there is a need to backport standard packages, a subdirectory could 
contain these modules.

> Basically the cleanup would entail:
> 
> - Cleaning ipmaker, with removing the current config handler as part of 
> this.
> 
> - Making Magic.py NOT be a mixin, but rather an attribute of the ipython 
> shell itself (self.MagicSystem = Magic() or somesuch).
> 
> - The 4 module removals mentioned above (requiring 2.3)
> 
> Please also take a look at the new_design document, which is where I try 
> to keep notes on this whole problem so everyone has access to them.  I 
> may even port this thread back to that document later.

Where can I find this document?  SF (seems to be down at the moment)?




-- 
Jeffery Collins (http://www.boulder.net/~jcollins)


From fperez at colorado.edu  Wed Oct  1 18:09:19 2003
From: fperez at colorado.edu (Fernando Perez)
Date: Wed, 01 Oct 2003 16:09:19 -0600
Subject: [IPython-dev] Re: ipython improvements
In-Reply-To: <3F7B4CD2.1090204@boulder.net>
References: <3F79CCCD.5070303@boulder.net> <3F79ECF2.8060901@colorado.edu>
	<3F7B4CD2.1090204@boulder.net>
Message-ID: <3F7B508F.8040702@colorado.edu>

Jeffery D. Collins wrote:
>>Please also take a look at the new_design document, which is where I try 
>>to keep notes on this whole problem so everyone has access to them.  I 
>>may even port this thread back to that document later.
> 
> 
> Where can I find this document?  SF (seems to be down at the moment)?

It's a pdf called new_design.pdf included with every distribution of ipython, 
in the doc/ directory.  So regardless of how you got ipyton (rpm, .tar.gz, 
CVS, etc) you should have that file already.  It gets put in the same 
directory as the manual.  Let me know if you have any problem with this.

And by the way, I don't use SF :)  Ipython is all hosted (including CVS) at 
scipy.org.

Cheers,

f


From hupp at cs.wisc.edu  Sat Oct  4 19:19:30 2003
From: hupp at cs.wisc.edu (Adam Hupp)
Date: Sat, 4 Oct 2003 18:19:30 -0500
Subject: [IPython-dev] Problem when sys.argv does not exist
Message-ID: <20031004231929.GA4669@upl.cs.wisc.edu>

I'm running into a problem embedding when sys.argv is not defined.
I'm running ipython from a C extension that does not set sys.argv.
The root of the problem is that DPyGetOpt unconditionally checks the
passed arguments against sys.argv.  With no sys.argv this raises an
exception and aborts the whole program.  Checked against 0.5.0.  The
following change to DPyGetOpt.py should fix it:

<               if args == sys.argv:
<                       args = sys.argv[1:]
---
>               if hasattr(sys, "argv"):
>                       if args == sys.argv:
>                               args = sys.argv[1:]

-Adam

From ralf at brainbot.com  Sun Oct  5 08:55:44 2003
From: ralf at brainbot.com (Ralf Schmitt)
Date: Sun, 05 Oct 2003 14:55:44 +0200
Subject: [IPython-dev] replacement for deep_reload.py
In-Reply-To: <3F7B2E9F.1050701@colorado.edu>
References: <3F7B1B45.5040907@brainbot.com> <3F7B2E9F.1050701@colorado.edu>
Message-ID: <3F8014D0.6070102@brainbot.com>

Fernando Perez wrote:

>
> Great!  Many thanks for this.  Please give it a bit more pounding, and 
> I'd encourage other users of dreload to also try it out.  The code is 
> definitely far simpler than the original dreload, but since I don't 
> understand that code too well, I'd like to tiptoe a bit on this 
> issue.  If it survives a bit of pounding and discussion, I'll 
> definitely be glad to put it in.

One of the things where it differ's from the old deep_reload is that 
when importing a submodule, say ipbug.vm, it will not reload 
ipbug/__init__.py. I've attached another version, which tries to do just 
that and I'm using that version currently without problems.
However I think there must be an even  less complicated version.  
Clearing sys.modules and 'reimporting' the module like in the following 
code seems to work ok.
====
import sys, bbutils.textproc.spellcheck
m=sys.modules.copy()
sys.modules.clear()
sys.modules['sys'] = sys
import bbutils.textproc.spellcheck
====

Currently I don't have the time to investigate this further, but in a 
week or two, I'll  have another look at this.

- Ralf

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: deep_reload.py
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20031005/36b660b3/attachment.ksh>

From fperez at colorado.edu  Mon Oct  6 13:28:00 2003
From: fperez at colorado.edu (Fernando Perez)
Date: Mon, 06 Oct 2003 11:28:00 -0600
Subject: [IPython-dev] Problem when sys.argv does not exist
In-Reply-To: <20031004231929.GA4669@upl.cs.wisc.edu>
References: <20031004231929.GA4669@upl.cs.wisc.edu>
Message-ID: <3F81A620.5050802@colorado.edu>

Adam Hupp wrote:
> I'm running into a problem embedding when sys.argv is not defined.
> I'm running ipython from a C extension that does not set sys.argv.
> The root of the problem is that DPyGetOpt unconditionally checks the
> passed arguments against sys.argv.  With no sys.argv this raises an
> exception and aborts the whole program.  Checked against 0.5.0.  The
> following change to DPyGetOpt.py should fix it:
> 
> <               if args == sys.argv:
> <                       args = sys.argv[1:]
> ---
> 
>>              if hasattr(sys, "argv"):
>>                      if args == sys.argv:
>>                              args = sys.argv[1:]

Cool!  Ipython _can_ run from within C extensions :)  A while back a user 
asked about it and said he had some problems, but I never heard back from him. 
  So I had no idea whether it worked or not.

I applied your change as:

                 if hasattr(sys, "argv") and args == sys.argv:
                     args = sys.argv[1:]

Given the short-circuiting behavior of 'and', this should be fully equivalent 
to your suggestion and slightly cheaper than a nested 'if'.

Thanks for the contribution, it's already in CVS.

Regards,

Fernando.


From fperez at colorado.edu  Mon Oct  6 13:35:29 2003
From: fperez at colorado.edu (Fernando Perez)
Date: Mon, 06 Oct 2003 11:35:29 -0600
Subject: [IPython-dev] replacement for deep_reload.py
In-Reply-To: <3F8014D0.6070102@brainbot.com>
References: <3F7B1B45.5040907@brainbot.com> <3F7B2E9F.1050701@colorado.edu>
	<3F8014D0.6070102@brainbot.com>
Message-ID: <3F81A7E1.5060806@colorado.edu>

Ralf Schmitt wrote:

>>Great!  Many thanks for this.  Please give it a bit more pounding, and 
>>I'd encourage other users of dreload to also try it out.  The code is 
>>definitely far simpler than the original dreload, but since I don't 
>>understand that code too well, I'd like to tiptoe a bit on this 
>>issue.  If it survives a bit of pounding and discussion, I'll 
>>definitely be glad to put it in.
> 
> 
> One of the things where it differ's from the old deep_reload is that 
> when importing a submodule, say ipbug.vm, it will not reload 
> ipbug/__init__.py. I've attached another version, which tries to do just 
> that and I'm using that version currently without problems.
> However I think there must be an even  less complicated version.  
> Clearing sys.modules and 'reimporting' the module like in the following 
> code seems to work ok.
> ====
> import sys, bbutils.textproc.spellcheck
> m=sys.modules.copy()
> sys.modules.clear()
> sys.modules['sys'] = sys
> import bbutils.textproc.spellcheck
> ====
> 
> Currently I don't have the time to investigate this further, but in a 
> week or two, I'll  have another look at this.

Great.  And don't worry: I have a trip this week which will shut me off from 
ipython for at least the rest of the week.  So take your time, and 0.5.1 will 
have quite a few nice goodies in it :)

Regards,

Fernando.


From anders at bruun-olsen.net  Tue Oct  7 04:28:38 2003
From: anders at bruun-olsen.net (Anders Bruun Olsen)
Date: Tue, 7 Oct 2003 10:28:38 +0200
Subject: [IPython-dev] Bugtracker?
Message-ID: <20031007082838.GE28465@elmer.skumleren.net>

Hi,

I filed a bug in the bugtracker almost 2 weeks ago but nobody seems to
have noticed it and since there seems to be no activity in the
bugtracker I just want to checkup on the state of affairs here. Have I
filed a bug the wrong place or what?

-- 
Anders
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS/O d--@ s:+ a-- C++ UL+++$ P++ L+++ E- W+ N(+) o K? w O-- M- V
PS+ PE@ Y+ PGP+ t 5 X R+ tv+ b++ DI+++ D+ G e- h !r y?
------END GEEK CODE BLOCK------
PGPKey: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x8BFECB41

From fperez at colorado.edu  Tue Oct  7 11:34:22 2003
From: fperez at colorado.edu (Fernando Perez)
Date: Tue, 07 Oct 2003 09:34:22 -0600
Subject: [IPython-dev] Bugtracker?
In-Reply-To: <20031007082838.GE28465@elmer.skumleren.net>
References: <20031007082838.GE28465@elmer.skumleren.net>
Message-ID: <3F82DCFE.9040607@colorado.edu>

Anders Bruun Olsen wrote:
> Hi,
> 
> I filed a bug in the bugtracker almost 2 weeks ago but nobody seems to
> have noticed it and since there seems to be no activity in the
> bugtracker I just want to checkup on the state of affairs here. Have I
> filed a bug the wrong place or what?
> 
  My bad.  I just checked it and assigned it to myself, I'll look into it as 
soon as I have a chance (next week, I'm leaving for a work trip soon).

You did the right thing with the bug tracker.  It's just that I haven't 
figured out how to make it email me when new bugs are filed.  I thought it was 
done, but obviously not.  I'll contact the bugtracker admin to find out how to 
do that.

Best regards,

Fernando


From anders at bruun-olsen.net  Tue Oct  7 15:18:05 2003
From: anders at bruun-olsen.net (Anders Bruun Olsen)
Date: Tue, 7 Oct 2003 21:18:05 +0200
Subject: [IPython-dev] Bugtracker?
In-Reply-To: <3F82DCFE.9040607@colorado.edu>
References: <20031007082838.GE28465@elmer.skumleren.net>
	<3F82DCFE.9040607@colorado.edu>
Message-ID: <20031007191809.GA15419@elmer.skumleren.net>

On Tue, Oct 07, 2003 at 09:34:22AM -0600, Fernando Perez wrote:
> >I filed a bug in the bugtracker almost 2 weeks ago but nobody seems to
> >have noticed it and since there seems to be no activity in the
> >bugtracker I just want to checkup on the state of affairs here. Have I
> >filed a bug the wrong place or what?
>  My bad.  I just checked it and assigned it to myself, I'll look into it as 
> soon as I have a chance (next week, I'm leaving for a work trip soon).
> You did the right thing with the bug tracker.  It's just that I haven't 
> figured out how to make it email me when new bugs are filed.  I thought it 
> was done, but obviously not.  I'll contact the bugtracker admin to find out 
> how to do that.

Twisted (twistedmatrix.com) uses Roundup also and it sends out mails, so
they Twisted admins might be able to help you if the Roundup admins are
unresponsive.

-- 
Anders
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS/O d--@ s:+ a-- C++ UL+++$ P++ L+++ E- W+ N(+) o K? w O-- M- V
PS+ PE@ Y+ PGP+ t 5 X R+ tv+ b++ DI+++ D+ G e- h !r y?
------END GEEK CODE BLOCK------
PGPKey: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x8BFECB41

From fperez at colorado.edu  Tue Oct  7 17:16:07 2003
From: fperez at colorado.edu (Fernando Perez)
Date: Tue, 07 Oct 2003 15:16:07 -0600
Subject: [IPython-dev] Bugtracker?
In-Reply-To: <20031007191809.GA15419@elmer.skumleren.net>
References: <20031007082838.GE28465@elmer.skumleren.net>
	<3F82DCFE.9040607@colorado.edu> <20031007191809.GA15419@elmer.skumleren.net>
Message-ID: <3F832D17.6030206@colorado.edu>

Anders Bruun Olsen wrote:

> Twisted (twistedmatrix.com) uses Roundup also and it sends out mails, so
> they Twisted admins might be able to help you if the Roundup admins are
> unresponsive.

No problem, Joe Cooper (the Scipy admin) just told me he'd enable email 
notifications later.  I simply had never asked him about it.

So this incident shouldn't happen again.  Thanks for reporting (both, the 
ipython bug and the email thingie),

f