From ipython at ml.schieke.net  Mon Aug  2 19:00:59 2004
From: ipython at ml.schieke.net (Jaco Schieke)
Date: Tue, 03 Aug 2004 01:00:59 +0200
Subject: [IPython-dev] Tab-completion of data/properties and win32com fix
	for IPython
Message-ID: <410EC7AB.3040806@ml.schieke.net>

I had some issues trying to get tab-completion working with COM objects' 
properties in Window$ using the win32com module & IPython and have been 
able to get things working with help from the python-win32 list.  I'm 
reposting the solution on here for inclusion if it makes sense into 
IPython.  The best place to fix this would however be in python's 
rlcompleter.py and not IPython's FlexCompleter.py, but previous requests 
to fix rlcompleter.py have gone unheeded - and this one's got an even 
lower chance as it is win32 specific.

(Patch follows at the end of email)

Regards,

Jaco Schieke

Thread from python-win32 follows:
-------- Original Message --------
Subject: 	Re: [python-win32] Tab-completion of data/properties and win32com
Date: 	Tue, 03 Aug 2004 00:37:49 +0200

	

	

	



Mark Hammond wrote:

>>I am trying to use IPython interactively to do some win32com
>>programming
>>and want to use the completion feature.  The problem is that
>>tab-completion only completes the object methods, and not the
>>associated
>>properties, for instance:
>>    
>>
>
>IDLE doesn't have special knowledge of COM objects.  Pythonwin does though -
>it shows the Excel properties in that example.
>
>You could try and find the place in IDLE where the list is built, and copy
>the code from pywin\scintilla\view.py - look for the comment "# The object
>may be a COM object with typelib support - lets see if we can get its
>props."  Then submit a patch for IDLE, and it may well be accepted.
>
>Another cute idea would be to hack makepy to use "properties" for 2.3 and
>later - this should make them look more "real" - but IDLE may not support
>that yet (and it would be non-trivial anyway)
>
>Mark.
>
>
>
>  
>
Mark,

Thanks for the advice.  We misunderstood each other, but I got it 
working.  What I ended up doing is to patch rlcompleter.py to allow it 
to get all the attributes of the COM object (as is done in pywin's 
view.py).  I'm currently using IPython (ipython.scipy.org) which is 
using a modified rlcompleter.py due to limitations of the exising 
rlcompleter.py implementation.  I'll submit a patch to IPython and see 
if it gets accepted there.

Would there be any point in submitting a patch against 2.3's 
rlcompleter?  This is a win32 specific feature, and if I understand you 
correctly, it can be solved in other ways by fixing makepy.

Regards & tx for the assistance,

Jaco



Index: ipython/IPython/FlexCompleter.py
===================================================================
RCS file: /home/cvsroot/world/ipython/IPython/FlexCompleter.py,v
retrieving revision 1.3
diff -r1.3 FlexCompleter.py
163d162
< 
170a170,175
>         def list2dict(l):
>             ret={}
>             for i in l:
> 	        ret[i] = None
>             return ret
> 
185a191,194
>         matches_dict = {}
>         matches_dict.update(list2dict(matches))
>         matches = matches_dict.keys()
>         matches.sort()
188a198
> 
192a203,208
>     try:
>         ret=ret+klass._prop_map_get_.keys()
>         ret=ret+klass._prop_map_put_.keys()
>         pass
>     except AttributeError:
>         pass





From Fernando.Perez at colorado.edu  Mon Aug  2 19:09:52 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 02 Aug 2004 17:09:52 -0600
Subject: [IPython-dev] Tab-completion of data/properties and win32com
	fix	for IPython
In-Reply-To: <410EC7AB.3040806@ml.schieke.net>
References: <410EC7AB.3040806@ml.schieke.net>
Message-ID: <410EC9C0.9030606@colorado.edu>

Jaco Schieke wrote:
> I had some issues trying to get tab-completion working with COM objects' 
> properties in Window$ using the win32com module & IPython and have been 
> able to get things working with help from the python-win32 list.  I'm 
> reposting the solution on here for inclusion if it makes sense into 
> IPython.  The best place to fix this would however be in python's 
> rlcompleter.py and not IPython's FlexCompleter.py, but previous requests 
> to fix rlcompleter.py have gone unheeded - and this one's got an even 
> lower chance as it is win32 specific.

Well, unfortunately there's no point in accepting your patch: FlexCompleter is 
slated for removal from ipython soon.  FlexCompleter was really just a copy of 
rlcompleter from python 2.1, back when rlcompleter did not support arbitrary 
namespaces.  But my patch to rlcompleter got accepted into mainline, so there 
really is no need anymore for FlexCompleter, which is now a copy of the 
official rlcompleter.

It's actually just a matter of my having been lazy that it's still around, 
since I should have removed it a while back (when 2.2 became a requirement for 
ipython -- I don't really remember when this happened).

Sorry to break these bad news to you.  As you say, the proper place for this 
fix is really in rlcompleter proper, so perhaps you could continue pounding on 
their door until at least you get a response.  That's how I got the patch 
which made FlexCompleter accepted into the official rlcompleter.

Regards,

f



From ipython at ml.schieke.net  Mon Aug  2 19:37:43 2004
From: ipython at ml.schieke.net (Jaco Schieke)
Date: Tue, 03 Aug 2004 01:37:43 +0200
Subject: [IPython-dev] Tab-completion of data/properties and win32com
	fix	for IPython
In-Reply-To: <410EC9C0.9030606@colorado.edu>
References: <410EC7AB.3040806@ml.schieke.net> <410EC9C0.9030606@colorado.edu>
Message-ID: <410ED047.7030500@ml.schieke.net>

Fernando,

I see though that the regexp line:
     m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text)
in FlexCompleter is not incorporated into rlcompleter for python 2.3.

The original reason for changing this from the vanilla rlcompleter was 
to allow for completions on arrays and other funny things, e.g. 
"".<tab>.  I was involved in this but took a 2 year leave of absence 
;-).  I read some comments (after posting here) on sourceforge on the 
original patch submitted for rlcompleter though and there are issues 
with arbitrary code execution 
(https://sourceforge.net/tracker/?func=detail&aid=547176&group_id=5470&atid=305470). 


Having said all of this ... I not quite sure what the solution is.  For 
the while being, I like typing as little as possible & living on the 
edge - so I'll stick with FlexCompleter.

Regards,

Jaco

Fernando Perez wrote:

> Jaco Schieke wrote:
>
>> I had some issues trying to get tab-completion working with COM 
>> objects' properties in Window$ using the win32com module & IPython 
>> and have been able to get things working with help from the 
>> python-win32 list.  I'm reposting the solution on here for inclusion 
>> if it makes sense into IPython.  The best place to fix this would 
>> however be in python's rlcompleter.py and not IPython's 
>> FlexCompleter.py, but previous requests to fix rlcompleter.py have 
>> gone unheeded - and this one's got an even lower chance as it is 
>> win32 specific.
>
>
> Well, unfortunately there's no point in accepting your patch: 
> FlexCompleter is slated for removal from ipython soon.  FlexCompleter 
> was really just a copy of rlcompleter from python 2.1, back when 
> rlcompleter did not support arbitrary namespaces.  But my patch to 
> rlcompleter got accepted into mainline, so there really is no need 
> anymore for FlexCompleter, which is now a copy of the official 
> rlcompleter.
>
> It's actually just a matter of my having been lazy that it's still 
> around, since I should have removed it a while back (when 2.2 became a 
> requirement for ipython -- I don't really remember when this happened).
>
> Sorry to break these bad news to you.  As you say, the proper place 
> for this fix is really in rlcompleter proper, so perhaps you could 
> continue pounding on their door until at least you get a response.  
> That's how I got the patch which made FlexCompleter accepted into the 
> official rlcompleter.
>
> Regards,
>
> f
>
>



From Fernando.Perez at colorado.edu  Tue Aug  3 00:10:30 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 02 Aug 2004 22:10:30 -0600
Subject: [IPython-dev] Tab-completion of data/properties and win32com
	fix	for IPython
In-Reply-To: <410ED047.7030500@ml.schieke.net>
References: <410EC7AB.3040806@ml.schieke.net> <410EC9C0.9030606@colorado.edu>
	<410ED047.7030500@ml.schieke.net>
Message-ID: <410F1036.5080803@colorado.edu>

Jaco Schieke wrote:
> Fernando,
> 
> I see though that the regexp line:
>      m = re.match(r"(\S+(\.\w+)*)\.(\w*)$", text)
> in FlexCompleter is not incorporated into rlcompleter for python 2.3.
> 
> The original reason for changing this from the vanilla rlcompleter was 
> to allow for completions on arrays and other funny things, e.g. 
> "".<tab>.  I was involved in this but took a 2 year leave of absence 
> ;-).  I read some comments (after posting here) on sourceforge on the 
> original patch submitted for rlcompleter though and there are issues 
> with arbitrary code execution 
> (https://sourceforge.net/tracker/?func=detail&aid=547176&group_id=5470&atid=305470). 

Thanks for pointing this to me.

Well, I guess then my solution will then be to keep FlexCompleter around, but 
I'll strip it to the bare essentials, overriding only the necessary methods 
from rlcompleter.

Cheers,

f



From Fernando.Perez at colorado.edu  Tue Aug  3 00:20:05 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 02 Aug 2004 22:20:05 -0600
Subject: [IPython-dev] Tab-completion of data/properties and win32com
	fix	for IPython
In-Reply-To: <410EC7AB.3040806@ml.schieke.net>
References: <410EC7AB.3040806@ml.schieke.net>
Message-ID: <410F1275.5070505@colorado.edu>

Jaco Schieke wrote:

> (Patch follows at the end of email)

I have some questions about this.  I'll also make some general comments about 
patches in a separate mail, please heed them for the future.

Since I may keep FlexCompleter around, I want to clarify a few things about 
your patch.

>>        def list2dict(l):
>>            ret={}
>>            for i in l:
>>	        ret[i] = None
>>            return ret
>>
> 
> 185a191,194
> 
>>        matches_dict = {}
>>        matches_dict.update(list2dict(matches))
>>        matches = matches_dict.keys()
>>        matches.sort()

What exactly is all of this for?  If you want to return the matches sorted, a 
simple

matches.sort()

before returning will do just fine, no?  Maybe I'm just tired now, but the 
list2dict contortions seem unnecessary to me.

Besides, I think readline sorts itself always, since I always see the proposed 
completions come out sorted.  Could you clarify exactly what this is doing? 
As far as I can tell, all of it is completely unnecessary.

> 188a198
> 
> 192a203,208
> 
>>    try:
>>        ret=ret+klass._prop_map_get_.keys()
>>        ret=ret+klass._prop_map_put_.keys()
>>        pass
>>    except AttributeError:
>>        pass

This could be acceptable, I imagine, for the benefit of win32 users.  But not 
in this form: all Win-specific code should be isolated at definition time, I 
have no intention of making Unix users pay at runtime for windows-only stuff. 
  So you'd need to write instead something like:

def get_class_members():
     --- current code ---

if on_windows:
   # redefine with win-specific hacks
   def get_class_members():
       --- your windows code ---

or something similar.  Be mindful of the details about cygwin vs non-cygwin 
environments when writing windows-specific code, please.  Also make sure you 
test it very well, since I can't test it myself.

Best,

f



From Fernando.Perez at colorado.edu  Tue Aug  3 01:56:44 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 02 Aug 2004 23:56:44 -0600
Subject: [IPython-dev] A simple patch policy for IPython contributions
Message-ID: <410F291C.901@colorado.edu>

Hi everyone,

as ipython's userbase grows, more and more people are sending me patches.  I
greatly appreciate this, and I hope the process continues.  However, more
patches mean more alien code for me to study, so I want to specify a minimal
'policy' on patch creation to try and make the process as smooth as possible.

This is just a set of simple guidelines aimed at ensuring that we all spend
our limited time on the code itself and not struggling with patch management.
I've said some of this before, but I want it in a separate message for easy
reference and archiving.

I don't want to sound patronizing or to anger anyone, it's just a matter of
making the process as simple and efficient as possible to all involved (with
my time counting double or triple, because I am the bottleneck).  Any comments
on this are very much welcome (except for any discussion on #1).

Ipython's minimal patch policy:

1. ***NEVER, EVER, EVER*** USE HARD TABS.  USE 4 SPACES FOR INDENTATION.

    I want to make sure that this one is as loudly and clearly spelled out as
    possible.  It is absolutely not open to discussion.  I've already had bugs
    creep in because of patches with hard tabs in them, so in the future I'll
    simply reject any patch with a single tab in it.

    IPython is indented (following python.org policy) with 4 spaces per level.

2. Please send patches as attachments, with a .diff extension.

    This way, there are no issues with manual copy/pasting from a mail client's
    window.  And they load them into XEmacs with syntax highlighting.

3. Make your patches against current CVS.

4. Use -u in diff, I find unified patches easier to visually parse.

That's it.

For those interested in more info about contributing to an open source
project, Here are some links I've found.  They are probably no better or worse
than other similar documents:

http://www.linux.com/howtos/Software-Release-Practice-HOWTO/index.shtml
http://www.kegel.com/academy/opensource.html

Best regards, and thanks again to all who have contributed in the past.

Fernando.



From ipython at ml.schieke.net  Tue Aug  3 19:22:01 2004
From: ipython at ml.schieke.net (Jaco Schieke)
Date: Wed, 04 Aug 2004 01:22:01 +0200
Subject: [IPython-dev] Tab-completion of data/properties and win32com
	fix	for IPython
In-Reply-To: <410F1275.5070505@colorado.edu>
References: <410EC7AB.3040806@ml.schieke.net> <410F1275.5070505@colorado.edu>
Message-ID: <41101E19.50803@ml.schieke.net>

Fernando,

Some clarifications:
1. The dictionary hack is simply to keep "matches" unique.  It took me a 
while to realize this, but pyreadline actually expects an unique array 
(see lib\site-packages\readline\pyreadline.py(788): if r and r not in 
completions:), otherwise it stops reading from the matches returned and 
simply exits.  This is not win32 specific and should be a usefull 
addition should be incorporated into rlcompleter.py.  An easy way to 
test this is to add matches.sort() just before you return matches in 
FlexCompleter.py.  Then try [].<tab> ... the results are disappointing.  
The reason for this is obviously that you first dir() the object and 
then the base classes, so you'd not notice this normally.  However 
sorting the array (or adding COM attributes after you dir()'ed the base 
classes) causes the issues, as duplicates appear very quickly.  It took 
me quite some time to figure this out. - so in short: matches must be 
unique'd and list2dict is the easiest way I know (I may be rusty.)  Let 
me know if you concur then I'll submit a bug report for rlcompleter.py.

2. I've rewritten the function to check for sys.platform and only run 
win32 hacks if on the right platform - patch follows..

Regards,

Jaco

Index: ipython/IPython/FlexCompleter.py
===================================================================
RCS file: /home/cvsroot/world/ipython/IPython/FlexCompleter.py,v
retrieving revision 1.3
diff -B -r1.3 FlexCompleter.py
78a79
 > import sys
156c157
<         instances, class members are are also considered.)
---
 >         instances, class members are are also considered.)  A unique 
array is returned
162c163,167
<         import re
---
 >         def list2dict(l):
 >             ret={}
 >             for i in l:
 >                 ret[i] = None
 >             return ret
163a169
 >         import re
185a192,196
 >
 >         matches_dict = {}
 >         matches_dict.update(list2dict(matches)) # ensures uniqueness
 >         matches = matches_dict.keys()
 >         matches.sort()
193a206,232
 >
 > if sys.platform == "win32":
 >     def get_class_members(klass):
 >     """Obtains class and base class members through dir()
 >
 >     Additional query is also performed in the case that the class is a
 >     COM object instantiated by the win32 module of Mark Hammond,
 >     as dir() does not reveal the properties associated with
 >     the COM object. 
 >    
 >     """
 >     ret = dir(klass)
 >
 >     if hasattr(klass,'__bases__'):
 >         for base in klass.__bases__:
 >         ret = ret + get_class_members(base)
 >    
 >     #Win32 specific code follows
 >     try:    # in case klass is a COM object from win32
 >         ret=ret+klass._prop_map_get_.keys()
 >         ret=ret+klass._prop_map_put_.keys()
 >         pass
 >     except AttributeError:
 >         pass
 >     #End of Win32 specific code
 >    
 >     return ret


Fernando Perez wrote:

> Jaco Schieke wrote:
>
>> (Patch follows at the end of email)
>
>
> I have some questions about this.  I'll also make some general 
> comments about patches in a separate mail, please heed them for the 
> future.
>
> Since I may keep FlexCompleter around, I want to clarify a few things 
> about your patch.
>
>>>        def list2dict(l):
>>>            ret={}
>>>            for i in l:
>>>             ret[i] = None
>>>            return ret
>>>
>>
>> 185a191,194
>>
>>>        matches_dict = {}
>>>        matches_dict.update(list2dict(matches))
>>>        matches = matches_dict.keys()
>>>        matches.sort()
>>
>
> What exactly is all of this for?  If you want to return the matches 
> sorted, a simple
>
> matches.sort()
>
> before returning will do just fine, no?  Maybe I'm just tired now, but 
> the list2dict contortions seem unnecessary to me.
>
> Besides, I think readline sorts itself always, since I always see the 
> proposed completions come out sorted.  Could you clarify exactly what 
> this is doing? As far as I can tell, all of it is completely unnecessary.
>
>> 188a198
>>
>> 192a203,208
>>
>>>    try:
>>>        ret=ret+klass._prop_map_get_.keys()
>>>        ret=ret+klass._prop_map_put_.keys()
>>>        pass
>>>    except AttributeError:
>>>        pass
>>
>
> This could be acceptable, I imagine, for the benefit of win32 users.  
> But not in this form: all Win-specific code should be isolated at 
> definition time, I have no intention of making Unix users pay at 
> runtime for windows-only stuff.  So you'd need to write instead 
> something like:
>
> def get_class_members():
>     --- current code ---
>
> if on_windows:
>   # redefine with win-specific hacks
>   def get_class_members():
>       --- your windows code ---
>
> or something similar.  Be mindful of the details about cygwin vs 
> non-cygwin environments when writing windows-specific code, please.  
> Also make sure you test it very well, since I can't test it myself.
>
> Best,
>
> f
>
>
>



From Fernando.Perez at colorado.edu  Tue Aug  3 20:07:26 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Tue, 03 Aug 2004 18:07:26 -0600
Subject: [IPython-dev] Tab-completion of data/properties and win32com
	fix	for IPython
In-Reply-To: <41101E19.50803@ml.schieke.net>
References: <410EC7AB.3040806@ml.schieke.net> <410F1275.5070505@colorado.edu>
	<41101E19.50803@ml.schieke.net>
Message-ID: <411028BE.2020909@colorado.edu>

Hi Jaco,

Jaco Schieke wrote:

> Some clarifications:
> 1. The dictionary hack is simply to keep "matches" unique.  It took me a 
> while to realize this, but pyreadline actually expects an unique array 
> (see lib\site-packages\readline\pyreadline.py(788): if r and r not in 
> completions:), otherwise it stops reading from the matches returned and 
> simply exits.  This is not win32 specific and should be a usefull 
> addition should be incorporated into rlcompleter.py.  An easy way to 
> test this is to add matches.sort() just before you return matches in 
> FlexCompleter.py.  Then try [].<tab> ... the results are disappointing.  

Mmh, I just tried this and I don't see any problem at all, at least not under 
Unix.  If this is something which is needed by pyreadline, then that's where 
the patch should be directed.  But I've never seen Unix readline needing any 
of this, so I don't want to add unnecessary functionality.

Sorry, but until you can show me a case where the Unix readline has problems 
with this setup, I'd rather not include this patch (and I suspect a similar 
answer may come from the rlcompleter crowd).  Obviously if you do have a usage 
case which shows how this is necessary on all platforms, then I'll gladly 
include it (and eventually it should work its way towards rlcompleter, where 
it probably really belongs; after all, FlexCompleter shouldn't even exist if 
rlcompleter had all we want).

But if the patch belongs in pyreadline (win32), then that's where it should 
go, not in ipython.

> 2. I've rewritten the function to check for sys.platform and only run 
> win32 hacks if on the right platform - patch follows..

This looks fine.  I'll wait until we settle the whole thing before applying 
the patch: if it's only going to be the win32 stuff, then I might ask you to 
resend me a standalone one so I don't accidentally mess things up applying and 
removing unneeded parts.

Also, please heed my comments about patches which I sent yesterday; your patch 
from yesterday had hard tabs, and I really prefer attachments created with -u.

Thanks for your efforts, this will eventually go into 0.6.3

Best,

f



From gb at cs.unc.edu  Wed Aug  4 18:33:25 2004
From: gb at cs.unc.edu (Gary Bishop)
Date: Wed, 04 Aug 2004 18:33:25 EDT
Subject: [IPython-dev] Tab-completion of data/properties and win32com	fix
	for IPython
Message-ID: <200408042233.i74MXQah000796@fafnir.cs.unc.edu>

I agree with Fernando. If pyreadline is different from GNU readline we 
need to fix it. I have NEVER used GNU readline on Linux. I played with 
it a bit on cygwin but didn't really know that I was doing when writing it.

I read the manual and tried to implement what I thought it does. Let's 
make pyreadline work as much like the GNU readline as we can. I'll be 
happy to accept patches to it.

gb



From ipython at ml.schieke.net  Thu Aug  5 17:35:50 2004
From: ipython at ml.schieke.net (Jaco Schieke)
Date: Thu, 05 Aug 2004 23:35:50 +0200
Subject: [IPython-dev] Tab-completion of data/properties and win32com
	fix	for IPython
In-Reply-To: <200408042233.i74MXQah000796@fafnir.cs.unc.edu>
References: <200408042233.i74MXQah000796@fafnir.cs.unc.edu>
Message-ID: <4112A836.2080607@ml.schieke.net>

OK.  I need to convince you guys (any myself) first that there is a 
problem though. Here is the patch for a proof of concept... I think my 
previous description of the POC may have been ambiguous, as the change 
could have been implemted in more than one place in FlexCompleter.py.  
Having applied this to FlexCompleter.py (or even rlcompleter.py at the 
right place with a vanilla python shell), things like sys.<tab> fail on 
my system (dont have access to a proper linux system to test.)  Please 
apply and verify.

My diagnosis is that the error lies in pyreadline at 
\lib\site-packages\readline\pyreadline.py(788)_get_completions() with 
the line "if r and r not in completions:", since this will break as soon 
as a duplicate entry is detected.  But before we discuss this further, 
please verify that my POC is not due to a wacky configuration on my side.

Regards,

Jaco

Proof of concept for duplicate entry failure.  After applying, do in 
IPython: import sys; sys.<tab>

$ cvs -z3 diff -ut IPython/FlexCompleter.py
Index: IPython/FlexCompleter.py
===================================================================
RCS file: /home/cvsroot/world/ipython/IPython/FlexCompleter.py,v
retrieving revision 1.3
diff -u -t -r1.3 FlexCompleter.py
--- IPython/FlexCompleter.py    22 Aug 2003 16:04:33 -0000      1.3
+++ IPython/FlexCompleter.py    5 Aug 2004 21:48:17 -0000
@@ -183,6 +183,7 @@
         for word in words:
             if word[:n] == attr and word != "__builtins__":
                 matches.append("%s.%s" % (expr, word))
+        matches.sort()
         return matches
 
 def get_class_members(klass):


Gary Bishop wrote:

> I agree with Fernando. If pyreadline is different from GNU readline we 
> need to fix it. I have NEVER used GNU readline on Linux. I played with 
> it a bit on cygwin but didn't really know that I was doing when 
> writing it.
>
> I read the manual and tried to implement what I thought it does. Let's 
> make pyreadline work as much like the GNU readline as we can. I'll be 
> happy to accept patches to it.
>
> gb
>
>



From Fernando.Perez at colorado.edu  Thu Aug  5 17:59:20 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 05 Aug 2004 15:59:20 -0600
Subject: [IPython-dev] Tab-completion of data/properties and win32com
	fix	for IPython
In-Reply-To: <4112A836.2080607@ml.schieke.net>
References: <200408042233.i74MXQah000796@fafnir.cs.unc.edu>
	<4112A836.2080607@ml.schieke.net>
Message-ID: <4112ADB8.1020006@colorado.edu>

Jaco Schieke wrote:
> OK.  I need to convince you guys (any myself) first that there is a 
> problem though. Here is the patch for a proof of concept... I think my 
> previous description of the POC may have been ambiguous, as the change 
> could have been implemted in more than one place in FlexCompleter.py.  
> Having applied this to FlexCompleter.py (or even rlcompleter.py at the 
> right place with a vanilla python shell), things like sys.<tab> fail on 
> my system (dont have access to a proper linux system to test.)  Please 
> apply and verify.

Works for me here (Linux).  After applying the matches.sort() patch:

In [1]: import sys

In [2]: sys.
sys._getframe              sys.exit                   sys.path_hooks
sys.api_version            sys.exitfunc               sys.path_importer_cache
sys.argv                   sys.getcheckinterval       sys.platform
sys.builtin_module_names   sys.getdefaultencoding     sys.prefix
sys.byteorder              sys.getdlopenflags         sys.setcheckinterval
sys.call_tracing           sys.getfilesystemencoding  sys.setdlopenflags
sys.callstats              sys.getrecursionlimit      sys.setprofile
sys.copyright              sys.getrefcount            sys.setrecursionlimit
sys.displayhook            sys.hexversion             sys.settrace
sys.exc_clear              sys.ipcompleter            sys.stderr
sys.exc_info               sys.maxint                 sys.stdin
sys.exc_type               sys.maxunicode             sys.stdout
sys.excepthook             sys.meta_path              sys.version
sys.exec_prefix            sys.modules                sys.version_info
sys.executable             sys.path                   sys.warnoptions


So if this is indeed a problem under WinXP, the ball would be in Gary's court, 
I'm afraid.

Best,

f



From ipython at ml.schieke.net  Thu Aug  5 18:23:03 2004
From: ipython at ml.schieke.net (Jaco Schieke)
Date: Fri, 06 Aug 2004 00:23:03 +0200
Subject: [IPython-dev] Tab-completion of data/properties and win32com
	fix	for IPython
In-Reply-To: <4112ADB8.1020006@colorado.edu>
References: <200408042233.i74MXQah000796@fafnir.cs.unc.edu>
	<4112A836.2080607@ml.schieke.net> <4112ADB8.1020006@colorado.edu>
Message-ID: <4112B347.50701@ml.schieke.net>

Fernando Perez wrote:

> Jaco Schieke wrote:
>
>> OK.  I need to convince you guys (any myself) first that there is a 
>> problem though. Here is the patch for a proof of concept... I think 
>> my previous description of the POC may have been ambiguous, as the 
>> change could have been implemted in more than one place in 
>> FlexCompleter.py.  Having applied this to FlexCompleter.py (or even 
>> rlcompleter.py at the right place with a vanilla python shell), 
>> things like sys.<tab> fail on my system (dont have access to a proper 
>> linux system to test.)  Please apply and verify.
>
>
> Works for me here (Linux).  After applying the matches.sort() patch:
>
You're right.  This is windows specific - due to the usage of 
pyreadline.  I'll rework the original COM patch for FlexCompleter.py (if 
you're still up for applying it) and see what Gary has to say on pyreadline.

Regards,

Jaco



From Fernando.Perez at colorado.edu  Thu Aug  5 18:30:00 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 05 Aug 2004 16:30:00 -0600
Subject: [IPython-dev] Tab-completion of data/properties and win32com
	fix	for IPython
In-Reply-To: <4112B347.50701@ml.schieke.net>
References: <200408042233.i74MXQah000796@fafnir.cs.unc.edu>
	<4112A836.2080607@ml.schieke.net> <4112ADB8.1020006@colorado.edu>
	<4112B347.50701@ml.schieke.net>
Message-ID: <4112B4E8.3060605@colorado.edu>

Jaco Schieke wrote:
> Fernando Perez wrote:
>>
>>Works for me here (Linux).  After applying the matches.sort() patch:
>>
> 
> You're right.  This is windows specific - due to the usage of 
> pyreadline.  I'll rework the original COM patch for FlexCompleter.py (if 
> you're still up for applying it) and see what Gary has to say on pyreadline.

Sure, that one, I have nothing against (since it doesn't affect Unix in any 
way :).  I'd like to hear the opinion of others more experienced with Windows, 
though.  I have zero basis for judging whether it's a good idea or not.

But in principle, unless someone objects on technical grounds (with an 
explanation), I can put that one in.

Thanks for heeding my call for -u patches, but please send them as 
attachments.  It's much easier for me to save a standalone diff file than to 
mess with either cut/paste (which always has problems with some email clients, 
and I need to use several different ones), or saving the whole email and then 
retrimming it.

For various reasons I use multiple mail clients, so I have no integrated tools 
for handling patches directly embedded into the messages.  Having them as 
attachments with a proper filename makes the process much smoother.  I know 
there are others who prefer inlined patches, this depends a lot on how your 
toolchain is setup.  Sorry to insist, but given that for contributors it's a 
small effort, but for me that extra time is multiplied by all patches sent, I 
really must.

Best,

f



From Fernando.Perez at colorado.edu  Sun Aug 22 18:56:38 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Sun, 22 Aug 2004 16:56:38 -0600
Subject: [IPython-dev] ipython, gtk & matplotlib support;
	end of current ipython development
Message-ID: <412924A6.1030003@colorado.edu>

Hi everyone,

sorry for the cross post to those of you who are on all these lists, but since 
this will affect ipython's future quite a bit, I want a significant heads-up 
to be seen by all potentially affected.

1. PyGTK & matplotlib
---------------------

Thanks to Antoon Pardon and John Hunter, ipython has nearly ready full support 
for interactive matplotlib with all backends.  In this process, we've also 
added GTK threading support, so you can now use ipython for pygtk development.

This code is now in IPython's CVS, and the matplotlib features require 
matplotlib CVS (for matplotlib use only; matplotlib has NOT become a general 
ipython requirement).  So those of you willing to bleed a little can use it, 
and now is your opportunity to let us know of any problems you find.

Our solution is simpler, but more limited in scope, than scipy's gui_thread. 
However it currently does NOT work with the WX backends, only with Tk and GTK 
(-AGG or not).  Help from any WX guru is welcome, the place to look is at the 
end of IPython/Shell.py.  I hope that we can find, at least in our more 
limited context, a working solution for WX which doesn't require all the 
complexity of gui_thread.

In order to use this, do an 'ipython -upgrade' after a cvs update; this will 
get the necessary support files into ~/.ipython.  You will then need to use 
the new threading options; copying from the docs:

  -gthread, -mpthread: these are special options, only one of which can be
  given, and which can ONLY be given as the FIRST option passed to IPython
  (they will have no effect in any other position).

  If -gthread is given, IPython starts running a separate thread for GTK
  operation, so that pyGTK-based programs can open GUIs without blocking
  IPython.

  The -mpthread option adds special support for the matplotlib library
  (http://matplotlib.sourceforge.net), allowing interactive usage of the GTK
  and WX backends.  This also modifies the @run command to correctly execute
  any matplotlib-based script which calls show() at the end, without blocking.

The most convenient way to use this is the new pylab profile, which should be 
invoked as follows (aliasing this in your shell may be a good idea):

	$ ipython -mpthread -p pylab

pylab will honor your choice of matplotlib backend, though currently it will 
revert (with a warning) WX to TkAgg, since WX is broken.  This will go away 
once we figure out the WX problems.


2. IPython's future
-------------------

Once this support for matplotlib is working to satisfaction, it will mean the 
end of the line for any more feature-related changes to ipython for quite a 
while.  Once this is reasonably shaken (I hope with at most one more release 
beyond 0.6.3, which will officially include this), I plan on beginning the 
long-awaited internal cleanup of ipython.

Given my very limited time, this will mean essentially ZERO new features on 
ipython for quite a while.  It will also mean that the new ipython will:

- require python 2.3: I want to deprecate as much redundant code as I can from 
the ipython distribution.  I'll use optparse and any other new module from the 
stdlib which can help shrink ipython.

- break backwards compatibility in many areas.  In particular, the ipythonrc 
files will become true python files.

- the internal class structure of ipython will drastically change.  If you 
have code which uses ipython via IPython.Shell, you should be fine, as I'll 
try to keep that API stable.  If you've been poking your dirty fingers into 
iplib or ipmaker directly, expect things to break badly, and don't even think 
about complaining :)

Hopefully once this is over, it will mean having a much cleaner ipython, with 
an easy path for including it into GUI shells (such as pycrust), and a sane 
internal code structure.  As the new design shakes down, we'll eventually have 
an ipython 1.0 at last ;)

Because of these changes coming down the pipe, if you have any further patches 
or changes for the current ipython which you'd like to see included, please 
send them NOW to me.  Once I shift gears to the cleanup project, I'll 
unfortunately have to drop most changes not going in that direction, simply 
for lack of time.

I'd like to thank everybody who has contributed to ipython's development so 
far, and to encourage others to join in.  The cleanup should not be too hard, 
and it will open the door for having ipython as the interactive core for very 
high quality python-based environments in the future.

Best regards,

f



From vivainio at kolumbus.fi  Mon Aug 23 11:51:02 2004
From: vivainio at kolumbus.fi (Ville Vainio)
Date: Mon, 23 Aug 2004 18:51:02 +0300
Subject: [IPython-dev] 
	Re: [IPython-user] ipython, gtk & matplotlib support; 
	end of current ipython development
In-Reply-To: <412924A6.1030003@colorado.edu>
References: <412924A6.1030003@colorado.edu>
Message-ID: <200408231851.02741.vivainio@kolumbus.fi>

On Monday 23 August 2004 01:56, Fernando Perez wrote:

> far, and to encourage others to join in.  The cleanup should not be too
> hard, and it will open the door for having ipython as the interactive core
> for very high quality python-based environments in the future.

Great. Keep us posted on your ideas as you proceed :-).



From jdhunter at ace.bsd.uchicago.edu  Mon Aug 23 11:40:23 2004
From: jdhunter at ace.bsd.uchicago.edu (John Hunter)
Date: Mon, 23 Aug 2004 10:40:23 -0500
Subject: [IPython-dev] potential pylab bug
Message-ID: <m2oel1zwuw.fsf@mother.paradise.lost>


First of all, I'm loving the new interactive shell.  For the first
time in 2 years of using python every day, I'm trying to do almost
everything interactively from the shell, including updating my package
from ipython w/o restarting ipython!

  cd ~/some/dir
  !sudo python setup.py install
  reload somemodule

very nice....

But I *may* have found a bug.  I have a script that produces an
errobar plot which I'm calling with run in the pylab config .  The
script imports matplotlib.matlab and nothing more.  On a call to
errorbar, the ipython session hung.  I could not control-c it control
control-z it.  A ps revealed

    hunter:~/python/projects/cluster/examples> ps
    PID TTY          TIME CMD
    29177 pts/0    00:00:00 tcsh
    29381 pts/0    00:28:51 ipython
    29405 pts/0    00:00:00 sh
    29406 pts/0    00:00:00 gnuplot
    29411 pts/0    00:00:00 ps

    hunter:~/python/projects/cluster/examples> killall -9 ipython
    hunter:~/python/projects/cluster/examples> ps
    PID TTY          TIME CMD
    29177 pts/0    00:00:00 tcsh
    29413 pts/0    00:00:00 ps

Note that gnuplot was running, and it was killed when I killed
ipython!  Is it possible that there is some vestigial gnuplot support
creaping in?

Note I have not been able to reproduce this behavior, either with the
original script or a minimal errorbar script.  I only encountered it
in the middle of a fairly long interactive session.  I just wanted to
mention FYI in case it triggers something in your mind about how
gnuplot might creap in.  I haven't used gnuplot in years so I would be
surprised if it is creaping in through on of my modules, but I can't
say for sure.

JDH



From Fernando.Perez at colorado.edu  Thu Aug 26 23:17:00 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 26 Aug 2004 21:17:00 -0600
Subject: [IPython-dev] potential pylab bug
In-Reply-To: <m2oel1zwuw.fsf@mother.paradise.lost>
References: <m2oel1zwuw.fsf@mother.paradise.lost>
Message-ID: <412EA7AC.5080109@colorado.edu>

John Hunter wrote:
> First of all, I'm loving the new interactive shell.  For the first
> time in 2 years of using python every day, I'm trying to do almost
> everything interactively from the shell, including updating my package
> from ipython w/o restarting ipython!
> 
>   cd ~/some/dir
>   !sudo python setup.py install
>   reload somemodule
> 
> very nice....

Great!  I'm glad you're liking it.  In my group, the one person already using 
matplotlib full time is also loving it.  I'll bring some comments he has to 
your attention next week.  I think we have something very nice in our hands here.

> But I *may* have found a bug.  I have a script that produces an
> errobar plot which I'm calling with run in the pylab config .  The
> script imports matplotlib.matlab and nothing more.  On a call to
> errorbar, the ipython session hung.  I could not control-c it control
> control-z it.  A ps revealed
> 
>     hunter:~/python/projects/cluster/examples> ps
>     PID TTY          TIME CMD
>     29177 pts/0    00:00:00 tcsh
>     29381 pts/0    00:28:51 ipython
>     29405 pts/0    00:00:00 sh
>     29406 pts/0    00:00:00 gnuplot
>     29411 pts/0    00:00:00 ps
> 
>     hunter:~/python/projects/cluster/examples> killall -9 ipython
>     hunter:~/python/projects/cluster/examples> ps
>     PID TTY          TIME CMD
>     29177 pts/0    00:00:00 tcsh
>     29413 pts/0    00:00:00 ps
> 
> Note that gnuplot was running, and it was killed when I killed
> ipython!  Is it possible that there is some vestigial gnuplot support
> creaping in?

This is very strange.  The only way ipython can start gnuplot is if you import 
IPython.GnuplotRuntime or IPython.GnuplotInteractive.  The default numeric 
profile DOES load the gnuplot support, but pylab doesn't.  Here's an example:

[~]> pylab
Using IPython.Shell.IPShellMatplotlibWX with the WXAgg backend.
Python 2.2.3 (#1, Oct 15 2003, 23:33:35)
[GCC 3.3.1 20030930 (Red Hat Linux 3.3.1-6)] on linux2
Type "copyright", "credits" or "license" for more information.
(MatplotlibShell)

In [1]: !ps aux | grep gnuplot
fperez   20987  0.0  0.1  4928  572 pts/12   S    21:09   0:00 grep gnuplot

Try the same thing and let me know what you see.  Let me also know exactly how 
you are invoking ipython (which profile) and what your profile does.  Are you 
using the pylab profile as supplied by me, or did you modify it?

I'm also puzzled by the fact that ps shows 'ipython', instead of 'python'. 
ipython is not a binary, and on my system ps shows python.  Are you on a BSD 
or linux box (I don't know if they report differently)?

Best,

f

ps.  I just got your mail now (Thursday evening), but the date on it is 3 days 
ago.  I wonder if there's a problem with the scipy list server (I also just 
got a mail I sent to scipy-user earlier this afternoon, ~6 hours ago).



From Fernando.Perez at colorado.edu  Mon Aug 30 22:47:46 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Mon, 30 Aug 2004 20:47:46 -0600
Subject: [IPython-dev] ANN: IPython 0.6.3 is out, with matplotlib support.
In-Reply-To: <m2wtzgedfh.fsf@mother.paradise.lost>
References: <m2wtzgedfh.fsf@mother.paradise.lost>
Message-ID: <4133E6D2.9070804@colorado.edu>

Hi all,

IPython 0.6.3 is out at: http://ipython.scipy.org/dist.

Some highlights of this release:

1. Matplotlib support

The major new feature this time is integrated support for matplotlib 
(http://matplotlib.sourceforge.net).  Thanks to the help of John Hunter 
(matplotlib's author) and others, IPython now has a new option -pylab.  With 
this flag, it loads and configures matplotlib for interactive use, 
automatically adapting thread handling to whatever backend is configured in 
your .matplotlibrc file.  This means that you can use Tk, GTK, or WX and 
IPython automatically handles threading for you so that the matplotlib figure 
windows do NOT block the shell.

The matplotlib features have been tested and work fairly well under Linux, 
though it's quite possible that bugs remain (threading code is particularly 
tricky to debug).  However, under Mac OSX and Windows we have run into 
difficulties which we don't know how to solve yet.  It appears that for these 
platforms, at this point only the Tk/TkAgg backends work well.  Anyone who 
works on these systems and knows about threads is welcome to help; the place 
to look is towards the end of Ipython/Shell.py, where all the thread-handling 
classes reside.

I'd like to thank John for all the time he spent helping me with this.  I 
think for those interested in scientific computing, the combination of 
matplotlib and ipython makes for a very interesting environment.

Please note that this requires matplotlib version 0.62.4 or newer, as John and 
I coordinated the development of these features, and it required changes to 
both ipython and matplotlib.

2. GTK & WXPython threading

There is also experimental, but disabled by default, support for generic (i.e. 
not specific to matplotlib) GTK and WX threading.  Ideally, IPython should 
allow you to run arbitrary GTK or WX programs without blocking, and new 
-gthread/-wthread options were introduced for this purpose.  However, this 
code is not working correctly and my threading knowledge is very limited.

If anyone is interested in this kind of functionality, and can help with 
debugging, simply modify the start() function at the end of IPython/Shell.py 
to re-enable those options (they are commented out).  At that point the 
threading code will become active again.  Any fixes for this will be most welcome.


3. As usual, a number of small bugfixes recently reported are also included.


Enjoy, and please report any problems.

Best,

Fernando Perez.