From fperez at colorado.edu  Thu Apr  1 04:48:43 2004
From: fperez at colorado.edu (Fernando Perez)
Date: Thu, 01 Apr 2004 02:48:43 -0700
Subject: [IPython-dev] Commits to CVS, fixes, and shell access (RFC)
Message-ID: <406BE57B.1080707@colorado.edu>

Hi all,

I'm trying to catch up a bit on ipython matters.  I just committed some 
bugfixes to CVS for various problems (among others, Solaris crashes caused by 
the curses module and crashes when users closed sys.stdin/out/err).

I also made some 'improvements' to the shell access mechanism, spurred by the 
recent discussions on the matter.   I'll try to say more about that discussion 
in a couple of days, but I'd appreciate some feedback on the changes I made.

Recapping, people have always wanted easy access to the shell, with a way to 
capture the result for further python use.  I added @sc as a magic to do this 
in January, and it's flexible with options.  But at the time, the '!!' syntax 
was also liked, and I realized that a very common usage mode is simply to want 
to loop over shell output in a line-oriented manner (as most unix tools are 
designed).

So I decided today to add (it's in CVS for you to play with) the following:

- @sx, a new magic to execute in the shell any command.  This one, in contrast 
to @sc, doesn't require a variable name.  Instead it _returns_ the output, 
split as a list on '\n', to the normal ipython namespace.  Since ipython 
remembers all output as _N, Out[N], and the last three as _,__ and ___, I 
realized that this might be a very useful thing to have, leveraging the 
existing output storage mechanism.

- !!cmd is a new syntax for '@sx cmd'.  It makes it very easy to get a command 
to the shell remembering and typing as little as possible.

Examples:

In [5]: !ls
IPython-0.5.0.tar.gz  pyamazonpres.tgz

In [6]: sx ls
Out[6]: ['IPython-0.5.0.tar.gz', 'pyamazonpres.tgz']

In [7]: !!ls  # identical to [6] above
Out[7]: ['IPython-0.5.0.tar.gz', 'pyamazonpres.tgz']

In [8]: sc a=ls  # nothing is printed, results in a

In [9]: print a
IPython-0.5.0.tar.gz
pyamazonpres.tgz

In [10]: sc -l a=ls  # results split as a list, then stored in a

In [11]: print a
['IPython-0.5.0.tar.gz', 'pyamazonpres.tgz']


In summary, you can now in ipython access the shell via (in addition to any 
aliases you may have defined):

- !cmd -> execute cmd, return None (output of cmd is printed)

- !!cmd, @sx cmd -> execute cmd and return output of cmd, split on '\n', as a 
python list.

- @sc [options] var=cmd -> execute cmd and store its output in var.  Returns 
None.  Option -l causes a split on '\n' before storing into var, and option -v 
causes the contents to be printed.

@sc is the one with the most flexibility for shell capture, while also 
requiring more typing and remembering options.


I see myself mostly using !cmd as today for quick shell access, and !!cmd when 
I want to pick up for example some filenames for further manipulation through 
python.  !!cmd is a natural enough extension of !cmd that I think I can 
remember it easily.

I'd appreciate feedback on these changes, esp. on whether they enhance 
ipython's functionality for shell access without detracting from its main 
python nature.  I'll try to address more radical (but optional) enhancements 
which others have proposed in another message in a few days.

Also, any reports of breakage (or successful improvements) from the recent CVS 
commits would be very welcome.

Cheers,

f.



From pythondev-dang at lazytwinacres.net  Thu Apr  1 11:26:39 2004
From: pythondev-dang at lazytwinacres.net (pythondev-dang)
Date: Thu, 01 Apr 2004 21:26:39 +0500
Subject: [IPython-dev] [IPython-user] Commits to CVS, fixes,
	and shell access (RFC)
Message-ID: <20040401162639.14508.qmail@server265.com>

This sounds great--similar to what I did for my "grep" shell access,
but generalized.  (I didn't look into storing the results in "hidden"
variables, but that looks like a good idea also.)

I'm having troubles accessing the CVS repository from here, so I 
won't be able to give usage feedback until after this weekend
at the soonest.

I've a question though.  My grep shell extension would capture
stdout and stderr separately, and wouldn't return stderr, but
just printed it.  It would then return whatever the command 
*had* sent to stdout (or None if nothing).  What does @sc do
for this?

    Thanks,
        --dang



   -------Original Message-------
   > From: Fernando Perez <fperez at colorado.edu>
   > Subject: [IPython-user] Commits to CVS, fixes, and shell access (RFC)
   > Sent: 01 Apr 2004 14:48:43
   >
   >  Hi all,
   >  
   >  I'm trying to catch up a bit on ipython matters.??I just committed some
   >  bugfixes to CVS for various problems (among others, Solaris crashes caused by
   >  the curses module and crashes when users closed sys.stdin/out/err).
   >  
   >  I also made some 'improvements' to the shell access mechanism, spurred by the
   >  recent discussions on the matter.?? I'll try to say more about that discussion
   >  in a couple of days, but I'd appreciate some feedback on the changes I made.
   >  
   >  Recapping, people have always wanted easy access to the shell, with a way to
   >  capture the result for further python use.??I added @sc as a magic to do this
   >  in January, and it's flexible with options.??But at the time, the '!!' syntax
   >  was also liked, and I realized that a very common usage mode is simply to want
   >  to loop over shell output in a line-oriented manner (as most unix tools are
   >  designed).
   >  
   >  So I decided today to add (it's in CVS for you to play with) the following:
   >  
   >  - @sx, a new magic to execute in the shell any command.??This one, in contrast
   >  to @sc, doesn't require a variable name.??Instead it _returns_ the output,
   >  split as a list on '\n', to the normal ipython namespace.??Since ipython
   >  remembers all output as _N, Out[N], and the last three as _,__ and ___, I
   >  realized that this might be a very useful thing to have, leveraging the
   >  existing output storage mechanism.
   >  
   >  - !!cmd is a new syntax for '@sx cmd'.??It makes it very easy to get a command
   >  to the shell remembering and typing as little as possible.
   >  
   >  Examples:
   >  
   >  In [5]: !ls
   >  IPython-0.5.0.tar.gz??pyamazonpres.tgz
   >  
   >  In [6]: sx ls
   >  Out[6]: ['IPython-0.5.0.tar.gz', 'pyamazonpres.tgz']
   >  
   >  In [7]: !!ls??# identical to [6] above
   >  Out[7]: ['IPython-0.5.0.tar.gz', 'pyamazonpres.tgz']
   >  
   >  In [8]: sc a=ls??# nothing is printed, results in a
   >  
   >  In [9]: print a
   >  IPython-0.5.0.tar.gz
   >  pyamazonpres.tgz
   >  
   >  In [10]: sc -l a=ls??# results split as a list, then stored in a
   >  
   >  In [11]: print a
   >  ['IPython-0.5.0.tar.gz', 'pyamazonpres.tgz']
   >  
   >  
   >  In summary, you can now in ipython access the shell via (in addition to any
   >  aliases you may have defined):
   >  
   >  - !cmd -> execute cmd, return None (output of cmd is printed)
   >  
   >  - !!cmd, @sx cmd -> execute cmd and return output of cmd, split on '\n', as a
   >  python list.
   >  
   >  - @sc [options] var=cmd -> execute cmd and store its output in var.??Returns
   >  None.??Option -l causes a split on '\n' before storing into var, and option -v
   >  causes the contents to be printed.
   >  
   >  @sc is the one with the most flexibility for shell capture, while also
   >  requiring more typing and remembering options.
   >  
   >  
   >  I see myself mostly using !cmd as today for quick shell access, and !!cmd when
   >  I want to pick up for example some filenames for further manipulation through
   >  python.??!!cmd is a natural enough extension of !cmd that I think I can
   >  remember it easily.
   >  
   >  I'd appreciate feedback on these changes, esp. on whether they enhance
   >  ipython's functionality for shell access without detracting from its main
   >  python nature.??I'll try to address more radical (but optional) enhancements
   >  which others have proposed in another message in a few days.
   >  
   >  Also, any reports of breakage (or successful improvements) from the recent CVS
   >  commits would be very welcome.
   >  
   >  Cheers,
   >  
   >  f.
   >  
   >  _______________________________________________
   >  IPython-user mailing list
   >  IPython-user at scipy.net
   >  http://scipy.net/mailman/listinfo/ipython-user
   -------Original Message-------



From Fernando.Perez at colorado.edu  Thu Apr  1 15:13:20 2004
From: Fernando.Perez at colorado.edu (Fernando Perez)
Date: Thu, 01 Apr 2004 13:13:20 -0700
Subject: [IPython-dev] Re: [IPython-user] Commits to CVS, fixes,
	and shell access (RFC)
In-Reply-To: <20040401162639.14508.qmail@server265.com>
References: <20040401162639.14508.qmail@server265.com>
Message-ID: <406C77E0.4060103@colorado.edu>

pythondev-dang wrote:
> This sounds great--similar to what I did for my "grep" shell access,
> but generalized.  (I didn't look into storing the results in "hidden"
> variables, but that looks like a good idea also.)

Indeed, your grep (with the caveats below) would now just be one more case of:

!!grep regexp files

For example:

!!grep assert *.py

would work, since all wildcards propagate down directly to the shell.

> I'm having troubles accessing the CVS repository from here, so I 
> won't be able to give usage feedback until after this weekend
> at the soonest.
> 
> I've a question though.  My grep shell extension would capture
> stdout and stderr separately, and wouldn't return stderr, but
> just printed it.  It would then return whatever the command 
> *had* sent to stdout (or None if nothing).  What does @sc do
> for this?

I was lazy and just used commands.getoutput(), which lumps them together.  But 
  what you are doing is the right thing, IMO.  I'll recode it later and 
recommit to CVS, so that both @sc and @sx function as follows:

- stdout is captured, either to the named var for @sc or to the normal ipython 
return value for @sx.

- stderr, if present, is just printed to the screen.

How does this sound?

Thanks for the feedback.

Cheers,

f.



From fperez at colorado.edu  Fri Apr  2 01:37:42 2004
From: fperez at colorado.edu (Fernando Perez)
Date: Thu, 01 Apr 2004 23:37:42 -0700
Subject: [IPython-dev] Re: [IPython-user] Commits to CVS, fixes,
	and shell access (RFC)
In-Reply-To: <20040401162639.14508.qmail@server265.com>
References: <20040401162639.14508.qmail@server265.com>
Message-ID: <406D0A36.2070904@colorado.edu>

pythondev-dang wrote:
> This sounds great--similar to what I did for my "grep" shell access,
> but generalized.  (I didn't look into storing the results in "hidden"
> variables, but that looks like a good idea also.)

These variables aren't really that 'hidden':  In[N] and Out[N] are always 
there for you, and hopefully the names of the prompts hint at their existence. 
  The magic stuff is that _N==Out[N], and _iN==In[N], and _,__ and ___ hold 
always Out[-1], Out[-2] and Out[-3].  When using ipython for scientific work, 
this kind of caching is incredibly useful.

> I've a question though.  My grep shell extension would capture
> stdout and stderr separately, and wouldn't return stderr, but
> just printed it.  It would then return whatever the command 
> *had* sent to stdout (or None if nothing).  What does @sc do
> for this?

Done.  I've uploaded changes so that now stderr is separately handled:

In [5]: d
lrwxrwxrwx  1 fperez     3 Apr  1 21:49 bar -> foo  ### broken link
-rw-r--r--  1 fperez   124 Oct  6 13:25 die.py
-rwxr-xr-x  1 fperez  9988 Jul  7  2003 divc*
-rwxr-xr-x  1 fperez 11671 Jul  7  2003 divf*

In [6]: !!grep ff *
grep: bar: No such file or directory
Out[6]: ['Binary file divc matches', 'Binary file divf matches']

Note stderr message printed to console before the output is returned.

Thanks again for the feedback.  Hopefully this will provide some relief for 
those interested in ipython having better system integration.

Cheers,

f



From a.schmolck at gmx.net  Thu Apr 29 15:53:02 2004
From: a.schmolck at gmx.net (Alexander Schmolck)
Date: Thu, 29 Apr 2004 20:53:02 +0100
Subject: [IPython-dev] minor ipython.el news
Message-ID: <yfs8yge4l41.fsf@black132.ex.ac.uk>

Hi,

After promising to look into the problems with CVS python-mode and my original
patch, I've lapsed into long silence -- sorry about that but I've first
discovered that I couldn't quickly determine the problem and then I
unfortunately fell ill for several weeks and have been really busy ever since
(so unless someone else has the time to work out how the new python.el screws
up C- -, don't expect a new patch any time soon; the original one I submitted
unfortunately didn't receive any attention and since the old python.el works
OK for me, I don't really have some spare hours to sacrifice for writing a
patch that might be ignored again; also other than breaking py-up-exception
etc. there seem to be no problems in applying the patch to the CVS
python-mode.el).

I have however recently discovered in an email exchange with a ipytho/emacs
user that the version of ipython.el in ipython-emacs.tgz is older than the one
I use myself (I've made some slight improvements to the traceback regexp and
added some dirtrack stuff, which might actually be redundant).

Anyway, here is the newest version:

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ipython.el
Type: application/emacs-lisp
Size: 13242 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/ipython-dev/attachments/20040429/19b27387/attachment.bin>
-------------- next part --------------

BTW, I think it would be nice if ipython's traceback output would be a bit
more uniform, i.e. always match this (or some other emacs-regexp) pattern:

 \(^[^\t ].+?\.py\).*\n   +[0-9]+[^\00]*?\n-+> \([0-9]+\) +

Syntax errors for example currently produce output that doesn't match the
abovee, so it's not possible to automatically jump to the right line in the
source code (and emacs regexp limitations + the way traceback parsing is
hardcoded in python-mode.el make it impossible to handle them seperately).

'as


P.S: There has been some effort in supplying shell like functionality to
ipython of late (not a bad idea in principle, I think, the fact that python
doesn't have syntax extension facilities is severely limiting here) -- one
thing that really often annoys me is the behavior of @run (which not only does
not perform any shell like globbing, but also doesn't seem to allow actually
passing parameters with spaces in them since there is no quoting or escaping
mechanism as far as I'm aware). Are there any plans to add globbing and
quoting or is there already an easy way to get the equivalent of

> ./script.py --title 'some title'

using @run?