From chris.mcavoy at gmail.com  Tue Nov  1 19:25:28 2005
From: chris.mcavoy at gmail.com (Chris McAvoy)
Date: Tue, 1 Nov 2005 12:25:28 -0600
Subject: [Chicago] Any topics?
Message-ID: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>

Hello All,

We've all been pretty quiet as of late...does anyone have anything they'd
like to present for November? I believe we have a place (not yet nailed
down, but ignore that for the time being). Assuming we have a room to go to,
anyone want to talk about anything?

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/chicago/attachments/20051101/7d38c5ab/attachment.html

From ianb at colorstudy.com  Tue Nov  1 19:25:56 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 01 Nov 2005 12:25:56 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
Message-ID: <4367B334.6090803@colorstudy.com>

Chris McAvoy wrote:
> Hello All,
> 
> We've all been pretty quiet as of late...does anyone have anything 
> they'd like to present for November?  I believe we have a place (not yet 
> nailed down, but ignore that for the time being).  Assuming we have a 
> room to go to, anyone want to talk about anything?

I could do a little thing on generic functions (aka RuleDispatch)

-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org

From chris.mcavoy at gmail.com  Tue Nov  1 19:33:47 2005
From: chris.mcavoy at gmail.com (Chris McAvoy)
Date: Tue, 1 Nov 2005 12:33:47 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <4367B334.6090803@colorstudy.com>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
	<4367B334.6090803@colorstudy.com>
Message-ID: <3096c19d0511011033i176ffee3j624056c48632ce0a@mail.gmail.com>

On 11/1/05, Ian Bicking <ianb at colorstudy.com> wrote:

>
> I could do a little thing on generic functions (aka RuleDispatch)
>
>
Sounds good to me. If other folks weigh in, we could do some extended blitz
type presentations. BOOM!

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/chicago/attachments/20051101/5809c9e1/attachment.html

From bray at sent.com  Tue Nov  1 19:38:52 2005
From: bray at sent.com (Brian Ray)
Date: Tue, 1 Nov 2005 12:38:52 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
Message-ID: <71A43F44-622F-497A-B01F-2E88054FCB63@sent.com>

On Nov 1, 2005, at 12:25 PM, Chris McAvoy wrote:

> Hello All,
>
> We've all been pretty quiet as of late...does anyone have anything  
> they'd like to present for November?  I believe we have a place  
> (not yet nailed down, but ignore that for the time being).   
> Assuming we have a room to go to, anyone want to talk about anything?


Robert said he could talk on a module.

Robert: which one? There are 994+ <http://cheeseshop.python.org/pypi>

From RCRamsdell at gldd.com  Tue Nov  1 20:19:52 2005
From: RCRamsdell at gldd.com (RCRamsdell@gldd.com)
Date: Tue, 1 Nov 2005 13:19:52 -0600
Subject: [Chicago] Any topics?
Message-ID: <C0B0D5759E27344788E6EB554ECF4B9101FB8EEA@NEWMAIL.internal.gldd.com>

I was thinking about something from the standard library.  Suggestions:

Random (Gives me a chance to brush up on statistics)
Pickle (I need to get into this for one of my projects anyways)
Zip+StringIO+csv (I had to figure this combo out for a project a while
back)

Or are these too trivial?  Feel free to vote/comment/suggest others.

Robert

> -----Original Message-----
> From: chicago-bounces at python.org 
> [mailto:chicago-bounces at python.org] On Behalf Of Brian Ray
> Sent: Tuesday, November 01, 2005 12:39 PM
> To: The Chicago Python Users Group
> Subject: Re: [Chicago] Any topics?
> 
> 
> On Nov 1, 2005, at 12:25 PM, Chris McAvoy wrote:
> 
> > Hello All,
> >
> > We've all been pretty quiet as of late...does anyone have anything
> > they'd like to present for November?  I believe we have a place  
> > (not yet nailed down, but ignore that for the time being).   
> > Assuming we have a room to go to, anyone want to talk about 
> anything?
> 
> 
> Robert said he could talk on a module.
> 
> Robert: which one? There are 994+ <http://cheeseshop.python.org/pypi>
> _______________________________________________
> Chicago mailing list
> Chicago at python.org http://mail.python.org/mailman/listinfo/chicago
> 

From chris.mcavoy at gmail.com  Tue Nov  1 20:26:46 2005
From: chris.mcavoy at gmail.com (Chris McAvoy)
Date: Tue, 1 Nov 2005 13:26:46 -0600
Subject: [Chicago] Next Meeting 11/10 7pm
Message-ID: <3096c19d0511011126k2bb365bdmcc30b56ad9b911c4@mail.gmail.com>

We are confirmed for a space here at my (and other's on this list) workplace
for 11/10 at 7pm. The address is:

Performics
180 N. Lasalle 12th floor.

I'll need to compile a list of who is coming. So please email me direct
chris.mcavoy at gmail.com with something in the subject along the lines of
"RSVP for ChiPy"

Please spread it around. The only restriction is the RSVP part.

Also, I know we're sort of skipping the 'burbs here. We should try and get
back out there...maybe for December?

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/chicago/attachments/20051101/9280750c/attachment.html

From bray at sent.com  Tue Nov  1 21:17:09 2005
From: bray at sent.com (Brian Ray)
Date: Tue, 1 Nov 2005 14:17:09 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <C0B0D5759E27344788E6EB554ECF4B9101FB8EEA@NEWMAIL.internal.gldd.com>
References: <C0B0D5759E27344788E6EB554ECF4B9101FB8EEA@NEWMAIL.internal.gldd.com>
Message-ID: <1C8552F3-A727-4AE4-AA21-FF3E0530AC47@sent.com>


On Nov 1, 2005, at 1:19 PM, <RCRamsdell at gldd.com>  
<RCRamsdell at gldd.com> wrote:

> Random (Gives me a chance to brush up on statistics)
> Pickle (I need to get into this for one of my projects anyways)
> Zip+StringIO+csv (I had to figure this combo out for a project a while
> back)

These any of these sound good to me.

If you do Random, you could probably put some math spin in there.

Random is my vote.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/chicago/attachments/20051101/fa455987/attachment.htm

From bray at sent.com  Tue Nov  1 21:20:52 2005
From: bray at sent.com (Brian Ray)
Date: Tue, 1 Nov 2005 14:20:52 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <1C8552F3-A727-4AE4-AA21-FF3E0530AC47@sent.com>
References: <C0B0D5759E27344788E6EB554ECF4B9101FB8EEA@NEWMAIL.internal.gldd.com>
	<1C8552F3-A727-4AE4-AA21-FF3E0530AC47@sent.com>
Message-ID: <8FC4909E-749E-40F2-B886-CF1A08A2F086@sent.com>

Maybe our meeting topic can *specifically* be "Random and Generic".

From chris.mcavoy at gmail.com  Tue Nov  1 21:32:46 2005
From: chris.mcavoy at gmail.com (Chris McAvoy)
Date: Tue, 1 Nov 2005 14:32:46 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <8FC4909E-749E-40F2-B886-CF1A08A2F086@sent.com>
References: <C0B0D5759E27344788E6EB554ECF4B9101FB8EEA@NEWMAIL.internal.gldd.com>
	<1C8552F3-A727-4AE4-AA21-FF3E0530AC47@sent.com>
	<8FC4909E-749E-40F2-B886-CF1A08A2F086@sent.com>
Message-ID: <3096c19d0511011232j6e06fe91r6f6b0d6ce09bebc1@mail.gmail.com>

On 11/1/05, Brian Ray <bray at sent.com> wrote:
>
> Maybe our meeting topic can *specifically* be "Random and Generic".



Damn, I love puns. +1.

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/chicago/attachments/20051101/02e3b9ff/attachment.html

From bray at sent.com  Tue Nov  1 21:42:58 2005
From: bray at sent.com (Brian Ray)
Date: Tue, 1 Nov 2005 14:42:58 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <4367B334.6090803@colorstudy.com>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
	<4367B334.6090803@colorstudy.com>
Message-ID: <3B877D6C-BA53-420A-AECF-D359602C64DB@sent.com>


On Nov 1, 2005, at 12:25 PM, Ian Bicking wrote:

> I could do a little thing on generic functions (aka RuleDispatch)


BTW, I had to go here <http://peak.telecommunity.com/> to figure out  
what RuleDispatch was.

$ python -c "import dispatch; print dispatch.__doc__"

Multiple/Predicate Dispatch Framework

This framework refines the algorithms of Chambers and Chen in their 1999
paper, "Efficient Multiple and Predicate Dispatching", to make them  
suitable
for Python, while adding a few other enhancements like incremental index
building and lazy expansion of the dispatch DAG.   Also, their algorithm
was designed only for class selection and true/false tests, while this
framework can be used with any kind of test, such as numeric ranges,  
or custom
tests such as categorization/hierarchy membership.

NOTE: this package is not yet ready for prime-time.  APIs are subject to
change randomly without notice.  You have been warned!

TODO

     * Support DAG-walking for visualization, debugging, and  
ambiguity detection


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/chicago/attachments/20051101/a3107cfd/attachment.htm

From chris.mcavoy at gmail.com  Tue Nov  1 21:46:21 2005
From: chris.mcavoy at gmail.com (Chris McAvoy)
Date: Tue, 1 Nov 2005 14:46:21 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <3B877D6C-BA53-420A-AECF-D359602C64DB@sent.com>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
	<4367B334.6090803@colorstudy.com>
	<3B877D6C-BA53-420A-AECF-D359602C64DB@sent.com>
Message-ID: <3096c19d0511011246o6a1ae518ta9b13b92c9f12e14@mail.gmail.com>

On 11/1/05, Brian Ray <bray at sent.com> wrote:
>
>
> On Nov 1, 2005, at 12:25 PM, Ian Bicking wrote:
>
> I could do a little thing on generic functions (aka RuleDispatch)
>
>
>
> BTW, I had to go here <http://peak.telecommunity.com/> to figure out what
> RuleDispatch was.
>


Yeah, me too. That page led me to this article:
http://www-128.ibm.com/developerworks/library/l-cppeak2/index.html

I think I saw that when it came out, but it fell off my radar. It's a neat
idea. I'm looking forward to the talk.

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/chicago/attachments/20051101/9500a260/attachment.html

From mtobis at gmail.com  Tue Nov  1 22:08:00 2005
From: mtobis at gmail.com (Michael Tobis)
Date: Tue, 1 Nov 2005 15:08:00 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <C0B0D5759E27344788E6EB554ECF4B9101FB8EEA@NEWMAIL.internal.gldd.com>
References: <C0B0D5759E27344788E6EB554ECF4B9101FB8EEA@NEWMAIL.internal.gldd.com>
Message-ID: <c3aa55550511011308t2e8b3ebcya67ea053ca60ae30@mail.gmail.com>

Hate to miss this month's meeting. Can we have it in Atlanta?

The good news is that I'm teaching a course in distributed systems at
Loyola, and I'm using Python exclusively.

(They have this odd half-semester system. I teach one night a week for
four hours, over eight weeks.)

If anyone wants to look at my slides, they first two marathons are at
http://webpages.cs.luc.edu/~mt/LEX0.1.html and
http://webpages.cs.luc.edu/~mt/LEX1.html

It's all in the original version of S5. The second one is output from
rst2s5, which is especially sweet. see
http://webpages.cs.luc.edu/~mt/LEX1.txt for my source file.

rst2s5 doesn't support the new 1.1 S5 yet. Someone appears to be
working on it, though.

Michael

From ianb at colorstudy.com  Tue Nov  1 22:12:51 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 01 Nov 2005 15:12:51 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <3B877D6C-BA53-420A-AECF-D359602C64DB@sent.com>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>	<4367B334.6090803@colorstudy.com>
	<3B877D6C-BA53-420A-AECF-D359602C64DB@sent.com>
Message-ID: <4367DA53.4050801@colorstudy.com>

Brian Ray wrote:
> On Nov 1, 2005, at 12:25 PM, Ian Bicking wrote:
> 
>> I could do a little thing on generic functions (aka RuleDispatch)
>>
> 
> 
> BTW, I had to go here <http://peak.telecommunity.com/> to figure out 
> what RuleDispatch was.

You are giving away my presentation!  Man, I only had about three 
sentences worth of content prepared, and now you've quoted all three 
verbatim.  And then Chris goes and refers to the other article, which 
was where I was going to appropriate the rest of the content...

Now I'm going to have to come up with my own ideas, blast.

-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org

From bray at sent.com  Tue Nov  1 22:28:33 2005
From: bray at sent.com (Brian Ray)
Date: Tue, 1 Nov 2005 15:28:33 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <4367DA53.4050801@colorstudy.com>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>	<4367B334.6090803@colorstudy.com>
	<3B877D6C-BA53-420A-AECF-D359602C64DB@sent.com>
	<4367DA53.4050801@colorstudy.com>
Message-ID: <9F5B0673-FDB5-402A-AA02-1341DB20D70B@sent.com>


On Nov 1, 2005, at 3:12 PM, Ian Bicking wrote:

> You are giving away my presentation!  Man, I only had about three
> sentences worth of content prepared, and now you've quoted all three
> verbatim.  And then Chris goes and refers to the other article, which
> was where I was going to appropriate the rest of the content...
>
> Now I'm going to have to come up with my own ideas, blast.


Trust me, you will have plenty of opportunity to clarify. ie convert  
to plain programmer talk.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/chicago/attachments/20051101/b45c211d/attachment.html

From chris.mcavoy at gmail.com  Tue Nov  1 22:36:00 2005
From: chris.mcavoy at gmail.com (Chris McAvoy)
Date: Tue, 1 Nov 2005 15:36:00 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <9F5B0673-FDB5-402A-AA02-1341DB20D70B@sent.com>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
	<4367B334.6090803@colorstudy.com>
	<3B877D6C-BA53-420A-AECF-D359602C64DB@sent.com>
	<4367DA53.4050801@colorstudy.com>
	<9F5B0673-FDB5-402A-AA02-1341DB20D70B@sent.com>
Message-ID: <3096c19d0511011336s15fb72aqd78d583fecf68ca1@mail.gmail.com>

On 11/1/05, Brian Ray <bray at sent.com> wrote:

>
> Trust me, you will have plenty of opportunity to clarify. ie convert to
> plain programmer talk.
>


Totally. You're confusing savvy Google use with understanding. I think the
Bible says something about "just because you have the link, doesn't mean you
get the concept."

Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/chicago/attachments/20051101/1f7626ce/attachment.html

From ianb at colorstudy.com  Tue Nov  1 22:39:29 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 01 Nov 2005 15:39:29 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <9F5B0673-FDB5-402A-AA02-1341DB20D70B@sent.com>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>	<4367B334.6090803@colorstudy.com>	<3B877D6C-BA53-420A-AECF-D359602C64DB@sent.com>	<4367DA53.4050801@colorstudy.com>
	<9F5B0673-FDB5-402A-AA02-1341DB20D70B@sent.com>
Message-ID: <4367E091.2010205@colorstudy.com>

Brian Ray wrote:
> 
> On Nov 1, 2005, at 3:12 PM, Ian Bicking wrote:
> 
>> You are giving away my presentation!  Man, I only had about three 
>>
>> sentences worth of content prepared, and now you've quoted all three 
>>
>> verbatim.  And then Chris goes and refers to the other article, which 
>>
>> was where I was going to appropriate the rest of the content...
>>
>>
>> Now I'm going to have to come up with my own ideas, blast.
>>
> 
> 
> Trust me, you will have plenty of opportunity to clarify. ie convert to 
> plain programmer talk.

What about "This framework refines the algorithms of Chambers and Chen 
in their 1999 paper, "Efficient Multiple and Predicate Dispatching", to 
make them suitable for Python," don't you understand?!?  I don't know if 
I can make it any clearer than that.  That, or Chambers and Chen will be 
very disappointed that you aren't familiar with their work!

-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org

From pfein at pobox.com  Tue Nov  1 23:24:21 2005
From: pfein at pobox.com (Peter Fein)
Date: Tue, 01 Nov 2005 16:24:21 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
Message-ID: <4367EB15.6070102@pobox.com>

Chris McAvoy wrote:
> Hello All,
> 
> We've all been pretty quiet as of late...does anyone have anything
> they'd like to present for November?  I believe we have a place (not yet
> nailed down, but ignore that for the time being).  Assuming we have a
> room to go to, anyone want to talk about anything?

I'd maybe want to do for *next* month, or even January:

Descriptors and Metaclasses at Home:
Cool things to do with metaprogramming besides writing ORMs

Basically, an explanation of how these work and how to use them in small
ad-hoc doses.

-- 
Peter Fein                 pfein at pobox.com                 773-575-0694

Basically, if you're not a utopianist, you're a schmuck. -J. Feldman

From david at graniteweb.com  Wed Nov  2 03:23:49 2005
From: david at graniteweb.com (David Rock)
Date: Tue, 1 Nov 2005 20:23:49 -0600
Subject: [Chicago] Next Meeting 11/10 7pm
In-Reply-To: <3096c19d0511011126k2bb365bdmcc30b56ad9b911c4@mail.gmail.com>
References: <3096c19d0511011126k2bb365bdmcc30b56ad9b911c4@mail.gmail.com>
Message-ID: <20051102022348.GB9417@wdfs.graniteweb.com>

* Chris McAvoy <chris.mcavoy at gmail.com> [2005-11-01 13:26]:
> We are confirmed for a space here at my (and other's on this list) workplace
> for 11/10 at 7pm. The address is:
> 
> Performics
> 180 N. Lasalle 12th floor.
> 
> I'll need to compile a list of who is coming. So please email me direct
> chris.mcavoy at gmail.com with something in the subject along the lines of
> "RSVP for ChiPy"
> 
> Please spread it around. The only restriction is the RSVP part.
> 
> Also, I know we're sort of skipping the 'burbs here. We should try and get
> back out there...maybe for December?

I can almost guarantee I won't be available for December (that's what I
get for being active on the homeowners' association board). If we could
play with the date a bit (like the third Thurs instead) I shouldn't have
a problem.

For that matter, I don't think I am a requirement to use the room.

-- 
David Rock
david at graniteweb.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.python.org/pipermail/chicago/attachments/20051101/3c6709dc/attachment.pgp

From david at graniteweb.com  Wed Nov  2 03:18:24 2005
From: david at graniteweb.com (David Rock)
Date: Tue, 1 Nov 2005 20:18:24 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <3096c19d0511011336s15fb72aqd78d583fecf68ca1@mail.gmail.com>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
	<4367B334.6090803@colorstudy.com>
	<3B877D6C-BA53-420A-AECF-D359602C64DB@sent.com>
	<4367DA53.4050801@colorstudy.com>
	<9F5B0673-FDB5-402A-AA02-1341DB20D70B@sent.com>
	<3096c19d0511011336s15fb72aqd78d583fecf68ca1@mail.gmail.com>
Message-ID: <20051102021824.GA9417@wdfs.graniteweb.com>

* Chris McAvoy <chris.mcavoy at gmail.com> [2005-11-01 15:36]:
> On 11/1/05, Brian Ray <bray at sent.com> wrote:
> 
> >
> > Trust me, you will have plenty of opportunity to clarify. ie convert to
> > plain programmer talk.
> >
> 
> 
> Totally. You're confusing savvy Google use with understanding. I think the
> Bible says something about "just because you have the link, doesn't mean you
> get the concept."

Which passage was that? I seem to have missed it somehow ;-)

-- 
David Rock
david at graniteweb.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.python.org/pipermail/chicago/attachments/20051101/8f08b7f1/attachment.pgp

From fawad at fawad.net  Wed Nov  2 15:56:00 2005
From: fawad at fawad.net (Fawad Halim)
Date: Wed, 02 Nov 2005 08:56:00 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
Message-ID: <4368D380.6090503@fawad.net>

I can do a lightning talk on the PYRO (Python Remote Objects) library if 
anyone's interested, but I probably won't have slides/cheat sheet ready 
by next week. We already have 2 topic for this month, so I can do that 
next month.

Regards
-fawad

Chris McAvoy wrote:
> Hello All,
> 
> We've all been pretty quiet as of late...does anyone have anything 
> they'd like to present for November?  I believe we have a place (not yet 
> nailed down, but ignore that for the time being).  Assuming we have a 
> room to go to, anyone want to talk about anything?
> 
> Chris
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago

From bray at sent.com  Wed Nov  2 18:20:10 2005
From: bray at sent.com (Brian Ray)
Date: Wed, 2 Nov 2005 11:20:10 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <4368D380.6090503@fawad.net>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
	<4368D380.6090503@fawad.net>
Message-ID: <DBC719E0-1AF0-4DE4-B734-000613A387A4@sent.com>


"Remote, Generic and Random"... sounds good to me!


On Nov 2, 2005, at 8:56 AM, Fawad Halim wrote:

> I can do a lightning talk on the PYRO (Python Remote Objects)  
> library if
> anyone's interested, but I probably won't have slides/cheat sheet  
> ready
> by next week. We already have 2 topic for this month, so I can do that
> next month.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/chicago/attachments/20051102/0004fb53/attachment.htm

From bray at sent.com  Wed Nov  2 18:26:26 2005
From: bray at sent.com (Brian Ray)
Date: Wed, 2 Nov 2005 11:26:26 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <c3aa55550511011308t2e8b3ebcya67ea053ca60ae30@mail.gmail.com>
References: <C0B0D5759E27344788E6EB554ECF4B9101FB8EEA@NEWMAIL.internal.gldd.com>
	<c3aa55550511011308t2e8b3ebcya67ea053ca60ae30@mail.gmail.com>
Message-ID: <32FE8E0C-06DE-4B0D-8EC4-84F0543AE203@sent.com>

On Nov 1, 2005, at 3:08 PM, Michael Tobis wrote:

>
> The good news is that I'm teaching a course in distributed systems at
> Loyola, and I'm using Python exclusively.

This is very good new, mt. Congratulations!

Those students are very fortunate to get to work with you and Python.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/chicago/attachments/20051102/7ea6b88a/attachment.html

From maney at two14.net  Wed Nov  2 20:53:42 2005
From: maney at two14.net (Martin Maney)
Date: Wed, 2 Nov 2005 13:53:42 -0600
Subject: [Chicago] Any topics?
In-Reply-To: <4367E091.2010205@colorstudy.com>
References: <3096c19d0511011025n6a433623lcd5aac6449baf382@mail.gmail.com>
	<4367B334.6090803@colorstudy.com>
	<3B877D6C-BA53-420A-AECF-D359602C64DB@sent.com>
	<4367DA53.4050801@colorstudy.com>
	<9F5B0673-FDB5-402A-AA02-1341DB20D70B@sent.com>
	<4367E091.2010205@colorstudy.com>
Message-ID: <20051102195342.GA20929@furrr.two14.net>

On Tue, Nov 01, 2005 at 03:39:29PM -0600, Ian Bicking wrote:
> Chambers and Chen will be very disappointed that you aren't familiar
> with their work!

I have a sneaking suspicion that they're used to that disappointment by
now.  :-)

-- 
[this space for rent]

From razeh at earthlink.net  Thu Nov  3 23:06:45 2005
From: razeh at earthlink.net (Robert Allan Zeh)
Date: Thu, 3 Nov 2005 16:06:45 -0600 (GMT-06:00)
Subject: [Chicago] Combining Tkinter and asyncore
Message-ID: <14232744.1131055605630.JavaMail.root@elwamui-lapwing.atl.sa.earthlink.net>

Does anyone have any experience combining Tkinter and asyncore?  

I've been looking for a clean way to do this; it would be nice if I only had one event loop.  After spending some time with google, I found the following:

        while 1:
            tkinter.dooneevent(tkinter.DONT_WAIT)
            asyncore.poll(0.01) 

Which is a bit unsatisfying --- I'd rather have one underlying call to select.

Thanks,
Robert Zeh



From bray at sent.com  Sat Nov  5 05:34:24 2005
From: bray at sent.com (Brian Ray)
Date: Fri, 4 Nov 2005 22:34:24 -0600
Subject: [Chicago] Book Review: Producing Open Source Software
Message-ID: <B7911422-EF00-443F-AA44-6AD01418294E@sent.com>



<http://brianray.chipy.org//Book%20Reviews/ProducingOSS.html>

If you wish to review a book hardcopy, ask chris.mcavoy at gmail.com.  
Write a review for O'Reilly and you get to keep the book for free.

From nttaylor at uchicago.edu  Tue Nov  8 01:12:14 2005
From: nttaylor at uchicago.edu (Noel Thomas Taylor)
Date: Mon, 7 Nov 2005 18:12:14 -0600 (CST)
Subject: [Chicago] capturing output from subprocesses
Message-ID: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>


Dear Chicago Python Hackers,

I just joined the list and hope to come to the meeting this Thursday. If I 
can't, however, I wonder if someone could help me with a very tricky 
problem...

A couple of months ago I started trying to find a way to launch a
subprocess in python that would allow the parent to capture the stdout
and stderr of the child, time the duration of the child, and abort the
child if it had failed to produce output after 'n' seconds. Also, the
solution should perform consistently across all Unices (Windows doesn't
matter) and if the child must be aborted, whatever output it produced up
to that point should still be captured.

There are several solutions out there, but most do not capture the output 
of the child in real time. You always have to wait for the child to 
terminate until you can see any of its output, and consequently if the 
child must be aborted because it gets stuck, you lose whatever output it 
generated before you killed it.

For several weeks I've been trying to use Noah Spurrier's "pexpect"
module, which uses pseudo-terminals to get around the above issue, but
pseudo-terminals do not perform consistently across platforms, and pexpect
fails under IRIX. Does anyone know if there is a way to solve the above 
problem with out pseudo-terminals?

I would greatly appreciate any knowledge any of you have to impart. Like I 
mentioned, I'm only concerned that this work on Unix/Linux/Irix, etc 
(though Mac OSX would be nice). The subprocess-launching code should be in 
python, but the child code might be python, fortran, or a shell script. 
What do you think?

with thanks in advance,

Noel Taylor

From ianb at colorstudy.com  Tue Nov  8 01:20:18 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Mon, 07 Nov 2005 18:20:18 -0600
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
Message-ID: <436FEF42.2050207@colorstudy.com>

Noel Thomas Taylor wrote:
> I just joined the list and hope to come to the meeting this Thursday. If I 
> can't, however, I wonder if someone could help me with a very tricky 
> problem...
> 
> A couple of months ago I started trying to find a way to launch a
> subprocess in python that would allow the parent to capture the stdout
> and stderr of the child, time the duration of the child, and abort the
> child if it had failed to produce output after 'n' seconds. Also, the
> solution should perform consistently across all Unices (Windows doesn't
> matter) and if the child must be aborted, whatever output it produced up
> to that point should still be captured.
> 
> There are several solutions out there, but most do not capture the output 
> of the child in real time. You always have to wait for the child to 
> terminate until you can see any of its output, and consequently if the 
> child must be aborted because it gets stuck, you lose whatever output it 
> generated before you killed it.

The subprocess module can do this.  It's a little funny, because it 
doesn't allow streaming using proc.communicate(), and I find it 
difficult to do normal piping because of the blocking.

But if you look at the bottom of this file there's a function 
proc_communicate allows you to communicate using Python file-like objects:

   http://svn.pythonpaste.org/Paste/trunk/paste/cgiapp.py

It's basically a slight rewrite of Proc.communicate().  There's already 
some patches filed for this issue, so hopefully subprocess in Python 2.5 
will make this easier.


-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org

From ianb at colorstudy.com  Tue Nov  8 09:24:19 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 08 Nov 2005 02:24:19 -0600
Subject: [Chicago] Chicago Python Users Group, Thurs Nov 10
Message-ID: <437060B3.3050302@colorstudy.com>

November topics are "Remote, Generic and Random", just like us.

We'll have presentations on PyRO (Python Remote Objects) by Fawad
Halim, generic functions (as implemented in RuleDispatch) by Ian
Bicking, and the standard library random module by Robert Ramsdell.

There will also be time to chat, and many opportunities to ask
questions. We encourage people at all levels to attend.

We'll be meeting at 7 PM, Thursday November 10, in downtown Chicago at
Performics 180 N. Lasalle 12th floor.  RSVP is necessary for building
security, email chris.mcavoy at gmail.com with a subject of "RSVP for
ChiPy" to be added to the list.  If you aren't sure if you can come,
email just in case.

About ChiPy (http://chipy.org):

ChiPy meets on the second Thursday of each month, at different
locations around Chicagoland.  If you are interested in goings-on,
please subscribe to our mailing list:
http://mail.python.org/mailman/listinfo/chicago


From maney at two14.net  Tue Nov  8 14:49:29 2005
From: maney at two14.net (Martin Maney)
Date: Tue, 8 Nov 2005 07:49:29 -0600
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
Message-ID: <20051108134929.GA23375@furrr.two14.net>

On Mon, Nov 07, 2005 at 06:12:14PM -0600, Noel Thomas Taylor wrote:
> For several weeks I've been trying to use Noah Spurrier's "pexpect"
> module, which uses pseudo-terminals to get around the above issue, but
> pseudo-terminals do not perform consistently across platforms, and pexpect
> fails under IRIX. Does anyone know if there is a way to solve the above 
> problem with out pseudo-terminals?

Nope.  The problem is that the child processes are internally buffering
their output when attached to a file-like sink, so there's nothing
coming out of them until (a) the internal buffer fills and gets
flushed, (b) the subprocess manually flushes the buffer, or (c) normal
termination casues all buffers to be flushed.  PTYs fix that because
the stdio library handles them differently (I think it goes to
line-oriented flushing mode, but that's an old, possibly nonstandard,
memory).

> I would greatly appreciate any knowledge any of you have to impart. Like I 
> mentioned, I'm only concerned that this work on Unix/Linux/Irix, etc 
> (though Mac OSX would be nice). The subprocess-launching code should be in 
> python, but the child code might be python, fortran, or a shell script. 
> What do you think?

Okay, one other way to fix it: if you can pass an option to tell the
program to use unbuffered (or line buffered) output (but that's rarely
available), or if you can alter the program itself, you can get
immediate output through a pipe.  Otherwise, PTYs are probably as
portable a solution as there is, to my knowledge.

-- 
People make secure systems insecure because
insecure systems do what people want and
secure systems don't.  -- James Grimmelmann


From RCRamsdell at gldd.com  Tue Nov  8 15:58:33 2005
From: RCRamsdell at gldd.com (RCRamsdell@gldd.com)
Date: Tue, 8 Nov 2005 08:58:33 -0600
Subject: [Chicago] Chicago Python Users Group, Thurs Nov 10
Message-ID: <C0B0D5759E27344788E6EB554ECF4B9101FB8F03@NEWMAIL.internal.gldd.com>

As always, I am driving in from Oakbrook, if anyone wants to ride
together.  My office is convenient to 290 and 294.  I usually leave
about 1 hour before the meeting, and promise to remember the directions
this time.

Robert

> -----Original Message-----
> From: chicago-bounces at python.org 
> [mailto:chicago-bounces at python.org] On Behalf Of Ian Bicking
> Sent: Tuesday, November 08, 2005 2:24 AM
> To: chicago at python.org
> Subject: [Chicago] Chicago Python Users Group, Thurs Nov 10
> 
> 
> November topics are "Remote, Generic and Random", just like us.
> 
> We'll have presentations on PyRO (Python Remote Objects) by 
> Fawad Halim, generic functions (as implemented in 
> RuleDispatch) by Ian Bicking, and the standard library random 
> module by Robert Ramsdell.
> 
> There will also be time to chat, and many opportunities to 
> ask questions. We encourage people at all levels to attend.
> 
> We'll be meeting at 7 PM, Thursday November 10, in downtown 
> Chicago at Performics 180 N. Lasalle 12th floor.  RSVP is 
> necessary for building security, email chris.mcavoy at gmail.com 
> with a subject of "RSVP for ChiPy" to be added to the list.  
> If you aren't sure if you can come, email just in case.
> 
> About ChiPy (http://chipy.org):
> 
> ChiPy meets on the second Thursday of each month, at 
> different locations around Chicagoland.  If you are 
> interested in goings-on, please subscribe to our mailing 
> list: http://mail.python.org/mailman/listinfo/chicago
> 
> _______________________________________________
> Chicago mailing list
> Chicago at python.org http://mail.python.org/mailman/listinfo/chicago
> 

From nttaylor at uchicago.edu  Tue Nov  8 18:59:08 2005
From: nttaylor at uchicago.edu (Noel Thomas Taylor)
Date: Tue, 8 Nov 2005 11:59:08 -0600 (CST)
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <436FEF42.2050207@colorstudy.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
Message-ID: <Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>


Dear ChiPy,

Thank you for your responses. Ian, I looked at the proc_communicate() 
function you pointed me to, and sent it a command to run a short python 
program "ticker", which just prints the word "tick" once a second for 10 
seconds.

But I couldn't make it capture the output as it was coming out - my only 
choice was to see all of the output at once after the process had 
finished.

Did I understand you correctly that proc_communicate() should be able to 
see unbuffered output as the subprocess is running?

I've tried to make this work in my own experiments by using fcntl to set 
the file descriptors on the child's stdout fileobject to non-blocking and 
then using select.select to report when the stdout file object is ready to 
be read from, but I always get a "resource temporarily unavailable" error 
when I try to actually read from stdout.

Martin Maney, who also responded to my question, wrote that such behavior 
is not possible without the use of pseudo-terminals (assuming you cannot 
alter the code of the child process itself to produce unbuffered output). 
In this case my experiments are doomed from the outset. Do you agree?

One more question: capturing the output "live" isn't so important by 
itself. The real issue is that if the child process hangs and needs to be 
aborted, whatever output it produced before it got hung can be preserved. 
Do you guys know if it is possible to get the child process's output out 
of whatever buffer it's in, in the event that the child has to be killed 
by the parent?

with thanks for your help,

Noel Taylor




On Mon, 7 Nov 2005, Ian Bicking wrote:

> Noel Thomas Taylor wrote:
>> I just joined the list and hope to come to the meeting this Thursday. If I
>> can't, however, I wonder if someone could help me with a very tricky
>> problem...
>>
>> A couple of months ago I started trying to find a way to launch a
>> subprocess in python that would allow the parent to capture the stdout
>> and stderr of the child, time the duration of the child, and abort the
>> child if it had failed to produce output after 'n' seconds. Also, the
>> solution should perform consistently across all Unices (Windows doesn't
>> matter) and if the child must be aborted, whatever output it produced up
>> to that point should still be captured.
>>
>> There are several solutions out there, but most do not capture the output
>> of the child in real time. You always have to wait for the child to
>> terminate until you can see any of its output, and consequently if the
>> child must be aborted because it gets stuck, you lose whatever output it
>> generated before you killed it.
>
> The subprocess module can do this.  It's a little funny, because it
> doesn't allow streaming using proc.communicate(), and I find it
> difficult to do normal piping because of the blocking.
>
> But if you look at the bottom of this file there's a function
> proc_communicate allows you to communicate using Python file-like objects:
>
>   http://svn.pythonpaste.org/Paste/trunk/paste/cgiapp.py
>
> It's basically a slight rewrite of Proc.communicate().  There's already
> some patches filed for this issue, so hopefully subprocess in Python 2.5
> will make this easier.
>
>
> -- 
> Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>

From ianb at colorstudy.com  Tue Nov  8 19:06:52 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 08 Nov 2005 12:06:52 -0600
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
Message-ID: <4370E93C.7000906@colorstudy.com>

Noel Thomas Taylor wrote:
> Dear ChiPy,
> 
> Thank you for your responses. Ian, I looked at the proc_communicate() 
> function you pointed me to, and sent it a command to run a short python 
> program "ticker", which just prints the word "tick" once a second for 10 
> seconds.
> 
> But I couldn't make it capture the output as it was coming out - my only 
> choice was to see all of the output at once after the process had 
> finished.
> 
> Did I understand you correctly that proc_communicate() should be able to 
> see unbuffered output as the subprocess is running?

I would have guessed so, but I never tried.  You might try printing a 
much bigger string every second -- one that should fill the buffer -- 
and see what happens.  I honestly don't know much about file buffering 
and the peculiarities of subprocess communication, maybe Martin will 
have more ideas than I.  I was just focused on something like 
Proc.communicate that wouldn't absolutely require getting the entirely 
stdout result at once as a string -- but I just wanted to save memory in 
that case.

-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org

From nttaylor at uchicago.edu  Tue Nov  8 19:37:19 2005
From: nttaylor at uchicago.edu (Noel Thomas Taylor)
Date: Tue, 8 Nov 2005 12:37:19 -0600 (CST)
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <4370E93C.7000906@colorstudy.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
Message-ID: <Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>


Hi Ian,

I could try that, but in the case of the real application whose output I 
want to capture, I have no control over how much output it produces.

Do you have any thoughts about recapturing the output of an aborted child 
process before the memory that is buffering its output gets blown away?

with thanks,

Noel Taylor

>> Did I understand you correctly that proc_communicate() should be able to
>> see unbuffered output as the subprocess is running?
>
> I would have guessed so, but I never tried.  You might try printing a
> much bigger string every second -- one that should fill the buffer --
> and see what happens.  I honestly don't know much about file buffering
> and the peculiarities of subprocess communication, maybe Martin will
> have more ideas than I.  I was just focused on something like
> Proc.communicate that wouldn't absolutely require getting the entirely
> stdout result at once as a string -- but I just wanted to save memory in
> that case.
>
> -- 
> Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>

From ianb at colorstudy.com  Tue Nov  8 19:40:04 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Tue, 08 Nov 2005 12:40:04 -0600
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>	<436FEF42.2050207@colorstudy.com>	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
Message-ID: <4370F104.2050205@colorstudy.com>

Noel Thomas Taylor wrote:
> Hi Ian,
> 
> I could try that, but in the case of the real application whose output I 
> want to capture, I have no control over how much output it produces.

I thought it would be an interesting test to understand exactly what is 
going on, even if it isn't exactly what you are expecting to receive.

> Do you have any thoughts about recapturing the output of an aborted child 
> process before the memory that is buffering its output gets blown away?

Since it is OS buffering, shouldn't the OS handle that for you?  I don't 
know; I would suggest starting with a test, then finding something that 
passes that test.

-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org

From jbalint at gmail.com  Tue Nov  8 20:47:43 2005
From: jbalint at gmail.com (Jess Balint)
Date: Tue, 8 Nov 2005 14:47:43 -0500
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <4370F104.2050205@colorstudy.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
Message-ID: <68cb949d0511081147t7a6d507crd56a0f6fd0ac6945@mail.gmail.com>

I made a prototype you can use. It's a simple combination of creating
pipe()s for stdin and stderr, then dup2 them into the streams after a
fork(), before an exec(). I've attach the python and a test shell
script (that kills itself). You should be able to see it capture the
output. (If there is a problem with the attachments, I will put them
on a web site or something.)

Jess

On 11/8/05, Ian Bicking <ianb at colorstudy.com> wrote:
> Noel Thomas Taylor wrote:
> > Hi Ian,
> >
> > I could try that, but in the case of the real application whose output I
> > want to capture, I have no control over how much output it produces.
>
> I thought it would be an interesting test to understand exactly what is
> going on, even if it isn't exactly what you are expecting to receive.
>
> > Do you have any thoughts about recapturing the output of an aborted child
> > process before the memory that is buffering its output gets blown away?
>
> Since it is OS buffering, shouldn't the OS handle that for you?  I don't
> know; I would suggest starting with a test, then finding something that
> passes that test.
>
> --
> Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: child_process.py
Type: application/octet-stream
Size: 883 bytes
Desc: not available
Url : http://mail.python.org/pipermail/chicago/attachments/20051108/20897b1f/child_process.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testsh.sh
Type: application/x-sh
Size: 116 bytes
Desc: not available
Url : http://mail.python.org/pipermail/chicago/attachments/20051108/20897b1f/testsh.sh

From maney at two14.net  Tue Nov  8 21:17:05 2005
From: maney at two14.net (Martin Maney)
Date: Tue, 8 Nov 2005 14:17:05 -0600
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <4370F104.2050205@colorstudy.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
Message-ID: <20051108201705.GA482@furrr.two14.net>

On Tue, Nov 08, 2005 at 12:40:04PM -0600, Ian Bicking wrote:
> Noel Thomas Taylor wrote:
> > Do you have any thoughts about recapturing the output of an aborted child 
> > process before the memory that is buffering its output gets blown away?
> 
> Since it is OS buffering, shouldn't the OS handle that for you?  I don't 
> know; I would suggest starting with a test, then finding something that 
> passes that test.

Ah, but that's just it - it's NOT the OS that's doing the bufferring. 
C's standard IO library uses bufferred IO for efficency - it's much
faster than doing a system call for every character.  Back in the old
days the buffer was often only 512 bytes, but I'm sure that modern
library implementations will allocate a larger buffer by default.

No, I'm very much afraid that the problem is that the apps are
internally buffering the output, and that buffer isn't flushed by an
abnormal termination (which is actually a feature, I'm afraid).  I have
since thought that you might be able to change this normal bufferring
in the C library by playing games with LD_PRELOAD and a modified (and
subsetted, I should think) library, but that's probably no more
portable, and a good deal more work, than PTYs.

If you're willing to use solutions that aren't portable, I have a vague
notion that the GNU libc, at least, might provide a hack to control
this buffering through the environment.  Though I can't find anything
about it now, so maybe that was some other implementation...

-- 
If software vendors have liability costs, they'll pass those
on to us.  It might not be cheaper than what we're paying today,
but as long as we're going to pay, we might as well
pay to fix the problem.  -- Bruce Schneier


From david at graniteweb.com  Tue Nov  8 21:34:21 2005
From: david at graniteweb.com (David Rock)
Date: Tue, 8 Nov 2005 14:34:21 -0600
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <20051108201705.GA482@furrr.two14.net>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<20051108201705.GA482@furrr.two14.net>
Message-ID: <20051108203421.GA17777@wdfs.graniteweb.com>

* Martin Maney <maney at two14.net> [2005-11-08 14:17]:
> On Tue, Nov 08, 2005 at 12:40:04PM -0600, Ian Bicking wrote:
> > Noel Thomas Taylor wrote:
> > > Do you have any thoughts about recapturing the output of an aborted child 
> > > process before the memory that is buffering its output gets blown away?
> > 
> > Since it is OS buffering, shouldn't the OS handle that for you?  I don't 
> > know; I would suggest starting with a test, then finding something that 
> > passes that test.
> 
> Ah, but that's just it - it's NOT the OS that's doing the bufferring. 
> C's standard IO library uses bufferred IO for efficency - it's much
> faster than doing a system call for every character.  Back in the old
> days the buffer was often only 512 bytes, but I'm sure that modern
> library implementations will allocate a larger buffer by default.

I thought there was a way to manually flush the buffer.  I can't think of
what it is offhand, though.

-- 
David Rock
david at graniteweb.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mail.python.org/pipermail/chicago/attachments/20051108/7f95236d/attachment.pgp

From jbalint at gmail.com  Tue Nov  8 21:38:30 2005
From: jbalint at gmail.com (Jess Balint)
Date: Tue, 8 Nov 2005 15:38:30 -0500
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <20051108203421.GA17777@wdfs.graniteweb.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<20051108201705.GA482@furrr.two14.net>
	<20051108203421.GA17777@wdfs.graniteweb.com>
Message-ID: <68cb949d0511081238w7d82d21ey8f01311cf9564a2f@mail.gmail.com>

> I thought there was a way to manually flush the buffer.  I can't think of
> what it is offhand, though.

fflush() for streams. However you need to call this in the program and
if you die without a chance to cleanup this becomes impossible.

Jess

From chris.mcavoy at gmail.com  Tue Nov  8 21:56:18 2005
From: chris.mcavoy at gmail.com (Chris McAvoy)
Date: Tue, 8 Nov 2005 14:56:18 -0600
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <20051108203421.GA17777@wdfs.graniteweb.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<20051108201705.GA482@furrr.two14.net>
	<20051108203421.GA17777@wdfs.graniteweb.com>
Message-ID: <3096c19d0511081256v4fa4695mdb2b975ba3c088bc@mail.gmail.com>

> I thought there was a way to manually flush the buffer.  I can't think of
> what it is offhand, though.

Jiggle the handle.

Chris

PS. Get it?  Flush?  It's a toilet joke.

From bray at sent.com  Tue Nov  8 22:29:16 2005
From: bray at sent.com (Brian Ray)
Date: Tue, 8 Nov 2005 15:29:16 -0600
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <3096c19d0511081256v4fa4695mdb2b975ba3c088bc@mail.gmail.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<20051108201705.GA482@furrr.two14.net>
	<20051108203421.GA17777@wdfs.graniteweb.com>
	<3096c19d0511081256v4fa4695mdb2b975ba3c088bc@mail.gmail.com>
Message-ID: <21130E02-65B1-4BE1-B024-58A38A8D569A@sent.com>


On Nov 8, 2005, at 2:56 PM, Chris McAvoy wrote:

>> I thought there was a way to manually flush the buffer.  I can't  
>> think of
>> what it is offhand, though.
>>
>
> Jiggle the handle.

First, insert Chris's head! :p

Anyway, Jess, this example is really cool.  And in Python, great.  I  
hope you can make it to the meeting.

I thought the subprocess module took care of some of this in 2.4-- 
although, I have not really checked because usually I only want to  
know if it worked or not. I tend to do the same thing with 'C++'  
quite often. Basically, you can stream stderr also. And you can set  
the buffer size <http://www.cplusplus.com/ref/cstdio/setvbuf.html>.

I am not sure if Threads will help you at all because it sounds like  
the process is not that of Python. If it is, Threads are really cool.  
They can also be controlled from Python's C API.

One possible work around with capturing errors with Python is to  
override the Error handling:

# got this from some other Python forum, can't recall where
# redirects errors to file

import os,sys

def eh(klass, inst, tb):
    sys.stderr = file('/tmp/pyerror.log', 'w')
    sys.__excepthook__(klass, inst, tb)
    sys.stderr.close()
    sys.stderr = sys.__stderr__

sys.excepthook = eh


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/chicago/attachments/20051108/b5445cfe/attachment.htm

From chris.mcavoy at gmail.com  Wed Nov  9 19:57:13 2005
From: chris.mcavoy at gmail.com (Chris McAvoy)
Date: Wed, 9 Nov 2005 12:57:13 -0600
Subject: [Chicago] Thursday Meeting
Message-ID: <3096c19d0511091057x8c4aaddra1cf7b731260e22f@mail.gmail.com>

For the presenters: please plan on not having internet access.  If
possible, we'll get something going for those that can't be offline
for a short time (seriously, don't you people go outside?).

However, for presentation purposes, plan on being offline.  If you
don't have a laptop of your own, bring your presentation on some sort
of media and we should be able to help you out.

The RSVP list is looking pretty good.  However, I think there's some
folks out there that are planning on coming, but haven't RSVP'd.  I'll
talk to building security about arranging to call me if people wander
in asking about Pythons.

Chris

From clint at robotic.com  Wed Nov  9 22:21:15 2005
From: clint at robotic.com (clint@robotic.com)
Date: Wed, 9 Nov 2005 16:21:15 -0500 (EST)
Subject: [Chicago] Thursday Meeting
In-Reply-To: <3096c19d0511091057x8c4aaddra1cf7b731260e22f@mail.gmail.com>
References: <3096c19d0511091057x8c4aaddra1cf7b731260e22f@mail.gmail.com>
Message-ID: <10765.66.84.150.221.1131571275.squirrel@webmail1.hrnoc.net>

I'm considering coming down from Milwaukee along with one other person.
Due to time involved, I can't be 100% sure we'll be there or that we'll be
there on time.

-- Clint Laskowski
-- clint at robotic.com


> For the presenters: please plan on not having internet access.  If
> possible, we'll get something going for those that can't be offline
> for a short time (seriously, don't you people go outside?).
>
> However, for presentation purposes, plan on being offline.  If you
> don't have a laptop of your own, bring your presentation on some sort
> of media and we should be able to help you out.
>
> The RSVP list is looking pretty good.  However, I think there's some
> folks out there that are planning on coming, but haven't RSVP'd.  I'll
> talk to building security about arranging to call me if people wander
> in asking about Pythons.
>
> Chris
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>



From nttaylor at uchicago.edu  Thu Nov 10 18:07:56 2005
From: nttaylor at uchicago.edu (Noel Thomas Taylor)
Date: Thu, 10 Nov 2005 11:07:56 -0600 (CST)
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <68cb949d0511081147t7a6d507crd56a0f6fd0ac6945@mail.gmail.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<68cb949d0511081147t7a6d507crd56a0f6fd0ac6945@mail.gmail.com>
Message-ID: <Pine.GSO.4.62.0511101052510.8603@harper.uchicago.edu>


Hi Jess,

Thanks for this great example. I've been experimenting with it, and it 
could be the answer to my prayers. One question about forking: I know that 
when you do a fork() call, the code gets duplicated in memory, the child 
gets its own pid as the return value of the fork, and the parent gets 
nothing.

But how much code gets duplicated, and can a single fork call 
significantly impact your memory? In your "child_process.py" for 
example, does the whole module get duplicated? If this function were 
just one in a giant file thousands of lines long, would that whole file 
get duplicated? If your code has a call to fork() in it, does that mean 
you should isolate it into a smaller module which you then import, or does 
that make a difference?

Or maybe the duplication is virtual and the two processes are really 
occupying the same memory space?

I'm sorry I can't make the meeting tonight.

with thanks,

Noel Taylor



On Tue, 8 Nov 2005, Jess Balint wrote:

> I made a prototype you can use. It's a simple combination of creating
> pipe()s for stdin and stderr, then dup2 them into the streams after a
> fork(), before an exec(). I've attach the python and a test shell
> script (that kills itself). You should be able to see it capture the
> output. (If there is a problem with the attachments, I will put them
> on a web site or something.)
>
> Jess
>
> On 11/8/05, Ian Bicking <ianb at colorstudy.com> wrote:
>> Noel Thomas Taylor wrote:
>>> Hi Ian,
>>>
>>> I could try that, but in the case of the real application whose output I
>>> want to capture, I have no control over how much output it produces.
>>
>> I thought it would be an interesting test to understand exactly what is
>> going on, even if it isn't exactly what you are expecting to receive.
>>
>>> Do you have any thoughts about recapturing the output of an aborted child
>>> process before the memory that is buffering its output gets blown away?
>>
>> Since it is OS buffering, shouldn't the OS handle that for you?  I don't
>> know; I would suggest starting with a test, then finding something that
>> passes that test.
>>
>> --
>> Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org
>> _______________________________________________
>> Chicago mailing list
>> Chicago at python.org
>> http://mail.python.org/mailman/listinfo/chicago
>>
>

From jbalint at gmail.com  Thu Nov 10 18:18:29 2005
From: jbalint at gmail.com (Jess Balint)
Date: Thu, 10 Nov 2005 12:18:29 -0500
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <Pine.GSO.4.62.0511101052510.8603@harper.uchicago.edu>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<68cb949d0511081147t7a6d507crd56a0f6fd0ac6945@mail.gmail.com>
	<Pine.GSO.4.62.0511101052510.8603@harper.uchicago.edu>
Message-ID: <68cb949d0511100918r42d63250r7d35ef2840826d48@mail.gmail.com>

That's a valid concern. Initially there will be an increase in memory.
On most systems, this is pretty efficient and all parent process
memory is copy-on-write in the child. However, since you exec() so
soon the memory will be replaced by that of the image of the other
program. So in effect the end result when you run the other program is
no more memory than system() (or possibly even less due to no /bin/sh
overhead if it's not a shell script). How many processes do predict
you will be running at one time?

Jess

On 11/10/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
>
> Hi Jess,
>
> Thanks for this great example. I've been experimenting with it, and it
> could be the answer to my prayers. One question about forking: I know that
> when you do a fork() call, the code gets duplicated in memory, the child
> gets its own pid as the return value of the fork, and the parent gets
> nothing.
>
> But how much code gets duplicated, and can a single fork call
> significantly impact your memory? In your "child_process.py" for
> example, does the whole module get duplicated? If this function were
> just one in a giant file thousands of lines long, would that whole file
> get duplicated? If your code has a call to fork() in it, does that mean
> you should isolate it into a smaller module which you then import, or does
> that make a difference?
>
> Or maybe the duplication is virtual and the two processes are really
> occupying the same memory space?
>
> I'm sorry I can't make the meeting tonight.
>
> with thanks,
>
> Noel Taylor
>
>
>
> On Tue, 8 Nov 2005, Jess Balint wrote:
>
> > I made a prototype you can use. It's a simple combination of creating
> > pipe()s for stdin and stderr, then dup2 them into the streams after a
> > fork(), before an exec(). I've attach the python and a test shell
> > script (that kills itself). You should be able to see it capture the
> > output. (If there is a problem with the attachments, I will put them
> > on a web site or something.)
> >
> > Jess
> >
> > On 11/8/05, Ian Bicking <ianb at colorstudy.com> wrote:
> >> Noel Thomas Taylor wrote:
> >>> Hi Ian,
> >>>
> >>> I could try that, but in the case of the real application whose output I
> >>> want to capture, I have no control over how much output it produces.
> >>
> >> I thought it would be an interesting test to understand exactly what is
> >> going on, even if it isn't exactly what you are expecting to receive.
> >>
> >>> Do you have any thoughts about recapturing the output of an aborted child
> >>> process before the memory that is buffering its output gets blown away?
> >>
> >> Since it is OS buffering, shouldn't the OS handle that for you?  I don't
> >> know; I would suggest starting with a test, then finding something that
> >> passes that test.
> >>
> >> --
> >> Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org
> >> _______________________________________________
> >> Chicago mailing list
> >> Chicago at python.org
> >> http://mail.python.org/mailman/listinfo/chicago
> >>
> >
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>

From nttaylor at uchicago.edu  Thu Nov 10 19:35:09 2005
From: nttaylor at uchicago.edu (Noel Thomas Taylor)
Date: Thu, 10 Nov 2005 12:35:09 -0600 (CST)
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <68cb949d0511100918r42d63250r7d35ef2840826d48@mail.gmail.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<68cb949d0511081147t7a6d507crd56a0f6fd0ac6945@mail.gmail.com>
	<Pine.GSO.4.62.0511101052510.8603@harper.uchicago.edu>
	<68cb949d0511100918r42d63250r7d35ef2840826d48@mail.gmail.com>
Message-ID: <Pine.GSO.4.62.0511101233560.21125@harper.uchicago.edu>


The program should never be running more than a single child-process at a 
time.

On Thu, 10 Nov 2005, Jess Balint wrote:

> That's a valid concern. Initially there will be an increase in memory.
> On most systems, this is pretty efficient and all parent process
> memory is copy-on-write in the child. However, since you exec() so
> soon the memory will be replaced by that of the image of the other
> program. So in effect the end result when you run the other program is
> no more memory than system() (or possibly even less due to no /bin/sh
> overhead if it's not a shell script). How many processes do predict
> you will be running at one time?
>
> Jess
>
> On 11/10/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
>>
>> Hi Jess,
>>
>> Thanks for this great example. I've been experimenting with it, and it
>> could be the answer to my prayers. One question about forking: I know that
>> when you do a fork() call, the code gets duplicated in memory, the child
>> gets its own pid as the return value of the fork, and the parent gets
>> nothing.
>>
>> But how much code gets duplicated, and can a single fork call
>> significantly impact your memory? In your "child_process.py" for
>> example, does the whole module get duplicated? If this function were
>> just one in a giant file thousands of lines long, would that whole file
>> get duplicated? If your code has a call to fork() in it, does that mean
>> you should isolate it into a smaller module which you then import, or does
>> that make a difference?
>>
>> Or maybe the duplication is virtual and the two processes are really
>> occupying the same memory space?
>>
>> I'm sorry I can't make the meeting tonight.
>>
>> with thanks,
>>
>> Noel Taylor
>>
>>
>>
>> On Tue, 8 Nov 2005, Jess Balint wrote:
>>
>>> I made a prototype you can use. It's a simple combination of creating
>>> pipe()s for stdin and stderr, then dup2 them into the streams after a
>>> fork(), before an exec(). I've attach the python and a test shell
>>> script (that kills itself). You should be able to see it capture the
>>> output. (If there is a problem with the attachments, I will put them
>>> on a web site or something.)
>>>
>>> Jess
>>>
>>> On 11/8/05, Ian Bicking <ianb at colorstudy.com> wrote:
>>>> Noel Thomas Taylor wrote:
>>>>> Hi Ian,
>>>>>
>>>>> I could try that, but in the case of the real application whose output I
>>>>> want to capture, I have no control over how much output it produces.
>>>>
>>>> I thought it would be an interesting test to understand exactly what is
>>>> going on, even if it isn't exactly what you are expecting to receive.
>>>>
>>>>> Do you have any thoughts about recapturing the output of an aborted child
>>>>> process before the memory that is buffering its output gets blown away?
>>>>
>>>> Since it is OS buffering, shouldn't the OS handle that for you?  I don't
>>>> know; I would suggest starting with a test, then finding something that
>>>> passes that test.
>>>>
>>>> --
>>>> Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org
>>>> _______________________________________________
>>>> Chicago mailing list
>>>> Chicago at python.org
>>>> http://mail.python.org/mailman/listinfo/chicago
>>>>
>>>
>> _______________________________________________
>> Chicago mailing list
>> Chicago at python.org
>> http://mail.python.org/mailman/listinfo/chicago
>>
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>

From nttaylor at uchicago.edu  Fri Nov 11 20:24:04 2005
From: nttaylor at uchicago.edu (Noel Thomas Taylor)
Date: Fri, 11 Nov 2005 13:24:04 -0600 (CST)
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <68cb949d0511100918r42d63250r7d35ef2840826d48@mail.gmail.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<68cb949d0511081147t7a6d507crd56a0f6fd0ac6945@mail.gmail.com>
	<Pine.GSO.4.62.0511101052510.8603@harper.uchicago.edu>
	<68cb949d0511100918r42d63250r7d35ef2840826d48@mail.gmail.com>
Message-ID: <Pine.GSO.4.62.0511111310550.28525@harper.uchicago.edu>


Hi Jess,

I've been experimenting with the code you sent, in which the child kills 
itself, and its output is captured by the parent.

But I have not been successful in having the parent kill the child in 
mid-run while still capturing whatever output it had produced up to that 
point.

I added a timeout parameter to the select call. If the timeout happens 
before the child appears in the lists returned by "select", the parent 
kills the child. When I look at the results afterward, however, nothing 
has been sent to the output stream.

I tried making the output and error streams non-blocking with 'fcntl'. 
I've seen this kind of thing in a lot of code I've found on the internet 
where people are discussing this kind of topic, but it's never helped my 
cause so far. I'm I barking up the wrong tree with this?

Please take a look at my code. I've attached 'my_child_process.py' which 
is just the script you sent me before plus the above modifications, and 
'ticker.py' which prints the word 'tick' once a second for 10 seconds.

In an ideal world, I'd be able to set the timeout for say, five seconds, 
and run the program. The program would then timeout after the child had 
produced five "ticks", and I'd be able to capture those "ticks" and still 
kill the child.

What do you think? Can this be done? Does 'fcntl' have anything to do 
with this problem? You sounded optimistic about it before, so I'm still 
holding out hope.

with great thanks,

Noel Taylor

On Thu, 10 Nov 2005, Jess Balint wrote:

> That's a valid concern. Initially there will be an increase in memory.
> On most systems, this is pretty efficient and all parent process
> memory is copy-on-write in the child. However, since you exec() so
> soon the memory will be replaced by that of the image of the other
> program. So in effect the end result when you run the other program is
> no more memory than system() (or possibly even less due to no /bin/sh
> overhead if it's not a shell script). How many processes do predict
> you will be running at one time?
>
> Jess
>
> On 11/10/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
>>
>> Hi Jess,
>>
>> Thanks for this great example. I've been experimenting with it, and it
>> could be the answer to my prayers. One question about forking: I know that
>> when you do a fork() call, the code gets duplicated in memory, the child
>> gets its own pid as the return value of the fork, and the parent gets
>> nothing.
>>
>> But how much code gets duplicated, and can a single fork call
>> significantly impact your memory? In your "child_process.py" for
>> example, does the whole module get duplicated? If this function were
>> just one in a giant file thousands of lines long, would that whole file
>> get duplicated? If your code has a call to fork() in it, does that mean
>> you should isolate it into a smaller module which you then import, or does
>> that make a difference?
>>
>> Or maybe the duplication is virtual and the two processes are really
>> occupying the same memory space?
>>
>> I'm sorry I can't make the meeting tonight.
>>
>> with thanks,
>>
>> Noel Taylor
>>
>>
>>
>> On Tue, 8 Nov 2005, Jess Balint wrote:
>>
>>> I made a prototype you can use. It's a simple combination of creating
>>> pipe()s for stdin and stderr, then dup2 them into the streams after a
>>> fork(), before an exec(). I've attach the python and a test shell
>>> script (that kills itself). You should be able to see it capture the
>>> output. (If there is a problem with the attachments, I will put them
>>> on a web site or something.)
>>>
>>> Jess
>>>
>>> On 11/8/05, Ian Bicking <ianb at colorstudy.com> wrote:
>>>> Noel Thomas Taylor wrote:
>>>>> Hi Ian,
>>>>>
>>>>> I could try that, but in the case of the real application whose output I
>>>>> want to capture, I have no control over how much output it produces.
>>>>
>>>> I thought it would be an interesting test to understand exactly what is
>>>> going on, even if it isn't exactly what you are expecting to receive.
>>>>
>>>>> Do you have any thoughts about recapturing the output of an aborted child
>>>>> process before the memory that is buffering its output gets blown away?
>>>>
>>>> Since it is OS buffering, shouldn't the OS handle that for you?  I don't
>>>> know; I would suggest starting with a test, then finding something that
>>>> passes that test.
>>>>
>>>> --
>>>> Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org
>>>> _______________________________________________
>>>> Chicago mailing list
>>>> Chicago at python.org
>>>> http://mail.python.org/mailman/listinfo/chicago
>>>>
>>>
>> _______________________________________________
>> Chicago mailing list
>> Chicago at python.org
>> http://mail.python.org/mailman/listinfo/chicago
>>
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>
-------------- next part --------------
#!/usr/bin/env python

from os import pipe, fork, dup2, execv, kill, read, close, O_NDELAY
import fcntl, sys, signal
from select import select

def runProcess(file, timeout=None):
  (outpi, outpo) = pipe()  # returns pair of file descriptors
  (errpi, errpo) = pipe()

  cpid = fork()

  if cpid:
    output = ""
    error = ""

    # parent process
    close(outpo)
    close(errpo)

    # is this necessary? does it do anything?
    outpiflags = fcntl.fcntl(outpi, fcntl.F_GETFL)
    fcntl.fcntl(outpi, fcntl.F_SETFL, outpiflags | O_NDELAY)
    errpiflags = fcntl.fcntl(errpi, fcntl.F_GETFL)
    fcntl.fcntl(errpi, fcntl.F_SETFL, errpiflags | O_NDELAY)

    while 1:
      (rl, wl, xl) = select([outpi, errpi], [], [outpi, errpi], timeout)
      end = 0
      if len(rl) == 0 and len(wl) == 0 and len(xl) == 0:
        print "child aborted"
        kill(cpid, signal.SIGKILL)
        end = 1
      else:
        for f in rl:
          if f == outpi:
            o = read(f, 100)
            if o: output += o
            else: end = 1
          elif f == errpi:
            e = read(f, 100)
            if e: error += e
            else: end = 1

      if end: break

    return (output, error)
  else:
    # child process
    close(outpi)
    close(errpi)
    sys.stdout.flush()
    sys.stderr.flush()
    dup2(outpo, sys.stdout.fileno())
    dup2(errpo, sys.stderr.fileno())
    execv(file, ())

(out, err) = runProcess("./ticker.py", 5)

print "Output stream:", out
print "Error stream:", err

-------------- next part --------------
#!/usr/bin/env python

import time

for i in range(10):
  print "tick"
  time.sleep(1)

From jbalint at gmail.com  Fri Nov 11 21:13:42 2005
From: jbalint at gmail.com (Jess Balint)
Date: Fri, 11 Nov 2005 15:13:42 -0500
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <Pine.GSO.4.62.0511111310550.28525@harper.uchicago.edu>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<68cb949d0511081147t7a6d507crd56a0f6fd0ac6945@mail.gmail.com>
	<Pine.GSO.4.62.0511101052510.8603@harper.uchicago.edu>
	<68cb949d0511100918r42d63250r7d35ef2840826d48@mail.gmail.com>
	<Pine.GSO.4.62.0511111310550.28525@harper.uchicago.edu>
Message-ID: <68cb949d0511111213v110d0e22qac66a2a19f3bf49b@mail.gmail.com>

What you are seeing here is the stdio lib buffering. When the output
is a terminal, you will get line buffering. Since we are using a pipe,
it will switch to block buffering. To get around this in a Python
program, you can use the '-u' command line option or set the env var
PYTHONUNBUFFERED (both documented in man python). This is the
equivalent of calling setbuf(1, NULL).

The non-blocking io is not relevant to any of this.

Jess

On 11/11/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
>
> Hi Jess,
>
> I've been experimenting with the code you sent, in which the child kills
> itself, and its output is captured by the parent.
>
> But I have not been successful in having the parent kill the child in
> mid-run while still capturing whatever output it had produced up to that
> point.
>
> I added a timeout parameter to the select call. If the timeout happens
> before the child appears in the lists returned by "select", the parent
> kills the child. When I look at the results afterward, however, nothing
> has been sent to the output stream.
>
> I tried making the output and error streams non-blocking with 'fcntl'.
> I've seen this kind of thing in a lot of code I've found on the internet
> where people are discussing this kind of topic, but it's never helped my
> cause so far. I'm I barking up the wrong tree with this?
>
> Please take a look at my code. I've attached 'my_child_process.py' which
> is just the script you sent me before plus the above modifications, and
> 'ticker.py' which prints the word 'tick' once a second for 10 seconds.
>
> In an ideal world, I'd be able to set the timeout for say, five seconds,
> and run the program. The program would then timeout after the child had
> produced five "ticks", and I'd be able to capture those "ticks" and still
> kill the child.
>
> What do you think? Can this be done? Does 'fcntl' have anything to do
> with this problem? You sounded optimistic about it before, so I'm still
> holding out hope.
>
> with great thanks,
>
> Noel Taylor
>
> On Thu, 10 Nov 2005, Jess Balint wrote:
>
> > That's a valid concern. Initially there will be an increase in memory.
> > On most systems, this is pretty efficient and all parent process
> > memory is copy-on-write in the child. However, since you exec() so
> > soon the memory will be replaced by that of the image of the other
> > program. So in effect the end result when you run the other program is
> > no more memory than system() (or possibly even less due to no /bin/sh
> > overhead if it's not a shell script). How many processes do predict
> > you will be running at one time?
> >
> > Jess
> >
> > On 11/10/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
> >>
> >> Hi Jess,
> >>
> >> Thanks for this great example. I've been experimenting with it, and it
> >> could be the answer to my prayers. One question about forking: I know that
> >> when you do a fork() call, the code gets duplicated in memory, the child
> >> gets its own pid as the return value of the fork, and the parent gets
> >> nothing.
> >>
> >> But how much code gets duplicated, and can a single fork call
> >> significantly impact your memory? In your "child_process.py" for
> >> example, does the whole module get duplicated? If this function were
> >> just one in a giant file thousands of lines long, would that whole file
> >> get duplicated? If your code has a call to fork() in it, does that mean
> >> you should isolate it into a smaller module which you then import, or does
> >> that make a difference?
> >>
> >> Or maybe the duplication is virtual and the two processes are really
> >> occupying the same memory space?
> >>
> >> I'm sorry I can't make the meeting tonight.
> >>
> >> with thanks,
> >>
> >> Noel Taylor
> >>
> >>
> >>
> >> On Tue, 8 Nov 2005, Jess Balint wrote:
> >>
> >>> I made a prototype you can use. It's a simple combination of creating
> >>> pipe()s for stdin and stderr, then dup2 them into the streams after a
> >>> fork(), before an exec(). I've attach the python and a test shell
> >>> script (that kills itself). You should be able to see it capture the
> >>> output. (If there is a problem with the attachments, I will put them
> >>> on a web site or something.)
> >>>
> >>> Jess
> >>>
> >>> On 11/8/05, Ian Bicking <ianb at colorstudy.com> wrote:
> >>>> Noel Thomas Taylor wrote:
> >>>>> Hi Ian,
> >>>>>
> >>>>> I could try that, but in the case of the real application whose output I
> >>>>> want to capture, I have no control over how much output it produces.
> >>>>
> >>>> I thought it would be an interesting test to understand exactly what is
> >>>> going on, even if it isn't exactly what you are expecting to receive.
> >>>>
> >>>>> Do you have any thoughts about recapturing the output of an aborted child
> >>>>> process before the memory that is buffering its output gets blown away?
> >>>>
> >>>> Since it is OS buffering, shouldn't the OS handle that for you?  I don't
> >>>> know; I would suggest starting with a test, then finding something that
> >>>> passes that test.
> >>>>
> >>>> --
> >>>> Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org
> >>>> _______________________________________________
> >>>> Chicago mailing list
> >>>> Chicago at python.org
> >>>> http://mail.python.org/mailman/listinfo/chicago
> >>>>
> >>>
> >> _______________________________________________
> >> Chicago mailing list
> >> Chicago at python.org
> >> http://mail.python.org/mailman/listinfo/chicago
> >>
> > _______________________________________________
> > Chicago mailing list
> > Chicago at python.org
> > http://mail.python.org/mailman/listinfo/chicago
> >
>
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>
>
>
>

From nttaylor at uchicago.edu  Fri Nov 11 21:33:06 2005
From: nttaylor at uchicago.edu (Noel Thomas Taylor)
Date: Fri, 11 Nov 2005 14:33:06 -0600 (CST)
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <68cb949d0511111213v110d0e22qac66a2a19f3bf49b@mail.gmail.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<436FEF42.2050207@colorstudy.com>
	<Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<68cb949d0511081147t7a6d507crd56a0f6fd0ac6945@mail.gmail.com>
	<Pine.GSO.4.62.0511101052510.8603@harper.uchicago.edu>
	<68cb949d0511100918r42d63250r7d35ef2840826d48@mail.gmail.com>
	<Pine.GSO.4.62.0511111310550.28525@harper.uchicago.edu>
	<68cb949d0511111213v110d0e22qac66a2a19f3bf49b@mail.gmail.com>
Message-ID: <Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu>


On Fri, 11 Nov 2005, Jess Balint wrote:

> What you are seeing here is the stdio lib buffering. When the output
> is a terminal, you will get line buffering. Since we are using a pipe,
> it will switch to block buffering. To get around this in a Python
> program, you can use the '-u' command line option or set the env var
> PYTHONUNBUFFERED (both documented in man python). This is the
> equivalent of calling setbuf(1, NULL).

The child process in the real application I want to use this for is
written in fortran, so I can't use the techniques you mentioned.

I had already sought out second and third opinions on this, so now I am 
really convinced that there is no way to get the effect I want without 
using pseudo-terminals.

> The non-blocking io is not relevant to any of this.

Too bad. If I can ask you one more thing: what is the proper purpose of 
using fcntl to set a file descriptor to non-blocking IO if not to achieve 
the behavior I'm looking for in this problem?

with thanks,

Noel Taylor

>
> Jess
>
> On 11/11/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
>>
>> Hi Jess,
>>
>> I've been experimenting with the code you sent, in which the child kills
>> itself, and its output is captured by the parent.
>>
>> But I have not been successful in having the parent kill the child in
>> mid-run while still capturing whatever output it had produced up to that
>> point.
>>
>> I added a timeout parameter to the select call. If the timeout happens
>> before the child appears in the lists returned by "select", the parent
>> kills the child. When I look at the results afterward, however, nothing
>> has been sent to the output stream.
>>
>> I tried making the output and error streams non-blocking with 'fcntl'.
>> I've seen this kind of thing in a lot of code I've found on the internet
>> where people are discussing this kind of topic, but it's never helped my
>> cause so far. I'm I barking up the wrong tree with this?
>>
>> Please take a look at my code. I've attached 'my_child_process.py' which
>> is just the script you sent me before plus the above modifications, and
>> 'ticker.py' which prints the word 'tick' once a second for 10 seconds.
>>
>> In an ideal world, I'd be able to set the timeout for say, five seconds,
>> and run the program. The program would then timeout after the child had
>> produced five "ticks", and I'd be able to capture those "ticks" and still
>> kill the child.
>>
>> What do you think? Can this be done? Does 'fcntl' have anything to do
>> with this problem? You sounded optimistic about it before, so I'm still
>> holding out hope.
>>
>> with great thanks,
>>
>> Noel Taylor
>>
>> On Thu, 10 Nov 2005, Jess Balint wrote:
>>
>>> That's a valid concern. Initially there will be an increase in memory.
>>> On most systems, this is pretty efficient and all parent process
>>> memory is copy-on-write in the child. However, since you exec() so
>>> soon the memory will be replaced by that of the image of the other
>>> program. So in effect the end result when you run the other program is
>>> no more memory than system() (or possibly even less due to no /bin/sh
>>> overhead if it's not a shell script). How many processes do predict
>>> you will be running at one time?
>>>
>>> Jess
>>>
>>> On 11/10/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
>>>>
>>>> Hi Jess,
>>>>
>>>> Thanks for this great example. I've been experimenting with it, and it
>>>> could be the answer to my prayers. One question about forking: I know that
>>>> when you do a fork() call, the code gets duplicated in memory, the child
>>>> gets its own pid as the return value of the fork, and the parent gets
>>>> nothing.
>>>>
>>>> But how much code gets duplicated, and can a single fork call
>>>> significantly impact your memory? In your "child_process.py" for
>>>> example, does the whole module get duplicated? If this function were
>>>> just one in a giant file thousands of lines long, would that whole file
>>>> get duplicated? If your code has a call to fork() in it, does that mean
>>>> you should isolate it into a smaller module which you then import, or does
>>>> that make a difference?
>>>>
>>>> Or maybe the duplication is virtual and the two processes are really
>>>> occupying the same memory space?
>>>>
>>>> I'm sorry I can't make the meeting tonight.
>>>>
>>>> with thanks,
>>>>
>>>> Noel Taylor
>>>>
>>>>
>>>>
>>>> On Tue, 8 Nov 2005, Jess Balint wrote:
>>>>
>>>>> I made a prototype you can use. It's a simple combination of creating
>>>>> pipe()s for stdin and stderr, then dup2 them into the streams after a
>>>>> fork(), before an exec(). I've attach the python and a test shell
>>>>> script (that kills itself). You should be able to see it capture the
>>>>> output. (If there is a problem with the attachments, I will put them
>>>>> on a web site or something.)
>>>>>
>>>>> Jess
>>>>>
>>>>> On 11/8/05, Ian Bicking <ianb at colorstudy.com> wrote:
>>>>>> Noel Thomas Taylor wrote:
>>>>>>> Hi Ian,
>>>>>>>
>>>>>>> I could try that, but in the case of the real application whose output I
>>>>>>> want to capture, I have no control over how much output it produces.
>>>>>>
>>>>>> I thought it would be an interesting test to understand exactly what is
>>>>>> going on, even if it isn't exactly what you are expecting to receive.
>>>>>>
>>>>>>> Do you have any thoughts about recapturing the output of an aborted child
>>>>>>> process before the memory that is buffering its output gets blown away?
>>>>>>
>>>>>> Since it is OS buffering, shouldn't the OS handle that for you?  I don't
>>>>>> know; I would suggest starting with a test, then finding something that
>>>>>> passes that test.
>>>>>>
>>>>>> --
>>>>>> Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org
>>>>>> _______________________________________________
>>>>>> Chicago mailing list
>>>>>> Chicago at python.org
>>>>>> http://mail.python.org/mailman/listinfo/chicago
>>>>>>
>>>>>
>>>> _______________________________________________
>>>> Chicago mailing list
>>>> Chicago at python.org
>>>> http://mail.python.org/mailman/listinfo/chicago
>>>>
>>> _______________________________________________
>>> Chicago mailing list
>>> Chicago at python.org
>>> http://mail.python.org/mailman/listinfo/chicago
>>>
>>
>> _______________________________________________
>> Chicago mailing list
>> Chicago at python.org
>> http://mail.python.org/mailman/listinfo/chicago
>>
>>
>>
>>
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>

From maney at two14.net  Fri Nov 11 21:59:21 2005
From: maney at two14.net (Martin Maney)
Date: Fri, 11 Nov 2005 14:59:21 -0600
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu>
References: <Pine.GSO.4.62.0511081141140.10134@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<68cb949d0511081147t7a6d507crd56a0f6fd0ac6945@mail.gmail.com>
	<Pine.GSO.4.62.0511101052510.8603@harper.uchicago.edu>
	<68cb949d0511100918r42d63250r7d35ef2840826d48@mail.gmail.com>
	<Pine.GSO.4.62.0511111310550.28525@harper.uchicago.edu>
	<68cb949d0511111213v110d0e22qac66a2a19f3bf49b@mail.gmail.com>
	<Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu>
Message-ID: <20051111205921.GA11363@furrr.two14.net>

On Fri, Nov 11, 2005 at 02:33:06PM -0600, Noel Thomas Taylor wrote:
> Too bad. If I can ask you one more thing: what is the proper purpose of 
> using fcntl to set a file descriptor to non-blocking IO if not to achieve 
> the behavior I'm looking for in this problem?

fcntl affects the behavior of the low-level (system call) interface. 
There may be some buffering there that would need to be addressed, but
I think not, given that you're concerned with writing to a pipe.  The
problem is the higher-level buffering - what's been referred to as
"stdio" buffering.  I don't know if FORTRAN actually uses the clib
routines behind the scenes, but if not there seems to be something
very, very similar in place.  And it is that buffering that you would
need to manipulate - inside the child program.

-- 
Some people hack for fun, some because they want things their way;
some don't because they can't, and some because they can't be bothered.
Some can make anything work, some could but would rather not,
and some would misconfigure a bowling ball.  -- unknown


From jbalint at gmail.com  Fri Nov 11 22:15:35 2005
From: jbalint at gmail.com (Jess Balint)
Date: Fri, 11 Nov 2005 16:15:35 -0500
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<4370E93C.7000906@colorstudy.com>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<68cb949d0511081147t7a6d507crd56a0f6fd0ac6945@mail.gmail.com>
	<Pine.GSO.4.62.0511101052510.8603@harper.uchicago.edu>
	<68cb949d0511100918r42d63250r7d35ef2840826d48@mail.gmail.com>
	<Pine.GSO.4.62.0511111310550.28525@harper.uchicago.edu>
	<68cb949d0511111213v110d0e22qac66a2a19f3bf49b@mail.gmail.com>
	<Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu>
Message-ID: <68cb949d0511111315q2b3aac3nf60ecd96ae504b54@mail.gmail.com>

> The child process in the real application I want to use this for is
> written in fortran, so I can't use the techniques you mentioned.
>
> I had already sought out second and third opinions on this, so now I am
> really convinced that there is no way to get the effect I want without
> using pseudo-terminals.

Yes, I think that's the only thing you have influence on at this
point. Using forkpty() and connecting it to the pipe might work.
Another option might be to use an LD_PRELOAD with an init routine that
calls setbuf(). I am trying to find a simple, non-intrusive way to do
this, I will let you know if I come up with something.

> > The non-blocking io is not relevant to any of this.
>
> Too bad. If I can ask you one more thing: what is the proper purpose of
> using fcntl to set a file descriptor to non-blocking IO if not to achieve
> the behavior I'm looking for in this problem?

Non-blocking IO is something that allows a read() or write() call to
return when a file descriptor is not ready. So if you are reading from
a network connection and nothing has come through, a normal read()
call would block (and wait for something). A non-blocking attribute
added to the socket will allow a read() call to return so you can do
other stuff until it is ready.

Jess

From jbalint at gmail.com  Sat Nov 12 00:47:33 2005
From: jbalint at gmail.com (Jess Balint)
Date: Fri, 11 Nov 2005 18:47:33 -0500
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <68cb949d0511111315q2b3aac3nf60ecd96ae504b54@mail.gmail.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<68cb949d0511081147t7a6d507crd56a0f6fd0ac6945@mail.gmail.com>
	<Pine.GSO.4.62.0511101052510.8603@harper.uchicago.edu>
	<68cb949d0511100918r42d63250r7d35ef2840826d48@mail.gmail.com>
	<Pine.GSO.4.62.0511111310550.28525@harper.uchicago.edu>
	<68cb949d0511111213v110d0e22qac66a2a19f3bf49b@mail.gmail.com>
	<Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu>
	<68cb949d0511111315q2b3aac3nf60ecd96ae504b54@mail.gmail.com>
Message-ID: <68cb949d0511111547v6618c66eo2649795356dee7a0@mail.gmail.com>

Ok, I've attached a pty version. I've tested it on Linux and Solaris
and it seems to work well. I left your select timeout in there for
illustration, but I'm still not sure what you were getting at with
that. I set the select timeout to '3' and the ticker.py to sleep the
number of seconds as the incrementer. This way you see a few 'ticks'
in the output before the parent kills it. Let me know if this one
works ok and makes sense.

Jess

On 11/11/05, Jess Balint <jbalint at gmail.com> wrote:
> > The child process in the real application I want to use this for is
> > written in fortran, so I can't use the techniques you mentioned.
> >
> > I had already sought out second and third opinions on this, so now I am
> > really convinced that there is no way to get the effect I want without
> > using pseudo-terminals.
>
> Yes, I think that's the only thing you have influence on at this
> point. Using forkpty() and connecting it to the pipe might work.
> Another option might be to use an LD_PRELOAD with an init routine that
> calls setbuf(). I am trying to find a simple, non-intrusive way to do
> this, I will let you know if I come up with something.
>
> > > The non-blocking io is not relevant to any of this.
> >
> > Too bad. If I can ask you one more thing: what is the proper purpose of
> > using fcntl to set a file descriptor to non-blocking IO if not to achieve
> > the behavior I'm looking for in this problem?
>
> Non-blocking IO is something that allows a read() or write() call to
> return when a file descriptor is not ready. So if you are reading from
> a network connection and nothing has come through, a normal read()
> call would block (and wait for something). A non-blocking attribute
> added to the socket will allow a read() call to return so you can do
> other stuff until it is ready.
>
> Jess
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: child_process_3.py
Type: application/octet-stream
Size: 1750 bytes
Desc: not available
Url : http://mail.python.org/pipermail/chicago/attachments/20051111/30f6a3de/child_process_3.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ticker.py
Type: application/octet-stream
Size: 142 bytes
Desc: not available
Url : http://mail.python.org/pipermail/chicago/attachments/20051111/30f6a3de/ticker.obj

From nttaylor at uchicago.edu  Mon Nov 14 22:02:57 2005
From: nttaylor at uchicago.edu (Noel Thomas Taylor)
Date: Mon, 14 Nov 2005 15:02:57 -0600 (CST)
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <Pine.GSO.4.62.0511121854460.29085@harper.uchicago.edu>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<Pine.GSO.4.62.0511081233180.16485@harper.uchicago.edu>
	<4370F104.2050205@colorstudy.com>
	<68cb949d0511081147t7a6d507crd56a0f6fd0ac6945@mail.gmail.com>
	<Pine.GSO.4.62.0511101052510.8603@harper.uchicago.edu>
	<68cb949d0511100918r42d63250r7d35ef2840826d48@mail.gmail.com>
	<Pine.GSO.4.62.0511111310550.28525@harper.uchicago.edu>
	<68cb949d0511111213v110d0e22qac66a2a19f3bf49b@mail.gmail.com>
	<Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu>
	<68cb949d0511111315q2b3aac3nf60ecd96ae504b54@mail.gmail.com>
	<68cb949d0511111547v6618c66eo2649795356dee7a0@mail.gmail.com>
	<Pine.GSO.4.62.0511121854460.29085@harper.uchicago.edu>
Message-ID: <Pine.GSO.4.62.0511141457001.27908@harper.uchicago.edu>


Hi Jess,

It's great to be making progress on this problem. I tried the code you 
sent and it also worked for me under Linux (but see below!)

You wrote that you weren't sure what I was getting at with the "timeout" 
concept. The idea is that the child process should be producing output 
fairly regularly. If some number of seconds go by during which it does not 
produce output, the parent should decide that the child is stuck and 
should kill it. Adding the timeout to the select call makes this possible. 
I don't know a better way.

A couple of questions: I notice you don't import the pty module anywhere, 
nor do you use os.forkpty(). I guess just opening the file in 
"/dev/pty..." makes it a pseudo-terminal? Isn't this a little dangerous? I 
would have thought the pty module would make things safer all around, but 
maybe it's not necessary at all?

Also, the code you sent unfortunately doesn't work under Irix. On that OS, 
the child's termination signal will never be picked up, and it will run 
indefinitely. The timeout in the select call is ineffective.

I mentioned that I'd been using Noah Spurrier's pexpect module, which 
contains a lot of the functionality I'm looking for. Originally, pexpect 
had the exact same problem under Irix that your code does, but I got in 
touch with Noah and he made a hack for it.

In his email to me he wrote:
   "Irix is a weird platform. It seems to have no reliable way to test if
    a child process is alive or not without blocking... I managed to find
    a hack for this, but it adds a 2 second timeout to every call to
    read_nonblocking(). That means you can't have a timeout less than 3
    seconds. That's probably not a big deal in general."

I orignally sent this email two days ago with a copy of "pexpect.py" 
attached to it, but it got held up by the list administrator for being too 
large, so rather than wait to see if it goes through, I'm canceling that 
original mail and sending this one instead, which has a link to the code 
instead of the code itself:

   http://people.cs.uchicago.edu/~nttaylor/pexpect.py

If you have time, please visit the above site and take a look. There's a 
good bit of code there, but I wonder if you might look over the part that 
specifically addresses the Irix hack (just search for the word 'irix' in 
the source). My ultimate goal is to understand exactly what's going on, 
and I think I stand a better chance with your code, which is much shorter 
and more focused on this one specific problem, than with pexpect, which is 
meant to be a much more general. If I could eventually make my own program 
work with a short piece of code that I throroughly understand, that is 
preferable to using the pexpect module, which is more of a black box.

And the short-term goal is to incorporate Noah's Irix hack into the code 
that you sent me. What do you think? I'm working on getting you a guest 
account on our Irix box for testing, in case you don't have one available.

with thanks for all your help,

Noel Taylor


> On Fri, 11 Nov 2005, Jess Balint wrote:
>
>> Ok, I've attached a pty version. I've tested it on Linux and Solaris
>> and it seems to work well. I left your select timeout in there for
>> illustration, but I'm still not sure what you were getting at with
>> that. I set the select timeout to '3' and the ticker.py to sleep the
>> number of seconds as the incrementer. This way you see a few 'ticks'
>> in the output before the parent kills it. Let me know if this one
>> works ok and makes sense.
>> 
>> Jess
>> 
>> On 11/11/05, Jess Balint <jbalint at gmail.com> wrote:
>>>> The child process in the real application I want to use this for is
>>>> written in fortran, so I can't use the techniques you mentioned.
>>>> 
>>>> I had already sought out second and third opinions on this, so now I am
>>>> really convinced that there is no way to get the effect I want without
>>>> using pseudo-terminals.
>>> 
>>> Yes, I think that's the only thing you have influence on at this
>>> point. Using forkpty() and connecting it to the pipe might work.
>>> Another option might be to use an LD_PRELOAD with an init routine that
>>> calls setbuf(). I am trying to find a simple, non-intrusive way to do
>>> this, I will let you know if I come up with something.
>>> 
>>>>> The non-blocking io is not relevant to any of this.
>>>> 
>>>> Too bad. If I can ask you one more thing: what is the proper purpose of
>>>> using fcntl to set a file descriptor to non-blocking IO if not to achieve
>>>> the behavior I'm looking for in this problem?
>>> 
>>> Non-blocking IO is something that allows a read() or write() call to
>>> return when a file descriptor is not ready. So if you are reading from
>>> a network connection and nothing has come through, a normal read()
>>> call would block (and wait for something). A non-blocking attribute
>>> added to the socket will allow a read() call to return so you can do
>>> other stuff until it is ready.
>>> 
>>> Jess
>>> 
>

From jbalint at gmail.com  Mon Nov 14 22:30:19 2005
From: jbalint at gmail.com (Jess Balint)
Date: Mon, 14 Nov 2005 16:30:19 -0500
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <Pine.GSO.4.62.0511141457001.27908@harper.uchicago.edu>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<Pine.GSO.4.62.0511101052510.8603@harper.uchicago.edu>
	<68cb949d0511100918r42d63250r7d35ef2840826d48@mail.gmail.com>
	<Pine.GSO.4.62.0511111310550.28525@harper.uchicago.edu>
	<68cb949d0511111213v110d0e22qac66a2a19f3bf49b@mail.gmail.com>
	<Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu>
	<68cb949d0511111315q2b3aac3nf60ecd96ae504b54@mail.gmail.com>
	<68cb949d0511111547v6618c66eo2649795356dee7a0@mail.gmail.com>
	<Pine.GSO.4.62.0511121854460.29085@harper.uchicago.edu>
	<Pine.GSO.4.62.0511141457001.27908@harper.uchicago.edu>
Message-ID: <68cb949d0511141330p57152b7k36cac4335dbc21ae@mail.gmail.com>

On 11/14/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
> concept. The idea is that the child process should be producing output
> fairly regularly. If some number of seconds go by during which it does not
> produce output, the parent should decide that the child is stuck and
> should kill it. Adding the timeout to the select call makes this possible.

Ok, that makes fine sense now. :)

> A couple of questions: I notice you don't import the pty module anywhere,
> nor do you use os.forkpty(). I guess just opening the file in
> "/dev/pty..." makes it a pseudo-terminal? Isn't this a little dangerous? I
> would have thought the pty module would make things safer all around, but
> maybe it's not necessary at all?

This is the standard way to open a pty. As far as the module goes, it
doesn't seem to have support on Solaris. The pty module only has the
"select" call and "os" doesn't have the "openpty" call. I don't
consider this dangerous at all. If you have some other evidence, I'd
be interested to hear it.

> Also, the code you sent unfortunately doesn't work under Irix. On that OS,
> the child's termination signal will never be picked up, and it will run
> indefinitely. The timeout in the select call is ineffective.

What do you mean the child's termination signal will not be picked up?
Are you saying the SIGCHLD isn't sent to the parent process? Or isn't
received for some reason? If there is a difference in the select
implementation, we can probably work around that with a timer, but I
think that's probably how they implement it... The other thing is if
the pty isn't closed correctly, we might use the exception fd's from
select to know when it's done. Either way, I'll have to take a closer
look at it.

> I mentioned that I'd been using Noah Spurrier's pexpect module, which
> contains a lot of the functionality I'm looking for. Originally, pexpect
> had the exact same problem under Irix that your code does, but I got in
> touch with Noah and he made a hack for it.
>
> In his email to me he wrote:
>    "Irix is a weird platform. It seems to have no reliable way to test if
>     a child process is alive or not without blocking... I managed to find
>     a hack for this, but it adds a 2 second timeout to every call to
>     read_nonblocking(). That means you can't have a timeout less than 3
>     seconds. That's probably not a big deal in general."

I'm not familiar with all the Python aspects of this or how
read_nonblocking is implemented. Maybe kill(pid, 0) will work? As I
said before maybe the exc. fd set will help here.

> I orignally sent this email two days ago with a copy of "pexpect.py"
> attached to it, but it got held up by the list administrator for being too
> large, so rather than wait to see if it goes through, I'm canceling that
> original mail and sending this one instead, which has a link to the code
> instead of the code itself:

I see what he's doing, but I don't have know the exact issue he is
trying to work around.

> If you have time, please visit the above site and take a look. There's a
> good bit of code there, but I wonder if you might look over the part that
> specifically addresses the Irix hack (just search for the word 'irix' in
> the source). My ultimate goal is to understand exactly what's going on,
> and I think I stand a better chance with your code, which is much shorter
> and more focused on this one specific problem, than with pexpect, which is
> meant to be a much more general. If I could eventually make my own program
> work with a short piece of code that I throroughly understand, that is
> preferable to using the pexpect module, which is more of a black box.
>
> And the short-term goal is to incorporate Noah's Irix hack into the code
> that you sent me. What do you think? I'm working on getting you a guest
> account on our Irix box for testing, in case you don't have one available.

Hopefully you have a C compiler on there too. ;)

Jess

From nttaylor at uchicago.edu  Tue Nov 15 00:48:57 2005
From: nttaylor at uchicago.edu (Noel Thomas Taylor)
Date: Mon, 14 Nov 2005 17:48:57 -0600 (CST)
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <68cb949d0511141330p57152b7k36cac4335dbc21ae@mail.gmail.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<Pine.GSO.4.62.0511101052510.8603@harper.uchicago.edu>
	<68cb949d0511100918r42d63250r7d35ef2840826d48@mail.gmail.com>
	<Pine.GSO.4.62.0511111310550.28525@harper.uchicago.edu>
	<68cb949d0511111213v110d0e22qac66a2a19f3bf49b@mail.gmail.com>
	<Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu>
	<68cb949d0511111315q2b3aac3nf60ecd96ae504b54@mail.gmail.com>
	<68cb949d0511111547v6618c66eo2649795356dee7a0@mail.gmail.com>
	<Pine.GSO.4.62.0511121854460.29085@harper.uchicago.edu>
	<Pine.GSO.4.62.0511141457001.27908@harper.uchicago.edu>
	<68cb949d0511141330p57152b7k36cac4335dbc21ae@mail.gmail.com>
Message-ID: <Pine.GSO.4.62.0511141733420.27908@harper.uchicago.edu>


Hi Jess,

I've set up a guest account for you on our Irix box. Please login at 
jess at cube.uchicago.edu. The password is 'password', which you'll probably 
want to change when you first login. Let me know if you have any problem.

You'll find a copy of your latest script, 'child_process_3.py', waiting 
for you in your home directory, along with a couple of 'ticker' programs 
and a copy of "pexpect.py", which has a hack for circumventing this 
problem on Irix (assuming it's the same problem)

> What do you mean the child's termination signal will not be picked up?
> Are you saying the SIGCHLD isn't sent to the parent process? Or isn't
> received for some reason? If there is a difference in the select
> implementation, we can probably work around that with a timer, but I
> think that's probably how they implement it... The other thing is if
> the pty isn't closed correctly, we might use the exception fd's from
> select to know when it's done. Either way, I'll have to take a closer
> look at it.

Unfortunately, I'm not sure myself exactly what's happening, only that the 
parent will wait on the child indefinitely. It will not abort when the 
timeout value passed to the select call is exceeded, nor will it report 
back when the child terminates normally.

> Hopefully you have a C compiler on there too. ;)

There is a C compiler, so hopefully that won't be a problem. Let me know 
if I can do anything else to help. I'm very grateful for all your 
assistance so far.

best,

Noel Taylor

From nttaylor at uchicago.edu  Tue Nov 15 01:07:25 2005
From: nttaylor at uchicago.edu (Noel Thomas Taylor)
Date: Mon, 14 Nov 2005 18:07:25 -0600 (CST)
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <1b5ac5df0511141558i7fa47488i10c1a41a79d73cfc@mail.google.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu> 
	<Pine.GSO.4.62.0511111310550.28525@harper.uchicago.edu> 
	<68cb949d0511111213v110d0e22qac66a2a19f3bf49b@mail.gmail.com> 
	<Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu> 
	<68cb949d0511111315q2b3aac3nf60ecd96ae504b54@mail.gmail.com> 
	<68cb949d0511111547v6618c66eo2649795356dee7a0@mail.gmail.com> 
	<Pine.GSO.4.62.0511121854460.29085@harper.uchicago.edu> 
	<Pine.GSO.4.62.0511141457001.27908@harper.uchicago.edu> 
	<68cb949d0511141330p57152b7k36cac4335dbc21ae@mail.gmail.com> 
	<Pine.GSO.4.62.0511141733420.27908@harper.uchicago.edu>
	<1b5ac5df0511141558i7fa47488i10c1a41a79d73cfc@mail.google.com>
Message-ID: <Pine.GSO.4.62.0511141804450.27908@harper.uchicago.edu>


Hi Joe,

Thanks for the heads-up. I realized what I'd done right after I sent the 
message, and went in and changed the password, then sent a private message 
to Jess Balint.

But hey, if anyone else wants to help work on this problem, I'll be happy 
to set up an account for you too!

with thanks,

Noel Taylor

On Mon, 14 Nov 2005, Joe LaPenna wrote:

> oops? Sent to the whole list.
> On 11/14/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
>>
>> Hi Jess,
>>
>> I've set up a guest account for you on our Irix box. Please login at
>> jess at cube.uchicago.edu. The password is 'password', which you'll probably
>> want to change when you first login. Let me know if you have any problem.
>>
>> You'll find a copy of your latest script, 'child_process_3.py', waiting
>> for you in your home directory, along with a couple of 'ticker' programs
>> and a copy of "pexpect.py", which has a hack for circumventing this
>> problem on Irix (assuming it's the same problem)
>>
>>> What do you mean the child's termination signal will not be picked up?
>>> Are you saying the SIGCHLD isn't sent to the parent process? Or isn't
>>> received for some reason? If there is a difference in the select
>>> implementation, we can probably work around that with a timer, but I
>>> think that's probably how they implement it... The other thing is if
>>> the pty isn't closed correctly, we might use the exception fd's from
>>> select to know when it's done. Either way, I'll have to take a closer
>>> look at it.
>>
>> Unfortunately, I'm not sure myself exactly what's happening, only that the
>> parent will wait on the child indefinitely. It will not abort when the
>> timeout value passed to the select call is exceeded, nor will it report
>> back when the child terminates normally.
>>
>>> Hopefully you have a C compiler on there too. ;)
>>
>> There is a C compiler, so hopefully that won't be a problem. Let me know
>> if I can do anything else to help. I'm very grateful for all your
>> assistance so far.
>>
>> best,
>>
>> Noel Taylor
>> _______________________________________________
>> Chicago mailing list
>> Chicago at python.org
>> http://mail.python.org/mailman/listinfo/chicago
>>
>
>
> --
> HWOps Ninja - Chicago
> 1 (312) 925-2805
>

From jbalint at gmail.com  Tue Nov 15 01:42:23 2005
From: jbalint at gmail.com (Jess Balint)
Date: Mon, 14 Nov 2005 19:42:23 -0500
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <Pine.GSO.4.62.0511141733420.27908@harper.uchicago.edu>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<Pine.GSO.4.62.0511111310550.28525@harper.uchicago.edu>
	<68cb949d0511111213v110d0e22qac66a2a19f3bf49b@mail.gmail.com>
	<Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu>
	<68cb949d0511111315q2b3aac3nf60ecd96ae504b54@mail.gmail.com>
	<68cb949d0511111547v6618c66eo2649795356dee7a0@mail.gmail.com>
	<Pine.GSO.4.62.0511121854460.29085@harper.uchicago.edu>
	<Pine.GSO.4.62.0511141457001.27908@harper.uchicago.edu>
	<68cb949d0511141330p57152b7k36cac4335dbc21ae@mail.gmail.com>
	<Pine.GSO.4.62.0511141733420.27908@harper.uchicago.edu>
Message-ID: <68cb949d0511141642q617ad48ei11b289caaa87e167@mail.gmail.com>

On 11/14/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
> You'll find a copy of your latest script, 'child_process_3.py', waiting
> for you in your home directory, along with a couple of 'ticker' programs
> and a copy of "pexpect.py", which has a hack for circumventing this
> problem on Irix (assuming it's the same problem)

The os.openpty() call seems to be working fine on your installation.
Maybe it was added after the 2.2 that I have on my Solaris box? I just
checked and it's there in 2.4 (ActiveState) on Solaris. So now it
seems to work OK on your IRIX box (Python 2.3.5) and the Solaris and
Linux boxes I have tested, both with Python 2.4.

> Unfortunately, I'm not sure myself exactly what's happening, only that the
> parent will wait on the child indefinitely. It will not abort when the
> timeout value passed to the select call is exceeded, nor will it report
> back when the child terminates normally.

It's putting the fd in the exception fd set. I just added a call to
check for it. The timeout was working it just re-entered the loop
because there was no condition to break if nothing was read.

Let me know how it's working for you now.

Jess

From RCRamsdell at gldd.com  Wed Nov 16 18:41:05 2005
From: RCRamsdell at gldd.com (RCRamsdell@gldd.com)
Date: Wed, 16 Nov 2005 11:41:05 -0600
Subject: [Chicago] FW: Playing with generics
Message-ID: <C0B0D5759E27344788E6EB554ECF4B9101FB8F36@NEWMAIL.internal.gldd.com>


I have a function that works on a pair of x, y coordinates.  I wanted to
be able to pass either a 2-tuple, or x and y.  Remembering Ian's
presentation, this sounded like a great case for generics!  

So I produced listing 1.  Wow, lots of lines there, and some
duplication.  Plus I have to add a dependency (to the dispatch module).

I had a similar case in another module from the same project, so I tried
it in-line (Listing 2).  Much shorter, particularly when you consider
that I am checking for an additional case, that of no argument passed
in.

Is this just a case where the requirement is too simple to warrant the
overhead?  Yes, but: if I later wanted to add support for (x,y,z)
coords, where betaad might have a quite different calculation, then the
dispatch would probably pay off.


# Listing 1:
@dispatch.generic()
    def canvasCoords(self, *args):
        """Stub function for canvasCoords"""
    @canvasCoords.when("len(args)==2")
    def canvasCoords(self, x, y):
        """
        Calculate the coordinates on the canvas of the given dredge
coordinates
        x, y are coordinates in the dredging scheme, center of rotation
is (0,0)
        """
        return x/self.scalefactor+self.xoffset,
y/self.scalefactor+self.yoffset
    @canvasCoords.when("len(args)==1")
    def canvasCoords(self, xy):
        """
        Calculate the coordinates on the canvas of the given dredge
coordinates
        (x, y) are coordinates in the dredging scheme, center of
rotation is (0,0)
        """
        return xy[0]/self.scalefactor+self.xoffset,
xy[1]/self.scalefactor+self.yoffset

# Listing 2: 
def betaad(self, *args):
        """betaa - The angle of the line through the anchor in degrees
            the args are either a 2-tuple with the anchor coords, or
anchor
            x and y
        """
        if len(args)==0:
            xa, ya = self.Xa, self.Ya
        elif len(args) == 1 and len(args[0])==2:
            xa, ya = args[0]
        elif len(args) == 2:
            xa, ya = args
        else:
            raise TypeError, "HyAnchor.betaad - Wrong number of
arguments, expects a pair or 2-tuple"
        betaa = atan(xa/ya)
        return betaa*180/pi

Robert C. Ramsdell III
Production Engineering Manager
Great Lakes Dredge & Dock Company
2122 York Road
Oakbrook, IL 60523
rcramsdell at gldd.com
Phone: (630) 574-3463
Mobile: (630) 805-1776 
Fax: (630) 574-2909


From nttaylor at uchicago.edu  Wed Nov 16 22:06:41 2005
From: nttaylor at uchicago.edu (Noel Thomas Taylor)
Date: Wed, 16 Nov 2005 15:06:41 -0600 (CST)
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <68cb949d0511141642q617ad48ei11b289caaa87e167@mail.gmail.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<Pine.GSO.4.62.0511111310550.28525@harper.uchicago.edu>
	<68cb949d0511111213v110d0e22qac66a2a19f3bf49b@mail.gmail.com>
	<Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu>
	<68cb949d0511111315q2b3aac3nf60ecd96ae504b54@mail.gmail.com>
	<68cb949d0511111547v6618c66eo2649795356dee7a0@mail.gmail.com>
	<Pine.GSO.4.62.0511121854460.29085@harper.uchicago.edu>
	<Pine.GSO.4.62.0511141457001.27908@harper.uchicago.edu>
	<68cb949d0511141330p57152b7k36cac4335dbc21ae@mail.gmail.com>
	<Pine.GSO.4.62.0511141733420.27908@harper.uchicago.edu>
	<68cb949d0511141642q617ad48ei11b289caaa87e167@mail.gmail.com>
Message-ID: <Pine.GSO.4.62.0511161445580.779@harper.uchicago.edu>


> The os.openpty() call seems to be working fine on your installation.
> Maybe it was added after the 2.2 that I have on my Solaris box? I just
> checked and it's there in 2.4 (ActiveState) on Solaris. So now it
> seems to work OK on your IRIX box (Python 2.3.5) and the Solaris and
> Linux boxes I have tested, both with Python 2.4.

Hi Jess,

I've tested it too and it works fine all around. This is the greatest. 
I'll be giving you big props in the final code.

I had a few closing questions. At the beginning of the parent fork, you 
close the file descriptor for writing to stdErr, since you won't need it; 
meanwhile in the child fork you close the file descriptor for reading from 
stdErr, since you won't need it either. This all makes sense, but can't 
you also close the corresponding file descriptors for writing to and 
reading from stdOut, just as you did for stdErr? I added these "closes" to 
the code and noticed no ill effects, but thought I'd ask in case there was 
some reason I wasn't seeing.

Were you using your own function 'mygetpty()' on platforms / versions of 
Python where os.getpty() is not supported, like the Python 2.2 you had on 
your Solaris machine? I'm not sure about just when Python introduced this 
function, or if it is supported under Solaris. I don't have a Solaris 
machine here to test, and Python's own documentation pages are not very 
specific about which platforms support this function.

Finally, I pretty much understand what's happening, but why do you have to 
make the calls to dup2? I've seen this in several process-spawning codes 
but have never figured out just why it's necessary.

with thanks,

Noel Taylor

From jbalint at gmail.com  Wed Nov 16 22:13:46 2005
From: jbalint at gmail.com (Jess Balint)
Date: Wed, 16 Nov 2005 16:13:46 -0500
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <Pine.GSO.4.62.0511161445580.779@harper.uchicago.edu>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu>
	<68cb949d0511111315q2b3aac3nf60ecd96ae504b54@mail.gmail.com>
	<68cb949d0511111547v6618c66eo2649795356dee7a0@mail.gmail.com>
	<Pine.GSO.4.62.0511121854460.29085@harper.uchicago.edu>
	<Pine.GSO.4.62.0511141457001.27908@harper.uchicago.edu>
	<68cb949d0511141330p57152b7k36cac4335dbc21ae@mail.gmail.com>
	<Pine.GSO.4.62.0511141733420.27908@harper.uchicago.edu>
	<68cb949d0511141642q617ad48ei11b289caaa87e167@mail.gmail.com>
	<Pine.GSO.4.62.0511161445580.779@harper.uchicago.edu>
Message-ID: <68cb949d0511161313jdd24117xe16e57dbc1ea211f@mail.gmail.com>

On 11/16/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
> I've tested it too and it works fine all around. This is the greatest.
> I'll be giving you big props in the final code.

Glad it's working.

> I had a few closing questions. At the beginning of the parent fork, you
> close the file descriptor for writing to stdErr, since you won't need it;
> meanwhile in the child fork you close the file descriptor for reading from
> stdErr, since you won't need it either. This all makes sense, but can't
> you also close the corresponding file descriptors for writing to and
> reading from stdOut, just as you did for stdErr? I added these "closes" to
> the code and noticed no ill effects, but thought I'd ask in case there was
> some reason I wasn't seeing.

If you don't close the other side of pipes, they won't return EOF (see
pipe(2) man page). The pty is a different story. You usually only open
the slave in the child, but since the openpty() does the whole thing,
we just take it as it is.

> Were you using your own function 'mygetpty()' on platforms / versions of
> Python where os.getpty() is not supported, like the Python 2.2 you had on
> your Solaris machine? I'm not sure about just when Python introduced this
> function, or if it is supported under Solaris. I don't have a Solaris
> machine here to test, and Python's own documentation pages are not very
> specific about which platforms support this function.

I just moved the mygetpty() code out to it's own function so I could
easily replace it with openpty() to see if it worked. As I said Python
2.2 (from Solaris companion CD) doesn't have openpty(), but
ActiveState 2.4 does. So you can make whatever changes you need in
that direction.

> Finally, I pretty much understand what's happening, but why do you have to
> make the calls to dup2? I've seen this in several process-spawning codes
> but have never figured out just why it's necessary.

dup2() is really the key to all this. You should read the man page for
details, but in a sentence it just redefines file descriptors. This
allows setting the new stdout and stderr for the child so it can be
captured.

Jess

From chris.mcavoy at gmail.com  Thu Nov 17 13:30:02 2005
From: chris.mcavoy at gmail.com (Chris McAvoy)
Date: Thu, 17 Nov 2005 06:30:02 -0600
Subject: [Chicago] Fwd: [Chicago Area Ruby Group] Django / Rails Meeting
In-Reply-To: <437C2BA3.3020401@johnwlong.com>
References: <437C2BA3.3020401@johnwlong.com>
Message-ID: <3096c19d0511170430r18e51e4cw6e4ddba477f7c2d9@mail.gmail.com>

Hi ChiPy,

A date and location have been set for the Django / Rails presentation
I mentioned last week.  December 3rd at DePaul.  The details are here:
http://snakesandrubies.com

This mini-conference is going to be a great opportunity for all the
local users groups to get together and compare web development notes. 
It's pretty exciting.

Chris

---------- Forwarded message ----------
From: John W. Long <rubygroup at johnwlong.com>
Date: Nov 17, 2005 1:05 AM
Subject: [Chicago Area Ruby Group] Django / Rails Meeting
To: Chicago Area Ruby Group <chicagogroup-members-list at rubyforge.org>


We're on!

http://snakesandrubies.com

Let the advertising campaign begin. :)

--
John

_______________________________________________
ChicagoGroup-Members-List mailing list
ChicagoGroup-Members-List at rubyforge.org
http://rubyforge.org/mailman/listinfo/chicagogroup-members-list

From ehs at pobox.com  Thu Nov 17 14:20:41 2005
From: ehs at pobox.com (Edward Summers)
Date: Thu, 17 Nov 2005 07:20:41 -0600
Subject: [Chicago] Fwd: [Chicago Area Ruby Group] Django / Rails Meeting
In-Reply-To: <3096c19d0511170430r18e51e4cw6e4ddba477f7c2d9@mail.gmail.com>
References: <437C2BA3.3020401@johnwlong.com>
	<3096c19d0511170430r18e51e4cw6e4ddba477f7c2d9@mail.gmail.com>
Message-ID: <E7F79AFF-409F-4DA6-956D-836A81A9DE0F@pobox.com>

On Nov 17, 2005, at 6:30 AM, Chris McAvoy wrote:
> The details are here: http://snakesandrubies.com

What a sharp looking site, and awesome work getting this thing jump  
started Chris & Ian. I agree it'll be nice to have a meeting that  
isn't just about one particular language but focuses more on the  
vibrant opensource community here in Chicago.

//Ed

From JRHuggins at thoughtworks.COM  Thu Nov 17 16:03:16 2005
From: JRHuggins at thoughtworks.COM (Jason R Huggins)
Date: Thu, 17 Nov 2005 09:03:16 -0600
Subject: [Chicago] Fwd: [Chicago Area Ruby Group] Django / Rails Meeting
In-Reply-To: <3096c19d0511170430r18e51e4cw6e4ddba477f7c2d9@mail.gmail.com>
Message-ID: <OF3BEF6530.13BDE543-ON862570BC.005245CA-862570BC.0052B2C2@thoughtworks.com>

I was going to ask "Which framework was used to build 
snakesandrubies.com?... But I can tell from the "Yellow Fade Technique" 
after posting a question and the style of Add a Question form that it's in 
Rails. Doh! 

- Jason

Chris McAvoy wrote on 11/17/2005 06:30:02 AM:
> Hi ChiPy,
> 
> A date and location have been set for the Django / Rails presentation
> I mentioned last week.  December 3rd at DePaul.  The details are here:
> http://snakesandrubies.com

From mtobis at gmail.com  Thu Nov 17 16:42:17 2005
From: mtobis at gmail.com (Michael Tobis)
Date: Thu, 17 Nov 2005 09:42:17 -0600
Subject: [Chicago] Fwd: [Chicago Area Ruby Group] Django / Rails Meeting
In-Reply-To: <OF3BEF6530.13BDE543-ON862570BC.005245CA-862570BC.0052B2C2@thoughtworks.com>
References: <3096c19d0511170430r18e51e4cw6e4ddba477f7c2d9@mail.gmail.com>
	<OF3BEF6530.13BDE543-ON862570BC.005245CA-862570BC.0052B2C2@thoughtworks.com>
Message-ID: <c3aa55550511170742m43de2e94sf3ac70b8442d5a75@mail.gmail.com>

Excellent! I'll be there.

On 11/17/05, Jason R Huggins <JRHuggins at thoughtworks.com> wrote:
> I was going to ask "Which framework was used to build
> snakesandrubies.com?

I want to know who came up with the brilliant logo! It's so cheerful!
That's mighty fine artwork for a one-off open-source event.

mt

From RCRamsdell at gldd.com  Thu Nov 17 16:55:06 2005
From: RCRamsdell at gldd.com (RCRamsdell@gldd.com)
Date: Thu, 17 Nov 2005 09:55:06 -0600
Subject: [Chicago] Fwd: [Chicago Area Ruby Group] Django / Rails Meeting
Message-ID: <C0B0D5759E27344788E6EB554ECF4B9101FB8F39@NEWMAIL.internal.gldd.com>

Nice site. Are the questions for david and Adrian supposed to show up on
the front page?  Because Adrian has one question, but you don't know it
until you click through.

Robert

> -----Original Message-----
> From: chicago-bounces at python.org 
> [mailto:chicago-bounces at python.org] On Behalf Of Chris McAvoy
> Sent: Thursday, November 17, 2005 6:30 AM
> To: The Chicago Python Users Group; ws at johnwlong.com
> Subject: [Chicago] Fwd: [Chicago Area Ruby Group] Django / 
> Rails Meeting
> 
> 
> Hi ChiPy,
> 
> A date and location have been set for the Django / Rails 
> presentation I mentioned last week.  December 3rd at DePaul.  
> The details are here: http://snakesandrubies.com
> 
> This mini-conference is going to be a great opportunity for 
> all the local users groups to get together and compare web 
> development notes. 
> It's pretty exciting.
> 
> Chris
> 
> ---------- Forwarded message ----------
> From: John W. Long <rubygroup at johnwlong.com>
> Date: Nov 17, 2005 1:05 AM
> Subject: [Chicago Area Ruby Group] Django / Rails Meeting
> To: Chicago Area Ruby Group <chicagogroup-members-list at rubyforge.org>
> 
> 
> We're on!
> 
http://snakesandrubies.com

Let the advertising campaign begin. :)

--
John

_______________________________________________
ChicagoGroup-Members-List mailing list
ChicagoGroup-Members-List at rubyforge.org
http://rubyforge.org/mailman/listinfo/chicagogroup-members-list
_______________________________________________
Chicago mailing list
Chicago at python.org http://mail.python.org/mailman/listinfo/chicago

From mtobis at gmail.com  Thu Nov 17 21:22:07 2005
From: mtobis at gmail.com (Michael Tobis)
Date: Thu, 17 Nov 2005 14:22:07 -0600
Subject: [Chicago] FW: Playing with generics
In-Reply-To: <C0B0D5759E27344788E6EB554ECF4B9101FB8F36@NEWMAIL.internal.gldd.com>
References: <C0B0D5759E27344788E6EB554ECF4B9101FB8F36@NEWMAIL.internal.gldd.com>
Message-ID: <c3aa55550511171222n1be9e19sdcae667a5d74d0bb@mail.gmail.com>

If I understand what you are after correctly, I would prefer
functional programming for this sort of thing.

How about:


####

from operator import add

def toSequence(item):
   try: iter(item)
   except: return [item]
   else: return item

def Flatten(listOfListsAndScalars):

   # nonrecursive: handles only one level of list nesting
   # see also http://aspn.activestate.com/ASPN/Mail/Message/python-tutor/2302137

   return reduce(add,[toSequence(item) for item in listOfListsAndScalars])

##########
from math import atan, pi
scale = 180./pi

def RobertCalc(*args):
    xa,ya = Flatten(args)[:2]
    return atan(xa/ya)*scale

##########
# -mt

From nttaylor at uchicago.edu  Thu Nov 17 22:14:16 2005
From: nttaylor at uchicago.edu (Noel Thomas Taylor)
Date: Thu, 17 Nov 2005 15:14:16 -0600 (CST)
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <68cb949d0511161313jdd24117xe16e57dbc1ea211f@mail.gmail.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<Pine.GSO.4.62.0511111427590.7155@harper.uchicago.edu>
	<68cb949d0511111315q2b3aac3nf60ecd96ae504b54@mail.gmail.com>
	<68cb949d0511111547v6618c66eo2649795356dee7a0@mail.gmail.com>
	<Pine.GSO.4.62.0511121854460.29085@harper.uchicago.edu>
	<Pine.GSO.4.62.0511141457001.27908@harper.uchicago.edu>
	<68cb949d0511141330p57152b7k36cac4335dbc21ae@mail.gmail.com>
	<Pine.GSO.4.62.0511141733420.27908@harper.uchicago.edu>
	<68cb949d0511141642q617ad48ei11b289caaa87e167@mail.gmail.com>
	<Pine.GSO.4.62.0511161445580.779@harper.uchicago.edu>
	<68cb949d0511161313jdd24117xe16e57dbc1ea211f@mail.gmail.com>
Message-ID: <Pine.GSO.4.62.0511171417250.876@harper.uchicago.edu>


Hi Jess and the ChiPy users group,

Jess, I know you've done a lot of work on this already, but I hope I may 
send this last round of questions about the child processes, and then I 
promise to be done.

Inside the main while loop you've got:

(rl, wl, xl) = select([outpi, errpi], [], [outpi, errpi], timeout)

try:
   # capture the output of the child
   for f in rl:
     if f == outpi:
       o = os.read(f, 100)
       if o: output += o
       else: end = 1
     elif f == errpi:
       e = os.read(f, 100)
       if e: error += e
       else: end = 1
except os.OSError, e:
   print "OSError: " + str(e)
   break

On Linux, execution enters the except block when the child process has 
terminated. For this reason, I'm taking the print statement out of my own 
code, since this seems to be the normal way for the parent to know that 
the child has finished, and so it's not technically an "error". right?

But I couldn't quite figure out just why this error is raised at all. For 
example, it doesn't seem to happen if I use os.open() and os.read() to 
just read from a text file. Under what circumstances will the return value 
from the "select()" call show that the child's file descriptor is "ready 
to be read" and yet raise an error when you try to read it?

Also, your code has a variable 'end' which you set to 1 if you got 
nothing from the child's stdOut or stdErr, but then nothing happens with 
this variable once it's set. Was this going to be some way out of the 
while loop other than through the except block? I noticed that if I 
add a print statement and a line: "if end: break" like this:
...

     elif f == errpi:
       e = os.read(f, 100)
       if e: error += e
       else:
         end = 1
         print "got nothing from stderr"
     if end: break
except os.OSError, e:
   print "OSError: " + str(e)
   break

...

that a normally terminating child causes the "got nothing from stderr" 
message to be printed, and the while loop exits cleanly without entering 
the "except" block. Did you have something like this in mind for 'end'? If 
that is the case, under what circumstances would execution enter the 
except block? I haven't found anything about os.read() raising an OSError.

with thanks,

Noel Taylor

PS Why did you choose 100 for the size of your read buffer? Would 1 be a 
bad choice?


On Wed, 16 Nov 2005, Jess Balint wrote:

> On 11/16/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
>> I've tested it too and it works fine all around. This is the greatest.
>> I'll be giving you big props in the final code.
>
> Glad it's working.
>
>> I had a few closing questions. At the beginning of the parent fork, you
>> close the file descriptor for writing to stdErr, since you won't need it;
>> meanwhile in the child fork you close the file descriptor for reading from
>> stdErr, since you won't need it either. This all makes sense, but can't
>> you also close the corresponding file descriptors for writing to and
>> reading from stdOut, just as you did for stdErr? I added these "closes" to
>> the code and noticed no ill effects, but thought I'd ask in case there was
>> some reason I wasn't seeing.
>
> If you don't close the other side of pipes, they won't return EOF (see
> pipe(2) man page). The pty is a different story. You usually only open
> the slave in the child, but since the openpty() does the whole thing,
> we just take it as it is.
>
>> Were you using your own function 'mygetpty()' on platforms / versions of
>> Python where os.getpty() is not supported, like the Python 2.2 you had on
>> your Solaris machine? I'm not sure about just when Python introduced this
>> function, or if it is supported under Solaris. I don't have a Solaris
>> machine here to test, and Python's own documentation pages are not very
>> specific about which platforms support this function.
>
> I just moved the mygetpty() code out to it's own function so I could
> easily replace it with openpty() to see if it worked. As I said Python
> 2.2 (from Solaris companion CD) doesn't have openpty(), but
> ActiveState 2.4 does. So you can make whatever changes you need in
> that direction.
>
>> Finally, I pretty much understand what's happening, but why do you have to
>> make the calls to dup2? I've seen this in several process-spawning codes
>> but have never figured out just why it's necessary.
>
> dup2() is really the key to all this. You should read the man page for
> details, but in a sentence it just redefines file descriptors. This
> allows setting the new stdout and stderr for the child so it can be
> captured.
>
> Jess
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>

From RCRamsdell at gldd.com  Fri Nov 18 19:19:04 2005
From: RCRamsdell at gldd.com (RCRamsdell@gldd.com)
Date: Fri, 18 Nov 2005 12:19:04 -0600
Subject: [Chicago] FW: Playing with generics
Message-ID: <C0B0D5759E27344788E6EB554ECF4B9101FB8F3F@NEWMAIL.internal.gldd.com>

Two concerns with this:
It is fairly particular to the case of 2-tuple versus x and y input.  It
is not that easily extensible to handle, for example, xyz inputs, or
zero inputs, in which case the method should pull from the object.

Good call on the scale variable.  After reading your code I realized
that I do 180/pi or pi/180 (conversion to and from radians and degrees)
in a number of places.  But I already have a perfectly good
UnitConversions module from another project with a variable deg_to_rad
that I should use instead.  I feel a refactoring coming on.

Robert

> -----Original Message-----
> From: chicago-bounces at python.org 
> [mailto:chicago-bounces at python.org] On Behalf Of Michael Tobis
> Sent: Thursday, November 17, 2005 2:22 PM
> To: The Chicago Python Users Group
> Subject: Re: [Chicago] FW: Playing with generics
> 
> 
> If I understand what you are after correctly, I would prefer 
> functional programming for this sort of thing.
> 
> How about:
> 
> 
> ####
> 
> from operator import add
> 
> def toSequence(item):
>    try: iter(item)
>    except: return [item]
>    else: return item
> 
> def Flatten(listOfListsAndScalars):
> 
>    # nonrecursive: handles only one level of list nesting
>    # see also 
> http://aspn.activestate.com/ASPN/Mail/Message/python-tutor/230
2137

   return reduce(add,[toSequence(item) for item in
listOfListsAndScalars])

##########
from math import atan, pi
scale = 180./pi

def RobertCalc(*args):
    xa,ya = Flatten(args)[:2]
    return atan(xa/ya)*scale

##########
# -mt
_______________________________________________
Chicago mailing list
Chicago at python.org http://mail.python.org/mailman/listinfo/chicago

From jbalint at gmail.com  Fri Nov 18 19:24:37 2005
From: jbalint at gmail.com (Jess Balint)
Date: Fri, 18 Nov 2005 13:24:37 -0500
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <Pine.GSO.4.62.0511171417250.876@harper.uchicago.edu>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<68cb949d0511111547v6618c66eo2649795356dee7a0@mail.gmail.com>
	<Pine.GSO.4.62.0511121854460.29085@harper.uchicago.edu>
	<Pine.GSO.4.62.0511141457001.27908@harper.uchicago.edu>
	<68cb949d0511141330p57152b7k36cac4335dbc21ae@mail.gmail.com>
	<Pine.GSO.4.62.0511141733420.27908@harper.uchicago.edu>
	<68cb949d0511141642q617ad48ei11b289caaa87e167@mail.gmail.com>
	<Pine.GSO.4.62.0511161445580.779@harper.uchicago.edu>
	<68cb949d0511161313jdd24117xe16e57dbc1ea211f@mail.gmail.com>
	<Pine.GSO.4.62.0511171417250.876@harper.uchicago.edu>
Message-ID: <68cb949d0511181024i110ac20cg2ff23d824b501831@mail.gmail.com>

On 11/17/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
> Jess, I know you've done a lot of work on this already, but I hope I may
> send this last round of questions about the child processes, and then I
> promise to be done.

No problem, keep asking until you understand. :)

> Inside the main while loop you've got:
>
> (rl, wl, xl) = select([outpi, errpi], [], [outpi, errpi], timeout)
>
> try:
>    # capture the output of the child
>    for f in rl:
>      if f == outpi:
>        o = os.read(f, 100)
>        if o: output += o
>        else: end = 1
>      elif f == errpi:
>        e = os.read(f, 100)
>        if e: error += e
>        else: end = 1
> except os.OSError, e:
>    print "OSError: " + str(e)
>    break
>
> On Linux, execution enters the except block when the child process has
> terminated. For this reason, I'm taking the print statement out of my own
> code, since this seems to be the normal way for the parent to know that
> the child has finished, and so it's not technically an "error". right?

Yes, as far as the program goes, there is no problem. There was a read
error because the child is gone, so we know it's done.

> But I couldn't quite figure out just why this error is raised at all. For
> example, it doesn't seem to happen if I use os.open() and os.read() to
> just read from a text file. Under what circumstances will the return value
> from the "select()" call show that the child's file descriptor is "ready
> to be read" and yet raise an error when you try to read it?

This is answered in the man page:

Quote from man select(2): Those listed in readfds will  be  watched 
to  see  if characters  become available for reading (more precisely,
to see if a read will not block - ***in particular, a file descriptor
is also ready on end-of-file***)

The reason we get the OSError is because it's a pty. pty's are wierd
about eof's because they are a terminal instead of a file or pipe
(etc, etc). So the error stream actually reads the eof (and we set
end). That's what "end" was originally used for when both streams were
pipes. Since we moved to using the pty, we now catch the OSError to
end instead.

> Also, your code has a variable 'end' which you set to 1 if you got
> nothing from the child's stdOut or stdErr, but then nothing happens with
> this variable once it's set. Was this going to be some way out of the
> while loop other than through the except block? I noticed that if I
> add a print statement and a line: "if end: break" like this:
> ...
>
>      elif f == errpi:
>        e = os.read(f, 100)
>        if e: error += e
>        else:
>          end = 1
>          print "got nothing from stderr"
>      if end: break
> except os.OSError, e:
>    print "OSError: " + str(e)
>    break
>
> ...
>
> that a normally terminating child causes the "got nothing from stderr"
> message to be printed, and the while loop exits cleanly without entering
> the "except" block. Did you have something like this in mind for 'end'? If
> that is the case, under what circumstances would execution enter the
> except block? I haven't found anything about os.read() raising an OSError.

Hopefully this makes sense from what is said above. If not, let me know.

> PS Why did you choose 100 for the size of your read buffer? Would 1 be a
> bad choice?

If you are thinking that it will let you "squeeze" more out of the
output of the process, it won't. A 1 byte read is terribly inefficient
because it requires the operating system to handle it. At least using
100 byte reads we get a reasonable amount of work done for the system
call. Also, if less than 100 bytes are available, it will still do the
read. You can add some debugging statements to the program to get a
better idea of what is going on with that.

So based on what was said above, this seems to be the better implementation:

# finished flag
end = 0

for f in rl:
	if f == outpi:
		# we might not be able to read the pty
		# so we trap the error and signal end, we
		# don't break because there might be something
		# able to be read from "error" stream
		try:
			o = read(f, 100)
		except OSError, e:
			end = 1
			continue
		output += o
	elif f == errpi:
		# error stream is a pipe so we get a proper EOF
		# and can signal end from that
		e = read(f, 100)
		if e: error += e
		else: end = 1

# exit the 'while 1' when nothing left to be read
if end:
	break

------------------------------------
Also, it's possible that if you have a very large amount of amount in
one of the buffers that the other one will signal 'end' before you
finish reading. So if you want it to be perfect, you should track
'end' for each (stdout and stderr) buffer individually and adjust the
read's and read fdset accordingly.

Jess

From ianb at colorstudy.com  Sat Nov 19 00:06:38 2005
From: ianb at colorstudy.com (Ian Bicking)
Date: Fri, 18 Nov 2005 17:06:38 -0600
Subject: [Chicago] FW: Playing with generics
In-Reply-To: <C0B0D5759E27344788E6EB554ECF4B9101FB8F3F@NEWMAIL.internal.gldd.com>
References: <C0B0D5759E27344788E6EB554ECF4B9101FB8F3F@NEWMAIL.internal.gldd.com>
Message-ID: <437E5E7E.8010807@colorstudy.com>

RCRamsdell at gldd.com wrote:
> Two concerns with this:
> It is fairly particular to the case of 2-tuple versus x and y input.  It
> is not that easily extensible to handle, for example, xyz inputs, or
> zero inputs, in which case the method should pull from the object.

I forgot to respond to the original.  Anyway, I would agree that it 
isn't a very good use of generic functions, since it's really a coersion 
that you are doing, you aren't using different algorithms depending on 
the arguments.

Generally using default arguments at the beginning of the signature is a 
pain.  In this case I don't think it's worth saving the ()'s, and you 
should always take a tuple of (x, y) as the first argument. 
Incidentally, you can do:

def calc((xa, ya), scale):
     return atan(xa/ya)*scale

Though it's a little weird looking, and doesn't lead to the best error 
messages when you get the signature wrong.



-- 
Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org

From pfein at pobox.com  Sat Nov 19 03:10:26 2005
From: pfein at pobox.com (Peter Fein)
Date: Fri, 18 Nov 2005 20:10:26 -0600
Subject: [Chicago] FW: Playing with generics
In-Reply-To: <437E5E7E.8010807@colorstudy.com>
References: <C0B0D5759E27344788E6EB554ECF4B9101FB8F3F@NEWMAIL.internal.gldd.com>
	<437E5E7E.8010807@colorstudy.com>
Message-ID: <437E8992.10101@pobox.com>

Ian Bicking wrote:

> def calc((xa, ya), scale):
>      return atan(xa/ya)*scale
> 
> Though it's a little weird looking, and doesn't lead to the best error 
> messages when you get the signature wrong.

Python's compound-tuple function arguments have to be one of the
stranger programming language features I've ever come across.  Unpacking
compound tuples in a single line[1], ok, but for function arguments,
it's just bizzare. Is this a leagacy from the days before classes?

--Pete

[1] I mean something like (a, b), c = ((1, 2), 3).  This is sometimes
useful.
-- 
Peter Fein                 pfein at pobox.com                 773-575-0694

Basically, if you're not a utopianist, you're a schmuck. -J. Feldman

From chris.mcavoy at gmail.com  Mon Nov 21 18:41:16 2005
From: chris.mcavoy at gmail.com (Chris McAvoy)
Date: Mon, 21 Nov 2005 11:41:16 -0600
Subject: [Chicago] Fwd: O'Reilly looking for UG Podcasts to Feature Online
In-Reply-To: <LYRIS-5657990-26604-2005.11.21-09.35.09--chris.mcavoy#gmail.com@newsletter.oreilly.com>
References: <LYRIS-5657990-26604-2005.11.21-09.35.09--chris.mcavoy#gmail.com@newsletter.oreilly.com>
Message-ID: <3096c19d0511210941qdc3bdeesb184dab12349722e@mail.gmail.com>

O'Reilly is getting into podcasts.

Chris "forwarding button pusher" McAvoy

---------- Forwarded message ----------
From: Marsee Henon <marsee at oreilly.com>
Date: Nov 21, 2005 11:35 AM
Subject: O'Reilly looking for UG Podcasts to Feature Online
To: chris.mcavoy at gmail.com


Hello--

I wanted to pass along the following news:

Daniel Steinberg has been working on podcasting for O'Reilly and he
called to let me know about two ways in which user group members might
want to get involved. First, if you have your own podcast, let us know
about it and we will link to it and feature your shows from time to time
on our podcasting home page (http://www.oreillynet.com/podcasts). Also,
if you have an interesting speaker coming up, you might want to do an
interview and submit it for inclusion in our magazine show "Distributing
the Future" (http://www.oreillynet.com/future). In any case, if you have
interesting ideas for how you might want to contribute to O'Reilly
podcasting, send Daniel an email at daniel at oreilly.com.

Thanks for your help,

Marsee Henon

================================================================
O'Reilly
1005 Gravenstein Highway North
Sebastopol, CA   95472
http://ug.oreilly.com/  http://www.oreilly.com
================================================================

From nttaylor at uchicago.edu  Wed Nov 23 00:49:10 2005
From: nttaylor at uchicago.edu (Noel Thomas Taylor)
Date: Tue, 22 Nov 2005 17:49:10 -0600 (CST)
Subject: [Chicago] capturing output from subprocesses
In-Reply-To: <68cb949d0511181024i110ac20cg2ff23d824b501831@mail.gmail.com>
References: <Pine.GSO.4.62.0511071805350.21726@harper.uchicago.edu>
	<68cb949d0511111547v6618c66eo2649795356dee7a0@mail.gmail.com>
	<Pine.GSO.4.62.0511121854460.29085@harper.uchicago.edu>
	<Pine.GSO.4.62.0511141457001.27908@harper.uchicago.edu>
	<68cb949d0511141330p57152b7k36cac4335dbc21ae@mail.gmail.com>
	<Pine.GSO.4.62.0511141733420.27908@harper.uchicago.edu>
	<68cb949d0511141642q617ad48ei11b289caaa87e167@mail.gmail.com>
	<Pine.GSO.4.62.0511161445580.779@harper.uchicago.edu>
	<68cb949d0511161313jdd24117xe16e57dbc1ea211f@mail.gmail.com>
	<Pine.GSO.4.62.0511171417250.876@harper.uchicago.edu>
	<68cb949d0511181024i110ac20cg2ff23d824b501831@mail.gmail.com>
Message-ID: <Pine.GSO.4.62.0511221737480.29639@harper.uchicago.edu>


Hi Jess,

I had not properly understood the difference before in how EOF is 
registered when reading from a pipe vs. when reading from a pty. All is 
much clearer now and I've implemented the code as you outlined below, 
including separate tracking of the stdOut and stdErr buffers.

By the way, I removed one level of nesting by replacing:

   for f in rl:
     if f == outpi:
       o = read(f, 100)

with:

   if outpi in rl:
     o = read(outpi, 100)

So far there seem to be no ill-effects, but I've been burned making 
seemingly innocuous changes before, so I thought I'd bring it up.

Everything is still running fine on all platforms, and my understanding of 
these functions is at an all-time high. Still, I'd like to ask you about 
one of the more subtle tricks with pseudo-terminals.

I've read that sometimes when you're passing data through a 
pseudo-terminal, it will change "\n" to "\r\n" and in this way your data, 
especially binary data, can be corrupted.

Since I'm always passing text in my implementation, this isn't a big 
problem, and I just replace all "\r" instances with the empty string.

   o = read(f, 100).replace("\r","")

But if I *did* want to pass binary data, I'd be in trouble right? The 
situation is made more complicated by the fact that the data might be 
coming from a remote host.

Do you know a smooth way around this problem?

with thanks and best wishes for the holiday,

Noel Taylor

> So based on what was said above, this seems to be the better implementation:
>
> # finished flag
> end = 0
>
> for f in rl:
> 	if f == outpi:
> 		# we might not be able to read the pty
> 		# so we trap the error and signal end, we
> 		# don't break because there might be something
> 		# able to be read from "error" stream
> 		try:
> 			o = read(f, 100)
> 		except OSError, e:
> 			end = 1
> 			continue
> 		output += o
> 	elif f == errpi:
> 		# error stream is a pipe so we get a proper EOF
> 		# and can signal end from that
> 		e = read(f, 100)
> 		if e: error += e
> 		else: end = 1
>
> # exit the 'while 1' when nothing left to be read
> if end:
> 	break
>
> ------------------------------------
> Also, it's possible that if you have a very large amount of amount in
> one of the buffers that the other one will signal 'end' before you
> finish reading. So if you want it to be perfect, you should track
> 'end' for each (stdout and stderr) buffer individually and adjust the
> read's and read fdset accordingly.
>
> Jess
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>

From RCRamsdell at gldd.com  Wed Nov 23 13:10:38 2005
From: RCRamsdell at gldd.com (RCRamsdell@gldd.com)
Date: Wed, 23 Nov 2005 06:10:38 -0600
Subject: [Chicago] FW: Playing with generics
Message-ID: <C0B0D5759E27344788E6EB554ECF4B910259076A@NEWMAIL.internal.gldd.com>



>I forgot to respond to the original.  Anyway, I would agree that it 
>isn't a very good use of generic functions, since it's really a coersion 
>that you are doing, you aren't using different algorithms depending on 
>the arguments.
Yup.  I was really just trying them out.


>Generally using default arguments at the beginning of the signature is a 
>pain.  In this case I don't think it's worth saving the ()'s, and you 
>should always take a tuple of (x, y) as the first argument. 
>Incidentally, you can do:

>def calc((xa, ya), scale):
>    return atan(xa/ya)*scale

>Though it's a little weird looking, and doesn't lead to the best error 
>messages when you get the signature wrong.
For some reason this didn't even occur to me, but it is probably the cleanest way.  I'll probably also lose the scale factor, on the theory that the internal units should be consistent, and scaling should be applied at input/output to human-readable form.

Robert
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/ms-tnef
Size: 2940 bytes
Desc: not available
Url : http://mail.python.org/pipermail/chicago/attachments/20051123/f6bdcdf9/attachment.bin