Python popenX() slowness on AIX?

[Stefaan A Eeckels]

On 24 Nov 2006 09:03:41 -0800
allenjo5 at mail.northgrum.com wrote:

> Stefaan A Eeckels wrote:
> > On 21 Nov 2006 13:02:14 -0800
> > allenjo5 at mail.northgrum.com wrote:
> >
> > > The fact that it does this in Python code instead of C is the main
> > > cause of the slowness.  So, unless Python is changed to do this
> > > in C, it's always going to be slow on AIX :-(
> >
> > I guess that the reason it's slow is that there are many
> > descriptors to try and close. Reducing them using ulimit -n could
> > improve the speed.
> >
> > AIX has a fcntl command to close all open file descriptors from a
> > descriptor onwards:
> >
> > fcntl(3, F_CLOSEM);
> >
> > This of course should be used instead of the loop:
> >
> > 10 happens to be the value of F_CLOSEM (from /usr/include/fcntl.h).
> > I've currently no access to an AIX system with Python, but it could
> > be worth trying.
> 
> Yes, very much worth it.  F_CLOSEM is _so_ much better than the loop,
> even in C.   Using your brilliant suggestion, I now have a simple
> patch to the python source that implements it for any OS that happens
> to have the fcntl F_CLOSEM option.   

The *BSDs and Solaris have "closefrom(3)" which does the same as
F_CLOSEM in AIX. As with AIX, the speedup is dramatic when there are a
lot of file descriptors to try and close.

> It is below in its entirety.  I believe I got the try: stuff correct,
> but since I'm new to Python, I'd appreciate any comments.

I'm no great Python specialist myself. I'll leave it to those better
qualified to comment.

> I have another patch to implement my os.rclose(x,y) method, which
> would improve the speed of popenX() for the OSes that don't have
> F_CLOSEM, by doing the close() loop in C instead of Python,  but I
> don't know if it would be as likely to be accepted as this probably
> would be.
> 
> Now, where do I send my proposed patch for consideration?

The README file in the Python distribution has the following to say:

Patches and contributions
-------------------------

To submit a patch or other contribution, please use the Python Patch
Manager at http://sourceforge.net/patch/?group_id=5470.  Guidelines
for patch submission may be found at http://www.python.org/patches/.

Take care,

-- 
Stefaan A Eeckels
-- 
Never explain by malice what can be adequately explained by stupidity.
However:
Sufficiently advanced stupidity is indistinguishable from malice.