Python popenX() slowness on AIX?
Stefaan A Eeckels
hoendech at ecc.lu
Fri Nov 24 23:08:28 CET 2006
On 24 Nov 2006 09:03:41 -0800
allenjo5 at mail.northgrum.com wrote:
> Stefaan A Eeckels wrote:
> > On 21 Nov 2006 13:02:14 -0800
> > allenjo5 at mail.northgrum.com wrote:
> > > The fact that it does this in Python code instead of C is the main
> > > cause of the slowness. So, unless Python is changed to do this
> > > in C, it's always going to be slow on AIX :-(
> > I guess that the reason it's slow is that there are many
> > descriptors to try and close. Reducing them using ulimit -n could
> > improve the speed.
> > AIX has a fcntl command to close all open file descriptors from a
> > descriptor onwards:
> > fcntl(3, F_CLOSEM);
> > This of course should be used instead of the loop:
> > 10 happens to be the value of F_CLOSEM (from /usr/include/fcntl.h).
> > I've currently no access to an AIX system with Python, but it could
> > be worth trying.
> Yes, very much worth it. F_CLOSEM is _so_ much better than the loop,
> even in C. Using your brilliant suggestion, I now have a simple
> patch to the python source that implements it for any OS that happens
> to have the fcntl F_CLOSEM option.
The *BSDs and Solaris have "closefrom(3)" which does the same as
F_CLOSEM in AIX. As with AIX, the speedup is dramatic when there are a
lot of file descriptors to try and close.
> It is below in its entirety. I believe I got the try: stuff correct,
> but since I'm new to Python, I'd appreciate any comments.
I'm no great Python specialist myself. I'll leave it to those better
qualified to comment.
> I have another patch to implement my os.rclose(x,y) method, which
> would improve the speed of popenX() for the OSes that don't have
> F_CLOSEM, by doing the close() loop in C instead of Python, but I
> don't know if it would be as likely to be accepted as this probably
> would be.
> Now, where do I send my proposed patch for consideration?
The README file in the Python distribution has the following to say:
Patches and contributions
To submit a patch or other contribution, please use the Python Patch
Manager at http://sourceforge.net/patch/?group_id=5470. Guidelines
for patch submission may be found at http://www.python.org/patches/.
Stefaan A Eeckels
Never explain by malice what can be adequately explained by stupidity.
Sufficiently advanced stupidity is indistinguishable from malice.
More information about the Python-list