Question about unreasonable slowness
steve at REMOVE.THIS.cybersource.com.au
Thu Nov 16 23:27:12 CET 2006
On Thu, 16 Nov 2006 12:45:18 -0800, allenjo5 wrote:
> [ Warning: I'm new to Python. Don't know it at all really yet, but had
> to examine some 3rd party code because of performance problems with it.
> Here's a code snippet:
> i = 0
> while (i < 20):
> i = i + 1
You probably want to change that to:
for i in range(20):
If 20 is just a place-holder, and the real value is much bigger, change
the range() to xrange().
> (shellIn, shellOut) = os.popen4("/bin/sh -c ':'") # for testing, the
> spawned shell does nothing
> print 'next'
> # for line in shellOut:
> # print line
> On my system (AIX 5.1 if it matters, with Python 2.4.3), this simple
> loop spawning 20 subshells takes .75 sec. Ok, that's reasonable. Now,
> if I uncomment the two commented lines, which loop over the empty
> shellOut array, the progam now takes 11 secs. That slowdown seems
> very hard to believe. Why should it slow down so much?
What are you using to time the code?
Replacing print statements with "pass", I get these results:
>>> import timeit
>>> def test():
... i = 0
... while (i < 20):
... i = i + 1
... (shellIn, shellOut) = os.popen4("/bin/sh -c ':'")
... pass # print 'next'
... for line in shellOut:
... pass # print line
>>> timeit.Timer("test()", "from __main__ import test\nimport os").timeit(1)
>>> timeit.Timer("test()", "from __main__ import test\nimport os").timeit(100)
About 0.5 second to open and dispose of 20 subshells, even with the "for
line in shellOut" loop.
I think you need some more fine-grained testing to determine whether the
slowdown is actually happening inside the "for line in shellOut" loop or
inside the while loop or when the while loop completes.
More information about the Python-list