[Twisted-Python] Twisted slow in Mac OS X fast on Windows
Hi, We use Twisted extensively in the IPython project for parallel and distributed computing. We have an extensive test suite that uses trial to test our servers and clients. Everything works great...but...the test suite takes much longer to run on OS X than windows. Here is what I mean: 1. 2.53 Ghz Intel Core 2 Duo, Mac OS X 10.5. Test suite takes 67 seconds 2. *Same machine*, running Windows XP as VM in VMWare. Test suite takes 17 seconds. I have verified that all the tests are being run on Windows. This is very surprising to me as I would expect, if anything, the Windows VM to be a bit slower, not 3x faster. This is the same hardware, the same code, the only thing that I can think of that is different is the reactor implementation. I am using the default reactor on both platforms. Questions: * Is the Windows reactor really that fast? * Is the default (I think it is select) reactor on OS X really that slow? * Why else would it be that slow on OS X? * What can I do to speed this up? Cheers, Brian
On Fri, 20 Feb 2009 11:11:24 -0800, Brian Granger <ellisonbg.net@gmail.com> wrote:
Hi,
We use Twisted extensively in the IPython project for parallel and distributed computing. We have an extensive test suite that uses trial to test our servers and clients. Everything works great...but...the test suite takes much longer to run on OS X than windows. Here is what I mean:
1. 2.53 Ghz Intel Core 2 Duo, Mac OS X 10.5. Test suite takes 67 seconds
2. *Same machine*, running Windows XP as VM in VMWare. Test suite takes 17 seconds.
I have verified that all the tests are being run on Windows. This is very surprising to me as I would expect, if anything, the Windows VM to be a bit slower, not 3x faster. This is the same hardware, the same code, the only thing that I can think of that is different is the reactor implementation. I am using the default reactor on both platforms.
Questions:
* Is the Windows reactor really that fast?
Probably not. But it depends what your test suite does.
* Is the default (I think it is select) reactor on OS X really that slow?
Probably not. But it depends what your test suite does.
* Why else would it be that slow on OS X? * What can I do to speed this up?
The only thing that comes to mind is that some hostname lookup functions used to be pretty slow on OS X. I have no idea if this is still the case. I suggest profiling the suite on OS X and Windows and comparing the results to get a better idea of where to look for the problem. Jean-Paul
Hi, I think it may just be the built-in Python on Leopard: On OS X: In [1]: def f(): ...: for i in range(100000): ...: a = 1 + i In [2]: %timeit f() 100 loops, best of 3: 11.8 ms per loop Same hardware, but Windows: In [1]: def f(): ...: for i in range(100000): ...: a = 1 + i In [2]: %timeit f() 100 loops, best of 3: 8.33 ms per loop That's depressing... Cheers, Brian On Fri, Feb 20, 2009 at 11:19 AM, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Fri, 20 Feb 2009 11:11:24 -0800, Brian Granger <ellisonbg.net@gmail.com> wrote:
Hi,
We use Twisted extensively in the IPython project for parallel and distributed computing. We have an extensive test suite that uses trial to test our servers and clients. Everything works great...but...the test suite takes much longer to run on OS X than windows. Here is what I mean:
1. 2.53 Ghz Intel Core 2 Duo, Mac OS X 10.5. Test suite takes 67 seconds
2. *Same machine*, running Windows XP as VM in VMWare. Test suite takes 17 seconds.
I have verified that all the tests are being run on Windows. This is very surprising to me as I would expect, if anything, the Windows VM to be a bit slower, not 3x faster. This is the same hardware, the same code, the only thing that I can think of that is different is the reactor implementation. I am using the default reactor on both platforms.
Questions:
* Is the Windows reactor really that fast?
Probably not. But it depends what your test suite does.
* Is the default (I think it is select) reactor on OS X really that slow?
Probably not. But it depends what your test suite does.
* Why else would it be that slow on OS X? * What can I do to speed this up?
The only thing that comes to mind is that some hostname lookup functions used to be pretty slow on OS X. I have no idea if this is still the case. I suggest profiling the suite on OS X and Windows and comparing the results to get a better idea of where to look for the problem.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (2)
-
Brian Granger -
Jean-Paul Calderone