Hello Everyone! My name is Dirk Moors, and since 4 years now, I've been involved in developing a cloud computing platform, using Python as the programming language. A year ago I discovered Twisted Python, and it got me very interested, upto the point where I made the decision to convert our platform (in progress) to a Twisted platform. One year later I'm still very enthousiastic about the overal performance and stability, but last week I encountered something I did't expect; It appeared that it was less efficient to run small "atomic" operations in different deferred-callbacks, when compared to running these "atomic" operations together in "blocking" mode. Am I doing something wrong here? To prove the problem to myself, I created the following example (Full source- and test code is attached): --------------------------------------------------------------------------------------------------------------------------------------------------------------------- import struct def int2binAsync(anInteger): def packStruct(i): #Packs an integer, result is 4 bytes return struct.pack("i", i) d = defer.Deferred() d.addCallback(packStruct) reactor.callLater(0, d.callback, anInteger) return d def bin2intAsync(aBin): def unpackStruct(p): #Unpacks a bytestring into an integer return struct.unpack("i", p)[0] d = defer.Deferred() d.addCallback(unpackStruct) reactor.callLater(0, d.callback, aBin) return d def int2binSync(anInteger): #Packs an integer, result is 4 bytes return struct.pack("i", anInteger) def bin2intSync(aBin): #Unpacks a bytestring into an integer return struct.unpack("i", aBin)[0] --------------------------------------------------------------------------------------------------------------------------------------------------------------------- While running the testcode I got the following results: (1 run = converting an integer to a byte string, converting that byte string back to an integer, and finally checking whether that last integer is the same as the input integer.) *** Starting Synchronous Benchmarks. *(No Twisted => "blocking" code)* -> Synchronous Benchmark (1 runs) Completed in 0.0 seconds. -> Synchronous Benchmark (10 runs) Completed in 0.0 seconds. -> Synchronous Benchmark (100 runs) Completed in 0.0 seconds. -> Synchronous Benchmark (1000 runs) Completed in 0.00399994850159 seconds. -> Synchronous Benchmark (10000 runs) Completed in 0.0369999408722 seconds. -> Synchronous Benchmark (100000 runs) Completed in 0.362999916077 seconds. *** Synchronous Benchmarks Completed in* 0.406000137329* seconds. *** Starting Asynchronous Benchmarks . *(Twisted => "non-blocking" code)* -> Asynchronous Benchmark (1 runs) Completed in 34.5090000629 seconds. -> Asynchronous Benchmark (10 runs) Completed in 34.5099999905 seconds. -> Asynchronous Benchmark (100 runs) Completed in 34.5130000114 seconds. -> Asynchronous Benchmark (1000 runs) Completed in 34.5859999657 seconds. -> Asynchronous Benchmark (10000 runs) Completed in 35.2829999924 seconds. -> Asynchronous Benchmark (100000 runs) Completed in 41.492000103 seconds. *** Asynchronous Benchmarks Completed in *42.1460001469* seconds. Am I really seeing factor 100x?? I really hope that I made a huge reasoning error here but I just can't find it. If my results are correct then I really need to go and check my entire cloud platform for the places where I decided to split functions into atomic operations while thinking that it would actually improve the performance while on the contrary it did the opposit. I personaly suspect that I lose my cpu-cycles to the reactor scheduling the deferred-callbacks. Would that assumption make any sense? The part where I need these conversion functions is in marshalling/protocol reading and writing throughout the cloud platform, which implies that these functions will be called constantly so I need them to be superfast. I always though I had to split the entire marshalling process into small atomic (deferred-callback) functions to be efficient, but these figures tell me otherwise. I really hope someone can help me out here. Thanks in advance, Best regards, Dirk Moors