Mailman 3 October 2009 - Twisted

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 17 Oct '09

17 Oct '09

1 0

[Twisted-Python] Newbie question: can't get getProcessOutput to run from inside a listenTCP service call-back
by Read Roberts 17 Oct '09

17 Oct '09

I have a Python server using twisted which responds to pyAMF calls from Flex/Air applications (following Bruce Eckels 5/1/2008 article "Concurrency with Python, Twisted, and Flex"). The service is basically as follows below. When the client calls a service API, the API needs to make a OS shell call. I see that if I call reactor.stop() after the getProcessOutput call (such as within the doLog and doError functions), the call happens and I see the output. How can can I retrieve the output without calling reactor.stop()? In the example below, the AIR app calls the callback helloWorldTest(), and the deferred.addCallback function is never called. import os from twisted.spread import pb from twisted.internet.utils import getProcessOutput from twisted.internet import reactor class FlexInterface(pb.Root): def __init__(self): self.result e def doLog(self, result): self.result = "%s" % (result) return result def doError(self, failure): self.result = "%s" % ( failure) return failure def helloWorldTest(self): deferred = getProcessOutput("echo", ["Hello World"], os.environ) deferred.addErrback(self.doError) deferred.addCallback(self.doLog) while (self.result == None) print "doLog still hasn't been called" return self.result def terminate(self, discardPostAag): reactor.callLater(1, reactor.stop) return "Terminating CubeGlyphServices" def run(): # Place the namespace mapping into a TwistedGateway: fi = FlexInterface() gateway = TwistedGateway({ "flexAppServer": fi }) # Publish the PyAMF gateway at the root URL: root = resource.Resource() root.putChild("", gateway) # Tell the twisted reactor to listen: reactor.listenTCP(8050, server.Site(root)) print "Local python server listening on localhost port 8050" reactor.run() if __name__=='__main__': run() I do know that If I used Python 2.6 and later, a twisted bug is fixed such that I could use subprocess.Popen, and that the next release of AIR will allow direct execv calls. However, for performance issues I still want to call one Python process running a server, and I have to work within a Python 2.5 environment. Thanks for any advice, Read Roberts

2 2

[Twisted-Python] maybeDeferred capability
by mardiros 16 Oct '09

16 Oct '09

As I am still a new user of Twisted, I have decide to use the maybeDeferred capability. The real aim is to cache sql queries, but I have written this example from the twisted mayneDeferred doc. I am happy with my code, but, I would like to share it to have some feed back Here it is: --- import types from datetime import datetime, timedelta from twisted.internet import defer from twisted.internet import reactor, task class asyncmemoize: def __init__(self, function,pool_time=10,cachelifetime=10): self.function = function self._cache = {} self._cachelifetime = timedelta(0,cachelifetime) self._cleaner = task.LoopingCall(self.cleanCache) self._cleaner.start(pool_time,now=False) self._cleanrun = False def __call__(self, *args, **kwargs): key = (tuple(args), frozenset(kwargs.items())) if key not in self._cache: f = self.function(*args, **kwargs).addCallback(self.callback,key) self._cache[key]=[datetime.today(), f] else: print "from cache" return self._cache[key][1] def callback(self,result,key): print "Put real value in cache" self._cache[key]=[datetime.today(), self._cachelifetime, result] return result def cleanCache(self): if not self._cleanrun: self._cleanrun = True print 'Clean the cache...' #log.msg('Clean the persistent cache...', logLevel=logging.DEBUG) expire = datetime.today() i = 0 for k,v in self._cache.items(): print v[0], "<", expire, "=>", v[0] > expire if v[0] < expire: i+=1 del self._cache[k] print '%i items removed' % i #log.msg('%i items removed' % i, logLevel=logging.DEBUG) self._cleanrun = False def saveInCache(result,key): print "saving to cache" cache[key] = result return result @asyncmemoize def asynchronousIsValidUser(user): print "Loading is authenticating from async" d = defer.Deferred() reactor.callLater(2, d.callback, user in ["Alice", "Angus", "Agnes"]) return d def synchronousIsValidUser(user): ''' Return true if user is a valid user, false otherwise ''' return user in ["Alice", "Angus", "Agnes"] def printResult(result,user): if result: print "User", user ,"is authenticated" else: print "User", user ,"is not authenticated" def authenticateUser(isValidUser, user): print "Authenticating user ", user d = defer.maybeDeferred(isValidUser, user) return d.addCallback(printResult,user) authenticateUser(asynchronousIsValidUser,"Alice") reactor.callLater(3,authenticateUser,asynchronousIsValidUser,"Alice") reactor.callLater(5,authenticateUser,asynchronousIsValidUser,"Alice") reactor.callLater(15,authenticateUser,asynchronousIsValidUser,"Alice") reactor.callLater(30,authenticateUser,asynchronousIsValidUser,"Alice") reactor.run() ---------------------------------------------------------------------------- Laposte.net fête ses 10 ans ! Gratuite, garantie à vie et déjà utilisée par des millions d'internautes... vous aussi, pour votre adresse e-mail, choisissez laposte.net. Laposte.net, bien + qu'une messagerie ----------------------------------------------------------------------------

1 0

[Twisted-Python] Replacement for CherryPy
by James Gregory 15 Oct '09

15 Oct '09

Hi all, I'm trying to replace a server that I've written with CherryPy to use Twisted. My reasons are that my particular app has the server waiting on sockets to third-party services a lot of the time, which is tying up all the worker-threads I have allocated for CherryPy => scalability nightmare. In short, I want to be able to write a class more or less like so: class C: def searchData(self, query, resultCount) : # produce a deferred that will eventually return the data to go back to the client, d. return d And have it exposed such that fetching http://server/api/searchData?query=query+string&resultCount=10 will call that searchData method, passing in the URL's GET parameters as the arguments to that method. It'd be awesome if it also worked for POSTed data too. I realise that I can probably build this myself using twisted.web etc, but am wondering if someone's already done the legwork here. Any suggestions? Thanks in advance. James.

1 0

[Twisted-Python] How to find out if exceptions are being raised in your errBack?
by Steve Steiner (listsin) 14 Oct '09

14 Oct '09

I've been hunting down a problem that I've finally found the cause of and I'd like to know what's the Twisted way to catch this "error within the code handling the error" type of error. Basically, in one branch of the errBack, there was a typo. A simple typo that caused an unhandled NameError exception, but only once in a few thousand runs. The exception got caught and "displayed" by Twisted, but it wasn't going anyplace anyone was looking (buried under zillions of lines of logging) and the app continues on as if nothing went wrong. I've put up a simple app that demonstrates the issue: http://pastebin.com/m59217f60 If you put in a 404 error URL, let it run through, you'll see the 404 error printed out, the exception will occur in the background, and the program just keeps on going. If you then hit Ctrl-C, you can see the traceback showing that Twisted caught the NameError . What is the best way to handle programming errors like this in deferreds so they don't slip by, unnoticed? Thanks, S (~/twisted_err)# ./errs_away.py URL: http://www.yahoo.com line = http://www.yahoo.com Got data, len == 9490 URL: http://thereisnodomainnamedthis.com line = http://thereisnodomainnamedthis.com Error: DNS lookup failed: address 'thereisnodomainnamedthis.com' not found: [Errno 8] nodename nor servname provided, or not known. URL: http://www.yahoo.com/non-existent-page line = http://www.yahoo.com/non-existent-page Error: 400 Bad Request <====== This triggers the code with the bad variable URL: ^C <============= manually stop the program ============= Then, you get to see the traceback ================== Unhandled error in Deferred: Traceback (most recent call last): File "/System/Library/Frameworks/Python.framework/Versions/2.6/ Extras/lib/python/twisted/web/client.py", line 143, in handleResponse self.status, self.message, response))) File "/System/Library/Frameworks/Python.framework/Versions/2.6/ Extras/lib/python/twisted/web/client.py", line 309, in noPage self.deferred.errback(reason) File "/System/Library/Frameworks/Python.framework/Versions/2.6/ Extras/lib/python/twisted/internet/defer.py", line 269, in errback self._startRunCallbacks(fail) File "/System/Library/Frameworks/Python.framework/Versions/2.6/ Extras/lib/python/twisted/internet/defer.py", line 312, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "/System/Library/Frameworks/Python.framework/Versions/2.6/ Extras/lib/python/twisted/internet/defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File "./errs_away.py", line 15, in printError print oops # variable's not defined... exceptions.NameError: global name 'oops' is not defined Thanks, S

4 7

Re: [Twisted-Python] Twisted-Python Digest, Vol 67, Issue 23
by Dirk Moors 13 Oct '09

13 Oct '09

Hello Reza, I tried the solution you provided and I have to say, that changed a lot! You gave me a better understanding of how things work with Twisted, and I really appreciate your response! Thanks for your help! Best regards, Dirk Moors 2009/10/13 <twisted-python-request(a)twistedmatrix.com> > Send Twisted-Python mailing list submissions to > twisted-python(a)twistedmatrix.com > > To subscribe or unsubscribe via the World Wide Web, visit > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > or, via email, send a message with subject or body 'help' to > twisted-python-request(a)twistedmatrix.com > > You can reach the person managing the list at > twisted-python-owner(a)twistedmatrix.com > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Twisted-Python digest..." > > > Today's Topics: > > 1. Re: Twisted Python vs. "Blocking" Python: Weird performance > on small operations. (Reza Lotun) > 2. Re: Twisted-Python Digest, Vol 67, Issue 22 (Dirk Moors) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 13 Oct 2009 15:04:06 +0100 > From: Reza Lotun <rlotun(a)gmail.com> > Subject: Re: [Twisted-Python] Twisted Python vs. "Blocking" Python: > Weird performance on small operations. > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > Message-ID: > <95bb10690910130704o7c0ff2besf00dcf5918990dcf(a)mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > Hi Dirk, > > I took a look at your code sample and got the async benchmark to run > with the following values: > *** Starting Asynchronous Benchmarks. > > -> Asynchronous Benchmark (1 runs) Completed in 0.000181913375854 seconds. > -> Asynchronous Benchmark (10 runs) Completed in 0.000736951828003 > seconds. > -> Asynchronous Benchmark (100 runs) Completed in 0.00641012191772 > seconds. > -> Asynchronous Benchmark (1000 runs) Completed in 0.0741751194 seconds. > -> Asynchronous Benchmark (10000 runs) Completed in 0.675071001053 > seconds. > -> Asynchronous Benchmark (100000 runs) Completed in 7.29738497734 > seconds. > > *** Asynchronous Benchmarks Completed in 8.16032314301 seconds. > > Which, though still quite a bit slower than the synchronous version, > is still much better than the 40 sec. mark that you were experiencing. > My modified version simply returned defer.succeed from your aync > block-compute functions. > > i.e. Instead of your initial example: > def int2binAsync(anInteger): > def packStruct(i): > #Packs an integer, result is 4 bytes > return struct.pack("i", i) > > d = defer.Deferred() > d.addCallback(packStruct) > > reactor.callLater(0, > d.callback, > anInteger) > > return d > > my version does: > > def int2binAsync(anInteger): > return defer.succeed(struct.pack('i', anInteger)) > > A few things to note in general however: > 1) Twisted shines for block I/O operations - i.e. networking. A > compute intesive process will not necessarily yield any gains in > performance by using Twisted since the Python GIL exists (a global > lock). > > 2) If you are doing computations that use a C module (unforunately > struct pre 2.6 I believe doesn't use a C module), there may be a > chance that the C module releases the GIL, allowing you to do those > computations in a thread. In this case you'd be better off using > deferToThread as suggested earlier. > > 3) There is some (usually minimal but it exists) overhead to using > Twisted. Instead of computing a bunch of stuff serially and returning > your answer as in your sync example, you're wrapping everything up in > deferreds and starting a reactor - it's definitely going to be a bit > slower than the pure synchronous version for this case. > > Hope that makes sense. > > Cheers, > Reza > > > -- > Reza Lotun > mobile: +44 (0)7521 310 763 > email: rlotun(a)gmail.com > work: reza(a)tweetdeck.com > twitter: @rlotun > > > > ------------------------------ > > Message: 2 > Date: Tue, 13 Oct 2009 16:18:35 +0200 > From: Dirk Moors <dirkmoors(a)gmail.com> > Subject: Re: [Twisted-Python] Twisted-Python Digest, Vol 67, Issue 22 > To: twisted-python(a)twistedmatrix.com > Message-ID: > <cf75a1410910130718m53645515oc65f0890366a12f2(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hello Valeriy, > > I tried the thing you suggested, and I attached the (updated) code. > Unfortunatly, the new code was even slower, producing the following > results: > > *** Starting Asynchronous Benchmarks. (Using Twisted, with > "deferred-decorator") > -> Asynchronous Benchmark (1 runs) Completed in 56.0279998779 seconds. > -> Asynchronous Benchmark (10 runs) Completed in 56.0130000114 seconds. > -> Asynchronous Benchmark (100 runs) Completed in 56.010999918 seconds. > -> Asynchronous Benchmark (1000 runs) Completed in 56.0410001278 seconds. > -> Asynchronous Benchmark (10000 runs) Completed in 56.3069999218 seconds. > -> Asynchronous Benchmark (100000 runs) Completed in 58.8910000324 > seconds. > *** Asynchronous Benchmarks Completed in 59.4659998417 seconds. > > I suspect that this would me more inefficient because with the > deferToThread > function in place, every single operation will be executed in its own > thread, which means: > (1 x 2) + (10 x 2) + (100 x 2) + (1000 x 2) + (10000 x 2) + (100000 x 2) > threads....which is...a lot. > > Maybe the problem lies in the way I test the code? I understand that using > the asynchronous testcode this way (generating the deferreds using a > FOR-loop), a lot of deferreds are generated before the reactor starts > calling the deferred-callbacks.....would there be another, better way to > test the code? > The reason I need to now which one is faster (async vs sync functions) is > because I need to decide on whetehr or not I should re-evaluate the code I > just recently finished building. > > Any other ideas maybe? > > Thanks in advance, > Dirk > > > > ________________________________________________________________________________________________________________________________________________________ > > Message: 3 > > Date: Tue, 13 Oct 2009 09:41:19 -0400 > > From: Valeriy Pogrebitskiy <vpogrebi(a)verizon.net> > > Subject: Re: [Twisted-Python] Twisted Python vs. "Blocking" Python: > > Weird performance on small operations. > > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > > Message-ID: <EDB2B354-B25D-4A98-AC9D-B9745CA6C3AB(a)verizon.net> > > Content-Type: text/plain; charset="us-ascii" > > > > Dirk, > > > > Using deferred directly in your bin2intAsync() may be somewhat less > > efficient than some other way described in Recipe 439358: [Twisted] > > From blocking functions to deferred functions > > > > recipe (http://code.activestate.com/recipes/439358/) > > > > You would get same effect (asynchronous execution) - but potentially > > more efficiently - by just decorating your synchronous methods as: > > > > from twisted.internet.threads import deferToThread > > deferred = deferToThread.__get__ > > .... > > @deferred > > def int2binAsync(anInteger): > > #Packs an integer, result is 4 bytes > > return struct.pack("i", anInteger) > > > > @deferred > > def bin2intAsync(aBin): > > #Unpacks a bytestring into an integer > > return struct.unpack("i", aBin)[0] > > > > > > > > > > Kind regards, > > > > Valeriy Pogrebitskiy > > vpogrebi(a)verizon.net > > > > > > > > > > On Oct 13, 2009, at 9:18 AM, Dirk Moors wrote: > > > > > Hello Everyone! > > > > > > My name is Dirk Moors, and since 4 years now, I've been involved in > > > developing a cloud computing platform, using Python as the > > > programming language. A year ago I discovered Twisted Python, and it > > > got me very interested, upto the point where I made the decision to > > > convert our platform (in progress) to a Twisted platform. One year > > > later I'm still very enthousiastic about the overal performance and > > > stability, but last week I encountered something I did't expect; > > > > > > It appeared that it was less efficient to run small "atomic" > > > operations in different deferred-callbacks, when compared to running > > > these "atomic" operations together in "blocking" mode. Am I doing > > > something wrong here? > > > > > > To prove the problem to myself, I created the following example > > > (Full source- and test code is attached): > > > > > > --------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > import struct > > > > > > def int2binAsync(anInteger): > > > def packStruct(i): > > > #Packs an integer, result is 4 bytes > > > return struct.pack("i", i) > > > > > > d = defer.Deferred() > > > d.addCallback(packStruct) > > > > > > reactor.callLater(0, > > > d.callback, > > > anInteger) > > > > > > return d > > > > > > def bin2intAsync(aBin): > > > def unpackStruct(p): > > > #Unpacks a bytestring into an integer > > > return struct.unpack("i", p)[0] > > > > > > d = defer.Deferred() > > > d.addCallback(unpackStruct) > > > > > > reactor.callLater(0, > > > d.callback, > > > aBin) > > > return d > > > > > > def int2binSync(anInteger): > > > #Packs an integer, result is 4 bytes > > > return struct.pack("i", anInteger) > > > > > > def bin2intSync(aBin): > > > #Unpacks a bytestring into an integer > > > return struct.unpack("i", aBin)[0] > > > > > > > > > --------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > > > > While running the testcode I got the following results: > > > > > > (1 run = converting an integer to a byte string, converting that > > > byte string back to an integer, and finally checking whether that > > > last integer is the same as the input integer.) > > > > > > *** Starting Synchronous Benchmarks. (No Twisted => "blocking" code) > > > -> Synchronous Benchmark (1 runs) Completed in 0.0 seconds. > > > -> Synchronous Benchmark (10 runs) Completed in 0.0 seconds. > > > -> Synchronous Benchmark (100 runs) Completed in 0.0 seconds. > > > -> Synchronous Benchmark (1000 runs) Completed in 0.00399994850159 > > > seconds. > > > -> Synchronous Benchmark (10000 runs) Completed in 0.0369999408722 > > > seconds. > > > -> Synchronous Benchmark (100000 runs) Completed in 0.362999916077 > > > seconds. > > > *** Synchronous Benchmarks Completed in 0.406000137329 seconds. > > > > > > *** Starting Asynchronous Benchmarks . (Twisted => "non-blocking" > > > code) > > > -> Asynchronous Benchmark (1 runs) Completed in 34.5090000629 > > > seconds. > > > -> Asynchronous Benchmark (10 runs) Completed in 34.5099999905 > > > seconds. > > > -> Asynchronous Benchmark (100 runs) Completed in 34.5130000114 > > > seconds. > > > -> Asynchronous Benchmark (1000 runs) Completed in 34.5859999657 > > > seconds. > > > -> Asynchronous Benchmark (10000 runs) Completed in 35.2829999924 > > > seconds. > > > -> Asynchronous Benchmark (100000 runs) Completed in 41.492000103 > > > seconds. > > > *** Asynchronous Benchmarks Completed in 42.1460001469 seconds. > > > > > > Am I really seeing factor 100x?? > > > > > > I really hope that I made a huge reasoning error here but I just > > > can't find it. If my results are correct then I really need to go > > > and check my entire cloud platform for the places where I decided to > > > split functions into atomic operations while thinking that it would > > > actually improve the performance while on the contrary it did the > > > opposit. > > > > > > I personaly suspect that I lose my cpu-cycles to the reactor > > > scheduling the deferred-callbacks. Would that assumption make any > > > sense? > > > The part where I need these conversion functions is in marshalling/ > > > protocol reading and writing throughout the cloud platform, which > > > implies that these functions will be called constantly so I need > > > them to be superfast. I always though I had to split the entire > > > marshalling process into small atomic (deferred-callback) functions > > > to be efficient, but these figures tell me otherwise. > > > > > > I really hope someone can help me out here. > > > > > > Thanks in advance, > > > Best regards, > > > Dirk Moors > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <twistedbenchmark.py>_______________________________________________ > > > Twisted-Python mailing list > > > Twisted-Python(a)twistedmatrix.com > > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > >

2 1

[Twisted-Python] python-twisted-akonadi and akonadi-gtk
by Stephen Kelly 13 Oct '09

13 Oct '09

Hi, Twisted is a framework for event driven applications. Typically client- server architectures can be implemented with Twisted. Existing servers and clients exist for a long list of protocols and communication devices including HTTP, SSH, and notably for my purpose, IMAP and UNIX sockets. It provides an event loop, and asynchronous Deferred objects which are similar to KJob objects. http://twistedmatrix.com/trac/ Akonadi is a cross platform PIM (personal information management) framework developed as part of the KDE4 platform. The goals of Akonadi include * Isolating user applications handing PIM data such as emails, contacts, notes etc from the protocols used to access or store that data (IMAP, POP, Maildir, Groupware, vcards, etc). * Provide a single point of storage (actually a cache) of PIM data accessible and manipulatable by any application written in any language on the target platform. http://pim.kde.org/akonadi/ Akonadi is designed as a client/server architecture. The server is written in Qt/C++, and we already have one client library for interfacing with the server written in C++ using the KDE platform. Notifications of changes to data are transmitted over D-Bus, and the actual data is transferred over a local socket (on Unix. On windows it's a named pipe). The protocol used for communication is IMAP with some non-standard extensions. And so the purpose emerges :). I have started an Akonadi client library written in python using twisted- imap with some extensions on top of it for Akonadi specific functionality. The code currently lives here: http://gitorious.org/python-twisted-akonadi There is the twisted-akonadi library a gtkAkonadi library containing some high level classes for PyGtk applications and a simple email reader and addressbook written in pygtk. Because Akonadi keeps everything in sync, you can change items in a KDE application, the gtk application, or the django application, and the other two will be instantly updated with the change. I've blogged about it twice already here: * http://steveire.wordpress.com/2009/10/09/holy-grail-no-thanks-weve- already-got-one/ * http://steveire.wordpress.com/2009/10/13/cross-platform-akonadi-video/ As you can see, this is only a proof of concept of the project. The aim is to create a library which feels pythonic and natural to use for twisted users. If you think I've started in the wrong way, or you have ideas for ways this API could be used, please let me know. I've just started with twisted, so I've probably not found some stuff which would make this task easier. Additionally, if you would like to contribute to the project, that would be very welcome. :) If you find the ideas here interesting and want to know more, the Akonadi developers are in #akonadi on Freenode and kde-pim at KDE.org, and I am already in #twisted. It isn't quite a twisted success story yet, but I think it has the potential to become one. All the best, Steve.

1 0

Re: [Twisted-Python] Twisted-Python Digest, Vol 67, Issue 22
by Dirk Moors 13 Oct '09

13 Oct '09

Hello Valeriy, I tried the thing you suggested, and I attached the (updated) code. Unfortunatly, the new code was even slower, producing the following results: *** Starting Asynchronous Benchmarks. (Using Twisted, with "deferred-decorator") -> Asynchronous Benchmark (1 runs) Completed in 56.0279998779 seconds. -> Asynchronous Benchmark (10 runs) Completed in 56.0130000114 seconds. -> Asynchronous Benchmark (100 runs) Completed in 56.010999918 seconds. -> Asynchronous Benchmark (1000 runs) Completed in 56.0410001278 seconds. -> Asynchronous Benchmark (10000 runs) Completed in 56.3069999218 seconds. -> Asynchronous Benchmark (100000 runs) Completed in 58.8910000324 seconds. *** Asynchronous Benchmarks Completed in 59.4659998417 seconds. I suspect that this would me more inefficient because with the deferToThread function in place, every single operation will be executed in its own thread, which means: (1 x 2) + (10 x 2) + (100 x 2) + (1000 x 2) + (10000 x 2) + (100000 x 2) threads....which is...a lot. Maybe the problem lies in the way I test the code? I understand that using the asynchronous testcode this way (generating the deferreds using a FOR-loop), a lot of deferreds are generated before the reactor starts calling the deferred-callbacks.....would there be another, better way to test the code? The reason I need to now which one is faster (async vs sync functions) is because I need to decide on whetehr or not I should re-evaluate the code I just recently finished building. Any other ideas maybe? Thanks in advance, Dirk ________________________________________________________________________________________________________________________________________________________ > Message: 3 > Date: Tue, 13 Oct 2009 09:41:19 -0400 > From: Valeriy Pogrebitskiy <vpogrebi(a)verizon.net> > Subject: Re: [Twisted-Python] Twisted Python vs. "Blocking" Python: > Weird performance on small operations. > To: Twisted general discussion <twisted-python(a)twistedmatrix.com> > Message-ID: <EDB2B354-B25D-4A98-AC9D-B9745CA6C3AB(a)verizon.net> > Content-Type: text/plain; charset="us-ascii" > > Dirk, > > Using deferred directly in your bin2intAsync() may be somewhat less > efficient than some other way described in Recipe 439358: [Twisted] > From blocking functions to deferred functions > > recipe (http://code.activestate.com/recipes/439358/) > > You would get same effect (asynchronous execution) - but potentially > more efficiently - by just decorating your synchronous methods as: > > from twisted.internet.threads import deferToThread > deferred = deferToThread.__get__ > .... > @deferred > def int2binAsync(anInteger): > #Packs an integer, result is 4 bytes > return struct.pack("i", anInteger) > > @deferred > def bin2intAsync(aBin): > #Unpacks a bytestring into an integer > return struct.unpack("i", aBin)[0] > > > > > Kind regards, > > Valeriy Pogrebitskiy > vpogrebi(a)verizon.net > > > > > On Oct 13, 2009, at 9:18 AM, Dirk Moors wrote: > > > Hello Everyone! > > > > My name is Dirk Moors, and since 4 years now, I've been involved in > > developing a cloud computing platform, using Python as the > > programming language. A year ago I discovered Twisted Python, and it > > got me very interested, upto the point where I made the decision to > > convert our platform (in progress) to a Twisted platform. One year > > later I'm still very enthousiastic about the overal performance and > > stability, but last week I encountered something I did't expect; > > > > It appeared that it was less efficient to run small "atomic" > > operations in different deferred-callbacks, when compared to running > > these "atomic" operations together in "blocking" mode. Am I doing > > something wrong here? > > > > To prove the problem to myself, I created the following example > > (Full source- and test code is attached): > > > --------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > import struct > > > > def int2binAsync(anInteger): > > def packStruct(i): > > #Packs an integer, result is 4 bytes > > return struct.pack("i", i) > > > > d = defer.Deferred() > > d.addCallback(packStruct) > > > > reactor.callLater(0, > > d.callback, > > anInteger) > > > > return d > > > > def bin2intAsync(aBin): > > def unpackStruct(p): > > #Unpacks a bytestring into an integer > > return struct.unpack("i", p)[0] > > > > d = defer.Deferred() > > d.addCallback(unpackStruct) > > > > reactor.callLater(0, > > d.callback, > > aBin) > > return d > > > > def int2binSync(anInteger): > > #Packs an integer, result is 4 bytes > > return struct.pack("i", anInteger) > > > > def bin2intSync(aBin): > > #Unpacks a bytestring into an integer > > return struct.unpack("i", aBin)[0] > > > > > --------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > > While running the testcode I got the following results: > > > > (1 run = converting an integer to a byte string, converting that > > byte string back to an integer, and finally checking whether that > > last integer is the same as the input integer.) > > > > *** Starting Synchronous Benchmarks. (No Twisted => "blocking" code) > > -> Synchronous Benchmark (1 runs) Completed in 0.0 seconds. > > -> Synchronous Benchmark (10 runs) Completed in 0.0 seconds. > > -> Synchronous Benchmark (100 runs) Completed in 0.0 seconds. > > -> Synchronous Benchmark (1000 runs) Completed in 0.00399994850159 > > seconds. > > -> Synchronous Benchmark (10000 runs) Completed in 0.0369999408722 > > seconds. > > -> Synchronous Benchmark (100000 runs) Completed in 0.362999916077 > > seconds. > > *** Synchronous Benchmarks Completed in 0.406000137329 seconds. > > > > *** Starting Asynchronous Benchmarks . (Twisted => "non-blocking" > > code) > > -> Asynchronous Benchmark (1 runs) Completed in 34.5090000629 > > seconds. > > -> Asynchronous Benchmark (10 runs) Completed in 34.5099999905 > > seconds. > > -> Asynchronous Benchmark (100 runs) Completed in 34.5130000114 > > seconds. > > -> Asynchronous Benchmark (1000 runs) Completed in 34.5859999657 > > seconds. > > -> Asynchronous Benchmark (10000 runs) Completed in 35.2829999924 > > seconds. > > -> Asynchronous Benchmark (100000 runs) Completed in 41.492000103 > > seconds. > > *** Asynchronous Benchmarks Completed in 42.1460001469 seconds. > > > > Am I really seeing factor 100x?? > > > > I really hope that I made a huge reasoning error here but I just > > can't find it. If my results are correct then I really need to go > > and check my entire cloud platform for the places where I decided to > > split functions into atomic operations while thinking that it would > > actually improve the performance while on the contrary it did the > > opposit. > > > > I personaly suspect that I lose my cpu-cycles to the reactor > > scheduling the deferred-callbacks. Would that assumption make any > > sense? > > The part where I need these conversion functions is in marshalling/ > > protocol reading and writing throughout the cloud platform, which > > implies that these functions will be called constantly so I need > > them to be superfast. I always though I had to split the entire > > marshalling process into small atomic (deferred-callback) functions > > to be efficient, but these figures tell me otherwise. > > > > I really hope someone can help me out here. > > > > Thanks in advance, > > Best regards, > > Dirk Moors > > > > > > > > > > > > > > > > > > > > > > > > > > > > <twistedbenchmark.py>_______________________________________________ > > Twisted-Python mailing list > > Twisted-Python(a)twistedmatrix.com > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > >

2 1

[Twisted-Python] Twisted Python vs. "Blocking" Python: Weird performance on small operations.
by Dirk Moors 13 Oct '09

13 Oct '09

Hello Everyone! My name is Dirk Moors, and since 4 years now, I've been involved in developing a cloud computing platform, using Python as the programming language. A year ago I discovered Twisted Python, and it got me very interested, upto the point where I made the decision to convert our platform (in progress) to a Twisted platform. One year later I'm still very enthousiastic about the overal performance and stability, but last week I encountered something I did't expect; It appeared that it was less efficient to run small "atomic" operations in different deferred-callbacks, when compared to running these "atomic" operations together in "blocking" mode. Am I doing something wrong here? To prove the problem to myself, I created the following example (Full source- and test code is attached): --------------------------------------------------------------------------------------------------------------------------------------------------------------------- import struct def int2binAsync(anInteger): def packStruct(i): #Packs an integer, result is 4 bytes return struct.pack("i", i) d = defer.Deferred() d.addCallback(packStruct) reactor.callLater(0, d.callback, anInteger) return d def bin2intAsync(aBin): def unpackStruct(p): #Unpacks a bytestring into an integer return struct.unpack("i", p)[0] d = defer.Deferred() d.addCallback(unpackStruct) reactor.callLater(0, d.callback, aBin) return d def int2binSync(anInteger): #Packs an integer, result is 4 bytes return struct.pack("i", anInteger) def bin2intSync(aBin): #Unpacks a bytestring into an integer return struct.unpack("i", aBin)[0] --------------------------------------------------------------------------------------------------------------------------------------------------------------------- While running the testcode I got the following results: (1 run = converting an integer to a byte string, converting that byte string back to an integer, and finally checking whether that last integer is the same as the input integer.) *** Starting Synchronous Benchmarks. *(No Twisted => "blocking" code)* -> Synchronous Benchmark (1 runs) Completed in 0.0 seconds. -> Synchronous Benchmark (10 runs) Completed in 0.0 seconds. -> Synchronous Benchmark (100 runs) Completed in 0.0 seconds. -> Synchronous Benchmark (1000 runs) Completed in 0.00399994850159 seconds. -> Synchronous Benchmark (10000 runs) Completed in 0.0369999408722 seconds. -> Synchronous Benchmark (100000 runs) Completed in 0.362999916077 seconds. *** Synchronous Benchmarks Completed in* 0.406000137329* seconds. *** Starting Asynchronous Benchmarks . *(Twisted => "non-blocking" code)* -> Asynchronous Benchmark (1 runs) Completed in 34.5090000629 seconds. -> Asynchronous Benchmark (10 runs) Completed in 34.5099999905 seconds. -> Asynchronous Benchmark (100 runs) Completed in 34.5130000114 seconds. -> Asynchronous Benchmark (1000 runs) Completed in 34.5859999657 seconds. -> Asynchronous Benchmark (10000 runs) Completed in 35.2829999924 seconds. -> Asynchronous Benchmark (100000 runs) Completed in 41.492000103 seconds. *** Asynchronous Benchmarks Completed in *42.1460001469* seconds. Am I really seeing factor 100x?? I really hope that I made a huge reasoning error here but I just can't find it. If my results are correct then I really need to go and check my entire cloud platform for the places where I decided to split functions into atomic operations while thinking that it would actually improve the performance while on the contrary it did the opposit. I personaly suspect that I lose my cpu-cycles to the reactor scheduling the deferred-callbacks. Would that assumption make any sense? The part where I need these conversion functions is in marshalling/protocol reading and writing throughout the cloud platform, which implies that these functions will be called constantly so I need them to be superfast. I always though I had to split the entire marshalling process into small atomic (deferred-callback) functions to be efficient, but these figures tell me otherwise. I really hope someone can help me out here. Thanks in advance, Best regards, Dirk Moors

3 2

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 10 Oct '09

10 Oct '09

1 0