Mailman 3 May 2012 - Twisted

[Twisted-Python] Synchronous calls using Twisted?
by Louis May 18, 2012

May 18, 2012

Hello, I am writing an application, which architecture is (I hope my beautiful drawing is not messed up): +-------------+ +------------+ +-----------+ | Core Server |---| Web Server |---| Web client| +-------------+ +------------+ +-----------+ That is, I have a core server (itself interacting with a database, but I think this is not important). I have a second piece of software, which is both a client regarding the core server, and a web server. At last, the web client is the web browser of your choice. The web server is a Django application, which means that (as far as I know) I cannot (easily) make it use asynchronous calls. At last, for the communication between the core server and the web server (which is, from this point of vue, a client), I was looking for something that let me use remote objects as if they were local. That is when I discovered Twisted. Twisted seems to be the kind of tool I want to use because it seems to be powerful, can use SSL, authentication, transmits exceptions, is meant to be used with Trial (I want to write tests), etc. The problem is that one of the strength of Twisted is its ability to deal with asynchronous tasks, which I am not interested in. Using blockingCallFromThread() [1], I managed to make a small package which looks (from my Twisted-ignorant point of vue) promising for my application, where the server is simply defined by (in a Twisted .tac file): # Server() is the object to be exported to the client application = server.get_application(Server(), "tcp:8800:interface=localhost") And the client can perform something like: remote = client.RemoteObject("localhost", 8800) # display() is a method executed on the server remote.display("Hello, world!") remote.disconnect() But then I noticed that I might have problem to test it using Trial (because an deadlock happens when blockingCallFromThread() and the reactor are waiting for each other in the same thread). I also had a look at inlineCallbacks [2], but although code looks synchronous inside the decorated function, the function itself still returns a deferred. I noticed quite a few other minor things. And I started to realize that Twisted is not meant to be used that way (and I read a bunch of threads of the Twisted mailing list with subjects containing "synchronous", including this long one [3]). My question is: What is your reaction? a) "Nice hack! Go on!" b) "Go on if you like, but you won't be able to use advanced Trial stuff (like timeouts), let alone Trial." c) "Go on if you wish, but you are missing the beauty of Twisted." d) "Stop it now. You have clearly misunderstood Twisted. You are wasting your time and ours." e) Any other... Do you think Twisted is the right tool for my use case, or am I doing an ugly hack to do what I want, which means I should rather use another tool? Sorry for the length of this post, and thanks in advance for your help. Louis [1] http://twistedmatrix.com/documents/current/api/twisted.internet.threads.htm… [2] http://twistedmatrix.com/documents/current/api/twisted.internet.defer.html#… [3] http://twistedmatrix.com/pipermail/twisted-python/2005-October/thread.html#…

3 3

Re: [Twisted-Python] Synchronous calls using Twisted?
by Andrew Francis May 17, 2012

May 17, 2012

Message: 1 Date: Thu, 17 May 2012 00:58:57 +0200 From: Louis <spalax(a)gresille.org> Subject: [Twisted-Python] Synchronous calls using Twisted? To: twisted-python(a)twistedmatrix.com Message-ID: <4FB43131.2030805(a)gresille.org> Content-Type: text/plain; charset=ISO-8859-1 > Hello, > I am writing an application, which architecture is (I hope my >beautiful drawing is not messed up): > +-------------+ +------------+ +-----------+ > | Core Server |---| Web Server |---| Web client| > +-------------+ +------------+ +-----------+ >... > Do you think Twisted is the right tool for my use case, or am I doing >an ugly hack to do what I want, which means I should rather use another >tool? I am not sure why inline generators won't work? Over the years, I use Stackless Python with Twisted. I use Twisted for many of the reasons you describe. For myself there is another dimension: I find that a lot of the code I write are orchestrations - that is in order to compute something, I have to make a few network calls sequentially. I use a technique that Christopher Armstrong (what happened to him?) called a blockOn. For the sake of being academic (roll your eyes here), this is an example of an obscure design pattern called "Half-sync/Half-async." Essentially one does the following: def blockOn(deferred): ch = stackless.channel() def cb(result): ch.send(result) deferred.addBoth(cb) return ch.receive() Of course, there is a bit more, like the Twisted reactor is running in its own tasklet. But this has the effect of nicely allowing one to use Twisted in a synchronous fashion. If you don't feel like installing Stackless Python, then you can use stackless.py (not the new version that uses continuations) with the greenlet package. Here is a link to a complete example http://andrewfr.wordpress.com/2011/11/30/the-santa-claus-problem-with-join-… Cheers, Andrew

3 2

[Twisted-Python] (no subject)
by exarkun＠twistedmatrix.com May 16, 2012

May 16, 2012

Hi all, A few us were discussing providing commercial support contracts for Twisted users, and it would be helpful to gauge potential interest. Support contracts might include things like: A certified version of Twisted for your platform Twisted bug fixes Twisted feature development Answers to questions about Twisted functionality Answers to questions about Twisted best practices and design patterns Answers to questions about performance and scaling In-depth support for your specific applications Additional documentation Training Code review If you or your organization might be interested in such a thing, we'd appreciate your answering a quick questionnaire: http://bit.ly/JxrIbu Thanks, Jean-Paul

2 2

[Twisted-Python] Declaring AMP responders away from
by Laurens Van Houtven May 16, 2012

May 16, 2012

Hi, I'm writing a service that exposes a bunch of methods over AMP. These methods are easily logically grouped into separate modules (i.e. message-related, picture-related…). So, I'd like to define the behavior for those methods in those modules. However, the amp.Command.responder decorator only works inside the class definition of an AMP subclass. (http://twistedmatrix.com/documents/current/api/twisted.protocols.amp.Comman…). Does that mean I should have multiple AMP subclasses? How do I get all of them to listen on the same port? Is this what the mysterious "routes" feature I've been hearing about does? Right now I'm doing it like this, which works but isn't very pretty or nice: user.py: ----- class Register(amp.Command): arguments = […] response = […] errors = […] def register(self, …): # do actual registering behavior here ----- api.py: ----- from project import auth class API(amp.AMP): register = auth.Register.responder(auth.register) ----- cheers lvh

2 1

[Twisted-Python] Cannot install Twisted version 8 through PIP
by Pascal Briet May 14, 2012

May 14, 2012

Hi, We're having some troubles today, because it seems that old packages of Twisted are not available anymore. We are currently relying on Twisted 8.2.0. pip install twisted==8.2.0 is failing (first version available seems to be 9.0.0) The package was still available last Friday. Is there a critical reason explaining that it has been removed from the servers ? Thanks very much, Best regards, -- Pascal BRIET Software engineer Optimor Labs http://www.billmonitor.com

2 4

[Twisted-Python] question : json.loads failed using LineReceiver data
by bino oetomo May 14, 2012

May 14, 2012

Dear All .. I'm trying to received json formated text using LineReceiver Here is my code ----START---- class ArduinoReceiver(LineReceiver): def __init__(self): print 'INIT : ArduinoReceiver' def lineReceived(self, line): line = line.strip() print 'Received line = '+line if 'Arduino' in line : return self.linedict=json.loads(line) ----STOP----- but I got error msg in the log as : ----START---- 2012-05-14 15:42:00+0700 [-] Received line = '{"a_flw_fwd":12,"a_flw_rtr":5029,"a_rpm":519,"b_flw_fwd":23,"b_flw_rtr":8,"b_rpm":913}' 2012-05-14 15:42:00+0700 [-] Unhandled Error Traceback (most recent call last): File "/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0_r33425-py2.6-linux-i686.egg/twisted/python/log.py", line 84, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0_r33425-py2.6-linux-i686.egg/twisted/python/log.py", line 69, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0_r33425-py2.6-linux-i686.egg/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0_r33425-py2.6-linux-i686.egg/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0_r33425-py2.6-linux-i686.egg/twisted/internet/posixbase.py", line 586, in _doReadOrWrite why = selectable.doRead() File "/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0_r33425-py2.6-linux-i686.egg/twisted/internet/_posixserialport.py", line 62, in doRead return fdesc.readFromFD(self.fileno(), self.protocol.dataReceived) File "/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0_r33425-py2.6-linux-i686.egg/twisted/internet/fdesc.py", line 94, in readFromFD callback(output) File "/usr/local/lib/python2.6/dist-packages/Twisted-11.1.0_r33425-py2.6-linux-i686.egg/twisted/protocols/basic.py", line 564, in dataReceived why = self.lineReceived(line) File "./kapal02.tac", line 48, in lineReceived self.linedict=json.loads(line) File "/usr/lib/python2.6/json/__init__.py", line 307, in loads return _default_decoder.decode(s) File "/usr/lib/python2.6/json/decoder.py", line 319, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.6/json/decoder.py", line 338, in raw_decode raise ValueError("No JSON object could be decoded") exceptions.ValueError: No JSON object could be decoded ----STOP----- I try to do json.loads based on received msg using python console, and it work : ---START--------- bino@bino-desktop:/var/log/rabbitmq$ python Python 2.6.6 (r266:84292, Sep 15 2010, 15:52:39) [GCC 4.4.5] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import json >>> aa='{"a_flw_fwd":12,"a_flw_rtr":5029,"a_rpm":519,"b_flw_fwd":23,"b_flw_rtr":8,"b_rpm":913}' >>> mydict=json.loads(aa) >>> print mydict['a_flw_fwd'] 12 ---STOP---------- Kindly please tell me why the 'json.loads' is failed in my tac file while it work in python console. Sincerely -bino-

3 2

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com May 12, 2012

May 12, 2012

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com May 5, 2012

May 5, 2012

1 0

[Twisted-Python] feedback on next version of Fireshark using twisted framework....
by Stephan May 4, 2012

May 4, 2012

Hi I have an open source project that is pretty well-known in the security community. http://www.fireshark.org it's a service wrapper that runs a headless version of firefox to visit malicious sites to store telemetry data. The old service used PERL for threading (very painful), so a while ago I changed everything to python using the twisted framework. the service's job is to launch a choose a firefox profile, launch the process and handle closes and crashes approprietly. the launch of version 2.5 is next week but I sure could use some feedback on the service code before releasing it. Keep in mind I haven't finishing commenting for public release. here are the files for the new service: 1) fireshark.py http://pastebin.mozilla.org/1614427 this is the main service. opens up a port to listen on, and uses the following two classes to manage profiles and launch a firefox process. 2) firesharkprofilemanager.py http://pastebin.mozilla.org/1614428 this is the firefox profile manager, it manages what firefox profiles are available. 3) firefoxprocess.py http://pastebin.mozilla.org/1614431 this is the firefox process class, to check if the launched process has been killed as well, as check if the logging from the internal plugin (another part of the fireshark project) has completed logging. Thanks in advance! Twisted has been a great Framework! Stephan

4 4

[Twisted-Python] (no subject)
by Dmitriy Komarov May 3, 2012

May 3, 2012

Hi All, Our team is setting up Twisted to access Mercurial over SSH. SSH connection, key exchange, and receiving large amount of data from the client work nicely, but sending server data back to the client ends up with a PuTTY Fatal Error "Incoming packet was garbled on decryption" at a random point. If plink compression is disabled, the error happens very rarely, but overall SSH performance becomes unacceptable. If plink uses compression, the error occurs within 1-10 seconds from starting data download. To make things more hectic, on a rare occasion, PuTTY reports "Incorrect MAC received on packet" instead of "garbled on decryption". If anyone had a similar issue before, I'd appreciate any pointers on resolving or avoiding it. Sincerely, - Dmitriy

5 7