Mailman 3 October 2008 - Twisted

[Twisted-Python] sending signal through SSHChannel
by Frans Mårtensson Oct. 24, 2008

Oct. 24, 2008

Hello! I have written a python program that executes shell scripts on remote computers using the twisted conch SSHChannel. Several shell scripts are executed at once on several machines and the scripts range in execution time from minutes to hours. My question is regarding the behavior of Conch when it is shut down. What i would like to do is to send a SIGINT or SIGKILL to the remote script so that it can terminate in an orderly fashion. I am using the channel.SSHChannel as a base class for … [View More]

2 1

[Twisted-Python] Consensus on speed of deferred call/errback-style execution?
by Terry Jones Oct. 24, 2008

Oct. 24, 2008

Last year someone mentioned to me that using defer.inlineCallbacks (versus writing a bunch of independent standalone callback functions) incurs some sort of (significant) speed penalty during execution. How accurate is that? More generally, there are many styles one can use to write and add callbacks to a deferred. Is there any consensus on the "best" way to do it, or are there rules of thumb that people have figured out, etc? I sometimes use defer.inlineCallbacks and then have a bunch of … [View More]

3 2

[Twisted-Python] SOAP Client?
by Don Smith Oct. 24, 2008

Oct. 24, 2008

Is there a SOAP client available in Twisted, or one that would be recommended to use with Twisted? Thanks, Don Smith

3 2

[Twisted-Python] Deferred in C++
by Jamu Kakar Oct. 22, 2008

Oct. 22, 2008

Hi, I've implemented Deferred in C++. There is room for improvement and optimization, but the basic core works. My focus has been on trying to make the best possible API experience for users. The internal implementation is confusing and full of complexity unrelated to the problems being solved. For example, because the implementation relies on templates so heavily, order-of-definition is an issue. Luckily most of the complexity stays hidden away leaving a mostly easy-to-use API, even in … [View More]tricky cases such as callback pipelines that change the result type being passed from one callback to another by the Deferred: int convert_string(std::string result) { std::cout << "Converting string " << result << std::endl; return ion::Convert::to_integer<int>(result); } int add_ten(int result) { std::cout << "Adding 10 to " << result << " and returning " << result + 10 << std::endl; return result + 10; } Deferred deferred = Deferred::create_deferred<std::string>(); deferred.add_callback(&convert_string); deferred.add_callback(&add_ten); std::cout << "Calling succeed" << std::endl; deferred.succeed(std::string("13")); std::cout << "Callbacks finished" << std::endl; Running this code will produce the following output: Calling succeed Converting string 13 Adding ten to 13 and returning 23 Callbacks finished It's possible to use method pointers as callbacks, too: class CallbackHandler { public: int handle_result(int result) { std::cout << "Handling " << result << std::endl; return result; } }; CallbackHandler handler; deferred.add_callback(&handler, &CallbackHandler::handle_result); One of the constraints that the API mostly hides is that the return and parameter types of every callback in the pipeline need to be known at compile time. add_callbacks, add_callback and add_errback are template methods that provide implementations for different combinations of function and method pointers. They infer return and parameter types, based on the inputs passed in, and use them to create internal structures. Inferring types breaks down when adding a callback that returns a Deferred because there's no way to infer the result type of a Deferred that may not exist yet. In such cases, the result type of the Deferred must be explicitly stated: class DeferredCallback { public: DeferredCallback() : deferred(Deferred::create_deferred<int>) {} Deferred get_deferred(int) { std::cout << "Returning deferred" << std::endl; return this->deferred; } Deferred deferred; }; Deferred deferred = Deferred::create_deferred<std::string>(); DeferredCallback callback; // Notice the explicit result type here: deferred.add_callback<int>(&callback, DeferredCallback::get_deferred); deferred.add_callback(&add_ten); std::cout << "Calling succeed" << std::endl; deferred.succeed(std::string("13")); std::cerr << "Calling deferred" << std::endl; callback.deferred.succeed(6) std::cout << "Callbacks finished" << std::endl; Running this code will produce the following output: Calling succeed Returning deferred Calling deferred Adding ten to 6 and returning 16 Callbacks finished When a callback returns a Deferred, processing of the pipeline pauses until a result comes available, which is why add_ten is only called after the inner Deferred succeeds. If you forget the explicit result type when adding a Deferred returning callback you'll get a compile-time error: ../../ion/defer.h: In member function 'void ion::Deferred::add_callbacks(Instance*, ion::Deferred (Instance::*)(Param), ion::Deferred (Instance::*)(ion::trule<ion::Failure, ion::DeallocObject>)) [with Instance = ion::tests::DeferredMethodHandler, Param = int32_t]': test-defer.cpp:416: instantiated from here ../../ion/defer.h:773: error: no matching function for call to 'ion::CompileTimeAssertion<false>::CompileTimeAssertion(ion::Deferred::add_callbacks(Instance*, ion::Deferred (Instance::*)(Param), ion::Deferred (Instance::*)(ion::trule<ion::Failure, ion::DeallocObject>)) [with Instance = ion::tests::DeferredMethodHandler, Param = int32_t]::STATIC_ASSERT_Deferred_result_type_must_be_specified&)' ../../ion/debug.h:164: note: candidates are: ion::CompileTimeAssertion<false>::CompileTimeAssertion() ../../ion/debug.h:164: note: ion::CompileTimeAssertion<false>::CompileTimeAssertion(const ion::CompileTimeAssertion<false>&) ../../ion/defer.h: At global scope: ../../ion/defer.h:772: warning: unused parameter 'instance' ../../ion/defer.h:772: warning: unused parameter 'method_callback' ../../ion/defer.h:772: warning: unused parameter 'method_errback' If you look *really* hard you'll see some helpful information starting with STATIC_ASSERT. Yes, that's a feature. I've implemented a Failure object, but failure handling over all is not very good. Exceptions are tricky because, unless you're expecting a particular kind of exception, their types become opaque as soon as they're thrown. There's room for improvement but there's no obvious path to a good solution yet. The behaviour of errbacks is slightly different than in the Python implementation. Errbacks receive a trule<Failure> (a scoped pointer that the errback owns) and return a result suitable for the next callback. They can either (a) recover from the error and return a result, (b) raise an exception to indicate a new error or (c) raise an UnhandledFailureError exception with the failure object. Callback processing continues after (a), while errback processing continues after (b) or (c). A new Failure object is created when (b) occurs. int recover(trule<Failure>) { std::cout << "Recovering from failure" << std::endl; return 43; } Deferred deferred = Deferred::create_deferred<int>(); deferred.add_errback(&recover); deferred.add_callback(&add_ten); std::cout << "Calling fail" << std::endl; deferred.fail(Exception("Error message!")); std::cout << "Callbacks finished" << std::endl; Running this code will produce the following output: Calling fail Recovering from failure Adding ten to 43 and returning 53 Callbacks finished I'm not sure how practical this component would be in a non-trivial application. The reliance on templates would negatively affect compile time, for one thing. More importantly, I've found deciphering the error messages produced by simple mistakes can be tricky. Nonetheless, my main motivation was to see if it was possible at all and so I'm pleased that it works in the end. In the future, I'd like to see what the performance difference is between this version and the Python version, but I haven't investigated that at all. There are also memory-management related features that could be added to ease using this in C++ that would be interesting to explore. Porting to Windows is another item on the list of things that I'd like to do. The code is part of a project called ion (I/O and Networking library), which I use for experiments like this. I haven't merged the deferred branch yet, but will soon. Until it's merged, it's on Launchpad: https://code.edge.launchpad.net/~jkakar/ion/deferred It will most likely only build on Ubuntu Intrepid right now. Thanks, J. [View Less]

3 5

[Twisted-Python] Available for short-term (non-typing) consulting
by Itamar Shtull-Trauring Oct. 21, 2008

Oct. 21, 2008

Need help with Twisted, distributed systems or or just plain old software architecture? Due to a repetitive strain injury I am unable to type, and so I'm taking some time off of work to recover (I'm writing this e-mail using voice dictation, thanks to Dragon NaturallySpeaking). As a result I am available for short-term consulting, training or design work for the next month, assuming of course it does not involve any typing. I've worked on Twisted itself, as well as projects ranging from … [View More]

1 0

[Twisted-Python] October 26 Sprint Reminder
by Jean-Paul Calderone Oct. 19, 2008

Oct. 19, 2008

There will be a Twisted sprint on Sunday, October 26 at the Divmod office in Porter Squarel, Cambridge, MA. The sprint will run from 10 AM until people get sick of it. Lunch will be provided, thanks to the generous contributions of the sponsors listed on the website. The sprint will focus on anything attendees like, but there will probably be at least some emphasis on tasks related to release automation. Jean-Paul

1 0

[Twisted-Python] Serious performance problem of twisted
by davy zhang Oct. 19, 2008

Oct. 19, 2008

a very simple server and multi-process code, I found there's about 500 milliseconds delay for every msg I sent to client I just don't why, did I made something wrong ? the code below using a process to handle the data and in the shared taskList, after done put the data to the shared msgList, a thread in main process to read the msg and send to the clients. the msg delay is significant, you can even see it with your eyes in telnet client I checked the time in the client, it's approximate to 0.… [View More]5 secs first I tested in java using the same logic with MINA, BlockingQueue stuff, the msg traveled from client to server and back is 0, so I pretty much sure the problem here is in the server side then I printed time in the server code, I found everything is as fast as it suppose to be, until the c.transport.write. After the write operation the client need 0.5secs to get the msg the code is below #!/usr/bin/env python #coding=utf-8 from twisted.internet.protocol import Factory,Protocol from twisted.internet import reactor from twisted.internet import error import cPickle import time from processing import Manager,Process import thread from time import time """flash policy file request""" policyRequest = "<policy-file-request/>" + chr(0) policyReturn = """<cross-domain-policy> <allow-access-from domain="*" to-ports="*" /> </cross-domain-policy>""" + chr(0) def handler(taskList,msgList): while 1: item = cPickle.loads(taskList.get()) print time() print 'item before handle ', item item['msg'] += ' hanlded done \r\n' msgList.put(cPickle.dumps(item)) def sender(): global msgList while 1: item = cPickle.loads(msgList.get()) print time() c = clients[item['cid']] c.transport.write(item['msg'] ) class DTProtocol(Protocol): global clients def connectionMade(self): clients.append(self) print "a new customer!" def dataReceived(self, data): global clients global taskList print 'msg received ',data if data == policyRequest: self.transport.write(policyReturn) self.transport.loseConnection() else: d = {'cid':clients.index(self),'msg':data,'done':False,'sent':False} taskList.put(cPickle.dumps(d)) print time() #self.transport.write(data) def connectionLost(self, reason): print reason if self in clients: clients.remove(self) print "bye!" def message(self,msg): self.transport.write(msg) class DTFactory(Factory): protocol = DTProtocol if __name__ == '__main__': port = 80 reactor.listenTCP(80,DTFactory()) clients = [] manager = Manager() taskList = manager.Queue() msgList = manager.Queue() h = Process(target=handler,args=(taskList,msgList)) h.start() thread.start_new_thread(sender,()) print 'server is runing at ',port reactor.run() [View Less]

3 3

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Oct. 19, 2008

Oct. 19, 2008

1 0

[Twisted-Python] problem with Twisted and Processing
by davy zhang Oct. 17, 2008

Oct. 17, 2008

I used Processing package for python 2.5 which became an official package of python 2.6 named multipleprocess. 'Cause the twisted for python 2.6 is not ready, I'm still using python 2.5 on windows Here's the problem the code below is pretty simple and works fine until I tried to get access of the first element in the list, I can even print the whole list, but i just can not get the elements inside. I guess there's something get conflicted in these two packages, anyone knows why? To reproduce … [View More]

3 4

[Twisted-Python] waiting for results from tcp
by Nima Ghanavatian Oct. 16, 2008

Oct. 16, 2008

Hi, I have a server which can handle multiple connections. At any given time, I want to send out a ping to see who is still listening to this server. For this I scan through my list of ip's and send out a msg with factory = ClientFactory() factory.protocol = caller factory.message = self.message factory.alive = 0 reactor.connectTCP(self.ip, self.port, factory) and wait on the response. This is done from within an application daemon, so no reactor.run() is required.… [View More]

2 1