Mailman 3 November 2008 - Twisted

Re: [Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones Nov. 21, 2008

Nov. 21, 2008

>>>>> "Drew" == Drew Smathers <drew.smathers(a)gmail.com> writes: Drew> On Fri, Nov 21, 2008 at 1:04 PM, Terry Jones <terry(a)jon.es> wrote: Drew> Why not just return a Deferred from the function and not decorate it. Drew> Or document the function as returning a value if it doesn't block. In that case (1) I'm considering what happens if you take the last yield out of a function decorated with inlineCallbacks. You might do it via commenting something out for testing, or you might do it just as a matter of course in changing the behavior of the function. If you do either of those, inlineCallbacks will raise an exception. With the suggestion I sent, it will just run as expected: the original function will be called, and you'll get the value it returns back as the result of the returned deferred. Drew> Essentially this is equivalent to _not_ decorating your function and Drew> returning a Deferred via defer.succeed(value) or defer.fail(error). Drew> I'm still not sure why it's necessary to stitch this kind of behavior Drew> into inlineCallbacks(). It's not necessary, it just makes inlineCallbacks gracefully cover two cases that it currently throws exceptions on, and it makes inlineCallbacks always return a deferred, which is arguably also an advantage. Drew> From my perspective inlineCallbacks() simply equates: "I want to do Drew> more than one asynchronous operation inline and here's some syntactic Drew> sugar using yield as an expression." Right. I'm just making what's happening underneath a little more forgiving and consistent in these two edge cases where things go awry. The main point, to me, is that code changes over time. So you write a function and use inlineCallbacks decorator for syntactic sugar. Then you, or someone else, comes along a while later and change a few things. If you take out the last yield without noticing, or your change throws an exception before the first yield, then you've broken things and it may not be clear why. In the first case, you get an exception thrown from deep inside inlineCallbacks, and many/most people would have some trouble figuring out what's going on. There's no need for that though. >> This has the advantage that (barring e.g., a KeyboardInterrupt in the >> middle of things) you'll *always* get a deferred back when you call an >> inlineCallbacks decorated function. That deferred might have already >> called or erred back (corresponding to cases 1 and 2 above). Drew> You will always get a Deferred back if the function is a generator. But if you take out the last deferred, it's no longer a generator. And if an exception happens before the first yield you don't get a deferred back because the exception is raised before the inlineCallbacks created function gets its first value from your generator. They're just two simple scenarios that can be dealt with more cleanly. If it wasn't clear, the inlineCallbacks created function is identical. Your code is still run identically. There's just a little pre-processing to see if it even needs to be run. Drew> That's what unit tests are for :) But I'm not sure how you could Drew> easily wind up in this scenario considering generators don't allow Drew> return with an argument. Right. But a function can just fall off the end, implicitly returning None. And I gave two examples, plus simple code, that illustrates how both of the problems I'm addressing can arise. It's not about having return in a function. One part is about turning a generator into a non-generator by removing the final yield. The other's just about introducing an exception into your code. I've run into both situations, more than once, and in both had the reaction that inlineCallbacks could have been more accomodating or helpful in how it behaves. Fortunately I can just use my own version! :-) Sorry if I wasn't being clear enough first time round. Regards, Terry

1 0

[Twisted-Python] Connectionless AMP?
by Peter Sabaini Nov. 21, 2008

Nov. 21, 2008

Hello, I want to implement a messageing protocol running on top of TIPC / SOCK_RDM to be able to do multicasting; ie. send messages in a connectionless manner. I implemented some TIPC support, but I now wonder how to go about the protocol. Originally, I thought about using AMP, but this seems to assume a connection-oriented transport. How would you go about this? Should it be possible to use a datagram-like, connection-less transport with AMP? Or should I rather take a more low-level approach, eg. by using IntNStringReceiver and roll my own? Thanks for any advice. peter.

4 8

[Twisted-Python] Connection Manager
by Reza Lotun Nov. 21, 2008

Nov. 21, 2008

Hi All, I've been thinking of implementing a connection manager in twisted. The app I'm working on has various parts or "modules" that individually make separate connections to various services and what not. They basically act independently of each other - as in they're all scheduled connection calls and what not. I'm interested in mediating these connections through a "Dispatcher" of sorts - what I'm thinking would be a thin wrapper over the reactor, with extra state. One of the things the Dispatcher would do is gauge network connection health - something like periodically attempting to ping/connect to well known internet sites. If the connection ever degrades or fails, the Dispatcher would change state. Fairly straightforward, conceptually at least. My questions are twofold - is the general idea sound - how is such a thing implemented elsewhere? Also, what would be the best strategy to implement the above, if the idea is sane? I was thinking something like this: my Dispatcher would have something like a connectTCP call which would in turn call the reactor.connectTCP with a special DispatcherFactory. As the returned IConnector calls startedConnecting on the DispatcherFactory, it would check internal state, figure out a policy and decide whether to continue with the connection and call the passed in factory's buildProctol, or call the IConnector stopConnecting to pause the connection attempt. If the situation improves later on, the IConnector.connect can be called to "resume" the connection attempt. Any feedback would be greatly appreciated. Thanks, Reza -- Reza Lotun Senior Software Engineer GetPeer Limited reza(a)getpeer.com

1 0

[Twisted-Python] Exception handling in t.e.adbapi
by Don Dwiggins Nov. 20, 2008

Nov. 20, 2008

I've been using adbapi (with pyodbc talking to MS SQL Server) successfully in a Windows service. I've discovered that occasionally, the network between the service and the database machine goes down and comes back up (either that or the database server itself cycles down and back up). When this happens, any pyodbc connections hanging around are corrupted, so that the next query gets an exception (class pyodbc.Error). Currently, the best thing I can do is to restart the service. I'm trying to rewrite the service to be able to catch, analyze, and respond to exceptions from pyodbc. In particular, in the case above, it's possible to reconnect and retry the query. If the network connection is still down, I'd like to log it, send a message to alert someone, or whatever. I've rewritten a ConnectionPool.runQuery call to use runInteraction to call a function (in a thread) that works with the cursor created in runInteraction to execute the query. The function wraps the cursor.execute(...) in a try-except. I can successfully catch the exception, and I try to recover by doing cursor.reconnect(), then cursor.reopen(), and finally cursor.execute(...) again. I'm finding that this fails with a "wrong connection for thread" exception. I haven't been able to figure out why that should happen; more importantly, I'm not sure that I'm going about this in the right way. Any insights or pointers to code that does this kind of thing would be appreciated. Thanks, -- Don Dwiggins Advanced Publishing Technology

2 2

[Twisted-Python] trial testing messages
by Gabriel Rossetti Nov. 20, 2008

Nov. 20, 2008

Hello, I am writing unit tests using trial. I am trying to make sure that the received messages are processed correctly and the output messages are correct. I tried using toXml(), but the XML attributes aren't necessaraly in the correct order, so a string comparison doesn't work. I had originally submitted a patch to allow domish.Element to be compared (ticket #3344), but my patch isn't liked. So my question is, how may I test this? Thank you, Gabriel

2 2

[Twisted-Python] Deferreds vs sys.getrecursionlimit()
by Brian Warner Nov. 18, 2008

Nov. 18, 2008

About once every six months, I wind up debugging a python stack overflow in my Deferred-using code. The symptom is usually a log message that ends with: File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 344, in _runCallbacks self.result = failure.Failure() File "/usr/lib/python2.5/site-packages/twisted/python/failure.py", line 265, in __init__ parentCs = reflect.allYourBase(self.type) File "/usr/lib/python2.5/site-packages/twisted/python/reflect.py", line 542, in allYourBase accumulateBases(classObj, l, baseClass) File "/usr/lib/python2.5/site-packages/twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) File "/usr/lib/python2.5/site-packages/twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) File "/usr/lib/python2.5/site-packages/twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) File "/usr/lib/python2.5/site-packages/twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) exceptions.RuntimeError: maximum recursion depth exceeded It's always something weird. This time, I took notes. I offer these hints to help future searchers find a starting point in their own debugging efforts. The executive summary: Certain patterns of using Deferreds work fine while testing, but will fail in mysterious ways later on as the application's workload grows larger. These notes are formatted for Trac, as they were originally written as a comment for http://allmydata.org/trac/tahoe/ticket/237 . Also, the "solutions" suggested require the eventual-send operator as provided by Foolscap, and until/unless reactor.eventually() makes it into Twisted proper, these solutions may not be convenient for projects that aren't already using Foolscap. cheers, -Brian == Problem One: long list of callbacks, all of them are ready == Each Deferred (we'll call the first one Deferred A) has a list of callback functions. Each time you do d.addCallback(), this list grows by one element. When Deferred A fires, the list is executed in a 'while' loop, in Deferred._runCallbacks. If the callbacks all return either a normal value or a Failure, then the list is completely consumed during the one call to _runCallbacks, and everything is fine. However, when a callback returns another Deferred B (chaining), the first Deferred A must wait for the second to finish. The code that does this looks like: {{{ if isinstance(self.result, Deferred): self.pause() self.result.addBoth(self._continue) break }}} The second Deferred B might have already been fired by this point, either because it was born ready (created with defer.succeed, or defer.maybeDeferred), or because whatever was being waited upon has already occurred. If this occurs, the subsequent callback in Deferred A's chain will fire (with B's result), but it will fire through a 6-frame recursive loop instead of firing on the next pass of the 'while' loop. As a result, each such ready-to-fire Deferred will add 6 stack frames. 166 such loops are enough to put more than 1000 frames on the stack, which will exceed Python's default sys.getrecursionlimit() . The 6-frame cycle is: {{{ File "twisted/internet/defer.py", line 214, in addBoth callbackKeywords=kw, errbackKeywords=kw) File "twisted/internet/defer.py", line 186, in addCallbacks self._runCallbacks() File "twisted/internet/defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File "twisted/internet/defer.py", line 289, in _continue self.unpause() File "twisted/internet/defer.py", line 285, in unpause self._runCallbacks() File "twisted/internet/defer.py", line 341, in _runCallbacks self.result.addBoth(self._continue) }}} The following sample code will cause this situation: {{{ import traceback from twisted.internet import defer def fire(res, which): #print "FIRE", which, "stack:", len(traceback.extract_stack()) #if which == 2: # traceback.print_stack() return defer.succeed(None) d = defer.Deferred() for i in range(170): d.addCallback(fire, i) d.callback("go") }}} The exception that this provokes is caught by the Deferred's Failure mechanisms, but then Twisted has an internal failure while trying to capture it. The actual Unhandled error in Deferred that gets put into the logs is: {{{ Unhandled error in Deferred: Traceback (most recent call last): File "twisted/internet/defer.py", line 285, in unpause self._runCallbacks() File "twisted/internet/defer.py", line 341, in _runCallbacks self.result.addBoth(self._continue) File "twisted/internet/defer.py", line 214, in addBoth callbackKeywords=kw, errbackKeywords=kw) File "twisted/internet/defer.py", line 186, in addCallbacks self._runCallbacks() --- <exception caught here> --- File "twisted/internet/defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File "twisted/internet/defer.py", line 289, in _continue self.unpause() File "twisted/internet/defer.py", line 285, in unpause self._runCallbacks() File "twisted/internet/defer.py", line 344, in _runCallbacks self.result = failure.Failure() File "twisted/python/failure.py", line 265, in __init__ parentCs = reflect.allYourBase(self.type) File "twisted/python/reflect.py", line 542, in allYourBase accumulateBases(classObj, l, baseClass) File "twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) File "twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) File "twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) File "twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) exceptions.RuntimeError: maximum recursion depth exceeded }}} This problem frequently shows up in code which returns a Deferred for generality (i.e. some day it might be async), but is using defer.succeed() or defer.maybeDeferred(some_immediate_call) in the meanwhile. == Problem Two: deep chain of callbacks, e.g. recursive delayed polling == The other kind of recursion-limit-violation failures that occurs with Deferreds involves a long chain that finally fires. The most common way to generate such a chain is with a recursive method that separates each call with a Deferred, such as a polling function that returns a Deferred: {{{ def wait_until_done(self, ignored=None): if self.done: return True else: d = Deferred() reactor.callLater(1.0, d.callback, None) d.addCallback(self.wait_until_done) return d }}} If this function must poll more than 331 times, the reactor tick which notices the expired timer and fires d.callback will see a recursion-depth-exceeded exception. The last Deferred fires, which triggers the _continue callback on the next-to-last Deferred, which allows it to fire, which triggers the {{{[-2]}}} Deferred, etc. This recursive cycle is of length 3 and has the following frames: {{{ File "twisted/internet/defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File "twisted/internet/defer.py", line 289, in _continue self.unpause() File "twisted/internet/defer.py", line 285, in unpause self._runCallbacks() }}} This one is trickier to find, because the root of the stack will be some internal reactor call rather than application code. In particular, the bottom of the stack will look like: {{{ File "/tmp/t.py", line 26, in <module> reactor.run() File "twisted/internet/base.py", line 1048, in run self.mainLoop() File "twisted/internet/base.py", line 1057, in mainLoop self.runUntilCurrent() File "twisted/internet/base.py", line 705, in runUntilCurrent call.func(*call.args, **call.kw) File "twisted/internet/defer.py", line 243, in callback self._startRunCallbacks(result) File "twisted/internet/defer.py", line 312, in _startRunCallbacks self._runCallbacks() File "twisted/internet/defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File "twisted/internet/defer.py", line 289, in _continue self.unpause() File "twisted/internet/defer.py", line 285, in unpause self._runCallbacks() }}} The other tricky thing about this failure is that the application code is sitting on the end of the stack: any callback that is attached to the Deferred that {{{wait_until_done}}} returns will run in a low-stack environment. As a result, recursion-depth-exceeded exceptions will be triggered by seemingly innocent application code. Note how the "DONE" number changes as you modify the self.count comparsion value in this example: {{{ #! /usr/bin/python import traceback from twisted.internet import reactor from twisted.internet.defer import Deferred class Poller: count = 0 def wait_until_done(self, ignored=None): self.count += 1 if self.count > 301: # 331 works, 332 fails. return True else: d = Deferred() reactor.callLater(0.0, d.callback, None) d.addCallback(self.wait_until_done) return d p = Poller() def done(res): #traceback.print_stack() print "DONE", len(traceback.extract_stack()) d = p.wait_until_done() d.addCallback(done) reactor.run() }}} When this fails, the traceback that shows up in the logs looks like: {{{ Unhandled error in Deferred: Traceback (most recent call last): File "twisted/internet/defer.py", line 285, in unpause self._runCallbacks() File "twisted/internet/defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File "twisted/internet/defer.py", line 289, in _continue self.unpause() File "twisted/internet/defer.py", line 285, in unpause self._runCallbacks() --- <exception caught here> --- File "twisted/internet/defer.py", line 328, in _runCallbacks self.result = callback(self.result, *args, **kw) File "twisted/internet/defer.py", line 289, in _continue self.unpause() File "twisted/internet/defer.py", line 285, in unpause self._runCallbacks() File "twisted/internet/defer.py", line 344, in _runCallbacks self.result = failure.Failure() File "twisted/python/failure.py", line 265, in __init__ parentCs = reflect.allYourBase(self.type) File "twisted/python/reflect.py", line 542, in allYourBase accumulateBases(classObj, l, baseClass) File "twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) File "twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) File "twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) File "twisted/python/reflect.py", line 550, in accumulateBases accumulateBases(base, l, baseClass) exceptions.RuntimeError: maximum recursion depth exceeded }}} == Combinations == Note that these two problems can interact. Each ready-to-fire callback attached to a single Deferred uses 6 stack frames, and each chained callback uses 3 stack frames. If X*6+Y*3 > 1000, the code will fail. == Solutions == For problem one, the requirement is that Deferreds never wind up with more than 166 callbacks that are ready to fire. In other words, there must be at least one not-ready-to-fire Deferred in each span of 166 callbacks. One way to accomplish this is to have every 100th call return {{{foolscap.eventual.fireEventually(result)}}} instead of {{{defer.succeed(result)}}}. Having every call do this works too, it just slows things down a bit. (note that the reactor must be running for fireEventually to work) {{{ def fire(res, which): return defer.fireEventually(None) d = defer.Deferred() for i in range(170): d.addCallback(fire, i) }}} For problem two, the requirement is that the depth of the tail-recursion chain not exceed 331 cycles, minus some room for the code you're eventually going to attach to the end. One way to accomplish this is to have every 300th call (or every single call, if you are willing to accept the slowdown) add an additional {{{fireEventually}}} to break up the stack. {{{ def wait_until_done(self, ignored=None): self.count += 1 if self.count > 301: # 331 works, 332 fails. return True else: d = Deferred() reactor.callLater(0.0, d.callback, None) d.addCallback(self.wait_until_done) d.addCallback(lambda res: fireEventually(res)) return d }}}

5 6

[Twisted-Python] Non-blocking UDP output?
by Peter Cai Nov. 18, 2008

Nov. 18, 2008

Hi all, Today I read this line in "Unix Networking Programming". It seems that UDP output might be blocking. We also said in Section 2.11 that there is no actual UDP socket send buffer. The kernel just copies the application data and moves it down the stack, prepending the UDP and IP headers. Therefore, an output operation on a blocking UDP socket (the default) will not block for the same reason as a TCP socket, but it is possible for output operations to block on some systems due to the buffering and flow control that happens within the networking code in the kernel. But as far as I know, when I write "transport.write" in twisted code, it actually called "sendto" on the socket without using "select". Doesn't this conflict with the "non-blocking" philosophy of twisted?

3 3

[Twisted-Python] Opening a fifo
by Paul Thomas Nov. 17, 2008

Nov. 17, 2008

Hi, I've run into a small problem where my twisted application needs to open a fifo to talk to another process. On Linux at least, the open blocks until the other end of the fifo is opened and this is blocking my reactor. Is there a straight-forward mechanism to asynchronously open the fifo? Or will I have to do something with threads? Thanks, pt.

2 5

[Twisted-Python] ANN: 8.2.0pre1
by Christopher Armstrong Nov. 17, 2008

Nov. 17, 2008

Hi everybody, Please check out the first pre-release of Twisted 8.2.0, the first release since April. The downloads are available at: http://twistedmatrix.com/users/radix/8.2.0pre1/ And the NEWS file is available at: http://twistedmatrix.com/users/radix/8.2.0pre1/NEWS.txt If there are no problems reported with this pre-release, it will be re-cut as Twisted 8.2.0. -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/ http://canonical.com/

1 1

[Twisted-Python] Re: [Twisted-commits] r25381 - oh my GOD a news file
by Jonathan Lange Nov. 16, 2008

Nov. 16, 2008

On Mon, Nov 17, 2008 at 8:03 AM, Christopher Armstrong <radix(a)wolfwood.twistedmatrix.com> wrote: > Author: radix > Date: Sun Nov 16 14:03:04 2008 > New Revision: 25381 > > Modified: > branches/releases/release-8.2.x-3540/NEWS > > Log: > oh my GOD a news file > > I didn't know we had one :) Would life be easier if we updated the NEWS file incrementally, on each trunk commit? jml

1 0

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Nov. 16, 2008

Nov. 16, 2008

1 0

[Twisted-Python] Released Ampoule - 0.1
by Valentino Volonghi Nov. 16, 2008

Nov. 16, 2008

Here we go with Ampoule 0.1, the first beta release after 5 alphas. Featuring this release is simply support for a timeout time to avoid having all processes in the pool stuck on some commands. Unfortunately today PyPI is a bit broken and I can't upload the release there yet, however it's available on launchpad: https://launchpad.net/ampoule/ampoule/0.1 Right now my launchpad issue tracker is empty, which means that I've now implemented all the features that I was originally looking for in a process pool. Now the ball is in your hands dear users! :). Give me features to implement and I'll be glad to work on them. For the bunch of you worried about maturity of the project: it's obviously very young but at adroll.com we are using it in production systems, specifically to drive a pool of gtkWebKit processes that take screenshots of web pages. In my view this is one of the worst cases for worker processes because web sites leak like hell, so far it's been working flawlessly for over a week. Maybe soon we'll release also that project as open source. So, have fun using it. Below a more detailed changeset. This release was sponsored by Super Happy Dev House ;P. Ampoule 0.1 (2008-11-15) ========================== Changes -------- - Fixed bug #276841: Add timeout for subprocess calls. The ProcessPool constructor now supports an additional timeout argument that specifies a pool level timeout, in seconds, before which every call ever made should return. Alternatively there is also a per-call mechanism. Currently this second per-call system is not available when using the remote pool service, so be sure to set the timeout command line parameter at an high enough level to satisfy all the possible commands. If a command doesn't require an answer the timeout doesn't apply, of course. The error returned when a call is timedout is 'error.ProcessTerminated'. On *nix systems the process is terminated with signal 9, on windows ampoule uses SIGTERM and returns error code 255. -- Valentino Volonghi aka Dialtone Now running MacOS X 10.5 Home Page: http://www.twisted.it http://www.adroll.com

1 0

Re: [Twisted-Python] Deferreds vs sys.getrecursionlimit()
by Terry Jones Nov. 15, 2008

Nov. 15, 2008

Hi Brian! Thanks for posting all the details digging into this. I've not run into those things, but you never know. Anyway I really appreciate you taking the time to spell it all out so clearly. Terry

1 0

[Twisted-Python] How to stop a twisted daemon application.
by Peter Cai Nov. 14, 2008

Nov. 14, 2008

Hi, After I have run "twistd -y xxx.py" How can I stop it? I've read through twistd manual and didn't find anything about this. I have to find the process and kill it. Any better approach?

3 2

[Twisted-Python] Released Ampoule - 0.0.5
by Valentino Volonghi Nov. 14, 2008

Nov. 14, 2008

Ampoule is a process pool implementation written on top of Twisted Matrix. Its name comes from the use of AMP as the default communication protocol between the pool and all its children. It's different from other alternative solutions because it provides an API very close to that of the Twisted ThreadPool. As an helper function it also provides a deferToAMPProcess function that creates the ProcessPool and submits jobs to it. There has been 2 released since the last announce so I'm going to report both of them here. Ampoule 0.0.5 (2008-11-12) ========================== Changes -------- - Fixed bug #259264, this fix introduces a number of changes in the architecture of the project: 1. Removed childReactor argument form the process pool and added a starter argument. 2. Introduced the concept of a starter object whose role is to start subprocesses with given parameters. This makes it easier to specify particular parameters (eg. new env variables) to the child processes without needing to override many methods in the pool using closures. 3. main.py is completely changed and now provides the ProcessStarter object which is a default implementation of IStarter. 4. IStarter interface currently documents only 2 methods: startAMPProcess startPythonProcess in the future it's possible that we will add an additional: startProcess that starts whichever process we want without requiring python, also this might end up with the separation of ProcessPool in at least 2 logical levels: the ProcessPool and a dispatcher that talks with the children, in this way it would be possible to create custom ProcessPools without changing much code or requiring any special requirement on the children. - Introduced a callRemote method on the ProcessPool that is basically the same as doWork. Introduced for symmetry between all the RPC libraries in Twisted. - reactor short name and ampoule child class are now passed as the 2 last arguments rather than the first 2. So if you have written any custom bootstrap code be sure to change sys.argv[1] and sys.argv[2] into sys.argv[-2] and sys.argv[-1] respectively. Ampoule 0.0.4 (2008-11-03) ========================== Changes -------- - Ampoule parent process and child processes now talk using FDs 3 (in) and 4 (out) in order to avoid problems with libraries that mistakenly send error lines to stdout (yes, I'm looking at you gtk+) -- Valentino Volonghi aka Dialtone Now running MacOS X 10.5 Home Page: http://www.twisted.it http://www.adroll.com

2 2

[Twisted-Python] ssh and python shell with twisted
by Turi Nov. 13, 2008

Nov. 13, 2008

Let me preface by saying, I am new to python and of course Twisted. I am trying to play with the examples already available http://www.devshed.com/c/a/Python/SSH-with-Twisted/. I believe that is the example from Oreilly book on Twisted. I looked at the custom ssh server script and then the providing python shell over ssh script http://www.devshed.com/c/a/Python/SSH-with-Twisted/3/. What I am curious about is how to combine these two? So I want to be able to set up the custom ssh server, which the first example explains, but have support for a command in the SSHDemoProtocol Class (like do_python_shell) which will invoke python shell over the existing ssh session. I am assuming you have to somehow do that over the existing channel, but how? Does my question make sense? --Turi

1 0

[Twisted-Python] Event notification and general RPC in Twisted...
by Blair Bethwaite Nov. 12, 2008

Nov. 12, 2008

Hi all, I'm thinking about using Twisted to rewrite some communications in a Grid computing application. E.g.: [JobServer]1 ---------- *[Proxy]1 ----------- *[Agents] Agents get jobs, report status, results, etc through Proxy to JobServer. Agents are often distributed across a private network that has no external interface, hence the proxy, which is run on a machine between the private network and internet. Also, the proxy may do some caching/queuing of particular messages - especially where the number of agents behind it is large. What we have at present is a TCP socket server however we're starting to hit scalability issues and on top of that our future plans necessitate a notification framework. I've been digging around on twistedmatrix.com and in the book but so far am having trouble finding any guidance or the level of detail that tells me whether Twisted has what we need, some bits of which are: - we'd like to use a persistent stream/connection, at least between the JobServer and Proxy (traffic frequency will be reasonably high) - it needs to be interoperable with java (is there PB for java?) - each end of the connection should be able to register for and get notifications from the other (e.g. Agent gives a heartbeat, JobServer tells Agent to stop) - sometimes the Proxy might be behind a firewall and only able to connect out, we need to be able to use that connection to go back as above - we want to dynamically configure streaming connections between Proxies Hopefully that's enough context for you to point at modules and tell whether these are appropriate questions: Can I re-use a TCP stream for multiple XML-RPC or PB operations? If so, would it make sense to have the client (e.g. Proxy or Agent) initiate the connection and then make a rpc (e.g. "notify me of anything relevant") to the server which would eventually return when something relevant came along - triggering a callback in the client... wash, rinse, repeat? Sorry if I'm overlooking something that already answers this, it doesn't seem like something too unique to have not been asked before... Regards, -Blair -- In science one tries to tell people, in such a way as to be understood by everyone, something that no one ever knew before. But in poetry, it's the exact opposite. - Paul Dirac

9 11

[Twisted-Python] client send twice but server reads once
by Aydın ŞEN Nov. 12, 2008

Nov. 12, 2008

Hi, In my code i'm sending sequential data from client like; self.transport.write('aydin') self.transport.write('adige') but server reads data like *'aydinadige' *How can i solve this, actually it is not clear how does dataReceived work for me best regards..

2 2

[Twisted-Python] checking connection
by Benny Nov. 12, 2008

Nov. 12, 2008

Hi, I have been trying to find out how to keep checking the connection once the connection is made. Says I have successfully connected to the server, and i want to maintain and check that connection. How can i do that?

3 5

[Twisted-Python] Import TypeError
by James Joplin Nov. 9, 2008

Nov. 9, 2008

I've been doing some embedded python work and everything has been working like a charm thusfar! Twisted has saved us tons of work, hats off to the development team. I have been having issues with shutting down twisted completely (I think). Even upon executing Py_Finalize or Py_EndInterpreter as necessary upon re-entering our browser plugin we hit an exception during twisted import. Here is my traceback : Traceback (most recent call last): File "<string>", line 1, in <module> File "ourCode.py", line 8, in <module> from twisted.spread import pb File "C:\Python25\lib\site-packages\twisted\spread\pb.py", line 69, in <module> from zope.interface import implements, Interface File "C:\Python25\lib\site-packages\zope\interface\__init__.py", line 58, in <module> _wire() File "C:\Python25\lib\site-packages\zope\interface\interface.py", line 809, in _wire classImplements(Attribute, IAttribute) File "C:\Python25\lib\site-packages\zope\interface\declarations.py", line 461, in classImplements spec = implementedBy(cls) File "C:\Python25\lib\site-packages\zope\interface\declarations.py", line 332, in implementedByFallback if isinstance(spec, Implements): TypeError: 'NoneType' object is not callable Changing the order of imports hasn't seemed to matter, looks to me like something in zope init blows up if it's already loaded. Is there a safe way to make sure the threads are dead? To clean imports out? Anything? I haven't been able to find much documentation to help me overcome the problem and I CANNOT forcibly unload our app from browser memory. If the C++ code we are using for startup / shutdown let me know and I will post it as well. Regards, James

2 2

[Twisted-Python] Running a client within a thread... do I need to rewrite?
by Andrew McClain Nov. 9, 2008

Nov. 9, 2008

I have an existing twisted client connecting to a jabber server in order to send notifications for our web app. I'd like to fold in a STOMP message client into the my existing code, so I can do something like this: web app --> STOMP queue --> STOMP client --> jabber client --> jabber server Problem is, the STOMP client library I'm using creates a socket, spawns a thread, and listens, calling a callback method when it gets information. In order to integrate this with my twisted jabber client, do I need to rewrite the whole client using deferreds? Or could I use something like defertothread to encapsulate the STOMP client? Best, Andrew -- -------- Tap the collective. http://www.fluther.com

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Nov. 9, 2008

Nov. 9, 2008

1 0

[Twisted-Python] freeing the reactor to do other jobs
by Jeff Dyke Nov. 7, 2008

Nov. 7, 2008

I'm using the XMLRPC server in twisted and a few methods call other, sometimes long running, functions/methods. I'm trying to get my brain around how to free the reactor to respond to other requests while this is happening. A scenario. A call is made to the server, which selects say 10K rows from a db and needs to check each row against a table and if they do not exist, insert them. """ Oversimplified version of the process """ def getData(self,user_id): rows = self.getUserData(user_id) for row in rows: if self.existsInQueue(row['some_id']): continue else: self.insertQueue(row) I want the caller to wait on a result from this process, but I also want the reactor to be able to handle other requests as they come in. This function is not directly registered in the xmlrpc server with xmlrpc_getData, but is called by that type of method after validation that it is allowed to run in this context. What i've seen when this has thousands of rows to process is that the reactor is tied up and can not respond to requests until complete. Which obviously leads to me believe that I'm not using twisted correctly/to its potential. I have read the deferred/asynchronous doc pages...but am having a hard time getting my head around it and would appreciate any advice. When i don't specifically need the caller to get the final result, i've been suing deferToThread, but feel in some of those instances i could possibly write better code, rather then sending to a thread. Thanks jd

5 6

[Twisted-Python] making twisted provide good feedback (was Re: Having reactor run at the main thread...)
by glyph＠divmod.com Nov. 7, 2008

Nov. 7, 2008

(Moving this off the twisted-web list because it is of more general interest and contains nothing web-specific.) On 09:48 am, jml(a)mumak.net wrote: >On the other hand, there's a related invalid argument that gets used a >lot by library and framework authors: "if we provide X, people might >misuse it, so we should not provide X". This argument is also bogus >(that way lies Java). Provide safe, well-documented alternatives for >the abusers and let those who want to shoot themselves in the foot do >so: sometimes they might actually know better than you. I find it peculiar that you think this is Java's disease. My perspective is that Java is unpleasant because it is remarkably tedious, redundant, and inconsistent, not because it lacks functionality. If anything, it's a mess of functionality, and there are too many ways to do the same thing. If you want to shoot yourself in the foot (or face, as the case may be) there's always the com.sun hierarchy, which despite being full of warnings signs and sparsely documented, is all public (at the language level) and accessible from Java programs which wish to go that way. In that sense it's relatively hacker friendly. For more fun with foot- shooting there's also AccessibleObject, which lets you call private methods and access private fields. I think the whole issue here is very complex, and so I'd like to repeat and perhaps more clearly explain my own views. I work at the bleeding edge of lots of software and am frequently frustrated when the authors of libraries that I use don't bother with "edge" cases that are necessary to the applications that I write. Twisted should provide excellent support for valid use-cases, no matter how obscure. On the other hand, it's equally frustrating to use a library where the incredibly obscure "you probably don't need this" and the 99%-of-the- time positive-path APIs are listed, without any particular ordering, in one giant flat list. It's particularly frustrating to google for "How do I (X)" and get a solution which uses an apparently reasonable API to do (X), only to discover months after using it that it's actually unsupported and you're not supposed to touch it and now your program is going to break horribly with the next minor version of the library. This isn't just a problem with documentation, because in reality nobody reads documentation. If you're writing a program, you're going to read _just_ enough of the API doc and do just enough groveling around with dir() and pydoc to get yourself to the point where something works. When it works, you will probably consider that sufficient proof that you used the APIs correctly, unless you had to type something that _looked_ particularly gross in the course of doing it, or you got a bunch of warnings out of your program when you ran it. This is the whole point of DeprecationWarning: somehow, your library code itself has to communicate with the the developer who is invoking it and tell them useful things about their use or misuse. I feel like Twisted has more of the latter problem than the former. There are a lot of methods which aren't really private (addReader/removeReader) that don't differentiate themselves as more or less private or refer you to the more portable and general APIs that you probably want (pauseProducing/resumeProducing). However, (modulo global reactors, of course) Twisted does support a lot of edge cases very nicely. Speaking from personal experience, there are a lot of people who show up in #twisted looking at <http://twistedmatrix.com/documents/8.1.0/api/twisted.internet._sslverify.ht…> when they should be looking at <http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.ssl.html>, or <http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.iocpreactor.t…> when they should be looking at <http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.interfaces.IR…>. In the long term, we need to write both better documentation and more code that gets these people back onto the right track. I know I've used a few libraries which provided exactly this kind of feedback and it was very satisfying and educational. It's really an art, one that's difficult to master, and I don't know that I really know the right way to do it in the general case. The danger that comes along with doing it wrong is very nicely explained here: http://blogs.msdn.com/oldnewthing/archive/2008/10/06/8969399.aspx - if you make an API which doesn't work except for the "special" cases, then everyone just starts using the "special" version.

2 1

[Twisted-Python] Silencing PotentialZombieWarning
by Reza Lotun Nov. 6, 2008

Nov. 6, 2008

Hi All, I'm working on a multi-platform (win32 and os x) wx + Twisted app, where I've found the best solution is to run both in a separate thread - the main thread is wx, and the other is Twisted. It basically works like a charm (though I'd be interested in hearing about other approaches. I've also played with wxreactor way back, but didn't find it as stable). Because I spawn a few processes (using spawnProcess), and must run my reactor with installSignalHandlers=0, I get PotentialZombieWarning errors every time I spawn. Now, I handle reaping of my processes manually for various reasons, and also install my own signal handlers (in the main thread; I also pass clearSigInt=False to wx). Is there any way I can silence these warnings? Thanks, Reza -- Reza Lotun Senior Software Engineer GetPeer Limited reza(a)getpeer.com

3 4