Mailman 3 October 2008 - Twisted

[Twisted-Python] Status of twisted-pair and python-eunuchs.
by Chris Dew 16 Oct '08

16 Oct '08

I'm trying some network programming using tun interfaces to tunnel WAN traffic through user-space programmes at each end, which will perform compression and other magic on tcp streams. The userspace programmes then communicate across the WAN, using less bandwidth than the uncompressed data would have. Looking at the documentation for twisted-pair and python-eunuchs it looks as if they haven't been maintained for years - eunuchs last changed about 5-6 years ago. Is it going to make my life easier to mock things up in Twisted, to get things algorithmically correct, before porting partly/entirely to C for performance (if necessary)? (I'm looking at supporting >10Mbit/s on Core2 processors, with >100 simultaneous connections.) If Twisted would cause me to spend a large amount of time wrestling with Twisted-pair/Eunuchs problems, I'm probably better of starting with C. (I'm familiar with both Twisted and C, but not the tun interface, nor the innerworkings of the tcp protocol yet - I was hoping that the source of twisted-pair might help educate me.) Has anyone used these packages recently (2.6 kernel)? Thanks, Chris. P.S. Can anyone recommend any good articles on using the tun interface and interpreting/processing tcp packets?

2 1

[Twisted-Python] A deferred class that keeps and can log its call/errback history
by Terry Jones 16 Oct '08

16 Oct '08

A few days ago I thought it might be nice to augment the Deferred class to allow you to look at the callback/errback chain to see what's going on. Tonight I went to do it. I was nearly done in about 15 minutes, and then it took another 4 hours to finish.... :-) It's still a bit rough, but it seems to work. The code comes with a history logging function that you can insert into a callback chain. When it's run, it prints the history of the chain, including also printing the history of any deferreds that were returned along the way, and then prints the call/errback functions yet to be called. You get to see the initial result passed to the first call/errback, and the results along the way. Output is still grungy. Here's what you get from the attached test code: History for deferred <TDeferred at 0xbf1e60 current result: 45> Initial result = True --- ALREADY FIRED --- CALLBACK: func <history> result = True ERRBACK: <same as CALLBACK> CALLBACK: func = <function d at 0xbf46e0> args = '()' kwargs = '{}' result = <TDeferred at 0xbf6290 current result: 45> ERRBACK: func <pass> History for deferred <TDeferred at 0xbf6290 current result: 45> Initial result = 33 --- ALREADY FIRED --- CALLBACK: func = <function d2 at 0xbf4758> args = '()' kwargs = '{}' result = 45 ERRBACK: func <pass> --- STILL TO BE FIRED --- CALLBACK: func = <function ok at 0xbf41b8> args = "('hello',)" kwargs = "{'color': 'black'}" ERRBACK: func <pass> CALLBACK: func <pass> ERRBACK: func = <function nok at 0xbf47d0> args = '()' kwargs = '{}' CALLBACK: func = <function ok2 at 0xbf4668> args = '()' kwargs = '{}' ERRBACK: func <pass> CALLBACK: func <history> ERRBACK: <same as CALLBACK> CALLBACK: func = <function stop at 0xbf4848> args = '()' kwargs = '{}' ERRBACK: <same as CALLBACK> If you examine this you'll see that I'm using <pass> and <history> to indicate uninteresting call/errbacks (these correspond to the defer.passthru and the history function itself). When the errback is the same as the callback (due to d.addBoth), that's indicated. If you look at the attached code you'll see I also do a bit of hiding of housekeeping callbacks: the _continue and my own (perhaps unnecessary) _historyPop. Those are things the caller isn't aware of, and they're just clutter, so I don't print them. I made the logDeferredHistory function be standalone (not a class method) to make it easier to log any deferred (not just the current instance) and also because it's then simpler to identify when the callback is the history function itself (to be able to print that concisely). You use the logging callback pretty much like any other. E.g.: if __name__ == '__main__': log.startLogging(sys.stdout) x = funcReturningADeferred() x.addCallback(d) x.addBoth(logDeferredHistory, x) x.addCallback(ok, 'hello', color='black') x.addBoth(logDeferredHistory, x) Right now it logs the history. But you could pass it a file descriptor or something else, like a list. It would be nice to be able to make this a trivial drop-in replacement for normal Deferreds for when you're debugging. In that case logDeferredHistory could test to see if its deferred arg is an instance of a history deferred and if not just act as a passthrough. Then people might be able to do something like this when debugging: from twisted..... import historyDeffered as defer and otherwise not alter their code. Or we could add an optional argument to Deferred.__init__ indicating that the deferred should maintain its history. It doesn't involve much extra work - mainly just pushing things onto a list during _runCallbacks. Terry from twisted.internet import defer from twisted.python import failure, log def succeed(result): d = TDeferred() d.callback(result) return d def fail(result=defer._nothing): if result is defer._nothing: result = failure.Failure() d = TDeferred() d.errback(result) return d def _historyPop(result): return result def _continue(result, d): d.result = result d.unpause() class TDeferred(defer.Deferred): def __init__(self): defer.Deferred.__init__(self) self.history = [] def _startRunCallbacks(self, result): self.initialResult = result defer.Deferred._startRunCallbacks(self, result) def _runCallbacks(self): if not self.paused: cb = self.cb = self.callbacks self.callbacks = [] while cb: item = cb.pop(0) isFailure = isinstance(self.result, failure.Failure) callback, args, kw = item[isFailure] args = args or () kw = kw or {} try: self.result = callback(self.result, *args, **kw) if isinstance(self.result, TDeferred): self.callbacks = cb # note: this will cause _runCallbacks to be called # "recursively" sometimes... this shouldn't cause any # problems, since all the state has been set back to # the way it's supposed to be, but it is useful to know # in case something goes wrong. deferreds really ought # not to return themselves from their callbacks. self.pause() self.result.addBoth(_historyPop) self.result.addBoth(_continue, self) break except: self.result = failure.Failure() finally: self.history.append(list(item) + [isFailure, self.result]) if isinstance(self.result, failure.Failure): self.result.cleanFailure() if self._debugInfo is None: self._debugInfo = defer.DebugInfo() self._debugInfo.failResult = self.result else: if self._debugInfo is not None: self._debugInfo.failResult = None def _addHistoryItem(isCallback, f, args, kwargs, indent): out = [indent + ('CALLBACK:' if isCallback else 'ERRBACK:')] if f is defer.passthru: out.append('%s func <pass>' % indent) elif f is logDeferredHistory: out.append('%s func <history>' % indent) else: out.extend([ '%s func = %r' % (indent, f), '%s args = %r' % (indent, str(args)), '%s kwargs = %r' % (indent, str(kwargs)), ]) return out def logDeferredHistory(result, d, indent=''): out = ['', '%sHistory for deferred %r' % (indent, d)] append, extend = out.append, out.extend if hasattr(d, 'initialResult'): append('%sInitial result = %r' % (indent, d.initialResult)) if d.history: append('%s--- ALREADY FIRED ---' % indent) for item in d.history: (cb, cba, cbk), (eb, eba, ebk), wasFailure, fResult = item if cb is _historyPop: return out out += _addHistoryItem(True, cb, cba, cbk, indent) if not wasFailure: append('%s result = %r' % (indent, fResult)) if (eb, eba, ebk) == (cb, cba, cbk): append('%sERRBACK: <same as CALLBACK>' % indent) else: out += _addHistoryItem(False, eb, eba, ebk, indent) if wasFailure: append('%s result = %r' % (indent, fResult)) if isinstance(fResult, TDeferred): out += logDeferredHistory(None, fResult, indent + '\t') if d.cb: needHeader = True for item in d.cb: (cb, cba, cbk), (eb, eba, ebk) = item if cb is _continue or cb is _historyPop: continue if needHeader: append('%s--- STILL TO BE FIRED ---' % indent) needHeader = False out += _addHistoryItem(True, cb, cba, cbk, indent) if (eb, eba, ebk) == (cb, cba, cbk): append('%sERRBACK: <same as CALLBACK>' % indent) else: out += _addHistoryItem(False, eb, eba, ebk, indent) log.msg('\n'.join(out)) return result #!/usr/bin/env python import sys from twisted.internet import reactor from twisted.python import log, failure from tdefer import TDeferred, logDeferredHistory, succeed def gen(x): D = TDeferred() if x: reactor.callLater(2, D.callback, x) else: reactor.callLater(2, D.errback, Exception("one failed!")) return D def ok(result, msg, color='red'): print "ok called with %r" % result raise Exception("ok raises") def ok2(result): print "ok2 called with %r" % result def d(result): print "d called" D = TDeferred() D.addCallback(d2) # Uncomment the next line to log the history of the deferred D # in the middle of running (and printing) the deferred chain # started by our original call to gen. # D.addCallback(logDeferredHistory, D) reactor.callLater(2, D.callback, 33) return D def d2(result): return 45 def nok(failure): print "nok called" return d(1) def stop(x): print 'stop received %r' % x if isinstance(x, failure.Failure): print x.getTraceback() reactor.callLater(0, reactor.stop) if __name__ == '__main__': log.startLogging(sys.stdout) x = gen(True) log.err('orig deferred is %r' % x) x.addBoth(logDeferredHistory, x) x.addCallback(d) x.addBoth(logDeferredHistory, x) x.addCallback(ok, 'hello', color='black') x.addErrback(nok) x.addCallback(ok2) x.addBoth(logDeferredHistory, x) x.addBoth(stop) reactor.run()

1 0

[Twisted-Python] mystery exception
by Vadim Zaliva 15 Oct '08

15 Oct '08

I am writing network proxy, based on proxy.py code. I am also using getPage() and deffered. From time to time I am getting this mystery exception, the source of which I could not track. Does anybody knows what could be causing it? I am pretty sure I am not closing any sokets twice in my code. The code writing 'x' seems weird to me. Traceback (most recent call last): File "/usr/lib64/python2.5/threading.py", line 460, in __bootstrap self.run() File "/usr/lib64/python2.5/threading.py", line 440, in run self.__target(*self.__args, **self.__kwargs) --- <exception caught here> --- File "/usr/lib/python2.5/site-packages/twisted/python/threadpool.py", line 161, in _worker context.call(ctx, function, *args, **kwargs) File "/usr/lib/python2.5/site-packages/twisted/python/context.py", li ne 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw ) File "/usr/lib/python2.5/site-packages/twisted/python/context.py", li ne 37, in callWithContext return func(*args,**kw) File "/usr/lib/python2.5/site-packages/twisted/internet/threads.py", line 29, in _putResultInDeferred reactor.callFromThread(deferred.callback, result) File "/usr/lib/python2.5/site-packages/twisted/internet/base.py", line 824, in callFromThread self.wakeUp() File "/usr/lib/python2.5/site-packages/twisted/internet/base.py", line 443, in wakeUp self.waker.wakeUp() File "/usr/lib/python2.5/site-packages/twisted/internet/posixbase.py", line 138, in wakeUp util.untilConcludes(os.write, self.o, 'x') File "/usr/lib/python2.5/site-packages/twisted/python/util.py", line 757, in untilConcludes return f(*a, **kw) exceptions.OSError: [Errno 32] Broken pipe My OS is: Linux xxxx 2.6.23.1-42.fc8 #1 SMP Tue Oct 30 13:18:33 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux and I am using latest version of Twisted. Sincerely, Vadim

3 4

[Twisted-Python] How to make twistd print to stdout
by Alec Matusis 15 Oct '08

15 Oct '08

I have an application that is start from a command line by a shell script that has this: /usr/local/bin/twistd --no_save --reactor=epoll --pidfile=logs/serv.pid --logfile=logs/serv.log --python=serv.py serv.py: application = twisted.application.service.Application('serv',uid=config.server_uid,gid=con fig.server_gid) my_service = service.MyService() my_service.setServiceParent(application) . In MyService class I have: def startService(self): if bad_config_file: #I want to print to stdout here end quit immediately?? twisted.internet.reactor.callLater(0, twisted.internet.reactor.stop) I cannot force twistd to print to stdout after that comment line, probably because it's daemonized by then. print 'bad error' writes it to --logfile=logs/serv.log I tried sys.stdout/stderr.write('bad error') - it's not written anywhere, not even to the logfile. sys.exit('bad error') exits, but does not print anywhere either. How do I make it print to stdout?

3 3

[Twisted-Python] Win32 binaries for Python 2.6
by Paul Moore 15 Oct '08

15 Oct '08

Does anyone know when Twisted installers for Python 2.6 on Windows are likely to be available? Thanks, Paul

4 4

[Twisted-Python] how to make output file include xml declaration in nevow web framework?
by Boern 14 Oct '08

14 Oct '08

Hi,all: I am a newbie for nevow and now I write a wap page ,but the output file don`t include xml declaration ,eg: <?xml version="1.0" encoding="utf-8"?> . how to resolve ? my wap browser need it to parse the xml context and how I make output file include xml declaration? -- Boern Parx

1 0

[Twisted-Python] what to do!
by shweta mani 14 Oct '08

14 Oct '08

i need to execute a shell script file through SSH twisted or paramiko. if it is a normal file then directly with the command sh <filename>.sh it is getting executed. self.conn.sendRequest(self, 'exec', common.NS(sh test1.sh), wantReply = 1) but if a shell scripts asks for user input, this particular way does not wait for user input. it just print the echo part . is there any way so that interactive type of scripts can be executed in Paramiko or Twisted Conch in a windows machine (remote login in a unix machine through Paramiko SSH or Twisted) i did try with another way in Paramiko which helps in opening a remote window by creating an instance of SSHClient and invoking the method invoke_shell(with some parametes) but remote shell window is not opening.the method specifies starts a terminal shell window in SSH server. To my interpretation it is somewaht opening a terminal window like putty.

3 2

[Twisted-Python] Basic question about the architecture of twisted.internet
by Peter Jacobi 13 Oct '08

13 Oct '08

Dear All, I've just started diving into Twisted, read the docs, run the samples, wrote some tests, but: I'm rather clueless about some architectural decisions in Twisted Core, and perhaps somebody can clarify whether these are just hictorical accidents or serve a purpose I didn't get yet. Specifically I don't understand why the Factory classes in internet/protocols.py rely on class variables instead of instance variables for, most importantly "protocol", but also for "noisy" and "numPorts"? Won't this introduce an unwanted coupling between different instances of Factory, and effectively make the creation of more than instance of any Factory-derived class pointless (or even incorrect). Sort of a singleton pattern relying on the trust principle? Regards, Peter

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 12 Oct '08

12 Oct '08

1 0

[Twisted-Python] ANN: starpy-1.0.0a13
by Godson Gera 11 Oct '08

11 Oct '08

Hi All, I am happy to announce new release of starpy (Asterisk protocols for Twisted http://starpy.sf.net ) on behalf of starpy team. starpy-1.0.0a13 can be downloaded from here https://sourceforge.net/project/showfiles.php?group_id=164040 This release contains addition of many manager commands to manager API. Following is a list of newly added commands. 1. agentLogoff 2. dbGet 3. dbPut 4. extensionState 5. getConfig 6. meetmeMute 7. meetmeUnmute 8. park 9. pauseMonitor 10. playDTMF 11. queuePause 12. sipShowPeers 13. unpauseMonitor 14. updateConfig (read AMI documentation before using this ) 15. userEvent 16. waitEvent 17. zapDNDoff 18. zapDNDon 19. zapDialOffhook 20. zapHangup 21. zapRestart 22. zapShowChannels 23. zapTransfer - Those are all the new manager Actions that were added. Queues action is there but it won't work cause AMI returns improperly formatted lines, same case with IAXPeers action, as AgentCallbackLogin is deprecated and is not functioning properly it was not included. - Bug fix in queueStatus manager API, in collectDeferred argument 'QueueStatusEnd' is changed to 'QueueStatusComplete'. Now the deferred fires properly. - Bug fix in fastagi playback (not a standard AGI command, was created by Mike as a workaround for buggy streamFile command back in Asterisk 1.2 days. Now streamFile is working properly so you might not be using this playback ). Those are all the changes that are involved in this release if you find any bugs or hope for any new features let me know. Have fun with starpy, Godson Gera

1 0