Mailman 3 September 2010 - Twisted

[Twisted-Python] Telnet MCCP Transport
by Vermeersch Simon 05 Sep '10

05 Sep '10

Hello everyone, I couldn't find an implementation of MCCP (Mud Client Compression Protocol) using Twisted so I made my own, which I'm sharing if anyone else has a use for it. You can find a description of MCCP here: http://www.mudstandards.org/MCCP_Specification Since I'm fairly new to Twisted I would appreciate any input about how to improve this if you find a better/cleaner way to do it. import zlib from twisted.conch.telnet import TelnetTransport from twisted.conch.telnet import DO, DONT COMPRESS2 = chr(86) class MCCPTelnetTransport(TelnetTransport): def connectionMade(self): self.zlib = None TelnetTransport.connectionMade(self) self.will(COMPRESS2) def commandReceived(self, command, argument): if argument == COMPRESS2: if command == DO: self.requestNegotiation(COMPRESS2, "") self.zlib = zlib.compressobj() elif command == DONT: pass else: TelnetTransport.commandReceived(self, command, argument) def write(self, data): data = data.replace('\n', '\r\n') if self.zlib: data = self.zlib.compress(data) data += self.zlib.flush(zlib.Z_SYNC_FLUSH) self.transport.write(data) -- Simon Vermeersch

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 04 Sep '10

04 Sep '10

1 0

Re: [Twisted-Python] multiple workers
by ruslan usifov 02 Sep '10

02 Sep '10

Why it is not supported? I want behaviour like nginx http://nginx.org/, and misunderstand why i can't implemented it throw twisted. Its' so easy. Every process have it's own set sockets, and they doesn't share this sockets between each other. "OnConnect" event happens only once and which process handle this event depend on operation system(select epoll, kqueue), in my case this happens like round robin(FreeBSD 7.2-RELEASE-p8). Where here is unsupported behaviour?

2 1

[Twisted-Python] multiple workers
by ruslan usifov 02 Sep '10

02 Sep '10

Hello I try to write twisted based daemon that work in multiple workers, like this: from twisted.internet import reactor; from proxy import FASTCGIServerProxyFactory; import os; reactor.listenUNIX("/tmp/twisted-fcgi.sock", FASTCGIServerProxyFactory()); for i in xrange(3): l_pid = os.fork(); if(l_pid == 0): break; reactor.run() I creatae 4 wokers(by num of CPU cores). Im my test all ok, but when i shutdown daemon i got follow error 3 times (because every worker try to unlink sock file: /tmp/twisted-fcgi.sock): Traceback (most recent call last): File "/usr/local/lib/python2.6/site-packages/twisted/internet/defer.py", line 249, in addCallbacks self._runCallbacks() File "/usr/local/lib/python2.6/site-packages/twisted/internet/defer.py", line 441, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/usr/local/lib/python2.6/site-packages/twisted/internet/base.py", line 426, in _continueFiring callable(*args, **kwargs) File "/usr/local/lib/python2.6/site-packages/twisted/internet/base.py", line 613, in disconnectAll failure.Failure(main.CONNECTION_LOST)) --- <exception caught here> --- File "/usr/local/lib/python2.6/site-packages/twisted/python/log.py", line 84, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/local/lib/python2.6/site-packages/twisted/python/log.py", line 69, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/local/lib/python2.6/site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/local/lib/python2.6/site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/usr/local/lib/python2.6/site-packages/twisted/internet/unix.py", line 134, in connectionLost os.unlink(self.port) exceptions.OSError: [Errno 2] No such file or directory: '/tmp/twisted-fcgi.sock' So my questions is How can i suppress this behaviour, so shutdown code executes only once in main process?

2 1

[Twisted-Python] Ldaptor: [PATCH] Extend test driver send_multiResponse() to return deferred and throw errors
by Anton Gyllenberg 02 Sep '10

02 Sep '10

The deferred returned by the LDAP client send_multiResponse() method was previously unused in all code covered by tests, and so the replacement method in the test driver just returned None. The deferred is now used in search() and this change makes the test driver also return a deferred instead of None in order to make things work when run within the test framework. To make it possible to test failures in the client send() and send_multiResponse() methods, the test driver is changed to accept Failure instances in place of lists of LDAPProtocolResponses. Doing this causes the errback on the deferred to be called with this failure. The LDAPSyntaxSearch and Bind test cases are augmented with one test each to use the new failure functionality in the client test driver. As the search() code a while back did not handle errors in the send_multiResponse() deferred chain the test case would time out if run against older code. Therefore a smaller timeout of 3 seconds is set for the LDAPSyntaxSearch test case. --- Discussion: With this change the old test cases pass and the code path introduced by my modifications to send_multiResponse() is tested by a new test case. I am still a bit unsure if I am testing the right thing and if the original fix is the right thing to do. Any comments welcome! Code published on http://github.com/antong/ldaptor/tree/pu ldaptor/test/test_ldapsyntax.py | 25 +++++++++++++++++++++++++ ldaptor/testutil.py | 18 +++++++++++++++++- 2 files changed, 42 insertions(+), 1 deletions(-) diff --git a/ldaptor/test/test_ldapsyntax.py b/ldaptor/test/test_ldapsyntax.py index b8bcf53..46be06c 100755 --- a/ldaptor/test/test_ldapsyntax.py +++ b/ldaptor/test/test_ldapsyntax.py @@ -7,6 +7,7 @@ from ldaptor import config, testutil, delta from ldaptor.protocols.ldap import ldapsyntax, ldaperrors from ldaptor.protocols import pureldap, pureber from twisted.internet import defer +from twisted.internet import error from twisted.python import failure from ldaptor.testutil import LDAPClientTestDriver @@ -366,6 +367,7 @@ class LDAPSyntaxAttributesModificationOnWire(unittest.TestCase): class LDAPSyntaxSearch(unittest.TestCase): + timeout = 3 def testSearch(self): """Test searches.""" @@ -641,6 +643,17 @@ class LDAPSyntaxSearch(unittest.TestCase): d.addCallbacks(testutil.mustRaise, eb) return d + def testSearch_err(self): + client=LDAPClientTestDriver([ + failure.Failure(error.ConnectionLost()) + ]) + o = ldapsyntax.LDAPEntry(client=client, dn='dc=example,dc=com') + d = o.search(filterText='(foo=a)') + def eb(fail): + fail.trap(error.ConnectionLost) + d.addCallbacks(testutil.mustRaise, eb) + return d + class LDAPSyntaxDNs(unittest.TestCase): def testDNKeyExistenceSuccess(self): client = LDAPClientTestDriver() @@ -1516,3 +1529,15 @@ class Bind(unittest.TestCase): fail.trap(ldaperrors.LDAPInvalidCredentials) d.addCallbacks(testutil.mustRaise, eb) return d + + def test_err(self): + client = LDAPClientTestDriver([ + failure.Failure(error.ConnectionLost())]) + + o=ldapsyntax.LDAPEntry(client=client, + dn='cn=foo,dc=example,dc=com') + d = defer.maybeDeferred(o.bind, 'whatever') + def eb(fail): + fail.trap(error.ConnectionLost) + d.addCallbacks(testutil.mustRaise, eb) + return d diff --git a/ldaptor/testutil.py b/ldaptor/testutil.py index 8307cb9..cb25aa3 100644 --- a/ldaptor/testutil.py +++ b/ldaptor/testutil.py @@ -1,6 +1,7 @@ """Utilities for writing Twistedy unit tests and debugging.""" from twisted.internet import defer +from twisted.python import failure from twisted.trial import unittest from twisted.test import proto_helpers from ldaptor import config @@ -36,23 +37,37 @@ class LDAPClientTestDriver: messages are stored in self.sent, so you can assert that the sent messages are what they are supposed to be. + It is also possible to include a Failure instance instead of a list + of LDAPProtocolResponses to cause which will cause the errback + to be called with the failure. + """ def __init__(self, *responses): self.sent=[] self.responses=list(responses) self.connected = None self.transport = FakeTransport(self) + def send(self, op): self.sent.append(op) l = self._response() assert len(l) == 1, \ "got %d responses for a .send()" % len(l) - return defer.succeed(l[0]) + r = l[0] + if isinstance(r, failure.Failure): + return defer.fail(r) + else: + return defer.succeed(r) + def send_multiResponse(self, op, handler, *args, **kwargs): + d = defer.Deferred() self.sent.append(op) responses = self._response() while responses: r = responses.pop(0) + if isinstance(r, failure.Failure): + d.errback(r) + break ret = handler(r, *args, **kwargs) if responses: assert not ret, \ @@ -60,6 +75,7 @@ class LDAPClientTestDriver: else: assert ret, \ "no more responses to give, but handler still wants more (got %r)." % ret + return d def send_noResponse(self, op): responses = self.responses.pop(0) -- 1.7.1.5.g49342

1 0

[Twisted-Python] Twisted and SOAP
by Einar S. Idsø 02 Sep '10

02 Sep '10

Hi, What is currently the recommended option for letting Twisted applications communicate with SOAP services? Cheers, Einar

4 5

[Twisted-Python] adbapi MySQL read/write splitting
by Jason J. W. Williams 01 Sep '10

01 Sep '10

Is anyone doing read/write splitting with adbapi? Looking for advice if anyone has tackled this before. Thank you in advance. -J

3 8

[Twisted-Python] Ldaptor: [PATCH] Reroute errback to deferred returned by search()
by Anton Gyllenberg 01 Sep '10

01 Sep '10

The LDAPEntryWithClient.search() method used to send the LDAP request with a call like self.client.send_multiResponse(... ) send_multiResponse() returns a deferred that was just discarded. If the operation causes an error then the errback fired on the discarded deferred will remain unhandled. The deferred returned by search() will then not have any errback fired and the caller of search() will be waiting forever. This change adds an errback to the deferred returned by send_multiResponse() and has the error rerouted to the errback chain of the deferred returned by search(). --- Discussion: This hits right into the internals of Ldaptor and I am not sure I understand this well enough to make an accurate fix. The problem I encountered is real and can be reproduced, and this change does seem to work around it. Any comments and suggestions welcome! ldaptor/protocols/ldap/ldapsyntax.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/ldaptor/protocols/ldap/ldapsyntax.py b/ldaptor/protocols/ldap/ldapsyntax.py index f54d2f8..471ffa6 100644 --- a/ldaptor/protocols/ldap/ldapsyntax.py +++ b/ldaptor/protocols/ldap/ldapsyntax.py @@ -683,7 +683,7 @@ class LDAPEntryWithClient(entry.EditableLDAPEntry): typesOnly=typesOnly, filter=filterObject, attributes=attributes) - self.client.send_multiResponse( + dsend = self.client.send_multiResponse( op, self._cbSearchMsg, d, cb, complete=not attributes, sizeLimitIsNonFatal=sizeLimitIsNonFatal) @@ -692,6 +692,11 @@ class LDAPEntryWithClient(entry.EditableLDAPEntry): else: if callback is None: d.addCallback(lambda dummy: results) + def rerouteerr(e): + d.errback(e) + # returning None will stop the error + # from being propagated and logged. + dsend.addErrback(rerouteerr) return d def lookup(self, dn): -- 1.7.1.5.g49342

2 3

[Twisted-Python] Ldaptor status and updates
by Anton Gyllenberg 01 Sep '10

01 Sep '10

I want to use the Ldaptor LDAP client for a project. Does anybody know whether and how Ldaptor is currently maintained and what the best upstream source is? I need to make some changes and wonder what to base the work on and where to discuss and publish changes. Is it OK to discuss Ldaptor on this list and to send patches here? The first google hit for ldaptor is http://webmail.inoi.fi/open/trac/ldaptor but the SVN repo on that site has last commit Sun May 7 2006. I found a git repo with newer commits at http://eagain.net/gitweb/?p=ldaptor.git, last commit Sat Aug 23 2008. I published some work-in-progress changes based on the git repo above at http://github.com/antong/ldaptor Best regards Anton

2 2