Mailman 3 August 2011 - Twisted

[Twisted-Python] 11.0 in PPA
by Алексей Кирпичников Sept. 27, 2011

Sept. 27, 2011

Hi all, I hope that maintainers of Twisted PPA are subscribed to this mailing list :) Is there any chance that 11.0 will be packaged and uploaded to the PPA anytime soon? -- Alex

8 12

[Twisted-Python] Problem Reading a Directory with Conch/SFTP
by Jeffrey Ollie Aug. 29, 2011

Aug. 29, 2011

I'm re-writing a client that downloads some data from a 3rd party using SFTP. The old client was written using Paramiko but I'd like to rewrite it using Twisted and Conch. Right now I'm running into an issue trying to get a directory listing from the remote server: 2011-08-22 13:35:03-0500 [SSHChannel session (0) on SSHService ssh-connection on _WrappingProtocol,client] Unhandled Error Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/twisted/python/log.py", line 84, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/usr/lib64/python2.7/site-packages/twisted/python/log.py", line 69, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/usr/lib64/python2.7/site-packages/twisted/python/context.py", line 81, in callWithContext return func(*args,**kw) --- <exception caught here> --- File "/usr/lib64/python2.7/site-packages/twisted/conch/ssh/filetransfer.py", line 53, in dataReceived f(data) File "/usr/lib64/python2.7/site-packages/twisted/conch/ssh/filetransfer.py", line 711, in packet_STATUS msg, data = getNS(data) File "/usr/lib64/python2.7/site-packages/twisted/conch/ssh/common.py", line 36, in getNS l, = struct.unpack('!L',s[c:c+4]) struct.error: unpack requires a string argument of length 4 The problem seems to be that the remote SFTP implementation isn't returning complete status response message - it doesn't include the error message and the language identifier. I made a quite ugly workaround: diff --git a/twisted/conch/ssh/filetransfer.py b/twisted/conch/ssh/filetransfer. index 81a86fd..ed55b27 100644 --- a/twisted/conch/ssh/filetransfer.py +++ b/twisted/conch/ssh/filetransfer.py @@ -708,8 +708,15 @@ class FileTransferClient(FileTransferBase): d, data = self._parseRequest(data) code, = struct.unpack('!L', data[:4]) data = data[4:] - msg, data = getNS(data) - lang = getNS(data) + if len(data) >= 4: + msg, data = getNS(data) + if len(data) >= 4: + lang = getNS(data) + else: + lang = '' + else: + msg = '' + lang = '' if code == FX_OK: d.callback((msg, lang)) elif code == FX_EOF: Looking through the Paramiko code[1] it looks like it pads SFTP messages that are shorter than expected with null bytes. From what I saw in the SFTP I-D[2], a status message that doesn't include the error message and language code could be construed as legal even though they are not specifically marked as optional. [1] https://github.com/robey/paramiko/blob/master/paramiko/message.py#L103 [2] http://tools.ietf.org/html/draft-ietf-secsh-filexfer-13#section-4 -- Jeff Ollie

2 3

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Aug. 28, 2011

Aug. 28, 2011

1 0

[Twisted-Python] Retrying function calls
by Terry Jones Aug. 28, 2011

Aug. 28, 2011

I posted to the list about a Twisted class for retrying function calls almost two years ago and blogged about it at http://bit.ly/ncnE72 I've just done the work to put that class into its own package, available from various places: http://pypi.python.org/pypi/txretry https://github.com/fluidinfo/txretry https://launchpad.net/txretry For now the only documentation is the above blog post. Terry

2 1

[Twisted-Python] RPC design questions
by Tobias Oberstein Aug. 27, 2011

Aug. 27, 2011

I'm currently making some light-weight RPC/PubSub over WebSockets using Twisted. I have two design-like questions where I'd be happy for advice/options ... The first is definitely Twisted related, the second not strictly, .. Tobias 1) I'd like to chain RPC calls, i.e. self.call(23, "square").addCallback(self.call, "sqrt").addCallback(self.show) https://github.com/oberstet/Autobahn/blob/dev-rpc/demo/rpc/simple/simple_cl… self.call(<arg>, <procedure>) will return a deferred, <arg> is the marshalled argument for the RPC, and <procedure> is the remote procedure identifier. Now, the nice thing is, I can chain the result from one call to the next like in the example above. What I find less nice is that I have to have that order : <arg>, <procedure> since the result of the first deferred will be passed as the first argument to the second, and only then will the additional arguments be passed (in above example the "sqrt") How can I retain the - which I find - natural order for arguments? self.call(<procedure>, <arg>) self.call("square", 6).addCallback(self.call, "sqrt").addCallback(self.show) => not working, the 2nd self.call receives 36, "sqrt" .. 2) The server side methods a hooked up using decorators, like class SimpleServerProtocol(AutobahnServerProtocol): @AutobahnRpc def square(self, arg): return arg * arg https://github.com/oberstet/Autobahn/blob/dev-rpc/demo/rpc/simple/simple_se… Here, AutobahnServerProtocol derives (indirectly) from Twisted Protocol. The decorator will register the method for RPC under "square". https://github.com/oberstet/Autobahn/blob/dev-rpc/lib/python/autobahn/autob… That is all nice and simple, however I am wondering if it's a good idea to do that auto-registering on a Protocol derived class. I mean, the alternative could be having the user call something like registerRpcObject(<any class instance with RPC decorators>, <base URI>) in Protocol.connectionMade(), which then would auto-register all decorated methods in <any class instance ..> What do you think would be better?

8 19

[Twisted-Python] SFTP Server sample
by Osborn Chan Aug. 24, 2011

Aug. 24, 2011

Hi, I am a newbie in twisted and python. I would like to create a python SFTP server with twisted, but I cannot find any sample. Can someone give me some advise? Thanks, Osborn

7 12

[Twisted-Python] Basic problem with deferreds
by mayvimmer＠fibertel.com.ar Aug. 23, 2011

Aug. 23, 2011

Hi guys, I'm new to twisted world and I'm facing what seems to be an easy problem but I cannot grasp it. I thought I have already undestood the deferred concept but obviously I still have a concept error I cannot pass: I have a method with the following parameters: 1- an userid, 2- a security code, 3- a list of tuples composed by an object id and an object quantity. I need to check in a database if the user exists and he's active, if the security code exists and belongs to the user and for each object if it exists and then insert the submitted quantity in the database. What I try to do is to check both user and security code just once and then loop the objects to verify them and insert. If user or sec_code are invalid, this should fail right away, otherwise perform the action and add the results to a list. No matter how I place the methods or how I call them when evaluating the user or the security code, I always get a Deferred object and I need the actual result so that the method continue to be executed or not, evaluating each object in the list. I thought that using the "callback" method this will fire the callbacks and give me the result, but I cannot find where I need to execute it. Here is my code: class ExampleObject(object): def get_userid(self, username): r1 = defer.Deferred() def cb(rs): if len(rs)==0: resp = u'The user is not in the database' else: resp = rs[0]['id'] r1.callback(resp) r1 = self.conn_pool.runQuery('select id from users where ' 'username=?', (sql_safe(username),)).addCallback(cb) return r1 def get_sec_code(self, sec_code): r2 = defer.Deferred() def cb(rs): if len(rs)==0: resp = u'The security code is invalid' else: resp = rs[0]['userid'] r2.callback(resp) r2 = self.conn_pool.runQuery('select user_id from sec_codes where ' 'sec_code=?', (sql_safe(sec_code),)).addCallback(cb) return r2 def grant_objs(self, username, sec_code, objects): errmsg = None def grant_obj(pobj, puser, psec_code): d = defer.Deferred() def cb(rs): if len(rs)==0: d.callback(False) else: d.callback(True) def cb2(rs): if rs: self.conn_pool.runOperation('insert into inventory '\ '(user_id, obj_id, quantity, dateadded) values '\ '(?,?,?,?)', (pobj, pitem[0], pitemp[1], datetime.datetime.now())).callback(True) else: d.callback('Object with id %d does not exist' % (pobj[0])) self.conn_pool.runQuery("select id, ? as quantity from objects where " "id=?", (sql_safe(pobj[0]), sql_safe(pobj[1]))).\ addCallback(cb).addCallback(cb2) defs = [] def cb(value): return value for i in enumerate(items): if i[0]==0: # # I need the result right now, however I always get a Deferred object. # u_ = self.get_userid(username).addCallback(cb) t_ = self.get_sec_code(sec_code).addCallback(cb) if not isinstance(u_, int): errmsg = u_ elif not isinstance(t_, int): errmsg = t_ elif u_!=t_: errmsg = u'Your security code does not belong to the user' if errmsg is not None: defs.append(errmsg) break res = grant_obj(i[1], u_, t_) defs.append(res) return defs TIA for any pointers you can give me!

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Aug. 21, 2011

Aug. 21, 2011

1 0

[Twisted-Python] Custom protocol : good practices and complexity?
by Mark "Ckaos" Moissette Aug. 20, 2011

Aug. 20, 2011

I am currently working on a control system for Arduino type devices using Twisted,and have a bit of a design issue Hello everyone! I am currently working on a control system for Arduino type devices using Twisted,and have a bit of a design issue Here is how things are currently: (sorry in advance, might be a bit long) 1. to handle different type of devices (each having a different firmware & communication protocol ) i have a designed a "driver" system : - each driver is made of : - a "hardware handler class" : a wrapper around Twsited's *serial* class with a few added helper methods - a custom serial protocol 2- While implementing drivers for Reprap 3d printers (also based on arduino, also using a serial connection) with rather specific protocols (generally containing comands like *enqueue* point, *set* temperature etc), i have started to wonder if i am placing the methods for handling those features (each having specific commands) in the right place.. This all leads me to a few questions: While all things are working right now,I am not quite sure about the "good practices" as far asTwisted protocols go , but having looked through the documentation / code of quite a few of them, it seems they tend to have relatively few methods/keep it simple. - is this always the case? should the protocol *only* be used for very low level functions and in/out formatting and communication ? - certain devices i want to manage have very clearly defined protocols (Makerbot etc), should i consider general protocol specifications to be a different thing then the actual Twisted protocol classes i am creating ? Any advice, tips and pointers are more than welcome ! Thanks in advance!

1 0

[Twisted-Python] [ANN] Turtl 0.1.1 throttler released
by Valentino Volonghi Aug. 18, 2011

Aug. 18, 2011

Hi, Turtl is an HTTP proxy whose purpose is to throttle connections to specific hostnames to avoid breaking terms of usage of those API providers (like del.icio.us, technorati and so on). At the core of turtl is a throttling deferred that works in a similar way as DeferredSemaphore() except that it will enforce also a rate (N calls every M seconds) at which deferreds added to it are fired. In the past few weeks it's been improved a couple obscure bugs have been ironed out. It's been running as a proxy for a couple of years and recently we started using it as a crawler rate limiter. Source code lives on bitbucket: https://bitbucket.org/adroll/turtl/overview Here's a small example of its usage: import time from twisted.internet import reactor, defer from twisted.protocols.policies import WrappingFactory from twisted.web import client, server, resource from turtl import engine throttle = engine.ThrottlingDeferred(concurrency=1, calls=2, interval=1) class FakeResource(resource.Resource): isLeaf = True def render(self, request): return "hello" def setupServer(): site = server.Site(FakeResource()) wrapper = WrappingFactory(site) port = reactor.listenTCP(0, wrapper, interface="127.0.0.1") portno = port.getHost().port return portno def stop(_): return reactor.stop() def makeUrl(port): return "http://localhost:%s/" % (port) def prinl(page): print time.time(), page port = setupServer() url = makeUrl(port) defer.DeferredList([throttle.run(client.getPage, url).addBoth(prinl) for i in xrange(1000)]).addBoth(stop) reactor.run() -- Valentino Volonghi http://www.adroll.com

3 8