Mailman 3 November 2012 - Twisted

[Twisted-Python] Getting my Cred interfaces right: IUsername(Hashed)?Password
by Laurens Van Houtven Nov. 14, 2012

Nov. 14, 2012

Hi, I'm trying to make sure that I have my cred interfaces right. Users log in using a username and password. They provide these credentials in plaintext (over a TLSd connection). The user password is stored using a secure key derivation function (in casu, scrypt). Currently I have this gumongous User object (an Axiom Item), and I'm trying to split it up into parts. IIUC, the checker's checked interface should be IUsernamePassword (that's already the case). However, the thing I adapt a User to to check it should be an IUsernameHashedPassword, right? In the end, I doubt this matters an awful lot, unless somebody ends up implementing a IUsernameHashedPassword checker that is smart enough to read both scrypt/bcrypt headers and /etc/shadow-style $-delimited entries. cheers lvh

2 2

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Nov. 11, 2012

Nov. 11, 2012

1 0

[Twisted-Python] defer.TimeoutError in tests
by Maxim Lacrima Nov. 9, 2012

Nov. 9, 2012

Hi! I have some tests that fail intermittently, i.e. sometimes they fail, sometimes they don't. These tests were written against real server. When I run these tests, the following errors may or may not occur: [ERROR] Traceback (most recent call last): Failure: twisted.internet.defer.TimeoutError: <...> still running at 120.0 secs [ERROR] Traceback (most recent call last): Failure: twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean. Selectables: <<class 'twisted.internet.tcp.Client'> to ('localhost', 8081) at 1eb1750> I don't understand these errors. The server is up and running, but for some reason connections are timed out. The code under test uses perspective broker to talk to the server. I can't figure out where the problem is -- on the server or in the code under test (the client). I have found some information that can help me to tackle the problem -- http://twistedmatrix.com/trac/ticket/2869 But what can be the reason of such intermittent errors? Sorry, right now I can't provide a self contained example that demonstrates the problem, because testing code is rather complex. At first, I would like to know if there is something I could pay attention to. Thanks in advance. -- with regards, Maxim

2 2

[Twisted-Python] Twisted Conch - Flow control
by Matt Freeman Nov. 7, 2012

Nov. 7, 2012

I asked this question on Stackoverflow (even offered a bounty) and IRC: http://stackoverflow.com/questions/13117502/twisted-conch-flow-control -- I have a Twisted Conch SSH server and the typical scenario is this: git via OpenSSH client >>--- WAN1 --->> Twisted conch svr >>--- WAN2 -->> Git server There will be occassions that the 'git push' is sending data faster over WAN1 than I can proxy it over WAN2, so I need to tell the client to slow down (well before any TCP packet loss causes adjustments the TCP window size) to avoid buffering too much on the Twisted server. Reading the RFC for SSH this is accomplished with not acknowledging via adj window this will then cause the git push to block on syscall write to the pipe backed by openssh. Looking at conch/ssh/connection.py:L216 in the method def ssh_CHANNEL_DATA(self, packet): I can accomplish this with setting localWindowSize to 0, and inflight data will still land as the predicate on 230 should still pass (give localWindowLeft). I am wondering if this is the correct approach or am I missing something blindly obvious with regards to flow control with Twisted SSH Conch? * Note: I acknowledge there are methods placeholders for stopWriting and startWriting on (channel) that I can override so I have hooks to control the other side of the transmission 'git pull', but Im interested in first leg. Also IPush/IPull producer dont seem applicable at this level and I cant see how I can tie in these higher abstraction without butchering conch? -- Matt Freeman - @nonuby - http://blog.nonuby.com

2 2

[Twisted-Python] Create client in Server factory
by Grégoire Leroy Nov. 6, 2012

Nov. 6, 2012

Hi, I have a server who receive data from clients A/B/C (remote machines). I want to use a client D (on the same machine than the server) to send this data to another server (remote). The difficulty is, I want to use the same client D connection for any client. I don't want open an new connection each time. First thing I would think is to create the connection in my server factory, and use client's methods in my server protocol, for example with [...] class LocalProxyFactory(Factory): def __init__(self): f = LocalProxyClientFactory() reactor.connectTCP("retenodus.net", 4242, f) reactor.run() [...] But in this case, code execution will stay in the reactor, right ? I have read that : http://twistedmatrix.com/trac/wiki/FrequentlyAskedQuestions#HowdoImakeinput… But I don't know how it can be applied to my situation. Thanks, Grégoire Leroy

3 5

[Twisted-Python] Request for help: throttling serving of static content
by Peter Inglesby Nov. 6, 2012

Nov. 6, 2012

Tomorrow I'm giving a talk to provide an introduction to asynchronous programming. I'll be mentioning Twisted in passing, but the meat of the talk will be about writing a web crawler for finding broken links, first using single-threaded blocking code, then multi-threaded blocking code, before demonstrating a non-blocking solution using a select loop. I can make the code and slides available if there's interest. For this to be effective, I was planning on writing a very slow webserver to serve static content locally. I'd expected I'd be able to simply wrap an instance of twisted.web.static.Site in a ThrottlingFactory, and serve away: import sys from twisted.internet import reactor from twisted.internet.endpoints import TCP4ServerEndpoint from twisted.protocols.policies import ThrottlingFactory from twisted.web.server import Site from twisted.web.static import File from twisted.python import log if __name__ == '__main__': log.startLogging(sys.stdout) resource = File(sys.argv[1]) factory = Site(resource) endpoint = TCP4ServerEndpoint(reactor, 4321) endpoint.listen(ThrottlingFactory(factory, maxConnectionCount=3, readLimit=20, writeLimit=20)) reactor.run() However, this fails with the following error because I've not registered a producer with my ThrottlingProtocol: 2012-11-05 21:01:34+0000 [-] Throttling writes on <twisted.protocols.policies.ThrottlingFactory instance at 0x1016497e8> 2012-11-05 21:01:34+0000 [-] Unhandled Error Traceback (most recent call last): File "slow_server.py", line 16, in <module> reactor.run() File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/twisted/internet/base.py", line 1169, in run self.mainLoop() File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/twisted/internet/base.py", line 1178, in mainLoop self.runUntilCurrent() --- <exception caught here> --- File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/twisted/internet/base.py", line 800, in runUntilCurrent call.func(*call.args, **call.kw) File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/twisted/protocols/policies.py", line 290, in checkWriteBandwidth self.throttleWrites() File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/twisted/protocols/policies.py", line 324, in throttleWrites p.throttleWrites() File "/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/twisted/protocols/policies.py", line 221, in throttleWrites self.producer.pauseProducing() exceptions.AttributeError: 'NoneType' object has no attribute 'pauseProducing' I'm sure it's not impossible to figure out what I need to do here, but I'd appreciate any pointers. Many thanks in advance!

4 4

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Nov. 4, 2012

Nov. 4, 2012

1 0

[Twisted-Python] twisted thumbnail server
by Paul Wiseman Nov. 2, 2012

Nov. 2, 2012

I hope this will be an easy question for some of you guys :) I'm trying to set up a simple server which will accept requests over GET to create a thumbnail for an image, and server it back as the response. The images are stored in two S3 buckets, the originals are in one bucket (store), and the generated thumbnails are stored in another (thumb) as a cache so that the work doesn't need to be repeated. Currently I'm checking if the thumbnail already exists in the thumb bucket. I'm redirecting the request if it is or if not I'm downloading the image from store, generating the thumb using PIL, uploading the thumbnail to the thumb bucket and then redirecting the request. I'm very new to twisted and was wondering if anyone who is more experienced would be able to take a look at what I have so far and let me know if anything is wrong/not ideal/will cause problems etc. or just general style pointers? The more critical the better, as I said I'm very new to this. I've just chucked it up on github: https://github.com/GP89/thumbs/blob/master/thumb.py There's a definite memory leak right now which I believe is PIL, or possibly StringIO objects not being disposed, hence all the random del statements trying to cure it (unsuccessfully). Maybe there's something I'm doing wrong in twisted that is causing things not to be cleaned up that I'm not aware of as well. I did try to use deferToThread, rather than my thread pool but the server seemed to block up- I probably should have left it incase it was because I was doing something obviously wrong. I think I'll make a branch quickly with my deferToThread version. Thanks very much for any time you can lend! Paul

3 4

[Twisted-Python] deferToThread and reactor loop
by Tobias Oberstein Nov. 2, 2012

Nov. 2, 2012

Hi, I've got a question regarding deferToThread / runWithConnection I have a network server that accepts RPCs and forwards those to a relational database calling stored procedures. The call of the stored procedures is happening via "runWithConnection" .. that is on a background thread pool (since that DBs driver is blocking). Now, when there is a large number of RPCs coming in roughly at once, the latency of each call forwarded to the RDBMs increased into the 10's of ms. I have created a self-contained test program without DB etc that I think reproduces the "issue": https://github.com/oberstet/scratchbox/blob/master/python/twisted/threadpoo… ************** Am I correct that "deferToThread" does not immediately forward the call to a background thread, but only the next time the reactor loop runs? ************** At least that is what I conclude from the test program's option https://github.com/oberstet/scratchbox/blob/master/python/twisted/threadpoo… which produces totally different timing results: $ python test.py 800 no 20 2012-10-29 18:41:11+0100 [-] Log opened. 2012-10-29 18:41:11+0100 [-] Using Twisted reactor class <class 'twisted.internet.iocpreactor.reactor.IOCPReactor'> 2012-10-29 18:41:11+0100 [-] Thread pool size suggested: 20 2012-10-29 18:41:11+0100 [-] Worker count: 800 2012-10-29 18:41:11+0100 [-] release to reactor between deferToThread: False 2012-10-29 18:41:11+0100 [-] total run-time: 121 ms 2012-10-29 18:41:11+0100 [-] avg. thread switch IN time: 61177 us 2012-10-29 18:41:11+0100 [-] max. thread switch IN time: 79436 us 2012-10-29 18:41:11+0100 [-] avg. thread switch OUT time: 16021 us 2012-10-29 18:41:11+0100 [-] max. thread switch OUT time: 23150 us 2012-10-29 18:41:11+0100 [-] Main loop terminated. $ python test.py 800 yes 20 2012-10-29 18:41:14+0100 [-] Log opened. 2012-10-29 18:41:14+0100 [-] Using Twisted reactor class <class 'twisted.internet.iocpreactor.reactor.IOCPReactor'> 2012-10-29 18:41:14+0100 [-] Thread pool size suggested: 20 2012-10-29 18:41:14+0100 [-] Worker count: 800 2012-10-29 18:41:14+0100 [-] release to reactor between deferToThread: True 2012-10-29 18:41:14+0100 [-] total run-time: 201 ms 2012-10-29 18:41:14+0100 [-] avg. thread switch IN time: 127 us 2012-10-29 18:41:14+0100 [-] max. thread switch IN time: 5131 us 2012-10-29 18:41:14+0100 [-] avg. thread switch OUT time: 166 us 2012-10-29 18:41:14+0100 [-] max. thread switch OUT time: 7886 us 2012-10-29 18:41:15+0100 [-] Main loop terminated. === Could someone pls enlighten me rgd the "issue"? What's the point, and how to interpret the timings? Any hints greatly appreciated! Thanks, Tobias Above timings are on Windows. Here other platforms: Results on FreeBSD ================ [oberstet@www ~/scm/scratchbox/python/twisted/threadpoollatency]$ python test.py 800 no 20 2012-10-29 17:53:37+0000 [-] Log opened. 2012-10-29 17:53:37+0000 [-] Using Twisted reactor class <class 'twisted.internet.kqreactor.KQueueReactor'> 2012-10-29 17:53:37+0000 [-] Thread pool size suggested: 20 2012-10-29 17:53:37+0000 [-] Worker count: 800 2012-10-29 17:53:37+0000 [-] release to reactor between deferToThread: False 2012-10-29 17:53:37+0000 [-] total run-time: 469 ms 2012-10-29 17:53:37+0000 [-] avg. thread switch IN time: 244311 us 2012-10-29 17:53:37+0000 [-] max. thread switch IN time: 300846 us 2012-10-29 17:53:37+0000 [-] avg. thread switch OUT time: 2347 us 2012-10-29 17:53:37+0000 [-] max. thread switch OUT time: 25703 us 2012-10-29 17:53:37+0000 [-] Main loop terminated. [oberstet@www ~/scm/scratchbox/python/twisted/threadpoollatency]$ python test.py 800 yes 20 2012-10-29 17:53:44+0000 [-] Log opened. 2012-10-29 17:53:44+0000 [-] Using Twisted reactor class <class 'twisted.internet.kqreactor.KQueueReactor'> 2012-10-29 17:53:44+0000 [-] Thread pool size suggested: 20 2012-10-29 17:53:44+0000 [-] Worker count: 800 2012-10-29 17:53:45+0000 [-] release to reactor between deferToThread: True 2012-10-29 17:53:45+0000 [-] total run-time: 707 ms 2012-10-29 17:53:45+0000 [-] avg. thread switch IN time: 410 us 2012-10-29 17:53:45+0000 [-] max. thread switch IN time: 6184 us 2012-10-29 17:53:45+0000 [-] avg. thread switch OUT time: 310 us 2012-10-29 17:53:45+0000 [-] max. thread switch OUT time: 1331 us 2012-10-29 17:53:45+0000 [-] Main loop terminated. Results on Linux ============= [webmq@webmqpgcentos threadpoollatency]$ python test.py 800 no 20 2012-10-29 18:56:13+0100 [-] Log opened. 2012-10-29 18:56:13+0100 [-] Using Twisted reactor class <class 'twisted.internet.epollreactor.EPollReactor'> 2012-10-29 18:56:13+0100 [-] Thread pool size suggested: 20 2012-10-29 18:56:13+0100 [-] Worker count: 800 2012-10-29 18:56:13+0100 [-] release to reactor between deferToThread: False 2012-10-29 18:56:13+0100 [-] total run-time: 178 ms 2012-10-29 18:56:13+0100 [-] avg. thread switch IN time: 77332 us 2012-10-29 18:56:13+0100 [-] max. thread switch IN time: 123225 us 2012-10-29 18:56:13+0100 [-] avg. thread switch OUT time: 22048 us 2012-10-29 18:56:13+0100 [-] max. thread switch OUT time: 43779 us 2012-10-29 18:56:13+0100 [-] Main loop terminated. [webmq@webmqpgcentos threadpoollatency]$ python test.py 800 yes 20 2012-10-29 18:56:14+0100 [-] Log opened. 2012-10-29 18:56:14+0100 [-] Using Twisted reactor class <class 'twisted.internet.epollreactor.EPollReactor'> 2012-10-29 18:56:14+0100 [-] Thread pool size suggested: 20 2012-10-29 18:56:14+0100 [-] Worker count: 800 2012-10-29 18:56:15+0100 [-] release to reactor between deferToThread: True 2012-10-29 18:56:15+0100 [-] total run-time: 370 ms 2012-10-29 18:56:15+0100 [-] avg. thread switch IN time: 694 us 2012-10-29 18:56:15+0100 [-] max. thread switch IN time: 3204 us 2012-10-29 18:56:15+0100 [-] avg. thread switch OUT time: 483 us 2012-10-29 18:56:15+0100 [-] max. thread switch OUT time: 7991 us 2012-10-29 18:56:15+0100 [-] Main loop terminated.

2 2