Mailman 3 August 2007 - Twisted

[Twisted-Python] Checking whether a server started successfully
by Maarten ter Huurne 22 Aug '07

22 Aug '07

Hi, I am starting a server process using "twistd -oy" (don't save state; application is described by Python source). If all is well, the server is started in the background. However, if all is not well, I'd like to detect that startup failed. The particular scenario that I'm using is testing a server using a test framework that does HTTP requests. Before the tests are run, the server is started and it tries to bind to a fixed port. If there is still an old instance of the server running, the bind will fail and any requests made will be handled by the old instance instead. That will lead to invalid test results. I worked around it currently by waiting for 3 seconds (probably sufficient time to try the binding) and then checking if the process of which the ID stored in "twisted.pid" is still running. However, there is no guarantee that 3 seconds is always enough time for the server to have bound the port. So I'm wondering if there is a reliable and elegant way of solving this. One approach would be to check which process has bound a particular port. I'm running this server on Linux and the "netstat" command line tool has access to the right information, but not in an easily parseable format. I guess the same info can be found in /proc somehow, but are those files guaranteed to stay compatible with newer kernels? Another approach would be to check in advance if something is listening on that particular port, but between the time this check is done and the time the server tries to bind, some other process might have grabbed the port. In any case, the port already being bound is just one of the reasons why the startup might fail; I'd prefer to have a way to detect startup failures for other reasons as well. One question is when exactly "startup" is finished. For my purpose, the moment the reactor is running would be the moment I consider startup to be successful. Maybe I can write a file somewhere to explicitly signal that, but that's a bit messy (where to put it? is the server process allowed to write there? who cleans up the file?). Maybe it would be useful to have twistd (optionally) wait until the reactor is running before returning. I don't want the test script to rely on a patched version of Twisted, so this would only be an option if it would make sense to integrate such a feature into Twisted itself. Have you encountered this problem as well? And if so, how did you deal with it? Bye, Maarten

3 2

[Twisted-Python] Adapter for a class, not for an instance
by Vincent Bernat 22 Aug '07

22 Aug '07

Hi ! Currently, adapters can transform an instance into another instance. I would like to use adapters to transform a class into an instance. Does something like that already exists ? Thanks.

3 3

[Twisted-Python] invoking Deferred synchronously
by Laurie Harper 21 Aug '07

21 Aug '07

I'm sure this is trivial enough, but it has me scratching my head... I need to call a method that returns a Deferred, then block until the deferred completes (or errback's) and return the final result of the deferred call. Specifically, something like: r = None, is_ok = None def ok(p): r, is_ok = p, True def fail(e): r, is_ok = e, False cf = pb.PBClientFactory() reactor.connectTCP(host, port, cf) d = cf.login(creds, client=self).addCallbacks(ok, fail) # BLOCK: wait until d has called 'ok' of 'fail' if is_ok: return r else: raise r If it helps, I'm trying to create a web application (using Nevow) that connects to another Twister service through the Persistence Broker. I'm trying to implement authentication, and so need a way to wait for the PB connection to succeed/fail so I can send the appropriate response to the HTTP request. Any pointers would be very much appreciated! :-) L.

2 2

[Twisted-Python] Questions about the Perspective Broker login mechanism
by Julien Peeters 21 Aug '07

21 Aug '07

Hi readers, I'm looking for the person who implements the actual version of the login mechanism in the Perspective Broker. More precisely, I wonder what are the reasons to have chosen a challenge/response strategy to do login. I use the Perspective Broker in a project and before modifying anything, I want to be sure of what I do. Thank you for your attention, Julien.

4 13

[Twisted-Python] locking threads when deferToThread is used
by Ladislav Andel 20 Aug '07

20 Aug '07

Hi, I'm writing an application which will be periodically testing servers. I will have a global list of these servers(domain names) and need to do few tasks. 1) DNS checks - I will use asynchronous twisted-names for it - in case there is a difference comparing to the list it should update the list(then also in DB) 2) ICMP pings - should be also possible to do it asynchronously 3) Blocking function which will be pinging with SIP requests - here I will use function deferToThread to make it non-blocking. Questions: 1) How do I lock each thread when writing to a global list in twisted? 2) How will I put together all three results mentioned above in the global list - is it by using function callLater ? 3) Could you help me with a rough skeleton of this application, please? Thank you, Lada

5 7

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 19 Aug '07

19 Aug '07

Bug summary ______________________ Summary for 2007-08-12 through 2007-08-19 Bugs opened: 9 Bugs closed: 5 Total open bugs: 1014 (+4) |== Type Changes |== Priority Changes |== Component Changes |Defect: +5 |Highest: -2 |Conch: +1 |Enhancement: +1 |Normal: +7 |Core: +2 |Lowest: +1 |Trial: +3 New / Reopened Bugs ______________________ ===== Highest ===== [#2679] twisted.conch.ssh.keys has poor tests. (opened by z3p) defect conch http://twistedmatrix.com/trac/ticket/2679 ===== Normal ===== [#2784] trial reporter plugins can't get real failures unless they subclass trial's reporter class (opened by exarkun) defect trial http://twistedmatrix.com/trac/ticket/2784 [#2785] twisted.trial.itrial.IReporter is missing addSkip (opened by exarkun) defect trial http://twistedmatrix.com/trac/ticket/2785 [#2786] Trial's --until-failure feature is fragile and under-tested (opened by jml) enhancement trial http://twistedmatrix.com/trac/ticket/2786 [#2787] IReactorProcess.spawnProcess doesn't document all exceptions which the implementation can raise (opened by exarkun) defect core http://twistedmatrix.com/trac/ticket/2787 [#2788] StringTransport does not support .host/.port (opened by Lacrymology) defect core http://twistedmatrix.com/trac/ticket/2788 [#2790] UDP Transport write() raises socket.error EWOULDBLOCK (opened by Peaker) defect core http://twistedmatrix.com/trac/ticket/2790 [#2786] Trial's --until-failure feature is fragile and under-tested (opened by jml) enhancement trial http://twistedmatrix.com/trac/ticket/2786 ===== Lowest ===== [#2789] Windows Process Pipes are polled with 100% cpu (opened by Peaker) defect core http://twistedmatrix.com/trac/ticket/2789 Closed Bugs ______________________ ===== Highest ===== [#2749] Add a utility method to trial to check the type of an object (opened by therve, closed by therve, fixed) enhancement trial http://twistedmatrix.com/trac/ticket/2749 [#2341] Merge twisted.internet.process.PTYProcess and Process (opened by itamarst, closed by therve, fixed) defect core http://twistedmatrix.com/trac/ticket/2341 [#2769] Update plugins howto to cover new API for <package>/plugins/__init__.py added in #2339 (opened by itamarst, closed by exarkun, fixed) defect core http://twistedmatrix.com/trac/ticket/2769 [#2679] twisted.conch.ssh.keys has poor tests. (opened by z3p, closed by z3p, ) defect conch http://twistedmatrix.com/trac/ticket/2679 ===== Normal ===== [#2786] Trial's --until-failure feature is fragile and under-tested (opened by jml, closed by jml, ) enhancement trial http://twistedmatrix.com/trac/ticket/2786 Ticket Lifetime Stats ______________________ Oldest open ticket - [#38] Should be a way to create documentation aliases (since 2003-07-12 16:08:03). Newest open ticket - [#2790] UDP Transport write() raises socket.error EWOULDBLOCK (since 2007-08-18 17:39:59). Mean open ticket age: 505 days, 15:14:17.121751. Median: 397 days, 19:31:32.214454. Standard deviation: 396 days, 9:06:38.504059. Interquartile range: 490 days, 3:14:30. Mean time between ticket creation and ticket resolution: 119 days, 13:28:12.330649. Median: 14 days, 3:38:33. Standard deviation is 222 days, 18:22:19.136294. The interquartile range is 125 days, 7:40:09. Mean time spent in review: 61 days, 21:07:27.431849. Median: 4 days, 0:03:10. Standard deviation: 134 days, 13:25:27.447625. Interquartile range: 29 days, 21:36:30.259426. Mean number of times a ticket is reviewed: 1.88461538462. Median: 1 Standard deviation: 1.46797671007. Interquartile range: 1. Contributor Stats ______________________ In the last 4 weeks, 20 unique ticket reporters 10 unique ticket reviewers 5 unique ticket resolvers In the last 24 weeks, 83 unique ticket reporters 14 unique ticket reviewers 17 unique ticket resolvers In the last 48 weeks, 156 unique ticket reporters 21 unique ticket reviewers 26 unique ticket resolvers

1 0

[Twisted-Python] scheduling timers with long interval
by Alec Matusis 17 Aug '07

17 Aug '07

I have a server that is running for weeks without restart, under significant load. I am considering adding a function to it that requires setting long timers like twisted.internet.reactor.callLater(604800, func) i.e. scheduling events one week ahead. There will be up to 500 timers active at any given time. I do not know how the scheduler works. Will having these timers with very long intervals significantly tax the performance, since the reactor will have to check on their status with every churn of the reactor loop? Thanks Alec

2 1

[Twisted-Python] scheduling timers with long intervals
by Alec Matusis 17 Aug '07

17 Aug '07

I have a server that is running for weeks without restart, under significant load. We are considering adding a function to it that requires setting long timers like twisted.internet.reactor.callLater(604800, func) i.e. scheduling events one week ahead. There will be up to 500 timers active at any given time. I do not know how the scheduler works. Will having these timers with very long intervals significantly tax the performance, since the reactor will have to check on their status with every churn of the reactor loop? Thanks Alec

1 0

[Twisted-Python] again deferToThreads
by Ladislav Andel 16 Aug '07

16 Aug '07

Hello, I have a two functions which I run in one thread called via deferToThread. There is function called siptest_f and function pingNode test. siptest_f returns a dictionary and pingNode returns the average time out of 3 received packets. Below is code I use. I run it through reactor event loop which is not shown here. My problem is that the function on its own returns correct results but if use it within following code I get always different results and mostly wrong. There should probably be some way of locking but I'm not sure how to do it correctly. If you have any other suggestions, there are welcomed. Lada ------------------------------------------------------------------------ def serverTest(dns_res, index): """ Function for SIP testing """ from twisted.internet import threads from siptest import siptest_f from icmp_ping2 import pingNode # import thread print "here will be siptest for ", dns_res #stdoutmutex = thread.allocate_lock() def test(dns_res, test_opts): if dns_res: icmpres = pingNode(alive=0, timeout=1.0, ipv6=0, number=3, node=dns_res, flood=0, size=56) sipres = siptest_f(dns_res, test_opts) # stdoutmutex.acquire() sipres['ICMP'] = icmpres # stdoutmutex.release() return sipres raise ValueError("Error occured") def sipResult(d): print d #return d def printError(failure): print failure sd = threads.deferToThread(test, dns_res, test_opts) sd.addCallback(sipResult) sd.addErrback(printError)

1 0

[Twisted-Python] Best way to write a modularized proxy?
by Andrew Warkentin 16 Aug '07

16 Aug '07

I am writing a modular multi-protocol proxy (initially for HTTP, and later HTTPS and possibly FTP as well) using Twisted. It will consist of a small core that handles initialization and loading of modules, a module for each protocol (which will consist of a part that accepts connections and a part that connects to the remote server), and several "filter" modules (which will be inserted in a "stack" between the two halves of the protocol module) for things like caching and filtering. Would there be a better way to structure it? Would the best way to handle the modules be to write a ServerFactory subclass that wraps the stack of modules? I understand the basic concepts behind Twisted, but am not totally familiar with the API, so I am not exactly sure what is the best way to do things.

2 2