Mailman 3 August 2005 - Twisted

Re: [Twisted-Python] t.trial.remote -- does anyone use it?
by Brian Warner 26 Aug '05

26 Aug '05

> Trial currently does some black magic to support remote reporters, > defined in t.trial.remote. What use cases does t.trial.remote > implement? Does anyone actually use it? Not me. > Rumour has it that it was originally written to make buildbot easier > to write. However, I'm told that buildbot just parses stdio output. Both rumors are correct. The buildbot currently just captures stdout/stderr and parses the last few lines for the test-count summaries. The original idea was to make a PB connection from the trial process to the parent buildbot slave, and send fine-grained test results over that connection. To accomplish this, we (slyphon and I) were planning to write a Reporter class that would actually be distributed with the buildbot, then use a trial option like --reporter-class=buildbot.trialreporter.RemoteReporter or something. Another direction was to use PB running over stdin/stdout, using the same scheme. The goal was to get a tuple of (testname, results, exceptions, log.msg output, stdout, stderr, starttime, finishtime) for each test, over on the buildslave, so that the buildbot status reporting could give extremely fine-grained information about the tests which failed (perhaps including full stack traces with local variables for the ERROR cases). The most important thing to me is that we can get a list of test names and their individual results, but having the trial.log text for each one separately would make debugging test failures a lot easier (and the buildbot could arrange to just mail the relevant parts of the logs to the person responsible for that test, instead of sending the whole multi-megabyte logfile). If I remember correctly, the extra Reporter functionality was there to give buildbot enough feedback about tests starting/stopping to provide useful status information (specifically to accurately update the ETA timing info: it could eventually figure out that 45 tests out of 90 total means that it's N% complete and therefore has an ETA of 2 minutes, etc). It's been a while since I looked at the code, though. I gave up on it at some point, mainly because the reactor cleanup routines were getting in the way. (somewhat embarrassing because I was the one who put many of those cleanup routines in there in the first place). Basically the PB connection that was established by our RemoteReporter was torn down with a vengeance by the clean-up-all-Selectables pass, flunking the first test and condemning the rest to be run with no hope of ever delivering their status to the buildslave. I decided to put off the whole clever machine-parseable status thing until the trial's reactor cleanup had a way to let specific selectables stick around. So if that code is in the way, don't keep it around on account of the buildbot. Having a pluggable reporter will be handy once reactor cleanup is more convenient, but as long as that feature is retained in some form I think we'll be able to figure out a way to get the status information to the buildslave later. cheers, -Brian PS: another approach (implemented in trial at some point, I think, but I don't believe I ever got buildbot to use it) was to jelly some status objects and fling them out stdout in the hopes of being picked up by whoever launched trial. This might have worked better, because one-way jelly over stdout wouldn't have involved a Selectable (unlike two-way PB over stdio), so the cleanup might not have stomped on it. However, I was uncomfortable with the potential version skew of the code creating these jellied objects (coming from twisted.trial) and the code that was hoping to interpret them (coming from buildbot). The pluggable Reporter allowed the buildbot to supply the code on both sides of the wire, which struck me as a better long-term solution.

2 1

[Twisted-Python] How to "monitor" parallel executions of Deferreds
by Pedro Sanchez 26 Aug '05

26 Aug '05

Hello, I need to implement a function (say, myfunc) that fires several independent Deferreds (using deferToThread), and that returns a deferred itself. A client of this function (say, client) is expected to do something like this: def client(): def done(success) if success: print 'd1 and d2 are done' d = myfunc() d.addCallback(done) ... myfuct() above does something like this: def myfunc(): def checkFunc1(success): ... def checkFunc2(success): ... d = defer.Deferred() d1 = deferToThread(func1).addCallback(checkFunc1) d2 = deferToThread(func2).addCallback(checkFunc2) d.addCallback(????) return d Clearly d, d1, and d2 in the above are disconnected. And I don't want to serialize d1 and d2 which would be a way to connect them all. The problem is I don't know how d should be defined. The idea is that the 'done' function in the client function is only called when both threads d1 and d2 are done. Can you give me any suggestions on how to implement myfunc()? Thank you, -- Pedro

5 5

[Twisted-Python] How to forcibly disconnect PB client?
by Steve Freitas 25 Aug '05

25 Aug '05

Hi, In some cases I need to force a client to disconnect. Is there some factory call that I can use to do this? On the client side, I know I can use factory._broker.transport.loseConnection(), but on the server side, though I've looked at the source a bit, I'm not sure how to get at the correct transport. Thanks! Steve

4 8

[Twisted-Python] t.trial.remote -- does anyone use it?
by Jonathan Lange 25 Aug '05

25 Aug '05

Hello, Trial currently does some black magic to support remote reporters, defined in t.trial.remote. What use cases does t.trial.remote implement? Does anyone actually use it? Rumour has it that it was originally written to make buildbot easier to write. However, I'm told that buildbot just parses stdio output. Let me know if you use it. If you don't, I'll remove the code (in my branch) at my next hacking session :) cheers, jml

3 3

[Twisted-Python] Re: URL length limits in twisted [wrong list]
by Lenny G Arbage 24 Aug '05

24 Aug '05

Argh. Sorry, thought I was posting to twisted-web. Please forgive and ignore. - Lenny ____________________________________________________ Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs

1 0

[Twisted-Python] URL length limits in twisted
by Lenny G Arbage 24 Aug '05

24 Aug '05

I'm noticing silent failures using HTTPClientFactory with len(url) >= 16400. The factory's deferred object immediately calls its associated callBack() method without sending any data (the server is likewise written with twisted python, but it doesn't appear to ever receive data). The errorBack() doesn't fire, either. A url with len <= 16398 works just fine, firing the callback /after/ sending the data. According to RFC 2616, arbitrary length URLs are allowed by HTTP, but I understand that there are practical limitations in many implementations. Is the 16400 url-length ceiling a known limitation in twisted, or am I banging up against something else? thanks, Lenny ____________________________________________________ Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs

2 1

RE: [Twisted-Python] CannotListenError
by Henning.Ramm＠mediapro-gmbh.de 24 Aug '05

24 Aug '05

>> I would like, but I cannot - I got no mail regarding my login. >> Must someone handle that manually? >Probably not; have you tried just logging in with the username and >password you gave? That didn't work. I registered again, and it worked. Issue opened. Thank you! Best regards, Henning Hraban Ramm Südkurier Medienhaus / MediaPro Support/Admin/Development Dept.

1 0

RE: [Twisted-Python] CannotListenError
by Henning.Ramm＠mediapro-gmbh.de 24 Aug '05

24 Aug '05

>provides. Open a bug in http://twistedmatrix.com/bugs/ and assign it to >me. I would like, but I cannot - I got no mail regarding my login. Must someone handle that manually? HR

2 1

[Twisted-Python] Why is Int23StringReceiver so slow?
by Brian Granger 24 Aug '05

24 Aug '05

Hello,, I am designing and implementing a custom protocol using Twisted. I have wanted to design the protocol to use int 32 prefixed strings. I wrote a test version of my server using Twisted's Int32StringReceiver protocol class but it seemed really slow. To test this out I have written and tested two echo servers: one that uses LineReceiver and one that uses Int32StringReceiver. I have benchmarked both of these echo servers by making multiple connections to each. I find that the LineReceiver based echo server is about 200 times as fast as the Int32StringReceiver! Is the Int32StringReceiver really that slow? Why? What am I missing? Are there hidden assumptions that I am missing? Can it be made faster? Thanks in advance! Brian

2 2

[Twisted-Python] CannotListenError
by Henning.Ramm＠mediapro-gmbh.de 24 Aug '05

24 Aug '05

If my server app crashes, it often leaves its ports open, so that I get a CannotListenError 'Address already in use'. (I'm on Win2k.) Is there any possibility to set this address free other than re-logon the user? Best regards, Henning Hraban Ramm Südkurier Medienhaus / MediaPro Support/Admin/Development Dept.

2 1