Mailman 3 January 2007 - Twisted

[Twisted-Python] Watching a File Descriptor
by Justin Mazzola Paluska 21 Jan '07

21 Jan '07

Hi, I’m playing with the inotify(7) interface on Linux. When I call inotify_init(), I get back a file descriptor that I’m supposed read from when there are file system events. What’s the best way to integrate the inotify file descriptor into the Twisted reactor? My question has two subparts: 1. How do I tell Twisted to check my fd for readability? (Is reactor.addReader() all I need? 2. How do I register a callback to my own code when my fd is readable? All in all, it seems what I really want is something like ProcessProtocol, but without spawning a process. Forgive me if I’ve missed something obvious, but searching google and the gmane mail archives hasn’t turned up anything. Thanks, —Justin

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 21 Jan '07

21 Jan '07

Bug summary ______________________ Summary for 2007-01-14 through 2007-01-21 Bugs opened: 10 Bugs closed: 17 Total open bugs: 821 (-7) |== Type Changes |== Priority Changes |== Component Changes |Defect: -3 |Highest: -2 |Conch: -1 |Enhancement: -2 |High: +0 |Core: +0 |Task: -2 |Normal: -6 |News: -1 |Low: +1 |Pb: -1 |Release Management: -1 |Trial: +1 |Web: -2 |Website: -2 |Words: +0 New / Reopened Bugs ______________________ ===== Highest ===== [#2403] ITerminalTransport missing functions for setting and resetting private modes (opened by exarkun) (CLOSED, fixed) defect conch http://twistedmatrix.com/trac/ticket/2403 [#2408] Twisted Words 0.5.0 breaks import compatibility for interfaces without deprecation (opened by ralphm) defect words http://twistedmatrix.com/trac/ticket/2408 ===== High ===== [#2405] Various tests require stderr behavior CPython cannot and will not provide (opened by exarkun) defect core http://twistedmatrix.com/trac/ticket/2405 [#2409] twisted.plugin should be quiet when it can't write cache file (opened by jknight) defect core http://twistedmatrix.com/trac/ticket/2409 [#2410] setup.py should generate a twisted.plugin dropin.cache file during install. (opened by jknight) defect core http://twistedmatrix.com/trac/ticket/2410 ===== Normal ===== [#2404] DST calculation problem in log.py/FileLogObserver.getTimezoneOffset (opened by thor) defect core http://twistedmatrix.com/trac/ticket/2404 [#2407] FilePath.alwaysCreate should be removed (opened by jknight) enhancement core http://twistedmatrix.com/trac/ticket/2407 [#2411] Diverse improvements to IResult interface (opened by jml) enhancement trial http://twistedmatrix.com/trac/ticket/2411 [#2412] twisted hangs on infinite loop redirection (opened by Rick Graves) defect web http://twistedmatrix.com/trac/ticket/2412 ===== Low ===== [#2406] Location of basic pb tutorial not found until after monkeying with finger client tutorial (opened by arch_o_median) defect pb http://twistedmatrix.com/trac/ticket/2406 Closed Bugs ______________________ ===== Highest ===== [#1649] unused parameters in IRCClient.register (opened by synapsis, closed by exarkun, fixed) defect words http://twistedmatrix.com/trac/ticket/1649 [#2403] ITerminalTransport missing functions for setting and resetting private modes (opened by exarkun, closed by exarkun, fixed) defect conch http://twistedmatrix.com/trac/ticket/2403 [#1996] Remove repr() and getDelayedCalls() usage from twisted.web.test.test_web.SessionTest.test_delayedCallCleanup (opened by exarkun, closed by therve, fixed) task web http://twistedmatrix.com/trac/ticket/1996 [#2382] twisted/web/test/test_http.py uses backticks (opened by exarkun, closed by therve, fixed) defect web http://twistedmatrix.com/trac/ticket/2382 ===== High ===== [#2267] twisted.conch.test.test_conch tests leave the reactor unclean (opened by jml, closed by jml, fixed) defect conch http://twistedmatrix.com/trac/ticket/2267 [#1102] _Dereferences showing up in cyclic data structures with pb, jelly, and Cacheable (opened by phenyx, closed by warner, wontfix) defect pb http://twistedmatrix.com/trac/ticket/1102 [#1195] twisted.test.test_pb calls reactor.iterate() (opened by exarkun, closed by warner, fixed) defect core http://twistedmatrix.com/trac/ticket/1195 ===== Normal ===== [#1512] macosx & twisted - unable to find a reactor (opened by cripcrip, closed by exarkun, invalid) enhancement release management http://twistedmatrix.com/trac/ticket/1512 [#2370] Cannot create new milestones. (opened by ralphm, closed by exarkun, fixed) defect website http://twistedmatrix.com/trac/ticket/2370 [#1792] qt step of reactors build fails due to Xlib spew on stderr (opened by spiv, closed by exarkun, wontfix) defect core http://twistedmatrix.com/trac/ticket/1792 [#1972] Python 2.5 warnings mangle output from child processes in test (opened by exarkun, closed by exarkun, wontfix) task core http://twistedmatrix.com/trac/ticket/1972 [#1390] Intermitent failure in newpb unit tests (opened by exarkun, closed by warner, worksforme) defect pb http://twistedmatrix.com/trac/ticket/1390 [#1531] QtReactor Problems with Modal Dialogs (opened by mustafa, closed by exarkun, wontfix) defect core http://twistedmatrix.com/trac/ticket/1531 [#2037] twisted.python.reflect.namedAny depends on undocumented import implementation details (opened by exarkun, closed by exarkun, duplicate) defect core http://twistedmatrix.com/trac/ticket/2037 [#1819] give permission to update the website to the people who occasionally need to do it (opened by dreid, closed by dreid, invalid) enhancement website http://twistedmatrix.com/trac/ticket/1819 [#1090] twisted/web/server.py : Session expire times are hardwired (opened by loic, closed by therve, fixed) enhancement web http://twistedmatrix.com/trac/ticket/1090 [#2364] Release TwistedNews-0.3.0 (opened by radix, closed by exarkun, fixed) enhancement news http://twistedmatrix.com/trac/ticket/2364

1 0

[Twisted-Python] Serial port problem
by Matthias Urlichs 20 Jan '07

20 Jan '07

Hello, I'm writing to a serial port. That may of course stall. But what if a delayed write results in an error? I didn't see any obvious way to catch that. As an aside, the error whioch prompts me to write this message was EINTR, caused by a child process dying at the right moment. Shouldn't that be retried, like EAGAIN? And shouldn't serial ports be set to be non-blocking? (Linux, Twisted 2.4) -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf(a)smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - Saints should always be judged guilty until they are proven innocent. - George Orwell

1 0

[Twisted-Python] Re: Hanging Deferreds in PB Paging code
by Ed Suominen 20 Jan '07

20 Jan '07

From: "Brian Granger" <ellisonbg.net(a)gmail.com> > We are using PB as an initial protocol for some IPython related > stuff. Overall, PB is working well, but we need to be able to send > larger things around so we have been trying to implement things using > the pb.util.Pager stuff. I have spent a fair amount of time > understanding how the Paging works. I looked at the PB paging stuff and decided to write my own simple solution for it, borrowing the original code as needed. At the server, just pass one of these to your PB client: > class Receiver(pb.Referenceable): > """ > Give a PB client a remote reference to an instance of me and it will > have a way to upload a file in chunks of whatever size it likes. > """ > def __init__(self, destPath): > self.file = open(destPath, 'w') > > def remote_chunk(self, data): > self.file.write(data) > return True > > def remote_done(self, ok): > self.file.close() At the client side, use the source path for the file and the reference to the Receiver object to construct an Uploader object: > from twisted.internet import defer, interfaces > from twisted.python.filepath import FilePath > > class Uploader(object): > """ > I upload the file at the specified I{sourcePath} to the I{referenced} PB > referenceable object, both of which are supplied to my constructor. > > @ivar d: A deferred that fires when the file has been completely uploaded. > > """ > implements(interfaces.IConsumer) > > def __init__(self, sourcePath, referenced): > self.ref = referenced > # The source file > fp = FilePath(sourcePath) > if not fp.isfile(): > raise OSError("'%s' is not a valid file path" % result) > self.basename = fp.basename() > self.bytes = [0, fp.getsize()] > self.fh = fp.open() > # Pull producer for reading the file locally > self.producer = FileSender() > # Set up the deferred that fires when the upload is done > d = self.d = defer.Deferred() > d.addCallback(lambda ok: self.ref.callRemote('done', ok)) > # Start the transfer at the file producer > self.producer.beginProducing(self.fh, self) > > def cancel(self): > if not self.d.called: > self.d.callback(False) > self.producer.stopProducing() > > #--- IConsumer API -------------------------------------------------------- > > def registerProducer(self, producer, streaming): > self.producer = producer > if not streaming: > self.producer.resumeProducing() > > def unregisterProducer(self): > if not self.d.called: > self.d.callback(True) > > def write(self, data): > """ > Writes the supplied chunk of file I{data}, no larger than > L{FileSender.CHUNK_SIZE} bytes, to the server using the referenceable > I've been given. > """ > def oops(failure): > failure.printTraceback() > self.cancel() > > def update(sent): > sent = self.bytes[0] + sent > self.p.setValue(sent) > self.bytes[0] = sent > > d = self.ref.callRemote('chunk', data) > d.addCallback(lambda _: update(len(data))) > d.addCallback(lambda _: self.producer.resumeProducing()) > d.addErrback(oops) I excerpted this from code that queues up the remote calls via my asynchronous task queue, and I'm not sure whether it will work well without it. The taskqueue subpackage is part of Twisted-Goodies, see http://foss.eepatents.com/Twisted-Goodies. Anyhow, given how much it borrows from Twisted's own code, I hereby license this code snippet for use under the same MIT-like license that Twisted itself uses. Best regards, Ed

2 1

[Twisted-Python] Authentication with multiple handshakes
by Alvin Wang 19 Jan '07

19 Jan '07

http://twistedmatrix.com/projects/core/documentation/howto/pb-cred.html The documentation above says that credentials should be able to do authentication with multiple passes. However, I have not been able to find any examples of it. As an alternative, I was going to implement a user object with state that determined what it was able to do. I could force the client to conduct multiple challenge responses to achieve the logged in state. I figured it would be better to ask the mail list for the proper way to do it first. Thanks -- Alvin

5 8

[Twisted-Python] trac / website woes
by Matthias Urlichs 19 Jan '07

19 Jan '07

I hate Trac. Especially the one on twistedmatrix. - My previous login ("smurfix") seems to suffer from a lost password. The website does not have an obvious place to request it. - I tried to create a new login. Reporting a bug with it requires an email address. The page to enter the email address does not have a Submit button. - Not entering login data results in an empty page. On to the real bug; somebody else can enter it into Trac. Or, even better, fix the source. It seems that Deferreds are tricky. --- /usr/lib/python2.4/site-packages/twisted/spread/pb.py 2006-05-12 15:50:13.000000000 +0200 +++ /tmp/pb.py 2007-01-13 18:48:50.508925924 +0100 @@ -651,6 +651,9 @@ def notifyOnDisconnect(self, notifier): """Call the given callback when the Broker disconnects.""" assert callable(notifier) - self.disconnects.append(notifier) + if self.disconnected: + notifier() + else: + self.disconnects.append(notifier) def notifyOnFail(self, notifier): ... as exemplified by this traceback: 2007/01/13 18:26 CET [Broker,0,127.0.0.1] Error: <twisted.python.failure.Failure twisted.spread.pb.PBConnectionLost> 2007/01/13 18:26 CET [Broker,0,127.0.0.1] [Failure instance: Traceback (failure with no frames): twisted.spread.pb.PBConnectionLost: [Failure instance: Traceback (failure with no frames): twisted.internet.error.ConnectionDone: Connection was closed cleanly. 2007/01/13 18:26 CET [Broker,0,127.0.0.1] ] 2007/01/13 18:26 CET [Broker,0,127.0.0.1] ] 2007/01/13 18:27 CET [Broker,1,127.0.0.1] Looking for 00:11:d8:54:e6:2d 2007/01/13 18:27 CET [-] Peer will receive following PB traceback: 2007/01/13 18:27 CET [-] Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 226, in mainLoop self.runUntilCurrent() File "/usr/lib/python2.4/site-packages/twisted/internet/base.py", line 534, in runUntilCurrent f(*a, **kw) File "/usr/lib/python2.4/site-packages/twisted/internet/defer.py", line 229, in callback self._startRunCallbacks(result) File "/usr/lib/python2.4/site-packages/twisted/internet/defer.py", line 294, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "/usr/lib/python2.4/site-packages/twisted/internet/defer.py", line 307, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/spread/pb.py", line 1747, in _loggedIn self.portalWrapper.broker.notifyOnDisconnect(logout) File "/usr/lib/python2.4/site-packages/twisted/spread/pb.py", line 654, in notifyOnDisconnect self.disconnects.append(notifier) exceptions.AttributeError: 'NoneType' object has no attribute 'append' -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf(a)smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - <dracus> Ctrl+Option+Command + P + R <Knghtbrd> dracus - YE GODS! That's worse than EMACS! <LauraDax> hehehehe <dracus> don't ask what that does :P

2 3

RE: [Twisted-Python] Server authentication
by Martin Evans 18 Jan '07

18 Jan '07

Many thanks to Volodymyr and Lee for the authentication pointers, both the ldap and win32security modules look like suitable candidates. At the end of the day it is whatever the client decides so it is great to have these alternatives. Martin

3 2

[Twisted-Python] Twisted sprint at SuperHappyDevHouse15?
by David Reid 18 Jan '07

18 Jan '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Is anyone in the bay area (or willing to come to the bay area) interested in sprinting on Twisted at Super Happy Dev House 15? - -David Begin forwarded message: > From: tomicles(a)gmail.com > Date: January 17, 2007 10:16:48 PM PST > To: "SuperHappyDevHouse Announcements" <SHDH- > Announce(a)googlegroups.com> > Subject: [SHDH] SuperHappyDevHouse15! > Reply-To: tomicles(a)gmail.com > List-Id: <SHDH-Announce.googlegroups.com> > > > You're Invited! SuperHappyDevHouse15 has been scheduled and will take > place at our south bay location in Los Gatos on February 3rd from > 1pm ~ > 2am. Bring your laptop, your hacker friends and some snacks or drinks > to share. More information can be found on our site and you can RSVP > on upcoming, both linked below. > > http://shdh.org > http://upcoming.org/event/141406/ > > -The Dev House Team > > > --~--~---------~--~----~------------~-------~--~----~ > You received this message because you are subscribed to the Google > Groups "SuperHappyDevHouse Announcements" group. > To post to this group, send email to SHDH-Announce(a)googlegroups.com > To unsubscribe from this group, send email to SHDH-Announce- > unsubscribe(a)googlegroups.com > For more options, visit this group at http://groups.google.com/ > group/SHDH-Announce?hl=en > -~----------~----~----~----~------~----~------~--~--- > David Reid http://dreid.org/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iD8DBQFFr76arsrO6aeULcgRAkX+AKCex6BFZ5ChywpbFDiaA2mDxR4A0gCbBg+0 02YgFdEiJIUuvj28ULnMbSk= =VxTs -----END PGP SIGNATURE-----

1 0

Re: [Twisted-Python] client, deferred and/or reactor hung on these 2 URL's
by Rick Graves 18 Jan '07

18 Jan '07

Eric, I ran wget on the 2 URL's, and what I got may lead to the root cause of the problem. Here is the end of what appeared in the terminal window: begin quote --11:56:34-- http://www.theproxy.be/link.php => `link.php' Connecting to www.theproxy.be|209.67.219.74|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: http://www.theproxy.be/link.php [following] 20 redirections exceeded. end quote I assume this is a case of endless loop redirection. I got this for both URL's. The complete transcript for both is here: http://www.advanced-app.com.hk/MiscJunk/wget.txt I am running kubuntu, "codenamed 'Dapper Drake'" ver. 6.06. For this version, I get ethereal not WireShark. I installed ethereal before running wget, but I am wondering whether wget might zero in on the problem. So I hope to find out before spending some time trying to figure ethereal out. Also, I have ADSL service from different ISP's at home and at the office. These two URL's hang for me at the office, but work OK for me at home. (They would be going through different ISP's and different proxies.) I have not filed a bug report yet. If the wget output does in fact zero in on the problem, I will do so right away. This is a resend -- see below. I sent this content on Sunday, but it never came back to me on the mailing list. Thanks, Rick --- Rick Graves <gravesricharde(a)yahoo.com> wrote: > Date: Sun, 14 Jan 2007 21:12:55 -0800 (PST) > From: Rick Graves <gravesricharde(a)yahoo.com> > Subject: Re: [Twisted-Python] client, deferred > and/or reactor hung on these 2 URL's > To: twisted-python(a)twistedmatrix.com > > Eric, > > I ran wget on the 2 URL's, and what I got may lead > to > the root cause of the problem. > {snip}

1 0

Re: [Twisted-Python] Twisted server is 5 times SLOWER on Solaris than Linux?
by glyph＠divmod.com 18 Jan '07

18 Jan '07

On 11:35 pm, jarrod(a)vertigrated.com wrote: >There is a "backend" C module that our Twisted server front ends, and it is >highly multi-threaded. >So the T1000 is PERFECT for our application, except that now Twisted is the >bottleneck. :-( This seems odd to me. If all the CPUs are going to be busy doing a multi-threaded back-end's work, and Twisted is just doing the I/O, then it seems the T1000 would still be a benefit. The benchmark you mentioned was completely static; there was no backend library, no multithreaded CPU load. Is the performance disparity similar when you're running actual workloads? Sure, Twisted isn't going to be able to dole out as much work as something optimized to balance the I/O management CPU across N+1 cores; but if those cores are going to be busy anyway in realistic use, then presumably having Twisted contending for all of them wouldn't be much of a performance boost. I actually do have a little experience with Twisted-*like* software on Solaris, although not Twisted itself. The proprietary system which originally inspired Twisted's networking core was actually designed to run on Solaris, and took Sparc hardware advantages into account. It still ran all of its I/O in a single thread. >So we either scrap our Twisted implementation and have to spend extra time >on another network handling layer, or run 5 times as many instances of our >server to service the same number of concurrent clients. Congratulations. For years, I've been warning people that Twisted cannot transparently take full advantage of vertical scaling with SMP. While I've heard a lot of uninformed whinging about how this is a huge problem, you are the first person to report an actual performance problem related to that fact :). Running 5 times as many instances of the server does make sense, and shouldn't have a significant downside. The parallelism strategy I've used pretty much everywhere is multiprocessing rather than multithreading, and it works well. If the issue is that you don't want to have that many different open ports on each machine, would it be possible to have a small front-end server accept()ing and sending sockets to N+1 (where N is the number of cores) other Twisted processes? I don't know how this might be accomplished on solaris, but if it's possible, it should be transparent to the clients and let Twisted itself take advantage of the hardware. It would take some work, but not as much as a rewrite. Again, it seems weird to me that this is necessary if the back-end library is really utilizing all the CPUs already and you are not I/O bound.

2 1