Mailman 3 January 2007 - Twisted

[Twisted-Python] trac / website woes
by Matthias Urlichs Jan. 19, 2007

Jan. 19, 2007

I hate Trac. Especially the one on twistedmatrix. - My previous login ("smurfix") seems to suffer from a lost password. The website does not have an obvious place to request it. - I tried to create a new login. Reporting a bug with it requires an email address. The page to enter the email address does not have a Submit button. - Not entering login data results in an empty page. On to the real bug; somebody else can enter it into Trac. Or, even better, fix the source. It seems that … [View More]Deferreds are tricky. --- /usr/lib/python2.4/site-packages/twisted/spread/pb.py 2006-05-12 15:50:13.000000000 +0200 +++ /tmp/pb.py 2007-01-13 18:48:50.508925924 +0100 @@ -651,6 +651,9 @@ def notifyOnDisconnect(self, notifier): """Call the given callback when the Broker disconnects.""" assert callable(notifier) - self.disconnects.append(notifier) + if self.disconnected: + notifier() + else: + self.disconnects.append(notifier) def notifyOnFail(self, notifier): ... as exemplified by this traceback: 2007/01/13 18:26 CET [Broker,0,127.0.0.1] Error: <twisted.python.failure.Failure twisted.spread.pb.PBConnectionLost> 2007/01/13 18:26 CET [Broker,0,127.0.0.1] [Failure instance: Traceback (failure with no frames): twisted.spread.pb.PBConnectionLost: [Failure instance: Traceback (failure with no frames): twisted.internet.error.ConnectionDone: Connection was closed cleanly. 2007/01/13 18:26 CET [Broker,0,127.0.0.1] ] 2007/01/13 18:26 CET [Broker,0,127.0.0.1] ] 2007/01/13 18:27 CET [Broker,1,127.0.0.1] Looking for 00:11:d8:54:e6:2d 2007/01/13 18:27 CET [-] Peer will receive following PB traceback: 2007/01/13 18:27 CET [-] Traceback (most recent call last): File "/usr/lib/python2.4/site-packages/twisted/internet/posixbase.py", line 226, in mainLoop self.runUntilCurrent() File "/usr/lib/python2.4/site-packages/twisted/internet/base.py", line 534, in runUntilCurrent f(*a, **kw) File "/usr/lib/python2.4/site-packages/twisted/internet/defer.py", line 229, in callback self._startRunCallbacks(result) File "/usr/lib/python2.4/site-packages/twisted/internet/defer.py", line 294, in _startRunCallbacks self._runCallbacks() --- <exception caught here> --- File "/usr/lib/python2.4/site-packages/twisted/internet/defer.py", line 307, in _runCallbacks self.result = callback(self.result, *args, **kw) File "/usr/lib/python2.4/site-packages/twisted/spread/pb.py", line 1747, in _loggedIn self.portalWrapper.broker.notifyOnDisconnect(logout) File "/usr/lib/python2.4/site-packages/twisted/spread/pb.py", line 654, in notifyOnDisconnect self.disconnects.append(notifier) exceptions.AttributeError: 'NoneType' object has no attribute 'append' -- Matthias Urlichs | {M:U} IT Design @ m-u-it.de | smurf(a)smurf.noris.de Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de - - <dracus> Ctrl+Option+Command + P + R <Knghtbrd> dracus - YE GODS! That's worse than EMACS! <LauraDax> hehehehe <dracus> don't ask what that does :P [View Less]

2 3

RE: [Twisted-Python] Server authentication
by Martin Evans Jan. 18, 2007

Jan. 18, 2007

Many thanks to Volodymyr and Lee for the authentication pointers, both the ldap and win32security modules look like suitable candidates. At the end of the day it is whatever the client decides so it is great to have these alternatives. Martin

3 2

[Twisted-Python] Twisted sprint at SuperHappyDevHouse15?
by David Reid Jan. 18, 2007

Jan. 18, 2007

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Is anyone in the bay area (or willing to come to the bay area) interested in sprinting on Twisted at Super Happy Dev House 15? - -David Begin forwarded message: > From: tomicles(a)gmail.com > Date: January 17, 2007 10:16:48 PM PST > To: "SuperHappyDevHouse Announcements" <SHDH- > Announce(a)googlegroups.com> > Subject: [SHDH] SuperHappyDevHouse15! > Reply-To: tomicles(a)gmail.com > List-Id: <SHDH-Announce.… [View More]

1 0

Re: [Twisted-Python] client, deferred and/or reactor hung on these 2 URL's
by Rick Graves Jan. 18, 2007

Jan. 18, 2007

Eric, I ran wget on the 2 URL's, and what I got may lead to the root cause of the problem. Here is the end of what appeared in the terminal window: begin quote --11:56:34-- http://www.theproxy.be/link.php => `link.php' Connecting to www.theproxy.be|209.67.219.74|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: http://www.theproxy.be/link.php [following] 20 redirections exceeded. end quote I assume this is a case of endless loop … [View More]

1 0

Re: [Twisted-Python] Twisted server is 5 times SLOWER on Solaris than Linux?
by glyph＠divmod.com Jan. 18, 2007

Jan. 18, 2007

On 11:35 pm, jarrod(a)vertigrated.com wrote: >There is a "backend" C module that our Twisted server front ends, and it is >highly multi-threaded. >So the T1000 is PERFECT for our application, except that now Twisted is the >bottleneck. :-( This seems odd to me. If all the CPUs are going to be busy doing a multi-threaded back-end's work, and Twisted is just doing the I/O, then it seems the T1000 would still be a benefit. The benchmark you mentioned was completely static; there … [View More]was no backend library, no multithreaded CPU load. Is the performance disparity similar when you're running actual workloads? Sure, Twisted isn't going to be able to dole out as much work as something optimized to balance the I/O management CPU across N+1 cores; but if those cores are going to be busy anyway in realistic use, then presumably having Twisted contending for all of them wouldn't be much of a performance boost. I actually do have a little experience with Twisted-*like* software on Solaris, although not Twisted itself. The proprietary system which originally inspired Twisted's networking core was actually designed to run on Solaris, and took Sparc hardware advantages into account. It still ran all of its I/O in a single thread. >So we either scrap our Twisted implementation and have to spend extra time >on another network handling layer, or run 5 times as many instances of our >server to service the same number of concurrent clients. Congratulations. For years, I've been warning people that Twisted cannot transparently take full advantage of vertical scaling with SMP. While I've heard a lot of uninformed whinging about how this is a huge problem, you are the first person to report an actual performance problem related to that fact :). Running 5 times as many instances of the server does make sense, and shouldn't have a significant downside. The parallelism strategy I've used pretty much everywhere is multiprocessing rather than multithreading, and it works well. If the issue is that you don't want to have that many different open ports on each machine, would it be possible to have a small front-end server accept()ing and sending sockets to N+1 (where N is the number of cores) other Twisted processes? I don't know how this might be accomplished on solaris, but if it's possible, it should be transparent to the clients and let Twisted itself take advantage of the hardware. It would take some work, but not as much as a rewrite. Again, it seems weird to me that this is necessary if the back-end library is really utilizing all the CPUs already and you are not I/O bound. [View Less]

2 1

[Twisted-Python] Twisted server is 5 times SLOWER on Solaris than Linux?
by Jarrod Roberson Jan. 17, 2007

Jan. 17, 2007

I we have a basic twisted server that we have doing nothing but dropping received bits on the floor, and sending back a static response. We have tested it on a Solaris 10 on a Sun T1000 ( 8 cores ) and on Solaris 8 on a Sun V210. We did initial load testing on a Dell 2850 ( 2 Hyperthreaded P4's ) using Red Hat WS 3 (Taroon Update 4) / ( Linux 2.4.21-27.ELsmp ) The Solaris boxes are all showing latency 5 times what we are seeing on the Dell 2850 with 1/5 as many clients to boot. On the Dell … [View More]

4 6

[Twisted-Python] Implementing parseargs or getopt using twisted trial
by Vishal Sethia Jan. 17, 2007

Jan. 17, 2007

Hi, How do I implement user defined command line options in addition to the default switches that are available while running twisted trial. For ex. if I have my own py file which I execute using the python compiler has the following line of code to support several command line options, how do I achieve this using twisted trial For eg, for o,a in opts: if o == "-d": debug = 1 if o in ["-h","--help"]: Usage() sys.exit(2) if o in [… [View More]

2 1

[Twisted-Python] Server authentication
by Martin Evans Jan. 17, 2007

Jan. 17, 2007

I'm currently implementing a proprietary set of protocols for interacting with twisted running on a Microsoft server. All is working fine so far using the basic line receiver / raw methods. I was hoping to add a form of authentication to the protocol. Is there anyway to interact with the existing accounts on the server? The client wishes to make use of some of the lock down facilities based on the Microsoft Server accounts. ie in the same way MS Telnet server allows you to logon and then … [View More]

3 2

[Twisted-Python] Hanging Deferreds in PB Paging code
by Brian Granger Jan. 17, 2007

Jan. 17, 2007

Hi, We are using PB as an initial protocol for some IPython related stuff. Overall, PB is working well, but we need to be able to send larger things around so we have been trying to implement things using the pb.util.Pager stuff. I have spent a fair amount of time understanding how the Paging works. Now the fun part. I have have two versions of our methods - one that uses Paging and one that doesn't. I am using trial to test these methods. The test passes for the non-Paging version, but … [View More]not for the Paging version. The problem with the Paging version it seems is that there are still unanswered PB requests (and associated Deferreds) that haven't completed when my tearDown method is run. Thus I see errors like: =============================================================================== [ERROR]: ipython1.test.test_enginepb.EnginePBTest.testSimplePushPull Traceback (most recent call last): Failure: twisted.spread.pb.PBConnectionLost: [Failure instance: Traceback (failure with no frames): twisted.internet.error.ConnectionDone: Connection was closed cleanly. which indicates that the connection was closed before all PB requests had finished. We have written lots of unittests using trial so we are _very_ used to making sure our Deferreds are cleaned up properly in tests. It appears that the problem is coming from this code in twisted.pb: def resumeProducing(self): """Called when the consumer attached to me runs out of buffer. """ # Go backwards over the list so we can remove indexes from it as we go for pageridx in xrange(len(self.pageProducers)-1, -1, -1): pager = self.pageProducers[pageridx] pager.sendNextPage() if not pager.stillPaging(): del self.pageProducers[pageridx] if not self.pageProducers: self.transport.unregisterProducer() Both the pager.sendNextPage and pager.stillPaging calls invoke PB calls to the other side. But, notice that the Deferreds for these calls are not dealt with in any way. Thus there is no promise that these PB requests have finished by the time the actual paged data has been received and passed down the callback chain the completes the Paged send (and test). My overall feeling is that the Paging mechanism in PB in not used that much and that I am likely running into uncharted territory/problems with the underlying implementation. We are trying to decide if we should continue to struggle with this or just move on to using a different protocol that is better suited to streaming large objects (such as http/1.1). Any thoughts on this dillema? Thanks Brian Granger [View Less]

1 0

[Twisted-Python] How can I mix the proxy function into a special protocol
by slowtech chen Jan. 16, 2007

Jan. 16, 2007

Hi guys, I tried to do this with multiple inheritances: class MyProxyServer(basic.Int32StringReceiver, ProxyServer): def dataReceived(self, data): basic.Int32StringReceiver.dataReceived(self, data) ProxyServer.dataReceived(self, data) but I got the following errors: self.peer.transport.write(data) exceptions.AttributeError: 'NoneType' object has no attribute 'transport' why the setPeer method of ProxyServer isn't called here? Are there any better … [View More]

2 1

[Twisted-Python] test maillist
by 铃音 Jan. 15, 2007

Jan. 15, 2007

twisted-python，您好！ test,test.. 　　　　　　　　致礼！　　　　　　　　铃音　　　　　　　　6b6(a)21cn.com 　　　　　　　　　　2007-01-16

1 0

[Twisted-Python] JOB OPPORTUNITY - ApplianSys - Coventry (UK)
by Richard Wall Jan. 15, 2007

Jan. 15, 2007

I am a Twisted enthusiast and support engineer working for a company called ApplianSys based in Coventry (UK), we develop *nix based hardware appliances. http://www.appliansys.com/ This year we are planning a rapid expansion of our product range as well as updating our underlying software and hardware platform. As such, we need a Lead Developer and three Product Developers all of whom must be able to demonstrate a high level of competency in Python. http://www.appliansys.com/company/… [View More]

1 0

[Twisted-Python] poll reactor question
by Christopher Zorn Jan. 15, 2007

Jan. 15, 2007

Hello, When I use the poll reactor I get the following traceback every now and then : Traceback (most recent call last): File "/usr/lib/python2.3/site-packages/twisted/application/app.py", line 278, in run runApp(config) File "/usr/lib/python2.3/site-packages/twisted/scripts/twistd.py", line 197, in runApp app.runReactorWithLogging(config, oldstdout, oldstderr) File "/usr/lib/python2.3/site-packages/twisted/application/app.py", line 131, … [View More]

3 3

Re: [Twisted-Python] client, deferred and/or reactor hung on these 2 URL's
by Rick Graves Jan. 15, 2007

Jan. 15, 2007

Eric, I ran wget on the 2 URL's, and what I got may lead to the root cause of the problem. Here is the end of what appeared in the terminal window: begin quote --11:56:34-- http://www.theproxy.be/link.php => `link.php' Connecting to www.theproxy.be|209.67.219.74|:80... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: http://www.theproxy.be/link.php [following] 20 redirections exceeded. end quote I assume this is a case of endless loop … [View More]

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Jan. 14, 2007

Jan. 14, 2007

1 0

Re: [Twisted-Python] client, deferred and/or reactor hung on these 2 URL's
by Rick Graves Jan. 14, 2007

Jan. 14, 2007

Eric, I ran the same file of almost 850 URL's again today, and again today, in the end the program was hung on the same two URL's. I tried the URL's with a program like WebCat.py from the Twisted book, which I also call WebCat.py. Like the one in the book, it uses client.getPage. It is different from the one in the book as follows: 1) The Errorback in the book did not work for me, so I wrote a simple one, and 2) I do not need to print the text of the page on the screen, my WebCat.py … [View More]

2 1

Re: [Twisted-Python] Ticket Hall of Shame - Current Loser: Brian Warner
by glyph＠divmod.com Jan. 14, 2007

Jan. 14, 2007

On 04:01 pm, jacob(a)internet24.de wrote: >On Fri, 2007-01-12 at 13:14 +0000, glyph(a)divmod.com wrote: >> Although we should all be happy for the Twisted 2.5 release, we still >> have an unfortunate track-record as far as responsiveness to bugs. >A little bit more flexibility on what patches are added might help. If nobody has said anything about a ticket in 2 years with no patches on it, I don't see how accepting crappy patches is going to help. By "responsiveness" I … [View More]meant responding to people who report bugs, and being clear on what we are vs. are not working on. The problem I am trying to address is best described here: http://www.jwz.org/doc/cadt.html By highlighting the least responsive developer (who, by the way, is Brian Warner) I hope to encourage people to keep their tickets up to date and respond when users report issues, even if the code in question has been rewritten. >If somebody posts a patch to a problem that is not yet >covered by a unit test (so that would be most patches) >then not absolutely insisting that the patcher provides a unit test as >well >could speed up things considerably. All the evidence I've seen (although I admit, there's not much of it) suggests that the opposite is true. Bugs have been fixed faster and better since the institution of these requirements. Most patches that get submitted without tests address a new ticket that was created specifically to hold the patch. Very few outside contributors seem to be interested in trying to fix _known_ problems with Twisted. Either they want a new feature or they want a new problem they've just discovered fixed. Those outside contributors who do contribute to existing tickets have generally become involved with the project enough to know that they should be developing fixes test-first. >If the developer responsible for >that module thinks the patch is plausible and possibly covers a way to >use twisted which >probably couldn't have worked anyway because of the reported problem, >there >is not all that much risk, is there? After all, the existing tests >should bark if the patch breaks anything else. As Mr. Armstrong already said, this idea doesn't make any sense. If anything, we need to require tests for existing, old functionality that wasn't completely tested before the patch was applied. >Twisted is really a very cool and effective tool, and that it can >actually be used in a stable production environment is definitely not >something I >want to miss, but maybe that is not so much due to a strict development >policy >as it is due to the skills of the core developers in spotting bogus >changes ;) How do you think the core developers got to be skilled in this way? By writing patches that don't have any tests or docs in them? :) >Just a thought... Thanks for trying to help with the suggestion. However, I can guarantee that as long as I am involved with the project, these requirements will continue to stay in place. You can continue talking about it if you like, but it will not change. >Maybe I got the wrong impression on just how things are done >by the twisted team. You seem to have a roughly correct impression. Tests and documentation are required before a patch will be accepted. >BTW, the patch in ticket #1794 is already included in 2.5, it probably >shouldn't be an active ticket anymore. Thanks for the heads up - it looks like Thomas Herve closed it in response. [View Less]

3 3

[Twisted-Python] Re: trac / website woes
by Matthias Urlichs Jan. 14, 2007

Jan. 14, 2007

Hi, Matthias Urlichs: > It seems that Deferreds are tricky. > Even more so -- apply on top of the last patch: --- /tmp/pb.py 2007-01-13 18:48:50.508925924 +0100 +++ /usr/lib/python2.4/site-packages/twisted/spread/pb.py 2007-01-13 19:20:15.000000000 +0100 @@ -795,6 +795,9 @@ }) return object + if not self.connected: + return deferred.fail(failure.Failure(DeadReferenceError())) + # XXX This call is NOT REENTRANT and … [View More]

1 0

[Twisted-Python] SSL
by Lee Connell Jan. 14, 2007

Jan. 14, 2007

I am using SSL with LineReceiver on my server and it is working fine, however when I start the server it asks me for the PEM key. Is there a way to automate this so the server can start with out user intervention? Thanks! -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.432 / Virus Database: 268.16.10/624 - Release Date: 1/12/2007 2:04 PM

1 0

[Twisted-Python] Re: client crashed
by Rick Graves Jan. 14, 2007

Jan. 14, 2007

Andrew, Thanks for the feedback. BTW, I am new to twisted but not to python. > Anyway, there's nothing Twisted specific here that I can see. I disagree -- I was trying to make a suggestion that would make future releases of twisted more bulletproof. Specifically, perhaps _parse() should be changed as follows (4 out of the last 5 lines are different): def _parse(url, defaultPort=None): parsed = urlparse.urlparse(url) scheme = parsed[0] path = urlparse.urlunparse(('','')+… [View More]

3 2

[Twisted-Python] client, deferred and/or reactor hung on these 2 URL's
by Rick Graves Jan. 13, 2007

Jan. 13, 2007

Hello, I am new to twisted. I was using client.downloadPage and reactor. Twisted was hung on the following two for a whole hour: http://75i.net/ http://www.theproxy.be/ I had to break out with Ctrl-C. When I did that, they finally fired the Errorback. These two stood out, as I had successfully (otherwise) run through a list of about 850 URL's. Is my reporting this helpful? Should I be reporting this elsewhere? Thanks, Rick Graves

4 3

[Twisted-Python] Twisted lineReceived (sender)
by Lee Connell Jan. 13, 2007

Jan. 13, 2007

I just started with twisted and want to write a simple chat server. This code here works fine, however I want to know who the data is coming from in the callbacks or events. You can see where I commented code out in def lineReceived as to what I want to do. How can this be done in the easiest way? Thanks! #!/usr/bin/env python from twisted.protocols import basic class MyChat(basic.LineReceiver): def connectionMade(self): print "Got new client!" self.factory.… [View More]

2 4

[Twisted-Python] documentation
by Lee Connell Jan. 13, 2007

Jan. 13, 2007

Didn’t mean it was scarce, I just am use to MSDN, I’ve managed to find what I’ve needed so far. Thanks! -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.432 / Virus Database: 268.16.10/624 - Release Date: 1/12/2007 2:04 PM

1 0

[Twisted-Python] client crashed
by Rick Graves Jan. 13, 2007

Jan. 13, 2007

Hello, I am new to twisted. I got one unhandled error (twice) in Deferred: In this code in client.py (you can skip to the 2nd line from the bottom): def _parse(url, defaultPort=None): parsed = urlparse.urlparse(url) scheme = parsed[0] path = urlparse.urlunparse(('','')+parsed[2:]) if defaultPort is None: if scheme == 'https': defaultPort = 443 else: defaultPort = 80 host, port = parsed[1], defaultPort if ':' in host: … [View More]

2 1

[Twisted-Python] Ticket Hall of Shame - Current Loser: Brian Warner
by glyph＠divmod.com Jan. 12, 2007

Jan. 12, 2007

Although we should all be happy for the Twisted 2.5 release, we still have an unfortunate track-record as far as responsiveness to bugs. I encourage all readers of this list to have a look at lucky report 13: http://twistedmatrix.com/trac/report/13 Note that there are still high-priority tickets that haven't received even so much as an update since 2004. Paul Swartz has recently removed himself as the ultimate loser of this non-contest, and Brian Warner has replaced him. Don't want to … [View More]

9 12