Mailman 3 November 2005 - Twisted

[Twisted-Python] problems with threadedselectreactor
by Antony Kummel Nov. 30, 2005

Nov. 30, 2005

Hi! I encountered two problems when using threadedselectreactor on win32. I am not versed enough with reactor technicalities to resolve them myself, but I would gladly conduct any tests the experts recommend. 1. I often get tracebacks similar to this: File "C:\Python23\Lib\site-packages\twisted\internet\threadedselectreactor.py", line 299, in _doReadOrWrite why = getattr(selectable, method)() File "C:\Python23\Lib\site-packages\twisted\internet\abstract.py", line 108, in doWrite l = self.writeSomeData(self.dataBuffer) File "C:\Python23\lib\site-packages\twisted\internet\tcp.py", line 358, in writeSomeData return self.socket.send(buffer(data, 0, self.SEND_LIMIT)) exceptions.AttributeError: 'Server' object has no attribute 'socket' To my understanding, this can only happen if the Server's connectionLost method had been called before the exception happened, but for some reason, writeSomeData (or sometimes another method) was still called. I hypothesize that the Server should have been removed from some list or another but wasn't, and that this is the reactor's responsibility. 2. Sometimes I get this error: Traceback (most recent call last): File "C:\Python23\Lib\site-packages\wx-2.6-msw-unicode\wx\_core.py", line 7493, in MainLoop wx.PyApp.MainLoop(self) File "C:\Python23\Lib\site-packages\wx-2.6-msw-unicode\wx\_core.py", line 6926, in MainLoop return _core_.PyApp_MainLoop(*args, **kwargs) File "C:\Python23\Lib\site-packages\wx-2.6-msw-unicode\wx\_core.py", line 13266, in <lambda> lambda event: event.callable(*event.args, **event.kw) ) File "C:\Python23\Lib\site-packages\twisted\internet\threadedselectreactor.py", line 241, in _interleave self.runUntilCurrent() --- <exception caught here> --- File "C:\Python23\Lib\site-packages\twisted\internet\base.py", line 541, in runUntilCurrent call.func(*call.args, **call.kw) File "C:\Python23\Lib\site-packages\twisted\internet\task.py", line 91, in __call__ self._reschedule() File "C:\Python23\Lib\site-packages\twisted\internet\task.py", line 108, in _reschedule self.call = reactor.callLater(delay, self) File "C:\Python23\Lib\site-packages\twisted\internet\threadedselectreactor.py", line 116, in callLater self.wakeUp() File "C:\Python23\Lib\site-packages\twisted\internet\threadedselectreactor.py", line 112, in wakeUp self.waker.wakeUp() File "C:\Python23\Lib\site-packages\twisted\internet\posixbase.py", line 85, in wakeUp self.w.send('x') socket.error: (10055, 'No buffer space available') This doesn't seem to be a threadedselectreactor problem, but rather a problem that stems from it's use with arbitrary event loops. I suspect that it happens when the main thread doesn't respond for some time. I think that the best way to deal with it is simply to ignore this error (in wakeUp), because if the buffer is full, then the main thread already knows (or about to find out) that it needs to wake up, although maybe the user should be somehow notified that there is a problem (log message?). Thanks, Antony Kummel --------------------------------- Yahoo! Personals Single? There's someone we'd like you to meet. Lots of someones, actually. Yahoo! Personals

2 2

[Twisted-Python] Results of stress testing the reactor
by Keith.Gunderson＠act.org Nov. 30, 2005

Nov. 30, 2005

I'm getting settled in with an experimental project using twisted. I posting some of my preliminary results in this message. Please let me know if this information is interesting or helpful. I plan on using twisted as a part of my performance testing simulations. I use the Mercury Interactive tools for the 'heavy lifting' of my work. I'll use my twisted designs to provide advanced simulation logic and timing features. That is, I'll create many virtual users that will talk HTTP to non-twisted web apps while measuring the results (i.e. performance testing). But, these virtual users will sometimes talk HTTP to my twisted web app to get whatever data I want them to have. So, I've got the following little app which takes three seconds to respond to any request. I'm trying test the Mercury tools on how they deal with concurrent requests. I'm not doing much here, just kicking in the tires a little. I'm using the latest twisted release on Win2000. import time,sys from twisted.internet import protocol, reactor def defFn( protObject): protObject.transport.write('Hello. Deferred:\n') sys.stdout.write('The time is %s\n' % time.time()) protObject.transport.loseConnection() class TimeProtocol(protocol.Protocol): def dataReceived(self, data): self.transport.write('Hello. Data received: \n' ) reactor.callLater(3, defFn, self ) class TimeFactory(protocol.ServerFactory): protocol = TimeProtocol reactor.listenTCP(9100, TimeFactory()) reactor.run() Everything works as expected until I get about 300 virtual users hitting the site somewhat simultaneously. (Approx. 200 hits per second.) At which point the reactor kind of explodes. The following traceback races from the console (probably in an infinite loop.) This may not be unexpected or significant, but I found it interesting. Is there anything I can do to prevent this, other than refraining from trying to overload things on purpose? Are there other tests that you gurus might find interesting? File "tw09.py", line 25, in ? reactor.run() File "C:\progs\python24\Lib\site-packages\twisted\internet\posixbase.py", line 199, in run self.mainLoop() File "C:\progs\python24\Lib\site-packages\twisted\internet\posixbase.py", line 210, in mainLoop self.doIteration(t) --- <exception caught here> --- File "C: \progs\python24\Lib\site-packages\twisted\internet\selectreactor.py", line 97, in doSelect [], timeout) File "C: \progs\python24\Lib\site-packages\twisted\internet\selectreactor.py", line 49, in win32select r, w, e = select.select(r, w, w, timeout) exceptions.ValueError: too many file descriptors in select() Traceback (most recent call last): File "tw09.py", line 25, in ? reactor.run() File "C:\progs\python24\Lib\site-packages\twisted\internet\posixbase.py", line 199, in run self.mainLoop() File "C:\progs\python24\Lib\site-packages\twisted\internet\posixbase.py", line 210, in mainLoop self.doIteration(t) --- <exception caught here> --- File "C: \progs\python24\Lib\site-packages\twisted\internet\selectreactor.py", line 97, in doSelect [], timeout) File "C: \progs\python24\Lib\site-packages\twisted\internet\selectreactor.py", line 49, in win32select r, w, e = select.select(r, w, w, timeout) exceptions.ValueError: too many file descriptors in select() Keith Gunderson Performance Test Analyst

2 1

[Twisted-Python] Re: [Twisted-commits] r15259 - Try again. Merge trial-deferred-logic
by Jean-Paul Calderone Nov. 30, 2005

Nov. 30, 2005

On Tue, 29 Nov 2005 23:43:31 -0700, Jonathan Lange <jml(a)wolfwood.twistedmatrix.com> wrote: >Author: jml >Date: Tue Nov 29 23:43:31 2005 >New Revision: 15259 > >Modified: > trunk/twisted/mail/test/test_mail.py > trunk/twisted/test/test_ftp.py > trunk/twisted/test/test_ssl.py > trunk/twisted/test/test_tcp.py > trunk/twisted/trial/reporter.py > trunk/twisted/trial/runner.py > trunk/twisted/trial/test/detests.py > trunk/twisted/trial/test/test_deferred.py > trunk/twisted/trial/test/test_failure_formatting.py > trunk/twisted/trial/test/test_keyboard.py > trunk/twisted/trial/test/timeouts.py > trunk/twisted/trial/unittest.py > trunk/twisted/trial/util.py > trunk/twisted/web/test/test_webclient.py >Log: >Try again. Merge trial-deferred-logic > >@@ -907,17 +900,22 @@ > self.client.transport.loseConnection() > return self.p.stopListening() > >+ def _delayDeferred(self, time, arg=None): >+ from twisted.internet import reactor >+ d = defer.Deferred() >+ reactor.callLater(time, d.callback, arg) >+ return d >+ > def testNoNotification(self): > client = self.client > f = self.f > client.transport.write("hello") > w = client.transport.write > client.transport.loseWriteConnection() >- reactor.iterate() >- reactor.iterate() >- reactor.iterate() >- self.assertEquals(f.protocol.data, "hello") >- self.assertEquals(f.protocol.closed, True) >+ d = self._delayDeferred(0.2, f.protocol) >+ d.addCallback(lambda x : self.assertEqual(f.protocol.data, 'hello')) >+ d.addCallback(lambda x : self.assertEqual(f.protocol.closed, True)) >+ return d > > def testShutdownException(self): > client = self.client The above hunk is pretty sketchy. The real solution is to give the protocol a Deferred which it fires at whatever point it would have set its closed attribute to True. As it stands, the test has a race and I can guarantee it will be going the wrong way quite frequently. Jean-Paul

1 0

RE: [Twisted-Python] My review of Abe's book "Twisted NetworkProgramming Essentials"
by Henning.Ramm＠mediapro-gmbh.de Nov. 30, 2005

Nov. 30, 2005

>Hi. I just finished a review of the new O'Reilly book on Twisted, >by Abe Fettig (hi Abe!), and thought you all might be interested in it: Thank you very much for the hints how to simplify the Python code! (I didn't knew there was dict.setdefault ;-) Greetlings, HR

2 1

[Twisted-Python] Weekly Bug Summary
by James Y Knight Nov. 30, 2005

Nov. 30, 2005

Bug summary ______________________ Since 2005-11-23 07:00:01.530000: Bugs opened: 6 Bugs closed: 3 Total open bugs: 484 (+3) Urgent: 1 (+0) Bug: 227 (+3) Feature: 173 (+0) Wish: 83 (+0) New / Reopened Bugs ______________________ ==== Bug ==== [CORE] ftp NLST fails with a file on a twisted ftp server (yesterday) http://twistedmatrix.com/bugs/issue1342 opened by micktwomey [CORE, TRIAL] Test case run() methods to use Deferreds internally (2 days ago) http://twistedmatrix.com/bugs/issue1341 opened by jml [CORE, TRIAL] Possible: t.i.process.Process.reapProcess fails when called after a successful call (5 days ago) CLOSED http://twistedmatrix.com/bugs/issue1340 opened by jml [DOCUMENTATION, WEBSITE] API docs on site should reflect recent releases (6 days ago) http://twistedmatrix.com/bugs/issue1339 opened by ralphm [CORE] Delayed calls (callLater) can be lost when multi-threaded (6 days ago) CLOSED http://twistedmatrix.com/bugs/issue1338 opened by jeffsilver twisted.web.html.output bug (IRC) (6 days ago) http://twistedmatrix.com/bugs/issue1337 opened by kyle Bugs Closed ______________________ ==== Bug ==== [CORE, TRIAL] Possible: t.i.process.Process.reapProcess fails when called after a successful call (5 days ago) http://twistedmatrix.com/bugs/issue1340 opened by jml, closed by jml [TRIAL] Incorrect traceback from trial (13 months ago) http://twistedmatrix.com/bugs/issue769 opened by exarkun, closed by jml [CORE] Delayed calls (callLater) can be lost when multi-threaded (6 days ago) http://twistedmatrix.com/bugs/issue1338 opened by jeffsilver, closed by exarkun

1 0

Re: [Twisted-Python] newpb authorization
by Brian Warner Nov. 29, 2005

Nov. 29, 2005

> There's an entry in the newpb-docs, newpb-jobs.txt or so, > where an example is given how this authorization might/will look like. > it seems that it is to be implemented in the negotiation-phase, before > the switch-to-banana. Is that interpretation right, or am i wrong about > this, and the authorization is meant to take place via a mediating > pb.referenceable, similar to oldpb? My current thought is the latter, keeping all notion of authorization at the Referenceable layer, out of the connection itself. I'm thinking that the Realm (in cred terms) is a Referenceable, and you send your credentials to it via a login() method. The Realm then returns another Referenceable (the Avatar), which you interact with to your hearts content. On the client side, you'd have some sort of higher-level object to which you would pass the PB-URL of the target Realm and your Credentials. When all was said and done, you'd wind up with a RemoteReference to the remote Avatar. You could have multiple Realms on the same host, distinguished by their URLs. I'm definitely leaning in a capabilities-based direction these days, so both the Realm (which anybody can access) and the Avatar (which can only be accessed by the person with the right credentials) are reachable with URLs, the only difference being that the Realm's URL is public knowledge, while the Avatar URLs are randomly-generated and unguessable. If you want to bypass the whole username/password thing, you just give the hopeful client the URL of their Avatar, and let them connect directly to that. [note that some of the things I wrote about cred-vs-newpb were before I was fully aware of the capabilities model. back in those days, I was vaguely thinking of having a "default object" for each connection, which could accept credentials and return Avatars, such that it might make sense to put things like username and passwords in the URLs. Also, back then it seemed reasonable to use Interfaces or RemoteInterfaces as ways to specify "who" or "what" you wanted to talk to. I've since come to my senses. newpb provides a layer of secure remote references, on top of which you can build whatever sort of authorization scheme you like] [also, back then I was expecting Glyph to swoop down and implement the whole thing in one afternoon's frantic coding binge, absolving me from needing to understand how it would actually work or get used. fortunately this has not come to pass, so I get to write something that can be comprehended by my own humble brain. muahahaha.] > What i'm thinking of is two distinct uses of cred in pb. one at an early > stage, affecting the whole connection to be or not to be, and one in > later phases, to have a basis for managing roles and permissions while > the app is running. It sounds like the capabilities-based scheme I have in mind fits the second of your use cases. As for the first one.. what sort of things connection-time things would you like to be controlled by authorization information? One thing that comes to mind is whether the connection will be accepted or not, presumeably to reduce resource utilization, but consider this: the TCP connection will be accepted regardless. until the first byte of data arrives, the only thing we can deduce about the client is their IP address, and we all know how unhelpful it is to make decisions upon purely that basis. given that tub.registerReference() returns an unguessable URL by default, the only objects that can be reached will be ones to which you have explicitly granted access, either by registering a well-known URL (like "/login") or by giving the URL to the client by some other channel. So allowing the "bad guy" to establish a connection doesn't actually let them access anything you don't want them to have. newpb's Schemas can be used to minimize resource utilization (specifically memory consumption) required by messages sent to those public objects. If the only object exposed to the world at large is the "/login" Realm, and if it only has a single remote_login(username, password) method, then a simple (username=str, password=str) schema will prevent that one permissible message from consuming too much space, limiting its utility in a DoS attack. Given those, what other connection-time authorization checks do you think might be useful? Having authorization take place entirely at the object layer also has the nice property that you can run arbitrary numbers of mutually-unaware services off the same connection. My reluctance to add authorization below that layer comes partially from the benefits of this property. As an example, in the Buildbot[1], there are several PB services provided to entities in the outside world (buildslaves, a status client, a debug client, and a Change inlet), each distinguished by the cred username/password provided by the various clients. The nominal API for these services (i.e., what you put in the config file to create each service) would allow you to choose an arbitrary TCP port for each, which may or may not overlap with each other. In practice, the hassle of creating a Realm for each PBServerFactory (some of which know about some usernames, others which don't, all of which could change each time the config file is changed, etc) was so great that I punted and just forced the user to make all those services share a single TCP port, hence using a single Realm which knows about everything. I haven't quite figured out how this would work if Buildbot used newpb, but I know that I'd expose, say, the status service with a URL that ended in "/status", and the debug service (which is supposed to be protected, and thus gets a user-specified password in oldpb) would just get an unguessable URL. The same TCP port could be used for different services that had different authorization requirements. If authorization took place before the object-level connection could be established, I'd have to coordinate another layer of usernames and passwords (or challenge-response pairs, or SRP exchanges, or whatever) without the benefit of using PB serialization or object references. Having already built one remote-method-invocation protocol this decade, I'm not keen on creating another one quite yet :). cheers, -Brian PS: Yay! Somebody is looking at newpb! :) [1] Buildbot: http://buildbot.sf.net/

1 0

[Twisted-Python] newpb authorization
by Johann Borck Nov. 28, 2005

Nov. 28, 2005

I'm writing a distributed app which depends on username/password authorization. There's an entry in the newpb-docs, newpb-jobs.txt or so, where an example is given how this authorization might/will look like. it seems that it is to be implemented in the negotiation-phase, before the switch-to-banana. Is that interpretation right, or am i wrong about this, and the authorization is meant to take place via a mediating pb.referenceable, similar to oldpb? In the latter case it's mostly just a matter of copy&paste from the oldpb implementation, which works for me now. But if there are some ideas on where and how to do it in an earlier stage, or considerations of advantages to do it some defined way, i'd be glad to know about. Brian Warner wrote: >cred has not yet been dragged into newpb, but it >won't be too much work once someone figures out what exactly they want out of >such a combination :) . What i'm thinking of is two distinct uses of cred in pb. one at an early stage, affecting the whole connection to be or not to be, and one in later phases, to have a basis for managing roles and permissions while the app is running. i will spend some work on that, again, any ideas are appreciated, and with some feedback it's more likely to result in something usable for more than just me. Johann

1 0

[Twisted-Python] Serving different resource types from the same directory
by Olivier Laurent Nov. 28, 2005

Nov. 28, 2005

I'm trying to serve 2 different kind of resources from the same directory. I can, with no problem, serve one kind or the other but not the two resource types at the same time. Examples abounds on the web on how to serve one kind of resources from subdirectory /foo and another kind from subdirectory /bar. But no example on how to serve different kind of resources from the same directory. Here is how I tried: # [...] PATH = '/var/www/' pt_resource = static.File(PATH) pt_resource.processors = { '.pt' : PTResource } pt_resource.indexNames = [ 'index' + '.pt' ] rst_resource = static.File( PATH ) rst_resource.processors = { '.rst' : RSTResource } rst_resource.indexNames = [ 'index' + '.rst' ] #root = Resource() # I tried this too root = static.File(PATH) root.putChild(PATH, pt_resource) root.putChild(PATH, rst_resource) reactor.listenTCP( 80, server.Site( root ) ) reactor.run( ) # this is the end What's wrong ? What did I forget ? Is it possible ? Thank you. -- Olivier Laurent

3 2

[Twisted-Python] Irc <-> Jabber gateway?
by Avleen Vig Nov. 27, 2005

Nov. 27, 2005

Has anyone written such a device using twisted? In theory it ought to be easy enough, I don't want to duplicate the work if someone's already done it though :-) -- Avleen Vig Systems Administrator Personal: www.silverwraith.com "Wickedness is a myth invented by good people to account for the curious attractiveness of others." - Oscar Wilde

2 1

[Twisted-Python] Re: [Twisted-commits] r15209 - basename('/') == '', not IndexError
by Jean-Paul Calderone Nov. 26, 2005

Nov. 26, 2005

On Fri, 25 Nov 2005 20:35:46 -0700, Stephen Thorne <stephen(a)wolfwood.twistedmatrix.com> wrote: >Author: stephen >Date: Fri Nov 25 20:35:45 2005 >New Revision: 15209 > >Modified: > branches/wsanchez/dav/twisted/vfs/pathutils.py >Log: >basename('/') == '', not IndexError > >Modified: branches/wsanchez/dav/twisted/vfs/pathutils.py >============================================================================== >--- branches/wsanchez/dav/twisted/vfs/pathutils.py (original) >+++ branches/wsanchez/dav/twisted/vfs/pathutils.py Fri Nov 25 20:35:45 2005 >@@ -51,7 +51,10 @@ > return currNode > > def basename(path, cwd='/'): >- return getAbsoluteSegments(path, cwd)[-1] >+ s = getAbsoluteSegments(path, cwd) >+ if s: >+ return s[-1] >+ return '' > > def dirname(path, cwd='/'): > return "/" + "/".join(getAbsoluteSegments(path, cwd)[:-1]) Any plans to revert this? Jean-Paul

1 0