Mailman 3 November 2006 - Twisted

[Twisted-Python] Strange blocking in twisted + quickfix service
by Robert Parrott 15 Nov '06

15 Nov '06

Hi Folks, I'm trying to wrap the quickfix engine (www.quickfix.org) with a twisted service, in order to create a standalone daemon that will respond to RPC-like commands (such as "start", "stop", "request quote", etc.) However, I'm finding some odd blocking issues that I need to resolve. Basically, the twisted service works fine until I start the quickfix engine (which runs, I believe in a separate thread). Beforehand, calls to the twisted protocol class I've created invoke methods in the service, and things don't block. When the quickfix engine is started, it runs fine, but the main thread, in which the twisted "wrapper" application runs, seems to block in funny ways. As an example of this behavior, a connection to the LineReceiver protocol class I've subclassed will start to execute, but hangs exactly 12 minutes before proceeding. I.e. the time between telnetting to the port on which the server is listening and entering a command, and the time when the logs record that the "lineReceived" method of the LineReceiver class is invoked, is 12 minutes. As far as I can tell, everything is running in the same thread except for the quickfix engine class, so I'm not sure what the hanging behavior is due to. My understanding is that the twisted.python.log module, which I'm using to log, is non-blobking, and so shouldn't be causing this behavior. Does anyone have any ideas into what might be causing the hang, and approaches to diagnosing the cause? Thanks so much, rob

2 1

[Twisted-Python] Available for Twisted training
by Itamar Shtull-Trauring 15 Nov '06

15 Nov '06

Hi, Due to RSI I am taking a month off from work, so I am available for Twisted or Python-related training or other consulting (e.g. design and architecture help) that would not involve typing. Ideally in Boston area, but I might be open to other locations. I've been using and developing core parts of Twisted since 2001. See http://itamarst.org/resume.html - not really updated since 2004, so doesn't cover more recent projects such as reliable multicast system with congestion control. To emphasize: I'm limited to typing 30 minutes a day, so any consulting cannot, unfortunately, involve computer use. -- Itamar Shtull-Trauring http://itamarst.org

1 0

[Twisted-Python] Foolscap-0.0.5 released (aka newpb)
by Brian Warner 15 Nov '06

15 Nov '06

I've uploaded another release of Foolscap, the next-generation remote-object communication system for Twisted. (actually I've uploaded two, but 0.0.4 had a brown-paper-bag bug and so it never got announced). The new release makes all Referenceable objects "giftable" by default, making it easier to share objects among multiple machines. There is a new "Reconnector" with which you can announce your intention to connect to a given target and have the Tub perform exponential-backoff retry attempts whenever the connection is lost. Also, notifyOnDisconnect() now accepts args and kwargs to be passed to the disconnect handler callback. This release also adds support for making debian packages, for sid, sarge, and dapper. Just type 'make debian-sid' (etc) to get an installable .deb package in the parent directory. The full list of user-visible changes is attached below. No wire-level compatibility changes were introduced in this release, but there will be some in the future. Please see http://twistedmatrix.com/trac/wiki/FoolsCap for complete information about Foolscap. The 0.0.5 release tarball is available from http://twistedmatrix.com/~warner/Foolscap/ . share and enjoy, -Brian User visible changes in Foolscap (aka newpb/pb2). * Release 0.0.5 (04 Nov 2006) ** add Tub.setOption, add logRemoteFailures and logLocalFailures These options control whether we log exceptions (to the standard twisted log) that occur on other systems in response to messages that we've sent, and that occur on our system in response to messages that we've received (respectively). These may be useful while developing a distributed application. All such log messages have each line of the stack trace prefixed by REMOTE: or LOCAL: to make it clear where the exception is happening. ** add sarge packaging, improve dependencies for sid and dapper .debs ** fix typo that prevented Reconnector from actually reconnecting * Release 0.0.4 (26 Oct 2006) ** API Changes *** notifyOnDisconnect() takes args/kwargs RemoteReference.notifyOnDisconnect(), which registers a callback to be fired when the connection to this RemoteReference is lost, now accepts args and kwargs to be passed to the callback function. Without this, application code needed to use inner functions or bound methods to close over any additional state you wanted to get into the disconnect handler. notifyOnDisconnect() returns a "marker", an opaque values that should be passed into the corresponding dontNotifyOnDisconnect() function to deregister the callback. (previously dontNotifyOnDisconnect just took the same argument as notifyOnDisconnect). For example: class Foo: def _disconnect(self, who, reason): print "%s left us, because of %s" % (who, reason) def connect(self, url, why): d = self.tub.getReference(url) def _connected(rref): self.rref = rref m = rref.notifyOnDisconnect(self._disconnect, who, reason=why) self.marker = m d.addCallback(_connected) def stop_caring(self): self.rref.dontNotifyOnDisconnect(self.marker) *** Reconnector / Tub.connectTo() There is a new connection API for applications that want to connect to a target and to reconnect to it if/when that connection is lost. This is like ReconnectingClientFactory, but at a higher layer. You give it a URL to connect to, and a callback (plus args/kwargs) that should be called each time a connection is established. Your callback should use notifyOnDisconnect() to find out when it is disconnected. Reconnection attempts use exponential backoff to limit the retry rate, and you can shut off reconnection attempts when you no longer want to maintain a connection. Use it something like this: class Foo: def __init__(self, tub, url): self.tub = tub self.reconnector = tub.connectTo(url, self._connected, "arg") def _connected(self, rref, arg): print "connected" assert arg == "arg" self.rref = rref self.rref.callRemote("hello") self.rref.notifyOnDisconnect(self._disconnected, "blag") def _disconnected(self, blag): print "disconnected" assert blag == "blag" self.rref = None def shutdown(self): self.reconnector.stopConnecting() Code which uses this pattern will see "connected" events strictly interleaved with "disconnected" events (i.e. it will never see two "connected" events in a row, nor two "disconnected" events). The basic idea is that each time your _connected() method is called, it should re-initialize all your state by making method calls to the remote side. When the connection is lost, all that state goes away (since you have no way to know what is happening until you reconnect). ** Behavioral Changes *** All Referenceable object are now implicitly "giftable" In 0.0.3, for a Referenceable to be "giftable" (i.e. useable as the payload of an introduction), two conditions had to be satisfied. #1: the object must be published through a Tub with Tub.registerReference(obj). #2: that Tub must have a location set (with Tub.setLocation). Once those conditions were met, if the object was sent over a wire from this Tub to another one, the recipient of the corresponding RemoteReference could pass it on to a third party. Another side effect of calling registerReference() is that the Tub retains a strongref to the object, keeping it alive (with respect to gc) until either the Tub is shut down or the object is explicitly de-registered with unregisterReference(). Starting in 0.0.4, the first condition has been removed. All objects which pass through a setLocation'ed Tub will be usable as gifts. This makes it much more convenient to use third-party references. Note that the Tub will *not* retain a strongref to these objects (merely a weakref), so such objects might disappear before the recipient has had a chance to claim it. The lifecycle of gifts is a subject of much research. The hope is that, for reasonably punctual recipients, the gift will be kept alive until they claim it. The whole gift/introduction mechanism is likely to change in the near future, so this lifetime issue will be revisited in a later release. ** Build Changes The source tree now has some support for making debian-style packages (for both sid and dapper). 'make debian-sid' and 'make debian-dapper' ought to create a .deb package.

2 2

Re : [Twisted-Python] twisted performance and reactor choice
by Stéphane Brault 14 Nov '06

14 Nov '06

Thanks Jean-Paul, my mistake ;-) I must confess that if I use twisted (and I must say I'm quite happy with it), I didn't have time to know it that well. Anyway, I was wondering if changing the reactor could help a little (my application runs on a debian sarge 3.1). In which case would the pollreactor be relevant or should I stick with the default one (selectreactor if I read the documentation right). I have a lot of simultaneous deferred at scheduled times (both database access using zalchemy and connections to external sites which can take some times). I know this might not be enough to make a decision, but any help would be welcomed. Thanks again, Stéphane ----- Message d'origine ---- De : Jean-Paul Calderone <exarkun(a)divmod.com> À : Twisted general discussion <twisted-python(a)twistedmatrix.com> Envoyé le : Mardi, 14 Novembre 2006, 18h03mn 31s Objet : Re: [Twisted-Python] twisted performance and reactor choice On Tue, 14 Nov 2006 09:14:39 +0000 (GMT), Stéphane Brault <stephane_brault(a)yahoo.fr> wrote: >Hi, > >My application has lots of concurrent connections, both in and out, since >it communicates with outsides sites. It seems that it is sometimes slowed >down, I guess because there are to many simultaneous deffered. I'd like to >know if I could switch reactor (my application is accessed by XML-RPC and >communicates with external applications through XML-RPC, Http requests and >access ftp sites). I also need to know how large can be the threadpool in >order to increase the number of simultaneous deferred. You've gone a bit too fast here. You guessed at the cause of the slowdown. Instead, you need to profile the application and find out what the cause really is. Your question about the size of the threadpool is a bit of a non sequitur. The threadpool, let alone its size, is unrelated to how many Deferreds might exist in your application at once. Unless you are explicitly using the threadpool (or doing a significant number of DNS lookups without using Twisted Names), the threadpool isn't even related to how many concurrent tasks might be in progress in your application. This is because Twisted only uses the threadpool for an extremely limited number of purposes. Basic network traffic isn't one of these purposes. Jean-Paul _______________________________________________ Twisted-Python mailing list Twisted-Python(a)twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ___________________________________________________________________________ Découvrez une nouvelle façon d'obtenir des réponses à toutes vos questions ! Profitez des connaissances, des opinions et des expériences des internautes sur Yahoo! Questions/Réponses http://fr.answers.yahoo.com

2 1

[Twisted-Python] Re: [Twisted-bugs] Syntax error when upgrading Debian Python 2.4
by Stephen Waterbury 14 Nov '06

14 Nov '06

trac(a)twistedmatrix.com wrote: > New submission from LonelyPixel <nospam(a)unclassified.de>: > > I have installed twisted 2.4 for Debian Python 2.4 and after a recent apt-get upgrade, when Python 2.4 was to be upgraded, I get this error: > > Setting up python2.4 (2.4.1-2sarge1) ... > Compiling python modules in /usr/lib/python2.4 ... > Compiling /usr/lib/python2.4/site-packages/twisted/test/stdio_test_halfclose.py ... > File "/usr/lib/python2.4/site-packages/twisted/test/stdio_test_halfclose.py", line 20 > ??? > ^ > SyntaxError: invalid syntax I'm amazed that this error is still there -- I saw it *months* ago! Sure makes Twisted look bad. Steve

5 8

[Twisted-Python] twisted performance and reactor choice
by Stéphane Brault 14 Nov '06

14 Nov '06

Hi, My application has lots of concurrent connections, both in and out, since it communicates with outsides sites. It seems that it is sometimes slowed down, I guess because there are to many simultaneous deffered. I'd like to know if I could switch reactor (my application is accessed by XML-RPC and communicates with external applications through XML-RPC, Http requests and access ftp sites). I also need to know how large can be the threadpool in order to increase the number of simultaneous deferred. Thanks, Stéphane ___________________________________________________________________________ Découvrez une nouvelle façon d'obtenir des réponses à toutes vos questions ! Profitez des connaissances, des opinions et des expériences des internautes sur Yahoo! Questions/Réponses http://fr.answers.yahoo.com

2 1

[Twisted-Python] advice on asynchronous LDAP?
by Shimon Rura 14 Nov '06

14 Nov '06

Hi folks, I'm a newbie writing a twisted server that needs to access an LDAP server. I started by using python-ldap, which worked fine and is well documented, but doesn't seem to have support for a callback mechanism that would make it appropriate for a twisted app. So I switched to ldaptor, which fits the twisted model, but doesn't seem to have much documentation. I'm looking for advice from people who have some familiarity with writing twisted apps that include an LDAP client. Ideally, I'd like to find either: (1) a way to use the standard python-ldap library in a multitasking twisted server (i.e. without busy wait; polling at timed intervals is a possibility but also seems bad), or (2) a hidden trove of ldaptor documentation, or a mailing list or IRC channel where I could ask questions of people who know ldaptor. Any hints in either of these directions would be most appreciated. And if you're not an expert but might be interested in forming a new venue to discuss twisted+LDAP, let me know -- maybe we can set something up. thanks, Shimon Rura

3 2

[Twisted-Python] Why does this simple server not work?
by apocalypznow 13 Nov '06

13 Nov '06

The following is run but almost immediately terminates without any errors. WHY? I used "twistd -noy server.tac". Here is the code: **************************** import twisted from twisted.spread import pb from twisted.cred import checkers,portal from twisted.internet import reactor from twisted.python import threadable from twisted.python import usage import threading import socket clients={} class ClientRemoteObject(pb.Avatar): global clients def __init__(self): print "ClientRemoteObject initialized." def attached(self): print "Client attached" return self def detached(self): print "Client detached" def perspective_initialDataExchange(self,clientreference,username): print "Client reference attached for:" + username clients[username]=clientreference data={} data['connected']=1 return data def perspective_broadcastMessage(self,msg): for user in clients: clients[user].callRemote("messageCB",msg + "s") class DefaultRealm: __implements__ = portal.IRealm def requestAvatar(self, avatarID, mind, *interfaces): assert pb.IPerspective in interfaces clientremoteobj = ClientRemoteObject() clientremoteobj.attached() return pb.IPerspective, clientremoteobj, lambda a=clientremoteobj:a.detached() def shutdownCleanUp(): print "Twisted reactor shutting down." application = twisted.application.service.Application("server01") myService = twisted.application.service.IServiceCollection(application) reactor.addSystemEventTrigger('before','shutdown',shutdownCleanUp) p = portal.Portal(DefaultRealm()) dbu=twisted.cred.checkers.InMemoryUsernamePasswordDatabaseDontUse() dbu.addUser("user1","pass1") p.registerChecker(dbu) myServer = twisted.application.internet.TCPServer(81,pb.PBServerFactory(p)) myServer.setServiceParent(myService) **************** It used to work with Twisted 1.3rc1, but it does not run with Twisted 2.2 (or 2.3 or 2.4). I am using Python 2.3, and running this from a DOS prompt in Windows.

2 1

[Twisted-Python] Converting twistd script to NT service using py2exe
by Alvin Wang 13 Nov '06

13 Nov '06

http://twistedmatrix.com/pipermail/twisted-python/2004-January/007021.html http://twistedmatrix.com/pipermail/twisted-python/2004-March/007311.html http://twistedmatrix.com/pipermail/twisted-python/2005-April/010038.html *There seems to be the implication that I need tap2ntsvc from the sandbox. Unfortunately, I can not seem to find it in the sandbox. * Is it out of date? Have the files been moved? Is there another way to accomplish the same objective? thanks -- Alvin

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 12 Nov '06

12 Nov '06

Bug summary ______________________ Summary for 2006-11-05 through 2006-11-12 Bugs opened: 9 Bugs closed: 3 Total open bugs: 783 (+6) |== Type Changes |== Priority Changes |== Component Changes |Defect: +5 |Highest: +1 |Core: +4 |Enhancement: +1 |High: +1 |Trial: +0 |Normal: +2 |Web: +1 |Low: +1 |Web2: +1 |Lowest: +1 New / Reopened Bugs ______________________ ===== Highest ===== [#2223] DeferredSemaphore should be marked "stable" (opened by spiv) enhancement core http://twistedmatrix.com/trac/ticket/2223 [#2225] microdom shouldn't bork on malformed tags when beExtremelyLenient is enabled (opened by jml) (CLOSED, fixed) enhancement web http://twistedmatrix.com/trac/ticket/2225 ===== High ===== [#2224] Web2: Behavior of Request.locateResource vs Resource.locateChild is not clearly defined. (opened by dreid) defect web2 http://twistedmatrix.com/trac/ticket/2224 ===== Normal ===== [#2219] isInIOThread() returns False when used in TestCases (opened by sifu) defect trial http://twistedmatrix.com/trac/ticket/2219 [#2220] [SIP] parsing bugs/limitations and patch (opened by antoine) defect core http://twistedmatrix.com/trac/ticket/2220 [#2222] t.i.task.Clock.advance should set the call.called attribute. (opened by ralphm) defect core http://twistedmatrix.com/trac/ticket/2222 ===== Low ===== [#2221] MotdResource gets stale in finger15.py (opened by gfalcon) (CLOSED, duplicate) defect core http://twistedmatrix.com/trac/ticket/2221 [#2227] sux uses "True" as the default value for an attribute (opened by jml) enhancement web http://twistedmatrix.com/trac/ticket/2227 ===== Lowest ===== [#2226] qt4reactor.stop() not closing connections and quitting python (opened by Tantalum) defect core http://twistedmatrix.com/trac/ticket/2226 Closed Bugs ______________________ ===== Highest ===== [#2225] microdom shouldn't bork on malformed tags when beExtremelyLenient is enabled (opened by jml, closed by jml, fixed) enhancement web http://twistedmatrix.com/trac/ticket/2225 ===== Normal ===== [#2144] Add Trial option to stop test run on the first failure (opened by jml, closed by exarkun, duplicate) enhancement trial http://twistedmatrix.com/trac/ticket/2144 ===== Low ===== [#2221] MotdResource gets stale in finger15.py (opened by gfalcon, closed by jerub, duplicate) defect core http://twistedmatrix.com/trac/ticket/2221

1 0