Mailman 3 April 2004 - Twisted

[Twisted-Python] All use cases of getComponent
by Itamar Shtull-Trauring April 20, 2004

April 20, 2004

I'd like to have a list of all classes that use getComponent (Componentized, Adapter - anything else?) and the expected behaviour - do people override it, etc.. I'm working on Zope3 port of components and trying to figure out how to do backwards-compat (Zope3 has __conforms__ instead of getComponent). -- Itamar Shtull-Trauring http://itamarst.org Looking for a job -- http://itamarst.org/resume.html

2 4

[Twisted-Python] UDP Support sanity check
by Richard Taylor April 20, 2004

April 20, 2004

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi I am trying to implement support for the UDP functionality I need. I think that I need to implement the following classes: For server support: UDPPort(udp.Port) - adding support for broadcast ConnectedUDPPort(UDPPort) - connected variant For client (connection) support: UDPConnectionBase(FileDescriptor) - creation of non-bound UDP sockets UDPConnection(UDPConnectionBase) - addition of write method to implement IUDPTransport UDPConnectedConnection(UDPConnectionBase) - support for 'connected' UDP socket and write method without host,port arguments. UDPConnector(base.Connector) - connector class that can create use the UDPConnection objects. UDPConnectedConnector(UDPConnector) - one that works with UDPConnectedConnection objects. Extend the IReactorUDP interface and reactor implementation to support connectXXX methods for the new connector types. I am trying to support the creation of UDP connectors that are meant for sending datagrams and will never be bound to a local port and listened on. The current support for UDP does not implement and UDP connector type objects only UDP Port type objects. This means that a bind is always performed even when the connection is only meant for sending. I have most of the implementation written (at least in outline). One question. It appears to me that for UDP connection objects that concept of a protocol does not make much sense. The IUDPTransport interface does have a read method (as you would expect) so the protocol would never be called. This also calls in to question the need for a 'protocol factory' for such connections. Am I correct in this or am I missing something? Richard - -- QinetiQ B009 Woodward Building St. Andrews Road Malvern Worcs WR14 3PS Jabber: RichardTaylor(a)jabber.org PGPKey: http://search.keyserver.net:11371/pks/lookup?op=get&search=0xA7DA9FD9 Key fingerprint = D051 A121 E7C3 485F 3C0E 1593 ED9E D868 A7DA 9FD9 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQFAhORk7Z7YaKfan9kRAiuzAJ9TlVM1puA5TIDNSfKTvH/xIJHxzwCeMrRE mzGko+L421THScL3FgbqKPU= =xiC0 -----END PGP SIGNATURE-----

2 1

[Twisted-Python] why deferred.setTimeout is not my favorite API method
by Jonathan Simms April 20, 2004

April 20, 2004

This is response to issue 178 on the twisted tracker: http://www.twistedmatrix.com/users/roundup.twistd/twisted/issue178, about the deferred method setTimeout. The reason why I added a "DON'T USE THIS" to the docstring was that I find that we are telling people this at least once a week. I think it has more gravitas coming from the API documentation, as there is this illusion that the people on IRC don't really know what they're talking about, but if the author has marked something in the documentation, it has a certain level of legitimacy. I agree with spiv in the sense that it would be nice to have a minimal interface that would provide this functionality. However, i think that the level of control that is needed to pull this off properly is best served by writing an explicit timeout method and accompanying call to callLater. now, I'd like to present the idiom that we are try to encourage users to follow every time we have to steer them away from setTimeout. ------------------------------------------------------ d = iReturnADeferred() def onTimeout(): handleTimeoutCondition() delayedCall = reactor.callLater(timeoutLen, onTimeout) def success(value): handleSuccess() def nonTimeoutFailure(reason): handleErrorCondition() def cancelTimeout(val): if delayedCall.active(): delayedCall.cancel() d.addBoth(cancelTimeout) d.addCallback(success) d.addErrback(nonTimeoutFailure) ------------------------------------------------------- Here you see that each step is very explicit. You can see what happens on success and on timeout (failure). I think one problem with setTimeout is the handling of the IDelayedCall is something that a new user could miss. For instance, someone who hadn't /read/ the code of setTimeout, would most likely miss the returned IDelayedCall, and wouldn't know to cancel the pending timeout call in the event of success OR other failure. This is _a crucial piece of the code path_, and setTimeout makes this easier to miss. As from the Zen of Python Programming: - Explicit is better than implicit setTimeout hides a small piece of code that contains all of the important conceptual elements that a user is required to understand so that they may handle this side-exit condition properly. - Special cases aren't special enough to break the rules Delayed calls are orthogonal to deferreds. Deferreds are a _reactive_ tool, they fire in response to an event delayed calls are a _proactive_ tool, they cause events to fire Mixing the two together for the special case of dealing with a timeout is at best hairy and at worst ugly and confusing. Timeout conditions are _not necessarily_ something that a user needs to deal with 80% of the time they are working with deferreds. - Simple is better than complex The deferred api should contain absolutely necessary functionality. Let's emulate the python language, not the python standard library - There should be one, and preferrably only one way to do it It is better to have users of our API use a widely-understood and flexible idiom to accomplish this rather than a half-baked method that saves them 8 lines of easily written and conceptually crucial code I would like to recommend that we document a best-practice idiom in the deferred api documentation, and remove setTimeout. - If an implementation is hard to explain it's a bad idea Explaining to users how to properly use setTimeout takes more effort than explaining to them how to use the tools that setTimeout employs to achieve its functionality. spiv said: "In hindsight, I should've thought harder about the implications of setTimeout before I committed it... but now it's there, I don't like the idea of ripping it out unless we have a clearly better solution." This is a less-than-optimal solution to an edge-case use of two orthogonal parts of our API. It makes no sense to leave a sharp, pointy thing lying around for users to hurt themselves with, and to constantly tell them "DON'T USE THAT". -Jonathan Simms slyphon AT twistedmatrix DOT com

4 4

[Twisted-Python] PyPAM really needed by Twisted?
by Jonathan Dill April 19, 2004

April 19, 2004

I'm following the INSTALL instructions for Twisted-1.2.0 and it claims that it depends on PyPAM. PyPAM appears not to have been developed since 1999 for Python 1.52. Are these install directions really up to date? Or should I try the Twisted 1.2.1alpha1 or Twisted out of CVS instead? Thanks, -- Jonathan Dill <jfdill(a)jfdill.com>

2 1

[Twisted-Python] more kudos of mine for Twisted, quote/point to at will
by Alex Martelli April 19, 2004

April 19, 2004

"Open Enterprise Trends" has published the second part of an interview with me (about Python's role in the enterprise) -- you can find it at URL: http://www.oetrends.com/news.php?action=view_record&idnum=321 Towards the end of the interview I got a chance to heap some praise upon Twisted and to urge "enterprise developers" to consider it. Since OEtrends enjoys some prestige in certain IT circles, being able to quote from it might help you making the case, to a manager or client, that Twisted should be used for a certain project. If so, please feel free to quote (e.g. such snippets as "For any Python network server or client application, a developer would be well advised to consider the free Twisted Matrix package") and/or link to this URL (unfortunately there are no anchors to ease referring to the specific spot, and the whole HTML page is otherwise unsatisfactory -- e.g., the "keywords" include several technologies I do not even mention nor allude to, such as jboss, netbeans abd jabber, but do NOT include twisted -- how silly). Alex

2 1

[Twisted-Python] Twisted split: tap plugins
by Christopher Armstrong April 19, 2004

April 19, 2004

So, Jp and I ran into our first Big Problem with the Twisted split. First, though, we *have* gotten the TwistedNews repository set up: svn://svn.twistedmatrix.com/svn/TwistedNews/trunk or svn+ssh://.... The problem is the tap plugin. Currently both TwistedNews and Twisted have a plugins.tml that contains a "news" tap plugin. We assume that we'll leave twisted.news in Twisted for a little while for backwards compatibility. The alternatives that we've come up with are like so: 1) Rename 'mktap news' to 'mktap newsnews' in TwistedNews. Rename it back to 'news' when we get rid of twisted.news. This is Jp's solution. 2) Add an optional argument to the register() call in plugins.tml: allowOverride=True. This would mean that another register() call with similar arguments would override this one, whether it gets executed before or after this one. We would pass this argument to our deprecated tap-plugins in Twisted's plugins.tml. This is my solution. I dislike the idea of a rename, because not only would we force users to change what they're doing now, we'd force them to change what they're doing later when we get rid of twisted.news. This problem is going to have to be solved now, because we'll have similar problem with pretty much all of the packages we'll be splitting from Twisted. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | Release Manager, Twisted Project ---------+ http://radix.twistedmatrix.com/

3 5

[Twisted-Python] New user, help needed with raw UDP packets
by Richard Taylor April 19, 2004

April 19, 2004

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi I am a completely new user of Twisted. I have read the tutorial and much of the manual and I have been searching the archives but I can't work out how to get started with my problem. I am writing a UDP proxy/relay that needs to make detailed changes to the internals of the UDP packets as they pass through. I intend to use 'Impacket' to access the packet internals. One of the things that I need is to 'fake' the destination address of packets leaving the proxy so that they look like they have come from the original sender, Impacket has methods to do this for 'raw' udp packets. I think that I need to use a sub-class of ConnectedDatagramProtocol to send my packets but the "self.transport.write" method takes only the packet payload not the headers. I think that I need to use a sub-class of DatagramProtocol to receive my packets but the datagramReceived method only gets the packet payload not the headers. How can I get at the 'raw' udp packets for receiving and sending? Sorry if this is obvious. Many thanks. Richard - -- QinetiQ B009 Woodward Building St. Andrews Road Malvern Worcs WR14 3PS Jabber: RichardTaylor(a)jabber.org PGPKey: http://search.keyserver.net:11371/pks/lookup?op=get&search=0xA7DA9FD9 Key fingerprint = D051 A121 E7C3 485F 3C0E 1593 ED9E D868 A7DA 9FD9 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQFAfU7B7Z7YaKfan9kRAtJRAKCRkDbM0TEtgI3K3107v7aXk7r9awCg5sz6 X7IT8BGjPiSW0a2aYUGhLcY= =LKvJ -----END PGP SIGNATURE-----

4 5

Re: [Twisted-Python] All use cases of getComponent
by exarkun＠divmod.com April 19, 2004

April 19, 2004

On Sun, 18 Apr 2004 20:46:35 -0400, Itamar Shtull-Trauring <itamar(a)itamarst.org> wrote: >I'd like to have a list of all classes that use getComponent > (Componentized, Adapter - anything else?) and the expected behaviour - > do people override it, etc.. > > I'm working on Zope3 port of components and trying to figure out how to > do backwards-compat (Zope3 has __conforms__ instead of getComponent). Atop implements getComponent twice: once on a subclass of Componentized to make it slightly more convenient to use (automatic referenceTo/getItem logic), and once as a full implementation on another class that does not inherit from Componentized. Atop would also like an IComponentized to be added, but I'm not sure if that has any implications for the port to Zope3. Jp

1 0

Re: [Twisted-Python] why deferred.setTimeout is not my favorite API method
by slyphon＠twistedmatrix.com April 19, 2004

April 19, 2004

>It looks like I've succeeded in rekindling interest in this issue :) > >> The reason why I added a "DON'T USE THIS" to the docstring was that I >> find that we are telling people this at least once a week. I think it >> has more gravitas coming from the API documentation, as there is this >> illusion that the people on IRC don't really know what they're talking >> about, but if the author has marked something in the documentation, it >> has a certain level of legitimacy. > >Well, I guess I'm a bit confused. Why not raise a DeprecationWarning as >well then? Is "DON'T USE THIS" meant to be a watered-down version of a >DeprecationWarning? Yes. It was a diplomatic move. I understand that you wrote this function, and that you feel strongly that it adds value to the twisted library. To me, a DeprecationWarning represents that there has been a consensus reached that a certain function is not going to be supported in the future. I wouldn't be so haughty as to decide that on my own, so I put a note that reflected the advice that we normally give users on IRC into the API docs. > >> I agree with spiv in the sense that it would be nice to have a minimal >> interface that would provide this functionality. However, i think that >> the level of control that is needed to pull this off properly is best >> served by writing an explicit timeout method and accompanying call to >> callLater. > >The tricky part isn't the callLater, it's the actual cancellation of >whatever the operation that would've produced a deferred result -- and >that's something defer.py can't help with, because it is app-specific. Indeed, i realised the flaws in my example about 20 minutes after sending it >> >> now, I'd like to present the idiom that we are try to encourage users to >> follow every time we have to steer them away from setTimeout. >> >> ------------------------------------------------------ >> >> d = iReturnADeferred() >> >> def onTimeout(): >> handleTimeoutCondition() >> >> delayedCall = reactor.callLater(timeoutLen, onTimeout) >> >> def success(value): >> handleSuccess() >> >> def nonTimeoutFailure(reason): >> handleErrorCondition() >> >> def cancelTimeout(val): >> if delayedCall.active(): >> delayedCall.cancel() >> >> d.addBoth(cancelTimeout) >> d.addCallback(success) >> d.addErrback(nonTimeoutFailure) >> >> ------------------------------------------------------- >> >First, there's a bug. Presumably, cancelTimeout should return val, >otherwise you swallow the result/failure, regardless of what actually >happens. Yes, indeed. > >What I don't see here is the most important part: presumably, >iReturnADeferred starts a long-running operation going, maybe a call to a >remote server. This code doesn't do anything to stop that operation, and >doesn't do anything to prevent an AlreadyCalledError happening in that code >when the long-running operation completes. Or does the mysterious >handleTimeoutCondition method do that? > >setTimeout doesn't provide that either -- but it was my understanding that >that was the main objection to it. Why is handling this yourself (which as >you demonstrate is error-prone!) an improvement over setTimeout? I apologise for the magicness implied by my example. What offends me about setTimeout is that it is a solution to a vague problem. It seems to me that there are two different issues here, and setTimeout doesn't solve either of them cleanly. The first is how to actually stop a long-running operation after a given amount of time. Indeed, this functionality must be provided by the framework author, perhaps exposing a public '.cancel()' method. There is no way to provide or deal with this behavior in a generic way. The other problem that I believe could be solved by a setTimeout-y method is how to indicate that the client isn't interested in the result of a long-running operation any more. I think this is an important distinction to make. It is a condition that happens at the interface between your code and iReturnADeferred(). Instead of trying to stop whatever iReturnADeferred() is doing, you come up with a way of saying, "after foo seconds, _i don't care about this result_ and this is what we should do..." > >> Deferreds are a _reactive_ tool, they fire in response to an event >> >> delayed calls are a _proactive_ tool, they cause events to fire >> >> Mixing the two together for the special case of dealing with a >> timeout is at best hairy and at worst ugly and confusing. > >I disagree. The event that Deferreds react to has to come from *somewhere*. Yes, but the deferred is a reaction to an event. My point was that you're adding event-creating code to an object whose job it is to react to events. All i'm saying is that this adds complexity. Without the current implementation of setTimeout, deferreds would be /completely reactive/, and i think we can find a way to make them _appear_ that way. > >What is problematic is that timeouts provide a *second* place that might >trigger the Deferred, in addition to whatever the usual place the creator >expects. With setTimeout, this means that AlreadyCalledErrors can arise >from calls to d.callback/d.errback where the original author didn't expect >them, just because a user of that module is setting timeouts to Deferreds. >I thought this was the primary objection to setTimeout? It's certainly the >biggest concern I have about it. indeed, i think this is the biggest problem I've faced when dealing with deferreds > >Ok, how about a compromise: if a creator a Deferred is able to cope with >timeouts, they should pass an "allowTimeouts=True" flag to the constructor. >Without it, the setTimeout method will raise an AssertionError. (For >backwards-compatibility, for a release it will merely raise a Warning rather >than an exception). I'm willing to implement this, and properly document >setTimeout in the Deferred HOWTO while I'm at it. This is now my preferred >option. Hrrrrrrrrrrrrrrrrrrrmmmmm. Not a baaaaaaad compromise, but there's something about it that doesn't sit with me. It seems that it would add complexity to the very elegant design of Deferreds. My overall qualm with setTimeout is that I think that if the API was in the state of stability it is now, and setTimeout was proposed, that it's current implementation would be rejected. I think that if we're going to provide this functionality, we should try to come up with as transparent a solution as we can. the following is an example of an idea i've been kicking around. Why can't we just ignore the deferred after the timeout happens? (I use the style of my example above because that is what i'm more comfortable with. If setTimeout would provide a similar functionality, then i'm all for it.) ----------------------------------------------------------------------- #!/usr/bin/env python2.3 from twisted.internet import defer, reactor iTimedOut = False defer.Deferred.debug = True def setTimeoutForDeferred(d, seconds): aNewDeferred = defer.Deferred() from twisted.internet import reactor def onTimeout(): # kill any further action on this deferred to avoid AlreadyCalledErrors d.callbacks = [] iTimedOut = True aNewDeferred.errback(defer.TimeoutError('deferred timed out')) delayed = reactor.callLater(seconds, onTimeout) def cb(value): if not iTimedOut: aNewDeferred.callback(value) def eb(reason): if not iTimedOut: aNewDeferred.errback(reason) def cancelTimeout(val): if delayed.active(): delayed.cancel() return val d.addCallbacks(cb, eb) d.addBoth(cancelTimeout) return aNewDeferred def iReturnADeferred(): # this is frameworky code d = defer.Deferred() reactor.callLater(5, d.callback, 'winnar!') return d def iSetATimeout(): # this represents client-programmer code origd = iReturnADeferred() d = setTimeoutForDeferred(origd, 2) def cb(val): # if we time out, this code should not be called print "hooray!iReturnADeferred returned: %s" % val def eb(reason): e = reason.trap(defer.TimeoutError) if e == defer.TimeoutError: print "woops! we timed out" else: print "uh-oh! we didn't time out and there was an error!" d.addCallbacks(cb, eb) return d if __name__ == "__main__": d = iSetATimeout() reactor.run() ----------------------------------------------------------------------- Again, this is just code to illustrate my idea. There's no reason why we have to hand the deferred that's returned by iReturnADeferred to the rest of our code. We can hold on to that one and can substitute our own deferred that is a bit more intelligent, it understands how to break the chain after a length of time has passed. The advantage here is that the author of iReturnADeferred doesn't need to plan specially for timeout conditions, and the client doesn't have to worry about AlreadyCalled errors (which to me are not something to pooh-pooh, I take them as a sign that there's something wrong with my design). >> I would like to recommend that we document a best-practice idiom >> in the deferred api documentation, and remove setTimeout. > >I don't believe it's easily written until you have a pretty advanced grasp >of both Deferreds and reactor.callLater. I don't see why allowing users of >Deferreds to just say "d.setTimeout(5)" is a bad thing? again, there's a difference between stopping a long-running or long-waiting operation, which is something that deferreds cannot handle at all, and indicating that after a certain amount of time, we don't care about the result of that operation and would like to consider it an erroneous condition. I don't think that setTimeout makes this distinction clear (which may, of course be a fault of the documentation/examples). Also, I want whatever solution is accepted to make it easier to avoid AlreadyCalled errors. > >setTimeout has the big big advantage that it's dead easy for *users* of >Deferreds to work with. The trouble with it is that it requires some work >from the *creators* of Deferreds to support safely -- but I see the same >flaw in every solution proposed, including your own of documenting and >entirely by-hand idiom. > okay, i can understand this, and can agree in some sense. >> spiv said: >> >> "In hindsight, I should've thought harder about the implications of >> setTimeout before I committed it... but now it's there, I don't like the >> idea of ripping it out unless we have a clearly better solution." > >I think my "allowTimeouts" proposal above is a clearly better solution. :) I don't like it. It just seems hacky. deferreds should handle this case, without any extra specification from the author, as I believe that my idea above would. > >I'm going to spend a little time pondering it, then I'll look at >implementing a patch and adding it to the bug. please, lets come to a consensus before committing anything. ;) Respectfully yours, Jonathan

3 3

Re: [Twisted-Python] addCallbacks issue
by Jonathan Simms April 18, 2004

April 18, 2004

I think that: d.addCallback(cb) d.addErrback(eb) is the more common idiom because it avoids the double-helix nature of deferred handling, opting instead for a single node-to-node chain of handling, which is less complex conceptually. --snip!-- > Another way of approaching the concept might be to say that each stage > in the chain has to deal with the errors in the chain before it - that > is, if you do: > > d.addCallbacks(cb, eb) > > then eb needs to handle any errors that arise from the data source, > while if you do: > > d.addCallback(cb) > d.addErrback(eb) > > then eb needs to handle any errors that arise from the data source, AND > any errors that arise from cb. > > I think that this chunk is the important concept to document. This explanation in addition to the imperitave example given by Stephen would ensure that this concept is documented and explained sufficiently. In fact, i'd never thought to consider deferreds as a try/except/else statement. very useful concept! It's funny, because when i read the original posting to this thread, I had the same confusion. "Waitasec, that errback /should/ get called!" this thread has helped to elucidate a subtle and important point about deferreds.

1 0