Mailman 3 January 2010 - Twisted

[Twisted-Python] WSGI Production ready
by adamjamesdrew same Jan. 20, 2010

Jan. 20, 2010

Is twisted WSGI production ready as compared to apache and mod_wsgi For a simple server with 4 cores would you simple start 4 instances and load balance between them? How would you recommend a production deploy of twisted wsgi?

3 2

[Twisted-Python] deferring result to PB a callRemote method
by Chris Laws Jan. 20, 2010

Jan. 20, 2010

> >* > Google is my friend but I could not find examples of this usage. Is *> >* there *> >* > any references to this usage in the twisted docs? *> >* *> >* *> >* Huh, surprisingly not in the primary PB docs. The Twisted “finger” *> >* tutorial *> >* does do this, though, if you read it carefully enough. See finger21.tac in *> >* < http://twistedmatrix.com/documents/current/core/howto/tutorial/pb.html>; *> >* the *&… [View More]

1 0

[Twisted-Python] [ANN] Foolscap-0.5.0 released
by Brian Warner Jan. 19, 2010

Jan. 19, 2010

I'm pleased to announce the release of Foolscap-0.5.0 . This is a fairly minor release: the biggest difference from the previous 0.4.2 is a change to the preferred main entry point: application code should use e.g. "from foolscap.api import Tub" instead of "from foolscap import Tub". A DeprecationWarning will be raised for code that continues to import from the main "foolscap" module, which will become an error in a future foolscap-0.6.0. (this was done to avoid annoying circular dependencies … [View More]within foolscap itself) In addition, many tools received minor improvements, and the "sets is deprecated in python2.6" warning was fixed. Bugs, docs, discussion: http://foolscap.lothar.com/trac Download: PyPI: http://pypi.python.org/pypi/foolscap/0.5.0 home page: http://foolscap.lothar.com/releases/foolscap-0.5.0.tar.gz Releases are signed with my GPG key 0x1514A7BD, recently renewed (expiration date extended to jan-2011, please re-download from keyservers) Foolscap is a Twisted-friendly remote object protocol, a descendant of Perspective Broker, with improved security properties, third-party references, adaptable serialization, remote logging, and other useful features. Please visit http://foolscap.lothar.com/trac for more details. have a importable day, -Brian * Release 0.5.0 (18 Jan 2010) ** Compatibility The wire format remains the same as in earlier releases. The preferred API import path has changed, see below. ** API changes: import statements, foolscap.api To reduce circular dependencies in Foolscap's internal code, a new "foolscap.api" module has been created. Applications should use: from foolscap.api import Tub instead of e.g. "from foolscap import Tub". Deprecation warnings will be raised for code which imports symbols directly from the "foolscap" module. These warnings will turn into errors in the 0.6.0 release. (see ticket #122 for details) The nearly-useless getRemoteURL_TCP() function was removed. ** setup.py is more windows-friendly The main setup.py script has been modified to use setuptools "entry_points=" on windows, which should help create runnable executables of "flogtool" and "flappserver", with proper extensions. Entry-point scripts are not used on non-windows platforms, but setuptools still creates fairly opaque executable scripts (which makes it hard to figure out that e.g. /usr/bin/flogtool wants to import the "foolscap" module). To get non-opaque scripts, install with "setup.py install --single-version-externally-managed". (#109) ** tool changes *** flappserver "flappserver create" now records the umask value from its environment, and uses it later when the server is started (since normally twistd resets the umask to a very restrictive value). A new --umask argument was added to override this. The server's base directory is chmod go-rwx to protect the private key from other users. The "flappserver start" command uses twisted.scripts.twistd.run(), instead of spawning an intermediate "twistd" process with os.execvp(). This should make things work better in environments where Twisted is not fully installed (especially on windows) and correctly launching "twistd" is non-trivial, such as when some other package is installing it as a setuptools dependency. "flappclient upload-file ~/foo.txt" will use os.path.expanduser() on the filename, even if your shell does not. This should make it easier to use from e.g. buildbot upload commands. (#134) *** logging The "flogtool dump" and "flogtool web-viewer" commands now have a --timestamps argument, which controls how timestamps are expressed (UTC vs localtime, ISO-9601, etc). The web-viewer HTML pages now have more timestamp and sorting options, and hyperlinks to select each. (#100) "flogtool web-viewer --open" will tell your local web browser to open to the correct page, using the Python stdlib "webbrowser" module. "flogtool dump" now emits a better error when told to open a missing file. *** examples Examples of running the Git version-control-system over a flappserver-based secure connection have been added to doc/examples/ . This enables remote-update authority to be expressed as a FURL with no other shell privileges. To accomplish the same with ssh "authorized_keys" command restrictions is annoying and error-prone. See doc/examples/git-proxy-flappclient for setup instructions. This will probably be simplified to a single "git-furl" executable in a later release. The xfer-client/xfer-server/command-client examples have been removed, obsoleted by the flappserver/flappclient tools. ** Other changes The DeprecationWarning for the obsolete "sets" module is now removed on python2.6 (#124) When a getReference() call fails because the remote Tub does not recognize the FURL, it now only emits the first two letters of the secret swissnum in the exception, instead of the whole thing. This reduces information leakage into e.g. stderr logs from a "flappclient --furlfile=X upload-file" command. DeadReferenceError now includes the remote tubid, interfacename, and remote method name of the message that was being sent when the lost connection was discovered, so log.err() calls which record a DeadReferenceError should include this information. This may make it easier to locate the code that provoked the error. [View Less]

1 0

[Twisted-Python] deferring result to PB a callRemote method
by Chris Laws Jan. 19, 2010

Jan. 19, 2010

I am writing a tool for work that will run various software development aids such as message capturing and diagnostic control of system processes. My current design strategy is to implement these aids as plugins (not twisted plugins) to a generic plugin runner rather than stand alone applications. The plugins are spawned as separate processes by the plugin-runner and communicate using Perspective Broker during the plugin installation and shutdown phases. Some plugins may take a while to … [View More]shutdown as they need to close sockets, finalize files, move large files, etc. I would like to instruct the plugin to shutdown and be told when it is finally ready to be shut down. To date all my callRemote methods have effectively returned immediately. For example: def remote_get_name(self): return self.name My question to the list was going to be: Is there a pattern/example I could follow where I could call plugin.callRemote("shutdown") on a plugin and not return a result to the plugin-runner until the plugin has completed all it's, potentially, long running activities? However, while I was trying to write a small code snippet that would demonstrate what I was wanting, I think I got it working. I think the simple answer to my question is to just return a deferred as the result to the callRemote("shutdown") method and trigger it as normal. Google is my friend but I could not find examples of this usage. Is there any references to this usage in the twisted docs? Below (and attached in case the formatting goes screwy) is a short example which emulates a long running shutdown activity performed by the plugin prior to shutdown. It seems to delay the processing of the callRemote("shutdown") result until the plugin has completed it's long running activity. I have omitted the separate process stuff as it didn't seem relevant for this snippet. Is the following code snippet the standard/normal way to defer the return result of a callRemote method call? If this is the normal way, how does triggering the deferred on the plugin (client) side also trigger the same/copy deferred returned to the plugin-runner (server)? Is this PB magic, somehow managing deferreds across the PB interface? Regards, Chris from twisted.internet import reactor, defer from twisted.spread import pb import datetime class PluginClient(pb.Referenceable): """ Plugin client interface exposed to PluginServer's """ def __init__(self, shutdownCallback): self.shutdownHandler = shutdownCallback def remote_shutdown(self): """ Instruct Plugin to shutdown """ print "plugin instructed to shutdown" d = defer.Deferred() self.shutdownHandler(d) return d class PluginServer(pb.Root): """ Plugin server interface exposed to PluginClient's """ def remote_set_client_perspective(self, pluginRef): print "plugin-runner got client reference" reactor.callLater(1.0, self.shutdown_plugin, pluginRef) def shutdown_plugin(self, pluginRef): def pluginShutdownCompleted(result, startTime): endTime = datetime.datetime.now() print "Plugin shutdown took %s to complete." % (endTime - startTime) return result print "plugin-runner asking plugin to shutdown" d = pluginRef.callRemote("shutdown") d.addCallback(pluginShutdownCompleted, datetime.datetime.now()) d.addCallback(self.shutdown) def shutdown(self, _): reactor.stop() def startPluginServer(port): """ Start a plugin communications server """ print "starting server" reactor.listenTCP(port=port, factory=pb.PBServerFactory(PluginServer()), interface='localhost') def startPluginClient(port, shutdownHandler): """ Start a plugin communications client """ def gotServerPerspective(serverPerspective, pluginPerspective): """ Give the plugin-runner this client's perspective """ serverPerspective.callRemote("set_client_perspective", pluginPerspective) return serverPerspective print "starting plugin" client = PluginClient(shutdownHandler) factory = pb.PBClientFactory() reactor.connectTCP(host='localhost', port=port, factory=factory) return factory.getRootObject().addCallback(gotServerPerspective, client) if __name__ == "__main__": port = 42155 def longRunningAction(d, countDown=10): """ Emulate long running shutdown activities """ print "shuting down in %i seconds" % countDown if countDown == 0: d.callback(True) else: countDown -= 1 reactor.callLater(1, longRunningAction, d, countDown) # start plugin-runner reactor.callWhenRunning(startPluginServer, port) # start plugin reactor.callLater(2.0, startPluginClient, port, longRunningAction) reactor.run() [View Less]

3 2

[Twisted-Python] Proxy support through urllib2?
by Gabriel Rossetti Jan. 19, 2010

Jan. 19, 2010

Hello everyone, Is it possible to use a proxy with Twisted? I can do this with urllib2 : proxy = urllib2.ProxyHandler({"http": "http://my.proxy.com:8080"}) opener = urllib2.build_opener(proxy) urllib2.install_opener(opener) will that suffice? I'd need SOCKSv5 and HTTP proxy support. Thanks, Gabriel

3 4

[Twisted-Python] Timeout with pb callRemote
by Allen Bierbaum Jan. 18, 2010

Jan. 18, 2010

I just tracked down a bug in one of our servers that uses twisted PB. The long and short of it was that the server made remote calls to clients that connected in and in some cases those clients would fall off the network (disconnected network cable, etc) but the server would not detect this. I tracked this down to TCP timeouts not telling twisted rapidly enough that the clients were offline. What I was going to use to solve this was to put a timeout on the remoteCall() by calling setTimeout … [View More]

2 1

[Twisted-Python] Plain windows sockets and twisted
by Daniel Griffin Jan. 18, 2010

Jan. 18, 2010

Hi, I am using twisted in a cross-platform manner. I have written both the client and server. The client is twisted and the server uses python threads. I am using the select reactor and plain old TCP connections. I encode my messages to JSON to send them. On Windows I get tons of un-clean connectionlost events, on Mac everything works perfectly. I am using the same data set in each case. If I use the threaded server on windows and the twisted client on mac then I still get tons of errors. 99%… [View More]

2 5

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Jan. 17, 2010

Jan. 17, 2010

1 0

[Twisted-Python] Accessing the request object from a different thread
by Red Daly Jan. 16, 2010

Jan. 16, 2010

I have an application that uses synchronous access to a MySQL database through SQLAlchemy. Unwilling to part with this synchronous access, our application uses threads to perform database I/O outside of the main Twisted thread. It would be nice to still have some access to the request object while processing a request in a new thread. Is there any thread-safe way to access the request object data? Here is a typical Resource subclass: def parseRequest(self, request, session): … [View More]

2 1

[Twisted-Python] Signalling twistd with SIGUSR1 to rotate also interrupts txAMQP connections
by Anthony Jan. 15, 2010

Jan. 15, 2010

We run some services that use txAMQP within twistd to connect to an AMQP bus. When we send twistd a SIGUSR1 to rotate its logs, it appears our application loses connection to the AMQP bus and needs to reconnect. This doesn't crash the app, but means the app may be offline momentarily when it should be available to process data. Is it possible that twistd is processing the SIGUSR1 and then passing it onto our app and txAMQP? We want to be able to force a log rotate without the entire app, or … [View More]

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Jan. 10, 2010

Jan. 10, 2010

2 1

Re: [Twisted-Python] A pseudo-deferred class that can be canceled
by Terry Jones Jan. 7, 2010

Jan. 7, 2010

>>>>> "Glyph" == Glyph Lefkowitz <glyph(a)twistedmatrix.com> writes: Glyph> This implies, to me, that the cancellation callback would be better Glyph> passed to addCallbacks(): effectively creating a third callback Glyph> chain going from invoker to responder rather than the other way Glyph> 'round as callbacks and errbacks do. After I went to bed I realized that someone is immediately going to want to have a cancel function that returns a deferred. And what … [View More]

2 1

[Twisted-Python] Assistance with using Twisted with "pyinstaller"
by Ken MacDonald Jan. 7, 2010

Jan. 7, 2010

Hi, we're trying to using pyinstaller to create single-directory distributions of our Twisted/python server on Linux. Wondering if anyone has accomplished this? I googled and checked the archives, but there seems to be no concrete information available. I'm real new to pyinstaller and not really familiar with twisted internals, so not sure where to start with this, particularly whether to package our_server.tac, or the twistd script, or the our_server.py as the starting point to bundle all of … [View More]

2 2

Re: [Twisted-Python] A pseudo-deferred class that can be canceled
by Terry Jones Jan. 7, 2010

Jan. 7, 2010

Hi Glyph It's very late here, so I'll limit myself to a few thousand lines of reply. >>>>> "Glyph" == Glyph Lefkowitz <glyph(a)twistedmatrix.com> writes: Glyph> On Jan 6, 2010, at 7:09 AM, Terry Jones wrote: Glyph> What I mean is, there are a lot of weird little edge-cases in how Glyph> multiple layers of the stack interact when they're dealing with a Glyph> shared Deferred, and if we're The above was truncated. Glyph> However, upon further inspection I … [View More]think that they key distinction Glyph> between what you've proposed and what I'm talking about is the Glyph> distinction between cancelling *one* layer of the callback chain and Glyph> cancelling *all* layers of the callback chain. Yes, that's right. I nearly made a diagram for people today, but didn't know if anyone would be interested. But here's one way to look at it. In today's deferred world, you have (in general) situations like this: func makes d -> c1 -> c2 -> c3 -> c4 -> c5 -> c6 -> c7 -> client -> c8 -> c9 I.e., the client makes a call, gets its hands on a deferred (which already has zero or more call/errbacks on its chain) and adds its own callbacks. At that point cancellation is very hard. Neither the client, nor the deferred itself, or the original function, can know how to cancel the operation. From the POV of the client and the deferred, that callback chain is just a bunch of indistinguishable functions. My ControllableDeferred class, if used by just the client, makes it possible for the client to cut the link between c7 and c8, either by iself (the client) calling the deferred it receives, which causes c8 to fire/err, or by deactivating it, thereby arranging that c8 is never called. So the ControllableDeferred in some sense introduces a cut point between two callbacks in the original chain. And the cut point is in a sensible logical position because the client added c8 and c9 and can be expected to know what to do to clean them up if it decides it's done waiting for the original d to fire. But the above picture is more uniform than the reality: it hides the fact that the callbacks were added to the deferred in groups (of zero or more). That is, the chain really looks like this: func makes d -> w1 -> w2 -> x1 -> x2 -> y1 -> y2 -> y3 -> client -> c8 -> c9 which is to say that the client in fact called function Y. Y called X. X called W, and W called something that returned a deferred. W then adds w1, and w2 to d and returns it to X. X adds x1 and x2 to d and returns it to Y. Y adds y1-3 and returns it to C. So you can imagine now that we insert a cut point at each logical boundary, and then the cancel information can flow back up the chain and each logical unit presumably knows how to discard / abort etc., whatever it may have in progress. That picture is mainly for clarity. I'm sure you're miles ahead already... Glyph> Your description (elided for brevity's sake) was very helpful. Glyph> You've got resources which your callbacks are consuming by way of Glyph> being "currently outstanding", and you want to be able to free Glyph> *those* resources, without necessarily worrying about However you were going to finish that sentence, I agree :-) I want to free the resources, and I want to be able to get on with whatever it is I'm supposed to be doing. >> Yes, agreed. I like the fact that the class is simple and that it deals >> with the client-side issues, allowing ignoring, timing out, early firing, >> etc. As you say, the much harder problem remains. But the harder problem >> is a bit less messy now (at least in my mind): it's "just" cancellation. >> Responsibilities are cleanly divided by my class - the client takes care of >> itself, and cancellation has *only* to deal with callbacks placed on a >> deferred that was generated by what the client called. Glyph> I don't think that you can completely separate the problems. You Glyph> seem to have a reasonable solution to the problem of one layer of Glyph> the Deferred stack, but once you're trying to deal with multiple Glyph> layers of the stack at once, interactions occur which can be Glyph> difficult to reconcile with the same API, many of which are already Glyph> documented in the ticket's discussion. It may be that there are interactions between W and Y (for example) in my above (2nd) diagram, but I expect that would be infrequent. E.g., W might decide to attach a callback to d after it has been returned to (and added to by) X, Y, etc. That seems to be a problem, but if W were to add those extra callbacks within another logical unit of the callback chain, it would be alerted of the cancellation in the normal fashion (twice). Make sense? >> Looked at from this POV, an approach to cancellation would be for code that >> is able to cancel operations it has begun to also provide a cancel method. >> One way to think about doing this would be to have the cancel method take a >> deferred as an argument. Glyph> This is a *very* interesting idea, although I don't like the API Glyph> that you propose for it. By separating the cancel method from the Glyph> Deferred itself, you remove the ability for a trivial client of that Glyph> Deferred to say "forget about it" without also maintaining a Glyph> reference to the thing that gave it the Deferred in the first place. I agree that's less desirable, but I'm not sure it's a necessary consequence of the approach. Or maybe it is. Today I modified my ControllableDeferred class to allow a cancelFunc argument. The __init__ is a tiny bit more clunky, but it has methods just like the old class, e.g. def callback(self, result): if not self._called: self._called = True if self.cancelFunc: self.cancelFunc(self._calld) defer.Deferred.callback(self, result) def deactivate(self): if not self._called: self._called = True if self.cancelFunc: self.cancelFunc(self._calld) This is just what you suggest - the cancel function is inside the deferred class (my ControllableDeferred is a subclass of Deferred, so that's literally true). The client, receiving an instance of this class, can just say "forget about it" and the cancel goes back to wherever it should go, if anywhere. So you can imagine writing a getPage function (or class) that returns ControllableDeferred instances. Calling the deferred or deactivating it would then result in a HTTPClientFactory instance calling transport.loseConnection. >> Something like my class could then hand the deferred back, effectively >> saying "my client is no longer interested in this deferred. You can >> call/errback it, or not, it makes no difference to us". If you've done >> that once, you can do it multiple times - by which I mean that I might >> write code that's a client of getPage, and getPage is a client of XXX, >> and XXX is a client of YYY, etc. Each could in turn pass the deferred it >> got back to the thing that created it. Glyph> This implies, to me, that the cancellation callback would be better Glyph> passed to addCallbacks(): effectively creating a third callback Glyph> chain going from invoker to responder rather than the other way Glyph> 'round as callbacks and errbacks do. Yes, I like that a lot, at least in a 5:30am superficial kinda way. A key difference between what I'd imagined and what you're suggesting is that in my approach, the cancel call goes directly to the thing (it would need to be a class instance, I suspect) that got the deferred. I.e. from my 2nd diagram, if the client calls cancel (or deactivate, as in the code), then the thing that added y1 to the chain is going to have its cancel method called (or some method that it asked to have called). So the control in a sense jumps back over y3, y2 and y1 to the root of the logical Y section. Your approach passes the signal back up the chain. Most secondary steps, like y3 and y2, will pass the call along without taking any action. But they don't have to, which is good. And the first callback of a logical unit can always do exactly what would have been done in my approach above. I think your approach is better. Glyph> I have stumbled in the direction of this thought a few times already Glyph> but this is the first time I've had a really clear grasp of how it Glyph> would work. Now I can see that each layer of the stack may have its Glyph> own resources that it might want to clean up... previously I thought Glyph> this could be done entirely with errbacks, but in this version, it Glyph> doesn't matter if the base deferred doesn't know how to kick off the Glyph> errback chain: all the resources on the *rest* of the callback chain Glyph> can be cleaned up. Yes. And the logical divisions of the call/errback chain are going to ignore each other in any case. Once a further-down-the-chain function has either called or deactivated the deferred (to put it simply - it's actually not just one deferred, at least in my implementation), it doesn't matter at all what the upstream (earlier) functions do - the result, if any, is not going through. Glyph> I'm going to need to figure out some good values for XXX and YYY Glyph> here in order to truly dispel the fog, though. I'm a bit foggy too. That's why I started playing with getPage to try to use a common example with at least a few levels of processing. But I didn't have time to think about it clearly. I wrote some foggy code, which I wont inflict on you. I'm pretty sure there's a clean solution in here though, that we can get to with a bit more back & forth. >> If there's no cancel method, then that's as far as can be gone with >> canceling. Glyph> This is one of the really tricky issues that has faced this feature Glyph> all along: what happens when some part of the chain involved doesn't Glyph> know what to do with a canceller? And your solution here seems like Glyph> it may be a very elegant hack: do exactly the same thing as other Glyph> parts of the callback chain. What I mean is: currently, if a Glyph> particular callback pair doesn't have a callback or an errback, the Glyph> behavior is to do nothing and pass the result through. Cancellation Glyph> could do exactly the same thing! Yes, that's great. That's *your* solution, btw :-) >> At that point the result is no longer passed because the first >> ControllableDeferred instance that's involved will effectively snip the >> link (or send an early result) in the sequence of steps that would >> originally have been done. Glyph> Severing the link seems like a problem though; if we do that, then Glyph> introducing any non-cancellation-aware Deferred - or callback, for Glyph> that matter - into a cancellation-aware pipeline will prevent Glyph> cancellations from propagating further up, and there should be no Glyph> reason to do that. Yes, agreed. >> And it keeps all code for doing cancellation out of the Deferred class. Glyph> Why is it that you want to keep the cancellation code out of Glyph> Deferred? It seems very useful to me to have one object that you Glyph> can say "stop" to, without necessarily knowing what's going on above Glyph> it or where it came from. Yes, I guess I didn't want to keep it out of there - especially since I already put it in today.... I guess what I really meant was that I wanted it to be clean / simple, because Deferreds are that way already (once you've spent a couple of years thinking about them). >> OK, sorry for so many words. I hope this seems like it's heading in a >> useful direction. It does to me. Glyph> Yes, this has been very useful. I hope we can distill this into Glyph> some useful conclusions soon. :) I think we can / will. It should be fairly easy to build an example based on getPage. I badly wanted to today, but we have a ton of stuff going on right now and I forced myself to put this aside for some hours. Terry [View Less]

1 0

Re: [Twisted-Python] A pseudo-deferred class that can be canceled
by Terry Jones Jan. 7, 2010

Jan. 7, 2010

>>>>> "Glyph" == Glyph Lefkowitz <glyph(a)twistedmatrix.com> writes: Glyph> And let's not forget, in long-running servers, it's possible to Glyph> "leak" operations, and the associated resources like file Glyph> descriptors. When the caller says that they're not interested in Glyph> the result of an operation any more, the library should clean up as Glyph> much as possible, otherwise you'll start running out of those Glyph> resources eventually. Yes, … [View More]

3 3

Re: [Twisted-Python] A pseudo-deferred class that can be canceled
by Terry Jones Jan. 7, 2010

Jan. 7, 2010

>>>>> "Glyph" == Glyph Lefkowitz <glyph(a)twistedmatrix.com> writes: Glyph> On Jan 5, 2010, at 4:53 PM, Terry Jones wrote: Glyph> I understand what you're saying: you're interested in a subset of Glyph> what I'm interested in, here. Yes. Glyph> The point I'm trying to make is that once you've gone to the trouble Glyph> of providing an API for *clients* of an operation to declare that Glyph> they are no longer interested in its results, then it's … [View More]wasteful for Glyph> the underlying engine to continue executing the operation only to Glyph> discard its result. Yes, agreed. But it's not at all clear to me how one should go about stopping things in general. That's a much harder task than ignoring a result or triggering a call/errback result yourself. My class just takes a couple of small client-side pieces out of the way. I know you know all this, I'm just trying to be clear for others / for the record, etc. Glyph> I think that coming up with a good API and semantics for "I no Glyph> longer care about the result here" has a huge amount of overlap with Glyph> this anyway. Hmmmm. I'm not sure about that. I agree if you s/huge amount/tiny/ :-) Naively perhaps, it seems like the client declaring they're no longer interested in a result is just a single bit of information. My class sits between the chain of steps (callbacks) leading to the production of a result and the chain of steps (callbacks) involved client-side in post-processing the result. It's a nice/simple way for the client to snip the overall chain. If I were going to try to push it further, I would start thinking about how to (figuratively) propagate this bit of information ("your result is no longer relevant / needed" back up the chain of as-yet unfired callbacks. If callbacks had access to that information, they could act accordingly. I don't think the running of the callback chain should be interrupted. As a small example, one callback function might do some setup operation (let's say open a file, or print an opening HTML tag) that a later one makes sure is wrapped up, using addBoth. It seems like there are two things that it would be nice to have: a way for a callback to know that its result will ultimately be ignored, and a mechanism for a callback to register a function to be called if an operation is canceled while it is running. In the case that an operation was canceled, its current callback would have its cancel function (if any) called, and subsequent callbacks would all find the resultWillBeIgnored to be True. Something like that. I'm happy to go into that discussion too, if you want. I do have a use for it as well (see below). Glyph> I grant that it may well be easier to implement without worrying Glyph> about the underlying operation though, and the semantics you've Glyph> defined by explicitly ignoring the actually-stopping case are much Glyph> simpler. Yes, agreed. I like the fact that the class is simple and that it deals with the client-side issues, allowing ignoring, timing out, early firing, etc. As you say, the much harder problem remains. But the harder problem is a bit less messy now (at least in my mind): it's "just" cancellation. Responsibilities are cleanly divided by my class - the client takes care of itself, and cancellation has *only* to deal with callbacks placed on a deferred that was generated by what the client called. Looked at from this POV, an approach to cancellation would be for code that is able to cancel operations it has begun to also provide a cancel method. One way to think about doing this would be to have the cancel method take a deferred as an argument. Something like my class could then hand the deferred back, effectively saying "my client is no longer interested in this deferred. You can call/errback it, or not, it makes no difference to us". If you've done that once, you can do it multiple times - by which I mean that I might write code that's a client of getPage, and getPage is a client of XXX, and XXX is a client of YYY, etc. Each could in turn pass the deferred it got back to the thing that created it. If there's no cancel method, then that's as far as can be gone with canceling. At that point the result is no longer passed because the first ControllableDeferred instance that's involved will effectively snip the link (or send an early result) in the sequence of steps that would originally have been done. Deferred producing code that's capable of cancellation might simply keep a dictionary of outstanding deferreds, and itself use a callback to remove things from its dictionary once a result is calculated and about to be passed on. My resizable dispatch queue code uses that approach. Or it could be more sophisticated and be a state machine, that transitions to new states as the callbacks that it put onto the deferred are called. Done this way, the business of cancellation is not handled by the Deferred class at all, which I think is a good thing because Deferreds are simple and don't have any idea of the operational connections between the functions in their call/errback chains. But the code that obtained the deferred and put callbacks onto it does. So, concretely, you could imagine something like this: - a getPageFactory class with a getPage method that returns a deferred, and a cancel method that accepts a deferred. - I'd add a cancelFunc argument (default None) to my ControllableDeferred class. If not None, that gets called with a deferred from getPage in case its client decides to ignore the operation or get an early result. That's pretty simple, I think. And it keeps all code for doing cancellation out of the Deferred class. Any client code that wants to be able to control the deferreds its getting can use a ControllableDeferred. And any code that produces deferreds and wants to offer the possibility of cancellation of underway operations can provide / advertise some kind of cancel function. That keeps cancellation split into logical pieces, wherein each piece doesn't know or care where its deferred came and safely knows that its result will be ignored. The important piece of the overall operation that's provided by ControllableDeferred is that it keeps all these pieces separate, only passing on results when they're ready. We'd generalize some of this, use an interface, etc. Glyph> But that also means that you still have to go to the trouble of Glyph> figuring out when you're no longer interested in the result any Glyph> more, but after going to the trobule ... what's the benefit? It's partly as you said in an earlier reply: releasing resources. It's also so that the client code can move on (see below). Glyph> I know what the use-cases are for stopping the underlying operation Glyph> (notifying the peer that you're not going to accept it, reclaiming Glyph> the resources); but if you're just going to let the operation Glyph> complete eventually anyway, why wouldn't you want to just finish Glyph> processing the result when it arrives regardless? Because it may not arrive at all, or it may arrive at a point when I'm no longer able to deal with a result. And because the non-arrival might be holding up some other part of a system (thereby consuming a resource). In the case I'm dealing with right now, I'm using my resizable dispatch queue code to process jobs that are submitted via a web form. To be more concrete, a user of my app can enter a Twitter user name and my code then goes and does a bunch of work. The processing of that Twitter user is a "job" here, and it may involve thousands of network API calls, to Twitter, to FluidDB, to other services. Because processing a job can create a lot of traffic, I use my dispatch queue class to keep a limit on the number of jobs that are being done in parallel. So there are a finite number of slots available for processing jobs. I know when a job is done because a deferred fires, and one of the callbacks in that deferred takes the job out of the list of currently outstanding jobs. It all works nicely. But the queue code just fires jobs and doesn't have any idea what they do. If they never fire, it continues to think the job is underway. This is generally true for any client that gets given a deferred - your continued processing is in the hands of the thing that made the deferred. In my case, a slot in a finite table is occupied. If I want to give up on it, I need to do some clean-up and may therefore not want the callbacks I added to the deferred to ever be called (e.g., data structs that could have processed the result may be changed or gone). I.e., I want my client to have more control over how / when / if its callbacks are called. That's totally not under my control if I call an unknown black-box deferred-producing code that may or may not call me back. In my particular case, it seems like I sometimes make a call to the Twitter API and that call never completes. I don't know what's going on, I suspect the connection is made and then held open and never processed by Twitter. So a deferred, or multiple deferreds, simply don't fire. I want to give up on them, clean up, and move on. From the client POV I don't know or care what went wrong with the call, I just know that it's time to do something else. I'm also interested in the wider problem (cancellation), because I'd like to tell a job to cancel itself. In my case this could in part take the form of a test inside a generator that was yielding deferreds to process the Twitter user. That would be a simple way to cancel the outstanding work for a job, even if I still couldn't reach out to the code that gave me a deferred to tell it to lose a connection etc. I hope that makes sense. OK, sorry for so many words. I hope this seems like it's heading in a useful direction. It does to me. Terry [View Less]

2 1

Re: [Twisted-Python] A pseudo-deferred class that can be canceled
by Terry Jones Jan. 5, 2010

Jan. 5, 2010

>>>>> "Tristan" == Tristan Seligmann <mithrandi(a)mithrandi.net> writes: Tristan> On Tue, Jan 5, 2010 at 6:29 AM, Terry Jones <terry(a)jon.es> wrote: >> - Once someone has made a function call, gotten a deferred, added >> �call/errbacks to it, etc., it's gone. It's in flight. Forget about it. Tristan> The thing is, this attitude isn't always reasonable. Yes, I was being a bit deliberately flippant, and was speaking in the context of deferred … [View More]

1 0

Re: [Twisted-Python] A pseudo-deferred class that can be canceled
by Terry Jones Jan. 5, 2010

Jan. 5, 2010

Hi Glyph I read through <http://twistedmatrix.com/trac/ticket/990> without trying to get all the details (some are not relevant to me, see below). I'll make a few comments here, then continue in the ticket, supposing there's interest. - It would simplify things to separate discussion of canceling deferreds from discussion of timeouts, which I think you were saying too. The timeout issue seemed to fade in that ticket, which is good. The stuff I posted is just about cancelation, nothing … [View More]

3 2

[Twisted-Python] lore2sphinx table handling?
by Kevin Horn Jan. 5, 2010

Jan. 5, 2010

Sphinx uses plain ReST tables, and from what I can tell, ReST has 4 (yes, 4!) ways of marking up tables: Grid tables http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#grid-tab… - ASCII-art style tables Simple Tables http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#simple-t… - A different style of ASCII-art table, simpler but less powerful than Grid Tables CSV Tables http://docutils.sourceforge.net/docs/ref/rst/directives.html#csv-table - data is provided … [View More]

5 6

[Twisted-Python] Some news about currently-pending code-reviews
by Tim Allen Jan. 5, 2010

Jan. 5, 2010

First, a public service announcement about code-reviews: I'm not a Twisted Developer, I'm just a guy who submitted a patch and got talked into helping out by reviewing other people's Twisted patches while I was waiting for other people to review mine. You don't have to be a Twisted expert to help out, and don't worry about whether you're experienced enough to spot every possible problem - most patches get reviewed *at least* two or three times by different people, and every problem you pick up … [View More]and the developer addresses is one less issue the next reviewer has to think about. As I've started reviewing over the past week or two, the developers whose code I've reviewed and the regulars in the #twisted IRC channel have all been super-helpful, and I highly recommend code-reviewing as a pleasant way to spend an idle afternoon. Everything you need to know to get started should be listed here: http://twistedmatrix.com/trac/wiki/ReviewProcess ---------------------------------------------------------------------- As I mentioned above, I've been doing Twisted code-reviews recently, and I've seen a few tickets in particular I thought the general Twisted community might like to know about, if not help out with. :) http://twistedmatrix.com/trac/ticket/3956 Add arraysize option to runQuery in adbapi - The "arraysize" property is the only writable cursor property defined by DBAPI2. Setting it can make the .fetchall() method faster with some DB modules, but Twisted's adbapi module offers no way to set it. - This ticket has a patch that adds an "cp_arraysize" keyword argument to ConnectionPool.runQuery() to set the .arraysize property, but this could interfere with DB modules that accept a "cp_arraysize" keyword parameter to cursor.execute(). - I looked up the API docs for as many Python DB modules as I could think of, and none of them attach any significance to a "cp_arraysize" keyword parameter. - However, at least cx_Oracle supports using keyword parameters for populating parameterised queries, like this: cursor.execute("select :name from dual;", name="Fred") So, my questions to the Twisted community are: - How many of you are using adbapi.ConnectionPool with cx_Oracle, or another DB module that supports arbitrary keyword arguments to cursor.execute? - Of those, how many of you are using parameterised queries or prepared statements with placeholders named "arraysize" or "cp_arraysize"? http://twistedmatrix.com/trac/ticket/4138 A fresh Twisted checkout should support "setup.py sdist" - Occasionally people ask about building RPMs of Twisted with "./setup.py bdist_rpm"; this ticket is the first step in getting that working - the rest of the process is #1696. - I'm not sure if it helps with other kinds of bdist, like "bdist_wininst" or "bdist_msi" - if it does, and you've been waiting for that, then you might want to help review the code and check it works properly on your platform! http://twistedmatrix.com/trac/ticket/4004 subunit reporter. FTW. - This ticket adds another test-reporter to Twisted Trial, one that outputs results in a format that can be parsed by the tools in the third-party "subunit" project, https://launchpad.net/subunit - Among other things, this would help integrate Python tests with test results from other systems (tap2subunit, subunit2junitxml), compare the results of different test-runs (subunit-diff), and enable cute GUI-based test-runners (subunit2gtk). - If subunit sounds like a tool you would find useful, you might want to help review the code! [View Less]

4 5

[Twisted-Python] Running twisted app in a different thread
by Devraj Mukherjee Jan. 5, 2010

Jan. 5, 2010

Hi all, I am new to Twisted, please be gentle :) The Twisted application I am dealing with is python-vnc-viewer (http://python-vnc.viewer.googlecode.com). We are attempting the RFB library implemented using Twisted in our application, to bring up a customer viewer. Our uses a threading model that is quite centric to our application. Is it possible to ask the Twisted application to run in a sea prate thread? Thanks. -- "The secret impresses no-one, the trick you use it for is everything" -… [View More]

3 2

Re: [Twisted-Python] A pseudo-deferred class that can be canceled
by Terry Jones Jan. 5, 2010

Jan. 5, 2010

Argh..... sorry, there was a trivial cut & paste error in the code I just posted. The errback method should of course be: def errback(self, fail=None): if not self._called: self._called = True defer.Deferred.errback(self, fail) Terry

1 0

Re: [Twisted-Python] A pseudo-deferred class that can be canceled
by Terry Jones Jan. 5, 2010

Jan. 5, 2010

Hi again Glyph, Andrew Below is a simpler and more flexible ControllableDeferred class. The idea is simple: you want a deferred that you can call/errback yourself, and which you can also deactivate so that it never fires. This is done by manipulating two normal deferreds (self, and a deferred received from maybeDeferred). This ControllableDeferred class sits in the middle and will (a) pass on the normal result (via chainDeferred) if it arrives first, or (b) if called by the user, fire with … [View More]

1 0

Re: [Twisted-Python] A pseudo-deferred class that can be canceled
by Terry Jones Jan. 5, 2010

Jan. 5, 2010

>>>>> "Andrew" == Andrew Bennetts <andrew(a)bemusement.org> writes: Andrew> I think the key thing is to remember there are two distinct parts: Andrew> 1) an operation (which eventually will have a result or failure) Andrew> 2) a result Andrew> A Deferred takes care of managing 2. It's a placeholder for that Andrew> result, and it can track callbacks that need to be called when it Andrew> arrives, etc. Andrew> We currently don't have any formal … [View More]

1 0

[Twisted-Python] A pseudo-deferred class that can be canceled
by Terry Jones Jan. 5, 2010

Jan. 5, 2010

I just wrote the below for fun. It's untested :-) It's a class that you initialize with a callable (f), and which gives you back a deferred (d) that will fire when f fires. Alternately, you can fire d yourself by calling 'callback' or 'errback' on the class instance and passing a value. That value is returned via d. The reason this is useful is that normally when you call a function that returns a deferred, you can't easily "cancel" the deferred because it is made and controlled by the … [View More]function you called. The callback or errback on the deferred is (very likely) going to be called at some point. OTOH, in the class below you get to "cancel" the deferred by triggering it yourself. If you fire d in this way, then when the original deferred fires (if ever), its result is ignored. I don't know why defer.Deferred.setTimeout is deprecated, but I guess it's partly to do with this control issue. If a timeout function you add to a deferred calls its errback (the default behavior), the deferred is still nevertheless going to be called by the code that created it. That code doesn't know and shouldn't have to check if a timeout happened from the POV of its caller. The setTimeout approach also only allows one timeout. With the below, any code can call the callback/errback function at any time. If you do want to cancel the deferred based simply on a timeout, you can do this (with as many different timeouts as you like - only the earliest will have any effect, supposing the original deferred still hasn't fired): cd = CancellableDeferred(func) deferred = cd.call() reactor.callLater(5, cd.errback, None) I post this partly because it seems cute, but more because it seems like issue that's probably been solved by many people. Is there a general utility in Twisted for doing this? A better way? Terry ---- from twisted.internet import defer class CancellableDeferred(object): def __init__(self, f, *args, **kw): self._f = f self._args = args self._kw = kw self._calld = None self.called = False def _fire(self, result): if not self.called: self.called = True self._calld.chainDeferred(self._resultd) def call(self): if self._calld: raise Exception('Already called.') self._resultd = defer.Deferred() self._calld = defer.maybeDeferred(self.f, *self._args, **self._kw) self._calld.addBoth(self._fire) return self._resultd def callback(self, value=None): if not self.called: self.called = True self._resultd.callback(value) def errback(self, value=None): if not self.called: self.called = True self._resultd.errback(value) [View Less]

3 2