[Twisted-Python] Synchronization techniques
Hello fellow twisters, I am developing a twisted application on Mac OS X (using Perspective Broker) in which the server component runs shell scripts to post orders to an accounting system. The accounting system is developed and maintained by a third party vendor, and is therefore beyond my control. For a given command the scripts must be executed in order, and all commands that affect the accounting system must be synchronized. For example: class AccountingService(pb.Root): def remote_post_order(self, data): self.lock.acquire() try: # put data in 'trans.dat' file # ... process = TwistedProcess() process.check_call(["load_dat.sh", "trans.dat"]) process.check_call(["post_trx.sh"]) finally: self.lock.release() The process.check_call() method has (nearly) the same semantics as the check_call() method in Python's built-in subprocess module, and the lock has (nearly) the same semantics as Python's built-in threading.Lock class. Unfortunately, I cannot use those built-in methods/classes because they do not play nice with Twisted--I get 'Interrupted system call' errors from subprocess.check_call(), and (I think) my server could deadlock if I used threading.Lock. So I developed a few helper classes to hack around these Twisted mis- features (questions follow the code): from Queue import Queue, Empty from twisted.internet.error import ProcessTerminated from twisted.internet.protocol import ProcessProtocol class ProcessError(Exception): pass class ProcessErrorHandler(ProcessProtocol): def __init__(self): self.errbuf = [] self.errors = [] self.waiting = True def errReceived(self, text): self.errbuf.append(text) def processEnded(self, status): if self.errbuf: self.errors.append("".join(self.errbuf)) if isinstance(status.value, ProcessTerminated): self.errors.append(status.getErrorMessage()) self.waiting = False def hasError(self): return bool(self.errors) def raiseError(self): raise ProcessError("\n".join(self.errors)) class TwistedProcess(object): def __init__(self, reactor=None): if reactor is None: from twisted.internet import reactor self.reactor = reactor def check_call(self, cmd): status = ProcessErrorHandler() self.reactor.spawnProcess(status, cmd[0], cmd, env=os.environ) while status.waiting: self.reactor.iterate(0.05) if status.hasError(): status.raiseError() class TwistedLock(object): """A lock object for Twisted The lock is instantiated in a released state. """ def __init__(self, reactor=None): if reactor is None: from twisted.internet import reactor self.reactor = reactor self.queue = Queue(1) self.release() def acquire(self): """Acquire the lock This method returns immediately when the lock is acquired. It "blocks" (i.e. iterates the reactor) indefinitely while it waits for the lock to be released. """ while True: try: self.queue.get_nowait() except Empty: # pseudo block (iterate the reactor) while waiting for the lock to be released self.reactor.iterate(0.05) else: break # the lock has been acquired def release(self): """Release the lock Raises Queue.Full if the lock was not acquired exactly once since it was last released. """ self.queue.put_nowait("TOKEN") Questions: 1. Is reactor.iterate() meant to be used this way (see TwistedLock.acquire() and TwistedProcess.check_call())? If not, what is the right way to do those things? I've read everything I can find on reactor.iterate() and while it seems to be the only way to do what I need to do, its use is often strongly discouraged. Why is it so difficult for twisted to handle non-async code? 2. Is my suspicion that the server could deadlock if I used threading.Lock correct? Note: the server is not multi-threaded (unless twisted is doing something with threads that I am not aware of). Example scenario: req 1: acquire lock req 1: spawn sub-process req 1: iterate reactor while waiting for process to complete req 2: acquire lock (block and wait) SERVER IS DEAD-LOCKED ? 3. Is there a better way to do what I'm trying to do here? (getting a new accounting system is not an option :) ~ Daniel
1. Is reactor.iterate() meant to be used this way (see TwistedLock.acquire() and TwistedProcess.check_call())? If not, what is the right way to do those things? I've read everything I can find on reactor.iterate() and while it seems to be the only way to do what I need to do, its use is often strongly discouraged. Why is it so difficult for twisted to handle non-async code?
You should never call reactor.iterate().
2. Is my suspicion that the server could deadlock if I used threading.Lock correct?
Yes. And the Queue model uses threading.Lock, so using it can also deadlock you.
3. Is there a better way to do what I'm trying to do here? (getting a new accounting system is not an option :)
twisted.internet.defer.DeferredLock and some of the related classes are what you ought to be using.
Thanks for the response Itamar,
1. Is reactor.iterate() meant to be used this way (see TwistedLock.acquire() and TwistedProcess.check_call())? If not, what is the right way to do those things? I've read everything I can find on reactor.iterate() and while it seems to be the only way to do what I need to do, its use is often strongly discouraged. Why is it so difficult for twisted to handle non-async code?
You should never call reactor.iterate().
There really needs to be a way to do this: d = getDeferred() result = reactor.waitFor(d) Or even better: d = getDeferred() result = d.waitFor() The waitFor() method returns the result of the deferred or raises an exception if the deferred failed. This would allow a much more intuitive way to do the same type of thing that inlineCallbacks() allows. Why is this not implemented?
2. Is my suspicion that the server could deadlock if I used threading.Lock correct?
Yes. And the Queue model uses threading.Lock, so using it can also deadlock you.
Right, but I'm using Queue.get_nowait() and Queue.put_nowait(), neither of which block, so that shouldn't be a problem.
3. Is there a better way to do what I'm trying to do here? (getting a new accounting system is not an option :)
twisted.internet.defer.DeferredLock and some of the related classes are what you ought to be using.
Unfortunately that only gets me half way there. DeferredLock.acquire () returns a deferred. How do I return the result of a deferred from a PB remote_xxx() function? Thanks, Daniel
On 4/3/07, Daniel Miller <daniel@keystonewood.com> wrote:
Thanks for the response Itamar,
1. Is reactor.iterate() meant to be used this way (see TwistedLock.acquire() and TwistedProcess.check_call())? If not, what is the right way to do those things? I've read everything I can find on reactor.iterate() and while it seems to be the only way to do what I need to do, its use is often strongly discouraged. Why is it so difficult for twisted to handle non-async code?
You should never call reactor.iterate().
There really needs to be a way to do this:
d = getDeferred() result = reactor.waitFor(d)
Or even better:
d = getDeferred() result = d.waitFor()
The waitFor() method returns the result of the deferred or raises an exception if the deferred failed. This would allow a much more intuitive way to do the same type of thing that inlineCallbacks() allows. Why is this not implemented?
Because it's not possible. -bob
On Tue, 2007-04-03 at 17:07 -0400, Daniel Miller wrote:
twisted.internet.defer.DeferredLock and some of the related classes are what you ought to be using.
Unfortunately that only gets me half way there. DeferredLock.acquire () returns a deferred. How do I return the result of a deferred from a PB remote_xxx() function?
Just return the Deferred from the remote_xxx() function. As for Queue - you may as well just use a list. But probably you want DeferredQueue.
On Apr 3, 2007, at 8:42 PM, Itamar Shtull-Trauring wrote:
On Tue, 2007-04-03 at 17:07 -0400, Daniel Miller wrote:
twisted.internet.defer.DeferredLock and some of the related classes are what you ought to be using.
Unfortunately that only gets me half way there. DeferredLock.acquire () returns a deferred. How do I return the result of a deferred from a PB remote_xxx() function?
Just return the Deferred from the remote_xxx() function.
Thanks, I didn't know I could return a deferred from a PB remote_xxx () method. That detail doesn't seem to be documented in the Perspective Broker documentation, which I have read quite a few times. Maybe this could be highlighted in the "Complete Example" [0] section of the PB usage documentation? The examples use the TwistedQuotes application, and the IQuoter.getQuote() method always returns a string (at least I couldn't find any implementations that return a deferred). However, that would require rewriting most if not all implementations of IQuoter to return deferred's and/or the code that calls IQuoter.getQuote(), which demonstrates the viral nature of twisted when used in conjunction with other libraries. So anyway, I rewrote my server-side library to do it the twisted way and return deferred's instead trying rig up some way of waiting for them. I still think it would be super useful to be able to pseudo- block on a deferred (i.e. allow the reactor to process other events while waiting for the deferred). It is very annoying to have to rewrite many layers of code when twisted is introduced into a program. I did find gthreadless.py, and maybe that would do it. Unfortunately discussion on that seems to have been dropped some time ago... For the record, I've included updated versions of the previously posted code below. I'd be happy if someone pointed out if I'm doing anything wrong (with respect to twisted) in this code. Thanks, Daniel [0] <http://twistedmatrix.com/projects/core/documentation/howto/pb- usage.html#auto1> ~~~~~~~~~ from twisted.internet.defer import Deferred, DeferredLock from twisted.internet.error import ProcessTerminated from twisted.internet.protocol import ProcessProtocol from twisted.python.failure import Failure class LockProxy(object): """A proxy that synchronizes method calls on a given subject""" def __init__(self, subject): self.lock = DeferredLock() self.subject = subject def __getattr__(self, name): value = getattr(self.subject, name) if hasattr(value, "im_self") and value.im_self is self.subject: def func(*args, **kw): def callback(lock, *args, **kw): try: result = value(*args, **kw) except: lock.release() raise if isinstance(result, Deferred): def release(arg, lock): lock.release() return arg result.addBoth(release, lock) else: lock.release() return result dfr = self.lock.acquire() dfr.addCallback(callback, *args, **kw) return dfr return func return value class ProcessError(Exception): pass class ProcessErrorHandler(ProcessProtocol): def __init__(self, deferred): self.deferred = deferred self.errbuf = [] self.errors = [] def errReceived(self, text): self.errbuf.append(text) def processEnded(self, status): if self.errbuf: self.errors.append("".join(self.errbuf)) if isinstance(status.value, ProcessTerminated): self.errors.append(status.getErrorMessage()) if self.errors: err = Failure(ProcessError("\n".join(self.errors))) self.deferred.errback(err) else: self.deferred.callback(self) self.deferred = None class TwistedProcess(object): def __init__(self, reactor=None, protocol=ProcessErrorHandler): if reactor is None: from twisted.internet import reactor self.reactor = reactor self.protocol = protocol def check_call(self, cmd): dfr = Deferred() protocol = self.protocol(dfr) self.reactor.spawnProcess(protocol, cmd[0], cmd, env=os.environ) return dfr
Daniel Miller wrote: -- snip --
So anyway, I rewrote my server-side library to do it the twisted way and return deferred's instead trying rig up some way of waiting for them. I still think it would be super useful to be able to pseudo- block on a deferred (i.e. allow the reactor to process other events while waiting for the deferred). It is very annoying to have to rewrite many layers of code when twisted is introduced into a program. I did find gthreadless.py, and maybe that would do it. Unfortunately discussion on that seems to have been dropped some time ago...
you can still use that method *if* you're willing to patch your cpython with the stackless changes and live with the requirement that this be done on all cpythons your code is going to run on. short of that, or using evil bytecode hacks, it's not really possible to implement - you need the ability to save, switch and restore stacks plus call into the middle of a function. while this can be done with generators, it is virtually equivalent to splitting your functions into a top/bottom half and is not directly supported, so it ends up looking ugly. if pypy ever gets to the point where it's usable in production (has the required c extensions ported and runs close in speed to cpython on non-trivial code), syntactic support for such usage could be added. please note that if you hadn't wanted to take advantage of being in an event loop, you could have easily stuck your code - unchanged - into a thread pool (via apis helpfully provided by twisted) and just written a few lines of code to adapt the interfaces. this has little to do with twisted, which - imo - does a wonderful job of making the (for some, abeit not me personally) difficult execution flow model palatable and clear. twisted might be a very good framework, but it isn't an async fairy that sneaks into your room at night and - if you've been good - rewrites your code to be all twisty and eventful and leaves a quarter under your pillow. with all that said, the sooner you start thinking of a program as a series of events being emitted and handled, the better for you and your twisted code - everything will become very natural at that point instead of looking like an idiosyncracy. while there are a few languages, like erlang for example, that express event loops with cooperative threading syntax (processes, messages and futures), i'm of the opinion that it's actually harmful to hide the event loop details from people who haven't started thinking that way yet - they should be simply syntactic sugar for when you've already grokked the model. it's inherently impossible to write good concurrent code without this insight, including preemptiver multithreading code as well. writing preemption and smp safe mt code boils down to understanding the chain of processing each 'event', where such event is access to shared data, goes through and the interaction between two or more such events descending down this chain - this is the only effective way i've found to think about memory barriers and cache coherency for example. just like grokking functional programming, grokking this will make you (the impersonal pronoun version) a better programmer (fwiw, it's made functional programming easier to get for me personally). cheers, -p
Paul G wrote:
while there are a few languages, like erlang for example, that express event loops with cooperative threading syntax (processes, messages and futures), i'm of the opinion that it's actually harmful to hide the event loop details from people who haven't started thinking that way yet - they should be simply syntactic sugar for when you've already grokked the model.
IIUC, the same holds true for inline callbacks, right? http://twistedmatrix.com/documents/current/api/twisted.internet.defer.html#i... -- Nicola Larosa - http://www.tekNico.net/ E-mail can be saved by your ISP or by the IT department in your corporation. Gmail, for example, saves everything, even if you delete it. -- Bruce Schneier, November 2006
Nicola Larosa wrote:
Paul G wrote:
while there are a few languages, like erlang for example, that express event loops with cooperative threading syntax (processes, messages and futures), i'm of the opinion that it's actually harmful to hide the event loop details from people who haven't started thinking that way yet - they should be simply syntactic sugar for when you've already grokked the model.
IIUC, the same holds true for inline callbacks, right?
http://twistedmatrix.com/documents/current/api/twisted.internet.defer.html#i...
nice, haven't seen that yet. it's not quite what i was agitating against - you still see an explicit async cutpoint and still have to realize what's going on. while i personally wouldn't recommend using this method all throughout, it does look to be very useful for situations where you'll never need to reuse the bottom half. -p
On 04:35 pm, daniel@keystonewood.com wrote:
On Apr 3, 2007, at 8:42 PM, Itamar Shtull-Trauring wrote:
On Tue, 2007-04-03 at 17:07 -0400, Daniel Miller wrote:
twisted.internet.defer.DeferredLock and some of the related classes are what you ought to be using.
Unfortunately that only gets me half way there. DeferredLock.acquire () returns a deferred. How do I return the result of a deferred from a PB remote_xxx() function?
Just return the Deferred from the remote_xxx() function.
Thanks, I didn't know I could return a deferred from a PB remote_xxx () method. That detail doesn't seem to be documented in the Perspective Broker documentation, which I have read quite a few times.
The PB documentation is not too great. Perhaps this paper would be helpful to you, if you haven't seen it: http://www.lothar.com/tech/papers/PyCon-2003/pb-pycon/pb.html#auto7 """ In addition, the remote method can itself return a Deferred instead of an actual return value. When that Deferreds fires, the data given to the callback will be serialized and returned to the original caller. """
Maybe this could be highlighted in the "Complete Example" [0] section of the PB usage documentation? The examples use the TwistedQuotes application, and the IQuoter.getQuote() method always returns a string (at least I couldn't find any implementations that return a deferred).
Please feel free to write some patches for the documentation, or open a doc bug describing this issue in more detail. It's definitely an under- documented feature of PB.
However, that would require rewriting most if not all implementations of IQuoter to return deferred's and/or the code that calls IQuoter.getQuote(), which demonstrates the viral nature of twisted when used in conjunction with other libraries.
I don't think that would really be the terrible burden that you suggest, considering the relatively small amount of tutorial documentation that implements or calls IQuoter. One could also propose a separate interface, IDeferredQuoter, to make the distinction clearer.
So anyway, I rewrote my server-side library to do it the twisted way and return deferred's instead trying rig up some way of waiting for them. I still think it would be super useful to be able to pseudo- block on a deferred (i.e. allow the reactor to process other events while waiting for the deferred). It is very annoying to have to rewrite many layers of code when twisted is introduced into a program. I did find gthreadless.py, and maybe that would do it. Unfortunately discussion on that seems to have been dropped some time ago...
I'm afraid that the feature you want doesn't make any sense and is, in a broad sense, impossible. There are some things like it which might be possible - for example, http://twistedmatrix.com/trac/ticket/2545 - but the reactor is not reentrant and in some sense could not be made reentrant. Consider this innocuous looking block of code: from twisted.internet.protocol import Protocol from make_believe import magicallyBlockOn class MagicalProtocol(Protocol): def dataReceived(self, data): commands = (self.buf + data).split() self.buf = commands[-1] for command in commands[:-1]: if command == 'QUIT': self.transport.loseConnection() return else: # Deferreds are hard, let's go shopping page = magicallyBlockOn(getPage("http://example.com/%s" % (command,))) self.transport.write("SIZE:"+len(page)) If you were using Deferreds to track the result of the 'getPage' operation, you could cancel the callbacks that write to the transport in connectionLost. However, with magical blocking, one dataReceived method might be interleaved with another. That means that every time through the loop, you have to check to see if the transport has already been disconnected - the code as presented here is buggy and will spuriously fail depending on the order of the connection being lost and the remote page being retrieved. In this example I've been careful to accumulate all the buffer- management and parsing logic at the top of the method, before any potential re-entrancy can happen, but other code (most everything in Twisted's existing protocol implementations, not to mention just about all application code) would not be so lucky. It might be quite feasible to implement a microthreaded runtime environment that lived on _top_ of Twisted and explicitly accounted for issues like these, but that would not really be substantially different than 2.5+inlineCallbacks.
For the record, I've included updated versions of the previously posted code below. I'd be happy if someone pointed out if I'm doing anything wrong (with respect to twisted) in this code.
Nothing immediately jumps out at me. I've had to write similar code in the past, though, and when I've had to do that, an explicit state machine for the state of the subprocess (or whatever asynchronous resource must be acquired) has been easier to deal with than a lock- oriented approach to it.
On Apr 4, 2007, at 1:43 PM, glyph@divmod.com wrote:
On 04:35 pm, daniel@keystonewood.com wrote:
On Apr 3, 2007, at 8:42 PM, Itamar Shtull-Trauring wrote:
On Tue, 2007-04-03 at 17:07 -0400, Daniel Miller wrote:
twisted.internet.defer.DeferredLock and some of the related classes are what you ought to be using.
Unfortunately that only gets me half way there. DeferredLock.acquire () returns a deferred. How do I return the result of a deferred from a PB remote_xxx() function?
Just return the Deferred from the remote_xxx() function.
Thanks, I didn't know I could return a deferred from a PB remote_xxx () method. That detail doesn't seem to be documented in the Perspective Broker documentation, which I have read quite a few times.
The PB documentation is not too great. Perhaps this paper would be helpful to you, if you haven't seen it:
http://www.lothar.com/tech/papers/PyCon-2003/pb-pycon/pb.html#auto7
""" In addition, the remote method can itself return a Deferred instead of an actual return value. When that Deferreds fires, the data given to the callback will be serialized and returned to the original caller. """
Thanks, I had not read that before, and that does explain it. Although it's such a short note that it could be easily missed. It would be much better to have a code example.
Maybe this could be highlighted in the "Complete Example" [0] section of the PB usage documentation? The examples use the TwistedQuotes application, and the IQuoter.getQuote() method always returns a string (at least I couldn't find any implementations that return a deferred).
Please feel free to write some patches for the documentation, or open a doc bug describing this issue in more detail. It's definitely an under-documented feature of PB.
I'll try to do that sometime soon.
However, that would require rewriting most if not all implementations of IQuoter to return deferred's and/or the code that calls IQuoter.getQuote(), which demonstrates the viral nature of twisted when used in conjunction with other libraries.
I don't think that would really be the terrible burden that you suggest, considering the relatively small amount of tutorial documentation that implements or calls IQuoter. One could also propose a separate interface, IDeferredQuoter, to make the distinction clearer.
Well of course it's no big deal to change IQuoter, but that specific case wasn't really my point. My point is that in the real world it's a BAD THING to have to rewrite perfectly good/working/tested code just because we want to use twisted. But this is exactly what happened to me when twisted was introduced into my project.
So anyway, I rewrote my server-side library to do it the twisted way and return deferred's instead trying rig up some way of waiting for them. I still think it would be super useful to be able to pseudo- block on a deferred (i.e. allow the reactor to process other events while waiting for the deferred). It is very annoying to have to rewrite many layers of code when twisted is introduced into a program. I did find gthreadless.py, and maybe that would do it. Unfortunately discussion on that seems to have been dropped some time ago...
I'm afraid that the feature you want doesn't make any sense and is, in a broad sense, impossible.
Maybe it's impossible for you to see things the way I see them because you have become drunk on Twisted Kool-Aide. In my specific case I am running twisted in a single-threaded environment with a single synchronized resource where each request that needs to access that resource must gain an exclusive lock before doing anything with it (a classic locking scenario). This is not "I'm being lazy and I do not want to learn how to use Deferreds." Rather, it is a requirement that is dictated by the system with which I am communicating (it does not support concurrent access through the API provided by the vendor). Thus, my code would be much simpler (both to write and maintain) if I had blockOn(), and it would not have any risk of dead lock or other such concurrency bugs. You might ask why I bother to use Twisted? -- Perspective Broker is the most elegant way I could find to call remote methods in Python. If it were abstracted from Twisted to become a fully synchronous library I would use that instead, but at this point it seems that if I want PB I am stuck with Twisted too. In short, this feature does "make sense" in my environment. Whether it's possible or not is another matter entirely.
There are some things like it which might be possible - for example, http://twistedmatrix.com/trac/ticket/2545 - but the reactor is not reentrant and in some sense could not be made reentrant.
Consider this innocuous looking block of code:
from twisted.internet.protocol import Protocol from make_believe import magicallyBlockOn
class MagicalProtocol(Protocol): def dataReceived(self, data): commands = (self.buf + data).split() self.buf = commands[-1] for command in commands[:-1]: if command == 'QUIT': self.transport.loseConnection() return else: # Deferreds are hard, let's go shopping page = magicallyBlockOn(getPage("http:// example.com/%s" % (command,))) self.transport.write("SIZE:"+len(page))
You're "Deferreds are hard" comment is an insult. You make it sound like I don't want to think. If I didn't want to think I wouldn't be be a software developer. This code obviously won't work because the getPage() has to wait and another dataReceived() call could come in with a QUIT command while the first one is still waiting for getPage(). Instead you'd need to accumulate the data in a buffer and then do your command processing logic after all data has been received--that is, if you want to use blockOn(getPage(...))--it probably wouldn't be the smartest way to do this because it would be nice to start getting pages before we receive all of the data. But this is just one case that doesn't work with blockOn(). I've never said that it would magically make every case easier, it just makes some less complicated cases very much simpler. Everything I've read about this issue suggests that the twisted developers just don't want to give people what they want because it would allow them to shoot themselves in the foot (for example, by using blockOn() in a multi-threaded environment or in inappropriate places such as the example above). But this is Python and we're consenting adults. With the proper warnings a feature like this could make twisted much more palatable for people with large existing projects that do not wish to rewrite entire sections of code just to work with deferreds. It would allow people to get the easiest thing working as quickly as possible, and then go back and write the optimal deferred implementation later when performance/blocking/etc. becomes an issue. Most people that would use blockOn() would probably use it in an entirely synchronous fashion where there would only be one deferred being processed at any given time. In these cases blockOn() would work just fine (if inefficiently). From your point of view that probably totally defeats the purpose of using twisted, but as I have pointed out above there are other useful features in twisted beside its deferred mechanism (PB). The concept that I am thinking of seems entirely possible, although I am sure it would require rewriting existing reactor implementations. However, in the long run that seems like a small cost if twisted could be more widely adopted because it would play nicer with existing non-async code.
If you were using Deferreds to track the result of the 'getPage' operation, you could cancel the callbacks that write to the transport in connectionLost. However, with magical blocking, one dataReceived method might be interleaved with another. That means that every time through the loop, you have to check to see if the transport has already been disconnected - the code as presented here is buggy and will spuriously fail depending on the order of the connection being lost and the remote page being retrieved.
In this example I've been careful to accumulate all the buffer- management and parsing logic at the top of the method, before any potential re-entrancy can happen, but other code (most everything in Twisted's existing protocol implementations, not to mention just about all application code) would not be so lucky.
It might be quite feasible to implement a microthreaded runtime environment that lived on _top_ of Twisted and explicitly accounted for issues like these, but that would not really be substantially different than 2.5+inlineCallbacks.
For the record, I've included updated versions of the previously posted code below. I'd be happy if someone pointed out if I'm doing anything wrong (with respect to twisted) in this code.
Nothing immediately jumps out at me.
Thanks for the review. ~ Daniel
So anyway, I rewrote my server-side library to do it the twisted way and return deferred's instead trying rig up some way of waiting for them. I still think it would be super useful to be able to pseudo- block on a deferred (i.e. allow the reactor to process other events while waiting for the deferred). It is very annoying to have to rewrite many layers of code when twisted is introduced into a program. I did find gthreadless.py, and maybe that would do it. Unfortunately discussion on that seems to have been dropped some time ago...
I'm afraid that the feature you want doesn't make any sense and is, in a broad sense, impossible.
Maybe it's impossible for you to see things the way I see them because you have become drunk on Twisted Kool-Aide. In my specific case I am running twisted in a single-threaded environment with a single synchronized resource where each request that needs to access that resource must gain an exclusive lock before doing anything with it (a classic locking scenario). This is not "I'm being lazy and I do not want to learn how to use Deferreds." Rather, it is a requirement that is dictated by the system with which I am communicating (it does not support concurrent access through the API provided by the vendor).
We have a very similar situation in IPython. We have a twisted server that is managing access to a bunch of other processes (talking over PB) that each don't support concurrent access.
Thus, my code would be much simpler (both to write and maintain) if I had blockOn(), and it would not have any risk of dead lock or other such concurrency bugs.
I do disagree with this. In our case, we simply use a FIFO queue based on Deferreds to manage multiple requests to a single resource that does not support concurrent access. It is very simple and explicit. Even if you had blockOn() you would still have to have queue to manage the multiple requests, right? I don't at all see why it would be simpler if blockOn existed.
You're "Deferreds are hard" comment is an insult. You make it sound like I don't want to think. If I didn't want to think I wouldn't be be a software developer.
Just for the record: I think Deferreds _are_ hard, even damn hard - at least if you want to do something non-trivial that has robust error handling. Some of the callback/errback decision trees we have in our code are insane and took days to get right and test fully. The point is that doing these complex things would be even more insane without twisted.
Everything I've read about this issue suggests that the twisted developers just don't want to give people what they want because it would allow them to shoot themselves in the foot (for example, by using blockOn() in a multi-threaded environment or in inappropriate places such as the example above).
Personally, I would love a completely robust blockOn to exist. I would use it in certain cases. But the bottom line is that many people have tried to do this, but that have all failed. Their collective wisdom (with which I agree) is that it can't be done without completely redesigning twisted's internals - if at all - without breaking the overall programming model in twisted. Most of us are not ready to throw the baby out with the bathwater.
Most people that would use blockOn() would probably use it in an entirely synchronous fashion where there would only be one deferred being processed at any given time. In these cases blockOn() would work just fine (if inefficiently). From your point of view that probably totally defeats the purpose of using twisted, but as I have pointed out above there are other useful features in twisted beside its deferred mechanism (PB).
I thought the same thing when I first wrote the version of blockOn that we tried in IPython. As time went along though, I quickly discovered that these assumptions are simply wrong. It doesn't work just fine.
The concept that I am thinking of seems entirely possible, although I am sure it would require rewriting existing reactor implementations. However, in the long run that seems like a small cost if twisted could be more widely adopted because it would play nicer with existing non-async code.
Currently my own gut feeling is that there is something intrinsic to Twisted's asynchronous programming model that makes a construct like blockOn impossible to implement (even if you re-wrote a reactor) without introducing new types of deadlocks and indeterminant behavior into the system. Thus it is not simply an issue of us not being smart enough to figure out how to do it. It seems more fundamental than that. Actually, I think I see why (at least in part) it is problematic. If blockOn exists, the following can happen: def compute(a, b): d = a.computeSomething() # Lets say that b.state = 1 as of here result = blockOn(d) # Because the reactor just ran for an iondeterminant amount of time, b.state could have # changed - or maybe not. # Thus the return value of this function is essentially a random result. return b.state + result To eliminate such indeterminacies, new constructs would need to be created to handle such situations: def compute(a, b): d = a.computeSomething() # Lets say that b.state = 1 as of here acquire(b.state) # This gets a lock on b.state result = blockOn(d) result += b.state # b.state =1 still release(b.state) # release the lock return result But now you can get deadlocks as blockOn switches to another code path. Things start to look just like threads at this point and the Kool-Aide starts to taste bitter. Brian
On 04:25 am, daniel@keystonewood.com wrote:
On Apr 4, 2007, at 1:43 PM, glyph@divmod.com wrote:
On 04:35 pm, daniel@keystonewood.com wrote:
On Apr 3, 2007, at 8:42 PM, Itamar Shtull-Trauring wrote:
On Tue, 2007-04-03 at 17:07 -0400, Daniel Miller wrote:
Well of course it's no big deal to change IQuoter, but that specific case wasn't really my point. My point is that in the real world it's a BAD THING to have to rewrite perfectly good/working/tested code just because we want to use twisted. But this is exactly what happened to me when twisted was introduced into my project.
Hmm. Well, I don't know about the "real world" - haven't visited in a while - but in the magical fairy kingdom where *I* live, it is generally considered a good idea to globally consider the implications of a new programming model on "good/working/tested code". Networks and concurrency, in particular, have this pesky habit of introducing entirely new error conditions into previously "working" code, breaking all of its assumptions. This isn't specific to Twisted, but Twisted does deal with networks and concurrency quite a bit.
I'm afraid that the feature you want doesn't make any sense and is, in a broad sense, impossible.
Maybe it's impossible for you to see things the way I see them because you have become drunk on Twisted Kool-Aide.
You are making it sound like you are bringing a fresh new idea to the discussion here, which I've never heard before and am unwilling to consider because I'm inflexible in my thinking. That's not what's happening. This is a frequently-debunked and well-understood issue in the Twisted community. It seems to come up about once per year. See my now apparently prescient blog post as of last April: http://glyf.livejournal.com/40037.html
In my specific case I am running twisted in a single-threaded environment with a single synchronized resource where each request that needs to access that resource must gain an exclusive lock before doing anything with it (a classic locking scenario). This is not "I'm being lazy and I do not want to learn how to use Deferreds." Rather, it is a requirement that is dictated by the system with which I am communicating (it does not support concurrent access through the API provided by the vendor). Thus, my code would be much simpler (both to write and maintain) if I had blockOn(), and it would not have any risk of dead lock or other such concurrency bugs.
You're confusing two things here. On the one hand, you want mutual exclusion for an external resource, which blocks. On the other, you want semantics for implementing that mutual exclusion via blocking in your own process. The former, as you have already demonstrated, can be implemented without the latter. The question is, would your code actually be simpler to write and to maintain if you had blockOn? Nothing you've said so far indicates that it would actually be more maintainable, and I've tried (although perhaps failed) to illustrate the high cost of the *apparent* simplicity at the moment of implementation. It strikes me that the process actually making the foreign API call could just block "for real" which would solve the mutual exclusion issue - callers into the PB layer would appear to be getting concurrent access, but responses would be implicitly queued up. Another solution here would be for Twisted to have a nice convenience API for dispatching tasks to a process pool. Right now setting up a process pool is conceptually easy but mechanically difficult; you have to do a lot of typing and make a lot of irrelevant decisions (AMP or PB or pickle? stdio or sockets?).
You might ask why I bother to use Twisted? -- Perspective Broker is the most elegant way I could find to call remote methods in Python. If it were abstracted from Twisted to become a fully synchronous library I would use that instead, but at this point it seems that if I want PB I am stuck with Twisted too.
This is another area where the feature request doesn't quite make sense. It would be possible to implement something that looked kinda-sorta like PB, which dealt with a single request/response pair over a single socket, in an apparently synchronous and blocking manner. However, PB itself is a fully symmetrical protocol where the server can send messages to the client at any time, so a full PB implementation is not really possible when any message can be replied to with a "busy, poor implementation doesn't allow me to answer that message in this state" error. For a lot of PB applications - those it was designed for, for example, online games - you absolutely need full two-way communication.
In short, this feature does "make sense" in my environment. Whether it's possible or not is another matter entirely.
I am still not convinced. You can feel free to stop trying to convince me though, or you can write a patch which we can then discuss.
You're "Deferreds are hard" comment is an insult. You make it sound like I don't want to think.
You've also insulted me by implication of not living in the "real world" and being "drunk" on "Kool-Aide [sic]". I think that this feature is a symptom of muddy thinking, since I've seen it dozens of times before, and I'm not going to apologize to you for thinking that. The difference between the jabs we're trading here is that I'm not using any software that *you* wrote, and I'm not insulting you at the same time I'm posting to a mailing list for that software while demanding impossible features.
If I didn't want to think I wouldn't be be a software developer.
I don't think that you "don't want to think", I think that you're mistaken. However, if indeed you didn't want to think, this is hardly a defense, as you'd clearly not be alone in the software development profession, such as it is. c.f. http://worsethanfailure.com/
This code obviously won't work because the getPage() has to wait and another dataReceived() call could come in with a QUIT command while the first one is still waiting for getPage(). Instead you'd need to accumulate the data in a buffer and then do your command processing logic after all data has been received--that is, if you want to use blockOn(getPage(...))--it probably wouldn't be the smartest way to do this because it would be nice to start getting pages before we receive all of the data. But this is just one case that doesn't work with blockOn(). I've never said that it would magically make every case easier, it just makes some less complicated cases very much simpler.
It makes some cases appear simpler *at the expense* of breaking lots of other, correctly-written code, which depends on not having 20 levels of naive "blockOn" calls above them on the stack. It's analogous to how there are restrictions on "user code" in UNIX and you're not allowed to handle interrupts directly because the point of the kernel is to allow multiple processes to run at the same time. The original point of Twisted was a high degree of frustration that so many libraries for speaking protocols implemented their own, incompatible event-loops.
Everything I've read about this issue suggests that the twisted developers just don't want to give people what they want because it would allow them to shoot themselves in the foot (for example, by using blockOn() in a multi-threaded environment or in inappropriate places such as the example above). But this is Python and we're consenting adults. With the proper warnings a feature like this could make twisted much more palatable for people with large existing projects that do not wish to rewrite entire sections of code just to work with deferreds. It would allow people to get the easiest thing working as quickly as possible, and then go back and write the optimal deferred implementation later when performance/blocking/etc. becomes an issue.
I agree that it would be nice to allow programs to get on the Twisted bandwagon slowly, and to integrate more cleanly with foreign concurrency mechanisms like microthreads and database transactions. This is exactly what Jim Fulton is working on with the multi-reactor stuff for ZEO. You can't have one reentrant reactor, but you *can*, at least conceptually, have one reactor start another reactor and wait for it to complete a particular operation. If you'd like to help other projects gradually adapt to Twisted, perhaps you would like to contribute something to ticket #2545. To follow my earlier analogy, this is like the hypervisor and user-mode- kernel facilities in various UNIXes; if you're not allowed to do something in the kernel, it's OK to start your own kernel.
Most people that would use blockOn() would probably use it in an entirely synchronous fashion where there would only be one deferred being processed at any given time. In these cases blockOn() would work just fine (if inefficiently). From your point of view that probably totally defeats the purpose of using twisted, but as I have pointed out above there are other useful features in twisted beside its deferred mechanism (PB).
... and as *I've* pointed out above, PB is only possible _because_ of Twisted's event loop. In fact Deferreds were directly extracted from PB - originally every PB method had "pbcallback" and "pberrback" keyword arguments, and the Deferred class was the encapsulation of that so that PB methods could be easily chained and their results passed to other systems.
The concept that I am thinking of seems entirely possible, although I am sure it would require rewriting existing reactor implementations. However, in the long run that seems like a small cost if twisted could be more widely adopted because it would play nicer with existing non- async code.
If you want to try and go implement this, you can discover just how small the cost is :). If, in the course of implementing such a thing, you manage to get clean, coherent semantics for "blockOn", and it passes the full test suite (etc etc) I would not reject such a thing out of hand. I am suggesting that it is impossible to get coherent semantics for blockOn, and if you submit an implementation I'll point out the specific brokenness of a particular approach, but my main point is that it's impossible because of specific problems, not that it's unacceptable.
On Apr 5, 2007, at 5:53 AM, glyph@divmod.com wrote:
I'm afraid that the feature you want doesn't make any sense and is, in a broad sense, impossible.
Maybe it's impossible for you to see things the way I see them because you have become drunk on Twisted Kool-Aide.
You are making it sound like you are bringing a fresh new idea to the discussion here, which I've never heard before and am unwilling to consider because I'm inflexible in my thinking. That's not what's happening.
I'm sorry I wrote that...it was inflammatory and did not bring any value to the conversation. Please accept my apology.
In my specific case I am running twisted in a single-threaded environment with a single synchronized resource where each request that needs to access that resource must gain an exclusive lock before doing anything with it (a classic locking scenario). This is not "I'm being lazy and I do not want to learn how to use Deferreds." Rather, it is a requirement that is dictated by the system with which I am communicating (it does not support concurrent access through the API provided by the vendor). Thus, my code would be much simpler (both to write and maintain) if I had blockOn(), and it would not have any risk of dead lock or other such concurrency bugs.
You're confusing two things here.
On the one hand, you want mutual exclusion for an external resource, which blocks.
On the other, you want semantics for implementing that mutual exclusion via blocking in your own process.
The external "blocking" resource is just a shell script that takes some time t run. It does not acquire any shared resources that would result in dead lock and it will always return (maybe with an error, but it will return) unless something terrible happens (e.g. plug is pulled on server, fire, etc.).
The former, as you have already demonstrated, can be implemented without the latter. The question is, would your code actually be simpler to write and to maintain if you had blockOn? Nothing you've said so far indicates that it would actually be more maintainable, and I've tried (although perhaps failed) to illustrate the high cost of the *apparent* simplicity at the moment of implementation.
It would be more maintainable because it would look just like normal sequential python code: lock.acquire() # uses blockOn() to acquire a DeferredLock try: process.check_call(['script1.sh']) # uses blockOn(spawnProcess (...)) internally process.check_call(['script2.sh']) finally: lock.release() This is very simple and very easy to maintain. It could be written with inlineCallbacks fairly easily as well: yield lock.acquire() try: yield process.check_call(...) yeild process.check_call(...) finally: lock.release() That's pretty nice (so nice I might just rewrite my code that way). My complaint is that the code must have knowledge of the twisted environment (why else would it yield the result of process.check_call ()?). I do not really see the conceptual difference between these two code blocks except one yields to and one calls into the reactor event loop. Is there some other inherent problem with the first example? Of course you need to make sure that the code inside the try/finally block does not try to acquire the lock again, but that's a basic concurrency problem which can even happen in the next example. Moving on, in a fully deferred world we have this: def deflock(func, *args, **kw): def callback(lock, *args, **kw): try: result = func(*args, **kw) except: lock.release() raise if isinstance(result, Deferred): def release(arg, lock): lock.release() return arg result.addBoth(release, lock) else: lock.release() return result dfr = self.lock.acquire() dfr.addCallback(callback, *args, **kw) return dfr def dostuff(): def deferproc(result, cmd): return process.check_call(cmd) # returns a deferred dfr = deferproc(None, ["script1.sh"]) dfr.addCallback(defproc, ["script2.sh"]) return dfr dfr = deflock(dostuff) ... you get the picture. Notice the code to acquire/release the lock--there are three different calls to lock.release() in there, and they all must be carefully sorted out to make sure that exactly one of them will be called in any given scenario--that's hard to maintain.
It strikes me that the process actually making the foreign API call could just block "for real" which would solve the mutual exclusion issue - callers into the PB layer would appear to be getting concurrent access, but responses would be implicitly queued up.
Right, that would work and that's exactly what subprocess.check_call () (the real python built-in version) would do. Unfortunately twisted does not work with the subprocess module--spawnProcess() is the only alternative I found that actually works and that means I have to use a deferred.
Another solution here would be for Twisted to have a nice convenience API for dispatching tasks to a process pool. Right now setting up a process pool is conceptually easy but mechanically difficult; you have to do a lot of typing and make a lot of irrelevant decisions (AMP or PB or pickle? stdio or sockets?).
That sounds nice.
You might ask why I bother to use Twisted? -- Perspective Broker is the most elegant way I could find to call remote methods in Python. If it were abstracted from Twisted to become a fully synchronous library I would use that instead, but at this point it seems that if I want PB I am stuck with Twisted too.
This is another area where the feature request doesn't quite make sense. It would be possible to implement something that looked kinda-sorta like PB, which dealt with a single request/response pair over a single socket, in an apparently synchronous and blocking manner. However, PB itself is a fully symmetrical protocol where the server can send messages to the client at any time, so a full PB implementation is not really possible when any message can be replied to with a "busy, poor implementation doesn't allow me to answer that message in this state" error.
I understand that PB is fully symmetrical. In my case I am only using half (client makes request, server responds). Would it make sense to relax the constraints when PB is used in this way?
Everything I've read about this issue suggests that the twisted developers just don't want to give people what they want because it would allow them to shoot themselves in the foot (for example, by using blockOn() in a multi-threaded environment or in inappropriate places such as the example above). But this is Python and we're consenting adults. With the proper warnings a feature like this could make twisted much more palatable for people with large existing projects that do not wish to rewrite entire sections of code just to work with deferreds. It would allow people to get the easiest thing working as quickly as possible, and then go back and write the optimal deferred implementation later when performance/blocking/etc. becomes an issue.
I agree that it would be nice to allow programs to get on the Twisted bandwagon slowly, and to integrate more cleanly with foreign concurrency mechanisms like microthreads and database transactions. This is exactly what Jim Fulton is working on with the multi-reactor stuff for ZEO. You can't have one reentrant reactor, but you *can*, at least conceptually, have one reactor start another reactor and wait for it to complete a particular operation. If you'd like to help other projects gradually adapt to Twisted, perhaps you would like to contribute something to ticket #2545.
This looks very interesting. I'll try to help out with this effort if I can find some time. Thanks for taking time to read my ramblings and understand the problems that I am having (even if we don't quite agree on the simplest solutions). Your input is valuable, and I am indebted to you for providing free support in your spare time. ~ Daniel
Daniel Miller wrote:
I'm sorry I wrote that...it was inflammatory and did not bring any value to the conversation. Please accept my apology.
That really shows some class. Thanks for saying it.
Another solution here would be for Twisted to have a nice convenience API for dispatching tasks to a process pool. Right now setting up a process pool is conceptually easy but mechanically difficult; you have to do a lot of typing and make a lot of irrelevant decisions (AMP or PB or pickle? stdio or sockets?).
That sounds nice.
There's been a lot of interest in this, including from Glyph Lefkowitz. The taskqueue API in my twisted-goodies package could be easily adapted to do it, using an implementation of IWorker that works with python interpreter processes rather than threads. (See http://foss.eepatents.com/api/twisted-goodies/taskqueue.workers.html.) Note that you can queue to remote workers with the API just as it stands now, see RemoteCallWorker and RemoteInterfaceWorker. If anyone wants to take a shot at a process pool using a ProcessWorker, I'll be happy to help. And if there's interest in having the result go into Twisted proper under an MIT license, I remain open to that. Best regards, Ed
On 01:53 pm, daniel@keystonewood.com wrote:
On Apr 5, 2007, at 5:53 AM, glyph@divmod.com wrote:
I'm afraid that the feature you want doesn't make any sense and is, in a broad sense, impossible.
Maybe it's impossible for you to see things the way I see them because you have become drunk on Twisted Kool-Aide.
You are making it sound like you are bringing a fresh new idea to the discussion here, which I've never heard before and am unwilling to consider because I'm inflexible in my thinking. That's not what's happening.
I'm sorry I wrote that...it was inflammatory and did not bring any value to the conversation. Please accept my apology.
Thank you, I very much appreciate the sentiment! I'm glad to see you also only quoted the actually useful / productive parts of my response too ;).
The external "blocking" resource is just a shell script that takes some time t run. It does not acquire any shared resources that would result in dead lock and it will always return (maybe with an error, but it will return) unless something terrible happens (e.g. plug is pulled on server, fire, etc.).
I thought I understood what was going on, but now I'm confused again. Why do you need mutual exclusion at all if it doesn't acquire any shared resources? Couldn't you just run it concurrently?
It would be more maintainable because it would look just like normal sequential python code:
Yes, it would *look* like sequential python code. But it wouldn't be :). There's a heck of a lot that can happen in acquire(); your whole application could run for ten minutes on that one line of code. Worst of all, it would only happen in extreme situations, so testing or debugging issues that are caused by it becomes even more difficult. <snip blocking code>
This is very simple and very easy to maintain. It could be written with inlineCallbacks fairly easily as well:
yield lock.acquire() try: yield process.check_call(...) yeild process.check_call(...) finally: lock.release()
That's pretty nice (so nice I might just rewrite my code that way).
I'm glad you think so. I was originally not too happy about inlineCallbacks (its predecessors did not do so well) but I keep seeing examples like this which it makes look much nicer.
My complaint is that the code must have knowledge of the twisted environment (why else would it yield the result of process.check_call ()?). I do not really see the conceptual difference between these two code blocks except one yields to and one calls into the reactor event loop. Is there some other inherent problem with the first example? Of course you need to make sure that the code inside the try/finally block does not try to acquire the lock again, but that's a basic concurrency problem which can even happen in the next example.
This is really the key thing. If you're running your code in the Twisted environment, and you want it to be correct, it really must know about the Twisted environment. The simple presence of the 'yield' keyword at every level where a Deferred is being returned forces you to acknowledge, "yes, I know that a context switch may occur here". Without it, any function could suddenly and radically change the assumptions that all of its callers were allowed to make.
Moving on, in a fully deferred world we have this:
<snip ugly stuff>
... you get the picture.
Notice the code to acquire/release the lock--there are three different calls to lock.release() in there, and they all must be carefully sorted out to make sure that exactly one of them will be called in any given scenario --that's hard to maintain.
There are other ways to deal with that. maybeDeferred, for example, will make sure you always get a Deferred back and that it looks vaguely correct.
Right, that would work and that's exactly what subprocess.check_call () (the real python built-in version) would do. Unfortunately twisted does not work with the subprocess module--spawnProcess() is the only alternative I found that actually works and that means I have to use a deferred.
The only thing I have to say about that is: http://twistedmatrix.com/trac/ticket/733
Another solution here would be for Twisted to have a nice convenience API for dispatching tasks to a process pool. Right now setting up a process pool is conceptually easy but mechanically difficult; you have to do a lot of typing and make a lot of irrelevant decisions (AMP or PB or pickle? stdio or sockets?).
That sounds nice.
Something I'd be doing in my copious spare time, if I had any.
I understand that PB is fully symmetrical. In my case I am only using half (client makes request, server responds). Would it make sense to relax the constraints when PB is used in this way?
I don't know if it would be feasible to do the work required for PB, due to other, less fundamental implementation issues. However, it was a design goal of AMP that it be possible to implement a "naive", only-a -few-lines-of-Python version for drop-in ease-of-use comparable to XMLRPC while still providing the actual "good" version in Twisted itself. I have heard rumors to the effect that Eric Mangold actually wrote such a thing, but I don't know where it is.
#2545.
This looks very interesting. I'll try to help out with this effort if I can find some time.
Thanks.
Thanks for taking time to read my ramblings and understand the problems that I am having (even if we don't quite agree on the simplest solutions). Your input is valuable, and I am indebted to you for providing free support in your spare time.
Thanks very much for taking the time to acknowledge this. You leave me here with the impression that writing these emails was time well spent. And, thanks in advance for working on any of those tickets I gave you links to ;-).
The external "blocking" resource is just a shell script that takes some time t run. It does not acquire any shared resources that would result in dead lock and it will always return (maybe with an error, but it will return) unless something terrible happens (e.g. plug is pulled on server, fire, etc.).
I thought I understood what was going on, but now I'm confused again. Why do you need mutual exclusion at all if it doesn't acquire any shared resources? Couldn't you just run it concurrently?
I guess I said that wrong. When I said "it does not acquire any shared resources" I was referring to the external system being manipulated by the shell script. That effectively means that the shell script is the shared resource and it can only be called in a synchronous manner. The script is essentially posting a transaction, which must be done in an atomic fashion with regard to my code. I know this is very ugly, and I'd love to fix it. Unfortunately it's not my system so I can't. We can keep going around and around about this but there's no need. My immediate problem was solved when I learned that I could return a deferred from a PB remote_xxx() method. What I would like to continue to discuss is whether all code that calls something that does deferred logic must be immediately aware of that fact.
It would be more maintainable because it would look just like normal sequential python code:
Yes, it would *look* like sequential python code. But it wouldn't be :). There's a heck of a lot that can happen in acquire(); your whole application could run for ten minutes on that one line of code. Worst of all, it would only happen in extreme situations, so testing or debugging issues that are caused by it becomes even more difficult.
This could happen with any deferred logic. As long as the code has the proper concurrency logic this is not a problem--even if it takes 10 minutes. In today's operating systems something like that could even happen in plain old synchronous single-threaded code if the OS decided to give some other process priority for that long (unlikely but possible).
My complaint is that the code must have knowledge of the twisted environment (why else would it yield the result of process.check_call ()?). I do not really see the conceptual difference between these two code blocks except one yields to and one calls into the reactor event loop. Is there some other inherent problem with the first example? Of course you need to make sure that the code inside the try/finally block does not try to acquire the lock again, but that's a basic concurrency problem which can even happen in the next example.
This is really the key thing. If you're running your code in the Twisted environment, and you want it to be correct, it really must know about the Twisted environment. The simple presence of the 'yield' keyword at every level where a Deferred is being returned forces you to acknowledge, "yes, I know that a context switch may occur here". Without it, any function could suddenly and radically change the assumptions that all of its callers were allowed to make.
So it's really a matter of being explicit...and it's true that "explicit is better than implicit" but then again, "practicality beats purity" :-) It would be super nice to be able to provide the exact interface of a normal python module/class/function and have twisted logic going on inside. When used properly it would be very powerful. Of course doing something like this is definitely not entirely innocent, and there should be warnings provided with implementations that may block (as there should be with any other piece of concurrency-related code that may block). But it's not nice to force everyone to use an awkward interface just to try to help them avoid mistakes. ~ Daniel
The issue brought up in this thread is one that we (the IPython dev team) has thought about a lot. There are really two questions about having true synchronization primitives (something like d.waitFor() or blockOn(d)): 1. Can they be implemented in python/twisted in a reliable and robust way? 2. Should you use them in your code? Unfortunately, the answer to (1) seems to be no - at least not without completely refactoring the core of twisted to support this capability from the ground level. The tempting way of doing it currently is to call reactor.iterate() to spin the event loop at a particular point. We have code in IPython that does this: http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipyth... [DON'T USE THIS CODE!] For very simple things this code works just fine. d = functionThatReturnsADeferred() r = blockOn(d) # spin the reactor until d fires. The problem is that if you start to use blockOn in various places in your code, a single call to blockOn (which calls reactor.iterate()) could trigger other calls to blockOn (which will attempt to call reactor.iterate() again). The twisted reactor is simply not designed to be doubly iterated like this - it leads to all sorts of really weird problems that are _impossible_ to track down. This is why people are saying "it can't be done." I should mention that it might be possible to implement these things in stackless. So what about (2)? Ignoring the fact that such constructs can't be implemented reliably (let's imagine they could be), should you want to use them? I think the answer is this: The design of Twisted reflects the realities of an asynchronous, event driven world where things can and do go wrong when you least expect it. The error handling decision tree of Deferreds are a reflection of this reality. If you try to make this stuff go away (we have tried ma ny times - we are slow learners and very stubborn) you will be punished and there will be "weeping and gnashing of teeth." This punishment will take the form of buggy code that is difficult to maintain and extend. With all that said, I have encountered a few highly unusual cases where I really did want blockOn to exist. These cases had the characteristic that they couldn't be done any other way in Twisted. The answer in this case is to ditch twisted and use a tool that is better suited to the problem. But in my experience these cases only pop up about 0.00001% of the time. Brian
Brian Granger wrote:
So what about (2)? Ignoring the fact that such constructs can't be implemented reliably (let's imagine they could be), should you want to use them? I think the answer is this:
The design of Twisted reflects the realities of an asynchronous, event driven world where things can and do go wrong when you least expect it. The error handling decision tree of Deferreds are a reflection of this reality. If you try to make this stuff go away (we have tried ma ny times - we are slow learners and very stubborn) you will be punished and there will be "weeping and gnashing of teeth." This punishment will take the form of buggy code that is difficult to maintain and extend.
agreed, it's what i wrote in the email i've just sent out. if you look at some parts of twisted, like pb, and actually try to use them, you'll find that it's extremely liberating and useful not to have things hidden from you because this sort of infantilization can only end in tears. continuing with the pb example, based on personal experience, not having the network communications disguised and hidden from you by ordinary function calls makes writing non-trivial code practical, nevermind easy. i can't tell you how many projects i've seen that used this hiding back when it was en vogue, and all of them to a t were either completely borked or barely functional. this is all about having the tools to deal with the problem domain using concepts that map it properly - you can't deal with asynchrony without having it reified, with every consequence that entails. i spent a non-trivial amount of time a long time ago playing with generators, bytecode hacks and preprocessors to hide this stuff (it's not as damaging to hide it selectively once you already thoroughly understand what's going on under the hood) and came to the conclusion that i was wrong to want that. now if i could do the same with cigarettes and beer .. ;] cheers, -p
On 05:25 pm, ellisonbg.net@gmail.com wrote:
The issue brought up in this thread is one that we (the IPython dev team) has thought about a lot. There are really two questions about having true synchronization primitives (something like d.waitFor() or blockOn(d)):
1. Can they be implemented in python/twisted in a reliable and robust way?
The real question here doesn't involve the words "in python/twisted" :).
2. Should you use them in your code?
Unfortunately, the answer to (1) seems to be no - at least not without completely refactoring the core of twisted to support this capability from the ground level. The tempting way of doing it currently is to call reactor.iterate() to spin the event loop at a particular point. We have code in IPython that does this:
http://projects.scipy.org/ipython/ipython/browser/ipython/branches/saw/ipyth... [DON'T USE THIS CODE!]
For very simple things this code works just fine.
I think you mean, "in some cases this code appears to work". Working "just fine" implies that it is robust and supported. This is an actively deprecated programming style and there are numerous known problems (besides the ones you've already documented here) with doing it.
d = functionThatReturnsADeferred() r = blockOn(d) # spin the reactor until d fires.
The problem is that if you start to use blockOn in various places in your code, a single call to blockOn (which calls reactor.iterate()) could trigger other calls to blockOn (which will attempt to call reactor.iterate() again). The twisted reactor is simply not designed to be doubly iterated like this - it leads to all sorts of really weird problems that are _impossible_ to track down.
These "weird" problems are the entirely predictable result of violating every assumption that code makes about its run-time environment when it is written and tested. An analogous operation would be to write C code to forcibly delete Python objects rather than go through the garbage collector because you *really know*, *just this one time*, that you want to free that memory. If you're careful to never touch that object again, you might be able to avoid a segfault, but I think most people would agree that all bets are off at that point. This might seem like an exaggerated problem, but I have actually seen code like that more than once written by dyed-in-the-wool C programmers who didn't "get" how Python's object model worked. It's the same with dyed-in-the-wool non-concurrent programmers approaching concurrent systems for the first time.
This is why people are saying "it can't be done." I should mention that it might be possible to implement these things in stackless.
If you were to implement these things in stackless, you would still have to deal with the inherently problematic issue of apparently "sequential" code being run recursively when it does not expect to be. In order to prevent this, you would likely have a completely different programming model where something properly event-driven, like Twisted itself, were scheduling "user code" which was written using a different programming style. For certain problems such things are a good approach. For example, in AI code with extremely deeply nested ad-hoc decision trees modeled as if statements and for loops, the cost of stack ripping becomes high both conceptually and performance-wise, and it is more natural to model individual agents as individual control flows (or "cooperative threads"). This sort of code, though, would be written in a style more like Erlang, with almost no shared state at all. Part of Twisted's appeal is that it makes mutable state-sharing between disparate systems straightforward. In other words, it is a different programming model for a different set of problems that would require a different pile of infrastructure. It may well be possible to implement such a layer on top of Twisted, but there is a curious thing that takes place when people begin to tackle this problem. Pretty much everyone eventually comes to the realization that this isn't a good idea for their problem domain, and what they *actually* want is to wish away the difficulties associated with concurrency and pretend that they can "just block" and everything will be OK. The ones who really, really need it (like people dealing with the aforementioned AI problems) already know their requirements and quietly go ahead and implement what they need, without any hand-wringing about how hard programming with Deferreds is or how they'd really like to block on one.
So what about (2)? Ignoring the fact that such constructs can't be implemented reliably (let's imagine they could be), should you want to use them? I think the answer is this:
The design of Twisted reflects the realities of an asynchronous, event driven world where things can and do go wrong when you least expect it. The error handling decision tree of Deferreds are a reflection of this reality. If you try to make this stuff go away (we have tried ma ny times - we are slow learners and very stubborn) you will be punished and there will be "weeping and gnashing of teeth." This punishment will take the form of buggy code that is difficult to maintain and extend.
It sounds like we broadly agree here :).
With all that said, I have encountered a few highly unusual cases where I really did want blockOn to exist. These cases had the characteristic that they couldn't be done any other way in Twisted. The answer in this case is to ditch twisted and use a tool that is better suited to the problem. But in my experience these cases only pop up about 0.00001% of the time.
I am very curious about your 0.00001% case. Not that I don't believe such cases exist, but in every case but one (twisted ticket #2545) the issue has actually been a documentation problem with Twisted, where it wasn't clear how to do something the "normal" way with Deferreds and such. I'd like to know if there is another such doc bug we should be filing :).
With all that said, I have encountered a few highly unusual cases where I really did want blockOn to exist. These cases had the characteristic that they couldn't be done any other way in Twisted. The answer in this case is to ditch twisted and use a tool that is better suited to the problem. But in my experience these cases only pop up about 0.00001% of the time.
I am very curious about your 0.00001% case. Not that I don't believe such cases exist, but in every case but one (twisted ticket #2545) the issue has actually been a documentation problem with Twisted, where it wasn't clear how to do something the "normal" way with Deferreds and such. I'd like to know if there is another such doc bug we should be filing :).
The cases I am thinking about is not an example of a doc bug. The most relevant one is related to using Twisted in an interactive python (really IPython most of the time) session. There are two difficulties we keep running into: 1. The interactive python interpreter is a completely synchronous universe - getting the reactor running in this context seems like a hack. The only way I have seen this done is by running the reactor in a different thread. The problem with this is that it is inevitable that you end up wanting to do things with Deferreds in the main thread where user code is running. But, as I understand it, Twisted is not thread safe, so at that point, you are playing with (threaded) fire. 2. Users expect certain things in an interactive python session that don't mesh well with Twisted and the asynchronous universe:
psi = computeWavefunctionForHydrogen() psi.getEnergy(1) -13.6 # here the user looks at the energy (a human if statement) and decides if they actually want to # make the following plot. If the answer were not -13.6, they would not make the plot. plot(psi.getState(1))
Even if you could get the reactor running in an interactive python session it would be crazy to have to write something like (in an interactive session):
d = computeWavefunctionForHydrogen() def printAndPlot(psi, n): print psi.getEnergy(n) if abs(psi.getEnergy(n) - (psi.getEnergy(n)) < 1.0e-4: plot(psi.getState(n)) d.addCallback(printAndPlot, 1)
The problem we are having is that we would like to use Twisted network clients in functions like computeWavefunctionForHydrogen. But we simply can't as there is no way of returning the actual result to the user. I can't emphasize enough that end users of such code (scientists) "just want the damn result" (not a deferred) and are willing to wait for it. Thus in classes/functions that need to block for an actual result, we don't use twisted - we use blocking sockets instead. In blockOn existed, the implementation of computeWavefunctionForHydrogen could look like: def computeWavefunctionForHydrogen(): d = doRemoteComputationOnServerUsingTwisted() return blockOn(d) The important point is that doRemoteComputationOnServerUsingTwisted is the only place where anything involving the network is happening in this process. This type of code occurs mainly in clients. In server code you are usually also listening on sockets, so there could be other asyncronous events that occur. But in interactive client code, the network event are often very contained and isolated. One thing we have done in our blocking client code is to create something we call a PendingResult object. It is basically a fully synchronous version of a deferred that allows a user to poll for or block on a result that is being computed elsewhere. It is designed for interactive usage, where only blocking sockets are used. Brian
On 08:32 pm, ellisonbg.net@gmail.com wrote:
I am very curious about your 0.00001% case. Not that I don't believe such cases exist, but in every case but one (twisted ticket #2545) the issue has actually been a documentation problem with Twisted, where it wasn't clear how to do something the "normal" way with Deferreds and such. I'd like to know if there is another such doc bug we should be filing :).
The cases I am thinking about is not an example of a doc bug. The most relevant one is related to using Twisted in an interactive python (really IPython most of the time) session. There are two difficulties we keep running into:
Hmm. I hope you're not the guy I talked to at PyCon (I didn't catch his/your name) because I'm about to feel very foolish repeating myself here, especially if at some point in the interim you responded to this and I didn't notice.
1. The interactive python interpreter is a completely synchronous universe - getting the reactor running in this context seems like a hack. The only way I have seen this done is by running the reactor in a different thread. The problem with this is that it is inevitable that you end up wanting to do things with Deferreds in the main thread where user code is running. But, as I understand it, Twisted is not thread safe, so at that point, you are playing with (threaded) fire.
IPython's shell is not the same as the standard Python interpreter. It's already its own program and therefore has a fair amount of freedom in what it does with user input. Run "python -m twisted/conch/stdio" for an example of an interactive session that is held in a non-blocking way. Not blocking in the code *doesn't* mean not blocking for the user - it just means having the option not to block.
2. Users expect certain things in an interactive python session that don't mesh well with Twisted and the asynchronous universe:
This is _exactly_ the documentation issue I was talking about :).
psi = computeWavefunctionForHydrogen() psi.getEnergy(1) -13.6 # here the user looks at the energy (a human if statement) and decides if they actually want to # make the following plot. If the answer were not -13.6, they would not make the plot. plot(psi.getState(1))
Even if you could get the reactor running in an interactive python session it would be crazy to have to write something like (in an interactive session):
d = computeWavefunctionForHydrogen() def printAndPlot(psi, n): print psi.getEnergy(n) if abs(psi.getEnergy(n) - (psi.getEnergy(n)) < 1.0e-4: plot(psi.getState(n)) d.addCallback(printAndPlot, 1)
Here's a screenshot of an interactive session using the command above: http://twistedmatrix.com/users/glyph/images/content/screenshots/psi- energy.png Unfortunately it doesn't animate, but there is a 1-second pause between "<Deferred #0>" and "Deferred #0 called back:". It's a bit crude since it is "deferred-oriented" right now, rather than result oriented, but a few clever implementation tricks could easily eliminate the distinction (like the "_.result" line). Is that the kind of user-interaction you want?
on 04.04.2007 23:40 glyph@divmod.com said the following: <snip>
Run "python -m twisted/conch/stdio" for an example of an interactive session that is held in a non-blocking way. Not blocking in the code *doesn't* mean not blocking for the user - it just means having the option not to block.
2. Users expect certain things in an interactive python session that don't mesh well with Twisted and the asynchronous universe:
This is _exactly_ the documentation issue I was talking about :).
psi = computeWavefunctionForHydrogen() psi.getEnergy(1) -13.6 # here the user looks at the energy (a human if statement) and decides if they actually want to # make the following plot. If the answer were not -13.6, they would not make the plot. plot(psi.getState(1))
Even if you could get the reactor running in an interactive python session it would be crazy to have to write something like (in an interactive session):
d = computeWavefunctionForHydrogen() def printAndPlot(psi, n): print psi.getEnergy(n) if abs(psi.getEnergy(n) - (psi.getEnergy(n)) < 1.0e-4: plot(psi.getState(n)) d.addCallback(printAndPlot, 1)
Here's a screenshot of an interactive session using the command above:
http://twistedmatrix.com/users/glyph/images/content/screenshots/psi-energy.p...
Unfortunately it doesn't animate, but there is a 1-second pause between "<Deferred #0>" and "Deferred #0 called back:". It's a bit crude since it is "deferred-oriented" right now, rather than result oriented, but a few clever implementation tricks could easily eliminate the distinction (like the "_.result" line).
Am I right that this is a screenshot of the manhole client? I just recently tried to embed the IPython shell into a twisted application. AFAIK this is currently only possible using a separate thread (there is an ASPN cookbook entry). I would like to list here what I think is necessary to make this possible without using threads. Hopefully, you can correct me where I am wrong. The main IPython class has a main loop (InteractiveShell.interact) that uses raw_input. Further, it first sets up pyreadline for handling command-line niceties inside raw_input. For embedding this in twisted, the twisted reactor needs to be the one calling IPython/pyreadline when necessary. On the IPython level, I think this should be "easy" by subclassing the Shell and splitting/rearranging the relevant code, so that it can be called back at the point where raw_input would normally return. (I did not look at the new IPython saw branch...) On the twisted side, I think that something like twisted.internet.stdio would be calling back? Or rather something like twisted.conch.insults? I suspect that it would be hard to recode pyreadline in a way that it could be called back by twisted? So to get readline support, something like readline would need to be implemented using t.i.stdio / insults. Does this seem possible, so far? :-) cheers, stefan
On 06:18 pm, list-ener@strank.info wrote:
on 04.04.2007 23:40 glyph@divmod.com said the following: <snip>
Run "python -m twisted/conch/stdio" for an example of an interactive <snip> Here's a screenshot of an interactive session using the command above:
Am I right that this is a screenshot of the manhole client?
Sort of. It's the manhole server (I don't know what you mean by "client") running on standard IO, run via 'python -m twisted/conch/stdio'.
For embedding this in twisted, the twisted reactor needs to be the one calling IPython/pyreadline when necessary. On the IPython level, I think this should be "easy" by subclassing the Shell and splitting/rearranging the relevant code, so that it can be called back at the point where raw_input would normally return. (I did not look at the new IPython saw branch...)
On the twisted side, I think that something like twisted.internet.stdio would be calling back?
Rather than IPython calling into raw_input, the loop that calls raw_input would have to be changed to a function that takes a string. This would almost certainly be the trickiest part, since it seems there are about two dozen places (at a quick glance) in ipython that call raw_input.
Or rather something like twisted.conch.insults?
insults will be getting its events from twisted.internet.stdio, so both will be involved.
I suspect that it would be hard to recode pyreadline in a way that it could be called back by twisted? So to get readline support, something like readline would need to be implemented using t.i.stdio / insults.
Does this seem possible, so far? :-)
To me, sure - although I hardly have the time to do it :). There have been a few forays into line-editing code for insults, although I'm not clear of the current state of the art, JP Calderone should be returning from vacation soon and will probably have some light to shed on the issue. I don't know much about pyreadline. There may be code there that's reusable, maybe not, but in any event it is a goal of the insults library to provide high-quality event-driven line-editing.
On Fri, 06 Apr 2007 21:28:49 -0000, glyph@divmod.com wrote:
On 06:18 pm, list-ener@strank.info wrote:
on 04.04.2007 23:40 glyph@divmod.com said the following: <snip>
Run "python -m twisted/conch/stdio" for an example of an interactive <snip> Here's a screenshot of an interactive session using the command above:
Am I right that this is a screenshot of the manhole client?
Sort of. It's the manhole server (I don't know what you mean by "client") running on standard IO, run via 'python -m twisted/conch/stdio'.
For embedding this in twisted, the twisted reactor needs to be the one calling IPython/pyreadline when necessary. On the IPython level, I think this should be "easy" by subclassing the Shell and splitting/rearranging the relevant code, so that it can be called back at the point where raw_input would normally return. (I did not look at the new IPython saw branch...)
On the twisted side, I think that something like twisted.internet.stdio would be calling back?
Rather than IPython calling into raw_input, the loop that calls raw_input would have to be changed to a function that takes a string. This would almost certainly be the trickiest part, since it seems there are about two dozen places (at a quick glance) in ipython that call raw_input.
Or rather something like twisted.conch.insults?
insults will be getting its events from twisted.internet.stdio, so both will be involved.
I suspect that it would be hard to recode pyreadline in a way that it could be called back by twisted? So to get readline support, something like readline would need to be implemented using t.i.stdio / insults.
Does this seem possible, so far? :-)
To me, sure - although I hardly have the time to do it :). There have been a few forays into line-editing code for insults, although I'm not clear of the current state of the art, JP Calderone should be returning from vacation soon and will probably have some light to shed on the issue. I don't know much about pyreadline. There may be code there that's reusable, maybe not, but in any event it is a goal of the insults library to provide high-quality event-driven line-editing.
I haven't looked at pyreadline before, although I did take a look at UNC a long time ago. Taking a quick look at the code, it looks like converting pyreadline to be Twisted friendly may be doable. The most obvious change that would be required is for each point in the code which calls getkeypress or getchar to be changed to instead respond to an event. The ideal way to do this would be with a nice state machine which accounts for all the legal inputs and states the input system can be in. A slightly less nice way which would probably be much easier to implement would be to add getkeypress and getchar implementations which return Deferreds and then use inlineCallbacks to avoid having to make significant changes to any of the callers. This is worse since it leaves the code a bit of a mess, but it might be useful as an experiment. As for insults, I expect it has a less complete implementation of all of the typical readline features you or your users are interested in, but it is gradually improving. What would actually be really excellent is if both insults and pyreadline used the same underlying readline model code which was devoid of input/output code, then each could focus on the part it should really care about (dealing with events or talking to a windows console or what have you) and improvements to the actually interesting behavior in the readline model could be shared between the two. To some extend insults already has this (I've tried to factor as much of the interesting behavior out away from the Twisted parts), and I see this is also how at least some of pyreadline is structured (eg lineeditor/history.py), so it might actually be feasible to merge the two backends into one shared thing. Jean-Paul
Jean-Paul Calderone <exarkun <at> divmod.com> writes:
I haven't looked at pyreadline before, although I did take a look at UNC a long time ago. Taking a quick look at the code, it looks like converting pyreadline to be Twisted friendly may be doable.
This isn't really a followup, but it seems worthwhile to mention that GNU readline itself has a callback interface (rl_callback_read_char) that would be suitable for using with Twisted. You'd have to use ctypes or something to get to it, but it's there. Cheers, mwh
On Mon, 9 Apr 2007 14:33:26 +0000 (UTC), Michael Hudson <micahel@gmail.com> wrote:
Jean-Paul Calderone <exarkun <at> divmod.com> writes:
I haven't looked at pyreadline before, although I did take a look at UNC a long time ago. Taking a quick look at the code, it looks like converting pyreadline to be Twisted friendly may be doable.
This isn't really a followup, but it seems worthwhile to mention that GNU readline itself has a callback interface (rl_callback_read_char) that would be suitable for using with Twisted. You'd have to use ctypes or something to get to it, but it's there.
Good point, I'd forgotten about that API. Jean-Paul
Jean-Paul Calderone <exarkun <at> divmod.com> writes:
On Mon, 9 Apr 2007 14:33:26 +0000 (UTC), Michael Hudson <micahel <at>
gmail.com> wrote:
Jean-Paul Calderone <exarkun <at> divmod.com> writes:
I haven't looked at pyreadline before, although I did take a look at UNC a long time ago. Taking a quick look at the code, it looks like converting pyreadline to be Twisted friendly may be doable.
This isn't really a followup, but it seems worthwhile to mention that GNU readline itself has a callback interface (rl_callback_read_char) that would be suitable for using with Twisted. You'd have to use ctypes or something to get to it, but it's there.
Good point, I'd forgotten about that API.
Make sure you use readline 5.1 (or newer) though, otherwise your app will hang when the user presses C-r :-) Cheers, mwh
Hi, on 08.04.2007 16:51 Jean-Paul Calderone said the following: <snip>
What would actually be really excellent is if both insults and pyreadline used the same underlying readline model code which was devoid of input/output code, then each could focus on the part it should really care about (dealing with events or talking to a windows console or what have you) and improvements to the actually interesting behavior in the readline model could be shared between the two.
To some extend insults already has this (I've tried to factor as much of the interesting behavior out away from the Twisted parts), and I see this is also how at least some of pyreadline is structured (eg lineeditor/history.py), so it might actually be feasible to merge the two backends into one shared thing.
That would be perfect. on 09.04.2007 16:33 Michael Hudson said the following: <snip>
This isn't really a followup, but it seems worthwhile to mention that GNU readline itself has a callback interface (rl_callback_read_char) that would be suitable for using with Twisted. You'd have to use ctypes or something to get to it, but it's there.
Seems even better. This interface, however, wants to consume the input itself, so a t.i.stdio-alike would need to poll/select for input events and then call rl_callback_read_char to actually consume them, right? Attached is my ugly first attempt at embedding IPython in twisted without threads. (no readline behaviour, only tried it on Windows) Hopefully this catches the interest of someone with more time/knowledge... ;-) Will also post on the IPython list to see if they would be willing to change their code to be more callback-friendly. cheers, stefan
Hello, Can anyone point me to some examples of using the TaskQueue code from Twisted-Goodies? Thanks, Scott Knutti
Scott Knutti wrote:
Can anyone point me to some examples of using the TaskQueue code from Twisted-Goodies?
As its author I've used it as follows, in no particular order: 1. In sAync, to do sequential database transactions in a single thread. See http://foss.eepatents.com/sAsync/browser/trunk/sasync/database.py 2. In DynamicSite (which serves foss.eepatents.com and some other sites) to run Trac stuff in a single thread. This implementation does some work to deal with occasional freezes of the Trac WSGI application. When the queue gets too backed up, Trac is rudely forced into finishing up and moving on. See http://foss.eepatents.com/DynamicSite/browser/trunk/dynamicsite/wsgi.py 3. In a GUI application (not publicly released) with PB to dispatch various commands of different priorities to a centralized server. 4. In the asynCluster sub-package of Twisted-Goodies itself to dispatch jobs from a server to available cluster nodes. See http://foss.eepatents.com/Twisted-Goodies/browser/trunk/twisted_goodies/asyn... Naturally, I would be very interested in hearing about any other applications that are out there or being worked on. :-) Best regards, Ed
participants (12)
-
Bob Ippolito -
Brian Granger -
Daniel Miller -
Ed Suominen -
glyph＠divmod.com -
Itamar Shtull-Trauring -
Jean-Paul Calderone -
Michael Hudson -
Nicola Larosa -
Paul G -
Scott Knutti -
Stefan Rank