[Twisted-Python] Synchronous Code Fishbowl
![](https://secure.gravatar.com/avatar/7905d47e9096477130b7055ab70d0f30.jpg?s=120&d=mm&r=g)
The dialogue copied below shows yet another example of the frustrations that arise when trying to bridge the chasm between Twisted and the synchronous rest of the world. This latest episode inspired the idea of a synchronous code fishbowl that offers well-behaved Twisted code a deferToQueue() method for running badly-behaved blocking code. See http://foss.eepatents.com/sAsync/browser/branches/syncbridge.py Comments? - Ed Suominen -------- Original Message -------- Michael Bayer wrote:
why are you using twisted *with* threads ? didnt we all agree that was sort of unnecessary ?
Twisted operates without needing to run *in* a thread by doing everything asynchronously. Every call made via the Twisted event loop must either return (synchronous) results very quickly or return an immediate reference to an eventual result (thus operating asynchronously). The objects that hold those immediate references are called "deferreds" in TWisted parlance. Twisted runs most all of its internal operations with appropriate chunking and use of select() to keep the asynchronous event loop humming along nicely, but it has no control over how other libraries do things. When an external library like SQLAlchemy presents a blocking call like select(...).execute.fetchall(), the only way to make that call "play nice" with Twisted by immediately returning a deferred to the eventual result is by having TWisted run it in a thread. It includes the deferToThread() function for that express purpose.
(this goes to my point that twisted is a pain in the butt...)
Perhaps, but asynchronous code is a whole different way of thinking that some of us actually find cleaner and more intuitive than blocking + threads. What I'm trying to do with sAsync is put the "pain in the butt" associated with making synchronous and asynchronous code work together firmly behind the scenes for Twisted users, at least as far as interacting with SQLAlchemy is concerned.
Mike, I remember you telling me a while ago that SA somehow figures out how to make things work with the engine. That was after I spent a lot of time trying to re-engineer the wheel, to great frustration on my part. I followed your advice, ditched my thread-local code, and got SA transactions just fine (based on the engine) to run in arbitrary threads. Now it's not working in SA 0.2. Could you point out what's changed in regard to the engine vs. threads vs. sqlite? I'd rather not change everything under the hood (e.g., using connections somehow) if I don't have to. The lesson of not trying to re-engineer what SA handled fine on its own is still with me. :-) Best regards, Ed
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Sat, 27 May 2006 18:48:59 -0700, Ed Suominen <general@eepatents.com> wrote:
syncbridge looks like an interesting module. I've often wanted to do something similar, to have a "one right way" for integrating with non-async-clean legacy code which also doesn't necessarily support threaded concurrency either. However, I notice that nothing sets up the 'shutdown' method to be called on reactor shutdown automatically; this is a very tricky area, since mismanaged thread-pool shutdown can lock a process up hard as it's exiting. Also, have you considered just using a twisted.python.threadpool of maximum size 1, rather than callInThread? The main reason I didn't document that as the aforementioned "one right way" was because of the associated shutdown issues. The major advantage of callInThread is that the reactor's own threadpool is definitely initialized and shut down at predictable points. HTH, -glyph
![](https://secure.gravatar.com/avatar/96a6fa70caad11789ce45b7096860447.jpg?s=120&d=mm&r=g)
I am rather new to Twisted. I am starting to build my first XMLRPC server and would like a way to shut it down remotely. Is there a way to do that? TIA, Chaz
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Tue, 2006-05-30 at 12:51 -0400, Chaz. wrote:
Add an xml-rpc command called e.g. "shutdown" that does reactor.stop(). -- Itamar Shtull-Trauring http://itamarst.org
![](https://secure.gravatar.com/avatar/96a6fa70caad11789ce45b7096860447.jpg?s=120&d=mm&r=g)
I've written a simple protocol, factory and service and can't seem to get it to work right. It is so simple that I can't seem to figure out the problem. Here is the code: --------------------------------------------------------- # Import standard python modules from random import random from cmath import exp # Import 3rd party modules from twisted.internet.protocol import DatagramProtocol, ServerFactory from twisted.internet import reactor from twisted.application import internet class NeedFilesServer(DatagramProtocol) : def datagramReceived(self, dgram, sourceIP) : if dgram == 'HELPNeedFiles' : self.transport.write(IAmHere",sourceIP) class NeedFilesFactory(ServerFactory): protocol = NeedFilesServer class NeedFilesService(internet.UDPServer): def __init__(self,wkp): internet.UDPServer.__init__(self,wkp,NeedFilesFactory()) ----------------------------------------------------------------- When I do the standard application thing... application = service.Application('foo') srv = service.MultiService() NeedFilesSrv = NeedFilesService(config['wkp']) NeedFilesSrv.setServiceParent(srv) srv.setServiceParent(service.IServiceCollection(application)) I get a the following trace: Traceback (most recent call last): 2006/06/14 15:23 EDT [-] File "/usr/bin/twistd", line 25, in ? 2006/06/14 15:23 EDT [-] run() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/scripts/twistd.py", line 205, in run 2006/06/14 15:23 EDT [-] app.run(runApp, ServerOptions) 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/app.py", line 278, in run 2006/06/14 15:23 EDT [-] runApp(config) 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/scripts/twistd.py", line 196, in runApp 2006/06/14 15:23 EDT [-] startApplication(config, application) 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/scripts/twistd.py", line 173, in startApplication 2006/06/14 15:23 EDT [-] service.IService(application).privilegedStartService() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/service.py", line 191, in privilegedStartService 2006/06/14 15:23 EDT [-] service.privilegedStartService() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/service.py", line 191, in privilegedStartService 2006/06/14 15:23 EDT [-] service.privilegedStartService() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/internet.py", line 68, in privilegedStartService 2006/06/14 15:23 EDT [-] self._port = self._getPort() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/internet.py", line 86, in _getPort 2006/06/14 15:23 EDT [-] return getattr(reactor, 'listen'+self.method)(*self.args, **self.kwargs) 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/internet/posixbase.py", line 307, in listenUDP 2006/06/14 15:23 EDT [-] p.startListening() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/internet/udp.py", line 85, in startListening 2006/06/14 15:23 EDT [-] self._connectToProtocol() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/internet/udp.py", line 105, in _connectToProtocol 2006/06/14 15:23 EDT [-] self.protocol.makeConnection(self) 2006/06/14 15:23 EDT [-] AttributeError: NeedFilesFactory instance has no attribute 'makeConnection' I've looked at the FTP protocol code and I seem to be doing the same things. Does anyone know what I am doing wrong? Thanks in advance, Chaz.
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Wed, 14 Jun 2006 16:01:35 -0400, "Chaz." <eprparadocs@gmail.com> wrote:
UDP servers don't use factories, since there is no network-level connection for which to maintain state. Instead, pass an instance of DatagramProtocol directly to listenUDP. Jean-Paul
![](https://secure.gravatar.com/avatar/96a6fa70caad11789ce45b7096860447.jpg?s=120&d=mm&r=g)
I don't know; is protocol a TCP protocol? Here is the example: class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) class EchoFactory(protocol.ServerFactory): protocol = Echo Chaz. Itamar Shtull-Trauring wrote:
![](https://secure.gravatar.com/avatar/96a6fa70caad11789ce45b7096860447.jpg?s=120&d=mm&r=g)
I am sorry I keep asking all these questions and hope you can tolerate at least one more. I have an application that has five services in it; two are factory based and three are protocol based (UDP). To shut the system down I need to blast out a couple of messages to one of the UDP connections. Is there an easy way to do this? Thanks once again, Chaz
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Wed, 14 Jun 2006 19:31:40 -0400, "Chaz." <eprparadocs@gmail.com> wrote:
See twisted.internet.interfaces.IReactorCore and twisted.application: http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.I... http://twistedmatrix.com/documents/current/api/twisted.application.service.S... Jean-Paul
![](https://secure.gravatar.com/avatar/96a6fa70caad11789ce45b7096860447.jpg?s=120&d=mm&r=g)
I have built a twistd application and now I want to pass in some arguments from the command line. The code I use is: class Options(usage.Options) : optParameters = [['wkp','Z','2200','The well known port']] config = Options() try : config.parseOptions() except usage.UsageError, e: print '%s: %s' % (sys.argv[0],e) print '%s: Try --help for usage details.' % (sys.argv[0]) sys.exit(1) I've tried running "twistd -ny --wkp=2200 foo.tac" and "twistd -ny foo.tac --wkp=2000" without success. I've decided I must not know how to get command line options to foo.tac. Can someone please enlighten me? Thanks, Chaz
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Thu, 15 Jun 2006 13:34:59 -0400, "Chaz." <eprparadocs@gmail.com> wrote:
Check out the following threads: http://twistedmatrix.com/pipermail/twisted-python/2006-June/013330.html http://twistedmatrix.com/pipermail/twisted-python/2005-May/010529.html http://twistedmatrix.com/pipermail/twisted-python/2005-June/010856.html Jean-Paul
![](https://secure.gravatar.com/avatar/96a6fa70caad11789ce45b7096860447.jpg?s=120&d=mm&r=g)
Yes, a second question in a row...I am extending my application (that runs under twistd) to use Manhole. It looks like a great extension. Unfortunately after doing a google on twisted manhole I came up with zilch. Does anyone have an example they can pass my way? Thanks in advance, Chaz
![](https://secure.gravatar.com/avatar/9ffe54dddf39980639aba5fae15c7082.jpg?s=120&d=mm&r=g)
On 6/15/06, Chaz. <eprparadocs@gmail.com> wrote:
Check out the sample chapter in the Oreilly Twisted Book[1]. It also includes how to do make it accessible over SSH. If you like the book, you should of course buy it :-). 1. http://oreilly.com/catalog/twistedadn/ -- - Henrik
![](https://secure.gravatar.com/avatar/d6328babd9f9a98ecc905e1ccac2495e.jpg?s=120&d=mm&r=g)
On Sat, 27 May 2006 18:48:59 -0700, Ed Suominen <general@eepatents.com> wrote:
syncbridge looks like an interesting module. I've often wanted to do something similar, to have a "one right way" for integrating with non-async-clean legacy code which also doesn't necessarily support threaded concurrency either. However, I notice that nothing sets up the 'shutdown' method to be called on reactor shutdown automatically; this is a very tricky area, since mismanaged thread-pool shutdown can lock a process up hard as it's exiting. Also, have you considered just using a twisted.python.threadpool of maximum size 1, rather than callInThread? The main reason I didn't document that as the aforementioned "one right way" was because of the associated shutdown issues. The major advantage of callInThread is that the reactor's own threadpool is definitely initialized and shut down at predictable points. HTH, -glyph
![](https://secure.gravatar.com/avatar/96a6fa70caad11789ce45b7096860447.jpg?s=120&d=mm&r=g)
I am rather new to Twisted. I am starting to build my first XMLRPC server and would like a way to shut it down remotely. Is there a way to do that? TIA, Chaz
![](https://secure.gravatar.com/avatar/d7875f8cfd8ba9262bfff2bf6f6f9b35.jpg?s=120&d=mm&r=g)
On Tue, 2006-05-30 at 12:51 -0400, Chaz. wrote:
Add an xml-rpc command called e.g. "shutdown" that does reactor.stop(). -- Itamar Shtull-Trauring http://itamarst.org
![](https://secure.gravatar.com/avatar/96a6fa70caad11789ce45b7096860447.jpg?s=120&d=mm&r=g)
I've written a simple protocol, factory and service and can't seem to get it to work right. It is so simple that I can't seem to figure out the problem. Here is the code: --------------------------------------------------------- # Import standard python modules from random import random from cmath import exp # Import 3rd party modules from twisted.internet.protocol import DatagramProtocol, ServerFactory from twisted.internet import reactor from twisted.application import internet class NeedFilesServer(DatagramProtocol) : def datagramReceived(self, dgram, sourceIP) : if dgram == 'HELPNeedFiles' : self.transport.write(IAmHere",sourceIP) class NeedFilesFactory(ServerFactory): protocol = NeedFilesServer class NeedFilesService(internet.UDPServer): def __init__(self,wkp): internet.UDPServer.__init__(self,wkp,NeedFilesFactory()) ----------------------------------------------------------------- When I do the standard application thing... application = service.Application('foo') srv = service.MultiService() NeedFilesSrv = NeedFilesService(config['wkp']) NeedFilesSrv.setServiceParent(srv) srv.setServiceParent(service.IServiceCollection(application)) I get a the following trace: Traceback (most recent call last): 2006/06/14 15:23 EDT [-] File "/usr/bin/twistd", line 25, in ? 2006/06/14 15:23 EDT [-] run() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/scripts/twistd.py", line 205, in run 2006/06/14 15:23 EDT [-] app.run(runApp, ServerOptions) 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/app.py", line 278, in run 2006/06/14 15:23 EDT [-] runApp(config) 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/scripts/twistd.py", line 196, in runApp 2006/06/14 15:23 EDT [-] startApplication(config, application) 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/scripts/twistd.py", line 173, in startApplication 2006/06/14 15:23 EDT [-] service.IService(application).privilegedStartService() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/service.py", line 191, in privilegedStartService 2006/06/14 15:23 EDT [-] service.privilegedStartService() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/service.py", line 191, in privilegedStartService 2006/06/14 15:23 EDT [-] service.privilegedStartService() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/internet.py", line 68, in privilegedStartService 2006/06/14 15:23 EDT [-] self._port = self._getPort() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/internet.py", line 86, in _getPort 2006/06/14 15:23 EDT [-] return getattr(reactor, 'listen'+self.method)(*self.args, **self.kwargs) 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/internet/posixbase.py", line 307, in listenUDP 2006/06/14 15:23 EDT [-] p.startListening() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/internet/udp.py", line 85, in startListening 2006/06/14 15:23 EDT [-] self._connectToProtocol() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/internet/udp.py", line 105, in _connectToProtocol 2006/06/14 15:23 EDT [-] self.protocol.makeConnection(self) 2006/06/14 15:23 EDT [-] AttributeError: NeedFilesFactory instance has no attribute 'makeConnection' I've looked at the FTP protocol code and I seem to be doing the same things. Does anyone know what I am doing wrong? Thanks in advance, Chaz.
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Wed, 14 Jun 2006 16:01:35 -0400, "Chaz." <eprparadocs@gmail.com> wrote:
UDP servers don't use factories, since there is no network-level connection for which to maintain state. Instead, pass an instance of DatagramProtocol directly to listenUDP. Jean-Paul
![](https://secure.gravatar.com/avatar/96a6fa70caad11789ce45b7096860447.jpg?s=120&d=mm&r=g)
I don't know; is protocol a TCP protocol? Here is the example: class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) class EchoFactory(protocol.ServerFactory): protocol = Echo Chaz. Itamar Shtull-Trauring wrote:
![](https://secure.gravatar.com/avatar/96a6fa70caad11789ce45b7096860447.jpg?s=120&d=mm&r=g)
I am sorry I keep asking all these questions and hope you can tolerate at least one more. I have an application that has five services in it; two are factory based and three are protocol based (UDP). To shut the system down I need to blast out a couple of messages to one of the UDP connections. Is there an easy way to do this? Thanks once again, Chaz
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Wed, 14 Jun 2006 19:31:40 -0400, "Chaz." <eprparadocs@gmail.com> wrote:
See twisted.internet.interfaces.IReactorCore and twisted.application: http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.I... http://twistedmatrix.com/documents/current/api/twisted.application.service.S... Jean-Paul
![](https://secure.gravatar.com/avatar/96a6fa70caad11789ce45b7096860447.jpg?s=120&d=mm&r=g)
I have built a twistd application and now I want to pass in some arguments from the command line. The code I use is: class Options(usage.Options) : optParameters = [['wkp','Z','2200','The well known port']] config = Options() try : config.parseOptions() except usage.UsageError, e: print '%s: %s' % (sys.argv[0],e) print '%s: Try --help for usage details.' % (sys.argv[0]) sys.exit(1) I've tried running "twistd -ny --wkp=2200 foo.tac" and "twistd -ny foo.tac --wkp=2000" without success. I've decided I must not know how to get command line options to foo.tac. Can someone please enlighten me? Thanks, Chaz
![](https://secure.gravatar.com/avatar/7ed9784cbb1ba1ef75454034b3a8e6a1.jpg?s=120&d=mm&r=g)
On Thu, 15 Jun 2006 13:34:59 -0400, "Chaz." <eprparadocs@gmail.com> wrote:
Check out the following threads: http://twistedmatrix.com/pipermail/twisted-python/2006-June/013330.html http://twistedmatrix.com/pipermail/twisted-python/2005-May/010529.html http://twistedmatrix.com/pipermail/twisted-python/2005-June/010856.html Jean-Paul
![](https://secure.gravatar.com/avatar/96a6fa70caad11789ce45b7096860447.jpg?s=120&d=mm&r=g)
Yes, a second question in a row...I am extending my application (that runs under twistd) to use Manhole. It looks like a great extension. Unfortunately after doing a google on twisted manhole I came up with zilch. Does anyone have an example they can pass my way? Thanks in advance, Chaz
![](https://secure.gravatar.com/avatar/9ffe54dddf39980639aba5fae15c7082.jpg?s=120&d=mm&r=g)
On 6/15/06, Chaz. <eprparadocs@gmail.com> wrote:
Check out the sample chapter in the Oreilly Twisted Book[1]. It also includes how to do make it accessible over SSH. If you like the book, you should of course buy it :-). 1. http://oreilly.com/catalog/twistedadn/ -- - Henrik
participants (6)
-
Chaz.
-
Ed Suominen
-
glyph@divmod.com
-
Henrik Thostrup Jensen
-
Itamar Shtull-Trauring
-
Jean-Paul Calderone