[Twisted-Python] Synchronous Code Fishbowl
The dialogue copied below shows yet another example of the frustrations that arise when trying to bridge the chasm between Twisted and the synchronous rest of the world. This latest episode inspired the idea of a synchronous code fishbowl that offers well-behaved Twisted code a deferToQueue() method for running badly-behaved blocking code. See http://foss.eepatents.com/sAsync/browser/branches/syncbridge.py Comments? - Ed Suominen -------- Original Message -------- Michael Bayer wrote:
why are you using twisted *with* threads ? didnt we all agree that was sort of unnecessary ?
Twisted operates without needing to run *in* a thread by doing everything asynchronously. Every call made via the Twisted event loop must either return (synchronous) results very quickly or return an immediate reference to an eventual result (thus operating asynchronously). The objects that hold those immediate references are called "deferreds" in TWisted parlance. Twisted runs most all of its internal operations with appropriate chunking and use of select() to keep the asynchronous event loop humming along nicely, but it has no control over how other libraries do things. When an external library like SQLAlchemy presents a blocking call like select(...).execute.fetchall(), the only way to make that call "play nice" with Twisted by immediately returning a deferred to the eventual result is by having TWisted run it in a thread. It includes the deferToThread() function for that express purpose.
(this goes to my point that twisted is a pain in the butt...)
Perhaps, but asynchronous code is a whole different way of thinking that some of us actually find cleaner and more intuitive than blocking + threads. What I'm trying to do with sAsync is put the "pain in the butt" associated with making synchronous and asynchronous code work together firmly behind the scenes for Twisted users, at least as far as interacting with SQLAlchemy is concerned.
anyway, SA 0.2 uses the same idea for sqlite threading as 0.1....it maintains each connection thread locally since you cant share a sqlite connection between threads. im also not sure what youre talking about with a "thread local copy of the engine", if thats something youre doing on your end, you might want to not do that. use Connection objects instead theyre much more portable.
Mike, I remember you telling me a while ago that SA somehow figures out how to make things work with the engine. That was after I spent a lot of time trying to re-engineer the wheel, to great frustration on my part. I followed your advice, ditched my thread-local code, and got SA transactions just fine (based on the engine) to run in arbitrary threads. Now it's not working in SA 0.2. Could you point out what's changed in regard to the engine vs. threads vs. sqlite? I'd rather not change everything under the hood (e.g., using connections somehow) if I don't have to. The lesson of not trying to re-engineer what SA handled fine on its own is still with me. :-) Best regards, Ed
On May 27, 2006, at 6:57 PM, Ed Suominen wrote:
My sAsync project relies on Twisted to maintain a thread pool. It runs database transactions in some available thread via Twisted's deferToThread() function. See the transact function in http://foss.eepatents.com/sAsync/browser/trunk/sasync/database.py for details.
That arrangement created no problem with SQLite and SA 0.1, even though SQLite does not allow sharing of connections between threads. Somehow, SA 0.1 made the transaction run in its thread with a thread-local copy of the engine and table that was created in the main thread. I never figured out how, but it just worked.
It doesn't work in SA 0.2, however. I either get the error when trying to use the main-thread table in the threaded transaction, or have no table available in the thread because it wasn't created there. Now I'm trying to figure out how to put humpty dumpty back together again. Any suggestions?
Best regards, Ed
On Sat, 27 May 2006 18:48:59 -0700, Ed Suominen <general@eepatents.com> wrote:
The dialogue copied below shows yet another example of the frustrations that arise when trying to bridge the chasm between Twisted and the synchronous rest of the world.
This latest episode inspired the idea of a synchronous code fishbowl that offers well-behaved Twisted code a deferToQueue() method for running badly-behaved blocking code.
See http://foss.eepatents.com/sAsync/browser/branches/syncbridge.py
Comments?
syncbridge looks like an interesting module. I've often wanted to do something similar, to have a "one right way" for integrating with non-async-clean legacy code which also doesn't necessarily support threaded concurrency either. However, I notice that nothing sets up the 'shutdown' method to be called on reactor shutdown automatically; this is a very tricky area, since mismanaged thread-pool shutdown can lock a process up hard as it's exiting. Also, have you considered just using a twisted.python.threadpool of maximum size 1, rather than callInThread? The main reason I didn't document that as the aforementioned "one right way" was because of the associated shutdown issues. The major advantage of callInThread is that the reactor's own threadpool is definitely initialized and shut down at predictable points. HTH, -glyph
I am rather new to Twisted. I am starting to build my first XMLRPC server and would like a way to shut it down remotely. Is there a way to do that? TIA, Chaz
On Tue, 2006-05-30 at 12:51 -0400, Chaz. wrote:
I am rather new to Twisted. I am starting to build my first XMLRPC server and would like a way to shut it down remotely. Is there a way to do that?
Add an xml-rpc command called e.g. "shutdown" that does reactor.stop(). -- Itamar Shtull-Trauring http://itamarst.org
I've written a simple protocol, factory and service and can't seem to get it to work right. It is so simple that I can't seem to figure out the problem. Here is the code: --------------------------------------------------------- # Import standard python modules from random import random from cmath import exp # Import 3rd party modules from twisted.internet.protocol import DatagramProtocol, ServerFactory from twisted.internet import reactor from twisted.application import internet class NeedFilesServer(DatagramProtocol) : def datagramReceived(self, dgram, sourceIP) : if dgram == 'HELPNeedFiles' : self.transport.write(IAmHere",sourceIP) class NeedFilesFactory(ServerFactory): protocol = NeedFilesServer class NeedFilesService(internet.UDPServer): def __init__(self,wkp): internet.UDPServer.__init__(self,wkp,NeedFilesFactory()) ----------------------------------------------------------------- When I do the standard application thing... application = service.Application('foo') srv = service.MultiService() NeedFilesSrv = NeedFilesService(config['wkp']) NeedFilesSrv.setServiceParent(srv) srv.setServiceParent(service.IServiceCollection(application)) I get a the following trace: Traceback (most recent call last): 2006/06/14 15:23 EDT [-] File "/usr/bin/twistd", line 25, in ? 2006/06/14 15:23 EDT [-] run() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/scripts/twistd.py", line 205, in run 2006/06/14 15:23 EDT [-] app.run(runApp, ServerOptions) 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/app.py", line 278, in run 2006/06/14 15:23 EDT [-] runApp(config) 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/scripts/twistd.py", line 196, in runApp 2006/06/14 15:23 EDT [-] startApplication(config, application) 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/scripts/twistd.py", line 173, in startApplication 2006/06/14 15:23 EDT [-] service.IService(application).privilegedStartService() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/service.py", line 191, in privilegedStartService 2006/06/14 15:23 EDT [-] service.privilegedStartService() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/service.py", line 191, in privilegedStartService 2006/06/14 15:23 EDT [-] service.privilegedStartService() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/internet.py", line 68, in privilegedStartService 2006/06/14 15:23 EDT [-] self._port = self._getPort() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/application/internet.py", line 86, in _getPort 2006/06/14 15:23 EDT [-] return getattr(reactor, 'listen'+self.method)(*self.args, **self.kwargs) 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/internet/posixbase.py", line 307, in listenUDP 2006/06/14 15:23 EDT [-] p.startListening() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/internet/udp.py", line 85, in startListening 2006/06/14 15:23 EDT [-] self._connectToProtocol() 2006/06/14 15:23 EDT [-] File "/usr/lib64/python2.4/site-packages/twisted/internet/udp.py", line 105, in _connectToProtocol 2006/06/14 15:23 EDT [-] self.protocol.makeConnection(self) 2006/06/14 15:23 EDT [-] AttributeError: NeedFilesFactory instance has no attribute 'makeConnection' I've looked at the FTP protocol code and I seem to be doing the same things. Does anyone know what I am doing wrong? Thanks in advance, Chaz.
On Wed, 14 Jun 2006 16:01:35 -0400, "Chaz." <eprparadocs@gmail.com> wrote:
I've written a simple protocol, factory and service and can't seem to get it to work right. It is so simple that I can't seem to figure out the problem.
Here is the code:
UDP servers don't use factories, since there is no network-level connection for which to maintain state. Instead, pass an instance of DatagramProtocol directly to listenUDP. Jean-Paul
How come the Echo server example shows a factory being used? Chaz. Jean-Paul Calderone wrote:
On Wed, 14 Jun 2006 16:01:35 -0400, "Chaz." <eprparadocs@gmail.com> wrote:
I've written a simple protocol, factory and service and can't seem to get it to work right. It is so simple that I can't seem to figure out the problem.
Here is the code:
UDP servers don't use factories, since there is no network-level connection for which to maintain state. Instead, pass an instance of DatagramProtocol directly to listenUDP.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
I don't know; is protocol a TCP protocol? Here is the example: class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) class EchoFactory(protocol.ServerFactory): protocol = Echo Chaz. Itamar Shtull-Trauring wrote:
On Wed, 2006-06-14 at 16:33 -0400, Chaz. wrote:
How come the Echo server example shows a factory being used?
Doesn't that use TCP, rather than UDP?
I am sorry I keep asking all these questions and hope you can tolerate at least one more. I have an application that has five services in it; two are factory based and three are protocol based (UDP). To shut the system down I need to blast out a couple of messages to one of the UDP connections. Is there an easy way to do this? Thanks once again, Chaz
On Wed, 14 Jun 2006 19:31:40 -0400, "Chaz." <eprparadocs@gmail.com> wrote:
I am sorry I keep asking all these questions and hope you can tolerate at least one more. I have an application that has five services in it; two are factory based and three are protocol based (UDP).
To shut the system down I need to blast out a couple of messages to one of the UDP connections. Is there an easy way to do this?
See twisted.internet.interfaces.IReactorCore and twisted.application: http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.I... http://twistedmatrix.com/documents/current/api/twisted.application.service.S... Jean-Paul
I have built a twistd application and now I want to pass in some arguments from the command line. The code I use is: class Options(usage.Options) : optParameters = [['wkp','Z','2200','The well known port']] config = Options() try : config.parseOptions() except usage.UsageError, e: print '%s: %s' % (sys.argv[0],e) print '%s: Try --help for usage details.' % (sys.argv[0]) sys.exit(1) I've tried running "twistd -ny --wkp=2200 foo.tac" and "twistd -ny foo.tac --wkp=2000" without success. I've decided I must not know how to get command line options to foo.tac. Can someone please enlighten me? Thanks, Chaz
On Thu, 15 Jun 2006 13:34:59 -0400, "Chaz." <eprparadocs@gmail.com> wrote:
I have built a twistd application and now I want to pass in some arguments from the command line. The code I use is:
class Options(usage.Options) : optParameters = [['wkp','Z','2200','The well known port']]
config = Options() try : config.parseOptions() except usage.UsageError, e: print '%s: %s' % (sys.argv[0],e) print '%s: Try --help for usage details.' % (sys.argv[0]) sys.exit(1)
I've tried running "twistd -ny --wkp=2200 foo.tac" and "twistd -ny foo.tac --wkp=2000" without success.
I've decided I must not know how to get command line options to foo.tac. Can someone please enlighten me?
Check out the following threads: http://twistedmatrix.com/pipermail/twisted-python/2006-June/013330.html http://twistedmatrix.com/pipermail/twisted-python/2005-May/010529.html http://twistedmatrix.com/pipermail/twisted-python/2005-June/010856.html Jean-Paul
Yes, a second question in a row...I am extending my application (that runs under twistd) to use Manhole. It looks like a great extension. Unfortunately after doing a google on twisted manhole I came up with zilch. Does anyone have an example they can pass my way? Thanks in advance, Chaz
On 6/15/06, Chaz. <eprparadocs@gmail.com> wrote:
Yes, a second question in a row...I am extending my application (that runs under twistd) to use Manhole. It looks like a great extension. Unfortunately after doing a google on twisted manhole I came up with zilch.
Does anyone have an example they can pass my way?
Check out the sample chapter in the Oreilly Twisted Book[1]. It also includes how to do make it accessible over SSH. If you like the book, you should of course buy it :-). 1. http://oreilly.com/catalog/twistedadn/ -- - Henrik
participants (6)
-
Chaz. -
Ed Suominen -
glyph＠divmod.com -
Henrik Thostrup Jensen -
Itamar Shtull-Trauring -
Jean-Paul Calderone