Mailman 3 June 2006 - Twisted

[Twisted-Python] Tracking twisted repository post 2.4.0?
by David Bolen 17 Jun '06

17 Jun '06

We've got some locally maintained modifications to Twisted that prior to 2.4.0 we were tracking by syncing our repository with the Twisted trunk (as a vendor) at the overall official tags. But with the tagging changes with 2.4.0 (and without a tag for the overall trunk), I was wondering if there were suggestions on how to adjust our setup. I can understand why there is no single tag for the trunk, since an overall "Sumo" release of twisted (and even the overall 2.4.0 concept) is really a combination of different releases of the various subpackages, but I can't figure out how to use the existing tags to track those pieces we are interested in. For example, we really use core and web, so I had thought I could just export, for example, the twisted-core-2.4.0 and twisted-web-0.6.0 tags from tags/releases, merge them together and call that our "vendor" dump for 2.4.0. But it would appear that the individual project tags are still based on tags for the complete trunk, and thus contain all trunk files as of when the subproject was tagged. Thus, if I try to use a checkout/export from twisted-web-0.6.0, then depending on when it was tagged, it may overlay files that came from twisted-core-2.4.0 with different versions that aren't really part of core 2.4.0. So how (at the subversion tag level) if at all can I isolate those files from beneath the twisted-web-0.6.0 tag that only apply to twisted-web, so that I can add them to the files from twisted-core-2.4.0 to yield my simulated vendor dump? Is anyone else tracking the repository (but not instant to instant at the trunk, but rather at release points) and if so, how are you doing it post 2.4.0? Thanks. -- David

3 3

[Twisted-Python] Returning a deferred, maybe
by Wilfredo Sánchez Vega 16 Jun '06

16 Jun '06

We have several functions in the code which return either a synchronous value or a deferred. In these cases, the caller has to either inspect the return type and do different things depending whether the value is available right then, or call the function using meybeDeferred. It seems like you can save the caller some grief by ensuring that you always return a deferred value (eg. by wrapping a synchronous result with defer.succeed(), as suggested in http://twistedmatrix.com/ projects/core/documentation/howto/gendefer.html). Should this be an encouraged practice? -wsv

4 5

[Twisted-Python] Bring multi service up gradually
by David Pratt 16 Jun '06

16 Jun '06

Hi. I have got a number of services that am bringing together at startup. In the past, I have added them to a parent and started them up all at one time. But now I am incorporating a service (the main service) that may be available from more than one address and I want to connect with one of these addresses first before starting the other services. What I am seeking is some general advice on this since I want the startup to be as efficient as possible. I was thinking that quickly testing the addresses first, before starting any twisted service might help increase the odds of a quick startup. But if this fails, I will still need to iterate with a backoff through each address to connect the main service, then add the other services when the main service is running. Many thanks. Regards, David

3 6

Re: [Twisted-Python] twisted.plugin question
by Terry Jones 15 Jun '06

15 Jun '06

>>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> On Thu, 15 Jun 2006 17:09:26 +0200, Terry Jones <terry(a)jon.es> wrote: >> sys.path[:0] = ['./twisted/plugins'] glyph> This line is unnecessary, and probably wrong. If you look in glyph> twisted/plugins/__init__.py, you will notice that it puts any glyph> additional entries on sys.path into its __path__; the fact that "." glyph> is already on your sys.path is enough for it to recognize this glyph> plugin. OK, I see now how that's working, thanks. >> def displayAllKnownMaterials(): >> for material in getPlugins(IMaterial): >> displayMaterial(material) glyph> This is going to look for plugins in twisted.plugins, which is glyph> probably not what you want. Did you look under the "alternate glyph> plugin packages" section of that document? I did, and it seemed relevant, but I didn't understand it. The page says "If passed in, the 2nd argument should be a module or package to be used instead of twisted.plugins as the plugin meta-package." I had no idea what such a module or package should do! Reading the section above seemed to make it clear that I could mkdir -p twisted/plugins and drop my code in there. >> from twisted.plugin import IPlugin >> from plugin import IMaterial glyph> There is some vagueness in naming here. Depending on your working glyph> directory (etc) twisted.plugin might be picked up as plugin. You glyph> probably want to change the name of your example module to something glyph> unique. OK, done. glyph> from myapp import main glyph> main() glyph> Python has the unfortunate habit of creating a special module called glyph> "__main__" and putting the code from the file it is passed onto the glyph> commandline into that module, rather than into the location where it glyph> would go if you imported it; having two copies of your "plugin" glyph> module in memory at once (the imported one, and the __main__ one) is glyph> probably causing at least some of your problems. And thanks for this too. I'm very new to python, and I had no idea. The code is now working. Terry

1 0

Re: [Twisted-Python] twisted.plugin question
by Terry Jones 15 Jun '06

15 Jun '06

>>>>> "JP" == Jean-Paul Calderone <exarkun(a)divmod.com> writes: JP> On Thu, 15 Jun 2006 17:09:26 +0200, Terry Jones <terry(a)jon.es> wrote: JP> One problem with the code attached is that it creates a plugin class JP> which implements plugin.IMaterial but then goes and searches for JP> plugins which provide __main__.IMaterial. Since these aren't the same JP> interface, even without the pickle error no results will be found. OK, thanks for the tip. I've fixed things - see next mail. >> BTW, the code on that page in the section Extending an Existing Program >> is missing a 'from zope.interface import implements'. Also, the >> assumption that code is in a file called imatsim.py, and the curious >> 'from matsim import imatsim' could be explained. I guess the later is >> just a harmless anachronism? JP> I've created a ticket in the tracker for expanding the howto to explain JP> these things. Thanks for pointing out that they were not well JP> explained. Thanks. I should have created the ticket myself. I just added another, but now see it was a duplicate. Terry

1 0

[Twisted-Python] General question on components, interfaces, etc.
by Terry Jones 15 Jun '06

15 Jun '06

I've been trying to properly understand components, interfaces, adaptors etc., as summarized here http://twistedmatrix.com/projects/core/documentation/howto/components.html and elsewhere. Although I feel like I understand the various bit & pieces, I still don't feel like I have a good grip on when these things are most useful, or that I can recognize patterns where they can be best applied. I do have the feeling that it's important though. So here's a specific example and question. Suppose I am implementing some functionality that I want to be able to easily replace with an alternate implementation, perhaps written by someone else. I can use some kind of plugin architecture, and I can specify an Interface that the plugin code is expected to provide. I can look for plugins (or have twisted.plugin do it for me), import them, allow a user config file to specify the preferred plugin to use to provide the functionality, and I can test the loaded plugin to see that it does provide the Interface. All that makes sense, but I'm not sure what it really buys me. An alternative would just be to document the expected interface (perhaps as an Interface), and then use __import__() to load the desired implementation. It seems that without needing adapters, the component architecture doesn't really buy me anything except a formalism (which I do think has value, though limited). Is this thinking more or less correct? How would people go about doing what I outline above? Regards, Terry

2 1

[Twisted-Python] Synchronous Code Fishbowl
by Ed Suominen 15 Jun '06

15 Jun '06

The dialogue copied below shows yet another example of the frustrations that arise when trying to bridge the chasm between Twisted and the synchronous rest of the world. This latest episode inspired the idea of a synchronous code fishbowl that offers well-behaved Twisted code a deferToQueue() method for running badly-behaved blocking code. See http://foss.eepatents.com/sAsync/browser/branches/syncbridge.py Comments? - Ed Suominen -------- Original Message -------- Michael Bayer wrote: > why are you using twisted *with* threads ? didnt we all agree that was > sort of unnecessary ? Twisted operates without needing to run *in* a thread by doing everything asynchronously. Every call made via the Twisted event loop must either return (synchronous) results very quickly or return an immediate reference to an eventual result (thus operating asynchronously). The objects that hold those immediate references are called "deferreds" in TWisted parlance. Twisted runs most all of its internal operations with appropriate chunking and use of select() to keep the asynchronous event loop humming along nicely, but it has no control over how other libraries do things. When an external library like SQLAlchemy presents a blocking call like select(...).execute.fetchall(), the only way to make that call "play nice" with Twisted by immediately returning a deferred to the eventual result is by having TWisted run it in a thread. It includes the deferToThread() function for that express purpose. > (this goes to my point that twisted is a pain in the butt...) Perhaps, but asynchronous code is a whole different way of thinking that some of us actually find cleaner and more intuitive than blocking + threads. What I'm trying to do with sAsync is put the "pain in the butt" associated with making synchronous and asynchronous code work together firmly behind the scenes for Twisted users, at least as far as interacting with SQLAlchemy is concerned. > anyway, SA 0.2 uses the same idea for sqlite threading as 0.1....it > maintains each connection thread locally since you cant share a sqlite > connection between threads. im also not sure what youre talking about > with a "thread local copy of the engine", if thats something youre doing > on your end, you might want to not do that. use Connection objects > instead theyre much more portable. Mike, I remember you telling me a while ago that SA somehow figures out how to make things work with the engine. That was after I spent a lot of time trying to re-engineer the wheel, to great frustration on my part. I followed your advice, ditched my thread-local code, and got SA transactions just fine (based on the engine) to run in arbitrary threads. Now it's not working in SA 0.2. Could you point out what's changed in regard to the engine vs. threads vs. sqlite? I'd rather not change everything under the hood (e.g., using connections somehow) if I don't have to. The lesson of not trying to re-engineer what SA handled fine on its own is still with me. :-) Best regards, Ed > On May 27, 2006, at 6:57 PM, Ed Suominen wrote: > >> My sAsync project relies on Twisted to maintain a thread pool. It runs >> database transactions in some available thread via Twisted's >> deferToThread() function. See the transact function in >> http://foss.eepatents.com/sAsync/browser/trunk/sasync/database.py for >> details. >> >> That arrangement created no problem with SQLite and SA 0.1, even though >> SQLite does not allow sharing of connections between threads. Somehow, >> SA 0.1 made the transaction run in its thread with a thread-local copy >> of the engine and table that was created in the main thread. I never >> figured out how, but it just worked. >> >> It doesn't work in SA 0.2, however. I either get the error when trying >> to use the main-thread table in the threaded transaction, or have no >> table available in the thread because it wasn't created there. Now I'm >> trying to figure out how to put humpty dumpty back together again. Any >> suggestions? >> >> Best regards, Ed

6 14

[Twisted-Python] twisted.plugin question
by Terry Jones 15 Jun '06

15 Jun '06

Is the Twisted plugin system described here http://twistedmatrix.com/projects/core/documentation/howto/plugin.html still supported / functional? I tried implementing the example on that page and couldn't make it work. Below is my simplified code from that example, in two files. BTW, the code on that page in the section Extending an Existing Program is missing a 'from zope.interface import implements'. Also, the assumption that code is in a file called imatsim.py, and the curious 'from matsim import imatsim' could be explained. I guess the later is just a harmless anachronism? Regards, Terry # ------------------- # Begin plugin.py: import sys from zope.interface import Interface from twisted.plugin import getPlugins sys.path[:0] = ['./twisted/plugins'] class IMaterial(Interface): def yieldStress(temperature): """ """ def displayMaterial(m): print 'A material with yield stress %s at 500 K' % (m.yieldStress(500),) def displayAllKnownMaterials(): for material in getPlugins(IMaterial): displayMaterial(material) if __name__ == '__main__': displayAllKnownMaterials() # End plugin.py: # ------------------- # Begin ./twisted/plugins/materials.py: from twisted.plugin import IPlugin from plugin import IMaterial from zope.interface import implements class SimpleMaterial(object): implements(IPlugin, IMaterial) def __init__(self, yieldStressFactor): self._yieldStressFactor = yieldStressFactor def yieldStress(self, temperature): return self._yieldStressFactory * temperature steelPlate = SimpleMaterial(2) brassPlate = SimpleMaterial(1) # End ./twisted/plugins/materials.py: When I run plugin.py, I get the error: $ python plugin.py Traceback (most recent call last): File "plugin.py", line 18, in ? displayAllKnownMaterials() File "plugin.py", line 14, in displayAllKnownMaterials for material in getPlugins(IMaterial): File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/twisted/plugin.py", line 210, in getPlugins allDropins = getCache(package) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/twisted/plugin.py", line 176, in getCache newCacheData = pickle.dumps(cache, 2) cPickle.PicklingError: Can't pickle <InterfaceClass plugin.IMaterial>: attribute lookup plugin.IMaterial failed

3 2

[Twisted-Python] pb and other protocol
by Stefano Canepa 14 Jun '06

14 Jun '06

Hi all, I have an application listening on a port that needs to connect to a pb server. As the reactor is running for the server part, how can I do somethink like the protocol.ClientCreator I use in other applications that are pb server and client for other protocols? TIA Stefano -- Stefano Canepa aka sc: sc(a)linux.it http://www.stefanocanepa.it Three great virtues of a programmer: laziness, impatience and hubris. Le tre grandi virtù di un programmatore: pigrizia, impazienza e arroganza. (Larry Wall)

1 0

Re: [Twisted-Python] recommended twisted database coding
by Terry Jones 13 Jun '06

13 Jun '06

>>>>> "Jesus" == Jesus Cea <jcea(a)argo.es> writes: Jesus> I recomend you "durus" Jesus> (http://www.mems-exchange.org/software/durus/) and, depending of Jesus> your usage profile, my storage backend: Jesus> <http://www.argo.es/~jcea/programacion/durus-berkeleydbstorage.htm> Jesus> Yes, durus supports transactions. Jesus> Durus is not async, but wrapping it in a thread should be easy enough. Jesus> I'm fairly involved with Durus, so I could help you there. Hi Jesus Thanks for the info. I think I've seen Durus before, but was not aware that you had implemented a BSD backend. Does that mean the qualification on the Durus home page "Durus is best suited to collections of less than a million instances with relatively stable state." does not apply here? After looking around at some Pyhon persistence options, I decided to use ZODB. I'd be happy to hear your thoughts on Durus+BSD versus ZODB. I was disappointed to see that ZODB had dropped BSD as a backend as BSD is battle hardened and I wanted to use its secondary DBs. BTW, I had no reply to email sent to greg(a)electricrain.com, the maintainer of the Python bsddb interface. I presume you are using that code? Anyway, I am using the BTrees provided by ZCatalog in Zope. Apart from things like robustness, speed, and number of objects I could store, I very much wanted to avoid having multiple persistence systems. Using ZODB makes this very easy. It sounds like you have that advantage too. How well supported is Durus? What about your interface? Are you in Spain? We can discuss that offline I guess. Regards, Terry

2 1