Mailman 3 April 2008 - Twisted

[Twisted-Python] Problems with trial in 8.0.1
by Brian Granger 11 Apr '08

11 Apr '08

Hi, I just got around to installing twisted 8.0.1 (from the tarball) and I see the following when I run trial twisted Traceback (most recent call last): File "/usr/local/bin/trial", line 5, in <module> pkg_resources.run_script('Twisted==8.0.1', 'trial') File "/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/pkg_resources.py", line 448, in run_script File "/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/pkg_resources.py", line 1160, in run_script execfile(script_filename, namespace, namespace) pkg_resources.ResolutionError: No script named 'trial' I did not use setuptools to install twisted (just plain old python setup.py install). Any ideas? Brian

2 2

[Twisted-Python] failures, errbacks and exception handling
by Atilla 09 Apr '08

09 Apr '08

Im a little confused when it comes to handling some errors in twisted. More specifically - when it comes to processing failures that are raised in adbapi. Now - when runInteraction succeeds, all is ok and things are commited, that's nice and clean. When it raises an error, I can do two things - either wrap my calls inside the callable passes to runInteraction in Try:Except, or attach an errback. I'm not exactly sure which one to take. I only need to catch specific exceptions - Mysql OperationalError (timeouts and such, in case I want to retry). Everything else is critical and should log & abort. The way I understood it, I should attach an errback where I call trap() on the passed Failure. So I'd do like handleError(falure): error = failure.trap(OperationalError) if error: log.err(error) myCall.addErrback(handleError) As far as I get it, if the error is different (say, syntax one) it'll be propagated. However afteri add the trap call() nothing else gets past that handleError, no Unhandled Errors are logged, and my error object contains nothing. I wonder what am I missing here.

3 4

[Twisted-Python] Re: [Stackless] Deferreds and Tasklets
by Simon Pickles 09 Apr '08

09 Apr '08

Carlos Eduardo de Paula wrote: > Hi Simon, > > Have you looked at the examples project? I made a small example using > PB in a producer/consumer demo where the producers and consumers > access a shared pool in a server using PB. > > You can see it here: > > http://code.google.com/p/stacklessexamples/wiki/StacklessTwisted Oh yeah! This is really great. I'd seen this a while back, but forget it was combining stackless/twisted so nicely! Thanks! > > http://stacklessexamples.googlecode.com/svn/trunk/examples/twisted/twistedP… > > http://stacklessexamples.googlecode.com/svn/trunk/examples/twisted/twistedP… > > > Regards, > > Carlos > > > On 06/04/2008, at 13:42, Simon Pickles wrote: > >> So I am still fiddling with interesting ways to combine stackless and >> twisted. Its fun, but I am probably naively hacking! (I always seem to >> want to use the deprecated code!- probably says a lot about me?) >> >> I'm using Perspective Broker to do remote calls to an app running >> Twisted-Stackless. PB is positively brilliant, btw. >> >> So say the a remote caller wants to get data from my app. It calls >> remote_GetData() in my PB app, which uses the actor model(each object >> has a channel.receive() running in its own tasklet). >> >> remote_GetData() has to send a message thru a stackless.channel() to get >> the required data, then it must stackless.schedule() >> >> Now it waits for the result to arrive at its own channel.receive(), >> which then needs to be passed back as the return value of >> remote_GetData(). >> >> Is this possible? >> >> Perhaps like this: >> >> from twisted.spread import pb >> from twisted.internet import reactor >> import stackless >> >> dbRx = stackless.channel() >> pbRx = stackless.channel() >> >> class DataSource: >> def Listen(self): >> while 1: >> msg = dbrx.receive() # a channel will arrive here >> msg.send(42) >> stackless.schedule() >> def __init__(self): >> stackless.tasklet(self.Listen)() >> >> >> class pbBroker(pb.Referenceable): >> def remote_GetData(self): >> dbRx.send(pbRx) >> result = pbRx.receive() >> # return 42 to remote caller thru pb >> return result >> >> This isn't ideal since I don't want pbBroker to block during the >> receive. I'd like it to closer resemble the Listen loop of the first >> class, so other objects can talk to pbBroker while it is waiting. Maybe >> like this: >> >> class pbBroker(pb.Referenceable): >> def Listen(self): >> while 1: >> msg = self.rx.receive() >> # do stuff with msg from other local objects >> stackless.schedule() >> def __init__(self): >> self.rx = stackless.channel() >> stackless.tasklet(self.Listen)() >> def remote_GetData(self): >> ch = stackless.channel() >> dbRx.send(ch) >> # can block since its a dedicated channel >> result = ch.receive() >> # return 42 to remote caller thru pb >> return result >> I guess the same problem arise when remote_GetData() has to use a >> deferred too. >> >> from twisted.enterprise import adbapi >> dbPool = adbapi.ConnectionPool("MySQLdb", db="db", user="user", >> passwd="pw") >> >> class pbBroker(pb.Referenceable): >> def remote_GetDeferredData(self): >> d = dbPool.runInteraction(DoQuery, "SELECT * FROM users") # >> DoQuery just wraps a query with try/except >> d.addCallback(self.GetDeferredData_Success) >> >> def GetDeferredData_Success(self, result): >> # return to remote caller thru pb >> return result ## WRONG >> >> How can I return a value for a remote_* call when I only get that value >> thru another callback? >> >> Sorry its a long and confusing one! >> >> Simon >> >> >> -- >> Linux user #458601 - http://counter.li.org. >> >> >> >> >> >> -- >> Linux user #458601 - http://counter.li.org. >> >> >> >> >> _______________________________________________ >> Stackless mailing list >> Stackless(a)stackless.com >> http://www.stackless.com/mailman/listinfo/stackless > > > -- Linux Counter: User# 424693

1 0

[Twisted-Python] April 20 Sprint
by glyph＠divmod.com 09 Apr '08

09 Apr '08

We're planning on having a sprint at the Divmod and Synthesis Studios offices in Cambridge, MA on April 20. While I know that some people who might be there will probably be working on twisted.conch, twisted.words, and twisted.web, the main purpose of these sprints is to help facilitate the expansion of the Twisted community. This sprint will have a little more planning than the last 3, which is to say "some, but not much". I'd like to get a feel for how many people would like to come, and maybe even get the foundation to buy us a pizza or something. I, for one, will definitely be there, of course. If you will be too, please reply to this message. I have specifically avoided mentioning any probable attendees by name so that you will all chime in and hopefully attract some more people :). Also, if you want to host a parallel sprint in a different city, it would be great to keep the momentum rolling all the way across the globe; we could reduce review latency by having reviews taking place at the sprints as well, after the earlier ones have ended. Sydney and the Bay Area, I'm talking to you. The sun never sets on the Twisted empire, after all. And that is why we must destroy it, so we can all get some rest.

13 19

[Twisted-Python] Problem with unittesting HTTP POST resource
by Teemu Harju 09 Apr '08

09 Apr '08

Hi, I'm kind of a newbie in what comes to using Twisted and I'm stuck with a problem that I cannot seem to solve. My problem is that I'm experimenting with "trial" to be able to test my code. (I've heard that it is a wise thing to do ;-)). What I have is a simple HTTP resource that allows POSTing data. I made this example to illustrate my problem... import urllib from twisted.internet import reactor from twisted.web import resource, server, client, error from twisted.trial import unittest class TestResource(resource.Resource): def render_POST(self, request): return str(request.args) class SimpleResourceTest(unittest.TestCase): def _listen(self, site): return reactor.listenTCP(0, site, interface="127.0.0.1") def setUp(self): r = resource.Resource() r.putChild("test", TestResource()) site = server.Site(r, timeout=None) self.port = self._listen(site) self.portno = self.port.getHost().port def tearDown(self): if self.port: return self.port.stopListening() def getURL(self, path): return "http://127.0.0.1:%d/%s" % (self.portno, path) def test_postdata(self): d = client.getPage(self.getURL("test"), method="POST", postdata=urllib.urlencode({"test": "data"})) d.addCallback(self.failIfEqual, "{}") return d I used some of the unittests in Twisted Web as an example and at least I think I've got this right. However, as you can see if you run this, the test fails. Which means that the request.args dictionary is empty. However, there should be something since I've set the postdata parameter on the client.getPage() method. I would really appreciate all the help here since I cannot figure out where the postdata disappears. Regards, Teemu -- Teemu Harju email/jabber: teemu.harju(a)gmail.com blog: http://www.teemuharju.net ~~ "A computer is like air conditioning: it becomes useless when you open windows." ~~

2 2

Proposed tracker tag (was Re: [Twisted-web] Re: [Twisted-Python] Thumbs up for TwistedWebPlan)
by Jonathan Lange 09 Apr '08

09 Apr '08

On Wed, Apr 9, 2008 at 10:03 AM, Phil Christensen <phil(a)bubblehouse.org> wrote: > I'd really love it if someone could help me with my web1 port of web2's > WSGI module. It's completely functional, and no longer has any dependencies > on web2, but it needs unit tests. > > http://twistedmatrix.com/trac/ticket/2753 > > I started working on imitating the test for the web2 version > (twisted/web2/test/test_wsgi.py), but it's been idling in my workspace for > some time, so I thought I'd just put a feeler out for anyone willing to give > it a try, or even (gasp!) write some tests. ;-) > I think a few tickets fall into this category. They have patches that fix code, but for one reason or another they lack tests. Could we, as a policy, add a 'patch' tag to such tickets when we see them? jml

1 0

Re: [Twisted-Python] April 20 Sprint
by Andrew Francis 08 Apr '08

08 Apr '08

Hi Glyph and Colleagues: >We're planning on having a sprint at the Divmod and >Synthesis Studios offices in Cambridge, MA on April >20. Boston is close and I have never been there. Not sure how busy I am that week. Also I am not sure what a sprint entails. Off the bat, stuff that would interest me is writing protocols (I am interested in MySQL), corotwine/stackless, SOA and Twisted. Cheers, Andrew ____________________________________________________________________________________ You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. http://tc.deals.yahoo.com/tc/blockbuster/text5.com

1 0

Re: [Twisted-Python] Deferreds and Tasklets
by Andrew Francis 07 Apr '08

07 Apr '08

Hi Simon: [quote] So I am still fiddling with interesting ways to combine stackless and twisted. Its fun, but I am probably naively hacking! (I always seem to want to use the deprecated code!- probably says a lot about me?) [/quote] Simon I am all for fiddling! I was lucky to be around some great fiddlers. "Adventures in Stackless Python / Twisted Integration" is about my fiddling. One of the things I tried to do with "Adventures" was show the how and why's of the basic techniques. If there is something you don't understand, please ask. That way I can improve the explanations. Some additional comments. 1) I try to learn from what others have done concerning Stackless and Twisted - i.e., Richard Tew and Christopher Armstrong. I found Richard's TinyMud easier to follow than blockOn but got that both were variations of the same approach. The main thing I took away from there was learning how deferred/callbacks/channels/tasklets interacted. Learning things on your own is cool. However sometimes you are just re-inventing the wheel.... 2) My knowledge increased when I understand more about what Twisted and Stackless did under the hood. One needs a descriptive model. Without that I would not have been able to solve the deadlock/starvation issue when making what is essentially a nested call. I am not sure if we are entirely out of the woods yet there (in part I want people to punch holes in the techniques). And there is much about Twisted that I need to learn. Take away, a deeper understanding prevents you from spinning your wheels. 3) I try to combine 1 & 2, with literature surveys to create a predictive model - how can we test? How can we improve things? I believe there is considerable room for improvement. Take away, you don't want your wheels stuck in a rut. Cheers, Andrew ____________________________________________________________________________________ You rock. That's why Blockbuster's offering you one month of Blockbuster Total Access, No Cost. http://tc.deals.yahoo.com/tc/blockbuster/text5.com

1 0

[Twisted-Python] Vertex or other P2P libraries for twisted?
by Vince Cate 07 Apr '08

07 Apr '08

I would like a library that I can use from Twisted to help with peer-to-peer communication. I have looked at and tried Vertex: http://divmod.org/trac/wiki/UsingVertex This is the kind of thing I want. But I am having some problems with it. Are there other similar libraries out there? What are people using? The problem we have with Vertex is sending from Windows to Linux. We can do Linux to Windows. The error we get is in file ptcp.py in class PTCP def connect(self, factory, host, port, pseudoPort=1) with factory <vertex.q2q.RPTCPConnectionAttempt instance at 0x010CEBE8> Any help appreciated. -- Vince Cate

5 7

[Twisted-Python] adbapi, transactions and threading
by Atilla 07 Apr '08

07 Apr '08

I hope it won't be too confusing. I'm basically trying to make sure I've grasped the concept - do correct me on anything I am wrong about or my way of thinking confuses you, please. I'm not used to running blocking calls and deferToThread a lot, so I wanted to make sure I understand that correctly. Basically, the way I understand it - adbapi is just a deferToThread wrapper around the normal python API, correct? As in - if I used something different to access my database, for example - sqlalchemy, I would only need to appropriately wrap that functionality with deferToThread, just as adbapi does? That's one thing. Now - what I'm trying to do in essence is to load some big chunk of data out of the DB, process it, and save it back. I'd use a nice chain of deffered calls - one runQuery, one for the processing, and one runOperation. However, I need transaction functionality, so unless I'm mistaken, my only choice is runInteraction. Since that's automatically ran in a separate thread, I see it as a monolithic piece of code -> query, processing call, saving query -> no deffereds can take place there. Am I wrong on that one? Even if so - it won't be that bad for me, since good part of the processing will be handled by an external library (GIL released - says so, more cores used, makes me happy). That leads me to the next question - when I've got long code to run and it happens to use an external, thread-safe, C library, releasing the GIL, I should probably always take care to defer it to thread, if I wanted to take advantage of multiple cores, correct? Otherwise, I wouldn't have any parallelism gains, which I can get, because of the GIL release. And let's say that my processing code can take a while sometimes. Which leads me to the other question - what should I do in the case where I need to occasionally run big chunks of code. No blocking calls, just crunch something down. Is deferToThread the only solution for that? Is the idea to compose that as big deffered chains so other processing might run normally, instead of wait for the big function to exit? Because deferToThread will only get me anything, if there's a blocking call inside, or if I mange to get parallelism out of it, if it's something handled by GIL-released code. Sorry if I sound too confusing, I'm trying to wrap it all in my head before I dive in handling the service.

3 3