Mailman 3 June 2009 - Twisted

[Twisted-Python] stop conditions and deferToThread()
by Thomas Jakobsen 12 Jun '09

12 Jun '09

Hi As discussed in a previous thread http://twistedmatrix.com/pipermail/twisted-python/2009-May/019717.html a task put in its own thread via deferToThread() won't stop even though the reactor has stopped. It has to carry out its own check that the reactor is running and stop if that is not the case. My question is: Is it possible to use anything else than the reactor.running as stop condition in this way? In my current twisted application I would like to keep the reactor running but still have some deferToThread jobs stop if e.g. an exception occurs in one of the jobs. In the code example below I try to use my own global field, running, as stop condition, but it doesn't work. If I let bar() call reactor.stop() after the time.sleep(3) and use reactor.running in foo() it works, however. import time from twisted.internet import reactor from twisted.internet.threads import deferToThread running = True def foo(): while running: print "working" time.sleep(1) def bar(): time.sleep(3) print "stopping" running = False d1 = deferToThread(foo) d2 = deferToThread(bar) reactor.run() Best regards, Thomas Jakobsen

4 4

Re: [Twisted-Python] Need working examples of imap4 client.
by Pywinder Singh 12 Jun '09

12 Jun '09

Tarjei, Yep. I actually grabbed it, but on perusal, but its use of twisted seemed a little too layered to help me at my current twisted expertise. Of course, now that I think I have the problem narrowed down to the login phase, it might help. Tnx. Tarjei wrote: > On 06/11/2009 02:03 AM, Pywinder Singh wrote: > > Can someone point me to a working example of imap4 client? > > > > I've tried the example off the site, code from Abe Fettig's book, and > > every example I could find on the web. (most of them seem outdated, as > > do most discussions I've found on the subject) > > Have you looked at chandler? AFAIK it uses (or used to use) Twisteds > IMAP code. > > > Regards, > Tarjei > > > > > All of them invariably end with: > > > > twisted.internet.error.TimeoutError: User timeout caused connection > > failure > > > > or > > > > twisted.internet.error.ConnectionLost: Connection to the other side > > was lost in a non-clean fashion. > > > > > > > > I've pointed the code at two different IMAP servers, both of which (one > > is gmail, the other is a private server) I am able to access with a > > hitch with python's own imaplib. I've tried the sample code on an Ubuntu > > system, OSX, and tried python versions 2.5 and 2.6. > > > > Ideally, I'd love to see a snipped which is able to log into an imap > > server and gets a list of mailboxes. If the example on the site works > > for other folks, I'd love a nudge in the right direction to modify the > > code to get a bit more debugging info. > > > > Thanks in advance! > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Twisted-Python mailing list > > Twisted-Python at twistedmatrix.com > > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > -- > Tarjei Huse > Mobil: 920 63 413 >

1 0

[Twisted-Python] Need working examples of imap4 client.
by Pywinder Singh 11 Jun '09

11 Jun '09

Can someone point me to a working example of imap4 client? I've tried the example off the site, code from Abe Fettig's book, and every example I could find on the web. (most of them seem outdated, as do most discussions I've found on the subject) All of them invariably end with: twisted.internet.error.TimeoutError: User timeout caused connection failure or twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion. I've pointed the code at two different IMAP servers, both of which (one is gmail, the other is a private server) I am able to access with a hitch with python's own imaplib. I've tried the sample code on an Ubuntu system, OSX, and tried python versions 2.5 and 2.6. Ideally, I'd love to see a snipped which is able to log into an imap server and gets a list of mailboxes. If the example on the site works for other folks, I'd love a nudge in the right direction to modify the code to get a bit more debugging info. Thanks in advance!

3 2

[Twisted-Python] Application Design help - Concurrent but not Protocols based.
by Senthil Kumaran 10 Jun '09

10 Jun '09

Hello Twisted Developers/Users, This is my first concurrent application design and my first trial with twisted. I have read the documentation and understand where twisted plays its part. Unfortunately, I could not directly relate it to my requirements and hence, could not go forward with designing and building on top using the examples as a reference. I need your guidance in helping me design an application. My Application Details: 1) I need to constantly monitor a particular directory for new files. 2) Whenever a new file is dropped; I read that file and get information on where to collect data from that is a) another machine b) machine2-different method c) database. 3) I collect data from those machines and store it. The data is huge and I need the three processes a, b, c to be non-blocking, and I can just do a function call like do_a(), do_b(), do_c() to perform them. For 1) to constantly monitor a particular directory for new files, I am doing something like this: while True: check_for_new_files() http://paste.pocoo.org/show/120824/ My Question: Can this be designed in way that looking for new files is also asynchronous activity? What will be the deferred in this case? # my ideas: - I might define a deferred as, whenever the contents of the directory is not matching the previous contents, return the new file which was added. - I can then add a callback to read the newfile. Now, after reading the contents, I will have to do a non-blocking call to fetch data, either using fun_a, fun_b or fun_b. How should I associate this requirement to deferred/callback pattern? Any guidance would be helpful. Thanks, Senthil

5 5

Re: [Twisted-Python] Application Design help - Concurrent but not Protocols based.
by Senthil Kumaran 10 Jun '09

10 Jun '09

On Wed, Jun 03, 2009 at 10:43:34AM -0700, Minesh Patel wrote: > For this, since your data is huge and seems like it will need a lot of > CPU utilization, you might have to deferToThread, since Twisted runs > in a single thread and will block AFAIK. > > d = threads.deferToThread(do_a, arg1_to_do_a, arg2...) > d.addCallback(do_a2, ...) > I delayed and forgot to respond to this thread. Sorry for that. Actually, I had asked the question on doing asynchronous monitoring of new file arrival and then based on the file contents, doing tasks a,b,c asynchronously. Thank you for your responses. I got a number of pointers to look at. * As a newbie to this world, I just started studying further. I realized that in my application, I have the control form the client on when I can notify that a particular file can be looked at. So, for the time being, I just skipped the inotify linux system related calls ( BTW, Twsited + inotify works great) and when ahead with just implementing a Twisted a Server which listens to a particular port and when receives data, will either read the file or parse the data to do some activity (like fetch logs from database). threads.deferToThread(func,[args]) is what I am using. http://paste.pocoo.org/show/122155/ I am still working out more details as doing this threads.deferToThread calls effectively, because there could be number of clients parallely requesting it and the data from my query (internal function) could be so big that it might take 30 minutes to service a request. * Another thing is when using something like "from datasources import query" I could not make it to application (.tac to run it from twistd), it could nto find datasources in its PYTHONPATH. Got to figure out how to do it, which I might pretty soon. Thanks again, Senthil

1 0

[Twisted-Python] copy domish element
by Jack Bates 09 Jun '09

09 Jun '09

i have a domish element - when i try the following, >>> message = element >>> del message['from'] >>> element['from'] - i get a key error to avoid this key error, i think i want to copy the element, so i can drop the 'from' attribute from one copy without affecting the other? but i didn't find anything to make a copy in the domish element api how should i avoid this key error?

3 3

[Twisted-Python] some callback function params problems
by hoooooosety 09 Jun '09

09 Jun '09

Hello, I have some callback function params problems. Here is a simple example. def deferDoSomeInsert(paramA, paramB): insertSQL = "..." db.runQuery(insertSQL) def deferDoSomeDelete(paramA, paramB): deleteSQL = "..." db.runQuery(deleteSQL) #these two functions are individual and basic, and then, I need compound function which first insert then delete. #so def deferCompoundFunc(paramA, paramB): return deferDoSomeInsert.addCallback(deferDoSomeDelete, paramA, paramB) here the problem comes, the first defer will return none as default so there're 3 pramas pass to the second function. Is there a common way to solve the problem like this. I mean I can still keep the separate functions as the look like now, and I can compound the basic function to do more complex action using addCallback to link them. 2009-06-09 hoooooosety

2 1

Re: [Twisted-Python] Twisted Words
by Alex Patino 09 Jun '09

09 Jun '09

I am looking for a chat system to add to my Django-based chat site. I don't need something ready-made but I don't want to build it on a framework that is no longer supported. What's the status of the Twisted Words project? Regards, Alex On Mon, Jun 8, 2009 at 11:25 AM, Alex Patino <patino(a)gmail.com> wrote: > I am looking for a chat system to add to my Django-based chat site. I > don't need something ready-made but I don't want to build it on a framework > that is no longer supported. What's the status of the Twisted Words > project? > > Regards, > Alex > > > On Mon, Jun 8, 2009 at 8:34 AM, Jean-Paul Calderone <exarkun(a)divmod.com>wrote: > >> On Sun, 7 Jun 2009 23:50:59 -0400, Alex Patino <patino(a)gmail.com> wrote: >> >>> I have been looking for a chat server/client to add to my Django-based >>> website. What is the current state of Twisted Words? Do you think it's >>> something I could use for this endeavor? >>> >> >> Twisted Words does not include a ready-made web-based chat system, so if >> that's what you're looking for, you'll have to look elsewhere. >> >> Please direct follow-up questions to twisted-python(a)twistedmatrix.com. >> >> Jean-Paul >> > >

3 2

Re: [Twisted-Python] inlineCallbacksDecorator
by Terry Jones 09 Jun '09

09 Jun '09

>>>>> "Drew" == Drew Smathers <drew.smathers(a)gmail.com> writes: Drew> On Sat, Jun 6, 2009 at 9:41 PM, Terry Jones<terry(a)jon.es> wrote: >> One inconvenience with inlineCallbacks is that you might have some kind >> of processing you want done no matter how the function returns, or >> wherever an error occurs. Two solutions here are 1) to put try/except >> calls around your various yields, and/or to perhaps do something else >> with various callbacks that might call defer.returnValue, or 2) expect >> each caller of your method to deal with the result. I don't like the >> first of those much (depending on the code), and don't like the second >> at all. Drew> How does inlineCallbacks preclude you from adding "processing you Drew> want done no matter how the function returns" or currying callbacks? Drew> From my understanding of the problem, this is already solved by Drew> attaching callbacks/errbacks in the regular way: Drew> @inlineCallbacks Drew> def foo(a, b, c): Drew> ... Drew> foo(a, b, c).addBoth(cb, a, b, c) Right, that's what I meant by the (2) in my paragraph. The reason that's not always what you want is that there may be things foo needs to get done before it returns, things that are best or can only be computed inside foo. In that case you don't really want to make all your callers have to add a callback to add functionality/logic that belongs inside foo. Under normal circumstances you'd have code like this: def foo(): d = someOtherFunc() d.addCallbacks(...) return d or def foo(): d = defer.Callback() # Set up some stuff that will fire d return d so the deferred that comes back from foo is under control of foo and if there's any logical last steps to be done to it, foo can easily add them. But with inlineCallbacks, foo never has its hands on the deferred that will be returned. Yes, you could do: @inlineCallbacks def foo(): try: # # All sorts of stuff, yielding deferreds # except ...: # yield somethingElse or raise something But that's not so attractive, and it only deals with errbacks. To deal with all returns, you could catch the undocumented _DefGen_Return, do your processing and then re-raise it but that's even more ugly. So I'm just trying to offer a clean way to always get hold of the deferred and be able to consistently process it using code in the same file (or class, given a self) as foo. Terry

1 0

[Twisted-Python] inlineCallbacksDecorator
by Terry Jones 09 Jun '09

09 Jun '09

I try to avoid using inlineCallbacks. There are two cases where I will happily use it though: when I write a method that would need more than a few callback functions, and, more importantly, when the logical flow of a method is non-trivial (i.e., it depends on the returned values or errors of several deferreds). One inconvenience with inlineCallbacks is that you might have some kind of processing you want done no matter how the function returns, or wherever an error occurs. Two solutions here are 1) to put try/except calls around your various yields, and/or to perhaps do something else with various callbacks that might call defer.returnValue, or 2) expect each caller of your method to deal with the result. I don't like the first of those much (depending on the code), and don't like the second at all. So I wrote a decorator specifically for inlineCallbacks decorated functions: from twisted.internet import defer def inlineCallbacksDecorator(callback, errback=defer.passthru): def wrap(f): def wrapper(*args, **kw): d = f(*args, **kw) if isinstance(d, defer.Deferred): return d.addCallbacks(callback, errback, callbackArgs=args, callbackKeywords=kw, errbackArgs=args, errbackKeywords=kw) # We were used to decorate a non-inlineCallbacks function. raise Exception( 'Function %r was decorated with inlineCallbacksDecorator but ' 'did not return a deferred (did you forget to also decorate ' 'with inlineCallbacks?)' % f.__name__) return mergeFunctionMetadata(f, wrapper) return wrap You can use it like this: def _cbok(result, a, b=None): print 'In _cbok: a=%r b=%r' % (a, b) return result def _cberr(failure, *args, **kw): pass # Do something.... and maybe also return the failure. @inlineCallbacksDecorator(_cbok, _cberr) @defer.inlineCallbacks def frog(a, b=None): print 'a = %r, b = %r' % (a, b) result = yield produceDeferred() # yield produceErr() defer.returnValue(a) The nice/interesting thing about this is that the callback and errback functions get called with the deferred result of calling frog (as decorated by inlineCallbacks) *and* the original arguments passed to frog. You can of course ignore the original args if you're not interested in them. You can argue that this doesn't really buy you anything. That's of course true. It's only a decorator that makes it neater to do certain things. Logging, accounting and error processing come immediately to mind. Just keeping code looking simpler/cleaner is enough of a reason for me. After writing this, I realized it deals with what I was clumsily trying to achieve here http://twistedmatrix.com/pipermail/twisted-python/2009-April/019492.html back in April. Terry

3 2