Mailman 3 November 2008 - Twisted

[Twisted-Python] Twisted Python 0.7.0 Prerelease Testing
by Glyph Lefkowitz Aug. 5, 2021

Aug. 5, 2021

hi there, folks: I'd really like to release 0.7.0 but I would like it to be at least a little bit tested before I do so. Could those of you with CVS trees check everything out and see if it performs as advertised? Deeper bugs than that will have to wait for the next release, but I'd at least like to know if it works for someone other than me. Thanks. ______ __ __ _____ _ _ | ____ | \_/ |_____] |_____| |_____| |_____ | | | | @ t w i s t e d m a t r i x . c o m http://www.twistedmatrix.com/~glyph/

4 5

[Twisted-Python] regarding xml elements
by Raj kumar Oct. 2, 2013

Oct. 2, 2013

Hi, I have an xml file in my application, I have created an element using Example goes like this......... document.createElement("abc") and i appeneded it by using append() method. But how i can reflect this change to my xml file? and one more thing is i want to create element with some other parameters.... like............. <abc m=" " n=" "> </abc> and i have m and n values as strings with me. can anybody help me to create this element and write it to the existing xml file as a child of an existing element? Thanks in advance.. Raj.. Forgot the famous last words? Access your message archive online at http://in.messenger.yahoo.com/webmessengerpromo.php

9 14

[Twisted-Python] Get IP of connecting client in XMLRPC server
by Remy C. Cool April 28, 2013

April 28, 2013

Hello, On 18 May 2004, the following message was posted to this mailinglist: Jp Calderone exarkun at divmod.com wrote: >Daniel Newton wrote: > > I have a simple XML-PRC server similar to the example below: > > from twisted.web import xmlrpc, server > > class Example(xmlrpc.XMLRPC): > """An example object to be published.""" > > def xmlrpc_add(self, a, b): > """Return sum of arguments.""" > return a + b def xmlrpc_whatIsMyAddress(self): return self.transport.getPeer().host > > if __name__ == '__main__': > from twisted.internet import reactor > r = Example() > reactor.listenTCP(7080, server.Site(r)) > reactor.run() > > I want to be able to get the address of the client that calls the xmlrpc > method can anyone help me with this? This solution didn't work because 'transport' isn't a property of the Example class. I'm currently in the process of changing from a customized SimpleXMLRPCServer to a twisted XMLRPC server solution and I need to insert the client IP into the attributes passed to the called xmlrpc method. Anyone who knows the answer and is willing to share the info? Regards, Remy

3 3

[Twisted-Python] An XMPP client with SSL support
by Pedro Sanchez Sept. 18, 2009

Sept. 18, 2009

Hello, I have a sample XMPP client with the following structure: from twisted.words.protocols.jabber import client, jid from twisted.internet import reactor ... factory = client.basicClientFactory(myJid, myPasswd) reactor.connectTCP(host,5222,factory) reactor.run() It works well, I can connect to the XMPP server. But I would appreciate if some one could give me some hints on what's required to get the client to connect using a secure SSL link. Thanks, -- Pedro

3 3

[Twisted-Python] SQLAlchemy and Twisted
by alex clemesha Feb. 13, 2009

Feb. 13, 2009

Hi, I didn't want to hijack the 'Twisted Presentation Materials..' thread, so I started this one ... First off, in the other thread, Glenn, you mentioned Storm. I have heard of Storm, but never got any deeper that noticing that it is still a work in progress (maybe things have settled down ?) If anyone has anything to say on how it might be a good choice with Twisted, please feel free to enlighten us! But, I hoping to solve some immediate problems with Twisted and SQLAlchemy, so ... >> >> * A main "App Server", that controls high level access with "twisted.cred" >> * Web frontend: "twisted.web2" >> * Data: some "twisted.enterprise.adbapi", moving to SQLAlchemy. > > Interesting; how are you handling asynchrony in SA? This is the issue that we're stilling unclear on, and I really would like to know of the best way of integrating Twisted with SQLAlchemy. Twisted devs / experts: Please, could you comment on the right way of using Twisted and SQLAlchemy together. Anyone: If anyone know of existing code that integrates Twisted and SQLAlchemy nicely, please share! RIght away I just went about the problem be wrapping each call to SQLAlchemy in a "deferToThread" inside my "DatabaseManager" class. At startup of my app, I make an instance of this class, and pass it to each "avatar" (in "requestAvatar", in my portal) Here is my Database Manager: (you will actually see that I 'turned off' the deferToThread wrapping for now): http://trac.knoboo.com/browser/branches/knoboo-sqlalchemy-take2/knoboo/data… but I really haven't convinced myself that this is the best way of doing things. Hell, it could be totally wrong, so if anyone has any advice, I would be very appreciative! > I'm aware of sAsync, > but haven't looked much into it. (I'm a step or two behind you, actively > using adbapi, and thinking about moving to SA when I get some breathing > room.) I've looked into SAsync as well, but it is not totally clear to me what the extra benefit of it is (again, anyone who has a good description, please tell) The main developer of sAsync use to post to this mailing list, but I haven't seen him post in a fairly long while. So, if others are thinking about using SQLAlchemy with Twisted, let's discuss it more. Thanks very much, Alex

5 5

Re: [Twisted-Python] What to do when a service fails to start, also, deferred and startService
by Terry Jones Dec. 21, 2008

Dec. 21, 2008

Hi Glyph Thanks for the detailed reply. >>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> On 27 Nov, 05:16 pm, terry(a)jon.es wrote: glyph> For me, baroque and elaborate start-up dances are a code smell. glyph> Services should be as independent as possible. Of course, sometimes glyph> some kind of initialization conversation is unavoidable, but I do glyph> like to try to keep it as short as possible. I do too. Sometimes it takes (me at least) a few iterations before you see how best to do that. glyph> I think you're misunderstanding what a "service" is. The word is, glyph> perhaps, a bit to lofty for its humble job. A service is just an glyph> event notification mechanism that tells you when it's time to start glyph> up, and when it's time to shut down. glyph> I can understand why it would be attractive to misunderstand in this glyph> way, though: IService doesn't do very much, you have requirements glyph> that it doesn't cover, and if it were the thing you understand it to glyph> be then it would cover those requirements. I'm sure that would be glyph> nicer for you :). glyph> This might seem a bit inconsistent, since stopService uses the glyph> return of a Deferred. However, this is for a very specific reason, glyph> not a generalized error-handling case: you may need to prevent the glyph> *rest* of the system (specifically, the reactor) from completely glyph> shutting down until you've managed to cleanly shut down whatever glyph> you're trying to shut down on potentially remote systems. glyph> startService has no such problem though; the service subsystem has glyph> told you "It's time to start up!" - its job is done, and the reactor glyph> isn't going away as part of service startup, so it's your glyph> responsibility as an application author to make sure your other glyph> dependencies are properly initialized. OK, this is helpful - I have been looking at it from a different point of view, as you've guessed. >> But if something does go wrong, you've got a failure propagating its way >> down a errback chain, eventually (unless an errback switches you back to >> the callback chain) popping out the end and causing the reactor to issue >> an Unhandled Error message. So you can't indicate that the service has >> failed to start by throwing, because the exception is going to pop >> harmlessly out the end of the deferred chain as a generic unhandled >> error and will not cause Twisted to know that the service couldn't >> start. glyph> The key question here is: indicate to whom? If you want to indicate glyph> it to some other object, well, try:except: or addErrback and call a glyph> method on that object. Nothing magic about it. I have code written as a Twisted plugin. So I have a class implementing IServiceMaker and IPlugin, and I create an instance of that class which gets found when I invoke twistd from the command line. So in my case I want to indicate to twistd that the service that my class creates a makeService method to create, but which I do not set in motion, has failed to start and that twistd should exit, or do something other than cheerfully tell me that there's been an Unhandled Error. Does that make more sense? Sorry, I should have said I was using twistd. glyph> In what way would you expect the service mechanism to "deal with" glyph> returning a Deferred? Stop starting other services? Print out some glyph> different log message? I'm not sure what should happen. I'm sitting at the command line, I've asked twistd to start something for me, there's clearly been a problem doing so (and this doesn't have to be baroque, maybe I just couldn't listen on a specific port I wanted, or maybe my code somehow raised an Exception), but I don't seem to have a mechanism for having twistd take any notice at all. I'm just talking about the case where startService calls something that returns a deferred and there's an Exception that comes back down the Deferred chain as a failure. I suppose if startService raises an Exception itself directly, something else happens - maybe twistd exits. glyph> IService is a very, very simple interface. If you want to respond glyph> to failures from startService (deferred failures, exceptions, or glyph> whatever else) in a useful way, then you can write your own glyph> implementation of it which manages startup order, keeps track of glyph> dependencies, and maintains a state machine that handles stopService glyph> appropriately if called in mid- startup. glyph> I don't think that having to implement an interface with 6 methods glyph> on it could be considered "cruel and unusual". If you think so you glyph> may want to investigate options other than Twisted: you will glyph> frequently be expected to implement interfaces with methods on them glyph> ;-). :-) glyph> There's no need to "track down and subclass" lots of things. Your glyph> IService wants the things that it contains to have a richer glyph> interface which allows for error handling, dependencies, and glyph> propagation, so simply write a single wrapper for simpler IService glyph> objects that expands the interface to do the other things that glyph> you're interested in. In the case of a service being started by twistd, it doesn't seem as simple as you describe, but maybe that's my lack of understanding again. I can easily subclass IService, but something else is calling the startService method of that subclass. And that thing, whatever it is, is not expecting me to return a deferred. So if my startService has for some reason got its hands on a deferred, it can't simply hand it back to its caller and have something (twistd in my case) see that an error occurred. It does feel like I have to track down what this something else might be. Either working from my IServiceMaker implementation or working from /usr/bin/twistd to find where startService is not trivial (you guys wrote it, I'm sure the logic is all much clearer to you). After looking through a few files I wind up at twisted/application/app.py, which has a startApplication function that calls service.IService(application).startService(). So I guess that's what is calling my startService. So I could make my own startApplication function, but I then have the same problem, I wind up with a deferred on my hands and my caller is not expecting me to return it. Plus, the startApplication function sits at the top level of twisted/application/app.py, so I have to find whatever is calling that. That seems to be twisted/scripts/_twistd_unix.py, which imports twisted.application.app and has a top-level startApplication that calls app.startApplication. But who is calling that? Looks like twisted/scripts/twistd.py is, and that's called by /usr/bin/twistd. So should I write my own twistd? All this doesn't seem to be a matter of simple subclassing. Plus, I can't just go in and start editing the top-level functions in twisted/application/app.py and twisted/scripts/_twistd_unix.py or code that imports app, etc. Sorry for so many questions - I really don't know if I'm missing something simple here. I do enjoy digging into all this, and I appreciate your apparently limitless patience. I wish I knew it all better. Twisted is complex and it's a pretty good bet that anything you think of or run into as a n00b has been thought of or encountered before, and that whatever way you think of to solve it will probably be non-optimal, or plain wrong, or in ignorance of a solution someone much more experienced has already implemented, or... etc. Hence my many questions. glyph> This all strikes me as totally straightforward and easy, and I don't glyph> think I'm any kind of super-genius for being able to write a few glyph> Python classes that call a few simple start/stop methods in the glyph> order that I want them to run in :). I should have mentioned that I want to use twistd. In fact I have something like a process pool running in one service and I talk to it from another machine. I say "hey, process pool, start me up the following service (a twistd service)" and I would then like to know if that service started, and if not then why not. So having twistd fail or report an error if it can't start a service would be useful. glyph> Doing either of those things would definitely be wrong. There's no glyph> reason to sys.exit or reactor.stop if your application can't start glyph> up, unless your management system specifically calls for such a glyph> thing. Maybe this is a case where it's (semi-)justified. At least if I called sys.exit the twistd process would go away, instead of sitting there acting as though nothing's wrong :-) I can also, of course, try interacting with the service I think I just started on the remote machine, and if I can't then I can tell the original process pool to kill the twistd process. But that seems a pretty roundabout alternative to just having twistd notice that something went awry when calling startService. glyph> In the future, even the Twisted plugin code might be starting some glyph> things in addition to your application. As I mentioned above, a glyph> good reason to do that is to perform diagnostics on failed startups glyph> :). I assume you really mean "startups" and not "services". In which case, I'm 100% sure there's something funny here, but I can't figure it out. I'd love to know, and I'm smiling broadly in any case. Too bad email loses so much humor..... we can always try though. Thanks again, Terry

2 1

[Twisted-Python] Syslog Listener UDP
by Marc Byrd Dec. 16, 2008

Dec. 16, 2008

Hi, I would like to build a syslog(udp port 514) listener in twisted. The goal is to make it compatible with syslog and syslog-ng, which may be configured on a "client" to send log entries (as they occur) via udp (fire and forget) to a remote server (my twisted server). Is there already something out there that would do this or that would be a good start? Thanks, Marc

4 4

Re: [Twisted-Python] A Python metaclass for Twisted allowing __init__ to return a Deferred
by Terry Jones Dec. 6, 2008

Dec. 6, 2008

>>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> On 02:33 am, terry(a)jon.es wrote: >> Am I missing something here? glyph> As a matter of fact, yes :). OK, thanks, I get it. That should have been obvious. BTW, I spent an interesting morning getting my head around *exactly* how inlineCallbacks does its thing... No summary of my thoughts could do it justice. I wonder how many people on the planet have been down that rabbit hole. Was the Twisted project always called Twisted? I was thinking it's a really good thing you picked that name - cause you definitely would have had to rename it :-) I hope people don't mind my peanut gallery commentary. We're all in this together, after all. A little gallows humor occasionally seems in order. Thanks again, Terry

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Nov. 30, 2008

Nov. 30, 2008

1 0

[Twisted-Python] What to do when a service fails to start, also, deferred and startService
by Terry Jones Nov. 28, 2008

Nov. 28, 2008

I've another of my pesky beginner questions. Note that this subject is somewhat covered in the thread started by Matt Goodall in Jan 2006: http://twistedmatrix.com/pipermail/twisted-python/2006-January/012380.html I imagine that it must be common that people write services that don't just simply launch things listening on sockets, but instead need to do a couple of things, one after another, in order to get going and to be ready to provide their service (or multiservice). If you do need to write something like that, it seems the chances are pretty high you're going to be calling code somewhere along the way that returns a deferred. And because the twisted/application/service.py code that calls startService doesn't handle deferreds being returned, this creates a real problem. At least as far as I understand things - which, as usual, may not be very far. If nothing goes wrong with the deferreds that startService is creating (via whatever its calling), then you'll probably get away with things even though your service will not really be up until after the deferreds fire, which can be some time after the code calling startService gets its deferred back (and ignores it). But if something does go wrong, you've got a failure propagating its way down a errback chain, eventually (unless an errback switches you back to the callback chain) popping out the end and causing the reactor to issue an Unhandled Error message. So you can't indicate that the service has failed to start by throwing, because the exception is going to pop harmlessly out the end of the deferred chain as a generic unhandled error and will not cause Twisted to know that the service couldn't start. This all feels quite ironic :-) Twisted leads you coyly into the dark and powerful world of working with and heavily depending on Deferreds. But then, right when you expect it to be there for you, covering your back, it throws up its hands as if to say "What!!? You expect me to deal with you returning a Deferred? You gotta be kidding, sucker." I could follow Moof's approach (last poster in the above thread), but that seems to just pass the problem on to a higher level, where something else is calling startService (or something earlier) and so on up until we reach the topmost point at which something is not allowing/expecting a deferred to come back. Should I track down and subclass all these things? That would seem cruel and unusual punishment to the faithful Deferred user, having to go in and subclass core classes because they don't deal with Deferreds. I could do something dramatic, like call reactor.stop or sys.exit in my errback chain, but those seem completely wrong. Apart from the (remote?) possibility that something other than Twisted plugin code is trying to start my service, it's also anachronistic because it will happen at some unpredictable time after startService has gotten back (and ignored) the deferred and Twisted has moved on (perhaps even to start other services). Terry

2 1

Re: [Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones Nov. 28, 2008

Nov. 28, 2008

Hi JP > Not quite: > > >>> def f(): > ... yield None > ... > >>> f(1, 2, 3) > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > TypeError: f() takes no arguments (3 given) > >>> Ah... thanks. > And unfortunately, there is essentially no way to differentiate this case > from a case where the call to `f´ succeeded but some code executed as a > result of that raised a `TypeError´. Right, not without looking at the text of the exception, which, apart from being extremely would be unreliable. Too bad there's not a Python exception for incorrect function/method arg passing. > See also <http://twistedmatrix.com/trac/ticket/2501>. Ha! There's pretty much the whole of this thread in there. I'll look more carefully in trac next time. I did look at some point, and saw another issue discussing what to do if a non-deferred is yielded. Thanks again. Terry

1 0

[Twisted-Python] Please test Twisted 8.2.0pre2 and tell me whether it works
by Christopher Armstrong Nov. 28, 2008

Nov. 28, 2008

I just uploaded 8.2.0pre2 tarballs, which include one bugfix since 8.2.0pre1 which was deemed important enough to be backported: - TLS Session Tickets are now disabled by default, allowing connections to certain servers which hang when an empty session ticket is received (like GTalk) (#3463) The tarballs are at http://twistedmatrix.com/users/radix/8.2.0pre2/ If you want to test 8.2.0pre2 on Windows, Thomas Hervé may be able to build installers some time tomorrow. Please test it out - I haven't heard any results from the pre1 release yet. If you don't test it then 8.2.0 will be horrible, and it will be your fault. You know, I've heard some people are using Twisted to manage puppy incubators. Think of the puppies. http://www.ustream.tv/channel/shiba-inu-puppy-cam -- Christopher Armstrong http://radix.twistedmatrix.com/ http://planet-if.com/ http://canonical.com/

4 5

[Twisted-Python] Twisted on Windows: "Python does not appear to be installed"
by Patrice RENAUD Nov. 26, 2008

Nov. 26, 2008

Hi all, Twisted does not install on my windows XP /python 2.6 computer. Twisted installer says: "Python does not appear to be installed" but python is really installed and runs well on my computer ! What is the solution ? thanks Patrice -- Patrice RENAUD Observatoire de Paris - Station de Radioastronomie de Nancay Tel: 02 48 51 87 21 Fax: 02 48 51 83 18 Email: Patrice.Renaud(a)obs-nancay.fr

2 1

[Twisted-Python] Running a GUI application based on PyQt
by Nahuel Defossé Nov. 26, 2008

Nov. 26, 2008

Hi, I'm trying to use a simple application based in twisted.internet.Protocol with PyQt4. PyQt has its own way of dealing with events, I've tried to run the reactor in a thread, but it says it only can acquire SIGINT from the main thread :( I've tried the code posted in qt4reactor.py but it doesn't work. It fails telling that a reactor's been already installed. Is it possible to use PyQt4 (v.4.4) with twisted? Thanks Nahuel

3 3

[Twisted-Python] SSL Client Certificates?
by asset Nov. 25, 2008

Nov. 25, 2008

Would I be right in assuming that the example echoserv_ssl.py and echoclient_ssl.py will only result in encryption from the client to server and not from the server back to the client? Im referencing these examples specifically, http://twistedmatrix.com/projects/core/documentation/examples/echoclient_ss… http://twistedmatrix.com/projects/core/documentation/examples/echoclient_ss… From what I know of SSL, the client connects to the server, and after the handshake it uses the cert to encrypt its communication to the server and the server uses the private key to decrypt it; but the server's traffic back to the client will not be encrypted without having to use a client certificate. So my question is how wrong am I on this? And if needed, how do I setup a client certificate to succeed in two way encryption? Thanks! TKiel

2 1

[Twisted-Python] sending file but LineReceiver doesnt run
by Aydın ŞEN Nov. 25, 2008

Nov. 25, 2008

Hi, I am trying to send file which is in the client's disk, i am using LineReceiver in the server side. I am sending and receiving data like login stuff but when i start to read & send file LineReceiver method doesnt run in the server side. Sending file code part is below, what can be cause of this? * i am reading 100 kb parts of file and send it, is it best choice? or reading file once and send it better? KB = 1024 * 100 while True: if (KB > file_size) | (KB == file_size): data = self.file.read(file_size) self.send_data(data) self.sended += file_size self.file.close() break else: data = self.file.read(KB) file_size -= KB self.send_data(data) self.sended += KB def send_data(self, data): self.transport.write(data + '\r\n') -- Aydın ŞEN Ege Üniversitesi Uluslararası Bilgisayar Enstitüsü

3 5

[Twisted-Python] Blocking reactor.connectSSL()
by asset Nov. 25, 2008

Nov. 25, 2008

Is there a workaround for reactor.connectSSL() blocking during the SSL handshake? I dug around and found some indications that it could be a problem fixed in Python 2.6 and 3.0, but also that its something that is worked around in Twisted, but I havent been able to find any reference to the issue with Twisted specifically. TKiel

2 1

Re: [Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones Nov. 24, 2008

Nov. 24, 2008

>>>>> "glyph" == glyph <glyph(a)divmod.com> writes: glyph> On 22 Nov, 05:50 pm, terry(a)jon.es wrote: >> def altInlineCallbacks(f): >> def unwindGenerator(*args, **kwargs): >> try: >> result = f(*args, **kwargs) >> except Exception, e: >> # f was not a generator. >> return failure.Failure() glyph> I hope you mean "defer.fail()". Yes, sorry. glyph> As with the other case we mistakenly diagnosed here, it doesn't glyph> actually raise. It returns a failed Deferred. Consider: Yes, that was my original case, passing a non-generator and getting an attribute error when _inlineCallbacks calls send. That one I knew goes back via the Deferred. glyph> The only thing that (potentially) needs to be done here is to glyph> produce a more useful error message. glyph> The other case, where inlineCallbacks decorates a function that glyph> itself raises an exception rather than returns an object, is the glyph> only way you won't get a Deferred. That's what I was addressing in the code above. If you call the function in unwindGenerator and you get an exception, you 1) know it's not a generator (that's what I didn't understand earlier - calling a function with a yield in it can never give you an exception, Python builds you a generator and gives you that) and 2) can immediately give the exception back via defer.fail (your correction above). That's a simple change and makes sure you always get a Deferred back. Terry

2 1

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com Nov. 23, 2008

Nov. 23, 2008

1 0

[Twisted-Python] TLS broken with twisted.words.protocols.jabber
by Jack Moffitt Nov. 23, 2008

Nov. 23, 2008

I would like to propose that #3463 (http://twistedmatrix.com/trac/ticket/3463) be additionally committed to the 8.1 branch and any other branches that still get point releases. It is a pretty critical workaround which fixes the fact that recent OpenSSL libraries cannot connect to Java based services. This means that most of the packaged versions of Twisted cannot talk to Google Talk or any Tigase or Openfire XMPP servers. Since packagers are often reluctant to upgrade very far in a particular distribution, I would like to get bug fix releases out for Twisted 8.1, etc, so that we have some hope that this problem will go away as soon as possible. jack.

5 9

Re: [Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones Nov. 23, 2008

Nov. 23, 2008

Hi Glyph >>>>> "glyph" == glyph <glyph(a)divmod.com> writes: >> inlineCallbacks appears to have a bug: 'raise' before 'yield' in a >> generator results in a synchronous exception rather than an errback, >> although its documentation does not explain this. glyph> Ugh, scratch that. No, it doesn't have this bug. Yes, you're right. glyph> I did some quick testing and saw some tracebacks, but apparently glyph> wasn't paying very close attention to them Nor was I, sorry! glyph> If func "raises before it gets to its first yield", we get the right glyph> behavior. If it just raises and doesn't yield *anywhere*, then it's glyph> not a generator and that's the same as your other case of glyph> accidentally-not- returning-a-generator. Still worth debugging, but glyph> not as serious. Agreed. I also agree with your earlier remarks about dropping the isinstance(result, GeneratorType). That leaves me with an alternative: def altInlineCallbacks(f): def unwindGenerator(*args, **kwargs): try: result = f(*args, **kwargs) except Exception, e: # f was not a generator. return failure.Failure() return _inlineCallbacks(None, result, Deferred()) return mergeFunctionMetadata(f, unwindGenerator) and still the problem that _inlineCallbacks raises if result doesn't have a send() method, etc. I'm also not sure of the best way to check for that, but don't think it should be in the _inlineCallbacks loop. --- BTW, there is a coding lesson here (at least for me): The reason I started to think there was a problem with Exceptions, tracebacks and _inlineCallbacks was from running code like this: @inlineCallbacks def f(): try: # something except Exception: # clean up raise When I ran it, the traceback of the Exception that propagated back to the attached errback would (sometimes) show the exception as having being triggered in _inlineCallbacks, like this: 2008/11/22 17:29 +0200 [-] File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 726, in _inlineCallbacks 2008/11/22 17:29 +0200 [-] result = g.send(result) The problem, as I guess will be obvious, is that my raise was delivering whatever sys.exc_info returned after the clean-up was done. In my case this was confusing as although the clean-up code had succeeded, it had called things that also made use of inlineCallbacks and the exception I was finally raising, and whose traceback I was printing, was a StopIteration raised inside *another* call of _inlineCallbacks! So it looked like _inlineCallbacks was somehow raising instead of sending a failure back to the errback... Re-raising the original exception made everything make sense. Thanks a lot for taking a look. Terry

2 1

[Twisted-Python] [JOB] Sr. Backend Developer, Eugene or Portland | 60-90k
by Beau Gould Nov. 22, 2008

Nov. 22, 2008

Responsibilities: * Work in a team building scalable web applications * Use test-driven development practices to ensure code correctness * Document code, track and fix bugs, respond rapidly to emergencies in production environment * Work with product group throughout development cycle to ensure quality of end product * Networking protocols such as TCP, UDP, HTTP, DNS, NAT, etc * Clustering and load balancing solutions for redundancy, high availability, scalability * Network programming (event-driven, asynchronous) * Experience analyzing large and complex systems for performance bottlenecks * Python, AJAX, PHP, C/C++ * LAMP * Scalable MySQL schema designs * Jabber/XMPP * Twisted Python * Spread Toolkit Requirements for this position: * Bachelors in Computer Science or related work experience. Masters degree is a plus * Ability to work on-site in Eugene, Oregon (preferred) or Portland, Oregon * 5+ years experience developing large scale software systems (10K+ concurrent users) To be considered, please submit your resume along with your salary requirements to bg @ capitalmarketsp.com Beau Gould Executive Advisor Capital Markets Placement www.cmp.jobs bg @ capitalmarketsp.com

1 0

[Twisted-Python] Libevent bindings and reactor
by Thomas Hervé Nov. 22, 2008

Nov. 22, 2008

Hi all! I've been working on a Libevent reactor for quite a long time now: http://twistedmatrix.com/trac/ticket/1930 (sheesh, 2 years). For different reasons I didn't manage (yet) to include something within Twisted itself, so I (re)started an external project to do it. For those who don't know libevent, it's an abstraction layer around select/poll/epoll/... to write portable applications. See http://www.monkey.org/~provos/libevent/. I thought it'll be easier for people who just want to use the libevent bindings to have an external project, and it's also a way to make people test it and get some feedback, without having to checkout a SVN branch. The project is now accessible on launchpad: https://launchpad.net/python-libevent (original project is recorded there: http://pypi.python.org/pypi/libevent-python). The latest version, 0.3, includes a libevent reactor, and a twisted plugins so that it's easily usable with trial and twistd (just pass -r libevent to both once installed). Note that these bindings only work with a recent enough version of Libevent itself (1.4). My PPA (https://launchpad.net/~therve/+archive) should help users of Hardy (although for some reasons the twisted plugin doesn't install correctly). Thanks for any comment/feedback. -- Thomas

1 0

[Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones Nov. 22, 2008

Nov. 22, 2008

Here's a suggestion for making inlineCallbacks more consistent and less confusing. Let's suppose you're writing something like this: @inlineCallbacks def func(): .... result = func() There are 2 things that could be better, IMO: 1. func may not yield. In that case, you either get an AttributeError when inlineCallbacks tries to send(). Or worse, the call to send might actually work, and do who knows what. I.e., func() could return an object with a send method but which is not a generator. For some fun, run some code that calls the following decorated function (see if you can figure out what will happen before you do): @defer.inlineCallbacks def f(): class yes(): def send(x, y): print 'yes' # accidentally_destroy_the_universe_too() return yes() 2. func might raise before it get to its first yield. In that case you'll get an exception thrown when the inlineCallbacks decorator tries to create the wrapper function: File "/usr/lib/python2.5/site-packages/twisted/internet/defer.py", line 813, in unwindGenerator return _inlineCallbacks(None, f(*args, **kwargs), Deferred()) There's a simple and consistent way to handle both of these. Just have inlineCallbacks do some initial work based on what it has been passed: def altInlineCallbacks(f): def unwindGenerator(*args, **kwargs): deferred = defer.Deferred() try: result = f(*args, **kwargs) except Exception, e: deferred.errback(e) return deferred if isinstance(result, types.GeneratorType): return defer._inlineCallbacks(None, result, deferred) deferred.callback(result) return deferred return mergeFunctionMetadata(f, unwindGenerator) This has the advantage that (barring e.g., a KeyboardInterrupt in the middle of things) you'll *always* get a deferred back when you call an inlineCallbacks decorated function. That deferred might have already called or erred back (corresponding to cases 1 and 2 above). I'm going to use this version of inlineCallbacks in my code. There's a case for it making it into Twisted itself: inlinecallbacks is already cryptic enough in its operation that anything we can do to make its operation more uniform and less surprising, the better. You might think that case 1 rarely comes up. But I've hit it a few times, usually when commenting out sections of code for testing. If you accidentally comment out the last yield in func, it no longer returns a generator and that causes a different error. And case 2 happens to me too. Having inlinecallbacks try/except the call to func is nicer because it means I don't have to be quite so defensive in coding. So instead of me having to write try: d = func() except Exception: # ??? and try to figure out what happened if an exception fired, I can just write d = func() and add errbacks as I please (they then have to figure out what happened). The (slight?) disadvantage to my suggestion is that with the above try/except fragment you can tell if the call to func() raised before ever yielding. You can detect that, if you need to, with my approach if you're not offended by looking at d.called immediately after calling func. The alternate approach also helps if you're a novice, or simply being lazy/careless/forgetful, and writing: d = func() d.addCallback(ok) d.addErrback(not_ok) thinking you have your ass covered, but you actually don't (due to case 2). There's some test code attached that illustrates all this. Terry import types from twisted.internet import defer, reactor from twisted.python.util import mergeFunctionMetadata def altInlineCallbacks(f): def unwindGenerator(*args, **kwargs): deferred = defer.Deferred() try: result = f(*args, **kwargs) except Exception, e: deferred.errback(e) return deferred if isinstance(result, types.GeneratorType): return defer._inlineCallbacks(None, result, deferred) deferred.callback(result) return deferred return mergeFunctionMetadata(f, unwindGenerator) @altInlineCallbacks def f0(): # This causes mayhem with the normal inlineCallbacks class yes(): def send(x, y): print 'yes' return yes() @altInlineCallbacks def f1(): return 'hey' @altInlineCallbacks def f2(): raise Exception('whoops!') @altInlineCallbacks def f3(): if True: raise Exception('ouch!') d = defer.Deferred() reactor.callLater(0, d.callback, None) yield d @altInlineCallbacks def f4(): d = defer.Deferred() reactor.callLater(0, d.callback, None) yield d def ok(result, name): print '%s ok: %s' % (name, result) def fail(failure, name): print '%s fail: %s' % (name, failure) def stop(_): reactor.stop() def main(): L = [] for func in f0, f1, f2, f3, f4: name = func.__name__ L.append(func().addCallbacks( ok, fail, callbackArgs=[name], errbackArgs=[name])) d = defer.DeferredList(L) d.addBoth(stop) if __name__ == '__main__': reactor.callLater(0, main) reactor.run()

3 3

Re: [Twisted-Python] A kinder and more consistent defer.inlineCallbacks
by Terry Jones Nov. 21, 2008

Nov. 21, 2008

Here's another way of looking at what I proposed. When you call a function, you can divide what happens into 3 possibilities: 1) An exception is raised 2) You get back a generator 3) You get back something else (Yes, I'm ignoring aborts, sys.exit, etc). As it stands, all three of these are handled inside _inlineCallbacks (not in inlineCallbacks). With the code I suggested, the call to f is made before _inlineCallbacks is called. Only in case 2, where we actually have a generator, do we ask _inlineCallbacks to deal with unwinding it. In case 1 we pass the exception back via a deferred errback with a failure and in case 2 we pass the result back in a deferred whose callback is already called. There's a good case (perhaps a stronger one) for just letting the Exception go on its merry way and raise back in the calling code. But it's hard to argue that knowing that we don't even have a generator result that passing it on to _inlineCallbacks in the knowledge that it's going to cryptically raise is better than just returning the result of the user's function, especially seeing as it has just successfully run to completion! Terry

1 0