[Twisted-Python] How to force synchronous behavior
Hello, I want to encapsulate deferreds into a function that looks like synchronous when invoked. Something like this: def mySyncFunc() x = 0 def done(data): global x x = data d = someCalculation() d.addCallback(done) <something here to hold until "done" is really done> return x print mySyncFunc() someCalculation() returns a deferred (for instance to accesses a DB). So, how can I write mySyncFunc() so that at the end I print the result of someCalculation() and not just 0? Thanks, -- Pedro
On Fri, 28 Oct 2005 17:40:21 -0400, Pedro Sanchez <psanchez@nortel.com> wrote:
I want to encapsulate deferreds into a function that looks like synchronous when invoked.
This is a bad idea. Don't do it. If you do somehow manage it, the results will be incredibly buggy and almost no-one will care to help you fix it, because people who know Twisted well don't do things this way. For more information, read this: http://www.livejournal.com/users/glyf/40037.html
On Sat, 29 Oct 2005 18:02:25 +0200, Marcin Kasperski <marcin.kasperski@softax.com.pl> wrote:
Because there seems to be so much confusion on this point, let me re-iterate Itamar's response: deferredResult is *broken*. It does not work now. It never worked in the past. If you use it, your program will suck and die and you will be sad and the only advise you will get to fix it will be to stop using deferredResult, which will probably mean rewriting a significant portion of your program, since you cannot write asynchronous programs synchronously and expect them to work. This is not a conditional point. It does not depend on anything. It is a simple, unavoidable fact. Disregard it at your own certain peril. Jean-Paul
On 29 Oct 2005, at 09:50, Jean-Paul Calderone wrote:
Why does twisted contain functions which are never to be used? If it can't be fixed, why not remove it? -- Scott Lamb <http://www.slamb.org/>
On Sat, 29 Oct 2005 10:41:45 -0700, Scott Lamb <slamb@slamb.org> wrote:
It will be removed shortly. deferredResult and deferredError have been deprecated since r14685 (Gee, I wish I knew what release that corresponded to - if I were to guess, I'd say 2.1), and will probably be removed in one of the next two releases. A similarly broken function, wait(), will be deprecated in the next release and removed sometime after that. Why were they added in the first place? It was mistakenly thought that they might be able to work. Jp
On Sat, 29 Oct 2005 14:58:13 -0400, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Sat, 29 Oct 2005 10:41:45 -0700, Scott Lamb <slamb@slamb.org> wrote:
Why does twisted contain functions which are never to be used? If it can't be fixed, why not remove it?
Why were they added in the first place? It was mistakenly thought that they might be able to work.
And why not remove it? The test cases still depend on this functionality (and are fragile and break non-deterministically in part because of it) and it is a significant effort to fix them all. This effort is underway.
On Sat, 29 Oct 2005 18:02:25 +0200, Marcin Kasperski <marcin.kasperski@softax.com.pl> wrote:
I want to encapsulate deferreds into a function that looks like synchronous when invoked.
This is a bad idea. Don't do it.
Depends on the purpose...
Nope. Unequivocally, a bad idea. The function you mentioned (whose name I will not repeat, even to quote) is SEVERELY deprecated. Don't use it. The poster sounds new to Twisted, which means that they have at least 2 years worth of regular asynchronous programming ahead of them before they can make effective and judicious use of shortcuts like deferredGenerator.
On Saturday 29 October 2005 20:59, glyph@divmod.com wrote:
Out of interest, what's your opinion on what we're doing with communicating generators in the project beginning with a "K" (*) ? As far as I'm concerned we're pretty much doing the same sorts of things as twisted, just written very differently. However, there are similarities in the way we write things to some of the ideas raised in this thread, and if you think we're doing something dumb (as opposed to ideas that could be ripped out at somepoint *IF* they prove useful), I'd be interested to hear (one of these days I'll learn to speak English properly as well). (*) Some people think "K" is competition, rather than trying out an alternate approach so I'll not mention it by the full name - a misspelt shrubbery if the allusion doesn't make sense. :) I suppose one risk point of our approach is that by trying to make it easier to work with we risk people thinking concurrency is naturally is easy (which is potentially quite a dangerous meme, since software concurrency isn't naturally easy). BTW, FWIW, I read the blog post you pointed at earlier in the thread and agree with it). I'll say one thing though, hopefully the original poster realises how much they have to do to make their code safe when they're faking synchronous behaviour and faking a single process look. If they don't then as everyone has said, they're setting themselves up for a world of pain. Best Regards, Michael.
On Sat, 29 Oct 2005 22:33:30 +0100, Michael <ms@cerenity.org> wrote:
It's not so much that "K" is doing something dumb (for reference: we are not talking about the successor to J, which is itself a successor to APL). With a project like "K", there would be as much focus on explaining correct usage of generators as there is in Twisted focusing on correct usage of Deferreds. It would be hard to write a program using "K" and not understand the ramifications of what you're doing.
I looked at "K". Reading code in it was powerfully weird - and this is coming from the author of Twisted ;-). It was enough like Erlang that it would only make things easy for experts, so I don't think that it would lead to this problem - on the other hand, I haven't seen how large communities react to it.
I have a lot of experience dealing with people that ask questions in the form that the poster did. As a rule they don't have any idea how hard solving concurrency problems is, and have been lead to believe (by Java or whatever) that concurrency is easy, just a matter of running 2 or more "simple" synchronous (-looking) programs at a time. While I don't know the OP personally, I am willing to risk their taking offense, so that I can continue to drive home the point to hordes of fledgeling developers out there that seem to desperately want to believe that massively concurrent programming is easy.
On Sunday 30 October 2005 06:31, glyph@divmod.com wrote:
It does seem to be the case, yes.
I looked at "K". Reading code in it was powerfully weird - and this is coming from the author of Twisted ;-).
Great :-) Can I quote you on that? :-D Seriously though, I did kinda expect that. I suspect the two different ways of writing code will be easier to different sorts of people. Personally I find the approach natural and normal. You can take any implication you like from that ;-)
Interesting viewpoint. KInda at odds with what we've seen so far, but then we haven't had a large community react to it yet, there's also the caveat above of it might be a mindset thing rather than skill level this, and mainly at the moment I don't think it's ready for a large community yet* - if one's appropriate (We're still in the process of finding the best way of writing systems using it). * We're not in a rush FWIW, it's more inspired by things which /aren't/ Erlang. I've not written anything in Erlang :) If it turns out though that you're right - only makes things easy for experts - then the project's failed (which won't be a problem, that's the point of research to try things and see what works/doesn't, just means I have to try again). As a result I hope you're wrong (for once :). Thanks for the feedback! Best Regards, Michael.
# #>> I want to encapsulate deferreds into a function that looks like #>> synchronous when invoked. #> #> I use a similar mechanism in my database wrapper module. here is an example of it in use: http://www.darkarts.co.za/repository/tada/doc/examples/taoBasicUsage.py (online syntax highlighted version of the same: http://www.rafb.net/paste/results/TmpcYr74.html) The code is part of my tada project at : http://www.darkarts.co.za/projects/tada/ I agree, callbacks are ugly, python 2.5 will be great :D -- Clive Crous http://www.darkarts.co.za/
Clive Crous wrote:
I agree, callbacks are ugly, python 2.5 will be great :D
What will make Python 2.5 better? I'm looking over PEPs now. PEP 342 looks amazing. I don't see a "Python-Version: 2.5" there, though. Is it expected to make it into 2.5? When might it actually get used in Twisted, given its policy of supporting old Python version? (I saw the recent debate on dropping Python 2.2 support.)
# Clive Crous wrote: #> I agree, callbacks are ugly, python 2.5 will be great :D #> # What will make Python 2.5 better? I'm looking over PEPs now. PEP 342 # looks amazing. I don't see a "Python-Version: 2.5" there, though. Is it # expected to make it into 2.5? When might it actually get used in # Twisted, given its policy of supporting old Python version? (I saw the # recent debate on dropping Python 2.2 support.) My appologies, i did in fact mean pep342, it somehow got into my head as '2.5 fact'. Whether it makes it into twisted or not is irrelivant, i will use it nonetheless ;) -- Clive Crous http://www.darkarts.co.za/
On 10/30/05, Clive Crous <clive@darkarts.co.za> wrote:
Well, it _is_ currently in Python's trunk, so unless it's reverted before 2.5 is released, it'll probably be in 2.5. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | w----v----w-+
On Sat, 29 Oct 2005 22:12:26 +0200 (SAST), Clive Crous <clive@darkarts.co.za> wrote:
the OP wanted a function that looks synchronous when *invoked* - the example you gave doesn't look synchronous to the caller at all - the functions decorated with "genAside" return deferreds. it's easy to pretend that they don't, if you decorate all of your functions similarly, but at some point some plain, upstream function is going to have to deal directly with a deferred (this is not the case with the example because it is a script, and nothing actually depends on the result of doit()). also, if any operation inside the doit() function raises an exception, the reactor will not stop. something like this would probably be better: doit().addErrback(log.err).addCallback(lambda ignore: reactor.stop()) i'd like to reiterate glyph's advice that suggesting generator based short-cuts to twisted beginners is not a good idea. it's *very* easy to use these things without understanding what is actually going on under the hood, and massive confusion frequently arises as soon as something crops up that doesn't fit into that programming style. -- Moe Aboulkheir
# On Sat, 29 Oct 2005 22:12:26 +0200 (SAST), Clive Crous # <clive@darkarts.co.za> wrote: #> #># #>#>> I want to encapsulate deferreds into a function that looks like #>#>> synchronous when invoked. #>#> #>#> #> #> #>I use a similar mechanism in my database wrapper module. #> #>here is an example of it in use: #>http://www.darkarts.co.za/repository/tada/doc/examples/taoBasicUsage.py # # the OP wanted a function that looks synchronous when *invoked* - the # example you gave doesn't look synchronous to the caller at all - the The origional question was: """ So, how can I write mySyncFunc() so that at the end I print the result of someCalculation() and not just 0? """ My example and proposed solution does that. And as such is usable by him. # functions decorated with "genAside" return deferreds. it's easy to # pretend that they don't, if you decorate all of your functions similarly, # but at some point some plain, upstream function is going to have to deal # directly with a deferred (this is not the case with the example because it # is a script, and nothing actually depends on the result of doit()). also, # if any operation inside the doit() function raises an exception, the # reactor will not stop. something like this would probably be better: # # doit().addErrback(log.err).addCallback(lambda ignore: reactor.stop()) # it is an example usage, not a real world implementation, of course if something fails it will die. # i'd like to reiterate glyph's advice that suggesting generator based # short-cuts to twisted beginners is not a good idea. it's *very* easy to # use these things without understanding what is actually going on under the # hood, and massive confusion frequently arises as soon as something crops # up that doesn't fit into that programming style. # Why do i hit this wall constantly when discussing twisted usage with twisted users or developers mostly on freenode's #twisted admitedly: Telling someone NOT to do something is not answering a question it is avoiding it. I pasted the url for the full source so that he could "look under the hood" if he feels uncomfortable with it's mechanism he needent use it. By your reasoning surely he should go through the entire twisted codebase aswell and know exactly what a deffered is doing "under the hood" and how twisted does it's asyncronousity before using it ? -- Clive Crous http://www.darkarts.co.za/
On Sun, 30 Oct 2005 00:11:07 +0200 (SAST), Clive Crous <clive@darkarts.co.za> wrote:
no, it really doesn't. here is the code snippet given by the original poster: def mySyncFunc() x = 0 def done(data): global x x = data d = someCalculation() d.addCallback(done) <something here to hold until "done" is really done> return x the desired functionality is "at the end i print the result of someCalculation() and not just 0". this could obviously be done by changing the first line of done() to "print data". but the "return x" betrays the intention that the function would ideally return the value of "x" (returning the _value_ of "x" is worlds away from returning a deferred that fires with the value of "x"). no defgen type solution is going to make this possible
the point here is that this particular question does not have a satisfying answer. it has been beaten to death recently that you cannot magically expose a synchronous interface to asynchronous code. so, in summary, the question was answered, by several people in this thread, and the answer is "you cannot do that and expect your code to work reliably". is that not a valid answer?
ideally yes. somebody unfamiliar with deferreds is going to be in no position to determine whether or not they are comfortable with a generator based mechanism for handling them. people already associate unfair amounts of magic with twisted, and i dont think suggesting such abstract interfaces to beginners is an effective way to mitigate this. -- Moe Aboulkheir
On Sun, 30 Oct 2005 00:11:07 +0200 (SAST), Clive Crous <clive@darkarts.co.za> wrote:
You are correct. I do not disagree with this fact, and it does not change my opinion of the behavior involved. Here is the ugly truth of the non-answers to such questions: I do not care about most of the posters to this list. I am happy that they use Twisted, and even more happy if they are pleased by it, but ultimately I am really just trying to improve the world I live in, and that means the average quality of available Python code. That means I am only going to answer questions which I believe will contribute positively to the state of affairs as it regards code quality in general, and Python code quality specifically. These people who people whose questions are answered (or not answered) on this list or on IRC will go and write applications or libraries based on the answers they receive. I may be stuck maintaining or using that code at some point. Now, maybe if I know, but don't tell them how to do some ridiculous thing they're trying to do, they'll miss their deadline at work or they won't turn on their homework assignment on time and I could have prevented that. They will be unhappy. Making them happy isn't my purpose here, so I don't care. Maybe they'll abandon their Python project and do it in Ruby instead because the Python community is so hostile. Many people think that this loss of contributing labor is a great tragedy for an open-source community, but IMHO it is really the best possible consequence. All the people who don't understand programming will end up in the more-polite Rails or PHP or Django or TurboGears communities and generate huge piles of code that don't work. Then all the guys who know what they're doing will stick around here and work on Twisted stuff. In the sense that we compete with those projects, this is great! It's like sabotage-by-proxy. When those projects are a smouldering ruin of inconsistent style and half-baked, buggy, insecure code, people who want a functional product will come ask those of us with Twisted expertise. A more realistic consequence, however, is that programmers new to Twisted will adopt a more consistent style, and avoid fighting the framework, and concentrate on solving their actual problems. Some will still go away, yes, but generally there is some reason they came to Twisted in the first place and it remains valid even if we won't make it easy to do things we collectively regard as bad practice. I have adopted this stance not merely because I am abraisive asshole, but based on long experience with IRC and with teaching programmers how to use things. There are several projects that were developed early on with Twisted that were utter disasters because I politely and pateiently answered all the authors' questions about how to make Deferreds appear to block, how to call reactor functions from threads, and how to invoke Twisted from C code, rather than stopping and saying "hey, what are you *really* trying to do?". (No, I will not name these projects. I do have *some* manners.) I can only assume that the other Twisted devs you've had problems with have gotten this habit from similar experiences, but they may have their own reasons.
It's not so much whether the OP is comfortable with your explanation or not. If I were confident that all programmers would go and investigate the solution first, I would not care to make a post such as this. It's whether, upon not understanding, the OP will encourage others to program using bad style with a tool that he is comfortable with but but does not *understand*. It is quite possible that the OP understands your solution perfectly well, in which case this conversation is useless to him and he can ignore it, but still socially useful in that it will appear in google searches when other, less experienced programmers have the bright idea that they can make Deferreds magically go away somehow. (Apologies to the OP: I really have no idea about you at all. I am really speaking of a hypothetical average programmer.)
#>Why do i hit this wall constantly when discussing twisted usage with #>twisted users or developers mostly on freenode's #twisted admitedly: #>Telling someone NOT to do something is not answering a question it is #>avoiding it. # # You are correct. I do not disagree with this fact, and it does not change # my opinion of the behavior involved. Here is the ugly truth of the # non-answers to such questions: # # I do not care about most of the posters to this list. I am happy that # they use Twisted, and even more happy if they are pleased by it, but # ultimately I am really just trying to improve the world I live in, and # that means the average quality of available Python code. That means I am # only going to answer questions which I believe will contribute positively # to the state of affairs as it regards code quality in general, and Python # code quality specifically. # # These people who people whose questions are answered (or not answered) on # this list or on IRC will go and write applications or libraries based on # the answers they receive. I may be stuck maintaining or using that code # at some point. # # Now, maybe if I know, but don't tell them how to do some ridiculous thing # they're trying to do, they'll miss their deadline at work or they won't # turn on their homework assignment on time and I could have prevented that. # They will be unhappy. Making them happy isn't my purpose here, so I # don't care. Maybe they'll abandon their Python project and do it in Ruby # instead because the Python community is so hostile. # # Many people think that this loss of contributing labor is a great tragedy # for an open-source community, but IMHO it is really the best possible # consequence. All the people who don't understand programming will end up # in the more-polite Rails or PHP or Django or TurboGears communities and # generate huge piles of code that don't work. Then all the guys who know # what they're doing will stick around here and work on Twisted stuff. In # the sense that we compete with those projects, this is great! It's like # sabotage-by-proxy. When those projects are a smouldering ruin of # inconsistent style and half-baked, buggy, insecure code, people who want a # functional product will come ask those of us with Twisted expertise. # # A more realistic consequence, however, is that programmers new to Twisted # will adopt a more consistent style, and avoid fighting the framework, and # concentrate on solving their actual problems. Some will still go away, # yes, but generally there is some reason they came to Twisted in the first # place and it remains valid even if we won't make it easy to do things we # collectively regard as bad practice. # # I have adopted this stance not merely because I am abraisive asshole, but # based on long experience with IRC and with teaching programmers how to use # things. There are several projects that were developed early on with # Twisted that were utter disasters because I politely and pateiently # answered all the authors' questions about how to make Deferreds appear to # block, how to call reactor functions from threads, and how to invoke # Twisted from C code, rather than stopping and saying "hey, what are you # *really* trying to do?". (No, I will not name these projects. I do have # *some* manners.) # # I can only assume that the other Twisted devs you've had problems with # have gotten this habit from similar experiences, but they may have their # own reasons. # I find it quite amusing, yet somewhat disturbing how you justify this behaviour to yourselves. I have worked on many projects in my life and each of them was unique in it's specific requirements. Let me give you two of my current projects as brief examples: The first, is for a client who would like a downloadable executable available on his website. The primary need, beside the obvious need that it in fact work, is that the executable be as tiny as possible to encourage potential clients in turn to download and use it. The second project is a set of modules for internal use by the company I now work for. The primary focus, again besides the obvious need for it to be functional, is that it be highly maintainable and easily modified by the senior most programmer or the newest, week-old junior developer. Anybody must be able to look at the code, understand it quickly and easily, and be able to modify parts as required. If I employed the same style of programming in both of these projects I would be a fool. Some of the styles implemented to make a small executable in any other circumstance would be considered incredibly bad practice, yet there it is, working perfectly and being incredibly small while doing so. Meanwhile some of the code within the second example is incredibly verbose and a seasoned coder would look at it and say "Why for the love of python are you doing this the long way round?" ... once again, it fulfills it's specification perfectly while doing it's job to the satisfaction of all concerned. When it comes to implementations of twisted, one must take into account the context in which the program is being developed and the specific use cases it requires. No two implementations will ever be the same, and no two implementations should ever use twisted in exactly the same way. To expect identical or even similar usage of a module in every single instance is poor practice in itself. If I come online, and ask: "How do i use A with B" and the response is "Dont!", as in this thread, it shows a high level of arrogance, extreme presumption and an incredible 'naivety to programming' by the person giving that "answer". There always will be people who have different needs to you. Enforcement of usage policy on users is the realm of restrictive corporate use policy and software distribution licensing as one expects from Microsoft et al. I have worked with programmers who hold the same mind-set as is being shown here. Some of the "programmers" with whom I have worked in the past, were insistent that things should be done in a particularly restrictive, stylized manner and that no-other-way-is-the-right-way. There is *NO* absolute "right way" when it comes to coding. There are good ways and bad ways however all are dependent on the context of the project at hand and it's particular specifications. The most common result of such narrow-minded programming mind-sets is the inability to complete the task at hand. Programs will be written, rewritten, re-factored, tossed out and restarted. This cycle will never end as long as this zenith of coding "nirvana" remains the goal. The tragic result of this is that projects repeatedly fail to reach a usable state, to the detriment of all. I hope twisted development does not go down this path and shows some maturity and the ability to sustain it's own usage. I really do love the twisted framework and would hate to see it dragged into a quagmire of self-indulgent disarray to the despair of a dedicated and loving user base (whom you freely admit not to care about at all). -- Clive Crous http://www.darkarts.co.za/
On Mon, 31 Oct 2005 00:02:47 +0200 (SAST), Clive Crous <clive@darkarts.co.za> wrote:
You're calling me arrogant and naive, and a fascist champion of policy enforcement: yet you have the gall to tell me what *I* must do with *my* freely donated time. I didn't say "only stupid people do this" when you answered the question, or delete your message from the archives, I merely offered my opinion that it is a bad thing to do. You are replying to my explanation of why we say things like that, not why *you* must never do so. However, now that we're name-calling, let me suggest instead that *you* are arrogant to presume that you know more about the questions people ask about Twisted than I do (how long do you think I've been answering these, anyway?), and that you are incredibly naive about education in general. When you have seen the same question asked a thousand times, you learn to see the question behind the question. You understand the unseen consequences of answering it straightforwardly. What can be a perfectly legitimate question from an intermediate or expert programmer is simply a set-up for disaster from a novice. For example, if a 12-year-old girl asks you "can I get pregnant from sex if I only do it once", this is most likely not idle curiosity speaking. You should not answer as accurately as possible, with equivocations about the probability. You say "YES, YOU ABSOLUTELY WILL, WHERE IS YOUR MOTHER". If a married couple in a fertility clinic asks you a similar question, about how difficult it is on average to conceive, it would be a good idea to just answer. The context of this mailing list is less obvious than either of those, but I have found over time that if the poster really knows what they're doing, they will respond to my refusal with an explanation and I can offer more detail at that point - quite often they are asking a very different question than "how can I make this look synchronous", it is something like "I need to integrate with a proprietary library that will call my code synchronously and I have no access to its source code" or "I would like to use a Twisted protocol implementation from within an existing Zope2 site". Neither of those questions are properly answered by "use deferredResult" or "use defgen". It's complicated.
This is exactly what happens with Twisted - you got everything right except the result, which is that *code quality goes up over time*. It is this dedication to quality that makes the project fun and interesting to use. Granted, breaking backwards compatibility is never fun, but we try hard to minimize the impact of that. For a project which has a strict deadline and budgetary constraints, this is obviously a terrible tragedy. I have seen it, I have even caused it (who hasn't?), and it's ugly. However, this is exactly what makes open source so much fun; you don't have to get it right the first time, you don't have to get it right in 6 months, you can just keep trying and trying and trying. Eventually you get it right and everyone thinks it's awesome. I am sure that no current users remember the massive upheaval and 1 1/2 years it took to get the transport/protocol split properly implemented. If we had not "rewritten, re-factored, tossed out and restarted" that code, Twisted would be nothing but a much set of libraries for Medusa today.
Luckily I am also a user and my career, my reputation and my income all depend quite explicitly upon its success. Many Twisted developers are in the same position now, so selfish motivations are more than adequate. Seriously though, my point about not caring about the community was slightly tongue-in-cheek. The point there is that I am not here to do your homework assignment, I am not here to do your job for you. If you think that's what I *am* here for, then no, I don't care about you. I am here to help you learn to use Twisted and learn to use it correctly. If you don't like my advice, the code is all free, there are other people to answer your questions, and there is quite a bit of freely-available documentation (and even a book from O'Reilly now!).
On Mon, 31 Oct 2005 00:02:47 +0200 (SAST), Clive Crous <clive@darkarts.co.za> wrote:
I find it equally arrogant to expect the community to behave in the way you would prefer. What is owed to you that justifies this, exactly?
First of all, no one is forcing anything on you. You're not locked in. Don't use Twisted if you don't want to. Don't use Twisted in the manner we are explaining is correct if you don't want to. Fork Twisted and take development in a better direction. Rip parts of Twisted out and drop them into your own project to use how you please. Spit on our works. Curse our names and our progeny to the seventh generation, if that's your thing. Who's going to stop you? Not me, and not anybody else on this list. I recognize that you probably don't want to do any of these things. You want to keep using Twisted, and you want the Twisted team to do things your way. I have this same expectation of other projects. Let's face it: it's an unreasonable expectation. Twisted is developed for you by a team of uncompensated developers working in their spare time. Likewise, support is being offered for free. It should not be too surprising that it is also being offered *conditionally*. A lot of us are happy to help with problems that don't directly benefit us, but a lot of us are not happy to help with solutions that we think are just as problematic as that which they are intended to resolve. Are you going to fault us for limiting the extent of our unpaid donations (frequently to commercial development)? Please. Furthermore, a perfectly legitimate response to "Don't!" is "But I have constraints X, Y, and Z." You may feel it is unreasonable to explain your situation, but that's pretty much too bad for you. If you want to be able to dictate people's behavior to this extent, negotiate a contract with them. Now, we may not be convinced by constraints X, Y, and Z, but in most cases where they are rational and correct (ie, not "I have to use threads because Cthulhu will rise and consume the world if I do not" - although if you are working on the project to prevent this, and are using Twisted, awesome! Can we put you on the successes page?), we will be. It is the refusal to explain them at all that is most annoying, and most likely to result in no assistance being rendered. I don't think this attitude is unique to the Twisted community. I don't even think it is unique to open source communities. Nor do I think it is a negative attitude. I think it is more open minded and generally productive than just presenting the precise answer to each exact question posed. In many cases, I have concrete evidence that not answering a direct question with a superficially correct response has helped the questioner measurable, and I've been thanked for this approach repeatedly.
This is a mischaracterization of the attitude I use when trying to help people, and from what I've witnessed of the interactions of other members of the Twisted community, I think it is not representitive of their behavior either. The interaction that gives rise to this impression, in my experience, most typically involves much more adament insistence and inflexibility on the part of the person seeking help than on the part of the person from which help is sought: the questioner *knows* they are already on the right track, and if this jerk he is talking to would just stop asking questions and answer them for a change, the problem would immediately be solved.
Which is why I stress the importance of discussing the context of a problem before trying to resolve it.
Twisted comes with a news server, a mail server, a chat server, a chat client, a remote REPL with multiple front ends, a plugin system, numerous persistence systems, a documentation framework with multiple input and output formats, an "Application" abstraction, an SSH client, an SSH server, an authentication and authorization system, a mature TCP, UDP, and SSL client and server API with numerous implementations for numerous platforms, an asynchronous DB-API 2.0 wrapper, an inetd-alike, a high-level remote method and object protocol, a unit test framework and rich command line tool for interacting with test suites, a web server, a web client, a DNS server, a DNS client, and FTP server, an FTP client, implementations of the echo, discard, chargen, ident, finger, QOTD, who, daytime, time, telnet, sip, toc, and oscar protocols, a filesystem path abstraction, a threadpool implementation, a much higher-quality module reloading system than the builtin reload(), a logging system, and quite a few other mature, stable features. Arguing that the Twisted team cannot complete tasks is ridiculous.
Twisted has been driven by hate for a long time. Hate and shared goal of the team to destroy the sun. There's no change in direction going on, at least with regard to the fundamental drive directing Twisted's development. Jean-Paul
I am the original poster of this thread. First of all, thank you for the different responses. However I still don't have a clear answer (other than what I'm thinking of doing is wrong, which is OK, it can be fixed). I'm working on a Twisted/Nevow application, and yes, I'm new to both and I can tell that if learning Twisted has a step curve, doing it in conjuction with Nevow is really almost vertical. But, nevertheless, my application is working well these days. The motivation for my question is the following: I added the guard/credentials stuff to my application. The issue is that in order to accept a login my application requires several things to happen before access can be granted. I therefore implemented my own AuthChecker() which fires several deferreds to acomplish its task. This is working well, including the use of deferToThread() in several places. But, the behavior of the authentication wrapper is synchronous. That is, access to the system cannot be granted until all deferred operations have happended. In this wrapper I have something like: def requestAvatar(self, avatarId, mind, *interfaces): for iface in interfaces: if iface is inevow.IResource: if avatarId is checkers.ANONYMOUS: resc = login.LoginPage() resc.realm = self return (inevow.IResource, resc, noLogout) else: resc = home.RootPage() resc.realm = self return (inevow.IResource, resc, self.createLogout(avatarId, mind)) else: raise NotImplementedError("Not supported.") So let me rephrase my original question. Given that the real user authentication is happening in my AuthChecker() implemented somewhere else, with deferreds and what not, how can I force this requestAvatar() thing to wait until the whole login process has finished? My idea was to force a synchronous behavior on AuthChecker() forcing this way the login process to wait for it to finish. While writting this I realize that I may be missunderstanding how the credentials stuff works ... who knows. Any suggestions are appreciated, Thank you, -- Pedro
On Mon, 31 Oct 2005 08:28:39 -0500, Pedro Sanchez <psanchez@nortel.com> wrote:
Cred supports Deferreds, so there's no need to make your requestAvatar or requestAvatarId methods synchronous. They can simply return Deferreds. If you have several Deferreds going in requestAvatarId, you may want to use a DeferredList, which behaves very similarly to a Deferred, but takes several other Deferreds as input and only fires when all those Deferreds have fired. requestAvatar won't be called until the Deferred you return from requestAvatarId fires. Jean-Paul
Stefan Behnel wrote:
I tried this already and couldn't make it work. In one of the functions called during the validation process I have this def registerSession(self, session): def _customerInit(self, customer): ... session.customerInit = True # initialize customer data d = defer.Deferred() d1 = defer.maybeDeferred(customerInit, session) d1.addCallback(_customerInit) d1.addErrback(errhandler, session) d.chainDeferred(d1) return d customerInit() fires several deferreds which I collect with a deferredList already. This works well. My goal therefore is to force "registerSession()" to behave synchronously[1] so that whoever is invoking it will block. I thought that by defining my own deferred d, and that by chaining it to d1 I would achive this. But I don't, the caller still doesn't block and goes along thinking that registerSsession() is done. I guess I don't understand how to use the chainDeferred() call. Hints are welcome; regardeless of what I end up doing, I'd like to understand who deferred chaining works. In a previous answer from Jean-Paul Calderone, he mentions that the requestAvatar and requestAvatarId methods don't have to be synchronous (thanks Jean-Paul!). This is no evident from the documentation, in fact, I dare to say that in all examples I've seen so far these calls are always synchronous, therefore my wrong assumption that they had to be. This is likely the right way to go to modify my program. I'll look into it. Thanks again! -- Pedro [1] Boy, I say this with hesitation after the long thread I spawned with my post, and after reading repeately that I shouldn't even attempt to do it. Which is fine, I take it, thanks for the imput. But I'm just trying to figure out the best way around it for my application.
Pedro Sanchez schrieb:
Ah, there we go. This simplifies to def registerSession(self, session): def _customerInit(self, customer): ... session.customerInit = True # initialize customer data d1 = defer.maybeDeferred(customerInit, session) d1.addCallback(_customerInit) d1.addErrback(errhandler, session) return d1 And then everyone who calls registerSession() receives a Deferred and can add callbacks for whatever else needs to be done after the session is successfully registered and _customerInit has been run. You get the difference? The registration is not done when registerSession /returns/, but when the Deferred that it returns /fires/. So keep adding callbacks to that Deferred instead of expecting things to have terminated. Twisted will then take care of running everything in a chain. Stefan
Stefan Behnel wrote:
Yes, I understand that very well. The problem is (was, I guess) my wrong assumption that the requestAvatar function (which at the end is the "caller" of registerSession() in my actual code) was supposed to be synchronous and not to deal with deferreds at all. Thanks, -- Pedro
Given that there has already been a long thread about how one should not do this, which I completely agree with, I'll not repeat it. Note that I do not recommend this solution to the OP, but rather post this just to appease to the people complaining about how the only answer given is "don't do that". "Don't do that" is the correct answer given the information so far available, but, here's a different non-answer, with code. On Oct 28, 2005, at 5:40 PM, Pedro Sanchez wrote:
The above code cannot work because for "done" to become done, the twisted event loop must get a chance to run. For the event loop to get a chance to run, you must return from that function. Basically, how can you "block" waiting for an async event, while letting the twisted event loop continue running? There is an answer: run your code in a separate thread. However, that doesn't work either, because the OP wants to use twisted APIs like ADBAPI or networking in someCalculation. So there really is no way to do literally what's asked in a working way. But, you can get something "like" the above by splitting your code into synch-like-code to be run in a separate thread, and async-twisted-using-code to be run in the twisted reactor thread. Here's a little example. Please note, again, that I do not recommend doing this except in circumstances where you absolutely must. import Queue from twisted.internet import reactor, defer from twisted.python import failure def callInReactor(__f, *__a, **__kw): # Called in other thread queue = Queue.Queue() reactor.callFromThread(_calledFromThread, queue, __f, __a, __kw) result = queue.get() if isinstance(result, failure.Failure): result.raiseException() return result def _calledFromThread(queue, f, a, kw): # Called in reactor thread result = defer.maybeDeferred(f, *a, **kw) result.addBoth(queue.put) def someCalculation(): # A demo "calculation" d = defer.Deferred() reactor.callLater(4, d.callback, 'hi') return d x = 0 def myAsyncFunctionInTwistedThread(): def done(data): global x x = data d = someCalculation() d.addCallback(done) return d def mySyncFuncInAnotherThread(): # NOTE: in this other thread you cannot call any twisted APIs besides reactor.callFromThread # and a very few select others. callInReactor(myAsyncFunctionInTwistedThread) # callInReactor blocks until the deferred returned by myAsyncFunctionInTwistedThread fires print x # Start up a thread to call your blocking function in reactor.callInThread(mySyncFuncInAnotherThread) # Run the reactor reactor.run() James
On Fri, 28 Oct 2005 17:40:21 -0400, Pedro Sanchez <psanchez@nortel.com> wrote:
I want to encapsulate deferreds into a function that looks like synchronous when invoked.
This is a bad idea. Don't do it. If you do somehow manage it, the results will be incredibly buggy and almost no-one will care to help you fix it, because people who know Twisted well don't do things this way. For more information, read this: http://www.livejournal.com/users/glyf/40037.html
On Sat, 29 Oct 2005 18:02:25 +0200, Marcin Kasperski <marcin.kasperski@softax.com.pl> wrote:
Because there seems to be so much confusion on this point, let me re-iterate Itamar's response: deferredResult is *broken*. It does not work now. It never worked in the past. If you use it, your program will suck and die and you will be sad and the only advise you will get to fix it will be to stop using deferredResult, which will probably mean rewriting a significant portion of your program, since you cannot write asynchronous programs synchronously and expect them to work. This is not a conditional point. It does not depend on anything. It is a simple, unavoidable fact. Disregard it at your own certain peril. Jean-Paul
On 29 Oct 2005, at 09:50, Jean-Paul Calderone wrote:
Why does twisted contain functions which are never to be used? If it can't be fixed, why not remove it? -- Scott Lamb <http://www.slamb.org/>
On Sat, 29 Oct 2005 10:41:45 -0700, Scott Lamb <slamb@slamb.org> wrote:
It will be removed shortly. deferredResult and deferredError have been deprecated since r14685 (Gee, I wish I knew what release that corresponded to - if I were to guess, I'd say 2.1), and will probably be removed in one of the next two releases. A similarly broken function, wait(), will be deprecated in the next release and removed sometime after that. Why were they added in the first place? It was mistakenly thought that they might be able to work. Jp
On Sat, 29 Oct 2005 14:58:13 -0400, Jean-Paul Calderone <exarkun@divmod.com> wrote:
On Sat, 29 Oct 2005 10:41:45 -0700, Scott Lamb <slamb@slamb.org> wrote:
Why does twisted contain functions which are never to be used? If it can't be fixed, why not remove it?
Why were they added in the first place? It was mistakenly thought that they might be able to work.
And why not remove it? The test cases still depend on this functionality (and are fragile and break non-deterministically in part because of it) and it is a significant effort to fix them all. This effort is underway.
On Sat, 29 Oct 2005 18:02:25 +0200, Marcin Kasperski <marcin.kasperski@softax.com.pl> wrote:
I want to encapsulate deferreds into a function that looks like synchronous when invoked.
This is a bad idea. Don't do it.
Depends on the purpose...
Nope. Unequivocally, a bad idea. The function you mentioned (whose name I will not repeat, even to quote) is SEVERELY deprecated. Don't use it. The poster sounds new to Twisted, which means that they have at least 2 years worth of regular asynchronous programming ahead of them before they can make effective and judicious use of shortcuts like deferredGenerator.
On Saturday 29 October 2005 20:59, glyph@divmod.com wrote:
Out of interest, what's your opinion on what we're doing with communicating generators in the project beginning with a "K" (*) ? As far as I'm concerned we're pretty much doing the same sorts of things as twisted, just written very differently. However, there are similarities in the way we write things to some of the ideas raised in this thread, and if you think we're doing something dumb (as opposed to ideas that could be ripped out at somepoint *IF* they prove useful), I'd be interested to hear (one of these days I'll learn to speak English properly as well). (*) Some people think "K" is competition, rather than trying out an alternate approach so I'll not mention it by the full name - a misspelt shrubbery if the allusion doesn't make sense. :) I suppose one risk point of our approach is that by trying to make it easier to work with we risk people thinking concurrency is naturally is easy (which is potentially quite a dangerous meme, since software concurrency isn't naturally easy). BTW, FWIW, I read the blog post you pointed at earlier in the thread and agree with it). I'll say one thing though, hopefully the original poster realises how much they have to do to make their code safe when they're faking synchronous behaviour and faking a single process look. If they don't then as everyone has said, they're setting themselves up for a world of pain. Best Regards, Michael.
On Sat, 29 Oct 2005 22:33:30 +0100, Michael <ms@cerenity.org> wrote:
It's not so much that "K" is doing something dumb (for reference: we are not talking about the successor to J, which is itself a successor to APL). With a project like "K", there would be as much focus on explaining correct usage of generators as there is in Twisted focusing on correct usage of Deferreds. It would be hard to write a program using "K" and not understand the ramifications of what you're doing.
I looked at "K". Reading code in it was powerfully weird - and this is coming from the author of Twisted ;-). It was enough like Erlang that it would only make things easy for experts, so I don't think that it would lead to this problem - on the other hand, I haven't seen how large communities react to it.
I have a lot of experience dealing with people that ask questions in the form that the poster did. As a rule they don't have any idea how hard solving concurrency problems is, and have been lead to believe (by Java or whatever) that concurrency is easy, just a matter of running 2 or more "simple" synchronous (-looking) programs at a time. While I don't know the OP personally, I am willing to risk their taking offense, so that I can continue to drive home the point to hordes of fledgeling developers out there that seem to desperately want to believe that massively concurrent programming is easy.
On Sunday 30 October 2005 06:31, glyph@divmod.com wrote:
It does seem to be the case, yes.
I looked at "K". Reading code in it was powerfully weird - and this is coming from the author of Twisted ;-).
Great :-) Can I quote you on that? :-D Seriously though, I did kinda expect that. I suspect the two different ways of writing code will be easier to different sorts of people. Personally I find the approach natural and normal. You can take any implication you like from that ;-)
Interesting viewpoint. KInda at odds with what we've seen so far, but then we haven't had a large community react to it yet, there's also the caveat above of it might be a mindset thing rather than skill level this, and mainly at the moment I don't think it's ready for a large community yet* - if one's appropriate (We're still in the process of finding the best way of writing systems using it). * We're not in a rush FWIW, it's more inspired by things which /aren't/ Erlang. I've not written anything in Erlang :) If it turns out though that you're right - only makes things easy for experts - then the project's failed (which won't be a problem, that's the point of research to try things and see what works/doesn't, just means I have to try again). As a result I hope you're wrong (for once :). Thanks for the feedback! Best Regards, Michael.
# #>> I want to encapsulate deferreds into a function that looks like #>> synchronous when invoked. #> #> I use a similar mechanism in my database wrapper module. here is an example of it in use: http://www.darkarts.co.za/repository/tada/doc/examples/taoBasicUsage.py (online syntax highlighted version of the same: http://www.rafb.net/paste/results/TmpcYr74.html) The code is part of my tada project at : http://www.darkarts.co.za/projects/tada/ I agree, callbacks are ugly, python 2.5 will be great :D -- Clive Crous http://www.darkarts.co.za/
Clive Crous wrote:
I agree, callbacks are ugly, python 2.5 will be great :D
What will make Python 2.5 better? I'm looking over PEPs now. PEP 342 looks amazing. I don't see a "Python-Version: 2.5" there, though. Is it expected to make it into 2.5? When might it actually get used in Twisted, given its policy of supporting old Python version? (I saw the recent debate on dropping Python 2.2 support.)
# Clive Crous wrote: #> I agree, callbacks are ugly, python 2.5 will be great :D #> # What will make Python 2.5 better? I'm looking over PEPs now. PEP 342 # looks amazing. I don't see a "Python-Version: 2.5" there, though. Is it # expected to make it into 2.5? When might it actually get used in # Twisted, given its policy of supporting old Python version? (I saw the # recent debate on dropping Python 2.2 support.) My appologies, i did in fact mean pep342, it somehow got into my head as '2.5 fact'. Whether it makes it into twisted or not is irrelivant, i will use it nonetheless ;) -- Clive Crous http://www.darkarts.co.za/
On 10/30/05, Clive Crous <clive@darkarts.co.za> wrote:
Well, it _is_ currently in Python's trunk, so unless it's reverted before 2.5 is released, it'll probably be in 2.5. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | w----v----w-+
On Sat, 29 Oct 2005 22:12:26 +0200 (SAST), Clive Crous <clive@darkarts.co.za> wrote:
the OP wanted a function that looks synchronous when *invoked* - the example you gave doesn't look synchronous to the caller at all - the functions decorated with "genAside" return deferreds. it's easy to pretend that they don't, if you decorate all of your functions similarly, but at some point some plain, upstream function is going to have to deal directly with a deferred (this is not the case with the example because it is a script, and nothing actually depends on the result of doit()). also, if any operation inside the doit() function raises an exception, the reactor will not stop. something like this would probably be better: doit().addErrback(log.err).addCallback(lambda ignore: reactor.stop()) i'd like to reiterate glyph's advice that suggesting generator based short-cuts to twisted beginners is not a good idea. it's *very* easy to use these things without understanding what is actually going on under the hood, and massive confusion frequently arises as soon as something crops up that doesn't fit into that programming style. -- Moe Aboulkheir
# On Sat, 29 Oct 2005 22:12:26 +0200 (SAST), Clive Crous # <clive@darkarts.co.za> wrote: #> #># #>#>> I want to encapsulate deferreds into a function that looks like #>#>> synchronous when invoked. #>#> #>#> #> #> #>I use a similar mechanism in my database wrapper module. #> #>here is an example of it in use: #>http://www.darkarts.co.za/repository/tada/doc/examples/taoBasicUsage.py # # the OP wanted a function that looks synchronous when *invoked* - the # example you gave doesn't look synchronous to the caller at all - the The origional question was: """ So, how can I write mySyncFunc() so that at the end I print the result of someCalculation() and not just 0? """ My example and proposed solution does that. And as such is usable by him. # functions decorated with "genAside" return deferreds. it's easy to # pretend that they don't, if you decorate all of your functions similarly, # but at some point some plain, upstream function is going to have to deal # directly with a deferred (this is not the case with the example because it # is a script, and nothing actually depends on the result of doit()). also, # if any operation inside the doit() function raises an exception, the # reactor will not stop. something like this would probably be better: # # doit().addErrback(log.err).addCallback(lambda ignore: reactor.stop()) # it is an example usage, not a real world implementation, of course if something fails it will die. # i'd like to reiterate glyph's advice that suggesting generator based # short-cuts to twisted beginners is not a good idea. it's *very* easy to # use these things without understanding what is actually going on under the # hood, and massive confusion frequently arises as soon as something crops # up that doesn't fit into that programming style. # Why do i hit this wall constantly when discussing twisted usage with twisted users or developers mostly on freenode's #twisted admitedly: Telling someone NOT to do something is not answering a question it is avoiding it. I pasted the url for the full source so that he could "look under the hood" if he feels uncomfortable with it's mechanism he needent use it. By your reasoning surely he should go through the entire twisted codebase aswell and know exactly what a deffered is doing "under the hood" and how twisted does it's asyncronousity before using it ? -- Clive Crous http://www.darkarts.co.za/
On Sun, 30 Oct 2005 00:11:07 +0200 (SAST), Clive Crous <clive@darkarts.co.za> wrote:
no, it really doesn't. here is the code snippet given by the original poster: def mySyncFunc() x = 0 def done(data): global x x = data d = someCalculation() d.addCallback(done) <something here to hold until "done" is really done> return x the desired functionality is "at the end i print the result of someCalculation() and not just 0". this could obviously be done by changing the first line of done() to "print data". but the "return x" betrays the intention that the function would ideally return the value of "x" (returning the _value_ of "x" is worlds away from returning a deferred that fires with the value of "x"). no defgen type solution is going to make this possible
the point here is that this particular question does not have a satisfying answer. it has been beaten to death recently that you cannot magically expose a synchronous interface to asynchronous code. so, in summary, the question was answered, by several people in this thread, and the answer is "you cannot do that and expect your code to work reliably". is that not a valid answer?
ideally yes. somebody unfamiliar with deferreds is going to be in no position to determine whether or not they are comfortable with a generator based mechanism for handling them. people already associate unfair amounts of magic with twisted, and i dont think suggesting such abstract interfaces to beginners is an effective way to mitigate this. -- Moe Aboulkheir
On Sun, 30 Oct 2005 00:11:07 +0200 (SAST), Clive Crous <clive@darkarts.co.za> wrote:
You are correct. I do not disagree with this fact, and it does not change my opinion of the behavior involved. Here is the ugly truth of the non-answers to such questions: I do not care about most of the posters to this list. I am happy that they use Twisted, and even more happy if they are pleased by it, but ultimately I am really just trying to improve the world I live in, and that means the average quality of available Python code. That means I am only going to answer questions which I believe will contribute positively to the state of affairs as it regards code quality in general, and Python code quality specifically. These people who people whose questions are answered (or not answered) on this list or on IRC will go and write applications or libraries based on the answers they receive. I may be stuck maintaining or using that code at some point. Now, maybe if I know, but don't tell them how to do some ridiculous thing they're trying to do, they'll miss their deadline at work or they won't turn on their homework assignment on time and I could have prevented that. They will be unhappy. Making them happy isn't my purpose here, so I don't care. Maybe they'll abandon their Python project and do it in Ruby instead because the Python community is so hostile. Many people think that this loss of contributing labor is a great tragedy for an open-source community, but IMHO it is really the best possible consequence. All the people who don't understand programming will end up in the more-polite Rails or PHP or Django or TurboGears communities and generate huge piles of code that don't work. Then all the guys who know what they're doing will stick around here and work on Twisted stuff. In the sense that we compete with those projects, this is great! It's like sabotage-by-proxy. When those projects are a smouldering ruin of inconsistent style and half-baked, buggy, insecure code, people who want a functional product will come ask those of us with Twisted expertise. A more realistic consequence, however, is that programmers new to Twisted will adopt a more consistent style, and avoid fighting the framework, and concentrate on solving their actual problems. Some will still go away, yes, but generally there is some reason they came to Twisted in the first place and it remains valid even if we won't make it easy to do things we collectively regard as bad practice. I have adopted this stance not merely because I am abraisive asshole, but based on long experience with IRC and with teaching programmers how to use things. There are several projects that were developed early on with Twisted that were utter disasters because I politely and pateiently answered all the authors' questions about how to make Deferreds appear to block, how to call reactor functions from threads, and how to invoke Twisted from C code, rather than stopping and saying "hey, what are you *really* trying to do?". (No, I will not name these projects. I do have *some* manners.) I can only assume that the other Twisted devs you've had problems with have gotten this habit from similar experiences, but they may have their own reasons.
It's not so much whether the OP is comfortable with your explanation or not. If I were confident that all programmers would go and investigate the solution first, I would not care to make a post such as this. It's whether, upon not understanding, the OP will encourage others to program using bad style with a tool that he is comfortable with but but does not *understand*. It is quite possible that the OP understands your solution perfectly well, in which case this conversation is useless to him and he can ignore it, but still socially useful in that it will appear in google searches when other, less experienced programmers have the bright idea that they can make Deferreds magically go away somehow. (Apologies to the OP: I really have no idea about you at all. I am really speaking of a hypothetical average programmer.)
#>Why do i hit this wall constantly when discussing twisted usage with #>twisted users or developers mostly on freenode's #twisted admitedly: #>Telling someone NOT to do something is not answering a question it is #>avoiding it. # # You are correct. I do not disagree with this fact, and it does not change # my opinion of the behavior involved. Here is the ugly truth of the # non-answers to such questions: # # I do not care about most of the posters to this list. I am happy that # they use Twisted, and even more happy if they are pleased by it, but # ultimately I am really just trying to improve the world I live in, and # that means the average quality of available Python code. That means I am # only going to answer questions which I believe will contribute positively # to the state of affairs as it regards code quality in general, and Python # code quality specifically. # # These people who people whose questions are answered (or not answered) on # this list or on IRC will go and write applications or libraries based on # the answers they receive. I may be stuck maintaining or using that code # at some point. # # Now, maybe if I know, but don't tell them how to do some ridiculous thing # they're trying to do, they'll miss their deadline at work or they won't # turn on their homework assignment on time and I could have prevented that. # They will be unhappy. Making them happy isn't my purpose here, so I # don't care. Maybe they'll abandon their Python project and do it in Ruby # instead because the Python community is so hostile. # # Many people think that this loss of contributing labor is a great tragedy # for an open-source community, but IMHO it is really the best possible # consequence. All the people who don't understand programming will end up # in the more-polite Rails or PHP or Django or TurboGears communities and # generate huge piles of code that don't work. Then all the guys who know # what they're doing will stick around here and work on Twisted stuff. In # the sense that we compete with those projects, this is great! It's like # sabotage-by-proxy. When those projects are a smouldering ruin of # inconsistent style and half-baked, buggy, insecure code, people who want a # functional product will come ask those of us with Twisted expertise. # # A more realistic consequence, however, is that programmers new to Twisted # will adopt a more consistent style, and avoid fighting the framework, and # concentrate on solving their actual problems. Some will still go away, # yes, but generally there is some reason they came to Twisted in the first # place and it remains valid even if we won't make it easy to do things we # collectively regard as bad practice. # # I have adopted this stance not merely because I am abraisive asshole, but # based on long experience with IRC and with teaching programmers how to use # things. There are several projects that were developed early on with # Twisted that were utter disasters because I politely and pateiently # answered all the authors' questions about how to make Deferreds appear to # block, how to call reactor functions from threads, and how to invoke # Twisted from C code, rather than stopping and saying "hey, what are you # *really* trying to do?". (No, I will not name these projects. I do have # *some* manners.) # # I can only assume that the other Twisted devs you've had problems with # have gotten this habit from similar experiences, but they may have their # own reasons. # I find it quite amusing, yet somewhat disturbing how you justify this behaviour to yourselves. I have worked on many projects in my life and each of them was unique in it's specific requirements. Let me give you two of my current projects as brief examples: The first, is for a client who would like a downloadable executable available on his website. The primary need, beside the obvious need that it in fact work, is that the executable be as tiny as possible to encourage potential clients in turn to download and use it. The second project is a set of modules for internal use by the company I now work for. The primary focus, again besides the obvious need for it to be functional, is that it be highly maintainable and easily modified by the senior most programmer or the newest, week-old junior developer. Anybody must be able to look at the code, understand it quickly and easily, and be able to modify parts as required. If I employed the same style of programming in both of these projects I would be a fool. Some of the styles implemented to make a small executable in any other circumstance would be considered incredibly bad practice, yet there it is, working perfectly and being incredibly small while doing so. Meanwhile some of the code within the second example is incredibly verbose and a seasoned coder would look at it and say "Why for the love of python are you doing this the long way round?" ... once again, it fulfills it's specification perfectly while doing it's job to the satisfaction of all concerned. When it comes to implementations of twisted, one must take into account the context in which the program is being developed and the specific use cases it requires. No two implementations will ever be the same, and no two implementations should ever use twisted in exactly the same way. To expect identical or even similar usage of a module in every single instance is poor practice in itself. If I come online, and ask: "How do i use A with B" and the response is "Dont!", as in this thread, it shows a high level of arrogance, extreme presumption and an incredible 'naivety to programming' by the person giving that "answer". There always will be people who have different needs to you. Enforcement of usage policy on users is the realm of restrictive corporate use policy and software distribution licensing as one expects from Microsoft et al. I have worked with programmers who hold the same mind-set as is being shown here. Some of the "programmers" with whom I have worked in the past, were insistent that things should be done in a particularly restrictive, stylized manner and that no-other-way-is-the-right-way. There is *NO* absolute "right way" when it comes to coding. There are good ways and bad ways however all are dependent on the context of the project at hand and it's particular specifications. The most common result of such narrow-minded programming mind-sets is the inability to complete the task at hand. Programs will be written, rewritten, re-factored, tossed out and restarted. This cycle will never end as long as this zenith of coding "nirvana" remains the goal. The tragic result of this is that projects repeatedly fail to reach a usable state, to the detriment of all. I hope twisted development does not go down this path and shows some maturity and the ability to sustain it's own usage. I really do love the twisted framework and would hate to see it dragged into a quagmire of self-indulgent disarray to the despair of a dedicated and loving user base (whom you freely admit not to care about at all). -- Clive Crous http://www.darkarts.co.za/
On Mon, 31 Oct 2005 00:02:47 +0200 (SAST), Clive Crous <clive@darkarts.co.za> wrote:
You're calling me arrogant and naive, and a fascist champion of policy enforcement: yet you have the gall to tell me what *I* must do with *my* freely donated time. I didn't say "only stupid people do this" when you answered the question, or delete your message from the archives, I merely offered my opinion that it is a bad thing to do. You are replying to my explanation of why we say things like that, not why *you* must never do so. However, now that we're name-calling, let me suggest instead that *you* are arrogant to presume that you know more about the questions people ask about Twisted than I do (how long do you think I've been answering these, anyway?), and that you are incredibly naive about education in general. When you have seen the same question asked a thousand times, you learn to see the question behind the question. You understand the unseen consequences of answering it straightforwardly. What can be a perfectly legitimate question from an intermediate or expert programmer is simply a set-up for disaster from a novice. For example, if a 12-year-old girl asks you "can I get pregnant from sex if I only do it once", this is most likely not idle curiosity speaking. You should not answer as accurately as possible, with equivocations about the probability. You say "YES, YOU ABSOLUTELY WILL, WHERE IS YOUR MOTHER". If a married couple in a fertility clinic asks you a similar question, about how difficult it is on average to conceive, it would be a good idea to just answer. The context of this mailing list is less obvious than either of those, but I have found over time that if the poster really knows what they're doing, they will respond to my refusal with an explanation and I can offer more detail at that point - quite often they are asking a very different question than "how can I make this look synchronous", it is something like "I need to integrate with a proprietary library that will call my code synchronously and I have no access to its source code" or "I would like to use a Twisted protocol implementation from within an existing Zope2 site". Neither of those questions are properly answered by "use deferredResult" or "use defgen". It's complicated.
This is exactly what happens with Twisted - you got everything right except the result, which is that *code quality goes up over time*. It is this dedication to quality that makes the project fun and interesting to use. Granted, breaking backwards compatibility is never fun, but we try hard to minimize the impact of that. For a project which has a strict deadline and budgetary constraints, this is obviously a terrible tragedy. I have seen it, I have even caused it (who hasn't?), and it's ugly. However, this is exactly what makes open source so much fun; you don't have to get it right the first time, you don't have to get it right in 6 months, you can just keep trying and trying and trying. Eventually you get it right and everyone thinks it's awesome. I am sure that no current users remember the massive upheaval and 1 1/2 years it took to get the transport/protocol split properly implemented. If we had not "rewritten, re-factored, tossed out and restarted" that code, Twisted would be nothing but a much set of libraries for Medusa today.
Luckily I am also a user and my career, my reputation and my income all depend quite explicitly upon its success. Many Twisted developers are in the same position now, so selfish motivations are more than adequate. Seriously though, my point about not caring about the community was slightly tongue-in-cheek. The point there is that I am not here to do your homework assignment, I am not here to do your job for you. If you think that's what I *am* here for, then no, I don't care about you. I am here to help you learn to use Twisted and learn to use it correctly. If you don't like my advice, the code is all free, there are other people to answer your questions, and there is quite a bit of freely-available documentation (and even a book from O'Reilly now!).
On Mon, 31 Oct 2005 00:02:47 +0200 (SAST), Clive Crous <clive@darkarts.co.za> wrote:
I find it equally arrogant to expect the community to behave in the way you would prefer. What is owed to you that justifies this, exactly?
First of all, no one is forcing anything on you. You're not locked in. Don't use Twisted if you don't want to. Don't use Twisted in the manner we are explaining is correct if you don't want to. Fork Twisted and take development in a better direction. Rip parts of Twisted out and drop them into your own project to use how you please. Spit on our works. Curse our names and our progeny to the seventh generation, if that's your thing. Who's going to stop you? Not me, and not anybody else on this list. I recognize that you probably don't want to do any of these things. You want to keep using Twisted, and you want the Twisted team to do things your way. I have this same expectation of other projects. Let's face it: it's an unreasonable expectation. Twisted is developed for you by a team of uncompensated developers working in their spare time. Likewise, support is being offered for free. It should not be too surprising that it is also being offered *conditionally*. A lot of us are happy to help with problems that don't directly benefit us, but a lot of us are not happy to help with solutions that we think are just as problematic as that which they are intended to resolve. Are you going to fault us for limiting the extent of our unpaid donations (frequently to commercial development)? Please. Furthermore, a perfectly legitimate response to "Don't!" is "But I have constraints X, Y, and Z." You may feel it is unreasonable to explain your situation, but that's pretty much too bad for you. If you want to be able to dictate people's behavior to this extent, negotiate a contract with them. Now, we may not be convinced by constraints X, Y, and Z, but in most cases where they are rational and correct (ie, not "I have to use threads because Cthulhu will rise and consume the world if I do not" - although if you are working on the project to prevent this, and are using Twisted, awesome! Can we put you on the successes page?), we will be. It is the refusal to explain them at all that is most annoying, and most likely to result in no assistance being rendered. I don't think this attitude is unique to the Twisted community. I don't even think it is unique to open source communities. Nor do I think it is a negative attitude. I think it is more open minded and generally productive than just presenting the precise answer to each exact question posed. In many cases, I have concrete evidence that not answering a direct question with a superficially correct response has helped the questioner measurable, and I've been thanked for this approach repeatedly.
This is a mischaracterization of the attitude I use when trying to help people, and from what I've witnessed of the interactions of other members of the Twisted community, I think it is not representitive of their behavior either. The interaction that gives rise to this impression, in my experience, most typically involves much more adament insistence and inflexibility on the part of the person seeking help than on the part of the person from which help is sought: the questioner *knows* they are already on the right track, and if this jerk he is talking to would just stop asking questions and answer them for a change, the problem would immediately be solved.
Which is why I stress the importance of discussing the context of a problem before trying to resolve it.
Twisted comes with a news server, a mail server, a chat server, a chat client, a remote REPL with multiple front ends, a plugin system, numerous persistence systems, a documentation framework with multiple input and output formats, an "Application" abstraction, an SSH client, an SSH server, an authentication and authorization system, a mature TCP, UDP, and SSL client and server API with numerous implementations for numerous platforms, an asynchronous DB-API 2.0 wrapper, an inetd-alike, a high-level remote method and object protocol, a unit test framework and rich command line tool for interacting with test suites, a web server, a web client, a DNS server, a DNS client, and FTP server, an FTP client, implementations of the echo, discard, chargen, ident, finger, QOTD, who, daytime, time, telnet, sip, toc, and oscar protocols, a filesystem path abstraction, a threadpool implementation, a much higher-quality module reloading system than the builtin reload(), a logging system, and quite a few other mature, stable features. Arguing that the Twisted team cannot complete tasks is ridiculous.
Twisted has been driven by hate for a long time. Hate and shared goal of the team to destroy the sun. There's no change in direction going on, at least with regard to the fundamental drive directing Twisted's development. Jean-Paul
I am the original poster of this thread. First of all, thank you for the different responses. However I still don't have a clear answer (other than what I'm thinking of doing is wrong, which is OK, it can be fixed). I'm working on a Twisted/Nevow application, and yes, I'm new to both and I can tell that if learning Twisted has a step curve, doing it in conjuction with Nevow is really almost vertical. But, nevertheless, my application is working well these days. The motivation for my question is the following: I added the guard/credentials stuff to my application. The issue is that in order to accept a login my application requires several things to happen before access can be granted. I therefore implemented my own AuthChecker() which fires several deferreds to acomplish its task. This is working well, including the use of deferToThread() in several places. But, the behavior of the authentication wrapper is synchronous. That is, access to the system cannot be granted until all deferred operations have happended. In this wrapper I have something like: def requestAvatar(self, avatarId, mind, *interfaces): for iface in interfaces: if iface is inevow.IResource: if avatarId is checkers.ANONYMOUS: resc = login.LoginPage() resc.realm = self return (inevow.IResource, resc, noLogout) else: resc = home.RootPage() resc.realm = self return (inevow.IResource, resc, self.createLogout(avatarId, mind)) else: raise NotImplementedError("Not supported.") So let me rephrase my original question. Given that the real user authentication is happening in my AuthChecker() implemented somewhere else, with deferreds and what not, how can I force this requestAvatar() thing to wait until the whole login process has finished? My idea was to force a synchronous behavior on AuthChecker() forcing this way the login process to wait for it to finish. While writting this I realize that I may be missunderstanding how the credentials stuff works ... who knows. Any suggestions are appreciated, Thank you, -- Pedro
On Mon, 31 Oct 2005 08:28:39 -0500, Pedro Sanchez <psanchez@nortel.com> wrote:
Cred supports Deferreds, so there's no need to make your requestAvatar or requestAvatarId methods synchronous. They can simply return Deferreds. If you have several Deferreds going in requestAvatarId, you may want to use a DeferredList, which behaves very similarly to a Deferred, but takes several other Deferreds as input and only fires when all those Deferreds have fired. requestAvatar won't be called until the Deferred you return from requestAvatarId fires. Jean-Paul
Stefan Behnel wrote:
I tried this already and couldn't make it work. In one of the functions called during the validation process I have this def registerSession(self, session): def _customerInit(self, customer): ... session.customerInit = True # initialize customer data d = defer.Deferred() d1 = defer.maybeDeferred(customerInit, session) d1.addCallback(_customerInit) d1.addErrback(errhandler, session) d.chainDeferred(d1) return d customerInit() fires several deferreds which I collect with a deferredList already. This works well. My goal therefore is to force "registerSession()" to behave synchronously[1] so that whoever is invoking it will block. I thought that by defining my own deferred d, and that by chaining it to d1 I would achive this. But I don't, the caller still doesn't block and goes along thinking that registerSsession() is done. I guess I don't understand how to use the chainDeferred() call. Hints are welcome; regardeless of what I end up doing, I'd like to understand who deferred chaining works. In a previous answer from Jean-Paul Calderone, he mentions that the requestAvatar and requestAvatarId methods don't have to be synchronous (thanks Jean-Paul!). This is no evident from the documentation, in fact, I dare to say that in all examples I've seen so far these calls are always synchronous, therefore my wrong assumption that they had to be. This is likely the right way to go to modify my program. I'll look into it. Thanks again! -- Pedro [1] Boy, I say this with hesitation after the long thread I spawned with my post, and after reading repeately that I shouldn't even attempt to do it. Which is fine, I take it, thanks for the imput. But I'm just trying to figure out the best way around it for my application.
Pedro Sanchez schrieb:
Ah, there we go. This simplifies to def registerSession(self, session): def _customerInit(self, customer): ... session.customerInit = True # initialize customer data d1 = defer.maybeDeferred(customerInit, session) d1.addCallback(_customerInit) d1.addErrback(errhandler, session) return d1 And then everyone who calls registerSession() receives a Deferred and can add callbacks for whatever else needs to be done after the session is successfully registered and _customerInit has been run. You get the difference? The registration is not done when registerSession /returns/, but when the Deferred that it returns /fires/. So keep adding callbacks to that Deferred instead of expecting things to have terminated. Twisted will then take care of running everything in a chain. Stefan
Stefan Behnel wrote:
Yes, I understand that very well. The problem is (was, I guess) my wrong assumption that the requestAvatar function (which at the end is the "caller" of registerSession() in my actual code) was supposed to be synchronous and not to deal with deferreds at all. Thanks, -- Pedro
Given that there has already been a long thread about how one should not do this, which I completely agree with, I'll not repeat it. Note that I do not recommend this solution to the OP, but rather post this just to appease to the people complaining about how the only answer given is "don't do that". "Don't do that" is the correct answer given the information so far available, but, here's a different non-answer, with code. On Oct 28, 2005, at 5:40 PM, Pedro Sanchez wrote:
The above code cannot work because for "done" to become done, the twisted event loop must get a chance to run. For the event loop to get a chance to run, you must return from that function. Basically, how can you "block" waiting for an async event, while letting the twisted event loop continue running? There is an answer: run your code in a separate thread. However, that doesn't work either, because the OP wants to use twisted APIs like ADBAPI or networking in someCalculation. So there really is no way to do literally what's asked in a working way. But, you can get something "like" the above by splitting your code into synch-like-code to be run in a separate thread, and async-twisted-using-code to be run in the twisted reactor thread. Here's a little example. Please note, again, that I do not recommend doing this except in circumstances where you absolutely must. import Queue from twisted.internet import reactor, defer from twisted.python import failure def callInReactor(__f, *__a, **__kw): # Called in other thread queue = Queue.Queue() reactor.callFromThread(_calledFromThread, queue, __f, __a, __kw) result = queue.get() if isinstance(result, failure.Failure): result.raiseException() return result def _calledFromThread(queue, f, a, kw): # Called in reactor thread result = defer.maybeDeferred(f, *a, **kw) result.addBoth(queue.put) def someCalculation(): # A demo "calculation" d = defer.Deferred() reactor.callLater(4, d.callback, 'hi') return d x = 0 def myAsyncFunctionInTwistedThread(): def done(data): global x x = data d = someCalculation() d.addCallback(done) return d def mySyncFuncInAnotherThread(): # NOTE: in this other thread you cannot call any twisted APIs besides reactor.callFromThread # and a very few select others. callInReactor(myAsyncFunctionInTwistedThread) # callInReactor blocks until the deferred returned by myAsyncFunctionInTwistedThread fires print x # Start up a thread to call your blocking function in reactor.callInThread(mySyncFuncInAnotherThread) # Run the reactor reactor.run() James
participants (14)
-
Christopher Armstrong -
Clive Crous -
glyph＠divmod.com -
Itamar Shtull-Trauring -
James Y Knight -
Jean-Paul Calderone -
Jeff Grimmett -
Jonathan Lange -
Marcin Kasperski -
Michael -
Moe Aboulkheir -
Pedro Sanchez -
Scott Lamb -
Stefan Behnel