[Twisted-Python] Evangelism notes...
As some of you know, I use Twisted to develop one of our products, a fairly large Cable modem management application. I've developed and released TwistedSNMP, and am, generally speaking comfortable with the package. I've also been using Twisted within the company for my other projects. I've just come back from a meeting where a decision was made to completely rewrite a *delivered* application on our own dime for the sole purpose of eliminating Twisted. I found this rather disappointing. I realise that the problem lay in part in my own "choice" (didn't actually choose, it just happened) of projects with which to introduce Twisted, but I felt it would be useful to share the experience with others. The Project: * Cross-protocol server intended to act as a "web services" intermediary for payment-processing services from a number of different banking systems. o Fairly straightforward networking protocols on all sides o All protocols were custom (i.e. we had to write them) save the client-side interface (which was XMLRPC) o Seems like a perfect problem domain for Twisted * Initially coded using Twisted by me, then pair programming with a colleague as he took over lead on the project o If my goal had initially been to evangelise, I should have *started* the project with pair programming so that the system wasn't already in-place and mystifyingly complex + Even the Finger sample is mystifyingly complex if you jump into it at the end of the process + Unfortunately, I didn't realise the project would be passed on to someone else before completion, I'd thought it would remain part of my portfolio within the company + Even if we did start out together, we really should have done some *formal* training on Twisted, rather than relying on knowledge transfer via pair programming. Again, it started out just being a few lines here and there, we didn't see much being done beyond the base system. o He never, in the entire process, saw anything really come from Twisted (other than problems where the complexity of the system got in the way of doing the rather simple task (from his point of view)), since the low-level coding was already done when he started working with the system. + Probably would have been better to start him on the much larger project, where Twisted represents far less of the overall complexity of the application. o In particular, test-driven development with threading in the background for blocking operations caused significant headaches * Project was delivered, paid for, and has been running in production for a few weeks, but 2 reported bugs cropped up o To fix one, needed to update to 2.0 (deferred object cleanup)... this caused rather a lot of upset, as the sysadmins were upset about such a huge package needing to be updated (and they tried to do just the core packages first, which doubled the work-load involved in preparing that update versus using Sumo) o Fixing the other was no big deal, except that something in 2.0 changed how the test-suite was running so that it now hung on completion... * Now, at this point there had been serious conversations just an hour earlier during a management meeting about replacing Twisted for 2 separate projects. The entire company is watching as we go to fix the *trivial* error while we're together after the meeting... but the test suite won't complete. Must be something wrong with the background thread not exiting thinks I. o What do we call to shutdown the threadpool? I don't remember, let's check the reference docs... 20 minutes of searching for the 2.0 reference docs (with the entire company watching this)... finally ask on IRC... told to build them myself and contribute them. Cute, but didn't really drive home the whole "robust, professional, well-documented platform on which to develop business systems" meme as much as I'd hoped. (Some of that is definitely my fault, I was pretty frustrated by the time I went to IRC to ask where the documentation was). o Finally go with the 1.3 docs, hoping there's not some change to the API that's causing the problem (though that looks very probable at this point, the change to 2.0 being the primary difference between working and not). * Finally get the 1.3 docs, hack together a reactor.threadpool.stop() call with a dummy last-run-test-suite and get the test suite to run-and-exit... o Keep in mind, we've just had to introduce a glaringly ugly and fragile hack to get around something trivial. There may be a better way to run the tests, but it's a bit late for thinking of that now... * Now that we can run the test suite and focus on what happens during it, we discover that the ESMTP client can't connect to a local SMTP host without authentication (or at least, it wouldn't with the code we were using across the network to a remote host)... no documentation on the ESMTP client, of course.[1] * At which point my colleague (lead on this project) got well and truly fed up and decided to dump Twisted and rewrite the project in plain-old-Python. There's lots of points where, had I anticipated the problems, I might have salvaged the project in its current state. If I'd anticipated that the problems would be non-trivial to fix, I could simply have returned to my workstation and fixed them in a relaxed and pressure-free environment. I could have spelunked through the code to figure out the interfaces in lieu of documentation. That's fine for me, I already understand Twisted enough to spelunk through and find what I want, but it doesn't make a great argument when someone's simply mystified by all the pieces scattered here and there that seem to be joined by nothing more than hope and bailing wire. I could have done better at evangelising. In the end, though, I had to concede that stripping out Twisted and writing the code himself would probably be faster and easier in the long run than trying to work around problems in an under-documented "foreign" system (again, from his viewpoint). Especially given the small size of the project, a few simple select loops and a thread can solve 90% of what Twisted is providing for him. Thinking back on that conversation; at that point the *only* failing feature was the ESMTP-sending, and I'm pretty sure it would have resolved to something like passing a None instead of a "" for the username and password; but when it came right down to it, he's the one maintaining the code, and as far as he can see, Twisted just gets in the way. Don't know if the other project will be rewritten as well. Given decision on this one, I'd imagine it will. I'd also predict that Twisted is now Techne non Grata wrt any new development where it's not an explicit requirement (given that we're going to be eating costs on rewriting a project to eliminate it). Certainly I'll have a hard sell going forward. So, anyway, just felt the need to share that. Sorry again for the sharp post to IRC. Peace, all, Mike [1] How did ESMTP get in there anyway, if it's a 1.3 codebase? Well, it shares common functionality with the big application, one piece of that being the automated use of ESMTP. -- ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
Itamar Shtull-Trauring wrote:
On Wed, 2005-05-04 at 16:08 -0400, Mike C. Fletcher wrote:
Mike, I'm sorry that Twisted was a part of such a horrible experience for you! Thanks for sharing the experience, but evangelism tip #1 that I get from the initial reading is from the real-time, public bit at the end. - Do not ever, ever use IRC in a public forum to solve a problem in real-time. In fact, do not ever use any support mechanism in a public forum unless you or your company has a paid relationship with the people on the other end of the wire. - Do not ever expect to be able to fix a threading bug in less than 2 days.
Glyph Lefkowitz wrote: ...
Wasn't a truly horrible experience. The company's fairly laid back, it was just disappointing.
Um, I hope you're joking. Do not *ever* use a public support mechanism without a paid relationship? So is everyone on IRC paid up? Or is it not a support mechanism? Or is it just unwise to ever *use* Twisted IRC; it should just be considered a waste of time? Absolutes like that are kind of extreme, no? Maybe you meant to say only for real-time problem solving... still seems to make the immediacy of the forum somewhat pointless, no? It did actually solve the problem, btw, and at least the search for the docs was cut short at 20 minutes. If that was intended more as a "who do you think you are, you've not paid us anything", okay, accepted. I'd been thinking of myself as a member of the community trying to sell "our" product to the world, I should have looked at it from the other side. We've (and I've) *tried* to contribute back to the Twisted community with introductory lectures at PyGTA, various pieces of documentation and releasing TwistedSNMP, but it's perfectly true that we've made no monetary arrangements for support. Fact is that we operate on a shoe-string budget to keep costs low for our customers and try to participate actively in the Open Source community to compensate others for the assistance we often get. I apologise if I went over the line in asking where the reference documents have moved ;) :) . Seriously, I didn't even think of asking in IRC, it was suggested by another colleague between stifling guffaws... maybe he was out to get me by tricking me into exactly this folly ;) .
- Do not ever expect to be able to fix a threading bug in less than 2 days.
:) That was obviously the root problem. Peace, out, Mike -- ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
On May 4, 2005, at 5:36 PM, Mike C. Fletcher wrote:
Presumably, this is what he meant: Don't CC the IRC conversation to the rest of your company ("in a public forum"). The S:N ratio and mannerisms of IRC aren't appropriate to broadcast to the rest of the world (unless they are also acquainted with how IRC works, etc.). Don't expect free help exactly when you need it ("in real-time"). #twisted is quite helpful, and often does end up being free real-time help, but there aren't any guarantees. -bob
Mike C. Fletcher wrote:
Glyph Lefkowitz wrote:
No no, you misunderstand! Your contributions have been substantial, and I didn't mean to impugn them. I don't mean, "don't use IRC" or "you've got no right", I mean, you are likely to get flak, humor, or random people being rude to you for no reason as you are to get support. This can get you in trouble in a meeting. #twisted is like a bunch of mechanics and automotive engineers (and various hangers-on) hanging around in a garage. It's not a slum, but it is a garage, there are going to be parts lying all over the place and people are informal. Going to IRC with the president of your company watching your screen is like inviting your auto mechanic and his girlfriend into your boss's limousine to discuss the company's car purchasing plans for the next year.
Hmm... sounds like maybe he was...
Glyph Lefkowitz wrote: ...
Our president, incidentally, is a leather-wearing biker (and a NetBSD developer), but I'm sure the mechanic and his girlfriend would get over their offense, eventually ;) . Point taken, though. No matter how laid back the individuals, when trying to convince a group about the viability of a product in a business setting it's not a good forum to which to turn. Have fun, Mike -- ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
Itamar Shtull-Trauring wrote:
Don't get too defensive, Itamar. Although I missed the IRC conversation, Mike's email certainly described a situation that anyone would have found frustrating -- and to me it did not appear to be an indictment of Twisted so much as simply a cautionary tale, and a useful one. It's reality. Mike knows (in retrospect) he would have done things differently if he had anticipated the docs not being there (i.e., go to his cubicle and quietly do the spelunking, away from skeptical eyes). I daresay if I were selling Twisted or handing off a Twisted app in my environment, the situation would be similar (God forbid I ever get to that ... which is part of the reason I haven't released yet ;). Twisted's documentation is constantly improving (thanks to Mary and all who contribute!), but it can still be daunting to the uninitiated, largely because Twisted is non-trivial and good technical writing is harder than it looks. Mike's experience was a confluence of many small dysfunctions. IOW, the makings of a classic tragedy -- not that it was a *huge* tragedy, but I'm sure Mike would prefer to work in an environment in which he can use Twisted and not be second-guessed (at best), and as he ruefully points out, the well is kind of poisoned now. So what's to be learned? The lesson for me is: if you are in a similar environment, go ahead and used Twisted -- to those who know it, it is reasonably robust and scalable -- but be very, very careful when demoing and/or introducing anyone else to your app. And if you encounter any bugs, no matter how insignificant the might at first appear, for heaven's sake go back to your cube for the spelunking, unless you are a Minion and know exactly what you're doing! ;) Cheers, Steve
Stephen Waterbury wrote:
A grim reality indeed. I myself have been in this situation (and even been accused of being a shyster!) and it is a very hard one to get out of. As Steve points out, there is a general-purpose learning here, but there is also the issue that given Twisted's technological maturity, users are beginning to expect a certain level of polish and sophistication from our web presence, our marketing materials and our documentation which just isn't there. Worse, this is a long-standing problem I don't see any effort underway to rectify it. Other open-source projects have gotten to this point and found huge, helpful teams of busy bees to fix the website, keep the documents up to date, wrangle the release notes and annotate the development process. Some projects are worse - I've recently had an experience with the linux cifs client project website that made me feel really good about our web presence :-) - but we could do better. That said, as it is I think the Twisted team is doing a great job at making Twisted, and we don't have any more resources or people to spend on other stuff. That's not to say we can't spot-fix the biggest problems with our website (like the API documentation problem) but ... where do these helpful people come from for other projects? I would estimate that Twisted has tens of thousands of users by now, but the community is still oddly silent. I'm genuinely baffled as to how to proceed. Is there anyone out there who can contribute some significant time to updating our "professional" presence on the web and elsewhere?
Glyph Lefkowitz wrote:
Lacking community volunteers with the necessary writing and twisted skillsets, the best option would be to have the documentation done professionally by *good* technical writers. How does one fund this? As you say, there are tens of thousands of twisted users by now, many writing code inside corporations/universities which regularly pay for software. So start a fund: - get an accurate estimate of the amount of money needed (ie., have *good* writers lined up and ready to start work under contract) - prominently display the fund status (thermometer?) on the twisted home page - solicit support from twisted users urging them to get their employers to contribute - recognize contributions I think you'd be pleasantly surprised by the response. $150k shouldn't be too hard to raise and, spent wisely, will go a long way towards solving the documentation/tutorial problems. Our group put $12.5k back into a specific open source project last year. Money that was earned thanks to that same project. Cheers, Darran. P.S. Of course a paper book would also do wonders ... -- Darran Edmundson (darran.edmundson@anu.edu.au) ANU Supercomputer Facility Vizlab Australian National University, Canberra, ACT 2600 tel: +61 2 6125-0517 fax: +61 2 6125-5088
Darran Edmundson wrote:
P.S. Of course a paper book would also do wonders ...
+1 here... I carry Alex Martelli's Python in a Nutshell around with me every day. Alex, if you're listening, how about a book on Twisted? Several books introduce Twisted positively (e.g., Goerzen's recent Foundations of Python Network Programming), but the lack of a complete print reference has made me a little hesitant to emphasize my reliance on it in recent customer projects. Heck, I have enough trouble convincing my customers that Python is supportable (e.g., "we're a Microsoft shop here..."), so I hide my use of Twisted in the packaging. Thanks again, Cory, for ntsvc... Keep up the good work, Norm
On Wed, May 04, 2005, Glyph Lefkowitz wrote:
What particular projects do you have in mind here? I ask because it could well be useful to actually go to them and say "hey, where did you find these people?" In particular, it would be useful to ask that question of things that will be used by programmers: Python itself, other languages developed/maintained by communities, the graphical toolkits and other big libraries... (answers from, say, the GNOME user documentation people might still be useful, but I think slightly less so, because that documentation can be written by non-programmers). I can think of all kinds of answers ("they just appeared!", "we don't merge features without review of their documentation," "we paid people," "we just love writing documentation," "there's these two crazy people who just love writing documentation," "there's this one company who employs our major developers and also employs some tech writers,") but I don't know what the correct ones are. -Mary
On Thu, May 05, 2005 at 08:27:57AM +1000, Mary Gardiner wrote:
In my experience (on Jabber and a few smaller projects), the hardest thing is finding people who want to write docs. Developers don't want to do it or don't have the time, and most folks who like to write are not especially technical. They might write a howto for end users, but documenting something like Twisted is beyond their ken. In the Jabber world I have been writing docs since mid-2000 but even so I mostly focus on protocol documentation because writing docs for various implementations (libraries, servers, clients) would be never-ending. My $0.02. Peter -- Peter Saint-Andre Jabber Software Foundation http://www.jabber.org/people/stpeter.shtml
On May 5, 2005, at 12:32 PM, Peter Saint-Andre wrote:
I think there's probably a lot of Twisted "end-user documentation" lying around in the form of presentations and tutorials given at various conferences. If someone aggregates the links to all of that stuff, it would probably be a useful resource to new-ish users. -bob
On Thu, 5 May 2005 12:54:46 -0400, Jeff Grimmett <grimmtooth@gmail.com> wrote:
There's lots of wiki software out there already. Writing more won't solve any pressing problem, I think (in fact, it would just have to be documented...). In the past, web-based content systems haven't been very successful with Twisted: people just don't update them frequently enough to be useful. If enough people wanted to make a serious committment to getting a Twisted wiki established, I'm sure some hosting could be arranged, but just tossing up a wiki and hoping people contribute to it probably isn't going to fly. Jp
well that is true. i am developing one "wiki like" software in twisted (and now nevow). but from the moment i started to write it i knew already on what machines will it be hosted, and now i have some 4 different servers (and around 100 sites) in different countries hosting it. but that is all. no help from others, no feedback, no any chance that somebody can use it and run it on his machine. not to mention the fact that i have to do all administrative stuff, all the template modificatons, all the "oh my... i lost my password" jobs :) anywho you can check it on http://tamtam.mi2.hr:8888/NoviTam/ and 2 years old version at http://tamtam.mi2.hr/TamTamDev/ -- http://aco.mi2.hr/
On 5/5/05, Jeff Grimmett <grimmtooth@gmail.com> wrote:
Whatever happened to the idea of a twisted Wiki, anyway? That's the sort of thing it's good at.
I was thinking more along the lines of an RSS/blog aggregator on the main site, taking in the developers' (and possibly some users') blogs, and filtering them for relevance. I know I get a ton of value out of reading various Python-, Twisted-, and PyObjC-related posts in the blogosphere (though I rather dislike RSS). There's a different feel to that kind of info; it tends to feel more personally relevant. It's relevant enough for *someone* to write about it, one hopes it's relevant enough for someone else to read :) Wouldn't require [much] new code to get working vs. a wiki, it'd be constantly updated, and once set up, nobody would have to write any more than they already do. J.
On Thu, May 05, 2005 at 12:06:41PM -0700, Jordan Krushen wrote:
Planet Twisted, anyone? http://www.planetplanet.org/ Ralph Meijer runs Planet Jabber, he might have some insights... http://planet.jabber.org/ /psa
On 5/5/05, Peter Saint-Andre <stpeter@jabber.org> wrote:
Planet Twisted, anyone?
Well, see, that needs a link on the Twisted site :) J.
On Thu, May 05, 2005, Jordan Krushen wrote:
The Planet software itself, unless it's had a very radical re-write very recently, does not use Twisted, so there's no reason to link to the software. http://planet.twistedmatrix.com/ does exist (as Itamar has already pointed out) and could probably do with a link from somewhere. -Mary
On Thu, 2005-05-05 at 14:26 -0500, Peter Saint-Andre wrote:
Planet Twisted, anyone?
http://planet.twistedmatrix.com has been up for quite a while :)
On Wednesday 04 May 2005 17:33, Glyph Lefkowitz wrote:
In Zope 3 we simply made it a policy that documentation has to be provided with any code that is checked in. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
On 5/5/05, Glyph Lefkowitz <glyph@divmod.com> wrote:
(This contains my thoughts on Twisted's state in general, this thread, and a response to this particular point from Glyph): As it stands, I'm pretty embarassed about the state of Twisted, and I'm really sympathetic for Mike. Personally, I'm truly sorry I had to drop a lot of polish from the Twisted 2.0 release; I decided to make it a higher priority to get it out the door (its release cycle was nearing Debian scales) than to get API docs building, for example. While there are a lot of docs, and people who claim "Twisted has no docs!" should be stabbed in the face, the level of usefulness in the docs isn't that, well, useful, despite Mary's excellent effort. Not only that, the code is also really nasty in certain areas, trial being particularly relevant, but certainly not the only area. Trial is something I've worried about for a while and have eventually just lost all hope of general usability for, if you're not a Twisted expert. To respond to your question, glyph, the only thing I can imagine helping Twisted a significant amount right now is if some company employing a Twisted hacker would give that Twisted hacker half a day a week to *generally* maintain Twisted. Just those 4.5 focussed hours a week would help an amazing amount, I reckon. Not just working on the web site and docs, but the code as well, like fixing bugs in the tracker. Of course, most of those companies that employ us do pay us for developing certain contributions to Twisted, but that's not really benefitting the project as a whole, it's just benefitting very small niches within it. Unfortunately, all of the companies employing Twisted hackers (Divmod, Nunatak, ITA, and some others) are either too poor or don't care enough about Twisted to make that offer. Barring that, maybe a horde of uni students with tons of time on their hands could help. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
On Thursday 05 May 2005 09:57, Christopher Armstrong wrote:
That's going to be a really hard thing to find. For instance, as far as I'm aware, there's no-one being paid to work on Python - I spend a small amount of work time on it, but certainly nowhere near half a day a week. (I generally take a half day or day off when it comes to cutting a release). If we can't find companies to fund work on Python (a much, much more visible project, with a whole pile of companies relying on it) I don't hold out much chance for someone funding Twisted as a whole. Having said that, I *could* see people being able to justify time on various Twisted subprojects because they need it for work. And yes, trial needs to be dropped off a bridge. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
Anthony Baxter <anthony@interlink.com.au> writes:
I understand that Guido has a certain amount of Python time written into his contract, but much of this is spent attending conferences and so on. Python appears to have more worker bees than Twisted who do the kind of general work radix was asking for, despite none of them being paid to do it; this is probably mostly a function of just being a bigger, more visible project (after all, every Twisted user is a Python user).
Of course, the PSF has a certain amount of money and while none of it has been spent directly on Python so far, some of it has gone on producing training material (for scientists). I personally doubt that the PSF would find funding the production of such material sufficiently in line with its interests but if noone asks, we'll never knows. Cheers, mwh -- <Aardappel> this "I hate c++" is so old <dash> it's as old as C++, yes -- from Twisted.Quotes
On Thu, May 05, 2005, Michael Hudson wrote:
This probably shouldn't be assumed though. If that's all it is, OK, damn. But if there's something about the Python community which is qualitatively different and which encourages the kind of work being discussed, Twisted should consider emulating it. -Mary
One detail: The social scene around Twisted currently features a lot of "verbal violence humor". This is very frequent on the IRC channel, and probably its practitioners don't notice how often it occurs (because shared social customs quickly become unnoticed by their practitioners). In this e-mail thread, someone wrote that people who say Twisted has no docs ought to be stabbed in the face. Obviously, this was intended as humor, but I believe that this social custom, along with as other related social customs in Twistedland, deter people from some other cultures from participating. NB: I'm not saying it's "bad". I'm not saying it's unfunny. I'm not even necessarily saying that you folks should try to change it -- maybe it works for you and the "social exclusion" factor is not a big enough problem to warrant a change. All I'm saying is that verbal violence humor, interpersonal aggression humor, and such customs limit the growth of your project by turning off people from outside your own specific (sub-)culture. Regards, Zooko
Mary Gardiner <mary-twisted@puzzling.org> writes:
Well, maybe it's just luck, or statistics of small numbers. After all, Python doesn't have very many worker bees compared to the long list of people who have checkin rights.
The (maybe just perceived, maybe real) barrier to entry is probably higher for twisted. Maybe you need more simple-to-fix bugs :) Cheers, mwh -- If you give someone Fortran, he has Fortran. If you give someone Lisp, he has any language he pleases. -- Guy L. Steele Jr, quoted by David Rush in comp.lang.scheme.scsh
On 5/4/05, Glyph Lefkowitz <glyph@divmod.com> wrote:
My feeling is that a big part of the problem is that Twisted is so darned obscure in places that most people don't feel up to the task. Other - far, far lesser, of course - projects get enthusiastic help because they are less complex and, well, mystical (for lack of a better word) than Twisted. For example: 1) I create a subclass of an existing Twisted class in order to add some helpful debugging. 2) I run my code but... the debugging code is not being executed! 3) But ... if that's the case, the code shouldn't even RUN! 4) Aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!!!! I had just such an experience. Grumpy, I was. But it illustrates the point: I'm no idiot (last I checked). But I cannot explain how it is possible that the code even worked, and thus I feel completely inadequate to the task of even STARTING to help with documentation. Where should I start? If I can't explain the workings of a ten-line web server, what hope do I have of providing accurate input to documentation? -- Best, Jeff
Itamar Shtull-Trauring wrote:
Oh, I didn't think you did it just to be annoying :) , but now that you bring up the possibility, I wonder ;) . How did you know that *I* would be the one to fall into the trap, that's what I want to know! You are a far more devious trickster than I would have imagined... :) I was just expressing my frustration at losing "mind-share" for Twisted within the company. Just another of those "all things being fine, no big deal, but not the right time to discover this" when people are in the middle of debating dropping the system. I'm not ticked off at you guys, you do what you have time to do. I *am* kind of annoyed at myself for not handling the situation better. I *would* suggest a link on the homepage just saying "documentation". Saying "old documents" implies that there are new ones somewhere. Having that link in the "Developers" menu would be good too, when the new docs are out, update those links to point to them. We would have been fine with the 1.3 docs (it's what we eventually used) for solving the first problem, but seeing "old documents" we had 3 people sitting around suggesting ways to find the new documents relating to the version we were using. Enjoy yourself, Mike ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
On 5/4/05, Itamar Shtull-Trauring <itamar@itamarst.org> wrote:
Do you have any suggestions on how we can improve things? Besides "write more docs" which is basically always a given.
OK, I'll bite at that one. Let me preface it with these two points: 1) On my development team, I'm the QA lead. We've encountered this same issue, with a different face, in the form of the lack of sufficient design documentation and code documentation of changes after the fact. It has caused serious issues down the road, both in software quality and in the quality of testing that QA provided. We've implemented some solutions that appear to work, and it's from that series of experiences that I am speaking. 2) I BELIEVE that this solution has been considered and discarded. I have suspicions why, which I will not speak of, but it's so OBVIOUS that I can't believe that a group of intelligent people would not have considered it. So I fear that I will be speaking to a wall here. So be it. My unsolicited advice: write the docs either before or during development. AFTER development is too late. Most developers have no interest in documentation (it's genetic or something) and want to move on to more "good stuff." It's OK, I understand, as everyone that enjoys coding understands it at a gut level. But the issue needs to be acknowledged before it can be dealt with. The gist of the response I often seen with regards to Twisted documentation is along the lines of "I don't see the issue here - if you don't like the docs, write some." It seems to be a denial of the responsibility to write good documentation. *** I'm not saying that is in fact the case! *** I am saying that there is a perception. As one of my past bosses told me, "perception is everything in some circumstances." Here are the pitfalls (as I see them, YMMV) of retroactively writing documentation: 1) If the person writing the docs is not the person that designed and implemented, the *intent* of the class/module/etc being documented is not clear. 2) If the person writing the docs is not the person that implemented, then the person that implemented must be found, interrogated, and gone back to multiple times to get the docs right. (And if getting it right is not the goal, why bother?) 3) If the intent of the module/class/etc is not known beforehand, how does one determine if it is working right? I don't mean to criticize, but this is in my eyes the greatest weakness of Twisted. My experiences with it, when it works, is that it works wonderfully. I can't commit to using it for anything critical, though, because I spend more time working around such gems as """I am a connector""" than I do in getting anything useful done. I tell you: if it gets to the point where I SERIOUSLY consider writing my own simple frame work to get a job done as a TIME SAVING MEASURE then there is a problem, and I do not believe I am alone in this perception. Take this in the spirit it is offered. I want to use Twisted for what I do, and I want it to succeed. -- Best, Jeff
Jeff Grimmett wrote:
My unsolicited advice: write the docs either before or during development.
dont know why you are so reluctant to say that. It also saves one from doing a lot of unproductive coding when the basic design can be put down on paper and analysed by peers to eliminate bad ideas from the beginning - I suppose that was the original motivation for something like UML - please don't shoot me :-)
That sums it up pretty well.
Twisted is a superb framework. Once I took a few days of and hacked my way through its enigmatic depths I also whispered to myself ... eureka... I'm not sure though that documentation is the *only* problem. My experience tells me that basic concepts around the "networking layer" is intimidating to a lot of developers.
I normally ask myself how long will it take to understand this stuff - given the state of the documentation. Then I compare it to how long it would take to write it myself. Lets be frank - "open source" assumes to an extent that you would actually "read the source" as part of the documentation. To me the detail (like the API) is not the problem. I would love to see high level conceptual documentation to give me a better idea of basic concepts like the "reactor", "application", "service", "transport","protocol", "factory" - what they are supposed to do and how they are to fit together. regards, Eugene Coetzee Web -> www.reedflute.com ===============================================
On 5/6/05, Eugene Coetzee <projects@reedflute.com> wrote:
Maybe maybe not. But right now the documentation is like a lightning rod. No matter what your agenda, if you want to (for whatever reason) discourage against the use of Twisted in a project, all you have to do is point to the documentation and the frazzled bunch of geeks trying to figure it all out :-) My experience tells me that basic concepts around the "networking layer"
is intimidating to a lot of developers.
It would not hurt to have some nice top level documents to address that. That's the sort of thing you'd derive from design documentation. I normally ask myself how long will it take to understand this stuff -
I would have to disagree on that on a fine point. Twisted is not "open source software that happens to be a network framework". It is a "network framework that is also open source software". I didn't start looking at Twisted because it was open source. I started looking at it because I was interested in a better answer than I currently had for networking. AS SUCH ... I expect the API, like any library I use, to be properly documented so I can start using it. I do not wish to investigate the source code as a recreational thing. I have a job to do and I want to get it done on time, on budget, and done well. Making source code delving a prerequisite to use of the software is bad form, in my opinion. Yes, I find having the source code useful. Usually it's because the documentation leaves a fine point unclear; examining the source may resolve the confusion. But the source code should not be a crutch to support incomplete documentation - not if you want to go toe to toe with the big dogs. I would love to see high level conceptual documentation to give me a
:: nod nod nod nod :: -- Best, Jeff
Jeff Grimmett wrote:
Perhaps we disagree radically about the usefulness of having the source code,being able to read it, and the ablility to recompile and rebuild the framework yourself.
But the source code should not be a crutch to support incomplete documentation - not if you want to go toe to toe with the big dogs.
I googled around with "networking framework" a few times in my life. Who are the "big dogs" when it comes to networking frameworks. In the past I wrote everything myself using raw C++ sockets, MFC or vanilla java.net. I have also been bitten badly by the "big dogs" - having to dig through a MSDN "knowledge base" of thousands of "known bugs" before I finally discovered that it *could* be fixed with next release. I honestly cannot say that Twisted is outclassed in this area, at least I can fix the bugs myself. Eugene Coetzee Web -> www.reedflute.com ===============================================
On 5/6/05, Eugene Coetzee <projects@reedflute.com> wrote:
I find it very useful. But isn't that beside the point of the discussion? Is Twisted or is it not a complete product? Why should I not expect the API to be well documented? Open source is not the reason for this. It provides, in this case, a possible way around an obstacle, but at a terrible cost in wasted time, multiplied by every person that uses the library. All those man-hours are lost to some project, somewhere. Twisted should not be considered a time sink. I find that distasteful. It's a wonderful framework that has potential in everything I do these days. It should not have this stigma attached to it.
But the source code should not be a crutch to support incomplete
documentation - not if you want to go toe to toe with the big dogs.
I googled around with "networking framework" a few times in my life. Who
Somewhere in this discussion I mentioned that perception is an important factor here. It matters not if .NET or Java or Ruby have lesser abilities than Twisted, not if potential developers are daunted by the perceived lack of information and an obscure channel towards enlightenment. Heck, I'm reading about "Rails" on PYTHON blogs, fercryinoutloud. -- Best, Jeff
On Fri, May 06, 2005 at 08:49:36PM +0200, Eugene Coetzee wrote: [...]
The glossary helps address this, although I think most people don't realise it's there: http://twistedmatrix.com/projects/core/documentation/howto/glossary.html I'd be interested to know if you think that helps, and whether you knew about it and had read it before. I agree that a nice overview (with a diagram or two) would help. -Andrew.
Andrew Bennetts wrote:
No - I didn't read it before and there is no excuse for that :-) It certainly helps, although it may be a bit minimalistic.
I agree that a nice overview (with a diagram or two) would help.
Certainly. I also find the "howto's" useful in attempting to explaining the design philosphy in a haphazard way - but there are a lot of thing that could better be explained in a more complete, systematic and elaborate overview. regards, Eugene Coetzee Web -> www.reedflute.com ===============================================
In the spirit of just having a go and testing out my twisted knowledge on this list I thought I'd start a questionaire, with answers, inspired by:
I am by no stretch of the imagination qualified to advertise these answers as authoratative. Buyer beware, and if you don't like mine give me a fat -ve 10 for my twisted score and show me your answer ;-) Authoritative intro material is at: http://twistedmatrix.com/projects/core/documentation/howto/basics.html http://twistedmatrix.com/projects/core/documentation/howto/glossary.html Q1. How do you run a twisted program ? In as many ways as there are pebbles on a beach ;-) There are lots of great examples in the docs that get you to the point where you have a python file that will create an application instance at global scope. Once you've got this far the simplest way to run your twisted program is to add the following to that file: from twisted.internet import reactor reactor.run() And then at the command prompt python myapplication.py Once you're comfortable with the basics of Services, the Application, Factory's and Protocols you're going to start caring about things like persistence, daemonization, privilege scheding, plugins etc. At that point it's time to look at 'twistd'. If you remove the lines from myapplication.py that were added above then you can run your application directly from the prompt with: twistd -oy myapplication.py Then you will get the almost the same result. The primary difference is that your now running your program as a daemon. twistd is a program that loads twisted applications from many different formats, including .py, and starts the application instance it finds there. Q2. What is an application ? Most twisted programs begin with an application instance. Twisted defines a helper function for creating an application instance so that it includes all the bits of functionality that allow it to be driven by the twisted framework. This function is found in the module twisted.application.service and is called 'Application'. Most twisted programs define a python file that makes a call to the 'Application' function at global scope. Just doing this, and nothing else, will not get you very interesting results, for that you need to look at adding 'Services'. If you've already got a python file that defines an application instance this way - in addition to some example code - and you just want to know the simplest way to run your program, then the answer you're after is almost certainly: # add these lines to myprogram.py from twisted.internet import reactor reactor.run() $ python myprogram.py But see Q1. for a little more on this. This is the most suitable way if you want to single step through your program in a debugger to find out how twisted works. There are many other ways to run your program but it's safe to ignore them until you decide you want them. Q3. What's a Service ? 'Services' collect together the bits that make it possible for cool networking stuff to happen in a single thread without blocking[1]. You don't have to use them to use the twisted framework but they can help. If you are using an application instance then you almost certainly want to use Services as well. You add Service's to your application instance (see Q2 for creating one). Each service typically manages a discrete network service. The twisted framework starts and stops all Services in an application in a well defined way when your program is executed. This includes provision for giving your services a chance to do stuff before privilege scheding happens. To define what your service is you will usually want to subclass twisted.application.service.Service and override any, or all of: privilegedStartService, startService, stopService. During a single run of your program each of these methods is called exactly once and in exactly that order. If you need to bind to privileged ports then privilegedStartService is where you need to do it. The bits that are used to piece together Service behavior are principally: Factory's, Protocols, and Defereds. The parameters involved in binding these bits together are things like hostnames, portnumbers, database passwords, and (arguably) the networking protocol the service is dealing with. Collecting these things together into a service can be done by hand, often, it can be easier, more flexible, and much more extensible if Adapters are used to do the binding. [1] Note carefully the word 'possible' in relation to 'without blocking'. The twisted framework can not guarantee your program will not block. Twisted just gives you a set of concurrency primitives to help you write your program so it doesn't block. To understand how to write a twisted program so it never blocks you need to learn to love 'Defered's. Concurrency primitives in python essentially boil down to a choice between callbacks or generators. The twisted framework supports both (as of 2.0) but the callback style is the most established. Generators didn't arrive in python until 2.2. Q4. What's a MultiService ? At some point you will almost certainly decide that subclassing MultiService is going to help you. It's not. To quote moshez: "duuuuuude. don't. inherit. multiservice." [And certainly don't inherit from both MultiService and Componentized at the same time ;-)] MultiService is an internal class that is used by the framework to manage the order in which all services, added to the application instance by you, are started and stoped. The fraze "added to the application instance" is potentially misleading but from the birds eye perspective this is essentially what you do when you add Services. You almost certainly do want to subclass Service. But don't be fooled into subclassing MultiService. Q4 What's a Factory ? Q5 What's a Protocol ? Q6 How do I get my Factory to open a connection ? Q7. How do I get my Factory to serve a connection ? Q8. How do I cancel my attempt to open a connection ? Q9. What's a plugin ? Q10. Ok, I'm feeling brave, what is an Application realy and why can't it be subclassable ? You can't subclass it because it is a function. But that doesn't tell you anything useful. The value returned by the function twisted.application.service.Appliction is an instance of twisted.python.components.Componentized. Componentized is a class that provides interface aggregation facilities in a persistable way. When you call Application you are creating an instance of Componentized that aggregates instances of the following together: * twisted.application.service.MultiService * twisted.application.service.Process * twisted.persisted.sob.Persitent The twisted framework uses interfaces and adapters to reduce the tendency for functional dependencies to snarl up the implementation of the framework. To do this it needs your application to support interface aggregation and to provide the basic interfaces that enable it to drive your program the Twisted way. In these terms the function Application produces an instance that supports the interfaces IService, IMultiService, IPersitable and IProcess. MultiService implements IService and IMultiService[1], Persistent implements IPersistable, and Process implements IProcess. These are the basic guarantees about the application instance that enable the twisted framework to drive your application. You can't specialize of the above instances either - unless you are prepared to mimic the behavior of Application. You're only route for specializing your application is by adding services. ie., via some variation of: IService(MySerivce()).setServiceParent(IMultiService(application)) It's not unusual or unreasonable to write a replacement for Application. A perfectly acceptable example is: def MyApplication(): a = MyService() a.setName("myapplication") ret = service.MultiService() a.setServiceParent(ret) return ret Reading up on adapters & interfaces in the python / Zope3 sense can give you an extra perspective when looking at the twisted docs on Adapters and interfaces. Twisted 2.0 actually uses Zope3 interfaces. And if you're reading this answer then there is a good chance taking the time to do this will help you. http://www.python.org/peps/pep-0246.html http://mail.zope.org/pipermail/zope3-dev/2005-January/013064.html [1]It is required that the IService interface and the IMultiService interface are implemented by the same underlying object (or at least it is by the application unit tests). You really probably don't want to read any more of this answer. For one thing Componentized looks like it's going to be superseded by Faceted which does similar things but much much better. From the doc string of Componentized: "I am a mixin to allow you to be adapted in various ways persistently." Componentized caches addaptions in a persitable way. Your applications root service, the MultiService instance above is effectively a persistably cached adaption from None to IMultiService and to IService. Well you did ask ;-> When you do: IService(application) You are retreiving the IService interface for you application that happens to be implemented by twisted.application.service.MultiService Depending on the feedback I'm keen to see/provide answers to Q4-Q8, Don't feel confident at all about Q9. Cheers, Robin Mary Gardiner wrote:
Mary Gardiner wrote:
I will much rather review http://twistedmatrix.com/documents/current/howto/tutorial/intro and http://twistedmatrix.com/documents/current/howto/tutorial/factory for you. I have a few issues with some some of the information in there.
Also, any suggestions on how to make the glossary easier to find?
I would suggest linking to http://twistedmatrix.com/documents/current/howto/glossary from http://twistedmatrix.com/documents/ regards, Eugene Coetzee -- -- =============================================== Reedflute Software Solutions Telephone -> +27 18 293 3236 General information -> info@reedflute.com Project information -> projects@reedflute.com Web -> www.reedflute.com ===============================================
On Wednesday 04 May 2005 20:40, Jeff Grimmett wrote:
My unsolicited advice: write the docs either before or during development.
That's exactely what we do in Zope 3 and I have been advocating this in all my mails with concrete suggestions. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
On Wednesday 04 May 2005 16:36, Itamar Shtull-Trauring wrote:
When we developed Zope 3, we put **huge** emphasis on documentation. As mentioned in the other mail, we made it a policy. Here are the things we did: - We (mainly Jim) used a slide show and later plain text files to document how we imagined a feature to work. Basically we wrote Sci-Fi stories. Thus documentation was available before code. - We also used a proposal process for larger features. This had a similar effect to the Sci-Fi presentation in that it documented the API before implementation. Many proposal authors made a point to update the proposal after the implementation, so that they would reflect the latest API. - We decided that Interfaces would be used for API documentation and public interfaces would always be found in the package's interfaces.py module, so people would always know where to go. - We later developed a very custom (on purpose) API doc tool that would not only document interfaces, but their interaction with the system. For every interface you can see its adapters and views, or which utility provides the inspected interface. The doc tool was later enhanced to also provide documentation for any object, ZCML and plain text files. - First we used regular unit tests for testing documentation, but eventually moved to file-based doctests, since they fulfill both XP requirements for tests: testing and documenting. I can't overstress the success. Every recently created or refactored package has now a README.txt file clearly documenting the API and it is never outdated, because it is part of the test suite. - We had two people write a book and find a publisher. My book is even semi-freely available as one can use it for non-commercial purposes. So, what's next? This E-mail is not suppose to show what's Zope 3 did better than Twisted or vice versa, but demonstrate some concrete things that can be done to improve the documentation situation: - I strongly suggest that Twisted starts using file-based doctests. I have demonstrated how this can be done with the current trial test runner in the following patch submission: http://twistedmatrix.com/bugs/issue1000 Also, I think twisted needs to start distinguishing between unit and functional tests. - Develop a tool that clearly shows the available adapters for a given interface. epyrun will not pick this up, so it is important. - Maybe it would be good to have a documentation sprint; since most (a lot) of the Twisted developers are in Boston, it would be good to do it here; I would be willing to come and help people getting started with writing file-based doctests. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
On May 4, 2005, at 4:08 PM, Mike C. Fletcher wrote:
That's why it's better if you don't install Twisted, just make its source code part of your project. That's what you should be doing for just about any dependency, especially those where you don't need extension modules.
Debugging with an audience? What the hell were you thinking? Don't do that.
The deferred/generator stuff is probably "plain-old-looking" enough to be used by a programmer who isn't already used to the Twisted style of programming, with caveats of course. -bob
Bob Ippolito wrote:
Could anyone offer advice on the best way to do this? For example, the setup.py of our twisted app creates a package hierarchy: - agdevicecontrol - agdevicecontrol.server - agdevicecontrol.client - agdevicecontrol.gui - agdevicecontrol.common - agdeviceconttrol.devices This is all under subversion control and (inspired by Twisted) a few hours away from having its tests run under Buildbot. So now if we want to include Twisted2.0 and the ZopeInterfaces dependencies in our source release, should we just take a source snapshot and add: - agdevicecontrol.twisted - agdevicecontrol.zopeinterfaces And do we try to keep these in synch with the respective CVS/SVN repositories? Or since this is going out to end users, stick with stable releases (almost rhetorical question). As always, your advice is much appreciated. Cheers, Darran. -- Darran Edmundson (darran.edmundson@anu.edu.au) ANU Supercomputer Facility Vizlab Australian National University, Canberra, ACT 2600 tel: +61 2 6125-0517 fax: +61 2 6125-5088
On May 4, 2005, at 5:42 PM, Darran Edmundson wrote:
Don't try and change the Python module hierarchy. Just make twisted a sibling of your package. zopeinterfaces is a bit different because it *does* have extension module(s), and doesn't evolve very quickly, so it might be best to just require that to be installed. If you only support one platform you could install a particular version of zope.interfaces with a --prefix that points somewhere in your repository (such that it's a sibling of your code) and just check that in. Or you can go with something a bit more complicated with multiple platforms where you have a bootstrap script that mangles the sys.path so that it finds the appropriate platform specific modules. This is a "__bootstrap__" module that I've used in the past that I have used before: import os, sys # change to current dir curdir = os.path.dirname(os.path.abspath(__file__)) os.chdir(curdir) # add dependencies and platform specific path for dependencies sitepkg = os.path.join(curdir, 'site-packages') uname = os.uname() platdir = '%s-%s' % (uname[0].replace(' ', '_'), uname[-1].replace(' ', '_')) sys.path[1:1] = [ sitepkg, os.path.join(sitepkg, platdir), os.path.join(sitepkg, platdir, 'PIL'), ] In this particular case, the layout looked like this: site-packages/ formless/ nevow/ twisted/ ... these three are just checkouts of their respective projects at some point in time plat-Darwin-Power_Macintosh/ mx/ PIL/ sqlite/ _sqlite.so plat-Linux-i686/ ... same stuff as in Darwin The way it bootstraps is that I had a "start.sh" script that calls "./ twistd -o -y something.tac", where twistd is just a local copy of the twistd shell script that is modified to import __bootstrap__ before it does anything else. If you didn't notice, zope.interfaces isn't here.. because this is a copy of Twisted 1.3. nevow worked better that way at the time. It still works just fine, and I don't have to worry about other applications on the server getting in the way because the environment is pretty isolated. I can svn co the project on (almost) any Mac OS X or Linux machine with a Python 2.3 interpreter and just run it, without worrying about any other dependencies.
That's up to you. You might want to map external code (even specific revisions of external code) as just svn:externals, but since you don't have control over other people's repositories (i.e. they might be down when you need to checkout) it's probably a good idea to "checkpoint" specific releases and make sure they live in your repo somewhere. -bob
Bob Ippolito wrote: ...
Hmm, can't see that flying with our sysadmins, they want standard packages with minimal apps that use them (NetBSD developer).
Debugging with an audience? What the hell were you thinking? Don't do that.
Pair programming *always* has an audience, in this particular case we just had the third programmer looking over our shoulder because we'd just been debating the merits of Twisted and he wanted to see how it compared to his own experiences with Twisted. Going back to my "cubicle" (actually apartment) would have involved making the pair programming remote (which is common in our company), the difference being that I could do the looking up/spelunking *before* I called over to start work ;) .
Quite possibly it would be, but I'm afraid the cow is already in the corn for this project. We'll see if they are won over by the beauty and majesty of the big Twisted project when/if they start working on that. Have fun, Mike ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
On May 4, 2005, at 5:54 PM, Mike C. Fletcher wrote:
Sysadmins be damned, the more self-contained an application is, the less trouble you'll have debugging, developing, and upgrading it. ESPECIALLY since Python can't reasonably do multiple-concurrent- versions (effectively our own DLL hell).
The way you said it kind of implied that more than 2 people were present, and that these were non-programmers. -bob
Bob Ippolito wrote:
Debugging with an audience? What the hell were you thinking? Don't do that.
Yeah- debugging is never pretty no matter if you are using Visual Basic "idiot proof" wizard tools or assembler written in encrypted sanskrit. What kind of IT outfit torture their developers by forcing them to debug "live" with the sword of managerial scrutiny hanging over their heads :-) For me the moral of the story is that frameworks tend to make simple tasks complex and complex tasks simple. Perhaps Twisted wasn't the best choice to start with? Web -> www.reedflute.com ===============================================
Bob Ippolito wrote:
Eww. -1 on that. As a Debian person, I tend to spend a lot of time dekludging things where upstream includes 5 libraries with the project, and will not use system-installed libraries. Not to speak of the bloat in the size of the source tarballs, which is multiplied by a _lot_ when it comes to Debian.
On 6-mei-2005, at 12:08, Sergio Trejo wrote:
When I install software in a production environment I prefer self- contained installations. That increases diskusage, but takes away some headages: what if server1 requires Twisted 1.3 and doesn't work with Twisted 2.0 and server2 requires Twisted 2.0? Ronald
On May 4, 2005, at 1:08 PM, Mike C. Fletcher wrote:
Did you ever figure out why? I have just run into similar behaviour myself after upgrading to 2.0. Here's an example that hangs for me on Mac OS X (10.4) after the final print: --------------- from twisted.internet import reactor, defer, threads from twisted.trial.util import deferredResult from twisted.web.client import getPage import time def itFired(result): print "Deferred fired: %s" % result return result d = getPage("http://www.example.com/").addBoth(itFired) print "Returned: %s" % deferredResult(d) print "Done; exiting" --------------- So far as I can tell, the python threading module has an atexit handler that's hanging. Also, if I replace the deferToThread() with a twisted.web.client.getPage(), I still get the same hang. --Grant Grant Baillie Open Source Applications Foundation
On May 5, 2005, at 11:49 AM, Grant Baillie wrote:
Er, I'd already replaced the deferToThread() in an earlier version of that file with twisted.web.client.getPage(), for anyone following along at home. I also found a hackaround, which is to insert the following before the call to getPage: #--------------- import threading # threading seems to install an atexit() handler that # can wedge if you've run the reactor. So, we import it # first.... import atexit def runAndStop(): if not reactor.running: reactor.callLater(0, reactor.stop) reactor.run() # ... then our handler will be called before threading's. atexit.register(runAndStop) #--------------- I'm not sure if this is a problem in python, in twisted's use of threading, or in trial.deferredResult, though. --Grant Grant Baillie Open Source Applications Foundation
On May 5, 2005, at 3:05 PM, Grant Baillie wrote:
Almost definitely a combination of 2 and 3, with 3 being the one really at fault. Trial is horribly, horribly broken by design and it's really just an accident that it works at all. The expectations a lot of the tests have about the reactor are also broken, but that's mostly just a consequence of trial sucking. -bob
On May 5, 2005, at 12:26 PM, Bob Ippolito wrote:
Thanks for the info. I'll go ahead and file a bug, unless someone tells me not to. FWIW, I'm happy with my workaround (I'm really only using it to keep my doc/unittests relatively uncomplicated). --Grant Grant Baillie Open Source Applications Foundation http://www.osafoundation.org
On Thu, 5 May 2005 11:49:20 -0700, Grant Baillie <grant@osafoundation.org> wrote:
So as to be entirely clear, I'm going to limit this response to just one idea. If more details are desired, I can post a followup later (as long as someone asks for one). Anyway, for now, here's the meat of this post: Do _____not_____ use deferredResult (or deferredError or wait) As someone who pushed for their existence, I apologize. Jp
Jp Calderone <exarkun@divmod.com> writes:
Just to be clear, IMHO, I'd qualify this by saying something like in final production code or non-test code or something. We make heavy use of these functions in our unit tests and they are invaluable ways to simplify the structure of such tests when each test is the only thing going on and there's no problem iterating the reactor beneath a blocking call. We've also cheated and used them for pure startup code in GUI apps that are quick 'n dirty example interfaces into our main Twisted system, and again they can simplify such efforts. But I do agree they really have no place in final production code. -- David
On May 6, 2005, at 5:11 PM, David Bolen wrote:
Sorry, but Jp is right. They have no place in any code. Their existence is a mistake and should be corrected ASAP. They severely violate the reactor interface. Something almost as easy would probably be waitDeferred and deferredGenerator in twisted.internet.defer (I don't know when they went in, but they're probably in 2.0). See the waitDeferred's doc string for information. The only way they could be "fixed" is if their implementation did something more like this: def deferredResult(d): """untested because this code should not exist""" res = [] def cb(result): res.append(result) reactor.stop() d.addBoth(cb) reactor.run() res = res.pop() if isinstance(res, Failure): failure.raiseException() return res However, if their implementation did that, then the tests would probably break (because the tests are WRONG) or be a hell of a lot slower and even less reliable because there is no more timeout. -bob
Bob Ippolito <bob@redivi.com> writes: with respect to deferredResult/deferredError:
Hmm, then perhaps it's the reactor interface that could use some improving rather than removing this high level functionality? Or is everyone just saying that the implementation (versus the concept) of the current functions is done wrong. Or maybe I'm just missing what major mess having this sort of interface is exposing. Given Glyph's recent response, I might also clarify that while our heaviest use of these functions are within unit tests, we aren't using trial, so any linkage between poor behavior of these implementations and trial is not something I'm referring to or worry about breaking. What I do think is that there is a very good and practical reason to be able to unwind deferrable operations into a blocking structure for a wide variety of test conditions, and losing that capability would make writing tested Twisted code (at least for our situation) much more fragile and error prone (the tests themselves). And that's not quite the same as the newer waitForDeferred stuff in 2.0 that integrates with generators. While slick, it still doesn't work as well for easily maintainable tests in many of our situations. If there's a problem with the current implementation, or with how the current implementation may abuse the public reactor interface, then I'm all for cleaning it up, fixing it or making it part of the reactor itself in some way, as long as the public mechanism remains. I mean, after all, at some point the reactor itself really is just sitting there iterating itself, so I find little reason not to permit some access to hook into that mechanism when appropriate. For example, in our system we may very heavy use of our own components all of which implement their entire API via deferrable interfaces. Interacting with multiple such components during a test is thus a multi-stage deferrable process. Having these blocking calls permits a test method to say something like: self.user = deferredResult(self.umgr.user()) self.user.email = 'test@dom.ain' deferredResult(self.umgr.saveUser(self.user)) in order to test being able to save a new user object into the system. Part of the test is that the calls themselves should succeed, so the natural result of deferredResult raising an exception on an errback is just right. And while clearly readable, the test still tests that the deferrable aspect of the interfaces will work properly (e.g, with deferredResult generating the exception if it doesn't get passed a deferred). Sure, I could write the user() and saveUser() calls with a callback chain, and then I'd have to nest the call to saveUser based on the user callback, probably something like: def fail(failure): self.fail(failure) def gotUser(user): self.user = user self.user.email = 'test@dom.ain' return self.umgr.saveUser(self.user) d = self.umgr.user() d.addCallback(gotUser) d.addErrback(fail) but I see no advantage (*in the context of such a test*) to doing it this way, and I see a real loss of clarity and maintainability for the test itself. Since each component in our system has a deferrable interface, any test which is going to interact with multiple components would quickly evolve a fairly deep callback system in the test, without much benefit that I can see. I'm all for fixing an implementation or cleaning up an interface if it's being misused, but I certainly find value in the particular external interface that deferred{Result,Error} deliver - so much so that we would (and in fact did) implement them locally before finding that they already existed. -- David
On May 6, 2005, at 6:51 PM, David Bolen wrote:
The reactor interface is not what needs improving, it's the way that these functions work. They iterate the reactor when the reactor is in a stopped state. The reactor should either be running, or not running. There are basically two fundamental issues: (1) Reactors can only be (meaningfully/predictably/etc) iterated if Twisted rules the universe AND the implementation of that reactor is amenable to that feature. This is not a tautology. (2) Reactors need to fire various startup/shutdown events. Reactors shouldn't be doing ANYTHING unless they are in a running state. The current deferredResult/deferredError breaks both of these conditions. (1) It iterates the reactor (which is a historically public, but conceptually broken interface) (2) It iterates the reactor in a STOPPED state. The reactor is never "running" during these tests. Startup/Shutdown does not happen! The fact that the doc strings talk about re-entrancy of certain reactor functions and whatnot scares the shit out of me. They should not have to be re-entrant. What their current implementation is doing is really really broken. The problem, for tests, is that using a properly written deferredResult / deferredError the reactor would startup/shutdown violently throughout the course of a single test, and will break the hell out of it if it has anything to do with services/etc. So, while for SOME deferreds it would work fine, but for others, it wouldn't. Basically, unless you can encapsulate the entire test in a single deferred, then it shouldn't be using deferredResult/deferredError. Therefore, what SHOULD happen is that trial should let you write tests that return deferreds, and it should let you write it in the deferredGenerator style (so it doesn't suck so much). Trial *could*, in theory, put the reactor into a "started" state at the beginning of every tests and a "stopped" state at the end, but then you're testing in a strange environment that doesn't really mimic how Twisted actually works, and it still breaks (1) which makes it unsuitable for testing the reactors where Twisted does not rule the universe. Testing this theory would require changes to the reactor interface to make some parts of if re-entrant (bleargh).
Well, it causes you to write two lines of code instead of one. That sucks, but so what? You don't have to write big callback chains.. It's conceptually identical, it's just that you have to throw in some extra boiler plate that says THIS IS A DEFERRED. If the way we say that wasn't so ugly, it might actually be a good thing from a readability standpoint :) I'm not *entirely* sure why "waitForDeferred" is really necessary at all, though "deferredGenerator" certainly is. In the interest of not- wearing-off-your-fingerprints one might write a "deferredGenerator" specifically for trial (or maybe in general) that just makes sure you're yielding a Deferred. I *think* that it does this so that it can assume the last thing you yield is the result. I would say that you should wrap the result instead of wrap every intermediate thing, since the intermediate things you do in such a deferred generator should far outnumber the times that you return from it. Perhaps whoever wrote it had a good reason that I just don't understand, but when I wrote something equivalent a few years ago I wrote it such that there wasn't quite so much boilerplate.
# let's assume I've aliased waitForDeferred to "wait" d = waitForDeferred(self.umgr.user()) yield d self.user = d.getResult() self.user.email = 'test@dom.ain' d = waitForDeferred(self.umgr.saveUser(self.user)) yield d d.getResult() So, the test is 6 lines instead of 3. Which sucks, but it's correct. For tests that do more stuff, it probably should be even less of a problem. If Python grows an extended iteration protocol, where yield becomes an expression, you could make it 3 lines again. This might actually happen in Python 2.5.
Well, let's say your database thing is a service, that maintains some kind of ephemeral state that's required in some way. If deferredResult were properly written, this ephemeral state would hit the bit bucket on each deferredResult, probably breaking your code even though the test are "correct". If you add such a component to the system, you might have to rewrite all of your tests, because it would not be possible to fix the way deferredResult works.
So, use deferredGenerator / waitForDeferred. -bob
Bob Ippolito <bob@redivi.com> writes:
This seems like an internal implementation issue to me - when a reactor is "running" (I've called run()), it's basically stuck in a loop doing runUntilCurrent and then doIteration. That's precisely what iterate does. Now, I agree that if you have tests that never initiate the run, that you skip over some startup events. But except for the uppermost tests, what is being tested is a specific unit test for a piece of the system that should be testable in isolation. If the item under test has a dependency on startup events, the test should arrange for them.
Not sure which doc strings those are, but I don't see a re-entrancy problem in the sort of scenario I'm looking at - rather than: reactor.run which calls reactor.runUntilCurrent/doIteration which causes my deferrable stuff to operate I have deferredResult which calls reactor.iterate which uses reactor.runUntilCurrent/doIteration which causes my deferrable stuff to operate I don't think there's any more room for re-entrancy issues than in a normal running reactor. Maybe this is a key difference with trial? If trial has a top level reactor.run that is always above any test, then I do see how you could get re-entrancy problems, since you'd be re-entering reactor.iterate from within an existing reactor.iterate call. I guess that's true of any nested use of deferredResult too, but we don't nest our deferredResult calls - no real need since any deferrable is directly wrapped with the deferredResult call, and deferredResult is only used in the tests, so what they are calling is always production code that is written properly with callbacks and what not.
I still don't necessarily see that (the last sentence). We use multiple deferrable operations in single tests, but never more than one at a time (e.g., no recursive or nested uses). But certainly more than a single deferrable operation within a single test. I would, however, agree that I'd prefer even more the ability to completely start/stop a reactor during the course of a test, but would still like to be able to iterate it manually during the test to provide a natural blocking flow to the test. But in my current scenario, the majority of my components under test are having interfaces tested that are not impacted by startup/shutdown (and we don't use any services, in the Twisted sense, for example).
I don't think I'd disagree with that - we pretty much stayed away from trial since I wasn't comfortable with it initially, so we're a pure Unittest approach, which certainly doesn't provide any framework for something like that.
Yeah, that's a tough problem, although one that would also simplify the fact that we often run the tests under the unittest GUI, and occasionally have to fight cross-test pollution from the reactor persisting across test runs, which is a wart from the current restrictions.
Well, seeing "deferredResult" sort of already says "THIS IS A DEFERRED" in the test. And it's less the number of lines than the inversion of the logic. In my example, it's tougher to read through that test and see the underlying core test of the user() and saveUser() calls. This is true really of any deferred code in general, so I find the ability to simplify things in the test level an improvement. As you say, the way we "say" this currently is ugly.
Except that you didn't include the additional code I'd now have to write to actually turn that into a generator which is iterated over by the test in order to actually process the deferrable yields. (Unless this is something trial does automatically somehow). That's what I meant by saying that the integration into generators is slick (and goes a good way to linearizing what is normally a callback chain), but still isn't quite as simple as the interface provided by the deferred{Result,Error} functions.
I might be getting lost on the "properly written" part, but if I were testing a component that did have state triggered during reactor startup/shutdown (which is what I think you're referring to), that test would likely be using direct calls to the component to trigger the startup/shutdown actions as part of the test setup/teardown, but without using the reactor. That's because the test would be focused on the component and not on Twisted itself (which in the context of such a test would be a "system" component that I'm trusting would do the right thing). This is clearly different than when testing Twisted itself. Now that would cover the lion's share of the component tests but somewhere there's the question of who plugs the components interface into the twisted events and is that done right. And I agree that's messy right now, but it's such a small bit of initialization code that even if it doesn't get full coverage in the tests it's not hard to validate. But I'd certainly welcome ways to test that more fully. So my tests should be fine for what they are testing (the component).
I'm not following this.
So, use deferredGenerator / waitForDeferred.
Once we move to 2.0 we might consider that (I'm not quite up for back-porting into 1.3 that we're using now), but I still think that would complicate the tests (in terms of the overhead to actually run each test like a generator) - but different people can have different views on what is more maintainable. Since we're not running our entire test bed under control of a single "reactor.run" call (I presume that's more of what trial does?), even using the generator would effectively be iterating the reactor somewhere along the line. I don't know - maybe my use case is just limited enough (non trial, no nesting, etc...) that I don't see any true exposures through deferred{Result,Error} while I'm getting benefits. -- David
On May 6, 2005, at 8:19 PM, David Bolen wrote:
This is an implementation detail. runUntilCurrent, doIteration, and iterate should not be public API.
The reactor itself needs startup/shutdown events. As you've noticed, the thread pool depends on them.
We all want things that aren't really possible to do :)
However if some implementation detail of twisted changes to take advantage of reactor startup/shutdown events internally then all of your tests would break even though the application would work. Having tests that fail in theory but work in practice is uh, bad.
Staying away from trial isn't really a bad idea. It's obviously broken. However, bringing the broken functionality from trial back into unittest doesn't make it any better :)
Trial could and should (but doesn't currently) do that automatically if your test function returns an iterator instead of None.
Using Twisted isn't as simple as using urllib to suck down a web page, and using ascii strings is easier than using unicode...
That's assuming you actually know about all of the services necessary to make something work, and that these services can startup/shutdown properly in this manner. The more special crap you write in a test the less useful the test is because it's testing something in an entirely different way than it actually works.
Move to 2.0 ASAP. Don't backport to 1.3.
Trial doesn't use reactor.run at all, that's why it's broken. Using the generator would be iterating the reactor but it would be not broken. A good implementation of test would start and stop the reactor at the beginning and end of every test so that you (probably) don't end up with side-effects due to the order that the tests run.
You're right, some things don't break in horrible ways when you use deferred{Result,Error}. Sometimes you can concatenate a str and a unicode and it Just Works too, but it only works if the str contains characters that can be decoded by the default encoding, whatever that happens to be at the time :) -bob
On 06 May 2005 20:19:45 -0400, David Bolen <db3l@fitlinxx.com> wrote:
It's an implementation detail, yes, but not an internal one. For the reactor to operate properly, it *must* be started up and shut down. That's simply a requirement of the interface. It's not even a particularly unusual one: many libraries require you call some initialization function before proceeding to other APIs they provide. With recent versions of Twisted, many reactors don't actually depend terribly heavily on being started up, so you may be able to call iterate() without calling run() first and see something resembling correct behavior. However, you should note that future versions of Twisted may introduce new startup requirements which break programs which assume reactors do not need an explicit startup event. Additionally, third-party reactors may depend on the startup event now, thus breaking when used with code that only uses iterate(). The shutdown event is much more important these days. People who have noticed Twisted programs and tests which hang at shutdown can attest to this.
This is one of the cases, yes.
Your application code all deals with Deferred callbacks. Why is it a stretch to have your test code do the same?
That you know of. With current releases of Twisted.
This is one reason you should use trial, even though it is broken. Trial will be fixed by Twisted developers who (at least occassionally) have a firm grasp of how it can correctly interact with the reactor, so by using it you benefit from this understanding. By using unittest, you are forced to get all the things right that trial will someday get right, or accept a broken test harness. If you have feature requirements that trial does not satisfy, make some feature requests :) We are quite aware that trial is not yet feature complete and will welcome suggestions for features that make it a more useful tool.
I don't think anyone is trying to suggest that it is simpler. However, it has the advantage of being correct. Given the choice between simplicity and correctness, I am inclined towards correctness.
You can definitely do this. The problem is that the reactor may have internal setup and tear down of which you are not aware and which will cause problems if skipped.
It may be the case that none of the bugs in trial synchronous result utilities cause problems for you. If they don't, I can understand how it would not be a big priority for you to move away from them (after all, I'm sure you have plenty of things that *are* causing you problems now that you would much rather spend time on). I recommend simply keeping this conversation in mind if and when you run into problems with your use of deferredResult and deferredError in the future and that you not let such problems take you by surprise. You can rest relatively easily in the knowledge that deferredResult and friends will not disappear from Twisted overnight. Broken as they are, they are clearly part of a public API and so will remain long enough to satisfy backwards compatibility requirements. Jp
Jp Calderone <exarkun@divmod.com> writes: (...)
Seems like as good a way as any to end this thread for me, since I definitely agree with the above. And yes, I'm aware (or believe I am) of the risks I may be incurring by using these methods, as well as using unittest rather than trial, and will certainly continue to follow Twisted actively to hopefully catch changes that may impact my current choices. -- David
On May 11, 2005, at 4:28 PM, David Bolen wrote:
I'll chime in with a "me, too" here. It might also be nice to add some kind of warning to the __doc__ strings for the 3 methods in question. There are caveats in wait()'s doc, but they weren't sufficient to scare me away, for one :). --Grant Grant Baillie Open Source Applications Foundation http://www.osafoundation.org
Grant Baillie wrote:
I apologize for the lack of enthusiasm of those warnings. The unfortunate problem is that currently _all_ of trial is subtly broken, so there is nothing that we could suggest as a decent alternative, and thus can only deprecate so hard :-(. The recent rewrite at least allows for returning Deferreds, which is what the future public API will eventually look like, but all it does when you return a Deferred is call wait() on it. (JP got me up to speed today on some bugs that are unique to returning Deferreds in this way, actually...)
On 06 May 2005 20:19:45 -0400, David Bolen <db3l@fitlinxx.com> wrote:
Uhhh, maybe I'm missing something here, but as I understand it (I probably should, being the guy who wrote deferredgenerators), that code is: testCase = deferredGenerator(testCase) after the definition of the method, or even: @deferredGenerator before the definition of the method. HTH -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
Christopher Armstrong <radeex@gmail.com> writes:
But then who actually performs the iteration over the test case to trigger the generator? Clearly some top level code needs to invoke the reactor.run() and at some point iterate over testCase while beneath a running reactor. We're just using unittest, so something would need to be added to serve as the basic framework for that sort of test. I'm guessing from your uncertainty that trial - which I haven't kept up with changes for 2.0 - probably provides a mode to do this automatically, so if I were using trial you'd be correct and it wouldn't be that much more work to integrate the new deferred generator support into the tests. Note that independent of use in tests, I'm definitely intrigued by these techniques (as I was in the early flow support in Twisted 1.x) so when we start the migration to 2.0 we'll probably look at incorporating this approach into the code base when reasonable. -- David
On 11 May 2005 19:34:15 -0400, David Bolen <db3l@fitlinxx.com> wrote:
deferredGenerator does the iteration. This is how defgen works: you define a generator. It should yield waitForDeferred instances. Then you wrap that generator function with deferredGenerator. deferredGenerator wraps that function such that, when called, returns a Deferred (not a generator object!). The original generator will be exhausted (doing the magic wait-for-deferred stuff along the way) and the Deferred returned from your function will result in the last item yielded from the generator. Since trial supports test methods returning deferreds, this Just Works. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
Hey Mike, this thread has been given a thrashing and should probably be let go off, but it hits a pretty emotional issue for me - so here goes my 2c's worth :) I've been in similiar situations before, always with selling (evangelising) python though - I dream of the day I can be selling twisted over vanilla python :) Mike C. Fletcher wrote:
+ Even the Finger sample is mystifyingly complex if you jump into it at the end of the process
certainly - that's why there is the Finger sample - your lead programmer never took 2 hours to work through the Finger sample? I often find this with python - me: 'these problems we're having - they go away if you use python' reluctant workplace: 'but we don't have the skills in house to support python' me: 'python's has be designed to be useful as an educational language - i've found programmers can be at least as productive as they are currently using python within 3 days. here are some great tutorial links.' reluctant workplace: 'we'll see' me: 'these problems we're having - they go away if you use python' reluctant workplace: 'but we don't have the skills in house to support python' me: 'did you even open one of those tutorial links i sent you?' reluctant workplace: 'no'
i've found it a losing battle trying to spoon feed even bright people who aren't eager to try out new and interesting ways of solving problems.
Not having a go at all you at all - but I don't believe there is a solution on earth, paid for, or volunteer, you would have come out ok in this situation. It sounds like your company had an axe to grind. I can't think of many paid support contracts I've had access to that could even get a problem being worked on within 20mins - let alone solved.
possibly, twisted can't always be the right hammer. vanilla python is still brilliant. just hoping your company hasn't shut the door for when twisted could be needed. good luck in the future, for both of us:) Andy.
Itamar Shtull-Trauring wrote:
On Wed, 2005-05-04 at 16:08 -0400, Mike C. Fletcher wrote:
Mike, I'm sorry that Twisted was a part of such a horrible experience for you! Thanks for sharing the experience, but evangelism tip #1 that I get from the initial reading is from the real-time, public bit at the end. - Do not ever, ever use IRC in a public forum to solve a problem in real-time. In fact, do not ever use any support mechanism in a public forum unless you or your company has a paid relationship with the people on the other end of the wire. - Do not ever expect to be able to fix a threading bug in less than 2 days.
Glyph Lefkowitz wrote: ...
Wasn't a truly horrible experience. The company's fairly laid back, it was just disappointing.
Um, I hope you're joking. Do not *ever* use a public support mechanism without a paid relationship? So is everyone on IRC paid up? Or is it not a support mechanism? Or is it just unwise to ever *use* Twisted IRC; it should just be considered a waste of time? Absolutes like that are kind of extreme, no? Maybe you meant to say only for real-time problem solving... still seems to make the immediacy of the forum somewhat pointless, no? It did actually solve the problem, btw, and at least the search for the docs was cut short at 20 minutes. If that was intended more as a "who do you think you are, you've not paid us anything", okay, accepted. I'd been thinking of myself as a member of the community trying to sell "our" product to the world, I should have looked at it from the other side. We've (and I've) *tried* to contribute back to the Twisted community with introductory lectures at PyGTA, various pieces of documentation and releasing TwistedSNMP, but it's perfectly true that we've made no monetary arrangements for support. Fact is that we operate on a shoe-string budget to keep costs low for our customers and try to participate actively in the Open Source community to compensate others for the assistance we often get. I apologise if I went over the line in asking where the reference documents have moved ;) :) . Seriously, I didn't even think of asking in IRC, it was suggested by another colleague between stifling guffaws... maybe he was out to get me by tricking me into exactly this folly ;) .
- Do not ever expect to be able to fix a threading bug in less than 2 days.
:) That was obviously the root problem. Peace, out, Mike -- ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
On May 4, 2005, at 5:36 PM, Mike C. Fletcher wrote:
Presumably, this is what he meant: Don't CC the IRC conversation to the rest of your company ("in a public forum"). The S:N ratio and mannerisms of IRC aren't appropriate to broadcast to the rest of the world (unless they are also acquainted with how IRC works, etc.). Don't expect free help exactly when you need it ("in real-time"). #twisted is quite helpful, and often does end up being free real-time help, but there aren't any guarantees. -bob
Mike C. Fletcher wrote:
Glyph Lefkowitz wrote:
No no, you misunderstand! Your contributions have been substantial, and I didn't mean to impugn them. I don't mean, "don't use IRC" or "you've got no right", I mean, you are likely to get flak, humor, or random people being rude to you for no reason as you are to get support. This can get you in trouble in a meeting. #twisted is like a bunch of mechanics and automotive engineers (and various hangers-on) hanging around in a garage. It's not a slum, but it is a garage, there are going to be parts lying all over the place and people are informal. Going to IRC with the president of your company watching your screen is like inviting your auto mechanic and his girlfriend into your boss's limousine to discuss the company's car purchasing plans for the next year.
Hmm... sounds like maybe he was...
Glyph Lefkowitz wrote: ...
Our president, incidentally, is a leather-wearing biker (and a NetBSD developer), but I'm sure the mechanic and his girlfriend would get over their offense, eventually ;) . Point taken, though. No matter how laid back the individuals, when trying to convince a group about the viability of a product in a business setting it's not a good forum to which to turn. Have fun, Mike -- ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
Itamar Shtull-Trauring wrote:
Don't get too defensive, Itamar. Although I missed the IRC conversation, Mike's email certainly described a situation that anyone would have found frustrating -- and to me it did not appear to be an indictment of Twisted so much as simply a cautionary tale, and a useful one. It's reality. Mike knows (in retrospect) he would have done things differently if he had anticipated the docs not being there (i.e., go to his cubicle and quietly do the spelunking, away from skeptical eyes). I daresay if I were selling Twisted or handing off a Twisted app in my environment, the situation would be similar (God forbid I ever get to that ... which is part of the reason I haven't released yet ;). Twisted's documentation is constantly improving (thanks to Mary and all who contribute!), but it can still be daunting to the uninitiated, largely because Twisted is non-trivial and good technical writing is harder than it looks. Mike's experience was a confluence of many small dysfunctions. IOW, the makings of a classic tragedy -- not that it was a *huge* tragedy, but I'm sure Mike would prefer to work in an environment in which he can use Twisted and not be second-guessed (at best), and as he ruefully points out, the well is kind of poisoned now. So what's to be learned? The lesson for me is: if you are in a similar environment, go ahead and used Twisted -- to those who know it, it is reasonably robust and scalable -- but be very, very careful when demoing and/or introducing anyone else to your app. And if you encounter any bugs, no matter how insignificant the might at first appear, for heaven's sake go back to your cube for the spelunking, unless you are a Minion and know exactly what you're doing! ;) Cheers, Steve
Stephen Waterbury wrote:
A grim reality indeed. I myself have been in this situation (and even been accused of being a shyster!) and it is a very hard one to get out of. As Steve points out, there is a general-purpose learning here, but there is also the issue that given Twisted's technological maturity, users are beginning to expect a certain level of polish and sophistication from our web presence, our marketing materials and our documentation which just isn't there. Worse, this is a long-standing problem I don't see any effort underway to rectify it. Other open-source projects have gotten to this point and found huge, helpful teams of busy bees to fix the website, keep the documents up to date, wrangle the release notes and annotate the development process. Some projects are worse - I've recently had an experience with the linux cifs client project website that made me feel really good about our web presence :-) - but we could do better. That said, as it is I think the Twisted team is doing a great job at making Twisted, and we don't have any more resources or people to spend on other stuff. That's not to say we can't spot-fix the biggest problems with our website (like the API documentation problem) but ... where do these helpful people come from for other projects? I would estimate that Twisted has tens of thousands of users by now, but the community is still oddly silent. I'm genuinely baffled as to how to proceed. Is there anyone out there who can contribute some significant time to updating our "professional" presence on the web and elsewhere?
Glyph Lefkowitz wrote:
Lacking community volunteers with the necessary writing and twisted skillsets, the best option would be to have the documentation done professionally by *good* technical writers. How does one fund this? As you say, there are tens of thousands of twisted users by now, many writing code inside corporations/universities which regularly pay for software. So start a fund: - get an accurate estimate of the amount of money needed (ie., have *good* writers lined up and ready to start work under contract) - prominently display the fund status (thermometer?) on the twisted home page - solicit support from twisted users urging them to get their employers to contribute - recognize contributions I think you'd be pleasantly surprised by the response. $150k shouldn't be too hard to raise and, spent wisely, will go a long way towards solving the documentation/tutorial problems. Our group put $12.5k back into a specific open source project last year. Money that was earned thanks to that same project. Cheers, Darran. P.S. Of course a paper book would also do wonders ... -- Darran Edmundson (darran.edmundson@anu.edu.au) ANU Supercomputer Facility Vizlab Australian National University, Canberra, ACT 2600 tel: +61 2 6125-0517 fax: +61 2 6125-5088
Darran Edmundson wrote:
P.S. Of course a paper book would also do wonders ...
+1 here... I carry Alex Martelli's Python in a Nutshell around with me every day. Alex, if you're listening, how about a book on Twisted? Several books introduce Twisted positively (e.g., Goerzen's recent Foundations of Python Network Programming), but the lack of a complete print reference has made me a little hesitant to emphasize my reliance on it in recent customer projects. Heck, I have enough trouble convincing my customers that Python is supportable (e.g., "we're a Microsoft shop here..."), so I hide my use of Twisted in the packaging. Thanks again, Cory, for ntsvc... Keep up the good work, Norm
On Wed, May 04, 2005, Glyph Lefkowitz wrote:
What particular projects do you have in mind here? I ask because it could well be useful to actually go to them and say "hey, where did you find these people?" In particular, it would be useful to ask that question of things that will be used by programmers: Python itself, other languages developed/maintained by communities, the graphical toolkits and other big libraries... (answers from, say, the GNOME user documentation people might still be useful, but I think slightly less so, because that documentation can be written by non-programmers). I can think of all kinds of answers ("they just appeared!", "we don't merge features without review of their documentation," "we paid people," "we just love writing documentation," "there's these two crazy people who just love writing documentation," "there's this one company who employs our major developers and also employs some tech writers,") but I don't know what the correct ones are. -Mary
On Thu, May 05, 2005 at 08:27:57AM +1000, Mary Gardiner wrote:
In my experience (on Jabber and a few smaller projects), the hardest thing is finding people who want to write docs. Developers don't want to do it or don't have the time, and most folks who like to write are not especially technical. They might write a howto for end users, but documenting something like Twisted is beyond their ken. In the Jabber world I have been writing docs since mid-2000 but even so I mostly focus on protocol documentation because writing docs for various implementations (libraries, servers, clients) would be never-ending. My $0.02. Peter -- Peter Saint-Andre Jabber Software Foundation http://www.jabber.org/people/stpeter.shtml
On May 5, 2005, at 12:32 PM, Peter Saint-Andre wrote:
I think there's probably a lot of Twisted "end-user documentation" lying around in the form of presentations and tutorials given at various conferences. If someone aggregates the links to all of that stuff, it would probably be a useful resource to new-ish users. -bob
On Thu, 5 May 2005 12:54:46 -0400, Jeff Grimmett <grimmtooth@gmail.com> wrote:
There's lots of wiki software out there already. Writing more won't solve any pressing problem, I think (in fact, it would just have to be documented...). In the past, web-based content systems haven't been very successful with Twisted: people just don't update them frequently enough to be useful. If enough people wanted to make a serious committment to getting a Twisted wiki established, I'm sure some hosting could be arranged, but just tossing up a wiki and hoping people contribute to it probably isn't going to fly. Jp
well that is true. i am developing one "wiki like" software in twisted (and now nevow). but from the moment i started to write it i knew already on what machines will it be hosted, and now i have some 4 different servers (and around 100 sites) in different countries hosting it. but that is all. no help from others, no feedback, no any chance that somebody can use it and run it on his machine. not to mention the fact that i have to do all administrative stuff, all the template modificatons, all the "oh my... i lost my password" jobs :) anywho you can check it on http://tamtam.mi2.hr:8888/NoviTam/ and 2 years old version at http://tamtam.mi2.hr/TamTamDev/ -- http://aco.mi2.hr/
On 5/5/05, Jeff Grimmett <grimmtooth@gmail.com> wrote:
Whatever happened to the idea of a twisted Wiki, anyway? That's the sort of thing it's good at.
I was thinking more along the lines of an RSS/blog aggregator on the main site, taking in the developers' (and possibly some users') blogs, and filtering them for relevance. I know I get a ton of value out of reading various Python-, Twisted-, and PyObjC-related posts in the blogosphere (though I rather dislike RSS). There's a different feel to that kind of info; it tends to feel more personally relevant. It's relevant enough for *someone* to write about it, one hopes it's relevant enough for someone else to read :) Wouldn't require [much] new code to get working vs. a wiki, it'd be constantly updated, and once set up, nobody would have to write any more than they already do. J.
On Thu, May 05, 2005 at 12:06:41PM -0700, Jordan Krushen wrote:
Planet Twisted, anyone? http://www.planetplanet.org/ Ralph Meijer runs Planet Jabber, he might have some insights... http://planet.jabber.org/ /psa
On 5/5/05, Peter Saint-Andre <stpeter@jabber.org> wrote:
Planet Twisted, anyone?
Well, see, that needs a link on the Twisted site :) J.
On Thu, May 05, 2005, Jordan Krushen wrote:
The Planet software itself, unless it's had a very radical re-write very recently, does not use Twisted, so there's no reason to link to the software. http://planet.twistedmatrix.com/ does exist (as Itamar has already pointed out) and could probably do with a link from somewhere. -Mary
On Thu, 2005-05-05 at 14:26 -0500, Peter Saint-Andre wrote:
Planet Twisted, anyone?
http://planet.twistedmatrix.com has been up for quite a while :)
On Wednesday 04 May 2005 17:33, Glyph Lefkowitz wrote:
In Zope 3 we simply made it a policy that documentation has to be provided with any code that is checked in. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
On 5/5/05, Glyph Lefkowitz <glyph@divmod.com> wrote:
(This contains my thoughts on Twisted's state in general, this thread, and a response to this particular point from Glyph): As it stands, I'm pretty embarassed about the state of Twisted, and I'm really sympathetic for Mike. Personally, I'm truly sorry I had to drop a lot of polish from the Twisted 2.0 release; I decided to make it a higher priority to get it out the door (its release cycle was nearing Debian scales) than to get API docs building, for example. While there are a lot of docs, and people who claim "Twisted has no docs!" should be stabbed in the face, the level of usefulness in the docs isn't that, well, useful, despite Mary's excellent effort. Not only that, the code is also really nasty in certain areas, trial being particularly relevant, but certainly not the only area. Trial is something I've worried about for a while and have eventually just lost all hope of general usability for, if you're not a Twisted expert. To respond to your question, glyph, the only thing I can imagine helping Twisted a significant amount right now is if some company employing a Twisted hacker would give that Twisted hacker half a day a week to *generally* maintain Twisted. Just those 4.5 focussed hours a week would help an amazing amount, I reckon. Not just working on the web site and docs, but the code as well, like fixing bugs in the tracker. Of course, most of those companies that employ us do pay us for developing certain contributions to Twisted, but that's not really benefitting the project as a whole, it's just benefitting very small niches within it. Unfortunately, all of the companies employing Twisted hackers (Divmod, Nunatak, ITA, and some others) are either too poor or don't care enough about Twisted to make that offer. Barring that, maybe a horde of uni students with tons of time on their hands could help. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
On Thursday 05 May 2005 09:57, Christopher Armstrong wrote:
That's going to be a really hard thing to find. For instance, as far as I'm aware, there's no-one being paid to work on Python - I spend a small amount of work time on it, but certainly nowhere near half a day a week. (I generally take a half day or day off when it comes to cutting a release). If we can't find companies to fund work on Python (a much, much more visible project, with a whole pile of companies relying on it) I don't hold out much chance for someone funding Twisted as a whole. Having said that, I *could* see people being able to justify time on various Twisted subprojects because they need it for work. And yes, trial needs to be dropped off a bridge. Anthony -- Anthony Baxter <anthony@interlink.com.au> It's never too late to have a happy childhood.
Anthony Baxter <anthony@interlink.com.au> writes:
I understand that Guido has a certain amount of Python time written into his contract, but much of this is spent attending conferences and so on. Python appears to have more worker bees than Twisted who do the kind of general work radix was asking for, despite none of them being paid to do it; this is probably mostly a function of just being a bigger, more visible project (after all, every Twisted user is a Python user).
Of course, the PSF has a certain amount of money and while none of it has been spent directly on Python so far, some of it has gone on producing training material (for scientists). I personally doubt that the PSF would find funding the production of such material sufficiently in line with its interests but if noone asks, we'll never knows. Cheers, mwh -- <Aardappel> this "I hate c++" is so old <dash> it's as old as C++, yes -- from Twisted.Quotes
On Thu, May 05, 2005, Michael Hudson wrote:
This probably shouldn't be assumed though. If that's all it is, OK, damn. But if there's something about the Python community which is qualitatively different and which encourages the kind of work being discussed, Twisted should consider emulating it. -Mary
One detail: The social scene around Twisted currently features a lot of "verbal violence humor". This is very frequent on the IRC channel, and probably its practitioners don't notice how often it occurs (because shared social customs quickly become unnoticed by their practitioners). In this e-mail thread, someone wrote that people who say Twisted has no docs ought to be stabbed in the face. Obviously, this was intended as humor, but I believe that this social custom, along with as other related social customs in Twistedland, deter people from some other cultures from participating. NB: I'm not saying it's "bad". I'm not saying it's unfunny. I'm not even necessarily saying that you folks should try to change it -- maybe it works for you and the "social exclusion" factor is not a big enough problem to warrant a change. All I'm saying is that verbal violence humor, interpersonal aggression humor, and such customs limit the growth of your project by turning off people from outside your own specific (sub-)culture. Regards, Zooko
Mary Gardiner <mary-twisted@puzzling.org> writes:
Well, maybe it's just luck, or statistics of small numbers. After all, Python doesn't have very many worker bees compared to the long list of people who have checkin rights.
The (maybe just perceived, maybe real) barrier to entry is probably higher for twisted. Maybe you need more simple-to-fix bugs :) Cheers, mwh -- If you give someone Fortran, he has Fortran. If you give someone Lisp, he has any language he pleases. -- Guy L. Steele Jr, quoted by David Rush in comp.lang.scheme.scsh
On 5/4/05, Glyph Lefkowitz <glyph@divmod.com> wrote:
My feeling is that a big part of the problem is that Twisted is so darned obscure in places that most people don't feel up to the task. Other - far, far lesser, of course - projects get enthusiastic help because they are less complex and, well, mystical (for lack of a better word) than Twisted. For example: 1) I create a subclass of an existing Twisted class in order to add some helpful debugging. 2) I run my code but... the debugging code is not being executed! 3) But ... if that's the case, the code shouldn't even RUN! 4) Aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!!!! I had just such an experience. Grumpy, I was. But it illustrates the point: I'm no idiot (last I checked). But I cannot explain how it is possible that the code even worked, and thus I feel completely inadequate to the task of even STARTING to help with documentation. Where should I start? If I can't explain the workings of a ten-line web server, what hope do I have of providing accurate input to documentation? -- Best, Jeff
Itamar Shtull-Trauring wrote:
Oh, I didn't think you did it just to be annoying :) , but now that you bring up the possibility, I wonder ;) . How did you know that *I* would be the one to fall into the trap, that's what I want to know! You are a far more devious trickster than I would have imagined... :) I was just expressing my frustration at losing "mind-share" for Twisted within the company. Just another of those "all things being fine, no big deal, but not the right time to discover this" when people are in the middle of debating dropping the system. I'm not ticked off at you guys, you do what you have time to do. I *am* kind of annoyed at myself for not handling the situation better. I *would* suggest a link on the homepage just saying "documentation". Saying "old documents" implies that there are new ones somewhere. Having that link in the "Developers" menu would be good too, when the new docs are out, update those links to point to them. We would have been fine with the 1.3 docs (it's what we eventually used) for solving the first problem, but seeing "old documents" we had 3 people sitting around suggesting ways to find the new documents relating to the version we were using. Enjoy yourself, Mike ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
On 5/4/05, Itamar Shtull-Trauring <itamar@itamarst.org> wrote:
Do you have any suggestions on how we can improve things? Besides "write more docs" which is basically always a given.
OK, I'll bite at that one. Let me preface it with these two points: 1) On my development team, I'm the QA lead. We've encountered this same issue, with a different face, in the form of the lack of sufficient design documentation and code documentation of changes after the fact. It has caused serious issues down the road, both in software quality and in the quality of testing that QA provided. We've implemented some solutions that appear to work, and it's from that series of experiences that I am speaking. 2) I BELIEVE that this solution has been considered and discarded. I have suspicions why, which I will not speak of, but it's so OBVIOUS that I can't believe that a group of intelligent people would not have considered it. So I fear that I will be speaking to a wall here. So be it. My unsolicited advice: write the docs either before or during development. AFTER development is too late. Most developers have no interest in documentation (it's genetic or something) and want to move on to more "good stuff." It's OK, I understand, as everyone that enjoys coding understands it at a gut level. But the issue needs to be acknowledged before it can be dealt with. The gist of the response I often seen with regards to Twisted documentation is along the lines of "I don't see the issue here - if you don't like the docs, write some." It seems to be a denial of the responsibility to write good documentation. *** I'm not saying that is in fact the case! *** I am saying that there is a perception. As one of my past bosses told me, "perception is everything in some circumstances." Here are the pitfalls (as I see them, YMMV) of retroactively writing documentation: 1) If the person writing the docs is not the person that designed and implemented, the *intent* of the class/module/etc being documented is not clear. 2) If the person writing the docs is not the person that implemented, then the person that implemented must be found, interrogated, and gone back to multiple times to get the docs right. (And if getting it right is not the goal, why bother?) 3) If the intent of the module/class/etc is not known beforehand, how does one determine if it is working right? I don't mean to criticize, but this is in my eyes the greatest weakness of Twisted. My experiences with it, when it works, is that it works wonderfully. I can't commit to using it for anything critical, though, because I spend more time working around such gems as """I am a connector""" than I do in getting anything useful done. I tell you: if it gets to the point where I SERIOUSLY consider writing my own simple frame work to get a job done as a TIME SAVING MEASURE then there is a problem, and I do not believe I am alone in this perception. Take this in the spirit it is offered. I want to use Twisted for what I do, and I want it to succeed. -- Best, Jeff
Jeff Grimmett wrote:
My unsolicited advice: write the docs either before or during development.
dont know why you are so reluctant to say that. It also saves one from doing a lot of unproductive coding when the basic design can be put down on paper and analysed by peers to eliminate bad ideas from the beginning - I suppose that was the original motivation for something like UML - please don't shoot me :-)
That sums it up pretty well.
Twisted is a superb framework. Once I took a few days of and hacked my way through its enigmatic depths I also whispered to myself ... eureka... I'm not sure though that documentation is the *only* problem. My experience tells me that basic concepts around the "networking layer" is intimidating to a lot of developers.
I normally ask myself how long will it take to understand this stuff - given the state of the documentation. Then I compare it to how long it would take to write it myself. Lets be frank - "open source" assumes to an extent that you would actually "read the source" as part of the documentation. To me the detail (like the API) is not the problem. I would love to see high level conceptual documentation to give me a better idea of basic concepts like the "reactor", "application", "service", "transport","protocol", "factory" - what they are supposed to do and how they are to fit together. regards, Eugene Coetzee Web -> www.reedflute.com ===============================================
On 5/6/05, Eugene Coetzee <projects@reedflute.com> wrote:
Maybe maybe not. But right now the documentation is like a lightning rod. No matter what your agenda, if you want to (for whatever reason) discourage against the use of Twisted in a project, all you have to do is point to the documentation and the frazzled bunch of geeks trying to figure it all out :-) My experience tells me that basic concepts around the "networking layer"
is intimidating to a lot of developers.
It would not hurt to have some nice top level documents to address that. That's the sort of thing you'd derive from design documentation. I normally ask myself how long will it take to understand this stuff -
I would have to disagree on that on a fine point. Twisted is not "open source software that happens to be a network framework". It is a "network framework that is also open source software". I didn't start looking at Twisted because it was open source. I started looking at it because I was interested in a better answer than I currently had for networking. AS SUCH ... I expect the API, like any library I use, to be properly documented so I can start using it. I do not wish to investigate the source code as a recreational thing. I have a job to do and I want to get it done on time, on budget, and done well. Making source code delving a prerequisite to use of the software is bad form, in my opinion. Yes, I find having the source code useful. Usually it's because the documentation leaves a fine point unclear; examining the source may resolve the confusion. But the source code should not be a crutch to support incomplete documentation - not if you want to go toe to toe with the big dogs. I would love to see high level conceptual documentation to give me a
:: nod nod nod nod :: -- Best, Jeff
Jeff Grimmett wrote:
Perhaps we disagree radically about the usefulness of having the source code,being able to read it, and the ablility to recompile and rebuild the framework yourself.
But the source code should not be a crutch to support incomplete documentation - not if you want to go toe to toe with the big dogs.
I googled around with "networking framework" a few times in my life. Who are the "big dogs" when it comes to networking frameworks. In the past I wrote everything myself using raw C++ sockets, MFC or vanilla java.net. I have also been bitten badly by the "big dogs" - having to dig through a MSDN "knowledge base" of thousands of "known bugs" before I finally discovered that it *could* be fixed with next release. I honestly cannot say that Twisted is outclassed in this area, at least I can fix the bugs myself. Eugene Coetzee Web -> www.reedflute.com ===============================================
On 5/6/05, Eugene Coetzee <projects@reedflute.com> wrote:
I find it very useful. But isn't that beside the point of the discussion? Is Twisted or is it not a complete product? Why should I not expect the API to be well documented? Open source is not the reason for this. It provides, in this case, a possible way around an obstacle, but at a terrible cost in wasted time, multiplied by every person that uses the library. All those man-hours are lost to some project, somewhere. Twisted should not be considered a time sink. I find that distasteful. It's a wonderful framework that has potential in everything I do these days. It should not have this stigma attached to it.
But the source code should not be a crutch to support incomplete
documentation - not if you want to go toe to toe with the big dogs.
I googled around with "networking framework" a few times in my life. Who
Somewhere in this discussion I mentioned that perception is an important factor here. It matters not if .NET or Java or Ruby have lesser abilities than Twisted, not if potential developers are daunted by the perceived lack of information and an obscure channel towards enlightenment. Heck, I'm reading about "Rails" on PYTHON blogs, fercryinoutloud. -- Best, Jeff
On Fri, May 06, 2005 at 08:49:36PM +0200, Eugene Coetzee wrote: [...]
The glossary helps address this, although I think most people don't realise it's there: http://twistedmatrix.com/projects/core/documentation/howto/glossary.html I'd be interested to know if you think that helps, and whether you knew about it and had read it before. I agree that a nice overview (with a diagram or two) would help. -Andrew.
Andrew Bennetts wrote:
No - I didn't read it before and there is no excuse for that :-) It certainly helps, although it may be a bit minimalistic.
I agree that a nice overview (with a diagram or two) would help.
Certainly. I also find the "howto's" useful in attempting to explaining the design philosphy in a haphazard way - but there are a lot of thing that could better be explained in a more complete, systematic and elaborate overview. regards, Eugene Coetzee Web -> www.reedflute.com ===============================================
In the spirit of just having a go and testing out my twisted knowledge on this list I thought I'd start a questionaire, with answers, inspired by:
I am by no stretch of the imagination qualified to advertise these answers as authoratative. Buyer beware, and if you don't like mine give me a fat -ve 10 for my twisted score and show me your answer ;-) Authoritative intro material is at: http://twistedmatrix.com/projects/core/documentation/howto/basics.html http://twistedmatrix.com/projects/core/documentation/howto/glossary.html Q1. How do you run a twisted program ? In as many ways as there are pebbles on a beach ;-) There are lots of great examples in the docs that get you to the point where you have a python file that will create an application instance at global scope. Once you've got this far the simplest way to run your twisted program is to add the following to that file: from twisted.internet import reactor reactor.run() And then at the command prompt python myapplication.py Once you're comfortable with the basics of Services, the Application, Factory's and Protocols you're going to start caring about things like persistence, daemonization, privilege scheding, plugins etc. At that point it's time to look at 'twistd'. If you remove the lines from myapplication.py that were added above then you can run your application directly from the prompt with: twistd -oy myapplication.py Then you will get the almost the same result. The primary difference is that your now running your program as a daemon. twistd is a program that loads twisted applications from many different formats, including .py, and starts the application instance it finds there. Q2. What is an application ? Most twisted programs begin with an application instance. Twisted defines a helper function for creating an application instance so that it includes all the bits of functionality that allow it to be driven by the twisted framework. This function is found in the module twisted.application.service and is called 'Application'. Most twisted programs define a python file that makes a call to the 'Application' function at global scope. Just doing this, and nothing else, will not get you very interesting results, for that you need to look at adding 'Services'. If you've already got a python file that defines an application instance this way - in addition to some example code - and you just want to know the simplest way to run your program, then the answer you're after is almost certainly: # add these lines to myprogram.py from twisted.internet import reactor reactor.run() $ python myprogram.py But see Q1. for a little more on this. This is the most suitable way if you want to single step through your program in a debugger to find out how twisted works. There are many other ways to run your program but it's safe to ignore them until you decide you want them. Q3. What's a Service ? 'Services' collect together the bits that make it possible for cool networking stuff to happen in a single thread without blocking[1]. You don't have to use them to use the twisted framework but they can help. If you are using an application instance then you almost certainly want to use Services as well. You add Service's to your application instance (see Q2 for creating one). Each service typically manages a discrete network service. The twisted framework starts and stops all Services in an application in a well defined way when your program is executed. This includes provision for giving your services a chance to do stuff before privilege scheding happens. To define what your service is you will usually want to subclass twisted.application.service.Service and override any, or all of: privilegedStartService, startService, stopService. During a single run of your program each of these methods is called exactly once and in exactly that order. If you need to bind to privileged ports then privilegedStartService is where you need to do it. The bits that are used to piece together Service behavior are principally: Factory's, Protocols, and Defereds. The parameters involved in binding these bits together are things like hostnames, portnumbers, database passwords, and (arguably) the networking protocol the service is dealing with. Collecting these things together into a service can be done by hand, often, it can be easier, more flexible, and much more extensible if Adapters are used to do the binding. [1] Note carefully the word 'possible' in relation to 'without blocking'. The twisted framework can not guarantee your program will not block. Twisted just gives you a set of concurrency primitives to help you write your program so it doesn't block. To understand how to write a twisted program so it never blocks you need to learn to love 'Defered's. Concurrency primitives in python essentially boil down to a choice between callbacks or generators. The twisted framework supports both (as of 2.0) but the callback style is the most established. Generators didn't arrive in python until 2.2. Q4. What's a MultiService ? At some point you will almost certainly decide that subclassing MultiService is going to help you. It's not. To quote moshez: "duuuuuude. don't. inherit. multiservice." [And certainly don't inherit from both MultiService and Componentized at the same time ;-)] MultiService is an internal class that is used by the framework to manage the order in which all services, added to the application instance by you, are started and stoped. The fraze "added to the application instance" is potentially misleading but from the birds eye perspective this is essentially what you do when you add Services. You almost certainly do want to subclass Service. But don't be fooled into subclassing MultiService. Q4 What's a Factory ? Q5 What's a Protocol ? Q6 How do I get my Factory to open a connection ? Q7. How do I get my Factory to serve a connection ? Q8. How do I cancel my attempt to open a connection ? Q9. What's a plugin ? Q10. Ok, I'm feeling brave, what is an Application realy and why can't it be subclassable ? You can't subclass it because it is a function. But that doesn't tell you anything useful. The value returned by the function twisted.application.service.Appliction is an instance of twisted.python.components.Componentized. Componentized is a class that provides interface aggregation facilities in a persistable way. When you call Application you are creating an instance of Componentized that aggregates instances of the following together: * twisted.application.service.MultiService * twisted.application.service.Process * twisted.persisted.sob.Persitent The twisted framework uses interfaces and adapters to reduce the tendency for functional dependencies to snarl up the implementation of the framework. To do this it needs your application to support interface aggregation and to provide the basic interfaces that enable it to drive your program the Twisted way. In these terms the function Application produces an instance that supports the interfaces IService, IMultiService, IPersitable and IProcess. MultiService implements IService and IMultiService[1], Persistent implements IPersistable, and Process implements IProcess. These are the basic guarantees about the application instance that enable the twisted framework to drive your application. You can't specialize of the above instances either - unless you are prepared to mimic the behavior of Application. You're only route for specializing your application is by adding services. ie., via some variation of: IService(MySerivce()).setServiceParent(IMultiService(application)) It's not unusual or unreasonable to write a replacement for Application. A perfectly acceptable example is: def MyApplication(): a = MyService() a.setName("myapplication") ret = service.MultiService() a.setServiceParent(ret) return ret Reading up on adapters & interfaces in the python / Zope3 sense can give you an extra perspective when looking at the twisted docs on Adapters and interfaces. Twisted 2.0 actually uses Zope3 interfaces. And if you're reading this answer then there is a good chance taking the time to do this will help you. http://www.python.org/peps/pep-0246.html http://mail.zope.org/pipermail/zope3-dev/2005-January/013064.html [1]It is required that the IService interface and the IMultiService interface are implemented by the same underlying object (or at least it is by the application unit tests). You really probably don't want to read any more of this answer. For one thing Componentized looks like it's going to be superseded by Faceted which does similar things but much much better. From the doc string of Componentized: "I am a mixin to allow you to be adapted in various ways persistently." Componentized caches addaptions in a persitable way. Your applications root service, the MultiService instance above is effectively a persistably cached adaption from None to IMultiService and to IService. Well you did ask ;-> When you do: IService(application) You are retreiving the IService interface for you application that happens to be implemented by twisted.application.service.MultiService Depending on the feedback I'm keen to see/provide answers to Q4-Q8, Don't feel confident at all about Q9. Cheers, Robin Mary Gardiner wrote:
Mary Gardiner wrote:
I will much rather review http://twistedmatrix.com/documents/current/howto/tutorial/intro and http://twistedmatrix.com/documents/current/howto/tutorial/factory for you. I have a few issues with some some of the information in there.
Also, any suggestions on how to make the glossary easier to find?
I would suggest linking to http://twistedmatrix.com/documents/current/howto/glossary from http://twistedmatrix.com/documents/ regards, Eugene Coetzee -- -- =============================================== Reedflute Software Solutions Telephone -> +27 18 293 3236 General information -> info@reedflute.com Project information -> projects@reedflute.com Web -> www.reedflute.com ===============================================
On Wednesday 04 May 2005 20:40, Jeff Grimmett wrote:
My unsolicited advice: write the docs either before or during development.
That's exactely what we do in Zope 3 and I have been advocating this in all my mails with concrete suggestions. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
On Wednesday 04 May 2005 16:36, Itamar Shtull-Trauring wrote:
When we developed Zope 3, we put **huge** emphasis on documentation. As mentioned in the other mail, we made it a policy. Here are the things we did: - We (mainly Jim) used a slide show and later plain text files to document how we imagined a feature to work. Basically we wrote Sci-Fi stories. Thus documentation was available before code. - We also used a proposal process for larger features. This had a similar effect to the Sci-Fi presentation in that it documented the API before implementation. Many proposal authors made a point to update the proposal after the implementation, so that they would reflect the latest API. - We decided that Interfaces would be used for API documentation and public interfaces would always be found in the package's interfaces.py module, so people would always know where to go. - We later developed a very custom (on purpose) API doc tool that would not only document interfaces, but their interaction with the system. For every interface you can see its adapters and views, or which utility provides the inspected interface. The doc tool was later enhanced to also provide documentation for any object, ZCML and plain text files. - First we used regular unit tests for testing documentation, but eventually moved to file-based doctests, since they fulfill both XP requirements for tests: testing and documenting. I can't overstress the success. Every recently created or refactored package has now a README.txt file clearly documenting the API and it is never outdated, because it is part of the test suite. - We had two people write a book and find a publisher. My book is even semi-freely available as one can use it for non-commercial purposes. So, what's next? This E-mail is not suppose to show what's Zope 3 did better than Twisted or vice versa, but demonstrate some concrete things that can be done to improve the documentation situation: - I strongly suggest that Twisted starts using file-based doctests. I have demonstrated how this can be done with the current trial test runner in the following patch submission: http://twistedmatrix.com/bugs/issue1000 Also, I think twisted needs to start distinguishing between unit and functional tests. - Develop a tool that clearly shows the available adapters for a given interface. epyrun will not pick this up, so it is important. - Maybe it would be good to have a documentation sprint; since most (a lot) of the Twisted developers are in Boston, it would be good to do it here; I would be willing to come and help people getting started with writing file-based doctests. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training
On May 4, 2005, at 4:08 PM, Mike C. Fletcher wrote:
That's why it's better if you don't install Twisted, just make its source code part of your project. That's what you should be doing for just about any dependency, especially those where you don't need extension modules.
Debugging with an audience? What the hell were you thinking? Don't do that.
The deferred/generator stuff is probably "plain-old-looking" enough to be used by a programmer who isn't already used to the Twisted style of programming, with caveats of course. -bob
Bob Ippolito wrote:
Could anyone offer advice on the best way to do this? For example, the setup.py of our twisted app creates a package hierarchy: - agdevicecontrol - agdevicecontrol.server - agdevicecontrol.client - agdevicecontrol.gui - agdevicecontrol.common - agdeviceconttrol.devices This is all under subversion control and (inspired by Twisted) a few hours away from having its tests run under Buildbot. So now if we want to include Twisted2.0 and the ZopeInterfaces dependencies in our source release, should we just take a source snapshot and add: - agdevicecontrol.twisted - agdevicecontrol.zopeinterfaces And do we try to keep these in synch with the respective CVS/SVN repositories? Or since this is going out to end users, stick with stable releases (almost rhetorical question). As always, your advice is much appreciated. Cheers, Darran. -- Darran Edmundson (darran.edmundson@anu.edu.au) ANU Supercomputer Facility Vizlab Australian National University, Canberra, ACT 2600 tel: +61 2 6125-0517 fax: +61 2 6125-5088
On May 4, 2005, at 5:42 PM, Darran Edmundson wrote:
Don't try and change the Python module hierarchy. Just make twisted a sibling of your package. zopeinterfaces is a bit different because it *does* have extension module(s), and doesn't evolve very quickly, so it might be best to just require that to be installed. If you only support one platform you could install a particular version of zope.interfaces with a --prefix that points somewhere in your repository (such that it's a sibling of your code) and just check that in. Or you can go with something a bit more complicated with multiple platforms where you have a bootstrap script that mangles the sys.path so that it finds the appropriate platform specific modules. This is a "__bootstrap__" module that I've used in the past that I have used before: import os, sys # change to current dir curdir = os.path.dirname(os.path.abspath(__file__)) os.chdir(curdir) # add dependencies and platform specific path for dependencies sitepkg = os.path.join(curdir, 'site-packages') uname = os.uname() platdir = '%s-%s' % (uname[0].replace(' ', '_'), uname[-1].replace(' ', '_')) sys.path[1:1] = [ sitepkg, os.path.join(sitepkg, platdir), os.path.join(sitepkg, platdir, 'PIL'), ] In this particular case, the layout looked like this: site-packages/ formless/ nevow/ twisted/ ... these three are just checkouts of their respective projects at some point in time plat-Darwin-Power_Macintosh/ mx/ PIL/ sqlite/ _sqlite.so plat-Linux-i686/ ... same stuff as in Darwin The way it bootstraps is that I had a "start.sh" script that calls "./ twistd -o -y something.tac", where twistd is just a local copy of the twistd shell script that is modified to import __bootstrap__ before it does anything else. If you didn't notice, zope.interfaces isn't here.. because this is a copy of Twisted 1.3. nevow worked better that way at the time. It still works just fine, and I don't have to worry about other applications on the server getting in the way because the environment is pretty isolated. I can svn co the project on (almost) any Mac OS X or Linux machine with a Python 2.3 interpreter and just run it, without worrying about any other dependencies.
That's up to you. You might want to map external code (even specific revisions of external code) as just svn:externals, but since you don't have control over other people's repositories (i.e. they might be down when you need to checkout) it's probably a good idea to "checkpoint" specific releases and make sure they live in your repo somewhere. -bob
Bob Ippolito wrote: ...
Hmm, can't see that flying with our sysadmins, they want standard packages with minimal apps that use them (NetBSD developer).
Debugging with an audience? What the hell were you thinking? Don't do that.
Pair programming *always* has an audience, in this particular case we just had the third programmer looking over our shoulder because we'd just been debating the merits of Twisted and he wanted to see how it compared to his own experiences with Twisted. Going back to my "cubicle" (actually apartment) would have involved making the pair programming remote (which is common in our company), the difference being that I could do the looking up/spelunking *before* I called over to start work ;) .
Quite possibly it would be, but I'm afraid the cow is already in the corn for this project. We'll see if they are won over by the beauty and majesty of the big Twisted project when/if they start working on that. Have fun, Mike ________________________________________________ Mike C. Fletcher Designer, VR Plumber, Coder http://www.vrplumber.com http://blog.vrplumber.com
On May 4, 2005, at 5:54 PM, Mike C. Fletcher wrote:
Sysadmins be damned, the more self-contained an application is, the less trouble you'll have debugging, developing, and upgrading it. ESPECIALLY since Python can't reasonably do multiple-concurrent- versions (effectively our own DLL hell).
The way you said it kind of implied that more than 2 people were present, and that these were non-programmers. -bob
Bob Ippolito wrote:
Debugging with an audience? What the hell were you thinking? Don't do that.
Yeah- debugging is never pretty no matter if you are using Visual Basic "idiot proof" wizard tools or assembler written in encrypted sanskrit. What kind of IT outfit torture their developers by forcing them to debug "live" with the sword of managerial scrutiny hanging over their heads :-) For me the moral of the story is that frameworks tend to make simple tasks complex and complex tasks simple. Perhaps Twisted wasn't the best choice to start with? Web -> www.reedflute.com ===============================================
Bob Ippolito wrote:
Eww. -1 on that. As a Debian person, I tend to spend a lot of time dekludging things where upstream includes 5 libraries with the project, and will not use system-installed libraries. Not to speak of the bloat in the size of the source tarballs, which is multiplied by a _lot_ when it comes to Debian.
On 6-mei-2005, at 12:08, Sergio Trejo wrote:
When I install software in a production environment I prefer self- contained installations. That increases diskusage, but takes away some headages: what if server1 requires Twisted 1.3 and doesn't work with Twisted 2.0 and server2 requires Twisted 2.0? Ronald
On May 4, 2005, at 1:08 PM, Mike C. Fletcher wrote:
Did you ever figure out why? I have just run into similar behaviour myself after upgrading to 2.0. Here's an example that hangs for me on Mac OS X (10.4) after the final print: --------------- from twisted.internet import reactor, defer, threads from twisted.trial.util import deferredResult from twisted.web.client import getPage import time def itFired(result): print "Deferred fired: %s" % result return result d = getPage("http://www.example.com/").addBoth(itFired) print "Returned: %s" % deferredResult(d) print "Done; exiting" --------------- So far as I can tell, the python threading module has an atexit handler that's hanging. Also, if I replace the deferToThread() with a twisted.web.client.getPage(), I still get the same hang. --Grant Grant Baillie Open Source Applications Foundation
On May 5, 2005, at 11:49 AM, Grant Baillie wrote:
Er, I'd already replaced the deferToThread() in an earlier version of that file with twisted.web.client.getPage(), for anyone following along at home. I also found a hackaround, which is to insert the following before the call to getPage: #--------------- import threading # threading seems to install an atexit() handler that # can wedge if you've run the reactor. So, we import it # first.... import atexit def runAndStop(): if not reactor.running: reactor.callLater(0, reactor.stop) reactor.run() # ... then our handler will be called before threading's. atexit.register(runAndStop) #--------------- I'm not sure if this is a problem in python, in twisted's use of threading, or in trial.deferredResult, though. --Grant Grant Baillie Open Source Applications Foundation
On May 5, 2005, at 3:05 PM, Grant Baillie wrote:
Almost definitely a combination of 2 and 3, with 3 being the one really at fault. Trial is horribly, horribly broken by design and it's really just an accident that it works at all. The expectations a lot of the tests have about the reactor are also broken, but that's mostly just a consequence of trial sucking. -bob
On May 5, 2005, at 12:26 PM, Bob Ippolito wrote:
Thanks for the info. I'll go ahead and file a bug, unless someone tells me not to. FWIW, I'm happy with my workaround (I'm really only using it to keep my doc/unittests relatively uncomplicated). --Grant Grant Baillie Open Source Applications Foundation http://www.osafoundation.org
On Thu, 5 May 2005 11:49:20 -0700, Grant Baillie <grant@osafoundation.org> wrote:
So as to be entirely clear, I'm going to limit this response to just one idea. If more details are desired, I can post a followup later (as long as someone asks for one). Anyway, for now, here's the meat of this post: Do _____not_____ use deferredResult (or deferredError or wait) As someone who pushed for their existence, I apologize. Jp
Jp Calderone <exarkun@divmod.com> writes:
Just to be clear, IMHO, I'd qualify this by saying something like in final production code or non-test code or something. We make heavy use of these functions in our unit tests and they are invaluable ways to simplify the structure of such tests when each test is the only thing going on and there's no problem iterating the reactor beneath a blocking call. We've also cheated and used them for pure startup code in GUI apps that are quick 'n dirty example interfaces into our main Twisted system, and again they can simplify such efforts. But I do agree they really have no place in final production code. -- David
On May 6, 2005, at 5:11 PM, David Bolen wrote:
Sorry, but Jp is right. They have no place in any code. Their existence is a mistake and should be corrected ASAP. They severely violate the reactor interface. Something almost as easy would probably be waitDeferred and deferredGenerator in twisted.internet.defer (I don't know when they went in, but they're probably in 2.0). See the waitDeferred's doc string for information. The only way they could be "fixed" is if their implementation did something more like this: def deferredResult(d): """untested because this code should not exist""" res = [] def cb(result): res.append(result) reactor.stop() d.addBoth(cb) reactor.run() res = res.pop() if isinstance(res, Failure): failure.raiseException() return res However, if their implementation did that, then the tests would probably break (because the tests are WRONG) or be a hell of a lot slower and even less reliable because there is no more timeout. -bob
Bob Ippolito <bob@redivi.com> writes: with respect to deferredResult/deferredError:
Hmm, then perhaps it's the reactor interface that could use some improving rather than removing this high level functionality? Or is everyone just saying that the implementation (versus the concept) of the current functions is done wrong. Or maybe I'm just missing what major mess having this sort of interface is exposing. Given Glyph's recent response, I might also clarify that while our heaviest use of these functions are within unit tests, we aren't using trial, so any linkage between poor behavior of these implementations and trial is not something I'm referring to or worry about breaking. What I do think is that there is a very good and practical reason to be able to unwind deferrable operations into a blocking structure for a wide variety of test conditions, and losing that capability would make writing tested Twisted code (at least for our situation) much more fragile and error prone (the tests themselves). And that's not quite the same as the newer waitForDeferred stuff in 2.0 that integrates with generators. While slick, it still doesn't work as well for easily maintainable tests in many of our situations. If there's a problem with the current implementation, or with how the current implementation may abuse the public reactor interface, then I'm all for cleaning it up, fixing it or making it part of the reactor itself in some way, as long as the public mechanism remains. I mean, after all, at some point the reactor itself really is just sitting there iterating itself, so I find little reason not to permit some access to hook into that mechanism when appropriate. For example, in our system we may very heavy use of our own components all of which implement their entire API via deferrable interfaces. Interacting with multiple such components during a test is thus a multi-stage deferrable process. Having these blocking calls permits a test method to say something like: self.user = deferredResult(self.umgr.user()) self.user.email = 'test@dom.ain' deferredResult(self.umgr.saveUser(self.user)) in order to test being able to save a new user object into the system. Part of the test is that the calls themselves should succeed, so the natural result of deferredResult raising an exception on an errback is just right. And while clearly readable, the test still tests that the deferrable aspect of the interfaces will work properly (e.g, with deferredResult generating the exception if it doesn't get passed a deferred). Sure, I could write the user() and saveUser() calls with a callback chain, and then I'd have to nest the call to saveUser based on the user callback, probably something like: def fail(failure): self.fail(failure) def gotUser(user): self.user = user self.user.email = 'test@dom.ain' return self.umgr.saveUser(self.user) d = self.umgr.user() d.addCallback(gotUser) d.addErrback(fail) but I see no advantage (*in the context of such a test*) to doing it this way, and I see a real loss of clarity and maintainability for the test itself. Since each component in our system has a deferrable interface, any test which is going to interact with multiple components would quickly evolve a fairly deep callback system in the test, without much benefit that I can see. I'm all for fixing an implementation or cleaning up an interface if it's being misused, but I certainly find value in the particular external interface that deferred{Result,Error} deliver - so much so that we would (and in fact did) implement them locally before finding that they already existed. -- David
On May 6, 2005, at 6:51 PM, David Bolen wrote:
The reactor interface is not what needs improving, it's the way that these functions work. They iterate the reactor when the reactor is in a stopped state. The reactor should either be running, or not running. There are basically two fundamental issues: (1) Reactors can only be (meaningfully/predictably/etc) iterated if Twisted rules the universe AND the implementation of that reactor is amenable to that feature. This is not a tautology. (2) Reactors need to fire various startup/shutdown events. Reactors shouldn't be doing ANYTHING unless they are in a running state. The current deferredResult/deferredError breaks both of these conditions. (1) It iterates the reactor (which is a historically public, but conceptually broken interface) (2) It iterates the reactor in a STOPPED state. The reactor is never "running" during these tests. Startup/Shutdown does not happen! The fact that the doc strings talk about re-entrancy of certain reactor functions and whatnot scares the shit out of me. They should not have to be re-entrant. What their current implementation is doing is really really broken. The problem, for tests, is that using a properly written deferredResult / deferredError the reactor would startup/shutdown violently throughout the course of a single test, and will break the hell out of it if it has anything to do with services/etc. So, while for SOME deferreds it would work fine, but for others, it wouldn't. Basically, unless you can encapsulate the entire test in a single deferred, then it shouldn't be using deferredResult/deferredError. Therefore, what SHOULD happen is that trial should let you write tests that return deferreds, and it should let you write it in the deferredGenerator style (so it doesn't suck so much). Trial *could*, in theory, put the reactor into a "started" state at the beginning of every tests and a "stopped" state at the end, but then you're testing in a strange environment that doesn't really mimic how Twisted actually works, and it still breaks (1) which makes it unsuitable for testing the reactors where Twisted does not rule the universe. Testing this theory would require changes to the reactor interface to make some parts of if re-entrant (bleargh).
Well, it causes you to write two lines of code instead of one. That sucks, but so what? You don't have to write big callback chains.. It's conceptually identical, it's just that you have to throw in some extra boiler plate that says THIS IS A DEFERRED. If the way we say that wasn't so ugly, it might actually be a good thing from a readability standpoint :) I'm not *entirely* sure why "waitForDeferred" is really necessary at all, though "deferredGenerator" certainly is. In the interest of not- wearing-off-your-fingerprints one might write a "deferredGenerator" specifically for trial (or maybe in general) that just makes sure you're yielding a Deferred. I *think* that it does this so that it can assume the last thing you yield is the result. I would say that you should wrap the result instead of wrap every intermediate thing, since the intermediate things you do in such a deferred generator should far outnumber the times that you return from it. Perhaps whoever wrote it had a good reason that I just don't understand, but when I wrote something equivalent a few years ago I wrote it such that there wasn't quite so much boilerplate.
# let's assume I've aliased waitForDeferred to "wait" d = waitForDeferred(self.umgr.user()) yield d self.user = d.getResult() self.user.email = 'test@dom.ain' d = waitForDeferred(self.umgr.saveUser(self.user)) yield d d.getResult() So, the test is 6 lines instead of 3. Which sucks, but it's correct. For tests that do more stuff, it probably should be even less of a problem. If Python grows an extended iteration protocol, where yield becomes an expression, you could make it 3 lines again. This might actually happen in Python 2.5.
Well, let's say your database thing is a service, that maintains some kind of ephemeral state that's required in some way. If deferredResult were properly written, this ephemeral state would hit the bit bucket on each deferredResult, probably breaking your code even though the test are "correct". If you add such a component to the system, you might have to rewrite all of your tests, because it would not be possible to fix the way deferredResult works.
So, use deferredGenerator / waitForDeferred. -bob
Bob Ippolito <bob@redivi.com> writes:
This seems like an internal implementation issue to me - when a reactor is "running" (I've called run()), it's basically stuck in a loop doing runUntilCurrent and then doIteration. That's precisely what iterate does. Now, I agree that if you have tests that never initiate the run, that you skip over some startup events. But except for the uppermost tests, what is being tested is a specific unit test for a piece of the system that should be testable in isolation. If the item under test has a dependency on startup events, the test should arrange for them.
Not sure which doc strings those are, but I don't see a re-entrancy problem in the sort of scenario I'm looking at - rather than: reactor.run which calls reactor.runUntilCurrent/doIteration which causes my deferrable stuff to operate I have deferredResult which calls reactor.iterate which uses reactor.runUntilCurrent/doIteration which causes my deferrable stuff to operate I don't think there's any more room for re-entrancy issues than in a normal running reactor. Maybe this is a key difference with trial? If trial has a top level reactor.run that is always above any test, then I do see how you could get re-entrancy problems, since you'd be re-entering reactor.iterate from within an existing reactor.iterate call. I guess that's true of any nested use of deferredResult too, but we don't nest our deferredResult calls - no real need since any deferrable is directly wrapped with the deferredResult call, and deferredResult is only used in the tests, so what they are calling is always production code that is written properly with callbacks and what not.
I still don't necessarily see that (the last sentence). We use multiple deferrable operations in single tests, but never more than one at a time (e.g., no recursive or nested uses). But certainly more than a single deferrable operation within a single test. I would, however, agree that I'd prefer even more the ability to completely start/stop a reactor during the course of a test, but would still like to be able to iterate it manually during the test to provide a natural blocking flow to the test. But in my current scenario, the majority of my components under test are having interfaces tested that are not impacted by startup/shutdown (and we don't use any services, in the Twisted sense, for example).
I don't think I'd disagree with that - we pretty much stayed away from trial since I wasn't comfortable with it initially, so we're a pure Unittest approach, which certainly doesn't provide any framework for something like that.
Yeah, that's a tough problem, although one that would also simplify the fact that we often run the tests under the unittest GUI, and occasionally have to fight cross-test pollution from the reactor persisting across test runs, which is a wart from the current restrictions.
Well, seeing "deferredResult" sort of already says "THIS IS A DEFERRED" in the test. And it's less the number of lines than the inversion of the logic. In my example, it's tougher to read through that test and see the underlying core test of the user() and saveUser() calls. This is true really of any deferred code in general, so I find the ability to simplify things in the test level an improvement. As you say, the way we "say" this currently is ugly.
Except that you didn't include the additional code I'd now have to write to actually turn that into a generator which is iterated over by the test in order to actually process the deferrable yields. (Unless this is something trial does automatically somehow). That's what I meant by saying that the integration into generators is slick (and goes a good way to linearizing what is normally a callback chain), but still isn't quite as simple as the interface provided by the deferred{Result,Error} functions.
I might be getting lost on the "properly written" part, but if I were testing a component that did have state triggered during reactor startup/shutdown (which is what I think you're referring to), that test would likely be using direct calls to the component to trigger the startup/shutdown actions as part of the test setup/teardown, but without using the reactor. That's because the test would be focused on the component and not on Twisted itself (which in the context of such a test would be a "system" component that I'm trusting would do the right thing). This is clearly different than when testing Twisted itself. Now that would cover the lion's share of the component tests but somewhere there's the question of who plugs the components interface into the twisted events and is that done right. And I agree that's messy right now, but it's such a small bit of initialization code that even if it doesn't get full coverage in the tests it's not hard to validate. But I'd certainly welcome ways to test that more fully. So my tests should be fine for what they are testing (the component).
I'm not following this.
So, use deferredGenerator / waitForDeferred.
Once we move to 2.0 we might consider that (I'm not quite up for back-porting into 1.3 that we're using now), but I still think that would complicate the tests (in terms of the overhead to actually run each test like a generator) - but different people can have different views on what is more maintainable. Since we're not running our entire test bed under control of a single "reactor.run" call (I presume that's more of what trial does?), even using the generator would effectively be iterating the reactor somewhere along the line. I don't know - maybe my use case is just limited enough (non trial, no nesting, etc...) that I don't see any true exposures through deferred{Result,Error} while I'm getting benefits. -- David
On May 6, 2005, at 8:19 PM, David Bolen wrote:
This is an implementation detail. runUntilCurrent, doIteration, and iterate should not be public API.
The reactor itself needs startup/shutdown events. As you've noticed, the thread pool depends on them.
We all want things that aren't really possible to do :)
However if some implementation detail of twisted changes to take advantage of reactor startup/shutdown events internally then all of your tests would break even though the application would work. Having tests that fail in theory but work in practice is uh, bad.
Staying away from trial isn't really a bad idea. It's obviously broken. However, bringing the broken functionality from trial back into unittest doesn't make it any better :)
Trial could and should (but doesn't currently) do that automatically if your test function returns an iterator instead of None.
Using Twisted isn't as simple as using urllib to suck down a web page, and using ascii strings is easier than using unicode...
That's assuming you actually know about all of the services necessary to make something work, and that these services can startup/shutdown properly in this manner. The more special crap you write in a test the less useful the test is because it's testing something in an entirely different way than it actually works.
Move to 2.0 ASAP. Don't backport to 1.3.
Trial doesn't use reactor.run at all, that's why it's broken. Using the generator would be iterating the reactor but it would be not broken. A good implementation of test would start and stop the reactor at the beginning and end of every test so that you (probably) don't end up with side-effects due to the order that the tests run.
You're right, some things don't break in horrible ways when you use deferred{Result,Error}. Sometimes you can concatenate a str and a unicode and it Just Works too, but it only works if the str contains characters that can be decoded by the default encoding, whatever that happens to be at the time :) -bob
On 06 May 2005 20:19:45 -0400, David Bolen <db3l@fitlinxx.com> wrote:
It's an implementation detail, yes, but not an internal one. For the reactor to operate properly, it *must* be started up and shut down. That's simply a requirement of the interface. It's not even a particularly unusual one: many libraries require you call some initialization function before proceeding to other APIs they provide. With recent versions of Twisted, many reactors don't actually depend terribly heavily on being started up, so you may be able to call iterate() without calling run() first and see something resembling correct behavior. However, you should note that future versions of Twisted may introduce new startup requirements which break programs which assume reactors do not need an explicit startup event. Additionally, third-party reactors may depend on the startup event now, thus breaking when used with code that only uses iterate(). The shutdown event is much more important these days. People who have noticed Twisted programs and tests which hang at shutdown can attest to this.
This is one of the cases, yes.
Your application code all deals with Deferred callbacks. Why is it a stretch to have your test code do the same?
That you know of. With current releases of Twisted.
This is one reason you should use trial, even though it is broken. Trial will be fixed by Twisted developers who (at least occassionally) have a firm grasp of how it can correctly interact with the reactor, so by using it you benefit from this understanding. By using unittest, you are forced to get all the things right that trial will someday get right, or accept a broken test harness. If you have feature requirements that trial does not satisfy, make some feature requests :) We are quite aware that trial is not yet feature complete and will welcome suggestions for features that make it a more useful tool.
I don't think anyone is trying to suggest that it is simpler. However, it has the advantage of being correct. Given the choice between simplicity and correctness, I am inclined towards correctness.
You can definitely do this. The problem is that the reactor may have internal setup and tear down of which you are not aware and which will cause problems if skipped.
It may be the case that none of the bugs in trial synchronous result utilities cause problems for you. If they don't, I can understand how it would not be a big priority for you to move away from them (after all, I'm sure you have plenty of things that *are* causing you problems now that you would much rather spend time on). I recommend simply keeping this conversation in mind if and when you run into problems with your use of deferredResult and deferredError in the future and that you not let such problems take you by surprise. You can rest relatively easily in the knowledge that deferredResult and friends will not disappear from Twisted overnight. Broken as they are, they are clearly part of a public API and so will remain long enough to satisfy backwards compatibility requirements. Jp
Jp Calderone <exarkun@divmod.com> writes: (...)
Seems like as good a way as any to end this thread for me, since I definitely agree with the above. And yes, I'm aware (or believe I am) of the risks I may be incurring by using these methods, as well as using unittest rather than trial, and will certainly continue to follow Twisted actively to hopefully catch changes that may impact my current choices. -- David
On May 11, 2005, at 4:28 PM, David Bolen wrote:
I'll chime in with a "me, too" here. It might also be nice to add some kind of warning to the __doc__ strings for the 3 methods in question. There are caveats in wait()'s doc, but they weren't sufficient to scare me away, for one :). --Grant Grant Baillie Open Source Applications Foundation http://www.osafoundation.org
Grant Baillie wrote:
I apologize for the lack of enthusiasm of those warnings. The unfortunate problem is that currently _all_ of trial is subtly broken, so there is nothing that we could suggest as a decent alternative, and thus can only deprecate so hard :-(. The recent rewrite at least allows for returning Deferreds, which is what the future public API will eventually look like, but all it does when you return a Deferred is call wait() on it. (JP got me up to speed today on some bugs that are unique to returning Deferreds in this way, actually...)
On 06 May 2005 20:19:45 -0400, David Bolen <db3l@fitlinxx.com> wrote:
Uhhh, maybe I'm missing something here, but as I understand it (I probably should, being the guy who wrote deferredgenerators), that code is: testCase = deferredGenerator(testCase) after the definition of the method, or even: @deferredGenerator before the definition of the method. HTH -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
Christopher Armstrong <radeex@gmail.com> writes:
But then who actually performs the iteration over the test case to trigger the generator? Clearly some top level code needs to invoke the reactor.run() and at some point iterate over testCase while beneath a running reactor. We're just using unittest, so something would need to be added to serve as the basic framework for that sort of test. I'm guessing from your uncertainty that trial - which I haven't kept up with changes for 2.0 - probably provides a mode to do this automatically, so if I were using trial you'd be correct and it wouldn't be that much more work to integrate the new deferred generator support into the tests. Note that independent of use in tests, I'm definitely intrigued by these techniques (as I was in the early flow support in Twisted 1.x) so when we start the migration to 2.0 we'll probably look at incorporating this approach into the code base when reasonable. -- David
On 11 May 2005 19:34:15 -0400, David Bolen <db3l@fitlinxx.com> wrote:
deferredGenerator does the iteration. This is how defgen works: you define a generator. It should yield waitForDeferred instances. Then you wrap that generator function with deferredGenerator. deferredGenerator wraps that function such that, when called, returns a Deferred (not a generator object!). The original generator will be exhausted (doing the magic wait-for-deferred stuff along the way) and the Deferred returned from your function will result in the last item yielded from the generator. Since trial supports test methods returning deferreds, this Just Works. -- Twisted | Christopher Armstrong: International Man of Twistery Radix | -- http://radix.twistedmatrix.com | Release Manager, Twisted Project \\\V/// | -- http://twistedmatrix.com |o O| | Founding Member, Hobart Hacking Society w----v----w-+ -- http://hackingsociety.org/chapters/hash
Hey Mike, this thread has been given a thrashing and should probably be let go off, but it hits a pretty emotional issue for me - so here goes my 2c's worth :) I've been in similiar situations before, always with selling (evangelising) python though - I dream of the day I can be selling twisted over vanilla python :) Mike C. Fletcher wrote:
+ Even the Finger sample is mystifyingly complex if you jump into it at the end of the process
certainly - that's why there is the Finger sample - your lead programmer never took 2 hours to work through the Finger sample? I often find this with python - me: 'these problems we're having - they go away if you use python' reluctant workplace: 'but we don't have the skills in house to support python' me: 'python's has be designed to be useful as an educational language - i've found programmers can be at least as productive as they are currently using python within 3 days. here are some great tutorial links.' reluctant workplace: 'we'll see' me: 'these problems we're having - they go away if you use python' reluctant workplace: 'but we don't have the skills in house to support python' me: 'did you even open one of those tutorial links i sent you?' reluctant workplace: 'no'
i've found it a losing battle trying to spoon feed even bright people who aren't eager to try out new and interesting ways of solving problems.
Not having a go at all you at all - but I don't believe there is a solution on earth, paid for, or volunteer, you would have come out ok in this situation. It sounds like your company had an axe to grind. I can't think of many paid support contracts I've had access to that could even get a problem being worked on within 20mins - let alone solved.
possibly, twisted can't always be the right hammer. vanilla python is still brilliant. just hoping your company hasn't shut the door for when twisted could be needed. good luck in the future, for both of us:) Andy.
participants (27)
-
Aleksandar Erkalovic -
Andrew Bennetts -
Andy Gayton -
Anthony Baxter -
Bob Ippolito -
Christopher Armstrong -
Darran Edmundson -
David Bolen -
Eugene Coetzee -
Glyph Lefkowitz -
Grant Baillie -
Itamar Shtull-Trauring -
Jeff Grimmett -
Jordan Krushen -
Jp Calderone -
Mary Gardiner -
Michael Hudson -
Mike C. Fletcher -
Norm Petterson -
Peter Saint-Andre -
Robin Bryce -
Ronald Oussoren -
Sergio Trejo -
Stephan Richter -
Stephen Waterbury -
Tommi Virtanen -
Zooko