Mailman 3 March 2006 - Twisted

[Twisted-Python] Nested/Aggregate Exceptions
by Justin Mazzola Paluska 29 Mar '06

29 Mar '06

Hello, I'm writing a Twisted application using Twisted 2.0.1 on Mandriva 2006 that uses PerspectiveBroker for communication between modules. Our application is based on a centralized Twisted server and web- and GTK-based client applications that connect to the Twisted server to manage a user database. Up until recently, most of methods exported by the Twisted server are essentially single-operation functions that operate on a single user (e.g., "update_user(user, **data)"). As such, exception handling in the clients is easy: if the operation throws an exception, we know what operation died and can report a reasonable error to the user running the client program. In the most recent version of our server, however, we've decided to add several aggregate functions to our server that take in multiple users (e.g., "update_users(users, **data)" where users is a Python list of user names). Essentially, these new aggregate functions repeatedly call their single-operation counterparts multiple times. Since each call that the aggregate function makes is independent from other calls, we store away exceptions that get raised and continue with the other operations. At the end of the function, if there were errors, we raise an AggregateException that contains the list of all exceptions raised by the aggregate function. This way our clients know what the specific exceptions and what arguments they failed on. The problem is, I can't figure out how to pass an AggregateException across PB. I know I can probably use pb.CopyableFailure, but my experiments usually died when I passed non-Failure-subclasses into my aggregate. Do the mavens on the list have any suggestions? Thanks! --Justin

3 2

[Twisted-Python] Twisted Sponsors
by Eric Mangold 28 Mar '06

28 Mar '06

So I'm compiling a web page listing everyone who has donated money, hardware, software, hosting, etc to Twisted. You can see the work-in-progress here: http://twistedmatrix.com/trac/wiki/TwistedSponsors As part of maintaining win32 I would like a place to thank people for all the stuff I am about to ask for ;) I'm probably leaving out a lot of people, so would whoever is in the know please response, and we can get a comprehensive list sorted out. -Eric

1 0

[Twisted-Python] Re: [Twisted-commits] r16486 - Merge epytext-cleanups-1545
by James Y Knight 27 Mar '06

27 Mar '06

> - zsh_extras = [":file to read from:action", ":file to write > to:action"] > + zsh_extras = [\":file to read from:action\", \":file to write > to:action\"] Why were changes like that made? Isn't it a no-op, since they were in triple-quoted strings anyhow? It would be nice if the docs were actually usable/readable by people viewing the source as well as people viewing the generated docs, where possible. James

5 5

Re: [Twisted-Python] Degrading under load
by Yitzchak Gale 26 Mar '06

26 Mar '06

Sorry, I guess my question wasn't clear enough. The most important things I need to know are: When running listenTCP, how often does twisted accept pending connections on the port? Is it only when the previous connection is finished processing, or every time the event loop gets control, or something in between? And when twisted does accept pending connections, does it accept ALL of them and queue them all for processing, or just one at a time? Thanks, Yitz My original post: > I need to set up a TCP service (on a linux box) > that will get something like a few hunderd connections > per minute at peak load. For each connection, I do > some XML processing, and possibly send a query > to another nearby machine and get a respone. > > Seems to me that twisted should be able to handle that. > > But what happens when I get the occasional burst > of connections, lets say tens of connections within > one second? What I need is: > > o Every client gets a socket connection promptly, so > no danger of TCP timeout. > o Under medium load, clients will have to wait a > bit longer for the response. > o Under heavy load, some clients will get a "busy" > response (defined in the protocol I am implementing) > and immediate socket close. > > What is the best way to do that in twisted? I envision > one of the following architectures: > > A. Just use twisted in the usual way. Watch twisted's > event queue for heavy load. > > B. Two processes: one to dish out connections and one > to queue requests and process them. > > C. Three processes: one to dish out connections, one > to queue requests and watch for load, and one to > process the requests. > > Which of these do I need to use to get the desired > effect under load? Or is there some better way?

11 21

[Twisted-Python] Getting splitted data with SSHChannel.dataReceived
by Mustafa Sakalsiz 26 Mar '06

26 Mar '06

Hi, I am using twisted conch to execute a remote program (say ProgA) through ssh. In ProgA, I print some datas and catch them in the dataReceived method of SSHChannel. Sometimes, the data comes partially, like this ProgA dataReceived ------------ -------------------------------- prints TWISTED gets TWISTED and appends it to TOTAL . . . prints TWISTED if len(TOTAL) is 1020 gets TWIS (gets at most 1024) => then gets TED (next time) I think there's a buffer of something else, so dataReceived couldn't get the datas as they are printed. They are splitted if TOTAL reaches 1024. I am looking for a solution to prevent getting splitted data. Any help is greatly appreciated. Greetings, Saki

3 3

[Twisted-Python] Weekly Bug Summary
by exarkun＠twistedmatrix.com 26 Mar '06

26 Mar '06

Bug summary ______________________ Summary for 2006-03-19 through 2006-03-26 Bugs opened: 15 Bugs closed: 27 Total open bugs: 571 (-12) |== Type Changes |== Priority Changes |== Component Changes |Defect: -3 |Highest: -6 |: -1 |Enhancement: -8 |High: -3 |Conch: -2 |Task: -1 |Normal: -2 |Core: -1 |Low: -1 |Mail: +0 |Release Management: -1 |Trial: +2 |Web: -3 |Web2: -1 |Website: -5 New / Reopened Bugs ______________________ ===== Highest ===== [#1604] Write More Doc Strings (opened by exarkun) enhancement core http://twistedmatrix.com/trac/ticket/1604 ===== High ===== [#1591] twisted.web2.dav.test.test_prop.PROP.test_PROPFIND fails if pyxattr is installed (opened by spiv) defect web2 http://twistedmatrix.com/trac/ticket/1591 [#1594] [PATCH] Errors from Trial TestCases don't report as failures when run by pyunit (opened by spiv) defect trial http://twistedmatrix.com/trac/ticket/1594 ===== Normal ===== [#1592] twisted.trial.test.test_loader.FileTest.test_packageNotInPath fails (opened by spiv) defect trial http://twistedmatrix.com/trac/ticket/1592 [#1593] twisted.test.test_udp.OldConnectedUDPTestCase.testDNSFailure fails on Ubuntu Breezy (opened by spiv) defect core http://twistedmatrix.com/trac/ticket/1593 [#1595] .txt file doctest support for trial (opened by radix) enhancement core http://twistedmatrix.com/trac/ticket/1595 [#1596] Leftover use of childNotFound() in t.w2.twcgi.CGIDirectory (opened by grimmtooth) defect web2 http://twistedmatrix.com/trac/ticket/1596 [#1598] Review API stability markers (opened by itamarst) enhancement core http://twistedmatrix.com/trac/ticket/1598 [#1599] t.p.c.Interface shouldn't warn when a user inherits from another interface (opened by jknight) enhancement core http://twistedmatrix.com/trac/ticket/1599 [#1602] smtp.SMTP would be faster using LineOnlyReceiver (opened by nickm) (CLOSED, fixed) enhancement mail http://twistedmatrix.com/trac/ticket/1602 [#1603] Typo in client howto (opened by exarkun) defect core http://twistedmatrix.com/trac/ticket/1603 [#1605] twisted.python.components.{getAdapterClass,getAdapterClassWithInheritance} mispell "deprecated" in docstring (opened by exarkun) defect core http://twistedmatrix.com/trac/ticket/1605 ===== Low ===== [#1597] Add query links to subprojects to quickly access tickets for that projet (opened by therve) (CLOSED, fixed) enhancement website http://twistedmatrix.com/trac/ticket/1597 [#1600] auto-reference ticket from branch with ticket number in name (opened by jknight) enhancement website http://twistedmatrix.com/trac/ticket/1600 [#1601] t.py.text.greedyWrap makes t.py.usage.Options.longdesc ugly (opened by dreid) enhancement core http://twistedmatrix.com/trac/ticket/1601 Closed Bugs ______________________ ===== Highest ===== [#1530] Document API documentation update process (opened by exarkun, closed by radix, fixed) task website http://twistedmatrix.com/trac/ticket/1530 [#1331] "Sumo" is confusing and unnecessary (opened by exarkun, closed by radix, fixed) defect core http://twistedmatrix.com/trac/ticket/1331 [#1561] Qt reactor fails two TCP tests (opened by exarkun, closed by exarkun, fixed) defect core http://twistedmatrix.com/trac/ticket/1561 [#1497] quick command-line way to run web2 server (opened by glyph, closed by dreid, fixed) defect web2 http://twistedmatrix.com/trac/ticket/1497 [#737 ] Minor code cleanups (opened by HelmutG, closed by exarkun, fixed) enhancement conch http://twistedmatrix.com/trac/ticket/737 [#1562] Permission denied errors on Win32 (opened by exarkun, closed by exarkun, fixed) defect core http://twistedmatrix.com/trac/ticket/1562 [#1578] setsockopt fails on joinAddr (opened by ghazel, closed by exarkun, fixed) defect core http://twistedmatrix.com/trac/ticket/1578 ===== High ===== [#551 ] [PATCH] Distrib throws exception when accessing /users/username/ where username exists but has no public_html (opened by Jerub, closed by radix, fixed) defect web http://twistedmatrix.com/trac/ticket/551 [#969 ] No .spec file in tarball as indicated by INSTALL file (opened by bjorn, closed by radix, fixed) defect release management http://twistedmatrix.com/trac/ticket/969 [#340 ] [PATCH]intermittent freebsd UDP failure (opened by warner, closed by exarkun, fixed) defect core http://twistedmatrix.com/trac/ticket/340 [#872 ] [PATCH] Unhandled exception for incomplete XMLRPC (opened by Shawn, closed by exarkun, fixed) defect http://twistedmatrix.com/trac/ticket/872 [#993 ] [PATCH] Excessive deprecation warnings from MetaInterface.__getattr__ (opened by jknight, closed by exarkun, fixed) defect core http://twistedmatrix.com/trac/ticket/993 ===== Normal ===== [#1555] website no longer features Twisted logo (opened by glyph, closed by radix, fixed) enhancement website http://twistedmatrix.com/trac/ticket/1555 [#1558] docextractor should parse C{name} etc in function summary listings (opened by spiv, closed by exarkun, fixed) enhancement website http://twistedmatrix.com/trac/ticket/1558 [#1582] move web2.dav.method.* to web2.dav.method.davfile.* (opened by wsanchez, closed by wsanchez, wontfix) enhancement web2 http://twistedmatrix.com/trac/ticket/1582 [#1556] API docs should have HTML title (opened by Tv, closed by mwh, fixed) enhancement website http://twistedmatrix.com/trac/ticket/1556 [#1557] apidocs should exclude "test" packages (opened by jknight, closed by mwh, fixed) enhancement website http://twistedmatrix.com/trac/ticket/1557 [#978 ] Document wxPython mess appropriately (opened by itamarst, closed by itamarst, fixed) enhancement core http://twistedmatrix.com/trac/ticket/978 [#477 ] More flexible profiling support (opened by exarkun, closed by exarkun, invalid) enhancement core http://twistedmatrix.com/trac/ticket/477 [#1474] [PATCH] Support for Basic Authentication in t.w.xmlrpc (opened by oubiwann, closed by exarkun, fixed) enhancement web http://twistedmatrix.com/trac/ticket/1474 [#688 ] [PATCH] robots.txt parser sub-class for use with Twisted (opened by mcfletch, closed by exarkun, wontfix) enhancement web http://twistedmatrix.com/trac/ticket/688 [#1325] [patch] Performance of Conch/SSH (opened by cooz, closed by exarkun, fixed) enhancement conch http://twistedmatrix.com/trac/ticket/1325 [#1602] smtp.SMTP would be faster using LineOnlyReceiver (opened by nickm, closed by exarkun, fixed) enhancement mail http://twistedmatrix.com/trac/ticket/1602 ===== Low ===== [#1548] get rid of epilepsy inducing entry page (opened by itamarst, closed by exarkun, fixed) enhancement website http://twistedmatrix.com/trac/ticket/1548 [#1471] replace StopTraversal with () or [] (opened by andrea, closed by dreid, wontfix) enhancement web2 http://twistedmatrix.com/trac/ticket/1471 [#1500] iocpreactor/udp.py:121 says 1234 when it could say ERROR_PORT_UNREACHABLE (opened by ghazel, closed by exarkun, fixed) enhancement core http://twistedmatrix.com/trac/ticket/1500 [#1597] Add query links to subprojects to quickly access tickets for that projet (opened by therve, closed by exarkun, fixed) enhancement website http://twistedmatrix.com/trac/ticket/1597

1 0

[Twisted-Python] Motivations of Twisted Developers & $200 Docstring Challenge
by Ed Suominen 25 Mar '06

25 Mar '06

"James Schend" <blakeyrat(a)gmail.com> wrote: > Look, all I want to be able to do is to write a program that can sit on a server for 10 years and run without me touching it. My users won't necessarily be technical and (much more likely) the program won't necessarily be maintained, so it is important that the libraries used, if any, won't break my code with updates. > > Maybe those are strange, unheard-of, constraints, but either way it looks like I have to write my own telnet server. I'd rather not, and I'm sure you'd rather have me as a user, but that's just the way it is. I'm an active user of Twisted, but even after several years, I'm still not sure whether the Twisted developers really *want* you as a user or not. I'm not saying that with any hostility; I'm actually a big fan of Twisted and those behind it. They certainly have been very helpful to me with my specific questions on IRC, and in that way it's one of the most responsive and "live" F/OSS projects around. Frankly, I think the Twisted developers understand Python and the theoretical underpinnings of software engineering better than 90% of the people developing with Python today. On the other hand, though, much of the style of development that goes on almost seems *intended* to turn off potential users, such as abandoning subprojects mid-stream in favor of a Next Big Thing that remains a moving and largely undocumented target for long periods of time, and failing, for reasons incomprehensible to me, to simply write even 2-3 lines explanations of what their New and Improved code is actually *doing* in the docstrings that Python so conveniently provides for that purpose. Ironically, one of the latest examples of this is a document extractor under development that is supposed to fix some alledgedly grevious flaws in epydoc (undetected by this user, by the way) that actually has very little source material to work with from the Twisted code itself! Another most unfortunate example is that the most powerful way to do networking with Python is without a recommendable HTTP server. The Twisted devs won't recommend twisted.web because it is "hopelessly broken" yet they have nothing released to support it. Twisted web2 remains largely undocumented and the developers advised those using it to do so at their own peril. Note to developers: serving HTTP is the biggest single use-case for Python networking! I've complained about these issues for a long time. Now I'm going to put my money where my mouth is. I hereby commit to a two hundred dollar ($200) payment to Twisted, c/o the exarkun(a)divmod.com account listed under the "Contributing" heading of http://twistedmatrix.com/trac/, in exchange for one or more of the Twisted developers committing SVN/trunk changes in the next two weeks that implement at least 100 meaningful class or method docstrings in new or active portions of the Twisted API. (I'd personally appreciate a focus on twisted.web2 as that's what I'm spending my time on right now, but this challenge is open to any active parts of the API.) That works out to $2 per docstring, and those "in the know" should be able to write them in just minutes apiece. Spend five hours writing all of them (improving "your" parts of Twisted in the process) at 20 docstrings per hour and pocket $40/hr for your efforts. It sounds like a pretty good deal to me, and hopefully to the Twisted devs as well. If this challenge is accepted and it works out well, I might even do it again -- I'm *that* interested in the ultimate success of this sometimes wonderfully crazy library/framework/Integratotron* Let me know. Best regards, Ed Suominen * See Glyph's somewhat amusing and thought-provoking blog entry of March 15: http://glyf.livejournal.com/54500.html

3 2

Re: [Twisted-Python] new pb, cPB and general PBness?
by Brian Warner 24 Mar '06

24 Mar '06

[Sorry to take so long to respond to this] Gabe Rudy <rudy(a)goldenhelix.com> writes: > So I'm happily using pb and am generally content with the performace, but > noticed the documentation mentioning that a C version of the jelly functions > might be forthcoming kind of like the cPickle library. > > What is the status of "c" pb? I also saw talk of a "new" pb implementation > (pb2?) and am wondering if the C version is being put on hold until the new > pb protocol stabilizes. The new pb talk has got me curious as well. What's > "new" or different about it? "c" PB, by which you're probably referrering to cBanana, is still around, I think. It replaces a small portion of the serialization process with a C extension. I have no idea how much this might help. The architecture of existing twisted.spread PB is not ideally suited towards replacing large parts of it with C extensions, so there is probably a limit as to how much it could be sped up this way. "newpb", which is the next-generation PB that lives in twisted.pb (in SVN), does not yet have any C extensions to speed it up, but I've tried to design it in such a way that it will be easier to write them, and to make it possible to replace more of the protocol implementation with C extensions than in oldpb. This will need to interact with the creactor/cppreactor code that Itamar has in his sandbox. But my hope is that newpb will be relatively straightforward to make really fast. I've just now created a trac/wiki page for newpb, with a brief list of features and some pointers to docs.. please take a look! http://twistedmatrix.com/trac/wiki/NewPB cheers, -Brian

2 1

Re: [Twisted-Python] **kwargs with PB
by Brian Warner 24 Mar '06

24 Mar '06

"Don Smith" <donwsmith(a)gmail.com> writes: > I'm experimenting with PB because I want to pass keyword arguments to > my remote methods, and I cannot do that with XML-RPC. But, I can't > seem to do it with PB either. I was wondering if anyone could confirm > or deny that **kwargs work with PB remote objects. Hmm, I was about to say that no, they don't work, but in looking at the code it appears that yes, they ought to work just fine. I haven't run a test to verify this, though. What kind of error message are you seeing? cheers, -Brian PS: fyi, newpb is all-kwargs, all the time. I think I put in code to handle positional args too, but only if you declare the method signature that will be used on the far end. If not, then you are restricted to using only keyword args, and positional args will cause an exception.

1 0

[Twisted-Python] on avoiding our policy: casual fixes branch
by glyph＠divmod.com 24 Mar '06

24 Mar '06

Right now, there are no exceptions to the rule of "make sure there's a ticket before you check any code in to trunk". However, several people raised some concerns on IRC over the additional friction this adds for minor fixes (the change in question being spiv's doc correction of "the the"). It's worth noting that if branching is difficult for you, it is also perfectly acceptable to attach a patch to a ticket. The important thing is that a ticket exist first, so that merging and reviewing can be done independently of committing, not that a branch actually be created in SVN. For now, because defining "trivial" is so hard (what if your docstring fix has a typo which actually changes some code which breaks some tests?), and it is always too tempting to classify "just this one" branch as "trivial" because the author wants to merge it really fast and it's not _too_ long, the proposed solution is to have a shared "minor fixes" ticket and branch, which should be reviewed once a week or so, and merged en-masse. To be clear: there are still *no exceptions* to this rule. However, please feel free to file a general-purpose ticket for minor fixes and work with as many other developers checking into the branch for that ticket.

6 5