Mailman 3 February 2006 - Twisted

Re: [Twisted-Python] Conch: multiple commands
by Brendan Simon 10 Feb '06

10 Feb '06

twisted-python-request(a)twistedmatrix.com wrote: > Date: Thu, 09 Feb 2006 10:14:20 -0500 > From: Paul Swartz <paulswartz(a)gmail.com> > > Brendan Simon wrote: > >> I'm trying to use and understand the latest Twisted and Conch (I'm a >> newbie). I can get teh sample apps going fine but they all seem to send >> one command and then exit. I'm trying to workout how to send multiple >> commands to the server from the same client application. >> >> The sshsimpleclient.py does send multiple commands (false, true, cat) >> with separate channels. >> Is that the preferred way to send multiple commands? >> Doesn't that require some setup overhead every time I want to issue a >> command to the server? >> > > That's the way SSH requires sending multiple commands. It does require > some server overhead, but there's no way around that. > So what does a normal ssh client (eg openssh) do when I connect to an sshd server to get to a bash session ??? Maybe I just answered my own question. The sshd server runs "login" (or something) which in turn looks in /etc/passwd for the user and shell, and then invokes that shell (eg. bash). Is that the way it works? i.e. the ssh client only sends _one_ command and as far as it is concerned is only invoked one program and it doesn't care that the server has fired off many apps. If that's the case, then maybe I can invoke bash as my command, and just write data to the channel and grab the results ??? Is that good or bad??? Thanks, Brendan.

2 1

[Twisted-Python] Conch: multiple commands
by Brendan Simon 09 Feb '06

09 Feb '06

I'm trying to use and understand the latest Twisted and Conch (I'm a newbie). I can get teh sample apps going fine but they all seem to send one command and then exit. I'm trying to workout how to send multiple commands to the server from the same client application. The sshsimpleclient.py does send multiple commands (false, true, cat) with separate channels. Is that the preferred way to send multiple commands? Doesn't that require some setup overhead every time I want to issue a command to the server? Is it possible to open _one_ channel, and send commands in a synchronously? ie. wait for commands to return output and then issue other commands if necesary?? Maybe this goes against the grain of the Twisted Asynchronous Framework ??? If I assume that I have to open a new channel for each command, and I have a 100 different commands, I could have 100 classes for each command (seems a bit extreme) or have one class that gets the command passed to it. The problem I see with the latter is that I don't know how to wait until the first command has got a reply and is finished. It seems that the channelOpen method returns straight away. How can I control the program execution to wait for the response, error or timeout? Example: def servicesStarted(self): self.openChannel(CommandChannel.openChannel('ls /tmp/dir1') # how do I wait for list of dir1 to finish? self.openChannel(CommandChannel.openChannel('ls /tmp/dir2') # how do I wait for list of dir2 to finish? self.openChannel(CommandChannel.openChannel('ls /tmp/dir3') # how do I wait for list of dir3 to finish? Thanks, Brendan.

4 5

Re: [Twisted-Python] Conch: multiple commands
by Brendan Simon 09 Feb '06

09 Feb '06

Paul Swartz <paulswartz(a)gmail.com> wrote: >> Is it possible to open _one_ channel, and send commands in a >> synchronously? ie. wait for commands to return output and then issue >> other commands if necesary?? >> Maybe this goes against the grain of the Twisted Asynchronous Framework ??? >> > You can send one command at a time, but you still have to write the code > asynchronously. > I think this is half my problem, working out how to use the asynchronous framework properly ;-) >> def servicesStarted(self): >> self.openChannel(CommandChannel.openChannel('ls /tmp/dir1') >> # how do I wait for list of dir1 to finish? >> self.openChannel(CommandChannel.openChannel('ls /tmp/dir2') >> # how do I wait for list of dir2 to finish? >> self.openChannel(CommandChannel.openChannel('ls /tmp/dir3') >> # how do I wait for list of dir3 to finish? >> > What I'd do is also pass a Deferred to CommandChannel, that is > call/errbacked when the command finishes. > > def servicesStarted(self): > d = defer.Deferred() > d.addCallback(self._cbFirstLs) > d.addErrback(self._ebFirstLs) > self.openChannel(CommandChannel(d, "ls /tmp/dir1")) > > def _cbFirstLs(self, result): > print 'directory listing', result > d = defer.Deferred() > self.openChannel(CommandChannel(d, "ls /tmp/dir2")) > > def _ebFirstLs(self, f): > log.err() > > etc., for all the commands you want to run. > Thanks. I'll play with this a little more. Guess I may need some kind of state machine in the callback function if different unrelated commands are to be sent at different times based on various application events. Does that sound like a good way to use Twisted??? Thanks, Brendan.

1 0

[Twisted-Python] sprinting at PyCon
by Duncan McGreggor 09 Feb '06

09 Feb '06

From an email exchange with dreid: > From: David Reid <dreid(a)dreid.org> > Date: February 9, 2006 1:04:30 PM MST > To: Duncan McGreggor <duncan.mcgreggor(a)gmail.com> > Subject: Re: t.w.xmlrpc Basic Auth patch > >>> Help is always great, unfortunately the web2 client isn't _done_ yet. >>> It has pretty decent test coverage and works fairly well but doesn't >>> have any high level api. I'm not sure when that stuff will get done. >>> If you're interested in the XML-RPC server side that'd be great. :) >> >> Awesome. Perfect for sprinting at pycon. > > I don't think foom or I are going to be there. Perhaps some benefit > might come of organizing a virtual sprint to coincide with pycon's > sprints. :( I will be at pycon this year. I had planned on sprinting with some of my own projects, but I can do that anytime. I'd much rather do some group work, even if it's virtual sprinting with those that can't be there. Is there any interest in a sprint/vsprint? In particular, I want to help with t.w2 xmlrpc (server, maybe some client work). Any thoughts? What about #twisted.pycon or some such? d

1 0

[Twisted-Python] Deferred.callback() v. Deferred.called
by Lenny G Arbage 09 Feb '06

09 Feb '06

Jean-Paul Calderone exarkun at divmod.com wrote: >>def waitForDeferredResult(d): >> while not d.called: >> reactor.iteratoe(0.1) >> return d.result >> > > This function is broken. d.called isn't really > public. d.result isn't really public. > reactor.iterate() isn't really public. > You cannot poll a Deferred like this and expect > it to work. Lots of people have tried. It > just can't be done. Okay, I have repented and waitForDeferredResult() has been terminated. I will return to saner methods. BTW, trial is really nice. You may find it somewhat ironic that I actually got the idea for using reactor.iterate() from the trial howto (http://twisted.sourceforge.net/TwistedDocs-1.2.0rc3/howto/test-standard.html) Lenny __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

[Twisted-Python] Re: [Twisted-commits] r15860 - Move checkPreconditions() from web2.dav.static.DAVFile to web2.resource.RenderMixIn.
by James Y Knight 09 Feb '06

09 Feb '06

On Feb 8, 2006, at 7:49 PM, Wilfredo Sanchez wrote: > Move checkPreconditions() from web2.dav.static.DAVFile to > web2.resource.RenderMixIn. > Add test suite (ported foom's tests for server.checkpreconditions()). > Fix some bugs found by the tests. (Holy cow, checkpreconditions > are a PITA.) I don't like this part. You've made the precondition check code much longer, IMO harder to understand (figuring out how the return values of one method influenced the rest of the execution is not easy), and pinned it to being part of Resource, rather than being reusable. I'm not sure of the need, given that the previous implementation was callable from Resource as it was. Why not just use it... Nextly, I'm somewhat concerned about the change to calling the precondition check before GET rendering instead of afterwards: I think your use cases are probably not typical. I believe most people will write resources which generate dynamic content, and if they're _really good_ about it, they'll simultaneously generate an etag and a last-modified time. This is also the only way that such information can come out of wsgi, cgi, proxies to other servers, etc. The way things were set up before made Not Modified responses just "magically" show up correctly when you set those headers in your response. No more interface was required than that, which means that the response could come from anywhere. Now, instead, you force the author to implement resource.etag() and resource.lastModified() in order to get that behavior. It's really easy for file-like resources to do so, but not so much so for most other kinds. And the thing is, it's OK to check afterwards for the kinds of requests a web browser makes: GET/HEAD/POST. GET/HEAD don't change things so it's acceptable to do their check at any time, and while POST does change things, as a practical matter nobody actually sends conditionals on post requests, because what would it be useful for? (and nothing would be set up to handle such a thing). However, I'm currently ignoring conditionals on PostableResource's http_POST, which while okay as far as real-world interoperability goes, it is certainly not _right_. And to make things worse, the post- processing filter doesn't confine itself to checking only GET/HEAD requests, so the client will get back a precondition failed message, even though the method was executed. At the very least a POST with an If-Unmodified-Since/If-Match should return precondition failed. If there was code on Resource to handle that automatically, I could not have forgetten to do it. So, um, I dunno what to conclude. :) James

1 1

[Twisted-Python] Deferred.callback() v. Deferred.called
by Lenny G Arbage 08 Feb '06

08 Feb '06

I think I'm seeing a difference between when a deferred's callback is invoked and when the .called member variable is set, and was hoping someone might be able to explain. (And let me apologize in advance if what I'm trying to do is hairbrained or in some way abusive to the methodology...) First, a little context. I have some code that I use to test with, and it follows the standard deferred chaining pattern, i.e., def alldone(res): pass def trial3(res, ...): ... d.addCallback(alldone) ... def trial2(res, ...): ... d.addCallback(trial3) ... def trial1(...): ... d = someFunc1(...) d.addCallback(trial2) ... def maintest(): trial1(...) ...and I wanted to try putting all the trial invocations down in maintest, but still wait for the result of a preceeding test before invoking the next, so I made up a little function that does, basically, def waitForDeferredResult(d): while not d.called: reactor.iteratoe(0.1) return d.result ...and then, def maintest(): d1 = trial1() res1 = waitForDeferredResult(d1) d2 = trial2(res1) res2 = waitForDeferredResult(d2) ... Perhaps this is hairbrained, as the first strategy works perfectly, but it sure would make it easier to comment out one or two (or all but one or two) tests when I just want to focus on hacking on that functionality. However, the while loop in waitForDeferredResult ends (.called == True) before the callback is actually called in the case where the the Deferred in the trial is also chained. For example, if someFunc1() is set up something like this: someFunc1(): d = someFunc1-1() return d someFunc1-1(): d = threads.deferToThread(someThing) d.addCallback(someFunc1-2) return d someFunc1-2(): d = threads.deferToThread(someOtherThing) return d ...then it appears that .called becomes true after the deferred in someFunc1-1() returns, but before someFunc1-2() is invoked. If, on the other hand, I chain from one trialX() to the next (as in the top example) the callback isn't invoked until after someFunc1-2() returns. Anyone know why? Thanks, Lenny __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

3 2

[Twisted-Python] Weekly Bug Summary
by James Y Knight 08 Feb '06

08 Feb '06

Bug summary ______________________ Since 2006-02-01 06:00:02.340000: Bugs opened: 13 Bugs closed: 5 Total open bugs: 553 (+8) Urgent: 3 (+0) Bug: 268 (+4) Feature: 188 (+1) Wish: 94 (+3) New / Reopened Bugs ______________________ ==== Bug ==== iocpreactor ignores pauseProducing from inside dataReceived (3 hours ago) http://twistedmatrix.com/bugs/issue1473 opened by ghazel [TESTS, TRIAL] test_reporter.TestErrorReporting.test_timing uses clock and tests many things (4 days ago) http://twistedmatrix.com/bugs/issue1470 opened by jml fix twisted.web2 log format to comply with apache (4 days ago) http://twistedmatrix.com/bugs/issue1469 opened by andrea [CONCH] Conch FileTransferClient breaks on STATUS packet without message (6 days ago) http://twistedmatrix.com/bugs/issue1463 opened by bwh fix exception in twisted.web2 makeUnsatisfiable (5 days ago) http://twistedmatrix.com/bugs/issue1465 opened by andrea [DOCUMENTATION] small bug in the example in documentation/howto (6 days ago) CLOSED http://twistedmatrix.com/bugs/issue1464 opened by andyl [WEB2] static.Data fix (6 days ago) CLOSED http://twistedmatrix.com/bugs/issue1462 opened by andrea win32reactor spawnProcess incorrectly escapes cmd line (6 months ago) http://twistedmatrix.com/bugs/issue1123 opened by justinj ==== Feature ==== [WEB2] Additional address(es) in access.log (4 days ago) http://twistedmatrix.com/bugs/issue1468 opened by jwashin ==== Wish ==== [CORE, WIN32] Consistent cross-platform handling of connectTCP(INADDR_ANY, ...) (yesterday) http://twistedmatrix.com/bugs/issue1472 opened by glyph replace StopTraversal with () or [] (yesterday) http://twistedmatrix.com/bugs/issue1471 opened by andrea [WEBSITE] TwistedWeb2 version mismatch (0.1 / 0.1.0) (4 days ago) http://twistedmatrix.com/bugs/issue1467 opened by doko [WEBSITE] one html page to get an overview of current released versions (4 days ago) http://twistedmatrix.com/bugs/issue1466 opened by doko Bugs Closed ______________________ ==== Bug ==== [CORE, WIN32] spawnProcess in default reactor for win32 is broken (1 week ago) http://twistedmatrix.com/bugs/issue1457 opened by justinj, closed by glyph [CORE, WIN32] twisted.internet.process.Process and twisted.internet.win32eventreactor.Process do not implement IConsumer - also they don't work (19 months ago) http://twistedmatrix.com/bugs/issue654 opened by glyph, closed by exarkun [DEFERRED, DOCUMENTATION] Error in documentation (deferred timeout example) at http://twistedmatrix.com/projects/core/documentation/howto/deferrecipes.html (3 months ago) http://twistedmatrix.com/bugs/issue1289 opened by szumo, closed by exarkun [DOCUMENTATION] small bug in the example in documentation/howto (6 days ago) http://twistedmatrix.com/bugs/issue1464 opened by andyl, closed by spiv [WEB2] static.Data fix (6 days ago) http://twistedmatrix.com/bugs/issue1462 opened by andrea, closed by wsanchez

1 0

[Twisted-Python] help with deferreds??
by john peter 07 Feb '06

07 Feb '06

i'm writing an app that makes two twisted client connections to the same server. After each connection is made, each client connection sends a session establishment request to the server (per Netbios over TCP/IP protocol ) in the Protocol.connectionMade() method. In the dataReceived method, I check if my request was successful. If not successful, I display a failure notice then call self.transport.loseConnection(). if successful, I flip an internal variable "sessionEstablished" to true, then the intention is for the successful connection to continually interact with the server in a loop. this means, however, that i would get into a send-receive data loop, which as i've observed, would prevent the other successful connection to get any work done.. would it still be appropriate for me to use twisted for such application? if so, how should i approach the sendDataToServer() blocking. I'm read the docs on deferreds, but i'm still unsure how to proceed. thanks in advance for any help/tips --------------------------------- Do you Yahoo!? With a free 1 GB, there's more in store with Yahoo! Mail.

3 7

[Twisted-Python] Re: Making DeferredList look like Deferred
by Lenny G Arbage 07 Feb '06

07 Feb '06

Thomas HERVE therve at free.fr wrote: > > But really, I don't see the point of this. > If you want to have an exception somewhere, > use fireOnOneErrback. Else there's no need > to build the result list, if you're > actually raising an Exception for *one* > error. What if the desired exception handling should occur *after* all the Deferreds have fired, regardless of whether they fired err or callback? This is particularly useful for p2p-style apps. For example, suppose I want to maintain a maximum of 20 outstanding requests for a given operation -- I don't want to spin off another 20 (or another X) until I know exactly how many of the original 20 failed and how many succeeded. (This type of operation is common, for instance, when you are doing erasure coding or shamir secret sharing, where you really only need Y out of Z requests to succeed, but where you do need to respawn W new requests if you didn't get Y after the first round.) fireOnOneErrback doesn't seem to give me that functionality -- it gets fired only once, for the first failure. I don't know which other of the 20 requests failed or will fail, and there may be outstanding requests that will yet succeed that I'll never learn about. I think I'm beginning to see, though, that this is all accounted for by just letting the DeferredList's callback get invoked, and wrapping it similarly to what you posted. My only complaint, as you pointed out, is that this seems a bit clumsy. However, cluttering up the API with another special flag to handle my border case where I want the DeferredList's errback to be invoked just like the callback, but only if => 1 of its Deferreds errback'ed, may be even worse. It sure would be handy for me, however, so I'll likely create my own SpecialDeferredList that does the wrapping behind the scenes and fires the back() that I expect. Unless, of course, there is a better way. Here's what I'm currently using to wrap: def DeferredListToDeferred(result): print " result was: %s" % result for code, res in result: if code == False: # assumes failed results have # (False, failure) pairs, and # only returns the first of # these. return res return result __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1