Re: [Twisted-Python] Serializing cftp and ssh sessions
On 17 Oct, 08:50 am, axel.rau@chaos1.de wrote:
Am 14.10.2014 um 17:07 schrieb Axel Rau <Axel.Rau@chaos1.de>:
We might need more to go on. Do you have a complete, minimal example? (See http://sscce.org/) The attached zip archive Did someone could spend the time to look at it? Is my goal utopian?
I can't tell if this represents a bug in Twisted or not. After a quick look through the code, I'm not really clear on why the program would ever exit (I didn't find the calls to `reactor.stop`, I guess they're hidden somewhere in conch "library" code), nor quite which code paths you expect to work and which you expect to fail. What is the expected behavior if I run test.py after editing it to point at a host I have access to? Is that the desired behavior? If there is a second case, what one edit can I make to observe that behavior? How does the behavior differ from what you're trying to achieve? Separately: * You should rewrite this code to not use twisted.conch.client.connect. That's some random library code no one ever expected application code to use. It's public, so you can use it if you want, but that's no promise it's any good. * Use twisted.conch.endpoints.SSHCommandClientEndpoint to perform `exec` requests * Consider pitching in to complete https://twistedmatrix.com/trac/ticket/6617 so you can use the new `SSHSubsystemClientEndpoint` API that ticket introduces to set up your SFTP connections. Jean-Paul
Am 19.10.2014 um 22:44 schrieb exarkun@twistedmatrix.com:
On 17 Oct, 08:50 am, axel.rau@chaos1.de wrote:
Am 14.10.2014 um 17:07 schrieb Axel Rau <Axel.Rau@chaos1.de>:
We might need more to go on. Do you have a complete, minimal example? (See http://sscce.org/) The attached zip archive Did someone could spend the time to look at it? Is my goal utopian?
I can't tell if this represents a bug in Twisted or not. After a quick look through the code, I’m not really clear on why the program would ever exit (I didn’t find the calls to `reactor.stop`, I guess they’re hidden somewhere in conch „library“ code), I must prevent this stopping of the reactor, because I’m unable to start it again. I’m always getting twisted.internet.error.ReactorNotRestartable after trying it while reactor.running is False.
nor quite which code paths you expect to work and which you expect to fail.
What is the expected behavior if I run test.py after editing it to point at a host I have access to? If you put a reactor.stop() at the end of ssh._ebExit, test.py should run to completion after doing the remote file operations and executing the remote commands.
Is that the desired behavior? No. I need a solution where the call to executeRemoteCommands() is delayed until all remote file operations are complete. Similar the next call to distributeFiles (with a different host) must wait until the previous call to executeRemoteCommands has no outstanding operations.
If there is a second case, what one edit can I make to observe that behavior?
No.
How does the behavior differ from what you’re trying to achieve? See above. I must synchronize operations at 2 points in a loop. I tried to do this with callbacks, called at connection shutdown. What do you think about this?
Separately:
* You should rewrite this code to not use twisted.conch.client.connect. That's some random library code no one ever expected application code to use. It's public, so you can use it if you want, but that's no promise it's any good.
* Use twisted.conch.endpoints.SSHCommandClientEndpoint to perform `exec` requests
* Consider pitching in to complete https://twistedmatrix.com/trac/ticket/6617 so you can use the new `SSHSubsystemClientEndpoint` API that ticket introduces to set up your SFTP connections. I looked at endpoints before, but failed to implement public key authentication and a callback when complete.
Thanks for looking into this, Axel ——- PGP-Key:29E99DD6 ☀ +49 151 2300 9283 ☀ computing @ chaos claudius
On 02:05 pm, axel.rau@chaos1.de wrote:
On 17 Oct, 08:50 am, axel.rau@chaos1.de wrote:
Am 14.10.2014 um 17:07 schrieb Axel Rau <Axel.Rau@chaos1.de>:
We might need more to go on. Do you have a complete, minimal example? (See http://sscce.org/) The attached zip archive Did someone could spend the time to look at it? Is my goal utopian?
I can't tell if this represents a bug in Twisted or not. After a quick look through the code, I’m not really clear on why the program would ever exit (I didn’t find the calls to `reactor.stop`, I guess they’re hidden somewhere in conch „library“ code), I must prevent this stopping of the reactor, because I’m unable to start it again. I’m always getting twisted.internet.error.ReactorNotRestartable after
Am 19.10.2014 um 22:44 schrieb exarkun@twistedmatrix.com: trying it while reactor.running is False.
Okay. I guess I misunderstood part of the problem statement. I thought you wanted it to stop at some point.
nor quite which code paths you expect to work and which you expect to fail.
What is the expected behavior if I run test.py after editing it to point at a host I have access to? If you put a reactor.stop() at the end of ssh._ebExit, test.py should run to completion after doing the remote file operations and executing the remote commands.
Is that the desired behavior? No. I need a solution where the call to executeRemoteCommands() is delayed until all remote file operations are complete. Similar the next call to distributeFiles (with a different host) must wait until the previous call to executeRemoteCommands has no outstanding operations.
If there is a second case, what one edit can I make to observe that behavior? No. How does the behavior differ from what you’re trying to achieve? See above. I must synchronize operations at 2 points in a loop. I tried to do this with callbacks, called at connection shutdown. What do you think about this?
That sounds like the correct solution to me. The simplest way to structure your code is to make the APIs for starting these two operations return a Deferred that fires when the operation is complete. Then you can use these Deferreds to construct the desired control flow for starting the operations.
Separately:
* You should rewrite this code to not use twisted.conch.client.connect. That's some random library code no one ever expected application code to use. It's public, so you can use it if you want, but that's no promise it's any good.
* Use twisted.conch.endpoints.SSHCommandClientEndpoint to perform `exec` requests
* Consider pitching in to complete https://twistedmatrix.com/trac/ticket/6617 so you can use the new `SSHSubsystemClientEndpoint` API that ticket introduces to set up your SFTP connections.
I looked at endpoints before, but failed to implement public key authentication and a callback when complete.
You should be able to do key-based authentication with the endpoint. SSHCommandClientEndpoint.newConnection accepts a `keys` argument if you want to specify the keys to use explicitly. It also accepts an `agentEndpoint` argument if you instead want to point it at an SSH key agent which will perform key operations for it. Jean-Paul
Am 20.10.2014 um 16:28 schrieb exarkun@twistedmatrix.com:
See above. I must synchronize operations at 2 points in a loop. I tried to do this with callbacks, called at connection shutdown. What do you think about this?
That sounds like the correct solution to me. The simplest way to structure your code is to make the APIs for starting these two operations return a Deferred that fires when the operation is complete. Then you can use these Deferreds to construct the desired control flow for starting the operations.
I have implemented this in a prototype, which does what I want, simply by calling the next top level operation when one has completed. Your suggestion would require to handle the Deferred down to the point of connection termination. Thanks for your support, Axel --- PGP-Key:29E99DD6 ☀ +49 151 2300 9283 ☀ computing @ chaos claudius
participants (2)
-
Axel Rau -
exarkun＠twistedmatrix.com