[Twisted-Python] Effects of calling transport.writeSomeData() ?

We're writing a Twisted 14.0.0 application (on Python 2.7.7, Mac OS 10.9.3) that uses Conch as an SSH client; this is working fine. However, we have the requirement that in an advanced mode of operation for power users that the application take advantage of OpenSSH connection multiplexing over an already-established-by-the-user OpenSSH ControlMaster session (via an OpenSSH ControlPath socket) instead of using Conch. OpenSSH requires its new session command and forwarded file descriptors to be sent over the socket in a very particular way: the command must be sent first, followed by message with a '\0' byte with each forwarded file descriptor. OpenSSH ignores the '\0' for each file descriptor, extracting the file descriptors themselves from the message's ancillary data. The following will not work because none of the calls to write() send their data until control is returned to the reactor, while sendFileDescriptor() queues up the descriptors such that they get sent them with the very next data that is sent -- which wind up being the first three bytes of the command rather than the three '\0' bytes. class OpenSSHMuxProtocol( protocol.Protocol ): # built via reactor.connectUNIX() def sendCommand( self, command ): # Does not work: self.transport.write( command ) self.transport.sendFileDescriptor( sys.stdin.fileno() ) self.transport.write( '\0' ) # payload for the stdin file descriptor self.transport.sendFileDescriptor( sys.stdout.fileno() ) self.transport.write( '\0' ) # payload for the stdout file descriptor self.transport.sendFileDescriptor( sys.stdout.fileno() ) self.transport.write( '\0' ) # payload for the stderr file descriptor # ^^^ Does not work But this next solution /does/ work: from socket import SOL_SOCKET from twisted.python.sendmsg import SCM_RIGHTS, send1msg class OpenSSHMuxProtocol( protocol.Protocol ): # built via reactor.connectUNIX() def sendCommand( self, command ): self.transport.writeSomeData( command ) # data is sent over the socket immediately send1msg( self.transport.socket.fileno(), "\0", 0, [ ( SOL_SOCKET, SCM_RIGHTS, pack( 'i', sys.stdin.fileno() ) ) ] ) send1msg( self.transport.socket.fileno(), "\0", 0, [ ( SOL_SOCKET, SCM_RIGHTS, pack( 'i', sys.stdout.fileno() ) ) ] ) send1msg( self.transport.socket.fileno(), "\0", 0, [ ( SOL_SOCKET, SCM_RIGHTS, pack( 'i', sys.stderr.fileno() ) ) ] ) My questions are: Is it bad to bypass the reactor and send data directly/immediately this way using writeSomeData() and send1msg()? Note that sendCommand() actually gets called in response to a OpenSSHMuxProtocol.dataReceived() event. If bypassing the reactor this way is bad, how bad is it and what are the consequences or effects? Is there a better way to get a working solution? I think I'd need some way to guarantee that the write of the command was actually sent to the OpenSSH server before the file descriptors are forwarded -- for example, if a Deferred was used whose first callback wrote the command and whose second callback forwarded the descriptors, would a call to the reactor to actually sent the command be guaranteed between the two callbacks? Any information and/or advice is appreciated! -- Mark Montague mark@catseye.org

On 20 Jun, 10:22 pm, mark@catseye.org wrote:
We're writing a Twisted 14.0.0 application (on Python 2.7.7, Mac OS 10.9.3) that uses Conch as an SSH client; this is working fine. However, we have the requirement that in an advanced mode of operation for power users that the application take advantage of OpenSSH connection multiplexing over an already-established-by-the-user OpenSSH ControlMaster session (via an OpenSSH ControlPath socket) instead of using Conch.
OpenSSH requires its new session command and forwarded file descriptors to be sent over the socket in a very particular way: the command must be sent first, followed by message with a '\0' byte with each forwarded file descriptor. OpenSSH ignores the '\0' for each file descriptor, extracting the file descriptors themselves from the message's ancillary data.
The following will not work because none of the calls to write() send their data until control is returned to the reactor, while sendFileDescriptor() queues up the descriptors such that they get sent them with the very next data that is sent -- which wind up being the first three bytes of the command rather than the three '\0' bytes.
class OpenSSHMuxProtocol( protocol.Protocol ): # built via reactor.connectUNIX() def sendCommand( self, command ): # Does not work: self.transport.write( command ) self.transport.sendFileDescriptor( sys.stdin.fileno() ) self.transport.write( '\0' ) # payload for the stdin file descriptor self.transport.sendFileDescriptor( sys.stdout.fileno() ) self.transport.write( '\0' ) # payload for the stdout file descriptor self.transport.sendFileDescriptor( sys.stdout.fileno() ) self.transport.write( '\0' ) # payload for the stderr file descriptor # ^^^ Does not work
But this next solution /does/ work:
from socket import SOL_SOCKET from twisted.python.sendmsg import SCM_RIGHTS, send1msg
class OpenSSHMuxProtocol( protocol.Protocol ): # built via reactor.connectUNIX() def sendCommand( self, command ): self.transport.writeSomeData( command ) # data is sent over the socket immediately send1msg( self.transport.socket.fileno(), "\0", 0, [ ( SOL_SOCKET, SCM_RIGHTS, pack( 'i', sys.stdin.fileno() ) ) ] ) send1msg( self.transport.socket.fileno(), "\0", 0, [ ( SOL_SOCKET, SCM_RIGHTS, pack( 'i', sys.stdout.fileno() ) ) ] ) send1msg( self.transport.socket.fileno(), "\0", 0, [ ( SOL_SOCKET, SCM_RIGHTS, pack( 'i', sys.stderr.fileno() ) ) ] )
My questions are:
Is it bad to bypass the reactor and send data directly/immediately this way using writeSomeData() and send1msg()? Note that sendCommand() actually gets
Yes. `writeSomeData` is not a method on any transport interface. It is an implementation detail of particular transports.
called in response to a OpenSSHMuxProtocol.dataReceived() event. If bypassing the reactor this way is bad, how bad is it and what are the consequences or effects?
This use is untested. There's no reason to expect it will continue to work with future Twisted releases (or, really, that it fully works now; since `writeSomeData` bypasses the transport's buffering layer, it seems like you're risking an out-of-order or partial send; probably these will only arise under load so you may not have observed them in your testing).
Is there a better way to get a working solution? I think I'd need some way to guarantee that the write of the command was actually sent to the OpenSSH server before the file descriptors are forwarded -- for example, if a Deferred was used whose first callback wrote the command and whose second callback forwarded the descriptors, would a call to the reactor to actually sent the command be guaranteed between the two callbacks?
The proper way to do this would be for OpenSSH to acknowledge the operation. At this point you would know it's safe to proceed to the next operation. Since you didn't mention anything about acknowledgements, I'm guessing there are none. Since you're already relying on `self.transport.socket.fileno()` and `send1msg` (basically, bypassing the transport abstraction and just doing socket operations yourself) one improvement you could make would be just to rely on that for the whole thing. Don't use `writeSomeData`. Use `socket.send(command)`. At least this way you're only relying on being able to treat a transport like a UNIX socket - not on the particulars of the transport's buffering implementation. A different approach you could take would be to implement this connection sharing feature for Conch. I can pretty much guarantee it's possible to implement since an older version of Conch actually did implement it. :) The implementation was removed because it was fragile, complicated, and poorly tested. It would be great to re-introduce the functionality with a higher quality implementation. Jean-Paul

On 2014-06-21, 10:24, exarkun@twistedmatrix.com wrote:
If bypassing the reactor this way is bad, how bad is it and what are the consequences or effects?
This use is untested. There's no reason to expect it will continue to work with future Twisted releases (or, really, that it fully works now; since `writeSomeData` bypasses the transport's buffering layer, it seems like you're risking an out-of-order or partial send; probably these will only arise under load so you may not have observed them in your testing).
Thanks for the very helpful reply! This is exactly the sort of thing I was hoping to find out.
Is there a better way to get a working solution?
The proper way to do this would be for OpenSSH to acknowledge the operation. At this point you would know it's safe to proceed to the next operation. Since you didn't mention anything about acknowledgements, I'm guessing there are none.
Correct, the order of events is: 0. client opens OpenSSH socket, sends hello, receives reply 1. user calls the protocol function to create a new OpenSSH session over the existing socket 2. client sends MUX_C_ALIVE_CHECK to OpenSSH 3. OpenSSH replies with MUX_S_ALIVE 4. client sends MUX_C_NEW_SESSION to OpenSSH, then immediately forwards the file descriptors 5. OpenSSH replies with either MUX_S_SESSION_OPENED, MUX_S_PERMISSION_DENIED, or MUX_S_FAILURE. 6. ...session is now created... 7. OpenSSH sends MUX_S_EXIT_MESSAGE when the session has finished 8. client can now close the OpenSSH socket It would make sense for OpenSSH to acknowledge the MUX_C_NEW_SESSION in step 4 before the file descriptors get sent, but I've read the source code and OpenSSH does not do this -- it expects the file descriptors to be sent as soon as the new session message is sent.
Since you're already relying on `self.transport.socket.fileno()` and `send1msg` (basically, bypassing the transport abstraction and just doing socket operations yourself) one improvement you could make would be just to rely on that for the whole thing. Don't use `writeSomeData`. Use `socket.send(command)`. At least this way you're only relying on being able to treat a transport like a UNIX socket - not on the particulars of the transport's buffering implementation.
A different approach you could take would be to implement this connection sharing feature for Conch. I can pretty much guarantee it's possible to implement since an older version of Conch actually did implement it. :) The implementation was removed because it was fragile, complicated, and poorly tested. It would be great to re-introduce the functionality with a higher quality implementation.
I'll look into what it might take to implement this in a robust way with tests for Conch. I think that's definitely a preferred solution, since it sounds like such a feature might be accepted (I wasn't considering this initially since Conch is a generic SSH transport, and piggybacking on top of an existing OpenSSH socket is a highly-specific thing.) If I can come up with a good way to do this, I'll submit a patch, otherwise I'll keep the functionality as it is now, in a standalone class that doesn't use Conch at all. -- Mark Montague mark@catseye.org

On Jun 24, 2014, at 10:01 AM, Mark Montague <mark@catseye.org> wrote:
(I wasn't considering this initially since Conch is a generic SSH transport, and piggybacking on top of an existing OpenSSH socket is a highly-specific thing.)
Not only was this feature previously in Conch (although I'm not sure if it actually spoke OpenSSH's ControlMaster protocol or something custom?), but there's existing precedent for integrating with OpenSSH-based client-side tooling: Conch can talk to OpenSSH's agent via the agent protocol. -glyph

On 24 Jun, 05:01 pm, mark@catseye.org wrote:
On 2014-06-21, 10:24, exarkun@twistedmatrix.com wrote:
A different approach you could take would be to implement this connection sharing feature for Conch. I can pretty much guarantee it's possible to implement since an older version of Conch actually did implement it. :) The implementation was removed because it was fragile, complicated, and poorly tested. It would be great to re- introduce the functionality with a higher quality implementation.
I'll look into what it might take to implement this in a robust way with tests for Conch. I think that's definitely a preferred solution, since it sounds like such a feature might be accepted (I wasn't considering this initially since Conch is a generic SSH transport, and piggybacking on top of an existing OpenSSH socket is a highly-specific thing.)
To clarify slightly, what was implemented in Conch before was the entire connection sharing feature. Conch could act as a master and share connections with other Conch clients. It could not talk to an OpenSSH master and to share a connection. Jean-Paul
participants (3)
-
exarkun@twistedmatrix.com
-
Glyph Lefkowitz
-
Mark Montague