[Twisted-Python] Understanding the IOCP reactor and adding spawnProcess
I am attempting to add spawnProcess to iocpreactor. In order to begin this task I've had to do a lot of reading on Windows network programming, specifically the various Windows I/O methods, to attempt to understand what win32eventreactor and iocpreactor are doing, and also just increase my understanding of how reactors work in general. To understand the various Winsock 2 methods that both of these reactors rely upon, I read chapters 1-5 of Network Programming for Microsoft Windows[1]. Before actually attempting to add spawnProcess, I would like to present how I think iocpreactor works and how I think I should add spawnProcess, and hopefully be corrected or confirmed in my understanding. If I'm too vague there's a good chance it's because I don't understand it very well. Please feel free to point out things that you might think are obvious but aren't sure I understand. How iocpreactor works --------------------------------- 1. Create an IO Completion Port. 2. Create a socket and associate it with the IOCP. This is the socket we will call AcceptEx (a non-blocking accept) on. The association with the IOCP is made via CreateIoCompletionPort. 3. Setup any scheduled tasks on the reactor. 4. Call AcceptEx (which doesn't block) on the socket. AcceptEx takes an overlapped structure as a parameter. Before making the call, we set two attributes of the struct: the callback and callback_args which will be called when an accept event completes on the socket. The Winsock 2 methods don't actually call the callback. The Winsock 2 methods handle copying data related to the network event that occurred on the socket into the overlapped structure and making that overlapped structure available to GetQueuedCompletionStatus. So when we handle events on sockets via GetQueuedCompletionStatus from within doIteration, we have access to the data related to the event as well as the callback and callback_args we call to handle that event. The callbacks are setup in the xxxOp classes in ops.py and always result in some transport method getting called (such as readDone, connectionDone, etc). 5. From within doIteration, call GetQueuedCompletionStatus (which does block) with a timeout of the time until the next scheduled task needs to be run. If any event occurs on the sockets currently associated with the IOCP before that time expires, GetQueuedCompletionStatus will return (stop blocking). Now we have access to the overlapped structure containing data associated with the event which was copied into the overlapped structure's buffer, such as data received from WSARecv calls, as well as the callback and callback_args. From within doIteration we call the callbacks passing in the data related to the event. Depending on the events we are handling, we may create new sockets (e.g. end point sockets in TCP connections) and associate them with the IOCP as well. All Winsock 2 API calls made are non-blocking accept for GetQueuedCompletionStatus. 6. Step 5 continues until the reactor stops. How to add spawnProcess --------------------------------------- 1. Create the processes via Windows APIs and associate their stdout/err with with the IOCP via CreateIoCompletionPort calls. 2. Close stdin. 3. Notify the ProcessProtocol via protocol.makeConnection (not sure why, looking at win32eventreactor) 4. Receive data from stdout/err via the completion port by calling GetQueuedCompletionStatus from within doIteration. Is this really possible? ProcessProtocol's methods won't get called appropriately by letting the existing callbacks in ops.py make calls to the transport (e.g. connectionDone, readDone)? Thanks for your help. Justin [1] http://www.amazon.com/exec/obidos/ASIN/0735615799
On Mon, 11 Jul 2005 10:52:14 -0500, Justin Johnson <justinjohnson@gmail.com> wrote:
I am attempting to add spawnProcess to iocpreactor. In order to begin this task I've had to do a lot of reading on Windows network programming, specifically the various Windows I/O methods, to attempt to understand what win32eventreactor and iocpreactor are doing, and also just increase my understanding of how reactors work in general. To understand the various Winsock 2 methods that both of these reactors rely upon, I read chapters 1-5 of Network Programming for Microsoft Windows[1]. Before actually attempting to add spawnProcess, I would like to present how I think iocpreactor works and how I think I should add spawnProcess, and hopefully be corrected or confirmed in my understanding. If I'm too vague there's a good chance it's because I don't understand it very well. Please feel free to point out things that you might think are obvious but aren't sure I understand. How iocpreactor works ---------------------------------
[snip]
How to add spawnProcess ---------------------------------------
1. Create the processes via Windows APIs and associate their stdout/err with with the IOCP via CreateIoCompletionPort calls. 2. Close stdin.
Why close stdin? How will you write to the spawned process?
3. Notify the ProcessProtocol via protocol.makeConnection (not sure why, looking at win32eventreactor)
makeConnection is the method that's actually part of IProtocol (even though ProcessProtocol doesn't implement IProtocol, it still adheres to its API in this regard). Generally, all it does is set the .transport attribute on the protocol instance and then call connectionMade. This might seem pointless in the case of processes, but since it maintains consistency with other kinds of transports, it is useful.
4. Receive data from stdout/err via the completion port by calling GetQueuedCompletionStatus from within doIteration. Is this really possible? ProcessProtocol's methods won't get called appropriately by letting the existing callbacks in ops.py make calls to the transport (e.g. connectionDone, readDone)?
Thanks for your help. Justin [1] http://www.amazon.com/exec/obidos/ASIN/0735615799
Jp
See response below. On 7/11/05, Jp Calderone <exarkun@divmod.com> wrote:
I am attempting to add spawnProcess to iocpreactor. In order to begin
On Mon, 11 Jul 2005 10:52:14 -0500, Justin Johnson < justinjohnson@gmail.com> wrote: this
task I've had to do a lot of reading on Windows network programming, specifically the various Windows I/O methods, to attempt to understand what win32eventreactor and iocpreactor are doing, and also just increase my understanding of how reactors work in general. To understand the various Winsock 2 methods that both of these reactors rely upon, I read chapters 1-5 of Network Programming for Microsoft Windows[1]. Before actually attempting to add spawnProcess, I would like to present how I think iocpreactor works and how I think I should add spawnProcess, and hopefully be corrected or confirmed in my understanding. If I'm too vague there's a good chance it's because I don't understand it very well. Please feel free to point out things that you might think are obvious but aren't sure I understand. How iocpreactor works ---------------------------------
[snip]
How to add spawnProcess ---------------------------------------
1. Create the processes via Windows APIs and associate their stdout/err with with the IOCP via CreateIoCompletionPort calls. 2. Close stdin.
Why close stdin? How will you write to the spawned process?
I see, so I won't close stdin and people can do that in their ProcessProtocol in connectionMade or something if they want.
3. Notify the ProcessProtocol via protocol.makeConnection (not sure
why, looking at win32eventreactor)
makeConnection is the method that's actually part of IProtocol (even though ProcessProtocol doesn't implement IProtocol, it still adheres to its API in this regard). Generally, all it does is set the .transport attribute on the protocol instance and then call connectionMade. This might seem pointless in the case of processes, but since it maintains consistency with other kinds of transports, it is useful.
So the Process class is the transport? In win32eventreactor, 3 threads are started that loop on reading stdout and stderr and writing data from the outQueue to stdin. It seems to me that I would not use these threads, but instead have each of these 3 file handles associated with the IOCP. Then the methods defined in ops.py would be used to handle events on those handles, resulting in transport methods being called. These transport methods would be methods on the Process class, which would result in methods on a single instance of the ProcessProtocol being called. This single instance of the ProcessProtocol would be shared by all of the 3 file handles. Am I understanding correctly? Are write and loseConnection the only methods the Process would need to implement from ITransport?
GetQueuedCompletionStatus from within doIteration. Is this really
4. Receive data from stdout/err via the completion port by calling possible?
ProcessProtocol's methods won't get called appropriately by letting the existing callbacks in ops.py make calls to the transport (e.g. connectionDone, readDone)?
Thanks for your help. Justin [1] http://www.amazon.com/exec/obidos/ASIN/0735615799
Jp
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Mon, 11 Jul 2005 12:16:35 -0500, Justin Johnson <justinjohnson@gmail.com> wrote:
On 7/11/05, Jp Calderone <exarkun@divmod.com> wrote:
[snip]
3. Notify the ProcessProtocol via protocol.makeConnection (not sure
why, looking at win32eventreactor)
makeConnection is the method that's actually part of IProtocol (even though ProcessProtocol doesn't implement IProtocol, it still adheres to its API in this regard). Generally, all it does is set the .transport attribute on the protocol instance and then call connectionMade. This might seem pointless in the case of processes, but since it maintains consistency with other kinds of transports, it is useful.
So the Process class is the transport? In win32eventreactor, 3 threads are started that loop on reading stdout and stderr and writing data from the outQueue to stdin. It seems to me that I would not use these threads, but instead have each of these 3 file handles associated with the IOCP. Then the methods defined in ops.py would be used to handle events on those handles, resulting in transport methods being called. These transport methods would be methods on the Process class, which would result in methods on a single instance of the ProcessProtocol being called. This single instance of the ProcessProtocol would be shared by all of the 3 file handles. Am I understanding correctly? Are write and loseConnection the only methods the Process would need to implement from ITransport?
That mostly sounds correct, except the last bit. What makes it a transport is that it implements /all/ of ITransport. Moreover, ProcessProtocols expect their transport to be an IProcessTransport, which adds several methods on top of ITransport. Everything in each of those interfaces needs to be implemented. For reference, <http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.I...> and <http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.I...>. Jp
On 7/11/05, Jp Calderone <exarkun@divmod.com> wrote:
On 7/11/05, Jp Calderone <exarkun@divmod.com> wrote:
[snip]
3. Notify the ProcessProtocol via protocol.makeConnection (not sure
why, looking at win32eventreactor)
makeConnection is the method that's actually part of IProtocol (even though ProcessProtocol doesn't implement IProtocol, it still adheres to its API in this regard). Generally, all it does is set the .transport attribute on the protocol instance and then call connectionMade. This might seem pointless in the case of processes, but since it maintains consistency with other kinds of transports, it is useful.
So the Process class is the transport? In win32eventreactor, 3 threads are started that loop on reading stdout and stderr and writing data from the outQueue to stdin. It seems to me that I would not use these threads, but instead have each of these 3 file handles associated with the IOCP. Then
methods defined in ops.py would be used to handle events on those handles, resulting in transport methods being called. These transport methods would be methods on the Process class, which would result in methods on a single instance of the ProcessProtocol being called. This single instance of the ProcessProtocol would be shared by all of the 3 file handles. Am I understanding correctly? Are write and loseConnection the only methods
On Mon, 11 Jul 2005 12:16:35 -0500, Justin Johnson < justinjohnson@gmail.com> wrote: the the
Process would need to implement from ITransport?
That mostly sounds correct, except the last bit. What makes it a transport is that it implements /all/ of ITransport. Moreover, ProcessProtocols expect their transport to be an IProcessTransport, which adds several methods on top of ITransport. Everything in each of those interfaces needs to be implemented.
Ahh... I missed those interfaces because I was starting with win32eventreactor.Process, which doesn't implement either ITransport or IProcessTransport (either with an implements() or by defining all of those methods). After some more thought about this I think I will have to define new xxxOps methods for stdout and stderr (and maybe stdin as well). These ops' ovDone methods will call the methods on the transport (Process) appropriate for ProcessProtocols. For reference, <
http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.I...> and < http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.I...
.
Jp
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 7/11/05, Justin Johnson <justinjohnson@gmail.com> wrote:
On 7/11/05, Jp Calderone <exarkun@divmod.com> wrote:
On Mon, 11 Jul 2005 12:16:35 -0500, Justin Johnson <justinjohnson@gmail.com
wrote: On 7/11/05, Jp Calderone <exarkun@divmod.com> wrote:
[snip]
That mostly sounds correct, except the last bit. What makes it a transport is that it implements /all/ of ITransport. Moreover, ProcessProtocols expect their transport to be an IProcessTransport, which adds several methods on top of ITransport. Everything in each of those interfaces needs to be implemented.
Ahh... I missed those interfaces because I was starting with win32eventreactor.Process, which doesn't implement either ITransport or IProcessTransport (either with an implements() or by defining all of those methods).
Hmm.. I somehow missed the obvious implements call. But I still don't see all methods defined (e.g. getPeer, getHost, writeSequence). After some more thought about this I think I will have to define new xxxOps
methods for stdout and stderr (and maybe stdin as well). These ops' ovDone methods will call the methods on the transport (Process) appropriate for ProcessProtocols.
For reference, <http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.I...>
and <http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.I...
.
Jp
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
I am attempting to add spawnProcess to iocpreactor. In order to begin
On Mon, 11 Jul 2005 10:52:14 -0500 Justin Johnson <justinjohnson@gmail.com> wrote: this
task I've had to do a lot of reading on Windows network programming, specifically the various Windows I/O methods, to attempt to understand what win32eventreactor and iocpreactor are doing, and also just increase my
understanding of how reactors work in general. To understand the various Winsock 2 methods that both of these reactors rely upon, I read chapters 1-5 of Network Programming for Microsoft Windows[1]. Before actually attempting to add spawnProcess, I would like to present how I think iocpreactor works and how I think I should add spawnProcess, and hopefully be corrected or confirmed in my understanding. If I'm too vague there's a good chance it's because I don't understand it very well. Please feel free to point out things that you might think are obvious but aren't sure I understand. How iocpreactor works ---------------------------------
1. Create an IO Completion Port. 2. Create a socket and associate it with the IOCP. This is the socket we will call AcceptEx (a non-blocking accept) on. The association with the IOCP is made via CreateIoCompletionPort. 3. Setup any scheduled tasks on the reactor. 4. Call AcceptEx (which doesn't block) on the socket. AcceptEx takes an overlapped structure as a parameter. Before making the call, we set two attributes of the struct: the callback and callback_args which will be called when an accept event completes on the socket. The Winsock 2 methods don't actually call the callback. The Winsock 2 methods handle copying data related to the network event that occurred on the socket into the overlapped structure and making that overlapped structure available to GetQueuedCompletionStatus. So when we handle events on sockets via GetQueuedCompletionStatus from within doIteration, we have access to the data related to the event as well as the callback and callback_args we call to handle that event. The callbacks are setup in the xxxOp classes in ops.py and always result in some transport method getting called (such as readDone, connectionDone, etc). 5. From within doIteration, call GetQueuedCompletionStatus (which does block) with a timeout of the time until the next scheduled task needs to be run. If any event occurs on the sockets currently associated with the IOCP before that time expires, GetQueuedCompletionStatus will return (stop blocking). Now we have access to the overlapped structure containing data associated with the event which was copied into the overlapped structure's buffer, such as data received from WSARecv calls, as well as the callback and callback_args. From within doIteration we call the callbacks passing in the data related to the event. Depending on the events we are handling, we may create new sockets (e.g. end point sockets in TCP connections) and associate them with the IOCP as well. All Winsock 2 API calls made are non-blocking accept for GetQueuedCompletionStatus. 6. Step 5 continues until the reactor stops.
This sounds about right. Note how this is different from the usual reactor thing -- iocp notifies you when the operation is _finished_, not when it can success without blocking.
How to add spawnProcess ---------------------------------------
1. Create the processes via Windows APIs and associate their stdout/err with with the IOCP via CreateIoCompletionPort calls. 2. Close stdin. 3. Notify the ProcessProtocol via protocol.makeConnection (not sure
why, looking at win32eventreactor) 4. Receive data from stdout/err via the completion port by calling GetQueuedCompletionStatus from within doIteration. Is this really possible? ProcessProtocol's methods won't get called appropriately by letting the existing callbacks in ops.py make calls to the transport (e.g. connectionDone, readDone)?
Hrm. Not quite. In iocp, you always have a read call pending (ReadFileEx, for stdout/err handles). When it completes, you get a notification in GetQueuedCompletionStatus, pass the data to your Protocol and schedule the read again. Do that for stdout and stderr. ops.py already has a wrapper for ReadFile, but it always calls readDone and readErr on your transport. You'll need to fix that.
On 7/11/05, Pavel Pergamenshchik <pp64@codelock.com> wrote:
On Mon, 11 Jul 2005 10:52:14 -0500 Justin Johnson <justinjohnson@gmail.com> wrote:
I am attempting to add spawnProcess to iocpreactor. In order to begin this task I've had to do a lot of reading on Windows network programming, specifically the various Windows I/O methods, to attempt to understand what win32eventreactor and iocpreactor are doing, and also just increase my
understanding of how reactors work in general. To understand the various Winsock 2 methods that both of these reactors rely upon, I read chapters 1-5 of Network Programming for Microsoft Windows[1]. Before actually attempting to add spawnProcess, I would like to present how I think iocpreactor works and how I think I should add spawnProcess, and hopefully be corrected or confirmed in my understanding. If I'm too vague there's a good chance it's because I don't understand it very well. Please feel free to point out things that you might think are obvious but aren't sure I understand. How iocpreactor works ---------------------------------
1. Create an IO Completion Port. 2. Create a socket and associate it with the IOCP. This is the socket we will call AcceptEx (a non-blocking accept) on. The association with the IOCP is made via CreateIoCompletionPort. 3. Setup any scheduled tasks on the reactor. 4. Call AcceptEx (which doesn't block) on the socket. AcceptEx takes an overlapped structure as a parameter. Before making the call, we set two attributes of the struct: the callback and callback_args which will be called when an accept event completes on the socket. The Winsock 2 methods don't actually call the callback. The Winsock 2 methods handle copying data related to the network event that occurred on the socket into the overlapped structure and making that overlapped structure available to GetQueuedCompletionStatus. So when we handle events on sockets via GetQueuedCompletionStatus from within doIteration, we have access to the data related to the event as well as the callback and callback_args we call to handle that event. The callbacks are setup in the xxxOp classes in ops.py and always result in some transport method getting called (such as readDone, connectionDone, etc). 5. From within doIteration, call GetQueuedCompletionStatus (which does block) with a timeout of the time until the next scheduled task needs to be run. If any event occurs on the sockets currently associated with the IOCP before that time expires, GetQueuedCompletionStatus will return (stop blocking). Now we have access to the overlapped structure containing data associated with the event which was copied into the overlapped structure's buffer, such as data received from WSARecv calls, as well as the callback and callback_args. From within doIteration we call the callbacks passing in the data related to the event. Depending on the events we are handling, we may create new sockets (e.g. end point sockets in TCP connections) and associate them with the IOCP as well. All Winsock 2 API calls made are non-blocking accept for GetQueuedCompletionStatus. 6. Step 5 continues until the reactor stops.
This sounds about right. Note how this is different from the usual reactor thing -- iocp notifies you when the operation is _finished_, not when it can success without blocking.
Right. Understood.
How to add spawnProcess
---------------------------------------
1. Create the processes via Windows APIs and associate their stdout/err with with the IOCP via CreateIoCompletionPort calls. 2. Close stdin. 3. Notify the ProcessProtocol via protocol.makeConnection (not sure
why, looking at win32eventreactor) 4. Receive data from stdout/err via the completion port by calling GetQueuedCompletionStatus from within doIteration. Is this really possible? ProcessProtocol's methods won't get called appropriately by letting the existing callbacks in ops.py make calls to the transport (e.g. connectionDone, readDone)?
Hrm. Not quite. In iocp, you always have a read call pending (ReadFileEx, for stdout/err handles). When it completes, you get a notification in GetQueuedCompletionStatus, pass the data to your Protocol and schedule the read again. Do that for stdout and stderr. ops.py already has a wrapper for ReadFile, but it always calls readDone and readErr on your transport. You'll need to fix that.
I think we're on the same page here. See my previous emails correcting my original idea on how this would work. At this point I've defined custom xxxOp classes as follows. class ReadOutOp(OverlappedOp): def ovDone(self, ret, bytes, (handle, buffer)): if ret or not bytes: #self.transport.readErr(ret, bytes) self.transport.outConnectionLost() else: #self.transport.readDone(bytes) self.transport.protocol.outReceived(bytes) def initiateOp(self, handle, buffer): self.reactor.issueReadFile(handle, buffer, self.ovDone, (handle, buffer)) class ReadErrOp(OverlappedOp): def ovDone(self, ret, bytes, (handle, buffer)): if ret or not bytes: #self.transport.readErr(ret, bytes) self.transport.errConnectionLost() else: #self.transport.readDone(bytes) self.transport.protocol.errReceived(bytes) def initiateOp(self, handle, buffer): self.reactor.issueReadFile(handle, buffer, self.ovDone, (handle, buffer)) class WriteInOp(OverlappedOp): def ovDone(self, ret, bytes, (handle, buffer)): # log.msg("WriteFileOp.ovDone", time.time()) if ret or not bytes: #self.transport.writeErr(ret, bytes) self.transport.inConnectionLost() else: #self.transport.writeDone(bytes) pass def initiateOp(self, handle, buffer): # log.msg("WriteFileOp.initiateOp", time.time()) self.reactor.issueWriteFile(handle, buffer, self.ovDone, (handle, buffer))
participants (3)
-
Jp Calderone -
Justin Johnson -
Pavel Pergamenshchik