[Twisted-Python] _dumbwin32proc and PIPE_NOWAIT
From the msdn: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/ipc/base/se... """PIPE_NOWAIT 0x00000001 Nonblocking mode is enabled. In this mode, ReadFile, WriteFile, and ConnectNamedPipe always return immediately. Note that nonblocking mode is supported for compatibility with Microsoft® LAN Manager version 2.0 and should not be used to achieve asynchronous input and output (I/O) with named pipes. """ P.S. I really can't believe that win32 support for asynchronous I/O works with named pipes but not with anonymous pipes... By the way, instead of using a polling, why not to use a separate thread that puts data into a DeferredQueue? P.S.2: From: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/ipc/base/an... """Anonymous pipes are implemented using a named pipe with a unique name. Therefore, you can often pass a handle to an anonymous pipe to a function that requires a handle to a named pipe. """ And from: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/bas... """A child process can inherit [...] * Open handles returned by the CreateFile function. This includes handles to files, console input buffers, console screen buffers, named pipes, serial communication devices, and mailslots. * Open handles to process, thread, mutex, event, semaphore, named-pipe, anonymous-pipe, and file-mapping objects. These are returned by the CreateProcess, CreateThread, CreateMutex, CreateEvent, CreateSemaphore, CreateNamedPipe, CreatePipe, and CreateFileMapping functions, respectively. """ So, why not to create a named pipe with an unique name, know to both the parent and the child process, as an example using an enviroment variable? I think that using named pipes can simplify the _dumbwin32proc. Moreover polling a named pipe is not an efficient operation (reading the author of "Windows System Programming"). Lastly, the parent process, after creating a named pipe instance, can wait for a client connection using ConnectNamedPipe. Regards Manlio Perillo
On Sat, 30 Sep 2006 01:01:06 +0200, Manlio Perillo <manlio_perillo@libero.it> wrote:
From the msdn:
Yep, it's unfortunate that we were forced to use such an ugly API. However, the *correct* way to do this is to use IOCP, and good IOCP support has been *years* in the making. We still haven't got a reliably working IOCP buildbot. You're welcome to contribute a patch which cleans this up on the select reactor, but frankly I don't se any approach which would be better.
I really can't believe that win32 support for asynchronous I/O works with named pipes but not with anonymous pipes...
Yeah, Windows is pretty terrible. It's only a problem in some versions, but I forget where and when it was fixed.
By the way, instead of using a polling, why not to use a separate thread that puts data into a DeferredQueue?
Because interrupting a blocked thread is a dangerous and error-prone task. It was hard enough getting the thing to just work with non-blocking pipe IO.
"""Anonymous pipes are implemented using a named pipe with a unique name. Therefore, you can often pass a handle to an anonymous pipe to a function that requires a handle to a named pipe. """
The word "often" is the killer there.
"""A child process can inherit [...] """
So, why not to create a named pipe with an unique name, know to both the parent and the child process, as an example using an enviroment variable?
Because that would not be usable as standard output using the usual APIs. This is not just for running other Twisted-using programs in a subprocess, but any program which attempts to adhere to UNIX conventions around standard IO.
I think that using named pipes can simplify the _dumbwin32proc. Moreover polling a named pipe is not an efficient operation (reading the author of "Windows System Programming").
The goal of _dumbwin32proc is obviously not efficiency, it is correctness in a configuration which is not well-supported on Windows. Why do you think its name begins with "dumb"?
Lastly, the parent process, after creating a named pipe instance, can wait for a client connection using ConnectNamedPipe.
... as long as it can wait for the completion event. Which it can't, because it's using select().
glyph@divmod.com ha scritto:
On Sat, 30 Sep 2006 01:01:06 +0200, Manlio Perillo <manlio_perillo@libero.it> wrote:
From the msdn:
Yep, it's unfortunate that we were forced to use such an ugly API. However, the *correct* way to do this is to use IOCP, and good IOCP support has been *years* in the making. We still haven't got a reliably working IOCP buildbot.
Unfortunately I don't know I/O completion ports.
You're welcome to contribute a patch which cleans this up on the select reactor, but frankly I don't se any approach which would be better.
I really can't believe that win32 support for asynchronous I/O works with named pipes but not with anonymous pipes...
Yeah, Windows is pretty terrible. It's only a problem in some versions, but I forget where and when it was fixed.
By the way, instead of using a polling, why not to use a separate thread that puts data into a DeferredQueue?
Because interrupting a blocked thread is a dangerous and error-prone task. It was hard enough getting the thing to just work with non-blocking pipe IO.
Not sure to understand here.
[...]
So, why not to create a named pipe with an unique name, know to both the parent and the child process, as an example using an enviroment variable?
Because that would not be usable as standard output using the usual APIs. This is not just for running other Twisted-using programs in a subprocess, but any program which attempts to adhere to UNIX conventions around standard IO.
The solution is simple. Windows *do not* supports POSIX ;-). If one want to use Windows, the win32eventreactor (or, in future, the iocp reactor) should be the default (and the only one allowed). One needs to write a specializd abstract.FileDescriptor class, that use the Overlapped structure for I/O. This should allow one to easily use asyncronous I/O support of Window, not only for sockets, but on all (not many...) 'handles' that support the asyncronous I/O API. One can even add support to asyncronous I/O from a console, since WaitForMultipleObjects supports console handles, and the number of event pending in the input queue can be retrieved using GetNumberOfConsoleInputEvents(). By the way: serialport already works only with win32eventreactor. Regards Manlio Perillo
On Sat, 30 Sep 2006 10:04:47 +0200, Manlio Perillo <manlio_perillo@libero.it> wrote:
glyph@divmod.com ha scritto:
On Sat, 30 Sep 2006 01:01:06 +0200, Manlio Perillo <manlio_perillo@libero.it> wrote:
Unfortunately I don't know I/O completion ports.
That's unfortunate, but I'm not surprised :).
Because interrupting a blocked thread is a dangerous and error-prone task. It was hard enough getting the thing to just work with non-blocking pipe IO.
Not sure to understand here.
"threads are bad"
The solution is simple. Windows *do not* supports POSIX ;-).
It does, however, support standard IO, at least well enough for a large chunk of applications. Although it's not as well supported as on "POSIX" platforms, there are plenty of programs that produce output which Twisted can happily parse semi-asynchronously using the current strategy. Why on earth would you want to remove it? It serves a purpose (cross-platform multi-process communication and control), it works, it's tested.
If one want to use Windows, the win32eventreactor (or, in future, the iocp reactor) should be the default (and the only one allowed).
win32eventreactor can only support 64 sockets. The IOCP reactor currently has several issues and is missing, e.g. SSL support. While true asynchronous I/O and GUI event loop integration have their advantages, the primary feature of the select reactor on Windows is that it actually *works* in most cases.
One needs to write a specializd abstract.FileDescriptor class, that use the Overlapped structure for I/O.
You mean like twisted.internet.iocpreactor.abstract.ConnectedSocket? Have you even read this code?
[snip more oblique allusions to various win32 APIs]
Frankly, I don't think you know what you're talking about. If you know enough to make iocpreactor, or even win32eventreactor, work on Windows with SSL and GUI support, great, do that and contribute it, and we can discuss the patch. If not, we're not going to break the *only* working, tested Windows reactor because you think one day it should be different.
This should allow one to easily use asyncronous I/O support of Window, not only for sockets, but on all (not many...) 'handles' that support the asyncronous I/O API.
If it's easy, feel free to do it :).
By the way: serialport already works only with win32eventreactor.
That is correct, but hardly relevant.
glyph@divmod.com ha scritto:
[...]
"threads are bad"
But they are used in twisted.enterprise.adbapi, and as I can see several developers prefer to use threads to interact with "foreign" APIs instead of develope native asyncronous API.
The solution is simple. Windows *do not* supports POSIX ;-).
It does, however, support standard IO, at least well enough for a large chunk of applications. Although it's not as well supported as on "POSIX" platforms, there are plenty of programs that produce output which Twisted can happily parse semi-asynchronously using the current strategy. Why on earth would you want to remove it? It serves a purpose (cross-platform multi-process communication and control), it works, it's tested.
This is true, for sockets it works well, but for pipes it uses an inefficient implementation.
[...]
One needs to write a specializd abstract.FileDescriptor class, that use the Overlapped structure for I/O.
You mean like twisted.internet.iocpreactor.abstract.ConnectedSocket? Have you even read this code?
Yes, but as the name suggests it works only for Sockets(?). The problem is simple: if I want to do asyncronous I/O with Windows I don't have an unique 'interface'. Only sockets are really integrated with the reactor. serialport has some specialized code for win32eventreactor integration, but if I want to use, as an example, named pipe, I have to write my own support (but it seems this is an issue for broken POSIX implementations, too).
[snip more oblique allusions to various win32 APIs]
Frankly, I don't think you know what you're talking about.
This is very likely ;-)
If you know enough to make iocpreactor, or even win32eventreactor, work on Windows with SSL and GUI support, great, do that and contribute it, and we can discuss the patch. If not, we're not going to break the *only* working, tested Windows reactor because you think one day it should be different.
Is win32eventreactor not working and not tested? You did not mention it in the "removing unsupported reactors in twisted" thread.
This should allow one to easily use asyncronous I/O support of Window, not only for sockets, but on all (not many...) 'handles' that support the asyncronous I/O API.
If it's easy, feel free to do it :).
I have write it, now stdio works ;-). Unfortunately it is not very efficient since: - it uses _pollingfile (not a real porblem) - it reads raw keyboard input and have to do all low level stuff like '\b' handling, '\r' conversion, and so Here is the main code. I suspect such a thing will never be accepted into Twisted ;-): class _PollableConsoleReader(_PollableResource): # XXX TODO add support for Window events (resize) implements(IPushProducer) def __init__(self, con, receivedCallback, lostCallback): self.con = con self.receivedCallback = receivedCallback self.lostCallback = lostCallback self.cp = "cp%d" % win32console.GetConsoleCP() # We need this self._stdout = win32console.GetStdHandle( win32console.STD_OUTPUT_HANDLE ) def checkWork(self): finished = 0 fullDataRead = [] info = self._stdout.GetConsoleScreenBufferInfo() Y = info["CursorPosition"].Y while 1: try: n = self.con.GetNumberOfConsoleInputEvents() if n == 0: break records = self.con.ReadConsoleInput(n) # We need to process input # XXX check me # XXX TODO rewrite in C(?), now this is very inefficient for record in records: if record.EventType != win32console.KEY_EVENT \ or not record.KeyDown: continue char = record.Char if char == '\b': # We need to handle this info = self._stdout.GetConsoleScreenBufferInfo() rowSize = info["MaximumWindowSize"].X cursorPosition = info["CursorPosition"] # Move the cursor if cursorPosition.X == 0: if cursorPosition.Y > Y: cursorPosition.Y -= 1 cursorPosition.X = rowSize - 1 else: cursorPosition.X -= 1 self._stdout.SetConsoleCursorPosition( cursorPosition ) self._stdout.WriteConsoleOutputCharacter(' ', cursorPosition) continue elif char == '\0': continue elif char == '\r': char = '\n' char = char * record.RepeatCount data = char.encode(self.cp) self._stdout.WriteConsole(data) # Do echo fullDataRead.append(data) except pywintypes.error: raise # XXX this should not happen finished = 1 break dataBuf = ''.join(fullDataRead) if dataBuf: self.receivedCallback(dataBuf) if finished: self.cleanup() return len(dataBuf) ... The ConsoleWriter does not need to support asyncronous I/O, so it is a simple wrapper for ConsoleWrite. This code still have some problems[1]... For now I have added this to _pollingfile module. The _win32stdio module can simply check if the stdin is attached to a terminal, so it (hopefully) can support both pipes and consoles. Many things can be added, like an emulation of POSIX terminal, callbacks for resize events, support for focus in/out, colored output, history editing, ... [1] How can be tested such a thing? Regards Manlio Perillo
On Sat, 30 Sep 2006 13:06:03 +0200, Manlio Perillo <manlio_perillo@libero.it> wrote:
[snip]
Here is the main code. I suspect such a thing will never be accepted into Twisted ;-):
[snip]
This looks like it could be an interesting beginning for better terminal support on Windows. It would be better if you included it in unified diff format in the issue tracker, though, along with the unit tests you developed with it. I don't know if anyone out there is interested enough to finish it, but if it correctly provides even a basic level of functionality, it might prove a useful starting place for some other developer. Comprehensive test coverage and developer-oriented documentation improve the chances of someone else picking up where you've left off.
The ConsoleWriter does not need to support asyncronous I/O, so it is a simple wrapper for ConsoleWrite.
Why? Can't writing block? What if you write a huge amount of data? What if the user clicks and holds on the terminal, or selects a menu or resizes the window? Won't those lead to a write blocking for an arbitrary amount of time?
This code still have some problems[1]... For now I have added this to _pollingfile module.
[snip]
[1] How can be tested such a thing?
Presumably by launching a process in a terminal and then communicating with it in order to assert things about its behavior. Jean-Paul
Jean-Paul Calderone ha scritto:
On Sat, 30 Sep 2006 13:06:03 +0200, Manlio Perillo <manlio_perillo@libero.it> wrote:
[snip]
Here is the main code. I suspect such a thing will never be accepted into Twisted ;-):
[snip]
This looks like it could be an interesting beginning for better terminal support on Windows. It would be better if you included it in unified diff format in the issue tracker, though, along with the unit tests you developed with it. I don't know if anyone out there is interested enough to finish it, but if it correctly provides even a basic level of functionality, it might prove a useful starting place for some other developer.
I just remembered that IPython came with a readline implementation in Python + ctypes. There is a complete console module with ansi terminal emulation. The idea is to write a conio module with an implementation for both Windows and POSIX (like serialport).
Comprehensive test coverage and developer-oriented documentation improve the chances of someone else picking up where you've left off.
I'll try to reuse the console module from IPython. Is it a problem the use of ctype? This is not really necessary with pywin32.
The ConsoleWriter does not need to support asyncronous I/O, so it is a simple wrapper for ConsoleWrite.
Why? Can't writing block? What if you write a huge amount of data? What if the user clicks and holds on the terminal, or selects a menu or resizes the window? Won't those lead to a write blocking for an arbitrary amount of time?
Ok. I'll just copy the code from _PollableWritePipe. If I'm not wrong it simply write a chunk of data at every tick of the timer.
This code still have some problems[1]... For now I have added this to _pollingfile module.
[snip]
[1] How can be tested such a thing?
Presumably by launching a process in a terminal and then communicating with it in order to assert things about its behavior.
I'll try to do something. Regards Manlio Perillo
The idea is to write a conio module with an implementation for both Windows and POSIX (like serialport).
Comprehensive test coverage and developer-oriented documentation improve the chances of someone else picking up where you've left off.
I'll try to reuse the console module from IPython. Is it a problem the use of ctype? This is not really necessary with pywin32.
Personally I'd prefer a switch to ctypes from pywin32. It's smaller, it's included in Python 2.5.
On Sat, 30 Sep 2006 13:09:58 -0700, Cory Dodt <corydodt@gmail.com> wrote:
Personally I'd prefer a switch to ctypes from pywin32. It's smaller, it's included in Python 2.5.
But it's not included with Python 2.3 and the rest of our win32 support. This sounds like an entirely separate issue.
I think I get it now. The actual issue you're talking about is console IO. The thing about the efficiency of polling is a side issue. On Sat, 30 Sep 2006 13:06:03 +0200, Manlio Perillo <manlio_perillo@libero.it> wrote:
glyph@divmod.com ha scritto:
The problem is simple: if I want to do asyncronous I/O with Windows I don't have an unique 'interface'.
Yes. Take it up with Microsoft :). Even within their native event loops, things are inconsistent.
Is win32eventreactor not working and not tested?
It's tested, but it's not working. See http://twistedmatrix.com/buildbot/ - you'll notice the "win32-win32er" column is always red. It's also limited because it still allows only 63 objects.
You did not mention it in the "removing unsupported reactors in twisted" thread.
It has maintainers, it has a buildbot, and it's been gradually decreasing its number of failing tests over time. It didn't seem like a candidate for removal.
I have write it, now stdio works ;-).
Unfortunately it is not very efficient since: - it uses _pollingfile (not a real porblem) - it reads raw keyboard input and have to do all low level stuff like '\b' handling, '\r' conversion, and so
It's very hard to do these things efficiently, so I'm not too worried about that. We also don't have a lot of programmers interested in Windows support. My interest in Windows support ends right about at the point where the servers I develop can be successfully tested and developed on win32 :). Providing functionality in an inefficient implementation at least provides some useful functionality in the meanwhile, and something to optimize later if it becomes important.
Here is the main code. I suspect such a thing will never be accepted into Twisted ;-):
As JP said, it might, with some tests and a patch in the tracker :).
The _win32stdio module can simply check if the stdin is attached to a terminal, so it (hopefully) can support both pipes and consoles.
That would be great.
Many things can be added, like an emulation of POSIX terminal, callbacks for resize events, support for focus in/out, colored output, history editing, ...
[1] How can be tested such a thing?
In addition to JP's suggestion, "mock" the actual API you will be testing with, by replacing it in the tests and asserting that the appropriate things are called on the console object that you are passing it. Both approaches make sense in different situations, and good test coverage will generally do both.
glyph@divmod.com ha scritto:
[...]
Is win32eventreactor not working and not tested?
It's tested, but it's not working. See http://twistedmatrix.com/buildbot/ - you'll notice the "win32-win32er" column is always red. It's also limited because it still allows only 63 objects.
I have just runned the test suite (with Twisted 2.4). I got FAILED (skips=76, expectedFailures=7, failures=6, errors=13, successes=868) Moreover about 2-3 time I got a peak to CPU and memory usage (up to 300 MB).
You did not mention it in the "removing unsupported reactors in twisted" thread.
It has maintainers, it has a buildbot, and it's been gradually decreasing its number of failing tests over time. It didn't seem like a candidate for removal.
I have write it, now stdio works ;-).
Unfortunately it is not very efficient since: - it uses _pollingfile (not a real porblem) - it reads raw keyboard input and have to do all low level stuff like '\b' handling, '\r' conversion, and so
It's very hard to do these things efficiently, so I'm not too worried about that. We also don't have a lot of programmers interested in Windows support. My interest in Windows support ends right about at the point where the servers I develop can be successfully tested and developed on win32 :). Providing functionality in an inefficient implementation at least provides some useful functionality in the meanwhile, and something to optimize later if it becomes important.
The implementation surely can be optimized.
Here is the main code.
[...] One more thing. As I have said I would like to provide a portable interface to terminal handling, so that scripts like conch can works on both POSIX and Window. There are several options: 1) Write modules like twised.python.fncl, twisted.python.termios and twisted.python.pty, that implements the same POSIX interface on Windows too. I suspect that this can be done but it is hard (and I'm lazy). Moreover there is the problem with SIGWINCH handling (Windows can - not yet tested - report events about console window resizing) 2) Write an high level module like twisted.python.tty that provides functions like enableRawMode(fd, trueOrFalse), enableEcho(fd, trueOrFalse), addWindowChangeCallback(cb), and so 3) Write an high level module like twisted.internet.conio that provides StdIOChannel (just to pick a name) objects. These objects will have the interface as in 2). Regards Manlio Perillo
glyph@divmod.com ha scritto:
[...]
Here is the main code. I suspect such a thing will never be accepted into Twisted ;-):
As JP said, it might, with some tests and a patch in the tracker :).
The _win32stdio module can simply check if the stdin is attached to a terminal, so it (hopefully) can support both pipes and consoles.
That would be great.
I have managed to have an acceptable solution: http://twistedmatrix.com/trac/attachment/ticket/2157/
Many things can be added, like an emulation of POSIX terminal, callbacks for resize events, support for focus in/out, colored output, history editing, ...
[1] How can be tested such a thing?
In addition to JP's suggestion, "mock" the actual API you will be testing with, by replacing it in the tests and asserting that the appropriate things are called on the console object that you are passing it.
Both approaches make sense in different situations, and good test coverage will generally do both.
For now in the test suit I simply put some characters in the console input buffer (with WriteConsoleInput), checking that they are read/handled correctly. I think that this should suffice. If this patch is accepted I will add a new module, twisted.internet.termios, for a portable termios support (to be used by conch script). My implementation supports raw mode and adding a callback when the window size changes. In future I will add history editing (as it works in cmd.exe) Regards Manlio Perillo
participants (4)
-
Cory Dodt -
glyph＠divmod.com -
Jean-Paul Calderone -
Manlio Perillo