Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile
Hello,
I have written replacements for a few of Mark Hammond's PyWin32 functions using ctypes to call upon the Windows kernel API. Code can be found here; http://pastebin.com/m46de01f . When calling ReadFile, it appears that the kernel API converts '/n' newlines '/r/n' newlines. I have not been able to find any information about this but as far as I can tell, there is nothing in my code that is causing the problem. Initially I suspected it related to newline translation but the function in subprocess.Popen for translating newlines only converts to '/n' newlines. When I replaced my ReadFile and WriteFile functions with the PyWin32 functions I was imitating, the problem still arose. I was hoping someone could confirm this problem for me from experience or by testing out my code. If you would like the see the functions used in full context, I have a Mercurial Google Code project that can be checked out at http://code.google.com/p/subprocdev/source/list(branch "python3k"). My work entails modifications to subprocess.py so when running the code, please update your PYTHONPATH variable so that my subprocess.py will be imported.
Eric
Hi,
2009/7/22 Eric Pruitt eric.pruitt@gmail.com:
Hello,
I have written replacements for a few of Mark Hammond's PyWin32 functions using ctypes to call upon the Windows kernel API. Code can be found here; http://pastebin.com/m46de01f . When calling ReadFile, it appears that the kernel API converts '/n' newlines '/r/n' newlines. I have not been able to find any information about this but as far as I can tell, there is nothing in my code that is causing the problem. Initially I suspected it related to newline translation but the function in subprocess.Popen for translating newlines only converts to '/n' newlines. When I replaced my ReadFile and WriteFile functions with the PyWin32 functions I was imitating, the problem still arose. I was hoping someone could confirm this problem for me from experience or by testing out my code. If you would like the see the functions used in full context, I have a Mercurial Google Code project that can be checked out at http://code.google.com/p/subprocdev/source/list (branch "python3k"). My work entails modifications to subprocess.py so when running the code, please update your PYTHONPATH variable so that my subprocess.py will be imported.
These questions should be redirected to comp.lang.python.
But as a quick response, the subprocess stdout is likely to be opened in text mode. So reading \r\n is not a surprise to me.
On Wed, Jul 22, 2009 at 12:33 PM, Amaury Forgeot d'Arcamauryfa@gmail.com wrote:
These questions should be redirected to comp.lang.python.
Eric seems to be working on a GSoC for PFS related to improving subprocess. Even in such case this list is not the right place to make these posts??
Lisandro Dalcin wrote:
Eric seems to be working on a GSoC for PFS related to improving subprocess. Even in such case this list is not the right place to make these posts??
Eric didn't say that he is working on a GSoC project for the PSF. Anyway the Python general mailing list might still be a better place. IMHO he can reach many more competent Python developers there who can help him with this particular problem.
By the way I don't think that ctypes is the right way to go here. ctypes is very handy if you need a quick solution. However I wouldn't use it as a permanent solution for the subprocess module. It's tricky to get ctypes based solutions right on multiple platforms (32 vs 64bit). It's harder to debug a ctypes solutions rather than a C extension, too. I assume that calling overhead is greater than a pure C extension. Can ctypes release the GIL for a function call?
Christian
Hi,
2009/7/23 Christian Heimes lists@cheimes.de:
Lisandro Dalcin wrote:
Eric seems to be working on a GSoC for PFS related to improving subprocess. Even in such case this list is not the right place to make these posts??
Eric didn't say that he is working on a GSoC project for the PSF. Anyway the Python general mailing list might still be a better place. IMHO he can reach many more competent Python developers there who can help him with this particular problem.
Furthermore it was more a Windows question than a Python one. Anyway we continued the discussion privately and I try to be helpful.
By the way I don't think that ctypes is the right way to go here. ctypes is very handy if you need a quick solution. However I wouldn't use it as a permanent solution for the subprocess module. It's tricky to get ctypes based solutions right on multiple platforms (32 vs 64bit). It's harder to debug a ctypes solutions rather than a C extension, too. I assume that calling overhead is greater than a pure C extension.
And some distributions may choose to not distribute the ctypes module. No stdlib module should require it. OTOH ctypes is the perfect tool for rapid development with the win32 api, specially when the compiler is far away.
Can ctypes release the GIL for a function call?
Yes, see http://docs.python.org/library/ctypes.html#ctypes.CFUNCTYPE
Amaury Forgeot d'Arc wrote:
Hi,
2009/7/23 Christian Heimes lists@cheimes.de:
Lisandro Dalcin wrote:
Eric seems to be working on a GSoC for PFS related to improving subprocess. Even in such case this list is not the right place to make these posts??
Eric didn't say that he is working on a GSoC project for the PSF. Anyway the Python general mailing list might still be a better place. IMHO he can reach many more competent Python developers there who can help him with this particular problem.
Furthermore it was more a Windows question than a Python one. Anyway we continued the discussion privately and I try to be helpful.
By the way I don't think that ctypes is the right way to go here. ctypes is very handy if you need a quick solution. However I wouldn't use it as a permanent solution for the subprocess module. It's tricky to get ctypes based solutions right on multiple platforms (32 vs 64bit). It's harder to debug a ctypes solutions rather than a C extension, too. I assume that calling overhead is greater than a pure C extension.
And some distributions may choose to not distribute the ctypes module. No stdlib module should require it.
A big advantage of using ctypes is that it works cross-implementation - on IronPython and PyPy already and on Jython soon. I'd like to see more standard library modules use it. Distributions that choose not to include it are crippling their Python distribution.
Michael Foord
OTOH ctypes is the perfect tool for rapid development with the win32 api, specially when the compiler is far away.
Can ctypes release the GIL for a function call?
Yes, see http://docs.python.org/library/ctypes.html#ctypes.CFUNCTYPE
Michael Foord wrote:
A big advantage of using ctypes is that it works cross-implementation - on IronPython and PyPy already and on Jython soon. I'd like to see more standard library modules use it. Distributions that choose not to include it are crippling their Python distribution.
Interesting, I didn't know that IronPython supports ctypes, too. I still find ctypes a bit problematic because it doesn't us header files for its types, structs and function definitions.
Christian
PS: I'n still reading your IronPython book. I hope I'll have some spare time to write a review anytime soon.
On Thu, 23 Jul 2009 14:21:38 +0200, Christian Heimes lists@cheimes.de wrote:
Michael Foord wrote:
A big advantage of using ctypes is that it works cross-implementation - on IronPython and PyPy already and on Jython soon. I'd like to see more standard library modules use it. Distributions that choose not to include it are crippling their Python distribution.
Interesting, I didn't know that IronPython supports ctypes, too. I still find ctypes a bit problematic because it doesn't us header files for its types, structs and function definitions.
This is indeed a big problem with ctypes. Fortunately, a project exists to correct it:
http://pypi.python.org/pypi/ctypes_configure/0.1
Anyone writing code with ctypes should be looking at ctypes_configure as well.
Jean-Paul
Jean-Paul Calderone schrieb:
On Thu, 23 Jul 2009 14:21:38 +0200, Christian Heimes lists@cheimes.de wrote:
Michael Foord wrote:
A big advantage of using ctypes is that it works cross-implementation - on IronPython and PyPy already and on Jython soon. I'd like to see more standard library modules use it. Distributions that choose not to include it are crippling their Python distribution.
Interesting, I didn't know that IronPython supports ctypes, too. I still find ctypes a bit problematic because it doesn't us header files for its types, structs and function definitions.
This is indeed a big problem with ctypes. Fortunately, a project exists to correct it:
http://pypi.python.org/pypi/ctypes_configure/0.1Anyone writing code with ctypes should be looking at ctypes_configure as well.
There is also another project that uses gccxml to parse the header files and generate ctypes-compatible code.
http://web.archive.org/web/20080115092648/starship.python.net/crew/theller/w...
Especially well on Windows works the dynamic, on-demand code generation.
http://web.archive.org/web/20080115092648/starship.python.net/crew/theller/w...
I do all win32 api programming with this stuff.
I should note that the ctypeslib project is not well maintained because I don't have the time for that any longer.
Christian Heimes wrote:
By the way I don't think that ctypes is the right way to go here. ctypes is very handy if you need a quick solution. However I wouldn't use it as a permanent solution for the subprocess module. It's tricky to get ctypes based solutions right on multiple platforms (32 vs 64bit). It's harder to debug a ctypes solutions rather than a C extension, too. I assume that calling overhead is greater than a pure C extension.
I see ctypes as largely useful when you want to call a native DLL but don't have any existing infrastructure for accessing native code from your project. A few lines of ctypes code is then a much better solution than adding a C or C++ compilation dependency just to access a couple of functions.
Of course, that definitely isn't the case for CPython - we not only have plenty of existing C infrastructure, but in the specific case of subprocess on Windows we already have a dedicated extension module (PC/_subprocess.c).
Cheers, Nick.
Nick Coghlan wrote:
I see ctypes as largely useful when you want to call a native DLL but don't have any existing infrastructure for accessing native code from your project. A few lines of ctypes code is then a much better solution than adding a C or C++ compilation dependency just to access a couple of functions.
Of course, that definitely isn't the case for CPython - we not only have plenty of existing C infrastructure, but in the specific case of subprocess on Windows we already have a dedicated extension module (PC/_subprocess.c).
You've hit the nail on the head! That's it.
Christian
On Thu, 23 Jul 2009 14:23:56 +0200, Christian Heimes lists@cheimes.de wrote:
Nick Coghlan wrote:
I see ctypes as largely useful when you want to call a native DLL but don't have any existing infrastructure for accessing native code from your project. A few lines of ctypes code is then a much better solution than adding a C or C++ compilation dependency just to access a couple of functions.
Of course, that definitely isn't the case for CPython - we not only have plenty of existing C infrastructure, but in the specific case of subprocess on Windows we already have a dedicated extension module (PC/_subprocess.c).
You've hit the nail on the head! That's it.
True, CPython has C infrastructure. What about the other Python runtimes, though? At the language summit, there was a lot of discussion (and, I thought, agreement) about moving the standard library to be a collaborative project between several of the major runtime projects. A ctypes-based solution seems more aligned with this goal than more C code.
Jean-Paul
On Thu, Jul 23, 2009 at 8:57 AM, Jean-Paul Calderoneexarkun@divmod.com wrote:
On Thu, 23 Jul 2009 14:23:56 +0200, Christian Heimes lists@cheimes.de wrote:
Nick Coghlan wrote:
I see ctypes as largely useful when you want to call a native DLL but don't have any existing infrastructure for accessing native code from your project. A few lines of ctypes code is then a much better solution than adding a C or C++ compilation dependency just to access a couple of functions.
Of course, that definitely isn't the case for CPython - we not only have plenty of existing C infrastructure, but in the specific case of subprocess on Windows we already have a dedicated extension module (PC/_subprocess.c).
You've hit the nail on the head! That's it.
True, CPython has C infrastructure. What about the other Python runtimes, though? At the language summit, there was a lot of discussion (and, I thought, agreement) about moving the standard library to be a collaborative project between several of the major runtime projects. A ctypes-based solution seems more aligned with this goal than more C code.
Jean-Paul
Yeah, I remember that too - all of us discussed "breaking out" the stdlib post mercurial migration to be shared amongst all of the implementations, marking CPython things as "CPython only" and so on. That way we all have a common base to work from.
jesse
On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderoneexarkun@divmod.com wrote:
True, CPython has C infrastructure. What about the other Python runtimes, though?
Perhaps these other Python runtimes could implement the functionality of PC/_subprocess.c and use ctypes for that ?
Lisandro Dalcin wrote:
On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderoneexarkun@divmod.com wrote:
True, CPython has C infrastructure. What about the other Python runtimes, though?
Perhaps these other Python runtimes could implement the functionality of PC/_subprocess.c and use ctypes for that ?
Is it sensibly possible to augment a C file with #ifdefs so that it can be compiled either as a directly importable module or as a ctypes-accessible shared library?
tjr
On Thu, Jul 23, 2009 at 5:42 PM, Terry Reedytjreedy@udel.edu wrote:
Lisandro Dalcin wrote:
On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderoneexarkun@divmod.com wrote:
True, CPython has C infrastructure. What about the other Python runtimes, though?
Perhaps these other Python runtimes could implement the functionality of PC/_subprocess.c and use ctypes for that ?
Is it sensibly possible to augment a C file with #ifdefs so that it can be compiled either as a directly importable module or as a ctypes-accessible shared library?
Of course it is posible... Moreover, you could likely reuse 100 % of code intended for ctypes in implementing the extension module calls intended for Python. Mac OS X could have some issues though (IIUC, .so ext modules are not the same as .dylib dynamic libs, though perhaps ctypes can still dlopen() .so files?)
However, how that would help these other Python runtimes without C infraestructure ??
Lisandro Dalcin wrote:
On Thu, Jul 23, 2009 at 5:42 PM, Terry Reedytjreedy@udel.edu wrote:
Lisandro Dalcin wrote:
On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderoneexarkun@divmod.com wrote:
True, CPython has C infrastructure. What about the other Python runtimes, though?
Perhaps these other Python runtimes could implement the functionality of PC/_subprocess.c and use ctypes for that ?
Is it sensibly possible to augment a C file with #ifdefs so that it can be compiled either as a directly importable module or as a ctypes-accessible shared library?
Of course it is posible... Moreover, you could likely reuse 100 % of code intended for ctypes in implementing the extension module calls intended for Python. Mac OS X could have some issues though (IIUC, .so ext modules are not the same as .dylib dynamic libs, though perhaps ctypes can still dlopen() .so files?)
However, how that would help these other Python runtimes without C infraestructure ??
I believe someone just posted that PyPy and IronPython have ctypes and Jython is working on it. That is what triggered this post.
participants (10)
-
Amaury Forgeot d'Arc -
Christian Heimes -
Eric Pruitt -
Jean-Paul Calderone -
Jesse Noller -
Lisandro Dalcin -
Michael Foord -
Nick Coghlan -
Terry Reedy -
Thomas Heller