Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile
Hello, I have written replacements for a few of Mark Hammond's PyWin32 functions using ctypes to call upon the Windows kernel API. Code can be found here; http://pastebin.com/m46de01f . When calling ReadFile, it appears that the kernel API converts '/n' newlines '/r/n' newlines. I have not been able to find any information about this but as far as I can tell, there is nothing in my code that is causing the problem. Initially I suspected it related to newline translation but the function in subprocess.Popen for translating newlines only converts to '/n' newlines. When I replaced my ReadFile and WriteFile functions with the PyWin32 functions I was imitating, the problem still arose. I was hoping someone could confirm this problem for me from experience or by testing out my code. If you would like the see the functions used in full context, I have a Mercurial Google Code project that can be checked out at http://code.google.com/p/subprocdev/source/list(branch "python3k"). My work entails modifications to subprocess.py so when running the code, please update your PYTHONPATH variable so that my subprocess.py will be imported. Eric
Hi, 2009/7/22 Eric Pruitt <eric.pruitt@gmail.com>:
These questions should be redirected to comp.lang.python. But as a quick response, the subprocess stdout is likely to be opened in text mode. So reading \r\n is not a surprise to me. -- Amaury Forgeot d'Arc
On Wed, Jul 22, 2009 at 12:33 PM, Amaury Forgeot d'Arc<amauryfa@gmail.com> wrote:
These questions should be redirected to comp.lang.python.
Eric seems to be working on a GSoC for PFS related to improving subprocess. Even in such case this list is not the right place to make these posts?? -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
Lisandro Dalcin wrote:
Eric didn't say that he is working on a GSoC project for the PSF. Anyway the Python general mailing list might still be a better place. IMHO he can reach many more competent Python developers there who can help him with this particular problem. By the way I don't think that ctypes is the right way to go here. ctypes is very handy if you need a quick solution. However I wouldn't use it as a permanent solution for the subprocess module. It's tricky to get ctypes based solutions right on multiple platforms (32 vs 64bit). It's harder to debug a ctypes solutions rather than a C extension, too. I assume that calling overhead is greater than a pure C extension. Can ctypes release the GIL for a function call? Christian
Hi, 2009/7/23 Christian Heimes <lists@cheimes.de>:
Furthermore it was more a Windows question than a Python one. Anyway we continued the discussion privately and I try to be helpful.
And some distributions may choose to not distribute the ctypes module. No stdlib module should require it. OTOH ctypes is the perfect tool for rapid development with the win32 api, specially when the compiler is far away.
Can ctypes release the GIL for a function call?
Yes, see http://docs.python.org/library/ctypes.html#ctypes.CFUNCTYPE -- Amaury Forgeot d'Arc
Amaury Forgeot d'Arc wrote:
A big advantage of using ctypes is that it works cross-implementation - on IronPython and PyPy already and on Jython soon. I'd like to see more standard library modules use it. Distributions that choose not to include it are crippling their Python distribution. Michael Foord
Michael Foord wrote:
Interesting, I didn't know that IronPython supports ctypes, too. I still find ctypes a bit problematic because it doesn't us header files for its types, structs and function definitions. Christian PS: I'n still reading your IronPython book. I hope I'll have some spare time to write a review anytime soon.
On Thu, 23 Jul 2009 14:21:38 +0200, Christian Heimes <lists@cheimes.de> wrote:
This is indeed a big problem with ctypes. Fortunately, a project exists to correct it: http://pypi.python.org/pypi/ctypes_configure/0.1 Anyone writing code with ctypes should be looking at ctypes_configure as well. Jean-Paul
Jean-Paul Calderone schrieb:
There is also another project that uses gccxml to parse the header files and generate ctypes-compatible code. http://web.archive.org/web/20080115092648/starship.python.net/crew/theller/w... Especially well on Windows works the dynamic, on-demand code generation. http://web.archive.org/web/20080115092648/starship.python.net/crew/theller/w... I do all win32 api programming with this stuff. I should note that the ctypeslib project is not well maintained because I don't have the time for that any longer. -- Thanks, Thomas
Christian Heimes wrote:
I see ctypes as largely useful when you want to call a native DLL but don't have any existing infrastructure for accessing native code from your project. A few lines of ctypes code is then a much better solution than adding a C or C++ compilation dependency just to access a couple of functions. Of course, that definitely isn't the case for CPython - we not only have plenty of existing C infrastructure, but in the specific case of subprocess on Windows we already have a dedicated extension module (PC/_subprocess.c). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Thu, 23 Jul 2009 14:23:56 +0200, Christian Heimes <lists@cheimes.de> wrote:
True, CPython has C infrastructure. What about the other Python runtimes, though? At the language summit, there was a lot of discussion (and, I thought, agreement) about moving the standard library to be a collaborative project between several of the major runtime projects. A ctypes-based solution seems more aligned with this goal than more C code. Jean-Paul
On Thu, Jul 23, 2009 at 8:57 AM, Jean-Paul Calderone<exarkun@divmod.com> wrote:
Yeah, I remember that too - all of us discussed "breaking out" the stdlib post mercurial migration to be shared amongst all of the implementations, marking CPython things as "CPython only" and so on. That way we all have a common base to work from. jesse
On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderone<exarkun@divmod.com> wrote:
True, CPython has C infrastructure. What about the other Python runtimes, though?
Perhaps these other Python runtimes could implement the functionality of PC/_subprocess.c and use ctypes for that ? -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
On Thu, Jul 23, 2009 at 5:42 PM, Terry Reedy<tjreedy@udel.edu> wrote:
Of course it is posible... Moreover, you could likely reuse 100 % of code intended for ctypes in implementing the extension module calls intended for Python. Mac OS X could have some issues though (IIUC, .so ext modules are not the same as .dylib dynamic libs, though perhaps ctypes can still dlopen() .so files?) However, how that would help these other Python runtimes without C infraestructure ?? -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
Hi, 2009/7/22 Eric Pruitt <eric.pruitt@gmail.com>:
These questions should be redirected to comp.lang.python. But as a quick response, the subprocess stdout is likely to be opened in text mode. So reading \r\n is not a surprise to me. -- Amaury Forgeot d'Arc
On Wed, Jul 22, 2009 at 12:33 PM, Amaury Forgeot d'Arc<amauryfa@gmail.com> wrote:
These questions should be redirected to comp.lang.python.
Eric seems to be working on a GSoC for PFS related to improving subprocess. Even in such case this list is not the right place to make these posts?? -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
Lisandro Dalcin wrote:
Eric didn't say that he is working on a GSoC project for the PSF. Anyway the Python general mailing list might still be a better place. IMHO he can reach many more competent Python developers there who can help him with this particular problem. By the way I don't think that ctypes is the right way to go here. ctypes is very handy if you need a quick solution. However I wouldn't use it as a permanent solution for the subprocess module. It's tricky to get ctypes based solutions right on multiple platforms (32 vs 64bit). It's harder to debug a ctypes solutions rather than a C extension, too. I assume that calling overhead is greater than a pure C extension. Can ctypes release the GIL for a function call? Christian
Hi, 2009/7/23 Christian Heimes <lists@cheimes.de>:
Furthermore it was more a Windows question than a Python one. Anyway we continued the discussion privately and I try to be helpful.
And some distributions may choose to not distribute the ctypes module. No stdlib module should require it. OTOH ctypes is the perfect tool for rapid development with the win32 api, specially when the compiler is far away.
Can ctypes release the GIL for a function call?
Yes, see http://docs.python.org/library/ctypes.html#ctypes.CFUNCTYPE -- Amaury Forgeot d'Arc
Amaury Forgeot d'Arc wrote:
A big advantage of using ctypes is that it works cross-implementation - on IronPython and PyPy already and on Jython soon. I'd like to see more standard library modules use it. Distributions that choose not to include it are crippling their Python distribution. Michael Foord
Michael Foord wrote:
Interesting, I didn't know that IronPython supports ctypes, too. I still find ctypes a bit problematic because it doesn't us header files for its types, structs and function definitions. Christian PS: I'n still reading your IronPython book. I hope I'll have some spare time to write a review anytime soon.
On Thu, 23 Jul 2009 14:21:38 +0200, Christian Heimes <lists@cheimes.de> wrote:
This is indeed a big problem with ctypes. Fortunately, a project exists to correct it: http://pypi.python.org/pypi/ctypes_configure/0.1 Anyone writing code with ctypes should be looking at ctypes_configure as well. Jean-Paul
Jean-Paul Calderone schrieb:
There is also another project that uses gccxml to parse the header files and generate ctypes-compatible code. http://web.archive.org/web/20080115092648/starship.python.net/crew/theller/w... Especially well on Windows works the dynamic, on-demand code generation. http://web.archive.org/web/20080115092648/starship.python.net/crew/theller/w... I do all win32 api programming with this stuff. I should note that the ctypeslib project is not well maintained because I don't have the time for that any longer. -- Thanks, Thomas
Christian Heimes wrote:
I see ctypes as largely useful when you want to call a native DLL but don't have any existing infrastructure for accessing native code from your project. A few lines of ctypes code is then a much better solution than adding a C or C++ compilation dependency just to access a couple of functions. Of course, that definitely isn't the case for CPython - we not only have plenty of existing C infrastructure, but in the specific case of subprocess on Windows we already have a dedicated extension module (PC/_subprocess.c). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Thu, 23 Jul 2009 14:23:56 +0200, Christian Heimes <lists@cheimes.de> wrote:
True, CPython has C infrastructure. What about the other Python runtimes, though? At the language summit, there was a lot of discussion (and, I thought, agreement) about moving the standard library to be a collaborative project between several of the major runtime projects. A ctypes-based solution seems more aligned with this goal than more C code. Jean-Paul
On Thu, Jul 23, 2009 at 8:57 AM, Jean-Paul Calderone<exarkun@divmod.com> wrote:
Yeah, I remember that too - all of us discussed "breaking out" the stdlib post mercurial migration to be shared amongst all of the implementations, marking CPython things as "CPython only" and so on. That way we all have a common base to work from. jesse
On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderone<exarkun@divmod.com> wrote:
True, CPython has C infrastructure. What about the other Python runtimes, though?
Perhaps these other Python runtimes could implement the functionality of PC/_subprocess.c and use ctypes for that ? -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
On Thu, Jul 23, 2009 at 5:42 PM, Terry Reedy<tjreedy@udel.edu> wrote:
Of course it is posible... Moreover, you could likely reuse 100 % of code intended for ctypes in implementing the extension module calls intended for Python. Mac OS X could have some issues though (IIUC, .so ext modules are not the same as .dylib dynamic libs, though perhaps ctypes can still dlopen() .so files?) However, how that would help these other Python runtimes without C infraestructure ?? -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594
participants (10)
-
Amaury Forgeot d'Arc -
Christian Heimes -
Eric Pruitt -
Jean-Paul Calderone -
Jesse Noller -
Lisandro Dalcin -
Michael Foord -
Nick Coghlan -
Terry Reedy -
Thomas Heller