...

win32all covers a huge number of API functions, more then would be sane to add to os. But would there be any mileage in added enough from win32all to allow problems like popen5 to be implemented?

There is already the _reg module that has some win32 functions in it on the standard install.

(Not sure if this was in response to my "let's do it in Python" post.)

How many APIs would have to be copied from win32all to enable implementing popen5 in Python?

I'm not sure since I haven't implemented much yet, but we'll need at least: win32.CreatePipe win32api.DuplicateHandle win32api.GetCurrentProcess win32process.CreateProcess win32process.STARTUPINFO win32gui.EnumThreadWindows win32event.WaitForSingleObject win32process.TerminateProcess -- /Peter Åstrand <astrand@lysator.liu.se>

Guido van Rossum <guido@python.org> writes:

How many APIs would have to be copied from win32all to enable implementing popen5 in Python?

My (so far relatively trivial) prototype implementation uses 7, but they are a fairly mixed bunch. In many cases, only one of a "group" is needed (GetStdHandle but not SetStdHandle, and WaitForSingleObject, but none of the other WaitForXXX functions). I'm not sure what to conclude from this... Paul. -- This signature intentionally left blank

Guido van Rossum said:

How many APIs would have to be copied from win32all to enable implementing popen5 in Python?

As long as the possibility of copying APIs from win32all is being entertained, would it be feasible to make os.kill do something meaningful on Windows? I'm no expert on Windows APIs, but it may be as simple as calling win32api.GenerateConsoleCtrlEvent or win32api.TerminateProcess. (Although I seem to remember reading that killing processes externally on Windows is something of a thorny issue. It may corrupt the state of DLLs, or something along those lines.) Matthew Barnes

[Guido, on os.kill() for Windows]

Clearly, until someone shows up with more expertise, nothing will happen.

We hash this out periodically here; e.g., http://mail.python.org/pipermail/python-dev/2002-December/030627.html Windows wasn't designed to support clean termination of one process from another, and there are lots of dangers. For Zope's ZRS, I wrapped TerminateProcess() in a teensy C extension so the ZRS test suite could kill the various Python processes it starts (hundreds before the test suite finishes). These are console-mode processes (no windows), and make no use of Windows features, so the dangers are minimal in doing TerminateProcess() on one of those. The general case is a bottomless pit (see the link referenced in the article referenced above for a taste). Recent versions of Windows (not Win9x) have a new API, CreateRemoteThread(), which allows process A to create a thread that runs in some other process B's virtual address space, as if B had started the thread. I read once somewhere that this API can be used by A to execute the safer ExitProcess() "from within" B.

Another possibility: we'd get a lot more mileage out of simply adding ctypes to the standard install. That would solve the immediate popen problem, let us get rid of _winreg entirely (and replace it with a pure Python version), and make future Win32 problems easier to solve.

I don't know that a Python version of _winreg using ctypes would be preferable over a C version. I'd expect it to be slower, less readable than the C version, and more susceptible to the possibility of causing segfaults. (As for speed, I actually have a windows registry application where speed of access is important.) That doesn't mean I don't think ctypes is a good idea -- just that I don't think applying it to _winreg would be useful. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum <guido@python.org> writes:

I don't know that a Python version of _winreg using ctypes would be preferable over a C version. I'd expect it to be slower, less readable than the C version, and more susceptible to the possibility of causing segfaults. (As for speed, I actually have a windows registry application where speed of access is important.)

That doesn't mean I don't think ctypes is a good idea -- just that I don't think applying it to _winreg would be useful.

Agreed on both counts. The main advantage of having ctypes in the core would be to bypass these "but we can't do a Windows version without writing a C wrapper round some API calls" issues. I prefer the idea of including ctypes in the core and writing popen5 (or whatever it gets called) using it, over writing a support extension which just wraps "some" API calls. There's no way of making such an extension self-consistent in the way that _winreg is. The mix of APIs needed for popen5 is too varied. If popen5 is going to have a C extension behind it, I'd say that extension should expose higher level functions coded specifically for popen5, and be named something like _popen5.pyd. Don't try to make it generic - it won't be. But these are implementation and packaging issues. Let's get the design right first (and prototype using whatever seems convenient). Paul. -- This signature intentionally left blank

[Dave]

...

...

Another possibility: we'd get a lot more mileage out of simply adding ctypes to the standard install. That would solve the immediate popen problem, let us get rid of _winreg entirely (and replace it with a pure Python version), and make future Win32 problems easier to solve.

[Guido]

...

I don't know that a Python version of _winreg using ctypes would be preferable over a C version.

Since there is a C version, there's no need to code it in Python again.

...

I'd expect it to be slower, less readable than the C version, and more susceptible to the possibility of causing segfaults.

Slower: yes, but I don't know how much. Less readable: no, more readable. Susceptible to segfaults: If written correctly, it should be bullet proof - exactly the same as a C version.

But the version that was referenced here before isn't bulletproof, right? Wouldn't the bullet-proofing reduce the readability?

...

That doesn't mean I don't think ctypes is a good idea -- just that I don't think applying it to _winreg would be useful.

Cool.

Should this be a Windows only version, or cross-platform?

I don't know enough about ctypes and its user community to answer that (I doubt I'd have much direct need for it myself). But in general I'm biased towards cross-platform tools. --Guido van Rossum (home page: http://www.python.org/~guido/)

There are several ways to fix these problems. A simple one would be to write this instead:

def CreateKey(baseKey, subKey): 'Creates/opens the given key and returns it' RCK = windll.advapi32.RegCreateKeyExA key = c_int(0) result = RCK(str(baseKey), str(subKey), 0, 0, REG_OPTION_NON_VOLATILE, KEY_ALL_ACCESS, 0, byref(key), 0): raise WindowsError(result) return key.value

But ctypes supports a kind of 'function prototypes' (which do automatic argument type checking and/or conversion= as well as automatic result checking:

def CheckNonNull(errcode): # if errcode is nonzero, raise a WindowsError if errcode: raise WindowsError(errcode)

RCK = windll.advapi32.RegCreateKeyExA # specify the argument types RCK.argtypes = (HKEY, LPCTSTR, DWORD, REGSAM, LPSECURITY_ATTRIBUTES, PMKEY, LPDWROD) # RegCreateKeyExA returns 0 on success, or a windows error code RCK.restype = CheckNonNull

def CreateKey(baseKey, subKey): 'Creates/opens the given key and returns it' key = c_int(0) RCK(baseKey, subKey, 0, 0, REG_OPTION_NON_VOLATILE, KEY_ALL_ACCESS, 0, byref(key), 0) return key.value

If no C code existed, this all sounds great -- but don't underestimate its obscurity. For _winreg.c, I don't see a reason to switch. If it's missing something you'd like to see, please submit a patch to the C code.

...

I don't know enough about ctypes and its user community to answer that (I doubt I'd have much direct need for it myself). But in general I'm biased towards cross-platform tools.

I had reports that it works on Solaris, Linux, MacOS, BSD. Maybe more systems.

The problem is that the non-windows version uses libffi, which is difficult to find and install - it seems to be maintained now as part of gcc, although the license is more BSD like.

I would like to get rid of libffi - but the only multi-architecture alternative I know of is Bruno Haible's ffcall (which is GPL).

There *may* be other options (including writing assembly code). Sam Rushing's calldll did this, AFAIK.

And, remember: you can write bulletproof code with ctypes, but you can also easily crash Python. Or other weird things.

OK, never mind. --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum <guido@python.org> writes:

...

Another possibility: we'd get a lot more mileage out of simply adding ctypes to the standard install. That would solve the immediate popen problem, let us get rid of _winreg entirely (and replace it with a pure Python version), and make future Win32 problems easier to solve.

I don't know that a Python version of _winreg using ctypes would be preferable over a C version. I'd expect it to be slower, less readable than the C version, and more susceptible to the possibility of causing segfaults. (As for speed, I actually have a windows registry application where speed of access is important.)

Late followup, a potentially interesting anecdote: I was challanged to find out how much slower ctypes coded registry calls would be than the _winreg calls. So I naively wrote this code: """ import _winreg, time from ctypes import * if 1: start = time.time() i = 0 name = c_buffer(256) cchName = sizeof(name) while 0 == windll.advapi32.RegEnumKeyA(_winreg.HKEY_CLASSES_ROOT, i, name, cchName): i += 1 print i, time.time() - start if 1: start = time.time() i = 0 try: while 1: _winreg.EnumKey(_winreg.HKEY_CLASSES_ROOT, i) i += 1 except WindowsError: pass print i, time.time() - start """ Would you have guessed the result? """ 5648 0.0700001716614 5648 73.7660000324 """ So the *ctypes* version was about 1000 times faster! The comparison is not really right, the 'name' buffer creation must be inside the loop, of course. The real cause is that the _winreg version does a call to RegQueryInfoKey() before calling RegEnumKey(), to find out the size of the larest subkey's name, and this (internally in Windows) probably does have to look at each subkey. So the loop above probably has quadratic runtime behaviour, which explains the bad performance. To get on-topic for python-dev again, it seems there is much room for improvement, at least for the _winreg.EnumKey function. Thomas