I have now started an initial patch for PEP 384, in the pep-0384 branch. This has the following features: - modules can be compiled under Py_LIMITED_API - Tools/scripts/abitype.py converts C code containing static PyTypeObject definitions to use the new API for type definitions. The following aspects are still missing: - there is no support for generating python3.dll on Windows yet - there has been no validation whether the API is actually feasible to use in extension modules. I started looking into porting the sqlite extension, and ran into these issues: - certain fields of PyTypeObject are called directly: pysqlite_NodeType.tp_alloc Py_TYPE(self)->tp_free - PyObject_Print is used, but can't be supported, as it uses a FILE* parameter For the first issue, it would be possible to provide a generic accessor function that fetches fields from a type object. Alternatively, each case could be considered, suggesting an alternative code for the desired effect. I'll be off the net for the next two weeks most of the time, so I might not be able to respond quickly. Anybody interested in advancing that patch, feel free to commit changes into the branch. Regards, Martin
2010/8/28 "Martin v. Löwis" <martin@v.loewis.de>:
I have now started an initial patch for PEP 384, in the pep-0384 branch. This has the following features: - modules can be compiled under Py_LIMITED_API - Tools/scripts/abitype.py converts C code containing static PyTypeObject definitions to use the new API for type definitions.
The following aspects are still missing: - there is no support for generating python3.dll on Windows yet - there has been no validation whether the API is actually feasible to use in extension modules.
I started looking into porting the sqlite extension, and ran into these issues: - certain fields of PyTypeObject are called directly: pysqlite_NodeType.tp_alloc Py_TYPE(self)->tp_free
This is from tp_new and tp_dealloc, right? I think we should probably provide assessors PyObject_Alloc and PyObject_FreeObject.
- PyObject_Print is used, but can't be supported, as it uses a FILE* parameter
I thought tp_print was supposed to have been removed. Anyway, if sqlite is already using FILE *, then won't it be afflicted by the Microsoft runtime version changes anyway? Maybe provide an extra flag to enable FILE* APIs for those extensions that want to risk it? -- Regards, Benjamin
This is from tp_new and tp_dealloc, right? I think we should probably provide assessors PyObject_Alloc and PyObject_FreeObject.
Correct, and yes, that sounds like a good approach.
- PyObject_Print is used, but can't be supported, as it uses a FILE* parameter
I thought tp_print was supposed to have been removed.
Yes - that should have happened for 3.0. Not sure how to deal with it now.
Anyway, if sqlite is already using FILE *, then won't it be afflicted by the Microsoft runtime version changes anyway? Maybe provide an extra flag to enable FILE* APIs for those extensions that want to risk it?
For the sqlite extension, that wouldn't be a problem: if they build with a different MSVC release, it will automatically link with a different CRT, which then will have a consistent set of FILE objects. The issue only arises if you pass FILE* across DLLs which in turn are linked with different CRTs. Regards, Martin
On Sun, Aug 29, 2010 at 4:52 AM, "Martin v. Löwis" <martin@v.loewis.de> wrote:
This is from tp_new and tp_dealloc, right? I think we should probably provide assessors PyObject_Alloc and PyObject_FreeObject.
Correct, and yes, that sounds like a good approach.
- PyObject_Print is used, but can't be supported, as it uses a FILE* parameter
I thought tp_print was supposed to have been removed.
Yes - that should have happened for 3.0. Not sure how to deal with it now.
tp_print actually is gone, but PyObject_Print was retained. It just relies on repr() and str() under the hood instead of the old tp_print slot. There are 4 operations performed on fp in that function: clearerr, ferror, fprintf and fwrite. There is also an implicit reference to errno (through PyErr_SetFromErrno) which can't be trusted in a mixed CRT world (PyErr_SetFromErrno() should be excluded from the Limited API, if it isn't already, and replaced with a PyErr_SetFromErrnoEx which takes the errno as an explicit argument. That assumes the CRTs will be sufficiently compatible that strerror() will give the correct answer for errno values from the other CRT. If that assumption is incorrect, the new function would also need to accept a string argument for the error description). Four options come to mind: - just leave it out of the limited API, extensions can do their own thing to print objects - leave PyObject_Print out of the limited API, but create a PyObject_PrintEx that takes a Python IO stream via PyObject* rather than a C level FILE*. - leave PyObject_Print out of the limited API, but create a PyObject_PrintEx that takes function pointers for the above 4 operations (so the FILE* pointer is only every operated on by functions from the extension module's CRT) - leave PyObject_Print out of the limited API, but create a PyObject_PRINT macro that does much the same thing with the logic rearranged so there is an inner function that figures out the string to be printed, but an outer macro that does all the operations on the FILE * object (so again, the FILE * is never passed to Python's CRT) The last option requires the fewest adjustments for extension authors, and it should be feasible to do it that way (even though it is a bit of a hack). Something along the lines of the following: #define PyObject_PRINT (obj, fp, flags, resultp) \ { \ int _result = -1; _t = _PyObject_PrintInner(obj, flags); \ if (_t != NULL) { \ clearerr(fp); \ fwrite(PyBytes_AS_STRING(_t), 1, PyBytes_GET_SIZE(_t), fp); \ Py_DECREF(_t); \ if (ferror(fp)) { \ PyErr_SetFromErrnoEx(PyExc_IOError, errno); \ clearerr(fp); \ } else { \ _result = 0; \ } \ } \ if (resultp != NULL) { \ *resultp = _result; \ } \ } Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, 29 Aug 2010 09:20:56 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Four options come to mind:
- just leave it out of the limited API, extensions can do their own thing to print objects - leave PyObject_Print out of the limited API, but create a PyObject_PrintEx that takes a Python IO stream via PyObject* rather than a C level FILE*. - leave PyObject_Print out of the limited API, but create a PyObject_PrintEx that takes function pointers for the above 4 operations (so the FILE* pointer is only every operated on by functions from the extension module's CRT) - leave PyObject_Print out of the limited API, but create a PyObject_PRINT macro that does much the same thing with the logic rearranged so there is an inner function that figures out the string to be printed, but an outer macro that does all the operations on the FILE * object (so again, the FILE * is never passed to Python's CRT)
Fifth option: - make PyObject_Print() an inline function (similar to your macro proposal), but only on Windows. This would retain the name and current signature. Apparently we could use something like "__forceinline" or "extern __forceinline"? Regards Antoine.
On Sun, Aug 29, 2010 at 6:24 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sun, 29 Aug 2010 09:20:56 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Four options come to mind:
- just leave it out of the limited API, extensions can do their own thing to print objects - leave PyObject_Print out of the limited API, but create a PyObject_PrintEx that takes a Python IO stream via PyObject* rather than a C level FILE*. - leave PyObject_Print out of the limited API, but create a PyObject_PrintEx that takes function pointers for the above 4 operations (so the FILE* pointer is only every operated on by functions from the extension module's CRT) - leave PyObject_Print out of the limited API, but create a PyObject_PRINT macro that does much the same thing with the logic rearranged so there is an inner function that figures out the string to be printed, but an outer macro that does all the operations on the FILE * object (so again, the FILE * is never passed to Python's CRT)
Fifth option: - make PyObject_Print() an inline function (similar to your macro proposal), but only on Windows. This would retain the name and current signature. Apparently we could use something like "__forceinline" or "extern __forceinline"?
I believe both that option, and my third option, would run into trouble due to the potential for errno confusion. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, 29 Aug 2010 18:41:45 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
On Sun, Aug 29, 2010 at 6:24 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sun, 29 Aug 2010 09:20:56 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Four options come to mind:
- just leave it out of the limited API, extensions can do their own thing to print objects - leave PyObject_Print out of the limited API, but create a PyObject_PrintEx that takes a Python IO stream via PyObject* rather than a C level FILE*. - leave PyObject_Print out of the limited API, but create a PyObject_PrintEx that takes function pointers for the above 4 operations (so the FILE* pointer is only every operated on by functions from the extension module's CRT) - leave PyObject_Print out of the limited API, but create a PyObject_PRINT macro that does much the same thing with the logic rearranged so there is an inner function that figures out the string to be printed, but an outer macro that does all the operations on the FILE * object (so again, the FILE * is never passed to Python's CRT)
Fifth option: - make PyObject_Print() an inline function (similar to your macro proposal), but only on Windows. This would retain the name and current signature. Apparently we could use something like "__forceinline" or "extern __forceinline"?
I believe both that option, and my third option, would run into trouble due to the potential for errno confusion.
I don't understand. What's the difference with the macro you proposed (the fourth option)? Antoine.
On Sun, Aug 29, 2010 at 7:10 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sun, 29 Aug 2010 18:41:45 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
I believe both that option, and my third option, would run into trouble due to the potential for errno confusion.
I don't understand. What's the difference with the macro you proposed (the fourth option)?
Oh, you meant combining it with the suggested change to the errno handling rather than merely inlining the existing PyObject_Print? Yes, that should work and would be even more seamless from the point of view of the extension modules. However, since even platforms other than Windows aren't immune to version upgrades of the standard C runtime, I'm still more comfortable with the idea that the strict ABI should refuse to pass FILE* pointers across extension module boundaries. The advantage of the preprocessor macro approach is that it allows us to provide assistance with operations on FILE* pointers while still obeying that restriction regardless of the specific compilers involved in creating the Python and extension module binaries. It would be nice to get rid of the resultp parameter by rewriting the internals of the macro as a collection of ternary and comma expressions instead of a scope, but I haven't managed to crack the problem of feeding the correct arguments to fwrite with that approach (it's been quite a while since I've attempted to abuse C expressions to this extent, so I've forgotten most of the tricks for avoiding temporary variables). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Aug 29, 2010, at 8:16 AM, Nick Coghlan wrote:
However, since even platforms other than Windows aren't immune to version upgrades of the standard C runtime
Aren't they? I don't know of any other platform that lets you have two versions of libc linked into a single address space. Linux has had incompatible libc updates in the past, but it was not possible to use both in one program. I believe BSD works the same way. James
On Sun, 29 Aug 2010 22:16:57 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
However, since even platforms other than Windows aren't immune to version upgrades of the standard C runtime, I'm still more comfortable with the idea that the strict ABI should refuse to pass FILE* pointers across extension module boundaries.
I'm not sure what the worry is. Under an Unix-like system, there shouldn't be any situation where two different versions of the libc are loaded in the same process. And since FILE* is an opaque pointer whose referent is only accessed by the libc, things should be fine. (actually, I'm baffled that Windows has such problems, and I would suggest that it's not Python's job to shield Windows application developers from Windows-specific development issues)
The advantage of the preprocessor macro approach is that it allows us to provide assistance with operations on FILE* pointers while still obeying that restriction regardless of the specific compilers involved in creating the Python and extension module binaries.
An inline function should provide the same advantage since it will be compiled with the calling library/executable, rather than the Python DLL. Regards Antoine.
On Mon, Aug 30, 2010 at 1:43 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Sun, 29 Aug 2010 22:16:57 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote: (actually, I'm baffled that Windows has such problems, and I would suggest that it's not Python's job to shield Windows application developers from Windows-specific development issues)
It depends on how you view different versions of a shared library - with relocatable addresses, there's nothing fundamental preventing you from allowing separately compiled components to depend on those different versions, with the dynamic linker placing them at different points in memory. If you start from an independent binary distribution mindset and an attitude that the easiest way to ensure backwards compatibility is to avoid forced upgrades of dependencies, allowing it makes perfect sense (whereas *nix systems tend work to with a "don't repeat yourself" philosophy at the system level and are quite happy to let old binaries and source code become unusable, or at least incompatible with modern components, as their dependencies update below them). With the benefit of hindsight, the Windows approach turns out to have some pretty severe downsides (the name "DLL hell" is well earned), but there are some positive aspects to the approach. Back on the main topic, I am not in a position to state that Windows is the only platform that currently, or will ever, allow two different versions of the C runtime to be loaded into the same address space (I thought you could even persuade *nix to do it if you really wanted to, although you tell me I'm wrong about that). Since part of the point of PEP 384 is to support multiple versions of the C runtime in a single process, relying on Windows-specific or post-C89 options seems unnecessary when there are platform neutral C89 compatible approaches that achieve the same end in a way that will be completely obvious to most C programmers. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, 30 Aug 2010 07:31:34 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Since part of the point of PEP 384 is to support multiple versions of the C runtime in a single process, [...]
I think that's quite a maximalist goal. The point of PEP 384 should be to define a standard API for Python, (hopefully) spanning multiple versions. Whether the API effectively guarantees a standard ABI can only depend on whether the system itself hasn't changed its own conventions (including, for example, function call conventions, or the binary representation of standard C types). In other words, PEP 384 should only care to stabilize the ABI as long as the underlying system doesn't change. It sounds a bit foolish for us to try to hide potential unstabilities in the underlying platform. And it's equally foolish to try to forbid people from using well-known system facilities such as FILE* or (worse) errno. So, perhaps the C API docs can simply mention the caveat of using FILE* (and perhaps errno, if there's a problem there as well) for C extensions under Windows. C extension writers are (usually) consenting adults, for all. Regards Antoine.
On Mon, Aug 30, 2010 at 6:43 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Mon, 30 Aug 2010 07:31:34 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Since part of the point of PEP 384 is to support multiple versions of the C runtime in a single process, [...]
I think that's quite a maximalist goal. The point of PEP 384 should be to define a standard API for Python, (hopefully) spanning multiple versions. Whether the API effectively guarantees a standard ABI can only depend on whether the system itself hasn't changed its own conventions (including, for example, function call conventions, or the binary representation of standard C types).
In other words, PEP 384 should only care to stabilize the ABI as long as the underlying system doesn't change. It sounds a bit foolish for us to try to hide potential unstabilities in the underlying platform. And it's equally foolish to try to forbid people from using well-known system facilities such as FILE* or (worse) errno.
So, perhaps the C API docs can simply mention the caveat of using FILE* (and perhaps errno, if there's a problem there as well) for C extensions under Windows. C extension writers are (usually) consenting adults, for all.
This significantly decrease the value of such an API, to the point of making it useless on windows, since historically different python versions are built with different runtimes. And I would think that windows is the platform where PEP 384 would be the most useful - at least it would for numpy/scipy, where those runtimes issues have bitten us several times (and are painful to debug, especially when you don't know windows so well). cheers, David
On Mon, Aug 30, 2010 at 7:43 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Mon, 30 Aug 2010 07:31:34 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Since part of the point of PEP 384 is to support multiple versions of the C runtime in a single process, [...]
I think that's quite a maximalist goal. The point of PEP 384 should be to define a standard API for Python, (hopefully) spanning multiple versions. Whether the API effectively guarantees a standard ABI can only depend on whether the system itself hasn't changed its own conventions (including, for example, function call conventions, or the binary representation of standard C types).
FILE* is very different from the other things you mention. Function call conventions and binary representations are defined in the C standard. FILE*, on the other hand, is explicitly called out as an opaque reference completely under the control of the specific C runtime implementation. Since the Linkage section of PEP 384 specifically states the availability of a generic "python3.dll" that dynamically redirects to the relevant "python3y.dll" to allow an extension module to run against any 3.2 or later Python version as a goal of the PEP, I would say that allowing mixing of C runtimes is definitely one of the PEP's goals. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le lundi 30 août 2010 à 22:18 +1000, Nick Coghlan a écrit :
FILE* is very different from the other things you mention. Function call conventions and binary representations are defined in the C standard. FILE*, on the other hand, is explicitly called out as an opaque reference completely under the control of the specific C runtime implementation.
Since the Linkage section of PEP 384 specifically states the availability of a generic "python3.dll" that dynamically redirects to the relevant "python3y.dll" to allow an extension module to run against any 3.2 or later Python version as a goal of the PEP, I would say that allowing mixing of C runtimes is definitely one of the PEP's goals.
Well, allowing it is one thing. Guaranteeing that it will always work is another one. That said, I have nothing against Python providing such a guarantee, as long as it doesn't complicate the work of developers on other platforms who don't care about Windows. Regards Antoine.
On Aug 30, 2010, at 07:31 AM, Nick Coghlan wrote:
Since part of the point of PEP 384 is to support multiple versions of the C runtime in a single process
Is it? That's certainly not explicit in the Rationale section of PEP 384. It seems to me that the PEP is all about inoculating extension builders from changes in Python's ABI across Python versions. That's a worthy goal. I'm not sure the same can be said about allowing multiple versions of the C runtime in a linked process. But then I'm developing on Windows these days, and I think for all practical purposes, it's a moot point on *nix. -Barry
On Aug 30, 2010, at 10:18 PM, Nick Coghlan wrote:
Since the Linkage section of PEP 384 specifically states the availability of a generic "python3.dll" that dynamically redirects to the relevant "python3y.dll" to allow an extension module to run against any 3.2 or later Python version as a goal of the PEP, I would say that allowing mixing of C runtimes is definitely one of the PEP's goals.
It should be explicit about that then, and provide detail about why the runtime is relevant to Windows programmers (and probably not relevant in practice for *nix programmers). -Barry
On 30/08/2010 17:35, Barry Warsaw wrote:
On Aug 30, 2010, at 10:18 PM, Nick Coghlan wrote:
Since the Linkage section of PEP 384 specifically states the availability of a generic "python3.dll" that dynamically redirects to the relevant "python3y.dll" to allow an extension module to run against any 3.2 or later Python version as a goal of the PEP, I would say that allowing mixing of C runtimes is definitely one of the PEP's goals. It should be explicit about that then, and provide detail about why the runtime is relevant to Windows programmers (and probably not relevant in practice for *nix programmers).
An extension compiled for one version of Python will be linked against the version of the C runtime used by that version of Python (if it is compiled with the same version of Visual Studio of course). If the extension binary is to remain compatible with a later version of Python, compiled against a different version of the C runtime, then it *must* be possible for multiple C runtimes to be loaded. If the stable ABI doesn't allow this then binaries will *still* have to be recompiled when we update the version of Visual Studio used to compile Python - defeating the purpose of the PEP. Right? If this is the case then I agree that it should be explicit in the PEP. All the best, Michael
-Barry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
On Tue, Aug 31, 2010 at 12:30 AM, Barry Warsaw <barry@python.org> wrote:
On Aug 30, 2010, at 07:31 AM, Nick Coghlan wrote:
Since part of the point of PEP 384 is to support multiple versions of the C runtime in a single process
Is it? That's certainly not explicit in the Rationale section of PEP 384.
It seems to me that the PEP is all about inoculating extension builders from changes in Python's ABI across Python versions. That's a worthy goal. I'm not sure the same can be said about allowing multiple versions of the C runtime in a linked process. But then I'm developing on Windows these days, and I think for all practical purposes, it's a moot point on *nix.
Yeah, I found the Rationale section a little lacking in that regard as well. It was more of a description of the status quo rather than an attempt to justify the proposed changes. The actual proposal and benefits were down in the Linkage section. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Aug 31, 2010 at 12:47 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
An extension compiled for one version of Python will be linked against the version of the C runtime used by that version of Python (if it is compiled with the same version of Visual Studio of course).
If the extension binary is to remain compatible with a later version of Python, compiled against a different version of the C runtime, then it *must* be possible for multiple C runtimes to be loaded. If the stable ABI doesn't allow this then binaries will *still* have to be recompiled when we update the version of Visual Studio used to compile Python - defeating the purpose of the PEP. Right?
If this is the case then I agree that it should be explicit in the PEP.
Ah, I knew it was explicit in the PEP somewhere. The following is currently mentioned in the "Excluded Functions" section: "In addition, functions expecting FILE* are not part of the ABI, to avoid depending on a specific version of the Microsoft C runtime DLL on Windows." To make that explicit in the Rationale section, I would append a second sentence to the final paragraph of that section (the first sentence is already there): "With this PEP, it will be possible to reduce the dependency of binary extension modules on a specific Python feature release, and applications embedding Python can be made work with different releases. To ensure this is also achieved on Windows, the ABI will be designed to allow the Python binary and extension modules or an embedding application to depend on different versions of the C runtime DLL." I would also make the following modification to the above quoted paragraph from the Excluded Functions section of the PEP: "To avoid depending on a specific version of the Microsoft C runtime DLL on Windows while still providing a consistent, cross-platform API when PY_LIMITED_API is defined, the following functions are also excluded: - Any functions accepting or returning FILE* (as this is defined in the C standard as an implementation dependent opaque reference. On Windows, it is known to contain reference to runtime-specific global data) - Any functions reliant on global or thread local state expected to be modified by C standard library calls rather than other Python API calls (e.g. PyErr_FromErrno will be unavailable. Comparable functionality will be provided by a new function PyErr_FromErrnoEx, with the latter accepting an explicit errno parameter) Where practical, equivalent functionality for the excluded functions will instead be provided using preprocessor macros. This ensures that the use of the C runtime dependent types and data remains separate without significantly complicating extension module development and the API remains compatible with any C89 compiler. Note that locale dependent operations will still be available, they just may not see locale changes made via the C standard library calls. Locale changes will need to be made via Python API calls to the locale module to ensure they are seen correctly by the interpreter." Hmm... that last point is a bit of any issue actually, since it also flows the other way (changes made via the locale module won't be visible to any extension modules using a different C runtime). So I suspect mixing C runtimes is still going to come with the caveat of potential locale related glitches. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Aug 31, 2010 at 6:54 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Hmm... that last point is a bit of any issue actually, since it also flows the other way (changes made via the locale module won't be visible to any extension modules using a different C runtime). So I suspect mixing C runtimes is still going to come with the caveat of potential locale related glitches.
As far as IO is concerned, FILE* is just a special case of a more generic issue, though, so maybe this could be a bit reworded. For example, file descriptor cannot be shared between runtimes either. cheers, David
On 31/08/2010 7:54 AM, Nick Coghlan wrote:
On Tue, Aug 31, 2010 at 12:47 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
An extension compiled for one version of Python will be linked against the version of the C runtime used by that version of Python (if it is compiled with the same version of Visual Studio of course).
If the extension binary is to remain compatible with a later version of Python, compiled against a different version of the C runtime, then it *must* be possible for multiple C runtimes to be loaded. If the stable ABI doesn't allow this then binaries will *still* have to be recompiled when we update the version of Visual Studio used to compile Python - defeating the purpose of the PEP. Right?
If this is the case then I agree that it should be explicit in the PEP.
Ah, I knew it was explicit in the PEP somewhere. The following is currently mentioned in the "Excluded Functions" section:
"In addition, functions expecting FILE* are not part of the ABI, to avoid depending on a specific version of the Microsoft C runtime DLL on Windows."
It would be interesting to know how, in practice, these FILE pointers come to life. In my experience they are generally obtained via fopen. If that is broadly true, then a middle-ground may be for Python to expose something like Py_fopen, Py_fclose and a PyFILE opaque "handle". API elements which currently take a FILE * could be exposed using a PyFILE * in the ABI. People who didn't care about this level of portability could continue to use the non-ABI FILE * functions, but people who do could use Py_fopen/Py_fclose in place of fopen/fclose but otherwise work as now. A downside is that as mentioned, in practice these FILE pointers may come from more than simply fopen, so few people would be able to leverage this. It also assumes that people open files before handing them to Python, but otherwise don't use that file - it would be a slippery-slope to wind up with Py_fread etc. Mark
On Tue, 31 Aug 2010 11:12:17 +0900 David Cournapeau <cournape@gmail.com> wrote:
Hmm... that last point is a bit of any issue actually, since it also flows the other way (changes made via the locale module won't be visible to any extension modules using a different C runtime). So I suspect mixing C runtimes is still going to come with the caveat of potential locale related glitches.
As far as IO is concerned, FILE* is just a special case of a more generic issue, though, so maybe this could be a bit reworded. For example, file descriptor cannot be shared between runtimes either.
Er, really? So it means that, for example, a FileIO object couldn't be shared between runtimes either? How about a socket object? Do you want to forbid FileIO and socket objects as part of the API? Again, I propose that FILE* functions are allowed in the API, but their use discouraged in the docs (with proper explanations from those who know how to word them). Regards Antoine.
Michael Foord wrote:
On 30/08/2010 17:35, Barry Warsaw wrote:
On Aug 30, 2010, at 10:18 PM, Nick Coghlan wrote:
Since the Linkage section of PEP 384 specifically states the availability of a generic "python3.dll" that dynamically redirects to the relevant "python3y.dll" to allow an extension module to run against any 3.2 or later Python version as a goal of the PEP, I would say that allowing mixing of C runtimes is definitely one of the PEP's goals. It should be explicit about that then, and provide detail about why the runtime is relevant to Windows programmers (and probably not relevant in practice for *nix programmers).
An extension compiled for one version of Python will be linked against the version of the C runtime used by that version of Python (if it is compiled with the same version of Visual Studio of course).
If the extension binary is to remain compatible with a later version of Python, compiled against a different version of the C runtime, then it *must* be possible for multiple C runtimes to be loaded.
Is it possible to have multiple versions of the lib C loaded on Windows ? I know that it is not possible on Linux. Instead, the glibc takes great care to version all sorts of APIs in the lib to make it possible to run applications using different versions within the same runtime (they include different ABI versions in the same lib). AFAIK, on Windows, the lib C is using a different approach: up until the invention of the manifests, they took great care not to break the APIs in incompatible ways. With manifests, things are more complicated, since the DLLs now explicitly reference a particular version of a DLL (down to the patch level) and if the versions are not installed, the application won't run. Not sure what effect that has on the way they engineered the new lib C versions... This document suggests that it is possible to have an application use multiple CRTs, but care has to be taken with respect to things that are initialized by the CRTs (env vars, locales, handles, etc.): http://msdn.microsoft.com/en-us/library/abx4dbyh(v=VS.90).aspx
If the stable ABI doesn't allow this then binaries will *still* have to be recompiled when we update the version of Visual Studio used to compile Python - defeating the purpose of the PEP. Right?
If we keep changing the main compiler version used for creating Python binaries on Windows: yes. Fortunately, we don't :-) On Unix this shouldn't be too much of a problem, though, so the PEP is still valid for those platforms.
If this is the case then I agree that it should be explicit in the PEP.
-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 27 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 18 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Hi, 2010/8/31 Antoine Pitrou <solipsis@pitrou.net>:
David Cournapeau <cournape@gmail.com> wrote:
As far as IO is concerned, FILE* is just a special case of a more generic issue, though, so maybe this could be a bit reworded. For example, file descriptor cannot be shared between runtimes either.
Er, really?
Yes, on Windows, file descriptors returned by open() or dup() cannot be shared between runtimes. What can be safely shared is the underlying "handle", you can get it with the _get_osfhandle() function.
So it means that, for example, a FileIO object couldn't be shared between runtimes either? How about a socket object? Do you want to forbid FileIO and socket objects as part of the API?
Python objects don't have this concern: all methods of FileIO are implemented in a single file (fileio.c), linked to a single C runtime.
Again, I propose that FILE* functions are allowed in the API, but their use discouraged in the docs (with proper explanations from those who know how to word them).
IMO the warnings you'd write there would be similar to the motivations of PEP 384. -- Amaury Forgeot d'Arc
So it means that, for example, a FileIO object couldn't be shared between runtimes either? How about a socket object? Do you want to forbid FileIO and socket objects as part of the API?
Python objects don't have this concern: all methods of FileIO are implemented in a single file (fileio.c), linked to a single C runtime.
Ah, right. But you still can call the fileno() method. Regards Antoine.
M.-A. Lemburg:
Is it possible to have multiple versions of the lib C loaded on Windows ?
Yes. It is possible not only to mix C runtimes from different vendors but different variants from a single vendor. Historically, each vendor has shipped their own C runtime libraries. This was also the case with CP/M and OS/2. Many applications can be extended with DLLs and if it were not possible to load DLLs which use different runtimes then that would limit which compilers can be used to extend particular applications. If Microsoft were to stop DLLs compiled with Borland or Intel from working inside Internet Explorer or Excel then there would be considerable controversy and likely anti-trust actions. Neil
On Tue, Aug 31, 2010 at 4:54 AM, M.-A. Lemburg <mal@egenix.com> wrote:
Is it possible to have multiple versions of the lib C loaded on Windows ?
Yes, and it's a pretty common situation. The fopen() that I call within a DLL may not be the same fopen() called by another DLL. When writing a DLL for Windows, the API must be designed with the assumption that anything returned by the C library cannot be passed a different C library. For example, suppose I a expose a function in my DLL that allocates some memory, populates it with useful information, and returns a pointer. I must also supply a function to free the memory. I cannot ask the caller to simply call free(), because their free() may not be using the same heap as my malloc(). Likewise, a FILE * isn't safe to pass around, unless I can guarantee that the application really is one big happy family compiled against the same version of the C library. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
Daniel Stutzbach wrote:
On Tue, Aug 31, 2010 at 4:54 AM, M.-A. Lemburg <mal@egenix.com> wrote:
Is it possible to have multiple versions of the lib C loaded on Windows ?
Yes, and it's a pretty common situation. The fopen() that I call within a DLL may not be the same fopen() called by another DLL. When writing a DLL for Windows, the API must be designed with the assumption that anything returned by the C library cannot be passed a different C library. For example, suppose I a expose a function in my DLL that allocates some memory, populates it with useful information, and returns a pointer. I must also supply a function to free the memory. I cannot ask the caller to simply call free(), because their free() may not be using the same heap as my malloc().
But isn't exactly that a major problem for Python ? An extension written for a different MS CRT version would not be able to safely free a buffer allocated by the Python DLL.
Likewise, a FILE * isn't safe to pass around, unless I can guarantee that the application really is one big happy family compiled against the same version of the C library.
According to the MS document I found this also applies to the OS environment and handles. Taking all these things together makes it sound like there's a rather small chance of actually getting PEP 384 working across Windows compiler upgrades. Then again, I haven't heard of many people actually running into those possible issues. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 27 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 18 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Neil Hodgson wrote:
M.-A. Lemburg:
Is it possible to have multiple versions of the lib C loaded on Windows ?
Yes. It is possible not only to mix C runtimes from different vendors but different variants from a single vendor.
Historically, each vendor has shipped their own C runtime libraries. This was also the case with CP/M and OS/2.
Many applications can be extended with DLLs and if it were not possible to load DLLs which use different runtimes then that would limit which compilers can be used to extend particular applications. If Microsoft were to stop DLLs compiled with Borland or Intel from working inside Internet Explorer or Excel then there would be considerable controversy and likely anti-trust actions.
Thanks for the feedback. This sounds like the issues such a mix can cause are mostly theoretical and don't really bother much in practice, so PEP 384 on Windows does have a chance :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 27 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 18 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Daniel Stutzbach wrote:
Likewise, a FILE * isn't safe to pass around, unless I can guarantee that the application really is one big happy family compiled against the same version of the C library.
Given that, it seems to me that it's a mistake for Python to provide any APIs that take a FILE* directly in the first place. Maybe PyObject_Print should be deprecated in favour of something that takes a Python I/O object instead. -- Greg
On Wed, Sep 1, 2010 at 7:49 AM, M.-A. Lemburg <mal@egenix.com> wrote:
Yes, and it's a pretty common situation. The fopen() that I call within a DLL may not be the same fopen() called by another DLL. When writing a DLL for Windows, the API must be designed with the assumption that anything returned by the C library cannot be passed a different C library. For example, suppose I a expose a function in my DLL that allocates some memory, populates it with useful information, and returns a pointer. I must also supply a function to free the memory. I cannot ask the caller to simply call free(), because their free() may not be using the same heap as my malloc().
But isn't exactly that a major problem for Python ?
An extension written for a different MS CRT version would not be able to safely free a buffer allocated by the Python DLL.
Correct, but we generally don't allow raw pointers across the API boundary like that. Instead, users are told to release memory with PyMem_Free, PyObject_Delete, reference decrementing, etc. That means the memory release happens in the runtime linked with the Python DLL and everything is fine.
Likewise, a FILE * isn't safe to pass around, unless I can guarantee that the application really is one big happy family compiled against the same version of the C library.
According to the MS document I found this also applies to the OS environment and handles.
Taking all these things together makes it sound like there's a rather small chance of actually getting PEP 384 working across Windows compiler upgrades.
Not really - we just need to keep an eye out for the places where those things can leak through and ensure they're instead abstracted so the extension module only talks to Python rather than being given direct access to the underlying C runtime details. FILE* is one, errno is another, locale we may just have to live with (alternatively, we could add a callback API to allow extension modules to be notified when the locale changes, which may be useful for other reasons... see issue9727). Antoine pointed out file descriptors as another possible problem, but I'm not sure we can prevent that at the API level (there are too many ways for Python code to get at file descriptors - it is likely better to just say that if you get a file descriptor from Python, only use it with Python APIs, not directly with the C runtime).
Then again, I haven't heard of many people actually running into those possible issues.
Even today you can use a different C runtime in a Python extension so long as you avoid the APIs that would cause problems (e.g. I have legacy SWIG-wrapped extensions built with VC6 that run quite happily even when linked against Python 2.4). The idea with the stable ABI is to explicitly exclude those functions so that you *know* you aren't going to run into those problems. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Aug 31, 2010 at 2:15 PM, Mark Hammond <skippy.hammond@gmail.com> wrote:
It would be interesting to know how, in practice, these FILE pointers come to life. In my experience they are generally obtained via fopen. If that is broadly true, then a middle-ground may be for Python to expose something like Py_fopen, Py_fclose and a PyFILE opaque "handle". API elements which currently take a FILE * could be exposed using a PyFILE * in the ABI. People who didn't care about this level of portability could continue to use the non-ABI FILE * functions, but people who do could use Py_fopen/Py_fclose in place of fopen/fclose but otherwise work as now.
No need for that - the Python-specific equivalent is our own I/O stack (and things like os.fdopen). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 01 Sep 2010 10:23:42 +1200 Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Daniel Stutzbach wrote:
Likewise, a FILE * isn't safe to pass around, unless I can guarantee that the application really is one big happy family compiled against the same version of the C library.
Given that, it seems to me that it's a mistake for Python to provide any APIs that take a FILE* directly in the first place.
What I think would be a mistake would be to define the API in terms of Windows-specific quirks. All this discussion seems bent on satisfying the needs of Windows developers at the expense of non-Windows developers. "FILE*" is a perfectly standard C feature (and a widely-used one at that). If Windows doesn't handle it correctly then it's a Windows issue and we shouldn't annoy other people by refusing access to that feature. Again, putting a warning in the documentation should be enough for those people who want to have perfect Windows support. And if the Microsoft compiler/linker doesn't bother warning the user when passing FILE* between various CRTs, then it's not Python's job to supplement it. After all, we don't usually try to workaround platform-specific bugs (not as a low level such as the C API level); at worse, we mention their existence in the docs. Regards Antoine.
M.-A. Lemburg wrote:
But isn't exactly that a major problem for Python ?
An extension written for a different MS CRT version would not be able to safely free a buffer allocated by the Python DLL.
Python provides its own set of memory alloc/free functions to deal with this exact problem. For every Python function that allocates something, there's a corresponding function for freeing it, and you do it any other way at your peril. -- Greg
On Wed, Sep 1, 2010 at 1:42 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
What I think would be a mistake would be to define the API in terms of Windows-specific quirks. All this discussion seems bent on satisfying the needs of Windows developers at the expense of non-Windows developers. "FILE*" is a perfectly standard C feature (and a widely-used one at that). If Windows doesn't handle it correctly then it's a Windows issue and we shouldn't annoy other people by refusing access to that feature.
As an alternative the stable ABI could use the native system file handle instead of FILE*. HANDLE (returned by CreateFile) on Windows and int (returned by open) on Linux (or even FILE*). The extension writer would have to write it's own abstraction over this. Python could provide a default wrapper (as a sample). Unfortunately you would also lose the buffering provided by fopen (unless the sample also does this). As a side note, I wrote a plugin for a big Windows application (Mastercam) once. There were at least 5 different CRTs running in that process - 3 different release versions and 2 different debug versions. Besides a few other statically linked into other plugin DLLs. This is quite common, especially since you can have foreign CRTs injected into your process by various shell extensions like TortoiseSVN.
On Wed, Sep 1, 2010 at 8:42 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
After all, we don't usually try to workaround platform-specific bugs (not as a low level such as the C API level); at worse, we mention their existence in the docs.
You persist in viewing the allowance of multiple C runtimes in a single process as a bug instead of a feature. This comes from the monolithic approach to software management in Unix systems (with the associated web of dependencies and implications for backwards compatibility management). The Windows heritage is different, allowing each application to be largely independent of everything other than the OS itself, so the execution model is also different. Windows is a supported platform for Python and to meet the promise of a stable ABI, we need to support multiple C runtimes in a single process. Since we aren't in the habit of passing low level objects around, the number of APIs affected is relatively tiny (the current tally is less than half a dozen that I know of - the two mentioned in this thread, plus a couple of of the PyRun_* APIs). There will be some cases (like file descriptors) where we just say "don't do that" but those should be fairly rare. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le mercredi 01 septembre 2010 à 22:43 +1000, Nick Coghlan a écrit :
On Wed, Sep 1, 2010 at 8:42 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
After all, we don't usually try to workaround platform-specific bugs (not as a low level such as the C API level); at worse, we mention their existence in the docs.
You persist in viewing the allowance of multiple C runtimes in a single process as a bug instead of a feature.
No, I view the fact that you can't share FILE structures as a bug. I'm sure there would be ways to have multiple C runtimes loaded in memory with compatible FILE structures (for example, by versioning the FILE structure itself, or by embedding inside the FILE structure a set of function pointers, so that fread(), fwrite() and friends always get redirected to the proper implementation). Please consider this: even without relying on PEP 384, using FILE* is /already/ dangerous; because you might compile an extension with a different compiler version than Python was compiled with. So, if we were following you, we should rip out PyObject_Print() of the whole C API, not only the limited subset which is defined by PEP 384. (now I have nothing against completely ripping out PyObject_Print() if we find out that it's not really useful...) Regards Antoine.
On 01/09/2010 13:54, Antoine Pitrou wrote:
[snip...] Please consider this: even without relying on PEP 384, using FILE* is /already/ dangerous; because you might compile an extension with a different compiler version than Python was compiled with. So, if we were following you, we should rip out PyObject_Print() of the whole C API, not only the limited subset which is defined by PEP 384.
Definitely. On Windows it is relatively common to configure distutils to compile extensions with Mingw, which is likely to have problems with FILE* from the Microsoft C runtime [1]. Michael [1] I say *likely* because I have vague memories of Mingw being able to compile against msvcrt, but I'm not sure about this. This is actually less of an issue at the moment as we are now using a Microsoft compiler for which a free version is available. This wasn't the case for a while (and still isn't for those using Python 2.5 and earlier).
(now I have nothing against completely ripping out PyObject_Print() if we find out that it's not really useful...)
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On Wed, Sep 1, 2010 at 10:54 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Please consider this: even without relying on PEP 384, using FILE* is /already/ dangerous; because you might compile an extension with a different compiler version than Python was compiled with. So, if we were following you, we should rip out PyObject_Print() of the whole C API, not only the limited subset which is defined by PEP 384.
(now I have nothing against completely ripping out PyObject_Print() if we find out that it's not really useful...)
I think it would be better if everything dealing with FILE* was a macro rather than a function, yes. The definition of the limited API is a chance to fix that without incurring the cost in backwards incompatibility that would otherwise arise. Since we have that opportunity, why not take it? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, 2 Sep 2010 07:04:31 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Sep 1, 2010 at 10:54 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Please consider this: even without relying on PEP 384, using FILE* is /already/ dangerous; because you might compile an extension with a different compiler version than Python was compiled with. So, if we were following you, we should rip out PyObject_Print() of the whole C API, not only the limited subset which is defined by PEP 384.
(now I have nothing against completely ripping out PyObject_Print() if we find out that it's not really useful...)
I think it would be better if everything dealing with FILE* was a macro rather than a function, yes. The definition of the limited API is a chance to fix that without incurring the cost in backwards incompatibility that would otherwise arise. Since we have that opportunity, why not take it?
Maybe I've missed your answer, but what would prevent the "inline" solution from working? (a macro with the result-as-a-pointer is quite ugly) Regards Antoine.
On Thu, Sep 2, 2010 at 7:19 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Maybe I've missed your answer, but what would prevent the "inline" solution from working?
Only PEP 7 (since standard support for inline is a C99 feature) On the other hand, if gcc (including cygwin/mingw) and MSVC support it (even with non-standard spellings), then that objection probably doesn't matter.
(a macro with the result-as-a-pointer is quite ugly) Agreed.
Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Sep 1, 2010 at 10:54 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Please consider this: even without relying on PEP 384, using FILE* is /already/ dangerous; because you might compile an extension with a different compiler version than Python was compiled with. So, if we were following you, we should rip out PyObject_Print() of the whole C API, not only the limited subset which is defined by PEP 384.
(now I have nothing against completely ripping out PyObject_Print() if we find out that it's not really useful...)
I finally took the obvious step of grepping the include directory to see what APIs were even affected by this question. Turns out there are more APIs than I thought, but most extension modules aren't going to need most of them (since they're related to code parsing, compilation and execution directly from files). For the remainder, I propose as a starting point that users of the stable ABI be directed to the Python equivalents, with the possibility of later adding inline functions for more commonly used API elements. PyObject_Print is the only one I would bother providing as part of the initial stable ABI implementation. Cheers, Nick. Search criteria: grep "FILE \?\*" * object.h:typedef int (*printfunc)(PyObject *, FILE *, int); object.h:PyAPI_FUNC(int) PyObject_Print(PyObject *, FILE *, int); - should provide inline equivalent fileobject.h:PyAPI_FUNC(char *) Py_UniversalNewlineFgets(char *, int, FILE*, PyObject *); marshal.h:PyAPI_FUNC(void) PyMarshal_WriteLongToFile(long, FILE *, int); marshal.h:PyAPI_FUNC(void) PyMarshal_WriteObjectToFile(PyObject *, FILE *, int); marshal.h:PyAPI_FUNC(long) PyMarshal_ReadLongFromFile(FILE *); marshal.h:PyAPI_FUNC(int) PyMarshal_ReadShortFromFile(FILE *); marshal.h:PyAPI_FUNC(PyObject *) PyMarshal_ReadObjectFromFile(FILE *); marshal.h:PyAPI_FUNC(PyObject *) PyMarshal_ReadLastObjectFromFile(FILE *); parsetok.h:PyAPI_FUNC(node *) PyParser_ParseFile (FILE *, const char *, grammar *, int, parsetok.h:PyAPI_FUNC(node *) PyParser_ParseFileFlags(FILE *, const char *, parsetok.h:PyAPI_FUNC(node *) PyParser_ParseFileFlagsEx(FILE *, const char *, pythonrun.h:PyAPI_FUNC(int) PyRun_AnyFileFlags(FILE *, const char *, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(int) PyRun_AnyFileExFlags(FILE *, const char *, int, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(int) PyRun_SimpleFileExFlags(FILE *, const char *, int, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(int) PyRun_InteractiveOneFlags(FILE *, const char *, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(int) PyRun_InteractiveLoopFlags(FILE *, const char *, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(struct _mod *) PyParser_ASTFromFile(FILE *, const char *, pythonrun.h:PyAPI_FUNC(struct _node *) PyParser_SimpleParseFileFlags(FILE *, const char *, pythonrun.h:PyAPI_FUNC(PyObject *) PyRun_FileExFlags(FILE *, const char *, int, pythonrun.h:PyAPI_FUNC(int) Py_FdIsInteractive(FILE *, const char *); pythonrun.h:PyAPI_FUNC(char *) PyOS_Readline(FILE *, FILE *, char *); pythonrun.h:PyAPI_DATA(char) *(*PyOS_ReadlineFunctionPointer)(FILE *, FILE *, char *); - omit from stable ABI, recommend equivalent Python calls (may add inline versions to stable ABI over time if requested by developers) abstract.h: int PyObject_Print(PyObject *o, FILE *fp, int flags); - comment, not declaration (actual declaration is in object.h) grammar.h:void printgrammar(grammar *g, FILE *fp); grammar.h:void printnonterminals(grammar *g, FILE *fp); object.h:PyAPI_FUNC(void) _Py_PrintReferences(FILE *); object.h:PyAPI_FUNC(void) _Py_PrintReferenceAddresses(FILE *); Python.h:PyAPI_FUNC(FILE *) _Py_wfopen(const wchar_t *path, const wchar_t *mode); Python.h:PyAPI_FUNC(FILE*) _Py_fopen(PyObject *unicode, const char *mode); - private, omit from stable ABI -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Sep 3, 2010 at 12:44 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
On 02/09/10 09:04, Nick Coghlan wrote:
I think it would be better if everything dealing with FILE* was a macro rather than a function, yes.
How would that help?
Macros (and, as Antoine pointed out, inline functions) will never cross C runtime boundaries, even on Windows. They'll get compiled in when the extension module is built and linked to a C runtime library from there, so it won't matter if that differs from the one that is linked to the Python binary. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
It would be interesting to know how, in practice, these FILE pointers come to life. In my experience they are generally obtained via fopen.
I think that experience can't be generalized. I personally guess that in most cases, the FILE* being passed across CRT boundaries is stdout.
If that is broadly true, then a middle-ground may be for Python to expose something like Py_fopen, Py_fclose and a PyFILE opaque "handle". API elements which currently take a FILE * could be exposed using a PyFILE * in the ABI. People who didn't care about this level of portability could continue to use the non-ABI FILE * functions, but people who do could use Py_fopen/Py_fclose in place of fopen/fclose but otherwise work as now.
I don't think this would solve the problem; see above. Regards, Martin
This sounds like the issues such a mix can cause are mostly theoretical and don't really bother much in practice, so PEP 384 on Windows does have a chance :-)
Actually, the CRT issues (FILE* in particular) have been causing real crashes for many years, for many people. Regards, Martin
What I think would be a mistake would be to define the API in terms of Windows-specific quirks. All this discussion seems bent on satisfying the needs of Windows developers at the expense of non-Windows developers. "FILE*" is a perfectly standard C feature (and a widely-used one at that). If Windows doesn't handle it correctly then it's a Windows issue and we shouldn't annoy other people by refusing access to that feature.
The point of the PEP is specifically to solve Python versioning problems *ON WINDOWS*. It would be pointless if it didn't have that effect. If you think it doesn't affect you, then you can just ignore the stable ABI.
After all, we don't usually try to workaround platform-specific bugs (not as a low level such as the C API level); at worse, we mention their existence in the docs.
Yes, and that policy is an ongoing annoyance, one that PEP 384 tries to remedy. Only after I wrote the PEP, I learned that it can have uses for Linux distributions, too. Regards, Martin
Please consider this: even without relying on PEP 384, using FILE* is /already/ dangerous; because you might compile an extension with a different compiler version than Python was compiled with.
It's only dangerous *if* you compile with a different compiler. That's why we take serious precautions that you never ever do. For example, distutils will, by default, refuse to use a different compiler. Regards, Martin
On Sat, 04 Sep 2010 15:13:55 +0200 "Martin v. Löwis" <martin@v.loewis.de> wrote:
Please consider this: even without relying on PEP 384, using FILE* is /already/ dangerous; because you might compile an extension with a different compiler version than Python was compiled with.
It's only dangerous *if* you compile with a different compiler. That's why we take serious precautions that you never ever do. For example, distutils will, by default, refuse to use a different compiler.
Thanks for the explanations. Regards Antoine.
Hello Martin, On Sat, 28 Aug 2010 12:04:10 +0200 "Martin v. Löwis" <martin@v.loewis.de> wrote:
I have now started an initial patch for PEP 384, in the pep-0384 branch. [...]
On http://bugs.python.org/issue9778 you elaborated on what the PEP would entail in its current state: “No, vice versa. The PEP promises that the ABI won't change until Python 4. For any change that might break the ABI, either a backwards-compatible solution needs to be found, or the change be deferred to Python 4.” This sounds like it could be detrimental by blocking desired improvements (the aforementioned issue is a potential example of this). Do you think it would complicate things a lot to version the ABI itself? Actually, PYTHON_API_VERSION is already used as some kind of ABI tag (since it's checked at module load time rather than at compile time). Regards Antoine.
"Martin v. Löwis" wrote:
This sounds like the issues such a mix can cause are mostly theoretical and don't really bother much in practice, so PEP 384 on Windows does have a chance :-)
Actually, the CRT issues (FILE* in particular) have been causing real crashes for many years, for many people.
Do you have some pointers ? I don't remember this being a real practical issue, at least not for different versions of the MS CRTs. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 07 2010)
Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 7 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Wed, Sep 8, 2010 at 2:48 AM, M.-A. Lemburg <mal@egenix.com> wrote:
"Martin v. Löwis" wrote:
This sounds like the issues such a mix can cause are mostly theoretical and don't really bother much in practice, so PEP 384 on Windows does have a chance :-)
Actually, the CRT issues (FILE* in particular) have been causing real crashes for many years, for many people.
Do you have some pointers ?
I don't have a bug report, but I have had issues in my audiolab package, where the package itself is built with visual studio, but the library it is linked against is built with mingw (libsndfile, which cannot be built with visual studio). The problem happens when you want to use the mingw-built library which accepts a file descriptor that you create with mkstemp inside code built with Visual Studio. This issue is actually quite easy to reproduce.
I don't remember this being a real practical issue, at least not for different versions of the MS CRTs.
I would turn the question around: what are the cases where you manage to mix CRT and not getting any issues ? This has never worked in my own experience in the context of python extensions, cheers, David
On Wed, Sep 8, 2010 at 8:34 AM, David Cournapeau <cournape@gmail.com> wrote:
I would turn the question around: what are the cases where you manage to mix CRT and not getting any issues ? This has never worked in my own experience in the context of python extensions,
I've done it quite a bit over the years with SWIG-wrapped libraries where all I did was recompile my extension for new versions of Python. It works fine, so long as you avoid the very APIs that PEP 384 is blocking from the stable ABI. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Sep 8, 2010 at 5:19 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Sep 8, 2010 at 8:34 AM, David Cournapeau <cournape@gmail.com> wrote:
I would turn the question around: what are the cases where you manage to mix CRT and not getting any issues ? This has never worked in my own experience in the context of python extensions,
I've done it quite a bit over the years with SWIG-wrapped libraries where all I did was recompile my extension for new versions of Python. It works fine, so long as you avoid the very APIs that PEP 384 is blocking from the stable ABI.
What do you mean by recompiling ? Using Mingw ? Normally, if you just recompile your extensions without using special distutils options, you use the same compiler as the one used by python, which generally avoids mixing runtimes. In other words, the problem mainly arises when you need to integrate libraries which you can not recompile with the compiler used by python, because the code is not visual-studio compatible, or because the library is only available in binary form. cheers, David
On Wed, Sep 8, 2010 at 6:34 PM, David Cournapeau <cournape@gmail.com> wrote:
In other words, the problem mainly arises when you need to integrate libraries which you can not recompile with the compiler used by python, because the code is not visual-studio compatible, or because the library is only available in binary form.
In my case, I was building on an old dev system which only has VC6, but needed to link against Python 2.4 (which was compiled with MSVC 2005). The build process didn't use distutils, so that didn't affect anything. It works, you just have to know what APIs you have to avoid. The C runtime independence aspects of PEP 384 are just about enlisting the compiler in the task of making sure you *have* avoided all those APIs. The rest of the PEP is about hiding the details of some of Python's own data structures to allow them to change without affecting the stable ABI. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Sep 8, 2010 at 7:59 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Sep 8, 2010 at 6:34 PM, David Cournapeau <cournape@gmail.com> wrote:
In other words, the problem mainly arises when you need to integrate libraries which you can not recompile with the compiler used by python, because the code is not visual-studio compatible, or because the library is only available in binary form.
In my case, I was building on an old dev system which only has VC6, but needed to link against Python 2.4 (which was compiled with MSVC 2005). The build process didn't use distutils, so that didn't affect anything.
ok, I was confused by "I just recompiled".
It works, you just have to know what APIs you have to avoid.
The critical point is that you cannot always do that. Retaking my example of mkstemp: I have a C library which has a fdopen-like function (built with one C runtime, not the same as python), there is no way that I know of to use this API with a file descriptor obtained from tempfile.mkstemp function. The only solution is to build my own C extension with C mkstemp, built with the same runtime as my library, and make that available to python. cheers, David
Am 07.09.2010 19:48, schrieb M.-A. Lemburg:
"Martin v. Löwis" wrote:
This sounds like the issues such a mix can cause are mostly theoretical and don't really bother much in practice, so PEP 384 on Windows does have a chance :-)
Actually, the CRT issues (FILE* in particular) have been causing real crashes for many years, for many people.
Do you have some pointers ?
You mean, other than FILE* :-? Unfortunately, I don't have time to search for reports, but they had been several on comp.lang.python over the years. As others say, this is *really* easy to reproduce.
I don't remember this being a real practical issue, at least not for different versions of the MS CRTs.
You probably had not been passing FILE* across CRT boundaries, then. Notice that this is an uncommon thing to do, except that the Python API has some functions that expect FILE*. As others have mentioned, CRT file handles are a problem, too, but less so: a) they aren't explicitly passed in any of the Python APIs (except for the obvious os.* functions), b) they don't cause crashes, just outputs to the wrong files, and c) if they are 0/1/2, they actually do the right thing. Regards, Martin
On http://bugs.python.org/issue9778 you elaborated on what the PEP would entail in its current state:
“No, vice versa. The PEP promises that the ABI won't change until Python 4. For any change that might break the ABI, either a backwards-compatible solution needs to be found, or the change be deferred to Python 4.”
This sounds like it could be detrimental by blocking desired improvements (the aforementioned issue is a potential example of this).
Notice that it's only potential: in the specific case, there would be an ABI-compatible way of introducing wide hashes, using a second type slot.
Do you think it would complicate things a lot to version the ABI itself?
It would miss the point. The ABI deliberately restricts itself to features that really have been stable, and we should be committed to keeping that subset stable. If you think this is too restrictive, please point out specific aspects that you think might need to change in the mid-term future. They should then be excluded from the ABI.
Actually, PYTHON_API_VERSION is already used as some kind of ABI tag (since it's checked at module load time rather than at compile time).
It's too late, though: when the module is being created, the code using the ABI is already running. As I wrote in the issue: the *real* ABI version is "python3.dll". Regards, Martin
On Sun, 12 Sep 2010 19:38:33 +0200 "Martin v. Löwis" <martin@v.loewis.de> wrote:
On http://bugs.python.org/issue9778 you elaborated on what the PEP would entail in its current state:
“No, vice versa. The PEP promises that the ABI won't change until Python 4. For any change that might break the ABI, either a backwards-compatible solution needs to be found, or the change be deferred to Python 4.”
This sounds like it could be detrimental by blocking desired improvements (the aforementioned issue is a potential example of this).
Notice that it's only potential: in the specific case, there would be an ABI-compatible way of introducing wide hashes, using a second type slot.
Yes, but it would add complication, and be potentially detrimental to performance.
If you think this is too restrictive, please point out specific aspects that you think might need to change in the mid-term future. They should then be excluded from the ABI.
I have no a priori knowledge of what might happen in the future :) That said, looking at the PEP, I was wondering whether fields such as ob_type, ob_refcnt, ob_size have to be directly accessible, rather than through a macro-turned-into-a-function such as Py_REFCNT().
That said, looking at the PEP, I was wondering whether fields such as ob_type, ob_refcnt, ob_size have to be directly accessible, rather than through a macro-turned-into-a-function such as Py_REFCNT().
That they are macros still is primarily for performance reasons. For ob_type, that may be questionable; for Py_INCREF, I hope you agree that it's really desirable for it to be inline. However, more importantly, user-defined objects need to start with the standard object header. I really see no way to let extensions define types without them also being able to access their own struct fields, which in turn requires compile-time (ABI) knowledge of PyObject_HEAD. This is also the reason why Py_TRACE_REFS can't be supported in the ABI: it increases sizeof(PyObject), and thus shifts all user fields in the object. I actually *do* have a priori knowledge of what might happen in the future :-): If we were to change the layout of the object header in a significant way, then the majority of extension modules would break - not only on the binary level, but also on the source level. So any change to this likely justifies calling it Python 4. (I would be that even Python 4 keeps that structure) Regards, Martin
participants (15)
-
"Martin v. Löwis" -
Adal Chiriliuc -
Amaury Forgeot d'Arc -
Antoine Pitrou -
Barry Warsaw -
Benjamin Peterson -
Daniel Stutzbach -
David Cournapeau -
Greg Ewing -
James Y Knight -
M.-A. Lemburg -
Mark Hammond -
Michael Foord -
Neil Hodgson -
Nick Coghlan