On Sun, Aug 29, 2010 at 4:52 AM, "Martin v. Löwis" wrote:

...

This is from tp_new and tp_dealloc, right? I think we should probably provide assessors PyObject_Alloc and PyObject_FreeObject.

Correct, and yes, that sounds like a good approach.

...

...

- PyObject_Print is used, but can't be supported, as it uses a FILE*  parameter

I thought tp_print was supposed to have been removed.

Yes - that should have happened for 3.0. Not sure how to deal with it now.

tp_print actually is gone, but PyObject_Print was retained. It just relies on repr() and str() under the hood instead of the old tp_print slot. There are 4 operations performed on fp in that function: clearerr, ferror, fprintf and fwrite. There is also an implicit reference to errno (through PyErr_SetFromErrno) which can't be trusted in a mixed CRT world (PyErr_SetFromErrno() should be excluded from the Limited API, if it isn't already, and replaced with a PyErr_SetFromErrnoEx which takes the errno as an explicit argument. That assumes the CRTs will be sufficiently compatible that strerror() will give the correct answer for errno values from the other CRT. If that assumption is incorrect, the new function would also need to accept a string argument for the error description). Four options come to mind: - just leave it out of the limited API, extensions can do their own thing to print objects - leave PyObject_Print out of the limited API, but create a PyObject_PrintEx that takes a Python IO stream via PyObject* rather than a C level FILE*. - leave PyObject_Print out of the limited API, but create a PyObject_PrintEx that takes function pointers for the above 4 operations (so the FILE* pointer is only every operated on by functions from the extension module's CRT) - leave PyObject_Print out of the limited API, but create a PyObject_PRINT macro that does much the same thing with the logic rearranged so there is an inner function that figures out the string to be printed, but an outer macro that does all the operations on the FILE * object (so again, the FILE * is never passed to Python's CRT) The last option requires the fewest adjustments for extension authors, and it should be feasible to do it that way (even though it is a bit of a hack). Something along the lines of the following: #define PyObject_PRINT (obj, fp, flags, resultp) \ { \ int _result = -1; _t = _PyObject_PrintInner(obj, flags); \ if (_t != NULL) { \ clearerr(fp); \ fwrite(PyBytes_AS_STRING(_t), 1, PyBytes_GET_SIZE(_t), fp); \ Py_DECREF(_t); \ if (ferror(fp)) { \ PyErr_SetFromErrnoEx(PyExc_IOError, errno); \ clearerr(fp); \ } else { \ _result = 0; \ } \ } \ if (resultp != NULL) { \ *resultp = _result; \ } \ } Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Sun, Aug 29, 2010 at 6:24 PM, Antoine Pitrou wrote:

On Sun, 29 Aug 2010 09:20:56 +1000 Nick Coghlan wrote:

...

Four options come to mind:

- just leave it out of the limited API, extensions can do their own thing to print objects - leave PyObject_Print out of the limited API, but create a PyObject_PrintEx that takes a Python IO stream via PyObject* rather than a C level FILE*. - leave PyObject_Print out of the limited API, but create a PyObject_PrintEx that takes function pointers for the above 4 operations (so the FILE* pointer is only every operated on by functions from the extension module's CRT) - leave PyObject_Print out of the limited API, but create a PyObject_PRINT macro that does much the same thing with the logic rearranged so there is an inner function that figures out the string to be printed, but an outer macro that does all the operations on the FILE * object (so again, the FILE * is never passed to Python's CRT)

Fifth option: - make PyObject_Print() an inline function (similar to your macro  proposal), but only on Windows. This would retain the name and  current signature. Apparently we could use something like  "__forceinline" or "extern __forceinline"?

I believe both that option, and my third option, would run into trouble due to the potential for errno confusion. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Sun, Aug 29, 2010 at 7:10 PM, Antoine Pitrou wrote:

On Sun, 29 Aug 2010 18:41:45 +1000 Nick Coghlan wrote:

...

I believe both that option, and my third option, would run into trouble due to the potential for errno confusion.

I don't understand. What's the difference with the macro you proposed (the fourth option)?

Oh, you meant combining it with the suggested change to the errno handling rather than merely inlining the existing PyObject_Print? Yes, that should work and would be even more seamless from the point of view of the extension modules. However, since even platforms other than Windows aren't immune to version upgrades of the standard C runtime, I'm still more comfortable with the idea that the strict ABI should refuse to pass FILE* pointers across extension module boundaries. The advantage of the preprocessor macro approach is that it allows us to provide assistance with operations on FILE* pointers while still obeying that restriction regardless of the specific compilers involved in creating the Python and extension module binaries. It would be nice to get rid of the resultp parameter by rewriting the internals of the macro as a collection of ternary and comma expressions instead of a scope, but I haven't managed to crack the problem of feeding the correct arguments to fwrite with that approach (it's been quite a while since I've attempted to abuse C expressions to this extent, so I've forgotten most of the tricks for avoiding temporary variables). Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Sun, 29 Aug 2010 22:16:57 +1000 Nick Coghlan wrote:

However, since even platforms other than Windows aren't immune to version upgrades of the standard C runtime, I'm still more comfortable with the idea that the strict ABI should refuse to pass FILE* pointers across extension module boundaries.

I'm not sure what the worry is. Under an Unix-like system, there shouldn't be any situation where two different versions of the libc are loaded in the same process. And since FILE* is an opaque pointer whose referent is only accessed by the libc, things should be fine. (actually, I'm baffled that Windows has such problems, and I would suggest that it's not Python's job to shield Windows application developers from Windows-specific development issues)

The advantage of the preprocessor macro approach is that it allows us to provide assistance with operations on FILE* pointers while still obeying that restriction regardless of the specific compilers involved in creating the Python and extension module binaries.

An inline function should provide the same advantage since it will be compiled with the calling library/executable, rather than the Python DLL. Regards Antoine.

On Mon, 30 Aug 2010 07:31:34 +1000 Nick Coghlan wrote:

Since part of the point of PEP 384 is to support multiple versions of the C runtime in a single process, [...]

I think that's quite a maximalist goal. The point of PEP 384 should be to define a standard API for Python, (hopefully) spanning multiple versions. Whether the API effectively guarantees a standard ABI can only depend on whether the system itself hasn't changed its own conventions (including, for example, function call conventions, or the binary representation of standard C types). In other words, PEP 384 should only care to stabilize the ABI as long as the underlying system doesn't change. It sounds a bit foolish for us to try to hide potential unstabilities in the underlying platform. And it's equally foolish to try to forbid people from using well-known system facilities such as FILE* or (worse) errno. So, perhaps the C API docs can simply mention the caveat of using FILE* (and perhaps errno, if there's a problem there as well) for C extensions under Windows. C extension writers are (usually) consenting adults, for all. Regards Antoine.

On Mon, Aug 30, 2010 at 7:43 AM, Antoine Pitrou wrote:

On Mon, 30 Aug 2010 07:31:34 +1000 Nick Coghlan wrote:

...

Since part of the point of PEP 384 is to support multiple versions of the C runtime in a single process, [...]

I think that's quite a maximalist goal. The point of PEP 384 should be to define a standard API for Python, (hopefully) spanning multiple versions. Whether the API effectively guarantees a standard ABI can only depend on whether the system itself hasn't changed its own conventions (including, for example, function call conventions, or the binary representation of standard C types).

FILE* is very different from the other things you mention. Function call conventions and binary representations are defined in the C standard. FILE*, on the other hand, is explicitly called out as an opaque reference completely under the control of the specific C runtime implementation. Since the Linkage section of PEP 384 specifically states the availability of a generic "python3.dll" that dynamically redirects to the relevant "python3y.dll" to allow an extension module to run against any 3.2 or later Python version as a goal of the PEP, I would say that allowing mixing of C runtimes is definitely one of the PEP's goals. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Aug 30, 2010, at 10:18 PM, Nick Coghlan wrote:

Since the Linkage section of PEP 384 specifically states the availability of a generic "python3.dll" that dynamically redirects to the relevant "python3y.dll" to allow an extension module to run against any 3.2 or later Python version as a goal of the PEP, I would say that allowing mixing of C runtimes is definitely one of the PEP's goals.

It should be explicit about that then, and provide detail about why the runtime is relevant to Windows programmers (and probably not relevant in practice for *nix programmers). -Barry

On Tue, Aug 31, 2010 at 12:47 AM, Michael Foord wrote:

An extension compiled for one version of Python will be linked against the version of the C runtime used by that version of Python (if it is compiled with the same version of Visual Studio of course).

If the extension binary is to remain compatible with a later version of Python, compiled against a different version of the C runtime, then it *must* be possible for multiple C runtimes to be loaded. If the stable ABI doesn't allow this then binaries will *still* have to be recompiled when we update the version of Visual Studio used to compile Python - defeating the purpose of the PEP. Right?

If this is the case then I agree that it should be explicit in the PEP.

Ah, I knew it was explicit in the PEP somewhere. The following is currently mentioned in the "Excluded Functions" section: "In addition, functions expecting FILE* are not part of the ABI, to avoid depending on a specific version of the Microsoft C runtime DLL on Windows." To make that explicit in the Rationale section, I would append a second sentence to the final paragraph of that section (the first sentence is already there): "With this PEP, it will be possible to reduce the dependency of binary extension modules on a specific Python feature release, and applications embedding Python can be made work with different releases. To ensure this is also achieved on Windows, the ABI will be designed to allow the Python binary and extension modules or an embedding application to depend on different versions of the C runtime DLL." I would also make the following modification to the above quoted paragraph from the Excluded Functions section of the PEP: "To avoid depending on a specific version of the Microsoft C runtime DLL on Windows while still providing a consistent, cross-platform API when PY_LIMITED_API is defined, the following functions are also excluded: - Any functions accepting or returning FILE* (as this is defined in the C standard as an implementation dependent opaque reference. On Windows, it is known to contain reference to runtime-specific global data) - Any functions reliant on global or thread local state expected to be modified by C standard library calls rather than other Python API calls (e.g. PyErr_FromErrno will be unavailable. Comparable functionality will be provided by a new function PyErr_FromErrnoEx, with the latter accepting an explicit errno parameter) Where practical, equivalent functionality for the excluded functions will instead be provided using preprocessor macros. This ensures that the use of the C runtime dependent types and data remains separate without significantly complicating extension module development and the API remains compatible with any C89 compiler. Note that locale dependent operations will still be available, they just may not see locale changes made via the C standard library calls. Locale changes will need to be made via Python API calls to the locale module to ensure they are seen correctly by the interpreter." Hmm... that last point is a bit of any issue actually, since it also flows the other way (changes made via the locale module won't be visible to any extension modules using a different C runtime). So I suspect mixing C runtimes is still going to come with the caveat of potential locale related glitches. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Tue, 31 Aug 2010 11:12:17 +0900 David Cournapeau wrote:

...

Hmm... that last point is a bit of any issue actually, since it also flows the other way (changes made via the locale module won't be visible to any extension modules using a different C runtime). So I suspect mixing C runtimes is still going to come with the caveat of potential locale related glitches.

As far as IO is concerned, FILE* is just a special case of a more generic issue, though, so maybe this could be a bit reworded. For example, file descriptor cannot be shared between runtimes either.

Er, really? So it means that, for example, a FileIO object couldn't be shared between runtimes either? How about a socket object? Do you want to forbid FileIO and socket objects as part of the API? Again, I propose that FILE* functions are allowed in the API, but their use discouraged in the docs (with proper explanations from those who know how to word them). Regards Antoine.

...

So it means that, for example, a FileIO object couldn't be shared between runtimes either? How about a socket object? Do you want to forbid FileIO and socket objects as part of the API?

Python objects don't have this concern: all methods of FileIO are implemented in a single file (fileio.c), linked to a single C runtime.

Ah, right. But you still can call the fileno() method. Regards Antoine.

On Tue, Aug 31, 2010 at 2:15 PM, Mark Hammond wrote:

It would be interesting to know how, in practice, these FILE pointers come to life.  In my experience they are generally obtained via fopen. If that is broadly true, then a middle-ground may be for Python to expose something like Py_fopen, Py_fclose and a PyFILE opaque "handle".  API elements which currently take a FILE * could be exposed using a PyFILE * in the ABI.  People who didn't care about this level of portability could continue to use the non-ABI FILE * functions, but people who do could use Py_fopen/Py_fclose in place of fopen/fclose but otherwise work as now.

No need for that - the Python-specific equivalent is our own I/O stack (and things like os.fdopen). Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Michael Foord wrote:

On 30/08/2010 17:35, Barry Warsaw wrote:

...

On Aug 30, 2010, at 10:18 PM, Nick Coghlan wrote:

...

Since the Linkage section of PEP 384 specifically states the availability of a generic "python3.dll" that dynamically redirects to the relevant "python3y.dll" to allow an extension module to run against any 3.2 or later Python version as a goal of the PEP, I would say that allowing mixing of C runtimes is definitely one of the PEP's goals. It should be explicit about that then, and provide detail about why the runtime is relevant to Windows programmers (and probably not relevant in practice for *nix programmers).

An extension compiled for one version of Python will be linked against the version of the C runtime used by that version of Python (if it is compiled with the same version of Visual Studio of course).

If the extension binary is to remain compatible with a later version of Python, compiled against a different version of the C runtime, then it *must* be possible for multiple C runtimes to be loaded.

Is it possible to have multiple versions of the lib C loaded on Windows ? I know that it is not possible on Linux. Instead, the glibc takes great care to version all sorts of APIs in the lib to make it possible to run applications using different versions within the same runtime (they include different ABI versions in the same lib). AFAIK, on Windows, the lib C is using a different approach: up until the invention of the manifests, they took great care not to break the APIs in incompatible ways. With manifests, things are more complicated, since the DLLs now explicitly reference a particular version of a DLL (down to the patch level) and if the versions are not installed, the application won't run. Not sure what effect that has on the way they engineered the new lib C versions... This document suggests that it is possible to have an application use multiple CRTs, but care has to be taken with respect to things that are initialized by the CRTs (env vars, locales, handles, etc.): http://msdn.microsoft.com/en-us/library/abx4dbyh(v=VS.90).aspx

If the stable ABI doesn't allow this then binaries will *still* have to be recompiled when we update the version of Visual Studio used to compile Python - defeating the purpose of the PEP. Right?

If we keep changing the main compiler version used for creating Python binaries on Windows: yes. Fortunately, we don't :-) On Unix this shouldn't be too much of a problem, though, so the PEP is still valid for those platforms.

If this is the case then I agree that it should be explicit in the PEP.

-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 27 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 18 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Neil Hodgson wrote:

M.-A. Lemburg:

...

Is it possible to have multiple versions of the lib C loaded on Windows ?

Yes. It is possible not only to mix C runtimes from different vendors but different variants from a single vendor.

Historically, each vendor has shipped their own C runtime libraries. This was also the case with CP/M and OS/2.

Many applications can be extended with DLLs and if it were not possible to load DLLs which use different runtimes then that would limit which compilers can be used to extend particular applications. If Microsoft were to stop DLLs compiled with Borland or Intel from working inside Internet Explorer or Excel then there would be considerable controversy and likely anti-trust actions.

Thanks for the feedback. This sounds like the issues such a mix can cause are mostly theoretical and don't really bother much in practice, so PEP 384 on Windows does have a chance :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 27 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 18 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

"Martin v. Löwis" wrote:

...

This sounds like the issues such a mix can cause are mostly theoretical and don't really bother much in practice, so PEP 384 on Windows does have a chance :-)

Actually, the CRT issues (FILE* in particular) have been causing real crashes for many years, for many people.

Do you have some pointers ? I don't remember this being a real practical issue, at least not for different versions of the MS CRTs. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 07 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 7 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Wed, Sep 8, 2010 at 8:34 AM, David Cournapeau wrote:

I would turn the question around: what are the cases where you manage to mix CRT and not getting any issues ? This has never worked in my own experience in the context of python extensions,

I've done it quite a bit over the years with SWIG-wrapped libraries where all I did was recompile my extension for new versions of Python. It works fine, so long as you avoid the very APIs that PEP 384 is blocking from the stable ABI. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Wed, Sep 8, 2010 at 6:34 PM, David Cournapeau wrote:

In other words, the problem mainly arises when you need to integrate libraries which you can not recompile with the compiler used by python, because the code is not visual-studio compatible, or because the library is only available in binary form.

In my case, I was building on an old dev system which only has VC6, but needed to link against Python 2.4 (which was compiled with MSVC 2005). The build process didn't use distutils, so that didn't affect anything. It works, you just have to know what APIs you have to avoid. The C runtime independence aspects of PEP 384 are just about enlisting the compiler in the task of making sure you *have* avoided all those APIs. The rest of the PEP is about hiding the details of some of Python's own data structures to allow them to change without affecting the stable ABI. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

Am 07.09.2010 19:48, schrieb M.-A. Lemburg:

"Martin v. Löwis" wrote:

...

...

This sounds like the issues such a mix can cause are mostly theoretical and don't really bother much in practice, so PEP 384 on Windows does have a chance :-)

Actually, the CRT issues (FILE* in particular) have been causing real crashes for many years, for many people.

Do you have some pointers ?

You mean, other than FILE* :-? Unfortunately, I don't have time to search for reports, but they had been several on comp.lang.python over the years. As others say, this is *really* easy to reproduce.

I don't remember this being a real practical issue, at least not for different versions of the MS CRTs.

You probably had not been passing FILE* across CRT boundaries, then. Notice that this is an uncommon thing to do, except that the Python API has some functions that expect FILE*. As others have mentioned, CRT file handles are a problem, too, but less so: a) they aren't explicitly passed in any of the Python APIs (except for the obvious os.* functions), b) they don't cause crashes, just outputs to the wrong files, and c) if they are 0/1/2, they actually do the right thing. Regards, Martin

Daniel Stutzbach wrote:

On Tue, Aug 31, 2010 at 4:54 AM, M.-A. Lemburg wrote:

...

Is it possible to have multiple versions of the lib C loaded on Windows ?

Yes, and it's a pretty common situation. The fopen() that I call within a DLL may not be the same fopen() called by another DLL. When writing a DLL for Windows, the API must be designed with the assumption that anything returned by the C library cannot be passed a different C library. For example, suppose I a expose a function in my DLL that allocates some memory, populates it with useful information, and returns a pointer. I must also supply a function to free the memory. I cannot ask the caller to simply call free(), because their free() may not be using the same heap as my malloc().

But isn't exactly that a major problem for Python ? An extension written for a different MS CRT version would not be able to safely free a buffer allocated by the Python DLL.

Likewise, a FILE * isn't safe to pass around, unless I can guarantee that the application really is one big happy family compiled against the same version of the C library.

According to the MS document I found this also applies to the OS environment and handles. Taking all these things together makes it sound like there's a rather small chance of actually getting PEP 384 working across Windows compiler upgrades. Then again, I haven't heard of many people actually running into those possible issues. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 27 2010)

...

...

Python/Zope Consulting and Support ... http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2010-08-19: Released mxODBC 3.1.0 http://python.egenix.com/ 2010-09-15: DZUG Tagung, Dresden, Germany 18 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

M.-A. Lemburg wrote:

But isn't exactly that a major problem for Python ?

An extension written for a different MS CRT version would not be able to safely free a buffer allocated by the Python DLL.

Python provides its own set of memory alloc/free functions to deal with this exact problem. For every Python function that allocates something, there's a corresponding function for freeing it, and you do it any other way at your peril. -- Greg

On Wed, 01 Sep 2010 10:23:42 +1200 Greg Ewing wrote:

Daniel Stutzbach wrote:

...

Likewise, a FILE * isn't safe to pass around, unless I can guarantee that the application really is one big happy family compiled against the same version of the C library.

Given that, it seems to me that it's a mistake for Python to provide any APIs that take a FILE* directly in the first place.

What I think would be a mistake would be to define the API in terms of Windows-specific quirks. All this discussion seems bent on satisfying the needs of Windows developers at the expense of non-Windows developers. "FILE*" is a perfectly standard C feature (and a widely-used one at that). If Windows doesn't handle it correctly then it's a Windows issue and we shouldn't annoy other people by refusing access to that feature. Again, putting a warning in the documentation should be enough for those people who want to have perfect Windows support. And if the Microsoft compiler/linker doesn't bother warning the user when passing FILE* between various CRTs, then it's not Python's job to supplement it. After all, we don't usually try to workaround platform-specific bugs (not as a low level such as the C API level); at worse, we mention their existence in the docs. Regards Antoine.

On Wed, Sep 1, 2010 at 8:42 AM, Antoine Pitrou wrote:

After all, we don't usually try to workaround platform-specific bugs (not as a low level such as the C API level); at worse, we mention their existence in the docs.

You persist in viewing the allowance of multiple C runtimes in a single process as a bug instead of a feature. This comes from the monolithic approach to software management in Unix systems (with the associated web of dependencies and implications for backwards compatibility management). The Windows heritage is different, allowing each application to be largely independent of everything other than the OS itself, so the execution model is also different. Windows is a supported platform for Python and to meet the promise of a stable ABI, we need to support multiple C runtimes in a single process. Since we aren't in the habit of passing low level objects around, the number of APIs affected is relatively tiny (the current tally is less than half a dozen that I know of - the two mentioned in this thread, plus a couple of of the PyRun_* APIs). There will be some cases (like file descriptors) where we just say "don't do that" but those should be fairly rare. Cheers, Nick. -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On 01/09/2010 13:54, Antoine Pitrou wrote:

[snip...] Please consider this: even without relying on PEP 384, using FILE* is /already/ dangerous; because you might compile an extension with a different compiler version than Python was compiled with. So, if we were following you, we should rip out PyObject_Print() of the whole C API, not only the limited subset which is defined by PEP 384.

Definitely. On Windows it is relatively common to configure distutils to compile extensions with Mingw, which is likely to have problems with FILE* from the Microsoft C runtime [1]. Michael [1] I say *likely* because I have vague memories of Mingw being able to compile against msvcrt, but I'm not sure about this. This is actually less of an issue at the moment as we are now using a Microsoft compiler for which a free version is available. This wasn't the case for a while (and still isn't for those using Python 2.5 and earlier).

(now I have nothing against completely ripping out PyObject_Print() if we find out that it's not really useful...)

Regards

Antoine.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...

-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Thu, 2 Sep 2010 07:04:31 +1000 Nick Coghlan wrote:

On Wed, Sep 1, 2010 at 10:54 PM, Antoine Pitrou wrote:

...

Please consider this: even without relying on PEP 384, using FILE* is /already/ dangerous; because you might compile an extension with a different compiler version than Python was compiled with. So, if we were following you, we should rip out PyObject_Print() of the whole C API, not only the limited subset which is defined by PEP 384.

(now I have nothing against completely ripping out PyObject_Print() if we find out that it's not really useful...)

I think it would be better if everything dealing with FILE* was a macro rather than a function, yes. The definition of the limited API is a chance to fix that without incurring the cost in backwards incompatibility that would otherwise arise. Since we have that opportunity, why not take it?

Maybe I've missed your answer, but what would prevent the "inline" solution from working? (a macro with the result-as-a-pointer is quite ugly) Regards Antoine.

On 02/09/10 09:04, Nick Coghlan wrote:

I think it would be better if everything dealing with FILE* was a macro rather than a function, yes.

How would that help? -- Greg

On Wed, Sep 1, 2010 at 10:54 PM, Antoine Pitrou wrote:

Please consider this: even without relying on PEP 384, using FILE* is /already/ dangerous; because you might compile an extension with a different compiler version than Python was compiled with. So, if we were following you, we should rip out PyObject_Print() of the whole C API, not only the limited subset which is defined by PEP 384.

(now I have nothing against completely ripping out PyObject_Print() if we find out that it's not really useful...)

I finally took the obvious step of grepping the include directory to see what APIs were even affected by this question. Turns out there are more APIs than I thought, but most extension modules aren't going to need most of them (since they're related to code parsing, compilation and execution directly from files). For the remainder, I propose as a starting point that users of the stable ABI be directed to the Python equivalents, with the possibility of later adding inline functions for more commonly used API elements. PyObject_Print is the only one I would bother providing as part of the initial stable ABI implementation. Cheers, Nick. Search criteria: grep "FILE \?\*" * object.h:typedef int (*printfunc)(PyObject *, FILE *, int); object.h:PyAPI_FUNC(int) PyObject_Print(PyObject *, FILE *, int); - should provide inline equivalent fileobject.h:PyAPI_FUNC(char *) Py_UniversalNewlineFgets(char *, int, FILE*, PyObject *); marshal.h:PyAPI_FUNC(void) PyMarshal_WriteLongToFile(long, FILE *, int); marshal.h:PyAPI_FUNC(void) PyMarshal_WriteObjectToFile(PyObject *, FILE *, int); marshal.h:PyAPI_FUNC(long) PyMarshal_ReadLongFromFile(FILE *); marshal.h:PyAPI_FUNC(int) PyMarshal_ReadShortFromFile(FILE *); marshal.h:PyAPI_FUNC(PyObject *) PyMarshal_ReadObjectFromFile(FILE *); marshal.h:PyAPI_FUNC(PyObject *) PyMarshal_ReadLastObjectFromFile(FILE *); parsetok.h:PyAPI_FUNC(node *) PyParser_ParseFile (FILE *, const char *, grammar *, int, parsetok.h:PyAPI_FUNC(node *) PyParser_ParseFileFlags(FILE *, const char *, parsetok.h:PyAPI_FUNC(node *) PyParser_ParseFileFlagsEx(FILE *, const char *, pythonrun.h:PyAPI_FUNC(int) PyRun_AnyFileFlags(FILE *, const char *, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(int) PyRun_AnyFileExFlags(FILE *, const char *, int, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(int) PyRun_SimpleFileExFlags(FILE *, const char *, int, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(int) PyRun_InteractiveOneFlags(FILE *, const char *, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(int) PyRun_InteractiveLoopFlags(FILE *, const char *, PyCompilerFlags *); pythonrun.h:PyAPI_FUNC(struct _mod *) PyParser_ASTFromFile(FILE *, const char *, pythonrun.h:PyAPI_FUNC(struct _node *) PyParser_SimpleParseFileFlags(FILE *, const char *, pythonrun.h:PyAPI_FUNC(PyObject *) PyRun_FileExFlags(FILE *, const char *, int, pythonrun.h:PyAPI_FUNC(int) Py_FdIsInteractive(FILE *, const char *); pythonrun.h:PyAPI_FUNC(char *) PyOS_Readline(FILE *, FILE *, char *); pythonrun.h:PyAPI_DATA(char) *(*PyOS_ReadlineFunctionPointer)(FILE *, FILE *, char *); - omit from stable ABI, recommend equivalent Python calls (may add inline versions to stable ABI over time if requested by developers) abstract.h: int PyObject_Print(PyObject *o, FILE *fp, int flags); - comment, not declaration (actual declaration is in object.h) grammar.h:void printgrammar(grammar *g, FILE *fp); grammar.h:void printnonterminals(grammar *g, FILE *fp); object.h:PyAPI_FUNC(void) _Py_PrintReferences(FILE *); object.h:PyAPI_FUNC(void) _Py_PrintReferenceAddresses(FILE *); Python.h:PyAPI_FUNC(FILE *) _Py_wfopen(const wchar_t *path, const wchar_t *mode); Python.h:PyAPI_FUNC(FILE*) _Py_fopen(PyObject *unicode, const char *mode); - private, omit from stable ABI -- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

On Sat, 04 Sep 2010 15:13:55 +0200 "Martin v. Löwis" wrote:

...

Please consider this: even without relying on PEP 384, using FILE* is /already/ dangerous; because you might compile an extension with a different compiler version than Python was compiled with.

It's only dangerous *if* you compile with a different compiler. That's why we take serious precautions that you never ever do. For example, distutils will, by default, refuse to use a different compiler.

Thanks for the explanations. Regards Antoine.

On Aug 30, 2010, at 07:31 AM, Nick Coghlan wrote:

Since part of the point of PEP 384 is to support multiple versions of the C runtime in a single process

Is it? That's certainly not explicit in the Rationale section of PEP 384. It seems to me that the PEP is all about inoculating extension builders from changes in Python's ABI across Python versions. That's a worthy goal. I'm not sure the same can be said about allowing multiple versions of the C runtime in a linked process. But then I'm developing on Windows these days, and I think for all practical purposes, it's a moot point on *nix. -Barry

On Sun, 12 Sep 2010 19:38:33 +0200 "Martin v. Löwis" wrote:

...

On http://bugs.python.org/issue9778 you elaborated on what the PEP would entail in its current state:

“No, vice versa. The PEP promises that the ABI won't change until Python 4. For any change that might break the ABI, either a backwards-compatible solution needs to be found, or the change be deferred to Python 4.”

This sounds like it could be detrimental by blocking desired improvements (the aforementioned issue is a potential example of this).

Notice that it's only potential: in the specific case, there would be an ABI-compatible way of introducing wide hashes, using a second type slot.

Yes, but it would add complication, and be potentially detrimental to performance.

If you think this is too restrictive, please point out specific aspects that you think might need to change in the mid-term future. They should then be excluded from the ABI.

I have no a priori knowledge of what might happen in the future :) That said, looking at the PEP, I was wondering whether fields such as ob_type, ob_refcnt, ob_size have to be directly accessible, rather than through a macro-turned-into-a-function such as Py_REFCNT().