What do PyAPI_FUNC & PyAPI_DATA mean?
Many (most?) of the function declarations in the CPython header files are annotated with the PyAPI_FUNC declaration. Similarly for data declarations and PyAPI_DATA What do they mean, exactly? From the name I would expect that they are a way of declaring a function or datum to be part of the API, but their usage seems to be more to do with linkage. The reason I am asking is that the new dictionary implementation declares a few functions used to communicate between dictobject.c, typeobject.c and ceval.c which cannot be static functions, but are not intended to be part of the API. Cheers, Mark.
IMHO, we are _much_ too generous at applying this to almost whatever gets exposed between .c files. I have created something called the "restricted" api for our custom python27.dll where I use different macros (PyAPI_RFUNC, pyAPI_RDATA) to mean that things aren't exported for "restricted" builds. We use it to remove some of the easier access points to the dll for hackers to exploit. Also, once declared exported this way, things become more bothersome to remove again, since once could always argue that someone out there is using these thigns. K
Am 23.04.2012 15:05, schrieb Kristján Valur Jónsson:
For this, PyAPI_FUNC doesn't really matter. A symbol that is listed in the header file is available on Unix even without such a declaration, so listing it in the public header file is already the step that makes it public, not specifying it as PyAPI_FUNC. I agree that too much API is public, but the right approach is to rename such API to _Py*, indicating to users that we don't want them to use it. For existing API, that's tricky; for new API, I think it should be private by default. See also PEP 384. Regards, Martin
You know that I'm speaking of Windows, right? IMHO, we shouldn't put the PyAPI* stuff on functions unless they are actual API functions. I don't know how the export tables for ELF .so objects is generated, but it surely can't export _everything_. Anyway, marking stuff as part of the API makes sense, and marking functions as being part of the API makes no sense and is wasteful when they are not. We might even have something similar for the stable api.
Quoting Kristján Valur Jónsson <kristjan@ccpgames.com>:
You know that I'm speaking of Windows, right?
Yes, but this may only be valid for CCP; for CPython, we certainly have to consider Unix as well.
It certainly does. Any global symbol in an ELF shared library gets exported. There are recent (10 years or so) ways to restrict this, by declaring symbols as hidden in the object file, but exporting everything is the default that Python uses.
There are cases where it's necessary: when an extension module uses a function that is not in the API.
We might even have something similar for the stable api.
I don't understand. Everything in the stable api is part of the API, and thus needs to be exported from the Python DLL. Regards, Martin
It means that they will be exported from the pythonXY.dll on Windows. In Windows DLLs, it's not sufficient to make a symbol global (non-static) to use it in an application, you also have to declare it as __declspec(dllexport), or list it in the linker definition file (which is not used today anymore). Likewise, to use a symbol from a DLL, you also need to declare it as __declspec(dllimport) in the using application. This will, in particular, arrange for a slot in the indirect-jump assembler section of the using DLL, so that the resulting executable will be position- independent (except for this procedure linkage section). As we have the same header files both for the implemenation and the usage, this macro tricky is necessary to sometimes say dllexport, sometimes dllimport. Even though it's strictly needed on Windows, and strictly only for API that we do want to expose, we apply it to all API that is public on Unix (i.e. all Py* API), in order to avoid API being available on Unix but not on Windows. Regards, Martin
Aha, so that is the rationale. Because the export table on unix is so generous, we force ourselves to be generous on windows too? I did some unix programming back in the day. IRIX, actually (a Sys V derivative). I'm pretty sure we had to explicitly specify our .so exports. But I might be mistaken. K
Quoting Kristján Valur Jónsson <kristjan@ccpgames.com>:
Aha, so that is the rationale. Because the export table on unix is so generous, we force ourselves to be generous on windows too?
Yes. If the code compiles and links on Unix, it shall also compile and link on Windows.
Maybe on IRIX, probably in a way that predates ELF. In the old days, on Linux, you had to globally request address space from Linus Torvalds for shared libraries. These days are long gone. ELF shared libraries are designed to give the same experience (roughly) as static libraries, wrt. source compatibility. Regards, Martin
IMHO, we are _much_ too generous at applying this to almost whatever gets exposed between .c files. I have created something called the "restricted" api for our custom python27.dll where I use different macros (PyAPI_RFUNC, pyAPI_RDATA) to mean that things aren't exported for "restricted" builds. We use it to remove some of the easier access points to the dll for hackers to exploit. Also, once declared exported this way, things become more bothersome to remove again, since once could always argue that someone out there is using these thigns. K
Am 23.04.2012 15:05, schrieb Kristján Valur Jónsson:
For this, PyAPI_FUNC doesn't really matter. A symbol that is listed in the header file is available on Unix even without such a declaration, so listing it in the public header file is already the step that makes it public, not specifying it as PyAPI_FUNC. I agree that too much API is public, but the right approach is to rename such API to _Py*, indicating to users that we don't want them to use it. For existing API, that's tricky; for new API, I think it should be private by default. See also PEP 384. Regards, Martin
You know that I'm speaking of Windows, right? IMHO, we shouldn't put the PyAPI* stuff on functions unless they are actual API functions. I don't know how the export tables for ELF .so objects is generated, but it surely can't export _everything_. Anyway, marking stuff as part of the API makes sense, and marking functions as being part of the API makes no sense and is wasteful when they are not. We might even have something similar for the stable api.
Quoting Kristján Valur Jónsson <kristjan@ccpgames.com>:
You know that I'm speaking of Windows, right?
Yes, but this may only be valid for CCP; for CPython, we certainly have to consider Unix as well.
It certainly does. Any global symbol in an ELF shared library gets exported. There are recent (10 years or so) ways to restrict this, by declaring symbols as hidden in the object file, but exporting everything is the default that Python uses.
There are cases where it's necessary: when an extension module uses a function that is not in the API.
We might even have something similar for the stable api.
I don't understand. Everything in the stable api is part of the API, and thus needs to be exported from the Python DLL. Regards, Martin
It means that they will be exported from the pythonXY.dll on Windows. In Windows DLLs, it's not sufficient to make a symbol global (non-static) to use it in an application, you also have to declare it as __declspec(dllexport), or list it in the linker definition file (which is not used today anymore). Likewise, to use a symbol from a DLL, you also need to declare it as __declspec(dllimport) in the using application. This will, in particular, arrange for a slot in the indirect-jump assembler section of the using DLL, so that the resulting executable will be position- independent (except for this procedure linkage section). As we have the same header files both for the implemenation and the usage, this macro tricky is necessary to sometimes say dllexport, sometimes dllimport. Even though it's strictly needed on Windows, and strictly only for API that we do want to expose, we apply it to all API that is public on Unix (i.e. all Py* API), in order to avoid API being available on Unix but not on Windows. Regards, Martin
Aha, so that is the rationale. Because the export table on unix is so generous, we force ourselves to be generous on windows too? I did some unix programming back in the day. IRIX, actually (a Sys V derivative). I'm pretty sure we had to explicitly specify our .so exports. But I might be mistaken. K
Quoting Kristján Valur Jónsson <kristjan@ccpgames.com>:
Aha, so that is the rationale. Because the export table on unix is so generous, we force ourselves to be generous on windows too?
Yes. If the code compiles and links on Unix, it shall also compile and link on Windows.
Maybe on IRIX, probably in a way that predates ELF. In the old days, on Linux, you had to globally request address space from Linus Torvalds for shared libraries. These days are long gone. ELF shared libraries are designed to give the same experience (roughly) as static libraries, wrt. source compatibility. Regards, Martin
participants (5)
-
"Martin v. Löwis" -
Benjamin Peterson -
Kristján Valur Jónsson -
Mark Shannon -
martin＠v.loewis.de