Re: Leading uderscore in C-API names (`_Py`): Private, or just a warning?
On 14.06.2022 11:54, Petr Viktorin wrote:
On 14. 06. 22 11:35, Marc-Andre Lemburg wrote:
On 14.06.2022 11:15, Petr Viktorin wrote:
On 13. 06. 22 17:36, Marc-Andre Lemburg wrote:
In the past we always said: "_Py* is an internal API. Use at your own risk.", which I guess is somewhere between the warning and the strict "don't use" policy you are describing.
The problem with the "don't use" policy is that in some cases, there are no public APIs available to do certain things and so the extension writers have to resort to the private ones to implement their logic.
E.g. to implement a free list for Python objects, you have to use _Py_NewReference() in order to create an object based on a memory area taken from the free list. If you want to create a bytes objects using overallocation, it's common to use _PyBytes_Resize() to resize the buffer to the final size.
What sometimes happens is that after a while the private APIs get their leading underscore removed to then become public ones.
It's not just about removing the underscore: when this happens the APIs should also get documentation, tests, and some expectation of stability (e.g. that we won't go randomly adding tstate parameters to them).
Of course; all public APIs should ideally have this :-)
This upwards migration path would be made impossible with the "don't use" policy.
Why not? I have no doubt people will use private API, no matter how explicitly we say that it can break at any time.
My proposal is making this more explicit. And yes, it's also putting some more pressure on core devs to expose proper API for use cases people have, and on people to report their use cases.
It would certainly be good to get more awareness for common uses of currently private APIs, but I'm not sure whether the proposed "don't use" policy would help with this.
I have a feeling that the effect would go in a different direction: with a strict "don't use" policy core devs would get a blanket permission to change exposed _Py* APIs at will, without any consideration about breaking possible use cases out there.
IMO, both parties should be aware of the issues around using/changing exposed APIs marked as private and ideally to the same extent.
OK, but what are those issues? How do we get to the point where both parties agree? We don't even know who the other party is.
Perhaps it would be better to leave the current "use at your own risk" approach in place and just add a new process for potentially having private APIs made public.
But if we say "use at your own risk", what's the risk? Isn't that the same as saying we don't support it at all and it might go away at any time?
Not really, since the bar for core devs is higher when making changes to such internal APIs. A blanket "don't use" policy would make it easy for core devs to argue that any change is acceptable.
If we provide a process to document external use of currently private APIs, both sides get to know each other and their use cases.
The "risk" for the external developer is that the APIs may not get approved as public APIs.
The process itself allows core devs to get a view into the use of those APIs and take more care when making changes to those APIs while the process of potentially making them public is still ongoing.
Overall, we'd get a better idea on what extension writers need, while at the same it'd be clearer which of the private APIs can indeed be changed without causing disruption.
If we say: please open a ticket when starting to use a private API, I think we'd get to a better C API in the long run.
E.g. the above two cases are potentially candidates for such a process. I have used both in code I have written, because, AFAIK, there's no other way to implement the functionality otherwise.
I'm pretty sure that fairly low level tools such as Cython will have similar cases.
Indeed. Documenting and testing _Py_NewReference and its caveats sounds like a good idea, and we might as well remove the underscore then.
-- Marc-Andre Lemburg eGenix.com
Professional Python Services directly from the Experts (#1, Jun 14 2022)
Python Projects, Coaching and Support ... https://www.egenix.com/ Python Product Development ... https://consulting.egenix.com/
::: We implement business ideas - efficiently in both time and costs :::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/
On 14. 06. 22 12:18, Marc-Andre Lemburg wrote:
On 14.06.2022 11:54, Petr Viktorin wrote:
On 14. 06. 22 11:35, Marc-Andre Lemburg wrote:
On 14.06.2022 11:15, Petr Viktorin wrote:
On 13. 06. 22 17:36, Marc-Andre Lemburg wrote:
In the past we always said: "_Py* is an internal API. Use at your own risk.", which I guess is somewhere between the warning and the strict "don't use" policy you are describing.
The problem with the "don't use" policy is that in some cases, there are no public APIs available to do certain things and so the extension writers have to resort to the private ones to implement their logic.
E.g. to implement a free list for Python objects, you have to use _Py_NewReference() in order to create an object based on a memory area taken from the free list. If you want to create a bytes objects using overallocation, it's common to use _PyBytes_Resize() to resize the buffer to the final size.
What sometimes happens is that after a while the private APIs get their leading underscore removed to then become public ones.
It's not just about removing the underscore: when this happens the APIs should also get documentation, tests, and some expectation of stability (e.g. that we won't go randomly adding tstate parameters to them).
Of course; all public APIs should ideally have this :-)
This upwards migration path would be made impossible with the "don't use" policy.
Why not? I have no doubt people will use private API, no matter how explicitly we say that it can break at any time.
My proposal is making this more explicit. And yes, it's also putting some more pressure on core devs to expose proper API for use cases people have, and on people to report their use cases.
It would certainly be good to get more awareness for common uses of currently private APIs, but I'm not sure whether the proposed "don't use" policy would help with this.
I have a feeling that the effect would go in a different direction: with a strict "don't use" policy core devs would get a blanket permission to change exposed _Py* APIs at will, without any consideration about breaking possible use cases out there.
IMO, both parties should be aware of the issues around using/changing exposed APIs marked as private and ideally to the same extent.
OK, but what are those issues? How do we get to the point where both parties agree? We don't even know who the other party is.
Perhaps it would be better to leave the current "use at your own risk" approach in place and just add a new process for potentially having private APIs made public.
But if we say "use at your own risk", what's the risk? Isn't that the same as saying we don't support it at all and it might go away at any time?
Not really, since the bar for core devs is higher when making changes to such internal APIs. A blanket "don't use" policy would make it easy for core devs to argue that any change is acceptable.
And that's why my proposal explicitly says: "We won't break this internal API for no reason, so if you use it now and have good tests you don't need stop using it right now."
If we provide a process to document external use of currently private APIs, both sides get to know each other and their use cases.
The "risk" for the external developer is that the APIs may not get approved as public APIs.
The process itself allows core devs to get a view into the use of those APIs and take more care when making changes to those APIs while the process of potentially making them public is still ongoing.
Overall, we'd get a better idea on what extension writers need, while at the same it'd be clearer which of the private APIs can indeed be changed without causing disruption.
If we say: please open a ticket when starting to use a private API, I think we'd get to a better C API in the long run.
Yes, that would be great! You'll find exactly that in my proposal: "If you're using private API and don't see a public alternative, you should contact CPython devs".
What do you think should happen if the user *doesn't* open the ticket?
E.g. the above two cases are potentially candidates for such a process. I have used both in code I have written, because, AFAIK, there's no other way to implement the functionality otherwise.
I'm pretty sure that fairly low level tools such as Cython will have similar cases.
Indeed. Documenting and testing _Py_NewReference and its caveats sounds like a good idea, and we might as well remove the underscore then.
participants (2)
-
Marc-Andre Lemburg -
Petr Viktorin