[Distutils]Re: The Wheel specification and compatibility tags on Windows
I would like to revive this 5 year old thread and see if we can stir things up a bit. Basically the problem is that, in the current state of the PEPs and build tools, it is still not possible to build and distribute a Windows wheel that includes an extension module compiled with Py_LIMITED_API. Setuptools can successfully build such extension module on Windows (with `.pyd` file extension and no extra specifiers in the module filename), and these seems to work at least on CPython 3.5 and above. However the `--py-limited-api cpXX` option of bdist_wheel command fails on Windows because it attempts to use the `abi3` tag but the latter is not in the list of compatible tags for that platform. One can work around this by creating a wheel with `none` as the abi tag, and `cp35.cp36.cp37` as the python implementation tag but this feels a bit hackish. Here are some unresolved questions from the old distutils-sig thread, quoting Paul Moore:
2. How should tools determine which ABIs a given Python supports? (This is the get_supported function in wheel and distlib). The "Use" section of the PEP (http://www.python.org/dev/peps/pep-0425/#id1) gives a Linux-based example, but nothing normative and nothing that is understandable to a Windows user.
And from Vinay Sajip
For Windows, we (eventually) need some low-level sysconfig-supported way to get the ABI information in an analogous way to how it happens on POSIX: and because that's not currently there, distlib doesn't provide any ABI information on Windows other than "none".
Other related links: https://github.com/pypa/pip/issues/4445 https://mail.python.org/pipermail/distutils-sig/2018-January/031856.html So.. what needs to be done here to allow distributing/installing Windows wheels with Py_LIMITED_API support? Thank you, Cosimo Lupo -- Cosimo Lupo
On 17 July 2018 at 16:59, Cosimo Lupo <cosimo@anthrotype.com> wrote:
I would like to revive this 5 year old thread and see if we can stir things up a bit.
Basically the problem is that, in the current state of the PEPs and build tools, it is still not possible to build and distribute a Windows wheel that includes an extension module compiled with Py_LIMITED_API. Setuptools can successfully build such extension module on Windows (with `.pyd` file extension and no extra specifiers in the module filename), and these seems to work at least on CPython 3.5 and above. However the `--py-limited-api cpXX` option of bdist_wheel command fails on Windows because it attempts to use the `abi3` tag but the latter is not in the list of compatible tags for that platform. One can work around this by creating a wheel with `none` as the abi tag, and `cp35.cp36.cp37` as the python implementation tag but this feels a bit hackish.
Here are some unresolved questions from the old distutils-sig thread, quoting Paul Moore:
2. How should tools determine which ABIs a given Python supports? (This is the get_supported function in wheel and distlib). The "Use" section of the PEP (http://www.python.org/dev/peps/pep-0425/#id1) gives a Linux-based example, but nothing normative and nothing that is understandable to a Windows user.
And from Vinay Sajip
For Windows, we (eventually) need some low-level sysconfig-supported way to get the ABI information in an analogous way to how it happens on POSIX: and because that's not currently there, distlib doesn't provide any ABI information on Windows other than "none".
Other related links: https://github.com/pypa/pip/issues/4445 https://mail.python.org/pipermail/distutils-sig/2018-January/031856.html
So.. what needs to be done here to allow distributing/installing Windows wheels with Py_LIMITED_API support?
IMO, the question I posed back then remains key. Vinay's response is fair, but I don't think that waiting for core Python to provide something via sysconfig is practical (it's not happened yet, so why would we expect things to change?). So I think the next step is probably for someone to propose an algorithm that can be used. Actually, what I'd like to see is a full end to end proposal of the process someone would use to build and install a limited-ABI extension. That would probably tease out a number of issues. I imagine the steps would be something like this: 1. Create an extension. Presumably you'd need to #define PY_LIMITED_ABI in the source. 2. Build a wheel from it - how would you do that? I gather it's possible to do this with plain setuptools - would it be necessary to do this with setuptools/bdist_wheel, or should there be a way to request a limited ABI build via pip? If we do want to be able to request this from a build tools like pip, do we need something in PEP 517? Are we only looking at the prebuilt wheel case, or do we need to support building from source? 3. What tags would the built wheel have? 4. Install that wheel - `pip install xxx`. Pip needs to be able to enumerate the full list of valid tags here (cp37-abi3, cp3-abi3, ...) There are also questions like - if there's a limited ABI wheel and a full ABI (version specific) wheel, which takes precedence? I don't honestly know how well the limited ABI actually achieves its goals - is "cp3-abi3-win_x86_64" a realistic tag to apply? Can limited ABI wheels really be used on any version of Python 3? That's a question for python-dev, rather than distutils-sig, but if we take the position that this is what's promised, and it later turns out not to be true, we've got a lot of wheel renaming to do when Python 3.10 comes out and it doesn't support the same limited ABI as 3.x for x < 10... Also, does the limited ABI work on other platforms? If it does, we should ensure that the Windows support works the same. If it doesn't, do we want a Windows-only solution (why is that OK?) or should we extend to (say) manylinux or OSX (at the risk of making the job too big for anyone to actually get anywhere with it). So to move this forward, I think someone needs to write up the process of building and using a limited ABI wheel, as described above, and document suggested answers to the various questions that will come out in the process of going through the details. Is that something you'd be willing to take on? From that, we'd have something concrete to debate. I'm not sure how many people have an interest in this topic, so getting people with the necessary knowledge to chime in might take some work (I'm interested, but I don't have detail understanding of build options and linking conventions). The ultimate goal would be some sort of PEP covering handling limited ABI extensions within the packaging infrastructure. Does that seem reasonable? Is that the sort of guidance you were looking for? I doubt anything is going to happen on this subject until someone with the interest in moving it forward steps up to do the work of making a proposal and collecting community views. Paul
On 17 Jul 2018, at 5:37 pm, Paul Moore <p.f.moore@gmail.com> wrote:
On 17 July 2018 at 16:59, Cosimo Lupo <cosimo@anthrotype.com> wrote:
I would like to revive this 5 year old thread and see if we can stir things up a bit.
Basically the problem is that, in the current state of the PEPs and build tools, it is still not possible to build and distribute a Windows wheel that includes an extension module compiled with Py_LIMITED_API. Setuptools can successfully build such extension module on Windows (with `.pyd` file extension and no extra specifiers in the module filename), and these seems to work at least on CPython 3.5 and above. However the `--py-limited-api cpXX` option of bdist_wheel command fails on Windows because it attempts to use the `abi3` tag but the latter is not in the list of compatible tags for that platform. One can work around this by creating a wheel with `none` as the abi tag, and `cp35.cp36.cp37` as the python implementation tag but this feels a bit hackish.
Here are some unresolved questions from the old distutils-sig thread, quoting Paul Moore:
2. How should tools determine which ABIs a given Python supports? (This is the get_supported function in wheel and distlib). The "Use" section of the PEP (http://www.python.org/dev/peps/pep-0425/#id1) gives a Linux-based example, but nothing normative and nothing that is understandable to a Windows user.
And from Vinay Sajip
For Windows, we (eventually) need some low-level sysconfig-supported way to get the ABI information in an analogous way to how it happens on POSIX: and because that's not currently there, distlib doesn't provide any ABI information on Windows other than "none".
Other related links: https://github.com/pypa/pip/issues/4445 https://mail.python.org/pipermail/distutils-sig/2018-January/031856.html
So.. what needs to be done here to allow distributing/installing Windows wheels with Py_LIMITED_API support?
IMO, the question I posed back then remains key. Vinay's response is fair, but I don't think that waiting for core Python to provide something via sysconfig is practical (it's not happened yet, so why would we expect things to change?). So I think the next step is probably for someone to propose an algorithm that can be used. Actually, what I'd like to see is a full end to end proposal of the process someone would use to build and install a limited-ABI extension. That would probably tease out a number of issues.
I imagine the steps would be something like this:
1. Create an extension. Presumably you'd need to #define PY_LIMITED_ABI in the source. 2. Build a wheel from it - how would you do that? I gather it's possible to do this with plain setuptools - would it be necessary to do this with setuptools/bdist_wheel, or should there be a way to request a limited ABI build via pip? If we do want to be able to request this from a build tools like pip, do we need something in PEP 517? Are we only looking at the prebuilt wheel case, or do we need to support building from source? 3. What tags would the built wheel have? 4. Install that wheel - `pip install xxx`. Pip needs to be able to enumerate the full list of valid tags here (cp37-abi3, cp3-abi3, ...) There are also questions like - if there's a limited ABI wheel and a full ABI (version specific) wheel, which takes precedence?
I don't honestly know how well the limited ABI actually achieves its goals - is "cp3-abi3-win_x86_64" a realistic tag to apply? Can limited ABI wheels really be used on any version of Python 3? That's a question for python-dev, rather than distutils-sig, but if we take the position that this is what's promised, and it later turns out not to be true, we've got a lot of wheel renaming to do when Python 3.10 comes out and it doesn't support the same limited ABI as 3.x for x < 10...
Also, does the limited ABI work on other platforms? If it does, we should ensure that the Windows support works the same. If it doesn't, do we want a Windows-only solution (why is that OK?) or should we extend to (say) manylinux or OSX (at the risk of making the job too big for anyone to actually get anywhere with it).
So to move this forward, I think someone needs to write up the process of building and using a limited ABI wheel, as described above, and document suggested answers to the various questions that will come out in the process of going through the details. Is that something you'd be willing to take on?
From that, we'd have something concrete to debate. I'm not sure how many people have an interest in this topic, so getting people with the necessary knowledge to chime in might take some work (I'm interested, but I don't have detail understanding of build options and linking conventions). The ultimate goal would be some sort of PEP covering handling limited ABI extensions within the packaging infrastructure.
Does that seem reasonable? Is that the sort of guidance you were looking for? I doubt anything is going to happen on this subject until someone with the interest in moving it forward steps up to do the work of making a proposal and collecting community views.
As an example, see... https://pypi.org/project/PyQt5/#files These are extension modules that use the limited ABI. As you can see I am anticipating they will also work with Python v3.8. They are created from my own build tools. Phil
So it appears the "supports abi3" check on Windows is simply python 3.5+? There is also the question of which visual studio CRT is in use, which could break compat apart from the Python ABI. This is the change to add abi3 to wheel including an extension using it. Must pass a define, which should probably be and argument to Extension (), as well as the flag. https://bitbucket.org/pypa/wheel/pull-requests/69/add-py-limited-api-flag-to... bdist_wheel could be more clever and enumerate all the extensions looking for the py_limited_api flag. On Tue, Jul 17, 2018, 12:58 Phil Thompson <phil@riverbankcomputing.com> wrote:
On 17 Jul 2018, at 5:37 pm, Paul Moore <p.f.moore@gmail.com> wrote:
On 17 July 2018 at 16:59, Cosimo Lupo <cosimo@anthrotype.com> wrote:
I would like to revive this 5 year old thread and see if we can stir
up a bit.
Basically the problem is that, in the current state of the PEPs and build tools, it is still not possible to build and distribute a Windows wheel
includes an extension module compiled with Py_LIMITED_API. Setuptools can successfully build such extension module on Windows (with `.pyd` file extension and no extra specifiers in the module filename), and these seems to work at least on CPython 3.5 and above. However the `--py-limited-api cpXX` option of bdist_wheel command fails on Windows because it attempts to use the `abi3` tag but the latter is not in the
things that list
of compatible tags for that platform. One can work around this by creating a wheel with `none` as the abi tag, and `cp35.cp36.cp37` as the python implementation tag but this feels a bit hackish.
Here are some unresolved questions from the old distutils-sig thread, quoting Paul Moore:
2. How should tools determine which ABIs a given Python supports? (This is the get_supported function in wheel and distlib). The "Use" section of the PEP (http://www.python.org/dev/peps/pep-0425/#id1) gives a Linux-based example, but nothing normative and nothing that is understandable to a Windows user.
And from Vinay Sajip
For Windows, we (eventually) need some low-level sysconfig-supported way to get the ABI information in an analogous way to how it happens on POSIX: and because that's not currently there, distlib doesn't provide any ABI information on Windows other than "none".
Other related links: https://github.com/pypa/pip/issues/4445
https://mail.python.org/pipermail/distutils-sig/2018-January/031856.html
So.. what needs to be done here to allow distributing/installing Windows wheels with Py_LIMITED_API support?
IMO, the question I posed back then remains key. Vinay's response is fair, but I don't think that waiting for core Python to provide something via sysconfig is practical (it's not happened yet, so why would we expect things to change?). So I think the next step is probably for someone to propose an algorithm that can be used. Actually, what I'd like to see is a full end to end proposal of the process someone would use to build and install a limited-ABI extension. That would probably tease out a number of issues.
I imagine the steps would be something like this:
1. Create an extension. Presumably you'd need to #define PY_LIMITED_ABI in the source. 2. Build a wheel from it - how would you do that? I gather it's possible to do this with plain setuptools - would it be necessary to do this with setuptools/bdist_wheel, or should there be a way to request a limited ABI build via pip? If we do want to be able to request this from a build tools like pip, do we need something in PEP 517? Are we only looking at the prebuilt wheel case, or do we need to support building from source? 3. What tags would the built wheel have? 4. Install that wheel - `pip install xxx`. Pip needs to be able to enumerate the full list of valid tags here (cp37-abi3, cp3-abi3, ...) There are also questions like - if there's a limited ABI wheel and a full ABI (version specific) wheel, which takes precedence?
I don't honestly know how well the limited ABI actually achieves its goals - is "cp3-abi3-win_x86_64" a realistic tag to apply? Can limited ABI wheels really be used on any version of Python 3? That's a question for python-dev, rather than distutils-sig, but if we take the position that this is what's promised, and it later turns out not to be true, we've got a lot of wheel renaming to do when Python 3.10 comes out and it doesn't support the same limited ABI as 3.x for x < 10...
Also, does the limited ABI work on other platforms? If it does, we should ensure that the Windows support works the same. If it doesn't, do we want a Windows-only solution (why is that OK?) or should we extend to (say) manylinux or OSX (at the risk of making the job too big for anyone to actually get anywhere with it).
So to move this forward, I think someone needs to write up the process of building and using a limited ABI wheel, as described above, and document suggested answers to the various questions that will come out in the process of going through the details. Is that something you'd be willing to take on?
From that, we'd have something concrete to debate. I'm not sure how many people have an interest in this topic, so getting people with the necessary knowledge to chime in might take some work (I'm interested, but I don't have detail understanding of build options and linking conventions). The ultimate goal would be some sort of PEP covering handling limited ABI extensions within the packaging infrastructure.
Does that seem reasonable? Is that the sort of guidance you were looking for? I doubt anything is going to happen on this subject until someone with the interest in moving it forward steps up to do the work of making a proposal and collecting community views.
As an example, see...
https://pypi.org/project/PyQt5/#files
These are extension modules that use the limited ABI. As you can see I am anticipating they will also work with Python v3.8. They are created from my own build tools.
Phil -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/Z...
The promise behind the limited ABI is exactly that if your extension works on 3.x, it will also work on 3.y, for y >= x. One thing to watch out for: normally extension modules on Linux and MacOS don't try to link to libpython. Instead they trust that someone else will make sure they only get loaded into a compatible python interpreter, and that all the symbols they need from python will be injected by the host process. On Windows, the way the dynamic loader works, you can't do this: every extension module has to explicitly say "I'm getting PyNumber_Add from the dll named: somethingsomething.dll" But traditionally, version X.Y of python includes a pythonXY.dll, so there's no consistent name across releases. So even if your library uses only the limited ABI and all of your imports could just as well come from python36.dll or python37.dll... you need some way to express that, and only Windows has this problem. I feel bad sending this without doing my own research, but I don't have access to a Windows box right now. Does anyone know if this problem has been solved? Is it still true that Windows python dlls always include the python version in their name? -n On Tue, Jul 17, 2018, 09:38 Paul Moore <p.f.moore@gmail.com> wrote:
I would like to revive this 5 year old thread and see if we can stir
up a bit.
Basically the problem is that, in the current state of the PEPs and build tools, it is still not possible to build and distribute a Windows wheel
includes an extension module compiled with Py_LIMITED_API. Setuptools can successfully build such extension module on Windows (with `.pyd` file extension and no extra specifiers in the module filename), and these seems to work at least on CPython 3.5 and above. However the `--py-limited-api cpXX` option of bdist_wheel command fails on Windows because it attempts to use the `abi3` tag but the latter is not in the
On 17 July 2018 at 16:59, Cosimo Lupo <cosimo@anthrotype.com> wrote: things that list
of compatible tags for that platform. One can work around this by creating a wheel with `none` as the abi tag, and `cp35.cp36.cp37` as the python implementation tag but this feels a bit hackish.
Here are some unresolved questions from the old distutils-sig thread, quoting Paul Moore:
2. How should tools determine which ABIs a given Python supports? (This is the get_supported function in wheel and distlib). The "Use" section of the PEP (http://www.python.org/dev/peps/pep-0425/#id1) gives a Linux-based example, but nothing normative and nothing that is understandable to a Windows user.
And from Vinay Sajip
For Windows, we (eventually) need some low-level sysconfig-supported way to get the ABI information in an analogous way to how it happens on POSIX: and because that's not currently there, distlib doesn't provide any ABI information on Windows other than "none".
Other related links: https://github.com/pypa/pip/issues/4445 https://mail.python.org/pipermail/distutils-sig/2018-January/031856.html
So.. what needs to be done here to allow distributing/installing Windows wheels with Py_LIMITED_API support?
IMO, the question I posed back then remains key. Vinay's response is fair, but I don't think that waiting for core Python to provide something via sysconfig is practical (it's not happened yet, so why would we expect things to change?). So I think the next step is probably for someone to propose an algorithm that can be used. Actually, what I'd like to see is a full end to end proposal of the process someone would use to build and install a limited-ABI extension. That would probably tease out a number of issues.
I imagine the steps would be something like this:
1. Create an extension. Presumably you'd need to #define PY_LIMITED_ABI in the source. 2. Build a wheel from it - how would you do that? I gather it's possible to do this with plain setuptools - would it be necessary to do this with setuptools/bdist_wheel, or should there be a way to request a limited ABI build via pip? If we do want to be able to request this from a build tools like pip, do we need something in PEP 517? Are we only looking at the prebuilt wheel case, or do we need to support building from source? 3. What tags would the built wheel have? 4. Install that wheel - `pip install xxx`. Pip needs to be able to enumerate the full list of valid tags here (cp37-abi3, cp3-abi3, ...) There are also questions like - if there's a limited ABI wheel and a full ABI (version specific) wheel, which takes precedence?
I don't honestly know how well the limited ABI actually achieves its goals - is "cp3-abi3-win_x86_64" a realistic tag to apply? Can limited ABI wheels really be used on any version of Python 3? That's a question for python-dev, rather than distutils-sig, but if we take the position that this is what's promised, and it later turns out not to be true, we've got a lot of wheel renaming to do when Python 3.10 comes out and it doesn't support the same limited ABI as 3.x for x < 10...
Also, does the limited ABI work on other platforms? If it does, we should ensure that the Windows support works the same. If it doesn't, do we want a Windows-only solution (why is that OK?) or should we extend to (say) manylinux or OSX (at the risk of making the job too big for anyone to actually get anywhere with it).
So to move this forward, I think someone needs to write up the process of building and using a limited ABI wheel, as described above, and document suggested answers to the various questions that will come out in the process of going through the details. Is that something you'd be willing to take on?
From that, we'd have something concrete to debate. I'm not sure how many people have an interest in this topic, so getting people with the necessary knowledge to chime in might take some work (I'm interested, but I don't have detail understanding of build options and linking conventions). The ultimate goal would be some sort of PEP covering handling limited ABI extensions within the packaging infrastructure.
Does that seem reasonable? Is that the sort of guidance you were looking for? I doubt anything is going to happen on this subject until someone with the interest in moving it forward steps up to do the work of making a proposal and collecting community views.
Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/F...
According to a link in this chain, virtualenv lately copies python3.dll (+ python3.7 e.g.) on Windows. So hopefully the extension links with that dll. On Tue, Jul 17, 2018, 15:51 Nathaniel Smith <njs@pobox.com> wrote:
The promise behind the limited ABI is exactly that if your extension works on 3.x, it will also work on 3.y, for y >= x.
One thing to watch out for: normally extension modules on Linux and MacOS don't try to link to libpython. Instead they trust that someone else will make sure they only get loaded into a compatible python interpreter, and that all the symbols they need from python will be injected by the host process.
On Windows, the way the dynamic loader works, you can't do this: every extension module has to explicitly say "I'm getting PyNumber_Add from the dll named: somethingsomething.dll"
But traditionally, version X.Y of python includes a pythonXY.dll, so there's no consistent name across releases. So even if your library uses only the limited ABI and all of your imports could just as well come from python36.dll or python37.dll... you need some way to express that, and only Windows has this problem.
I feel bad sending this without doing my own research, but I don't have access to a Windows box right now. Does anyone know if this problem has been solved? Is it still true that Windows python dlls always include the python version in their name?
-n
On Tue, Jul 17, 2018, 09:38 Paul Moore <p.f.moore@gmail.com> wrote:
I would like to revive this 5 year old thread and see if we can stir
up a bit.
Basically the problem is that, in the current state of the PEPs and build tools, it is still not possible to build and distribute a Windows wheel
includes an extension module compiled with Py_LIMITED_API. Setuptools can successfully build such extension module on Windows (with `.pyd` file extension and no extra specifiers in the module filename), and these seems to work at least on CPython 3.5 and above. However the `--py-limited-api cpXX` option of bdist_wheel command fails on Windows because it attempts to use the `abi3` tag but the latter is not in the
On 17 July 2018 at 16:59, Cosimo Lupo <cosimo@anthrotype.com> wrote: things that list
of compatible tags for that platform. One can work around this by creating a wheel with `none` as the abi tag, and `cp35.cp36.cp37` as the python implementation tag but this feels a bit hackish.
Here are some unresolved questions from the old distutils-sig thread, quoting Paul Moore:
2. How should tools determine which ABIs a given Python supports? (This is the get_supported function in wheel and distlib). The "Use" section of the PEP (http://www.python.org/dev/peps/pep-0425/#id1) gives a Linux-based example, but nothing normative and nothing that is understandable to a Windows user.
And from Vinay Sajip
For Windows, we (eventually) need some low-level sysconfig-supported way to get the ABI information in an analogous way to how it happens on POSIX: and because that's not currently there, distlib doesn't provide any ABI information on Windows other than "none".
Other related links: https://github.com/pypa/pip/issues/4445
https://mail.python.org/pipermail/distutils-sig/2018-January/031856.html
So.. what needs to be done here to allow distributing/installing Windows wheels with Py_LIMITED_API support?
IMO, the question I posed back then remains key. Vinay's response is fair, but I don't think that waiting for core Python to provide something via sysconfig is practical (it's not happened yet, so why would we expect things to change?). So I think the next step is probably for someone to propose an algorithm that can be used. Actually, what I'd like to see is a full end to end proposal of the process someone would use to build and install a limited-ABI extension. That would probably tease out a number of issues.
I imagine the steps would be something like this:
1. Create an extension. Presumably you'd need to #define PY_LIMITED_ABI in the source. 2. Build a wheel from it - how would you do that? I gather it's possible to do this with plain setuptools - would it be necessary to do this with setuptools/bdist_wheel, or should there be a way to request a limited ABI build via pip? If we do want to be able to request this from a build tools like pip, do we need something in PEP 517? Are we only looking at the prebuilt wheel case, or do we need to support building from source? 3. What tags would the built wheel have? 4. Install that wheel - `pip install xxx`. Pip needs to be able to enumerate the full list of valid tags here (cp37-abi3, cp3-abi3, ...) There are also questions like - if there's a limited ABI wheel and a full ABI (version specific) wheel, which takes precedence?
I don't honestly know how well the limited ABI actually achieves its goals - is "cp3-abi3-win_x86_64" a realistic tag to apply? Can limited ABI wheels really be used on any version of Python 3? That's a question for python-dev, rather than distutils-sig, but if we take the position that this is what's promised, and it later turns out not to be true, we've got a lot of wheel renaming to do when Python 3.10 comes out and it doesn't support the same limited ABI as 3.x for x < 10...
Also, does the limited ABI work on other platforms? If it does, we should ensure that the Windows support works the same. If it doesn't, do we want a Windows-only solution (why is that OK?) or should we extend to (say) manylinux or OSX (at the risk of making the job too big for anyone to actually get anywhere with it).
So to move this forward, I think someone needs to write up the process of building and using a limited ABI wheel, as described above, and document suggested answers to the various questions that will come out in the process of going through the details. Is that something you'd be willing to take on?
From that, we'd have something concrete to debate. I'm not sure how many people have an interest in this topic, so getting people with the necessary knowledge to chime in might take some work (I'm interested, but I don't have detail understanding of build options and linking conventions). The ultimate goal would be some sort of PEP covering handling limited ABI extensions within the packaging infrastructure.
Does that seem reasonable? Is that the sort of guidance you were looking for? I doubt anything is going to happen on this subject until someone with the interest in moving it forward steps up to do the work of making a proposal and collecting community views.
Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/F...
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/Y...
On 17 Jul 2018, at 8:57 pm, Daniel Holth <dholth@gmail.com> wrote:
According to a link in this chain, virtualenv lately copies python3.dll (+ python3.7 e.g.) on Windows. So hopefully the extension links with that dll.
Yes. Historically at least, Anaconda does not include that DLL and (IMHO) is inherently broken. Phil
On Tue, Jul 17, 2018, 15:51 Nathaniel Smith <njs@pobox.com> wrote: The promise behind the limited ABI is exactly that if your extension works on 3.x, it will also work on 3.y, for y >= x.
One thing to watch out for: normally extension modules on Linux and MacOS don't try to link to libpython. Instead they trust that someone else will make sure they only get loaded into a compatible python interpreter, and that all the symbols they need from python will be injected by the host process.
On Windows, the way the dynamic loader works, you can't do this: every extension module has to explicitly say "I'm getting PyNumber_Add from the dll named: somethingsomething.dll"
But traditionally, version X.Y of python includes a pythonXY.dll, so there's no consistent name across releases. So even if your library uses only the limited ABI and all of your imports could just as well come from python36.dll or python37.dll... you need some way to express that, and only Windows has this problem.
I feel bad sending this without doing my own research, but I don't have access to a Windows box right now. Does anyone know if this problem has been solved? Is it still true that Windows python dlls always include the python version in their name?
-n
On Tue, Jul 17, 2018, 09:38 Paul Moore <p.f.moore@gmail.com> wrote: On 17 July 2018 at 16:59, Cosimo Lupo <cosimo@anthrotype.com> wrote:
I would like to revive this 5 year old thread and see if we can stir things up a bit.
Basically the problem is that, in the current state of the PEPs and build tools, it is still not possible to build and distribute a Windows wheel that includes an extension module compiled with Py_LIMITED_API. Setuptools can successfully build such extension module on Windows (with `.pyd` file extension and no extra specifiers in the module filename), and these seems to work at least on CPython 3.5 and above. However the `--py-limited-api cpXX` option of bdist_wheel command fails on Windows because it attempts to use the `abi3` tag but the latter is not in the list of compatible tags for that platform. One can work around this by creating a wheel with `none` as the abi tag, and `cp35.cp36.cp37` as the python implementation tag but this feels a bit hackish.
Here are some unresolved questions from the old distutils-sig thread, quoting Paul Moore:
2. How should tools determine which ABIs a given Python supports? (This is the get_supported function in wheel and distlib). The "Use" section of the PEP (http://www.python.org/dev/peps/pep-0425/#id1) gives a Linux-based example, but nothing normative and nothing that is understandable to a Windows user.
And from Vinay Sajip
For Windows, we (eventually) need some low-level sysconfig-supported way to get the ABI information in an analogous way to how it happens on POSIX: and because that's not currently there, distlib doesn't provide any ABI information on Windows other than "none".
Other related links: https://github.com/pypa/pip/issues/4445 https://mail.python.org/pipermail/distutils-sig/2018-January/031856.html
So.. what needs to be done here to allow distributing/installing Windows wheels with Py_LIMITED_API support?
IMO, the question I posed back then remains key. Vinay's response is fair, but I don't think that waiting for core Python to provide something via sysconfig is practical (it's not happened yet, so why would we expect things to change?). So I think the next step is probably for someone to propose an algorithm that can be used. Actually, what I'd like to see is a full end to end proposal of the process someone would use to build and install a limited-ABI extension. That would probably tease out a number of issues.
I imagine the steps would be something like this:
1. Create an extension. Presumably you'd need to #define PY_LIMITED_ABI in the source. 2. Build a wheel from it - how would you do that? I gather it's possible to do this with plain setuptools - would it be necessary to do this with setuptools/bdist_wheel, or should there be a way to request a limited ABI build via pip? If we do want to be able to request this from a build tools like pip, do we need something in PEP 517? Are we only looking at the prebuilt wheel case, or do we need to support building from source? 3. What tags would the built wheel have? 4. Install that wheel - `pip install xxx`. Pip needs to be able to enumerate the full list of valid tags here (cp37-abi3, cp3-abi3, ...) There are also questions like - if there's a limited ABI wheel and a full ABI (version specific) wheel, which takes precedence?
I don't honestly know how well the limited ABI actually achieves its goals - is "cp3-abi3-win_x86_64" a realistic tag to apply? Can limited ABI wheels really be used on any version of Python 3? That's a question for python-dev, rather than distutils-sig, but if we take the position that this is what's promised, and it later turns out not to be true, we've got a lot of wheel renaming to do when Python 3.10 comes out and it doesn't support the same limited ABI as 3.x for x < 10...
Also, does the limited ABI work on other platforms? If it does, we should ensure that the Windows support works the same. If it doesn't, do we want a Windows-only solution (why is that OK?) or should we extend to (say) manylinux or OSX (at the risk of making the job too big for anyone to actually get anywhere with it).
So to move this forward, I think someone needs to write up the process of building and using a limited ABI wheel, as described above, and document suggested answers to the various questions that will come out in the process of going through the details. Is that something you'd be willing to take on?
From that, we'd have something concrete to debate. I'm not sure how many people have an interest in this topic, so getting people with the necessary knowledge to chime in might take some work (I'm interested, but I don't have detail understanding of build options and linking conventions). The ultimate goal would be some sort of PEP covering handling limited ABI extensions within the packaging infrastructure.
Does that seem reasonable? Is that the sort of guidance you were looking for? I doubt anything is going to happen on this subject until someone with the interest in moving it forward steps up to do the work of making a proposal and collecting community views.
Paul -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/F... -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/Y... -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/H...
This is no longer accurate -- since early 2017, the Windows Python builds include python3.dll alongside the major.minor DLL. On 7/17/18, 4:24 PM, "Phil Thompson" <phil@riverbankcomputing.com> wrote: On 17 Jul 2018, at 8:57 pm, Daniel Holth <dholth@gmail.com> wrote: > > According to a link in this chain, virtualenv lately copies python3.dll (+ python3.7 e.g.) on Windows. So hopefully the extension links with that dll. Yes. Historically at least, Anaconda does not include that DLL and (IMHO) is inherently broken. Phil > On Tue, Jul 17, 2018, 15:51 Nathaniel Smith <njs@pobox.com> wrote: > The promise behind the limited ABI is exactly that if your extension works on 3.x, it will also work on 3.y, for y >= x. > > One thing to watch out for: normally extension modules on Linux and MacOS don't try to link to libpython. Instead they trust that someone else will make sure they only get loaded into a compatible python interpreter, and that all the symbols they need from python will be injected by the host process. > > On Windows, the way the dynamic loader works, you can't do this: every extension module has to explicitly say "I'm getting PyNumber_Add from the dll named: somethingsomething.dll" > > But traditionally, version X.Y of python includes a pythonXY.dll, so there's no consistent name across releases. So even if your library uses only the limited ABI and all of your imports could just as well come from python36.dll or python37.dll... you need some way to express that, and only Windows has this problem. > > I feel bad sending this without doing my own research, but I don't have access to a Windows box right now. Does anyone know if this problem has been solved? Is it still true that Windows python dlls always include the python version in their name? > > -n > > On Tue, Jul 17, 2018, 09:38 Paul Moore <p.f.moore@gmail.com> wrote: > On 17 July 2018 at 16:59, Cosimo Lupo <cosimo@anthrotype.com> wrote: > > I would like to revive this 5 year old thread and see if we can stir things > > up a bit. > > > > Basically the problem is that, in the current state of the PEPs and build > > tools, it is still not possible to build and distribute a Windows wheel that > > includes an extension module compiled with Py_LIMITED_API. > > Setuptools can successfully build such extension module on Windows (with > > `.pyd` file extension and no extra specifiers in the module filename), and > > these seems to work at least on CPython 3.5 and above. However the > > `--py-limited-api cpXX` option of bdist_wheel command fails on Windows > > because it attempts to use the `abi3` tag but the latter is not in the list > > of compatible tags for that platform. > > One can work around this by creating a wheel with `none` as the abi tag, and > > `cp35.cp36.cp37` as the python implementation tag but this feels a bit > > hackish. > > > > Here are some unresolved questions from the old distutils-sig thread, > > quoting Paul Moore: > > > >> 2. How should tools determine which ABIs a given Python supports? > >> (This is the get_supported function in wheel and distlib). The "Use" > >> section of the PEP (https://urldefense.proofpoint.com/v2/url?u=http-3A__www.python.org_dev_peps_pep-2D0425_-23id1&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=CI-50dMn_QF7jh6LxyY6q37nHnL8Zn0dP_GnjnXZ6iE&e=) > >> gives a Linux-based example, but nothing normative and nothing that is > >> understandable to a Windows user. > > > > And from Vinay Sajip > > > >> For Windows, we (eventually) need some low-level sysconfig-supported way > >> to get the ABI information in an analogous way to how it happens on POSIX: > >> and > >> because that's not currently there, distlib doesn't provide any ABI > >> information > >> on Windows other than "none". > > > > Other related links: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_pypa_pip_issues_4445&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=FbWIuKimM7I-r1z0drqZNQuxpypWBonCVRz9Jc8ryJ8&e= > > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_pipermail_distutils-2Dsig_2018-2DJanuary_031856.html&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=c5RVqW1Zda4NUInIER9iAEAH3fFfeBSvoOkf_KSKtsw&e= > > > > So.. what needs to be done here to allow distributing/installing Windows > > wheels with Py_LIMITED_API support? > > IMO, the question I posed back then remains key. Vinay's response is > fair, but I don't think that waiting for core Python to provide > something via sysconfig is practical (it's not happened yet, so why > would we expect things to change?). So I think the next step is > probably for someone to propose an algorithm that can be used. > Actually, what I'd like to see is a full end to end proposal of the > process someone would use to build and install a limited-ABI > extension. That would probably tease out a number of issues. > > I imagine the steps would be something like this: > > 1. Create an extension. Presumably you'd need to #define > PY_LIMITED_ABI in the source. > 2. Build a wheel from it - how would you do that? I gather it's > possible to do this with plain setuptools - would it be necessary to > do this with setuptools/bdist_wheel, or should there be a way to > request a limited ABI build via pip? If we do want to be able to > request this from a build tools like pip, do we need something in PEP > 517? Are we only looking at the prebuilt wheel case, or do we need to > support building from source? > 3. What tags would the built wheel have? > 4. Install that wheel - `pip install xxx`. Pip needs to be able to > enumerate the full list of valid tags here (cp37-abi3, cp3-abi3, ...) > There are also questions like - if there's a limited ABI wheel and a > full ABI (version specific) wheel, which takes precedence? > > I don't honestly know how well the limited ABI actually achieves its > goals - is "cp3-abi3-win_x86_64" a realistic tag to apply? Can limited > ABI wheels really be used on any version of Python 3? That's a > question for python-dev, rather than distutils-sig, but if we take the > position that this is what's promised, and it later turns out not to > be true, we've got a lot of wheel renaming to do when Python 3.10 > comes out and it doesn't support the same limited ABI as 3.x for x < > 10... > > Also, does the limited ABI work on other platforms? If it does, we > should ensure that the Windows support works the same. If it doesn't, > do we want a Windows-only solution (why is that OK?) or should we > extend to (say) manylinux or OSX (at the risk of making the job too > big for anyone to actually get anywhere with it). > > So to move this forward, I think someone needs to write up the process > of building and using a limited ABI wheel, as described above, and > document suggested answers to the various questions that will come out > in the process of going through the details. Is that something you'd > be willing to take on? > > From that, we'd have something concrete to debate. I'm not sure how > many people have an interest in this topic, so getting people with the > necessary knowledge to chime in might take some work (I'm interested, > but I don't have detail understanding of build options and linking > conventions). The ultimate goal would be some sort of PEP covering > handling limited ABI extensions within the packaging infrastructure. > > Does that seem reasonable? Is that the sort of guidance you were > looking for? I doubt anything is going to happen on this subject until > someone with the interest in moving it forward steps up to do the work > of making a proposal and collecting community views. > > Paul > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-leave@python.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_mailman3_lists_distutils-2Dsig.python.org_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=M8BIYDHADRC2DG3p_8p4E-jtXEvvKIrEAFpzMVko0is&e= > Message archived at https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_archives_list_distutils-2Dsig-40python.org_message_F3QCJTGKFZJFWWV4CLLQFQ6XGBAQNNFX_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=ZwMkKEX6br2nQj9KrQhvui3DANLGlR32_BOwALjytVw&e= > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-leave@python.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_mailman3_lists_distutils-2Dsig.python.org_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=M8BIYDHADRC2DG3p_8p4E-jtXEvvKIrEAFpzMVko0is&e= > Message archived at https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_archives_list_distutils-2Dsig-40python.org_message_YXMNZEF4DGCNFFE3M66LVBOUT73HCPTF_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=b6Ex8zkpUUKFFdzBdGn9uEVciRHgK5a8RMHLccIB9dk&e= > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-leave@python.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_mailman3_lists_distutils-2Dsig.python.org_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=M8BIYDHADRC2DG3p_8p4E-jtXEvvKIrEAFpzMVko0is&e= > Message archived at https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_archives_list_distutils-2Dsig-40python.org_message_H2DHMDLYA77BDGDRYLFEFY6ZFS7QVNWI_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=j402Afyg9SFbc2UK-S8ZlUlw8SDTncSNL9Ais3t-ISY&e= -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_mailman3_lists_distutils-2Dsig.python.org_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=M8BIYDHADRC2DG3p_8p4E-jtXEvvKIrEAFpzMVko0is&e= Message archived at https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_archives_list_distutils-2Dsig-40python.org_message_UORLWC6PNWHKOCL4TO5ZAXZD5HTPAU6J_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=NLh87pQi9YHUUWCZtAma8wk6VHqgd93f9wIPVfUEubw&e=
Yes, it seems like a number of the ABI3 prerequisites have caught up since 2016. Now is a great time to revisit the issue. On Tue, Jul 17, 2018 at 4:29 PM Shaun Walbridge <SWalbridge@esri.com> wrote:
This is no longer accurate -- since early 2017, the Windows Python builds include python3.dll alongside the major.minor DLL.
On 7/17/18, 4:24 PM, "Phil Thompson" <phil@riverbankcomputing.com> wrote:
On 17 Jul 2018, at 8:57 pm, Daniel Holth <dholth@gmail.com> wrote: > > According to a link in this chain, virtualenv lately copies python3.dll (+ python3.7 e.g.) on Windows. So hopefully the extension links with that dll.
Yes. Historically at least, Anaconda does not include that DLL and (IMHO) is inherently broken.
Phil
> On Tue, Jul 17, 2018, 15:51 Nathaniel Smith <njs@pobox.com> wrote: > The promise behind the limited ABI is exactly that if your extension works on 3.x, it will also work on 3.y, for y >= x. > > One thing to watch out for: normally extension modules on Linux and MacOS don't try to link to libpython. Instead they trust that someone else will make sure they only get loaded into a compatible python interpreter, and that all the symbols they need from python will be injected by the host process. > > On Windows, the way the dynamic loader works, you can't do this: every extension module has to explicitly say "I'm getting PyNumber_Add from the dll named: somethingsomething.dll" > > But traditionally, version X.Y of python includes a pythonXY.dll, so there's no consistent name across releases. So even if your library uses only the limited ABI and all of your imports could just as well come from python36.dll or python37.dll... you need some way to express that, and only Windows has this problem. > > I feel bad sending this without doing my own research, but I don't have access to a Windows box right now. Does anyone know if this problem has been solved? Is it still true that Windows python dlls always include the python version in their name? > > -n > > On Tue, Jul 17, 2018, 09:38 Paul Moore <p.f.moore@gmail.com> wrote: > On 17 July 2018 at 16:59, Cosimo Lupo <cosimo@anthrotype.com> wrote: > > I would like to revive this 5 year old thread and see if we can stir things > > up a bit. > > > > Basically the problem is that, in the current state of the PEPs and build > > tools, it is still not possible to build and distribute a Windows wheel that > > includes an extension module compiled with Py_LIMITED_API. > > Setuptools can successfully build such extension module on Windows (with > > `.pyd` file extension and no extra specifiers in the module filename), and > > these seems to work at least on CPython 3.5 and above. However the > > `--py-limited-api cpXX` option of bdist_wheel command fails on Windows > > because it attempts to use the `abi3` tag but the latter is not in the list > > of compatible tags for that platform. > > One can work around this by creating a wheel with `none` as the abi tag, and > > `cp35.cp36.cp37` as the python implementation tag but this feels a bit > > hackish. > > > > Here are some unresolved questions from the old distutils-sig thread, > > quoting Paul Moore: > > > >> 2. How should tools determine which ABIs a given Python supports? > >> (This is the get_supported function in wheel and distlib). The "Use" > >> section of the PEP ( https://urldefense.proofpoint.com/v2/url?u=http-3A__www.python.org_dev_peps_pep-2D0425_-23id1&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=CI-50dMn_QF7jh6LxyY6q37nHnL8Zn0dP_GnjnXZ6iE&e= ) > >> gives a Linux-based example, but nothing normative and nothing that is > >> understandable to a Windows user. > > > > And from Vinay Sajip > > > >> For Windows, we (eventually) need some low-level sysconfig-supported way > >> to get the ABI information in an analogous way to how it happens on POSIX: > >> and > >> because that's not currently there, distlib doesn't provide any ABI > >> information > >> on Windows other than "none". > > > > Other related links: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_pypa_pip_issues_4445&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=FbWIuKimM7I-r1z0drqZNQuxpypWBonCVRz9Jc8ryJ8&e= > > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_pipermail_distutils-2Dsig_2018-2DJanuary_031856.html&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=c5RVqW1Zda4NUInIER9iAEAH3fFfeBSvoOkf_KSKtsw&e= > > > > So.. what needs to be done here to allow distributing/installing Windows > > wheels with Py_LIMITED_API support? > > IMO, the question I posed back then remains key. Vinay's response is > fair, but I don't think that waiting for core Python to provide > something via sysconfig is practical (it's not happened yet, so why > would we expect things to change?). So I think the next step is > probably for someone to propose an algorithm that can be used. > Actually, what I'd like to see is a full end to end proposal of the > process someone would use to build and install a limited-ABI > extension. That would probably tease out a number of issues. > > I imagine the steps would be something like this: > > 1. Create an extension. Presumably you'd need to #define > PY_LIMITED_ABI in the source. > 2. Build a wheel from it - how would you do that? I gather it's > possible to do this with plain setuptools - would it be necessary to > do this with setuptools/bdist_wheel, or should there be a way to > request a limited ABI build via pip? If we do want to be able to > request this from a build tools like pip, do we need something in PEP > 517? Are we only looking at the prebuilt wheel case, or do we need to > support building from source? > 3. What tags would the built wheel have? > 4. Install that wheel - `pip install xxx`. Pip needs to be able to > enumerate the full list of valid tags here (cp37-abi3, cp3-abi3, ...) > There are also questions like - if there's a limited ABI wheel and a > full ABI (version specific) wheel, which takes precedence? > > I don't honestly know how well the limited ABI actually achieves its > goals - is "cp3-abi3-win_x86_64" a realistic tag to apply? Can limited > ABI wheels really be used on any version of Python 3? That's a > question for python-dev, rather than distutils-sig, but if we take the > position that this is what's promised, and it later turns out not to > be true, we've got a lot of wheel renaming to do when Python 3.10 > comes out and it doesn't support the same limited ABI as 3.x for x < > 10... > > Also, does the limited ABI work on other platforms? If it does, we > should ensure that the Windows support works the same. If it doesn't, > do we want a Windows-only solution (why is that OK?) or should we > extend to (say) manylinux or OSX (at the risk of making the job too > big for anyone to actually get anywhere with it). > > So to move this forward, I think someone needs to write up the process > of building and using a limited ABI wheel, as described above, and > document suggested answers to the various questions that will come out > in the process of going through the details. Is that something you'd > be willing to take on? > > From that, we'd have something concrete to debate. I'm not sure how > many people have an interest in this topic, so getting people with the > necessary knowledge to chime in might take some work (I'm interested, > but I don't have detail understanding of build options and linking > conventions). The ultimate goal would be some sort of PEP covering > handling limited ABI extensions within the packaging infrastructure. > > Does that seem reasonable? Is that the sort of guidance you were > looking for? I doubt anything is going to happen on this subject until > someone with the interest in moving it forward steps up to do the work > of making a proposal and collecting community views. > > Paul > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-leave@python.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_mailman3_lists_distutils-2Dsig.python.org_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=M8BIYDHADRC2DG3p_8p4E-jtXEvvKIrEAFpzMVko0is&e= > Message archived at https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_archives_list_distutils-2Dsig-40python.org_message_F3QCJTGKFZJFWWV4CLLQFQ6XGBAQNNFX_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=ZwMkKEX6br2nQj9KrQhvui3DANLGlR32_BOwALjytVw&e= > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-leave@python.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_mailman3_lists_distutils-2Dsig.python.org_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=M8BIYDHADRC2DG3p_8p4E-jtXEvvKIrEAFpzMVko0is&e= > Message archived at https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_archives_list_distutils-2Dsig-40python.org_message_YXMNZEF4DGCNFFE3M66LVBOUT73HCPTF_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=b6Ex8zkpUUKFFdzBdGn9uEVciRHgK5a8RMHLccIB9dk&e= > -- > Distutils-SIG mailing list -- distutils-sig@python.org > To unsubscribe send an email to distutils-sig-leave@python.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_mailman3_lists_distutils-2Dsig.python.org_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=M8BIYDHADRC2DG3p_8p4E-jtXEvvKIrEAFpzMVko0is&e= > Message archived at https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_archives_list_distutils-2Dsig-40python.org_message_H2DHMDLYA77BDGDRYLFEFY6ZFS7QVNWI_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=j402Afyg9SFbc2UK-S8ZlUlw8SDTncSNL9Ais3t-ISY&e= -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org
https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_mailman3_lists_distutils-2Dsig.python.org_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=M8BIYDHADRC2DG3p_8p4E-jtXEvvKIrEAFpzMVko0is&e= Message archived at https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mm3_archives_list_distutils-2Dsig-40python.org_message_UORLWC6PNWHKOCL4TO5ZAXZD5HTPAU6J_&d=DwICAg&c=n6-cguzQvX_tUIrZOS_4Og&r=YFaRLkcUCdDkLrpTbNOUV9J1CwYBCTMwgm5tdQkRSm4&m=xlQFE7IsUpY_J8xwu4_W--sNGrS38xw5c9B0ZJmewgE&s=NLh87pQi9YHUUWCZtAma8wk6VHqgd93f9wIPVfUEubw&e=
participants (6)
-
Cosimo Lupo
-
Daniel Holth
-
Nathaniel Smith
-
Paul Moore
-
Phil Thompson
-
Shaun Walbridge