Including soabi tag in --user path?
Hi all, A user on the pywavelets mailing list seems to have gotten themselves into a mess: AFAICT they did 'pip install --user pywavelets' using a ucs4 python, and then started a ucs2 python and tried to import it, which of course failed because these two python builds have incompatible ABIs, even though they share the same --user directory. In retrospect, this seems like an obvious oversight: right now the --user path (on Linux) is something like ~/.local/lib/python3.5/site-packages Wouldn't it make more sense for it to be something like ~/.local/lib/cpython-35m-x86_64-linux-gnu/site-packages so that Pythons with incompatible ABIs can't see each other's packages? (That's sysconfig.get_config_var("SOABI").) Obviously this can't really be fixed for old versions, but is this a change that should be made in 3.7? -n -- Nathaniel J. Smith -- https://vorpus.org
On Jul 15, 2017, at 00:22, Nathaniel Smith <njs@pobox.com> wrote:
A user on the pywavelets mailing list seems to have gotten themselves into a mess: AFAICT they did 'pip install --user pywavelets' using a ucs4 python, and then started a ucs2 python and tried to import it, which of course failed because these two python builds have incompatible ABIs, even though they share the same --user directory.
In retrospect, this seems like an obvious oversight: right now the --user path (on Linux) is something like
~/.local/lib/python3.5/site-packages
Wouldn't it make more sense for it to be something like
~/.local/lib/cpython-35m-x86_64-linux-gnu/site-packages
so that Pythons with incompatible ABIs can't see each other's packages? (That's sysconfig.get_config_var("SOABI").)
Obviously this can't really be fixed for old versions, but is this a change that should be made in 3.7?
I'm confused. This shouldn't be an issue for Python 3.*. First, there is no ucs2 / ucs4 option since Python 3.3 and PEP 393's flexible string representation. But even before that, PEP 3149 (in 3.2) is supposed to ensure that extension modules have unique names based on ABI, for example: _psutil_osx.cpython-35m-darwin.so This *is* a problem in Python 2.7 but that's not going to change now. -- Ned Deily nad@python.org -- []
On Fri, Jul 14, 2017 at 9:37 PM, Ned Deily <nad@python.org> wrote:
On Jul 15, 2017, at 00:22, Nathaniel Smith <njs@pobox.com> wrote:
A user on the pywavelets mailing list seems to have gotten themselves into a mess: AFAICT they did 'pip install --user pywavelets' using a ucs4 python, and then started a ucs2 python and tried to import it, which of course failed because these two python builds have incompatible ABIs, even though they share the same --user directory.
In retrospect, this seems like an obvious oversight: right now the --user path (on Linux) is something like
~/.local/lib/python3.5/site-packages
Wouldn't it make more sense for it to be something like
~/.local/lib/cpython-35m-x86_64-linux-gnu/site-packages
so that Pythons with incompatible ABIs can't see each other's packages? (That's sysconfig.get_config_var("SOABI").)
Obviously this can't really be fixed for old versions, but is this a change that should be made in 3.7?
I'm confused. This shouldn't be an issue for Python 3.*. First, there is no ucs2 / ucs4 option since Python 3.3 and PEP 393's flexible string representation.
Right, but there are still SOABI variations, e.g. debug builds versus regular builds.
But even before that, PEP 3149 (in 3.2) is supposed to ensure that extension modules have unique names based on ABI, for example:
_psutil_osx.cpython-35m-darwin.so
Right, but this only helps if you install multiple builds of the same package on top of each other. Which Debian is able to arrange, so PEP 3149 solves their problem. But the --user dir is managed by pip, and if you try to install a cpython-35dm package on top of a cpython-35m package, then pip will helpfully remove the first before installing the second. (And this is really the only reasonable thing for pip to do.) So in practice any given package in the --user dir can only support one SOABI variant, even though all the SOABI variants use the same dir. -n -- Nathaniel J. Smith -- https://vorpus.org
On Jul 15, 2017, at 00:41, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Jul 14, 2017 at 9:37 PM, Ned Deily <nad@python.org> wrote:
On Jul 15, 2017, at 00:22, Nathaniel Smith <njs@pobox.com> wrote:
A user on the pywavelets mailing list seems to have gotten themselves into a mess: AFAICT they did 'pip install --user pywavelets' using a ucs4 python, and then started a ucs2 python and tried to import it, which of course failed because these two python builds have incompatible ABIs, even though they share the same --user directory.
In retrospect, this seems like an obvious oversight: right now the --user path (on Linux) is something like
~/.local/lib/python3.5/site-packages
Wouldn't it make more sense for it to be something like
~/.local/lib/cpython-35m-x86_64-linux-gnu/site-packages
so that Pythons with incompatible ABIs can't see each other's packages? (That's sysconfig.get_config_var("SOABI").)
Obviously this can't really be fixed for old versions, but is this a change that should be made in 3.7?
I'm confused. This shouldn't be an issue for Python 3.*. First, there is no ucs2 / ucs4 option since Python 3.3 and PEP 393's flexible string representation.
Right, but there are still SOABI variations, e.g. debug builds versus regular builds.
And those are covered in the file names, e.g. a debug build extension module would be: _psutil_osx.cpython-35dm-darwin.so
But even before that, PEP 3149 (in 3.2) is supposed to ensure that extension modules have unique names based on ABI, for example:
_psutil_osx.cpython-35m-darwin.so
Right, but this only helps if you install multiple builds of the same package on top of each other. Which Debian is able to arrange, so PEP 3149 solves their problem. But the --user dir is managed by pip, and if you try to install a cpython-35dm package on top of a cpython-35m package, then pip will helpfully remove the first before installing the second. (And this is really the only reasonable thing for pip to do.) So in practice any given package in the --user dir can only support one SOABI variant, even though all the SOABI variants use the same dir.
Well, that sounds like either a pip problem or a packager's problem. The point of PEP 3149 was exactly to allow multiple builds of extension modules built with different ABIs to co-exist in the same directory. You would have the same behavior with packages installed to the system site-packergs dir, right? So what's special about --user? -- Ned Deily nad@python.org -- []
On Fri, Jul 14, 2017 at 9:48 PM, Ned Deily <nad@python.org> wrote:
On Jul 15, 2017, at 00:41, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Jul 14, 2017 at 9:37 PM, Ned Deily <nad@python.org> wrote:
On Jul 15, 2017, at 00:22, Nathaniel Smith <njs@pobox.com> wrote:
A user on the pywavelets mailing list seems to have gotten themselves into a mess: AFAICT they did 'pip install --user pywavelets' using a ucs4 python, and then started a ucs2 python and tried to import it, which of course failed because these two python builds have incompatible ABIs, even though they share the same --user directory.
In retrospect, this seems like an obvious oversight: right now the --user path (on Linux) is something like
~/.local/lib/python3.5/site-packages
Wouldn't it make more sense for it to be something like
~/.local/lib/cpython-35m-x86_64-linux-gnu/site-packages
so that Pythons with incompatible ABIs can't see each other's packages? (That's sysconfig.get_config_var("SOABI").)
Obviously this can't really be fixed for old versions, but is this a change that should be made in 3.7?
I'm confused. This shouldn't be an issue for Python 3.*. First, there is no ucs2 / ucs4 option since Python 3.3 and PEP 393's flexible string representation.
Right, but there are still SOABI variations, e.g. debug builds versus regular builds.
And those are covered in the file names, e.g. a debug build extension module would be:
_psutil_osx.cpython-35dm-darwin.so
But even before that, PEP 3149 (in 3.2) is supposed to ensure that extension modules have unique names based on ABI, for example:
_psutil_osx.cpython-35m-darwin.so
Right, but this only helps if you install multiple builds of the same package on top of each other. Which Debian is able to arrange, so PEP 3149 solves their problem. But the --user dir is managed by pip, and if you try to install a cpython-35dm package on top of a cpython-35m package, then pip will helpfully remove the first before installing the second. (And this is really the only reasonable thing for pip to do.) So in practice any given package in the --user dir can only support one SOABI variant, even though all the SOABI variants use the same dir.
Well, that sounds like either a pip problem or a packager's problem. The point of PEP 3149 was exactly to allow multiple builds of extension modules built with different ABIs to co-exist in the same directory. You would have the same behavior with packages installed to the system site-packergs dir, right? So what's special about --user?
For system-managed packages, we tend to get away with it because Debian [1] is able to enforce that there's one .deb file that ships all of the pure python files and all of the SOABI variants for extension modules together, so everything remains consistent. For pip managed packages this doesn't really make sense. You have to somehow ensure that two different pip runs, using different versions of python, are both allowed to install two copies of the same package on top of each other into the same directory, and that the only difference between them is the extension modules. And what do you do if one SOABI Python install package X version Y, but then a different SOABI Python tries to upgrade to package X version (Y+1)? Another possible option would be to have both an SOABI-tagged directory and a non-tagged directory on sys.path, and install packages that contain extensions into one and pure-python packages into the other. Allegedly this is what the purelib/platlib distinction in wheel metadata is supposed to enable, I think, but in practice there are definitely wheels out there that break this (e.g. tensorflow is allegedly a purelib wheel), so I don't know if it works in practice. And yeah, it might well make sense to do this for system site packages too, but then there might be additional complications with Debian etc. so I figured I'd start with the obvious case :-). -n [1] or whoever is distributing system packages, but PEP 3149 came from Debian. -- Nathaniel J. Smith -- https://vorpus.org
On 15 July 2017 at 14:41, Nathaniel Smith <njs@pobox.com> wrote:
_psutil_osx.cpython-35m-darwin.so
Right, but this only helps if you install multiple builds of the same package on top of each other. Which Debian is able to arrange, so PEP 3149 solves their problem. But the --user dir is managed by pip, and if you try to install a cpython-35dm package on top of a cpython-35m package, then pip will helpfully remove the first before installing the second. (And this is really the only reasonable thing for pip to do.) So in practice any given package in the --user dir can only support one SOABI variant, even though all the SOABI variants use the same dir.
Well, sort of - it's not intrinsic to *pip* or *Python* that this happens, it's an artifact of the fact that even though the compatibility tagging scheme allows for it, folks don't tend to publish "multi-SOABI" wheel files (where the 35dm and 35m binaries are in the same wheel), so installing one will uninstall the other. Rather than attempting to change how site-packages works (which wouldn't help until 3.7+ anyway), the main missing piece in the status quo is the equivalent of a "mergewheel" operation that creates those multi-SOABI files and thus allows both sets of binaries to be added to the same environment (the same way Linux distros do for the system site-packages directory). Cheers, Nick. P.S. Folks on Python 2.7 are pretty much stuck, though - while importlib2 and/or filefinder2 *might* be able to help, several of the pieces needed to fix this properly are Python 3 only. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 15, 2017, at 01:51, Nick Coghlan <ncoghlan@gmail.com> wrote:
Well, sort of - it's not intrinsic to *pip* or *Python* that this happens, it's an artifact of the fact that even though the compatibility tagging scheme allows for it, folks don't tend to publish "multi-SOABI" wheel files (where the 35dm and 35m binaries are in the same wheel), so installing one will uninstall the other.
Right, which is why I asserted it's a packager's problem.
Rather than attempting to change how site-packages works (which wouldn't help until 3.7+ anyway), the main missing piece in the status quo is the equivalent of a "mergewheel" operation that creates those multi-SOABI files and thus allows both sets of binaries to be added to the same environment (the same way Linux distros do for the system site-packages directory).
Without thinking too much about it, perhaps another option would be for packagers to provide two wheels: one containing all files needed for the non-debug case and a second wheel that only contains the debug extension modules (with their usual unique fila names) and that depends on the first wheel. In any case, I agree that this isn't something that has come up often in the past but perhaps some best practices should be documented in the packaging user guide. I added a comment to that effect in open issue #284: https://github.com/pypa/python-packaging-user-guide/issues/284 -- Ned Deily nad@python.org -- []
On 15 July 2017 at 16:00, Ned Deily <nad@python.org> wrote:
On Jul 15, 2017, at 01:51, Nick Coghlan <ncoghlan@gmail.com> wrote:
Rather than attempting to change how site-packages works (which wouldn't help until 3.7+ anyway), the main missing piece in the status quo is the equivalent of a "mergewheel" operation that creates those multi-SOABI files and thus allows both sets of binaries to be added to the same environment (the same way Linux distros do for the system site-packages directory).
Without thinking too much about it, perhaps another option would be for packagers to provide two wheels: one containing all files needed for the non-debug case and a second wheel that only contains the debug extension modules (with their usual unique fila names) and that depends on the first wheel.
This is essentially what Linux distros do - the debug packages are distinct from the regular ones, but they also have different names, so there's nothing keeping you from having both installed on a given system at the same time. However, there's a missing primitive for that in our case: we expect wheel archives to define independent package trees, with namespaces package being the only means of injecting your own files into another wheel's packages, so the debug and non-debug builds conflict by default. This means that for the specific problem of injecting debug versions of packages into a project, we're probably better off encouraging the use of package shadowing idioms that are more akin to the way LD_LIBRARY_PATH works than we are changing the way user site-packages works in general. Conveniently, we also already offer the key building block for that: virtual environments with --system-site-packages enabled. That is, if folks want to run the debug versions of their Python extensions without having to duplicate all their pure Python modules, we'd encourage them to: 1. Create a virtualenv with --system-site-packages and SOABI=cp35dm (for example) 2. Install the debug versions of any components with extension modules into that virtual environment (since SOABI=cp35m modules won't load) 3. Run the debug version from that venv One of the key benefits of encouraging that model is that it's general: it works for *any* case where you want to override the default version of a package, not just those where you want to run it against a different SOABI definition. We're missing a useful building block for *sharing* debug binaries between different venvs, but that's also a problem that's amenable to more general solutions (with the main one that comes to mind being a way to "chain" virtual environments, such that you can easily add a *.pth file to a venv that injects the site-packages for one of more other venvs into sys.path) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Jul 14, 2017 at 10:51 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 15 July 2017 at 14:41, Nathaniel Smith <njs@pobox.com> wrote:
_psutil_osx.cpython-35m-darwin.so
Right, but this only helps if you install multiple builds of the same package on top of each other. Which Debian is able to arrange, so PEP 3149 solves their problem. But the --user dir is managed by pip, and if you try to install a cpython-35dm package on top of a cpython-35m package, then pip will helpfully remove the first before installing the second. (And this is really the only reasonable thing for pip to do.) So in practice any given package in the --user dir can only support one SOABI variant, even though all the SOABI variants use the same dir.
Well, sort of - it's not intrinsic to *pip* or *Python* that this happens, it's an artifact of the fact that even though the compatibility tagging scheme allows for it, folks don't tend to publish "multi-SOABI" wheel files (where the 35dm and 35m binaries are in the same wheel), so installing one will uninstall the other.
I guess that's... sort of true? But it seems to me that the goal should be that Python shouldn't be finding broken packages on its search path, and AFAICT there are only two ways to accomplish that: - give Pythons with different SOABIs different search paths - mandate that *all* wheels be "multi-SOABI" wheel files that include every possible SOABI Installing a single-SOABI wheel into a multi-SOABI search path is just asking for trouble. Also AFAICT PyPy also shares the same --user path as the corresponding CPython release, so this isn't just restricted to debug builds of Python. -n -- Nathaniel J. Smith -- https://vorpus.org
On 15 July 2017 at 17:44, Nathaniel Smith <njs@pobox.com> wrote:
On Fri, Jul 14, 2017 at 10:51 PM, Nick Coghlan <ncoghlan@gmail.com> wrote: I guess that's... sort of true? But it seems to me that the goal should be that Python shouldn't be finding broken packages on its search path, and AFAICT there are only two ways to accomplish that:
- give Pythons with different SOABIs different search paths - mandate that *all* wheels be "multi-SOABI" wheel files that include every possible SOABI
Installing a single-SOABI wheel into a multi-SOABI search path is just asking for trouble.
Not really - it just means that that particular package will break if the SOABI doesn't match. The only way to avoid that outcome is to run a fully isolated virtualenv that can't see *either* of the system site-packages directories and hence has its own copy of everything it needs.
Also AFAICT PyPy also shares the same --user path as the corresponding CPython release, so this isn't just restricted to debug builds of Python.
Sure, and that's why __pycache__ works the way it does. So Ned's probably right that we need to provide written guidance that single-SOABI wheels implicitly assume the use of virtual environments to handle parallel installation, so if you want to fully support system and user level installations (rather than leaving that problem to redistributors), you'll need multi-SOABI wheel files that include all the necessary binaries in one archive. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 15, 2017, at 3:44 AM, Nathaniel Smith <njs@pobox.com> wrote:
- give Pythons with different SOABIs different search paths - mandate that *all* wheels be "multi-SOABI" wheel files that include every possible SOABI
Installing a single-SOABI wheel into a multi-SOABI search path is just asking for trouble.
It seems like the first of the options is a discussion for python-dev not for distutils-sig, since we don’t have any control over what Python itself does on this list. The second option seems like a non-starter to me, I don’t think we can reasonably expect every single package to publish a multi SOABI wheel file that includes every possible SOABI (particularly if other Pythons are reusing the same directory— which they shouldn’t be IMO). Better tooling for multi-SOABI seems like a good thing in general though. — Donald Stufft
On 16 July 2017 at 04:39, Donald Stufft <donald@stufft.io> wrote:
It seems like the first of the options is a discussion for python-dev not for distutils-sig, since we don’t have any control over what Python itself does on this list. The second option seems like a non-starter to me, I don’t think we can reasonably expect every single package to publish a multi SOABI wheel file that includes every possible SOABI (particularly if other Pythons are reusing the same directory— which they shouldn’t be IMO).
Better tooling for multi-SOABI seems like a good thing in general though.
A comment on the pip "default to --user" issue reminded me that PYTHONUSERBASE does exist as an environment variable: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUSERBASE I'd genuinely forgotten about that, but it seems to me that there could be some interesting potential use cases for that in combination with suitably populated virtual environments. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (4)
-
Donald Stufft -
Nathaniel Smith -
Ned Deily -
Nick Coghlan