Packages that have problems being installed from wheels
tl; dr; If you know of a project that can't be successfully installed with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let me know the details? One of the longer-term goals of the introduction of wheels was to split the build and install steps for a package - specifically, in pip, to restrict "pip install" to installing from wheels, and when no wheel is available, to (transparently) run "pip wheel" on the sdist followed by "pip install on the generated wheel". During a discussion yesterday, I realised that I don't know how close we are to that goal. So I'm looking for information on packages which don't install properly from wheels at the moment. If anyone has examples of packages where replacing "pip install foo" with "pip wheel -w /tmp/xxx; pip install /tmp/xxx/*.whl" does not result in an equivalent install, could they post them here with details of how & why they fail? Things I already know about (but would like specific examples): 1. Post-install steps included in setup.py. That should be covered by the support in Metadata 2.0. I'd also be interested in how much of an issue omitting the postinstall would be in practice (for instance many such steps just set up "Start Menu" type shortcuts, which are not essential for the package to be usable). 2. Actually, that's the only one :-) Things that should not be a problem (but might be): 1. Numpy (and the scipy stack) need better tagging facilities for wheels - but that wouldn't matter for a wheel that's built, used, then thrown away. 2. Some things are complex to build - but I don't know of any cases where building a wheel is *more* complex than installing, and I don't see how it could be, in theory. 3. Projects that customise setup.py so much that they aren't compatible with setuptools and bdist_wheel. Such projects are quite probably already incompatible with pip (which injects setuptools when running setup.py anyway) and so not relevant for this discussion. But if any do work with "pip install" but not "pip wheel", I'd like to hear about them. Thanks in advance for any information. Paul
On Aug 27, 2014, at 2:50 AM, Paul Moore <p.f.moore@gmail.com> wrote:
tl; dr; If you know of a project that can't be successfully installed with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let me know the details?
One of the longer-term goals of the introduction of wheels was to split the build and install steps for a package - specifically, in pip, to restrict "pip install" to installing from wheels, and when no wheel is available, to (transparently) run "pip wheel" on the sdist followed by "pip install on the generated wheel". During a discussion yesterday, I realised that I don't know how close we are to that goal. So I'm looking for information on packages which don't install properly from wheels at the moment. If anyone has examples of packages where replacing "pip install foo" with "pip wheel -w /tmp/xxx; pip install /tmp/xxx/*.whl" does not result in an equivalent install, could they post them here with details of how & why they fail?
Things I already know about (but would like specific examples):
1. Post-install steps included in setup.py. That should be covered by the support in Metadata 2.0. I'd also be interested in how much of an issue omitting the postinstall would be in practice (for instance many such steps just set up "Start Menu" type shortcuts, which are not essential for the package to be usable). 2. Actually, that's the only one :-)
Things that should not be a problem (but might be):
1. Numpy (and the scipy stack) need better tagging facilities for wheels - but that wouldn't matter for a wheel that's built, used, then thrown away. 2. Some things are complex to build - but I don't know of any cases where building a wheel is *more* complex than installing, and I don't see how it could be, in theory. 3. Projects that customise setup.py so much that they aren't compatible with setuptools and bdist_wheel. Such projects are quite probably already incompatible with pip (which injects setuptools when running setup.py anyway) and so not relevant for this discussion. But if any do work with "pip install" but not "pip wheel", I'd like to hear about them.
Thanks in advance for any information. Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Numpy (and other SciPy stack) has the problem too that it uses numpy.distutils so I'm not even sure if ``pip wheel`` actually works on it at all or not. They might have their own support for ``setup.py bdist_wheel`` though I don't know. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Scipy (the package) doesn't work due to the build requirement on numpy (and the lack of metadata stating this), however some of the others have included https://bitbucket.org/dholth/setup-requires (or something equivalent) to install using numpy. Effectively, if the package can be pip installed, then wheeling should work (I created a script a while ago which uses distlib to modify scipy wheels such that they contain the correct metadata, and I haven't run into any issues). Matplotlib works, however some of the graphics toolkits aren't wheelable (e.g. pygtk and pyqt). One case that I know of where wheels do not replace a straight install is PySide (there needs to be rpath modification), however the PySide developers are aware of this and in their installation instructions they explicitly mention to run a post install script if installing from a wheel (The post install probably could be got rid of if $ORIGIN was used, but that would involve digging into the cmake install, and checking that it doesn't break anything. Also, I'm not sure how widely $ORIGIN is supported). I'd guess that any package which creates shared libraries which link to each other would have this problem also, though how many of those are pip installable is probably almost none. James On 27 August 2014 17:29, Donald Stufft <donald@stufft.io> wrote:
On Aug 27, 2014, at 2:50 AM, Paul Moore <p.f.moore@gmail.com> wrote:
tl; dr; If you know of a project that can't be successfully installed with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let me know the details?
One of the longer-term goals of the introduction of wheels was to split the build and install steps for a package - specifically, in pip, to restrict "pip install" to installing from wheels, and when no wheel is available, to (transparently) run "pip wheel" on the sdist followed by "pip install on the generated wheel". During a discussion yesterday, I realised that I don't know how close we are to that goal. So I'm looking for information on packages which don't install properly from wheels at the moment. If anyone has examples of packages where replacing "pip install foo" with "pip wheel -w /tmp/xxx; pip install /tmp/xxx/*.whl" does not result in an equivalent install, could they post them here with details of how & why they fail?
Things I already know about (but would like specific examples):
1. Post-install steps included in setup.py. That should be covered by the support in Metadata 2.0. I'd also be interested in how much of an issue omitting the postinstall would be in practice (for instance many such steps just set up "Start Menu" type shortcuts, which are not essential for the package to be usable). 2. Actually, that's the only one :-)
Things that should not be a problem (but might be):
1. Numpy (and the scipy stack) need better tagging facilities for wheels - but that wouldn't matter for a wheel that's built, used, then thrown away. 2. Some things are complex to build - but I don't know of any cases where building a wheel is *more* complex than installing, and I don't see how it could be, in theory. 3. Projects that customise setup.py so much that they aren't compatible with setuptools and bdist_wheel. Such projects are quite probably already incompatible with pip (which injects setuptools when running setup.py anyway) and so not relevant for this discussion. But if any do work with "pip install" but not "pip wheel", I'd like to hear about them.
Thanks in advance for any information. Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Numpy (and other SciPy stack) has the problem too that it uses numpy.distutils so I'm not even sure if ``pip wheel`` actually works on it at all or not. They might have their own support for ``setup.py bdist_wheel`` though I don't know.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Don't send me files in proprietary formats (.doc(x), .xls, .ppt etc.). It isn't good enough for Tim Berners-Lee <http://opendotdotdot.blogspot.com/2010/04/rms-and-tim-berners-lee-separated-at.html>, and it isn't good enough for me either. For more information visit http://www.gnu.org/philosophy/no-word-attachments.html. Truly great madness cannot be achieved without significant intelligence. - Henrik Tikkanen If you're not messing with your sanity, you're not having fun. - James Tocknell In theory, there is no difference between theory and practice; In practice, there is.
Tell me more about setup-requires. It's nice to hear it has users. Should we promote it to a pypa project? On Aug 27, 2014 7:45 AM, "James Tocknell" <aragilar@gmail.com> wrote:
Scipy (the package) doesn't work due to the build requirement on numpy (and the lack of metadata stating this), however some of the others have included https://bitbucket.org/dholth/setup-requires (or something equivalent) to install using numpy. Effectively, if the package can be pip installed, then wheeling should work (I created a script a while ago which uses distlib to modify scipy wheels such that they contain the correct metadata, and I haven't run into any issues). Matplotlib works, however some of the graphics toolkits aren't wheelable (e.g. pygtk and pyqt).
One case that I know of where wheels do not replace a straight install is PySide (there needs to be rpath modification), however the PySide developers are aware of this and in their installation instructions they explicitly mention to run a post install script if installing from a wheel (The post install probably could be got rid of if $ORIGIN was used, but that would involve digging into the cmake install, and checking that it doesn't break anything. Also, I'm not sure how widely $ORIGIN is supported). I'd guess that any package which creates shared libraries which link to each other would have this problem also, though how many of those are pip installable is probably almost none.
James
On 27 August 2014 17:29, Donald Stufft <donald@stufft.io> wrote:
On Aug 27, 2014, at 2:50 AM, Paul Moore <p.f.moore@gmail.com> wrote:
tl; dr; If you know of a project that can't be successfully installed with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let me know the details?
One of the longer-term goals of the introduction of wheels was to split the build and install steps for a package - specifically, in pip, to restrict "pip install" to installing from wheels, and when no wheel is available, to (transparently) run "pip wheel" on the sdist followed by "pip install on the generated wheel". During a discussion yesterday, I realised that I don't know how close we are to that goal. So I'm looking for information on packages which don't install properly from wheels at the moment. If anyone has examples of packages where replacing "pip install foo" with "pip wheel -w /tmp/xxx; pip install /tmp/xxx/*.whl" does not result in an equivalent install, could they post them here with details of how & why they fail?
Things I already know about (but would like specific examples):
1. Post-install steps included in setup.py. That should be covered by the support in Metadata 2.0. I'd also be interested in how much of an issue omitting the postinstall would be in practice (for instance many such steps just set up "Start Menu" type shortcuts, which are not essential for the package to be usable). 2. Actually, that's the only one :-)
Things that should not be a problem (but might be):
1. Numpy (and the scipy stack) need better tagging facilities for wheels - but that wouldn't matter for a wheel that's built, used, then thrown away. 2. Some things are complex to build - but I don't know of any cases where building a wheel is *more* complex than installing, and I don't see how it could be, in theory. 3. Projects that customise setup.py so much that they aren't compatible with setuptools and bdist_wheel. Such projects are quite probably already incompatible with pip (which injects setuptools when running setup.py anyway) and so not relevant for this discussion. But if any do work with "pip install" but not "pip wheel", I'd like to hear about them.
Thanks in advance for any information. Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Numpy (and other SciPy stack) has the problem too that it uses numpy.distutils so I'm not even sure if ``pip wheel`` actually works on it at all or not. They might have their own support for ``setup.py bdist_wheel`` though I don't know.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Don't send me files in proprietary formats (.doc(x), .xls, .ppt etc.). It isn't good enough for Tim Berners-Lee <http://opendotdotdot.blogspot.com/2010/04/rms-and-tim-berners-lee-separated-at.html>, and it isn't good enough for me either. For more information visit http://www.gnu.org/philosophy/no-word-attachments.html.
Truly great madness cannot be achieved without significant intelligence. - Henrik Tikkanen
If you're not messing with your sanity, you're not having fun. - James Tocknell
In theory, there is no difference between theory and practice; In practice, there is.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 28 Aug 2014 05:56, "Daniel Holth" <dholth@gmail.com> wrote:
Tell me more about setup-requires. It's nice to hear it has users. Should
we promote it to a pypa project? That would be cool - bootstrapping as much as we can *without* metadata 2.0 has the virtue of working in many more environments :) Cheers, Nick.
1. Post-install steps included in setup.py. That should be covered by the support in Metadata 2.0. I'd also be interested in how much of an issue omitting the postinstall would be in practice (for instance many such steps just set up "Start Menu" type shortcuts, which are not essential for the package to be usable).
On windows, we use wheel convert to convert binary installers like pywin32 to wheels. But the post-install step in the setup is mandatory for pywin32 to work (it is not a mere Start Menu customization). So this is what we do: wheel convert pywin32-217.win32-py2.7.exe pip install pywin32-217-cp27-none-win32.whl In this case pywin32_postinstall.py is not run and we need to run it manually. I do not know the specifics, but pywin32 predates wheels, I don't know if it can be made to be installed so simply. D.
On 27 August 2014 16:42, David Genest <david.genest@ubisoft.com> wrote:
On windows, we use wheel convert to convert binary installers like pywin32 to wheels. But the post-install step in the setup is mandatory for pywin32 to work (it is not a mere Start Menu customization).
For pywin32, my understanding is that (ignoring start menu stuff) the postinstall registers the DLLs for COM and things like that. Which is essential, but only for the system installation (you don't want every virtualenv registering as "the" Python COM server). So pywin32 would need work to make it virtualenv-friendly anyway, and while that's being done, it can probably be made wheel-friendly too. For now, though, my experience is that using converted wheels in virtualenvs works fine as long as you don't write services or COM servers. But anyway, pywin32 isn't hosted anywhere pip can find, so "pip install" doesn't work from sdist either. Thanks for the information, nevertheless. Paul
On 27 August 2014 07:50, Paul Moore <p.f.moore@gmail.com> wrote:
tl; dr; If you know of a project that can't be successfully installed with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let me know the details?
I should have added: but can be installed from source with "pip install proj" Paul
There are cases where projects do this in setup.py: for scheme in INSTALL_SCHEMES.values(): scheme['data'] = scheme['purelib'] e.g. https://github.com/celery/django-celery/blob/master/setup.py#L80 so for the sdist install, data ends up relative to site-packages, whereas for the whl install, it ends up relative to sys.prefix On Tue, Aug 26, 2014 at 11:50 PM, Paul Moore <p.f.moore@gmail.com> wrote:
tl; dr; If you know of a project that can't be successfully installed with "pip wheel proj; pip install /path/to/the/wheel.whl" can you let me know the details?
One of the longer-term goals of the introduction of wheels was to split the build and install steps for a package - specifically, in pip, to restrict "pip install" to installing from wheels, and when no wheel is available, to (transparently) run "pip wheel" on the sdist followed by "pip install on the generated wheel". During a discussion yesterday, I realised that I don't know how close we are to that goal. So I'm looking for information on packages which don't install properly from wheels at the moment. If anyone has examples of packages where replacing "pip install foo" with "pip wheel -w /tmp/xxx; pip install /tmp/xxx/*.whl" does not result in an equivalent install, could they post them here with details of how & why they fail?
Things I already know about (but would like specific examples):
1. Post-install steps included in setup.py. That should be covered by the support in Metadata 2.0. I'd also be interested in how much of an issue omitting the postinstall would be in practice (for instance many such steps just set up "Start Menu" type shortcuts, which are not essential for the package to be usable). 2. Actually, that's the only one :-)
Things that should not be a problem (but might be):
1. Numpy (and the scipy stack) need better tagging facilities for wheels - but that wouldn't matter for a wheel that's built, used, then thrown away. 2. Some things are complex to build - but I don't know of any cases where building a wheel is *more* complex than installing, and I don't see how it could be, in theory. 3. Projects that customise setup.py so much that they aren't compatible with setuptools and bdist_wheel. Such projects are quite probably already incompatible with pip (which injects setuptools when running setup.py anyway) and so not relevant for this discussion. But if any do work with "pip install" but not "pip wheel", I'd like to hear about them.
Thanks in advance for any information. Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Aug 27, 2014, at 2:09 PM, Marcus Smith <qwcode@gmail.com> wrote:
There are cases where projects do this in setup.py:
for scheme in INSTALL_SCHEMES.values(): scheme['data'] = scheme['purelib']
e.g. https://github.com/celery/django-celery/blob/master/setup.py#L80 <https://github.com/celery/django-celery/blob/master/setup.py#L80>
so for the sdist install, data ends up relative to site-packages, whereas for the whl install, it ends up relative to sys.prefix
Oh right I forgot about that. Older versions of Django did it (not newer versions) and some stuff in the django ecosystem copied it. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
participants (7)
-
Daniel Holth -
David Genest -
Donald Stufft -
James Tocknell -
Marcus Smith -
Nick Coghlan -
Paul Moore