local platform tags and pip wheel caching
So the pip wheel caching feature has promoted a few latent problems
into the limelight.
One is that many wheels projects built are not actually as compatible
as thought. https://github.com/pypa/pip/issues/2882
One is that wheels don't support writing datafiles outside their
environment. https://github.com/pypa/pip/issues/2874
And the one I want to talk about is that we don't generate
sufficiently good platform markers on Linux.
https://github.com/pypa/pip/issues/2875
That bug describes the scenario of using a wheel cache shared across
multiple Linux ABIs - e.g. LXC containers with a bind mounted home
dir, or NFS on a cluster. pip 7's implicit wheel cache triggers this
situation for people.
The OP has suggested that being able to control the platform tag that
pip a) looks for and b) instructs wheel to use would let them avoid
issues in this situation without having to solve the general problem
of 'the universe of Linuxs can't agree on anything'.
e.g, in each context they could set a /etc/pip.conf containing
[global]
wheel_platform = contextname
and then pip would use that and instruct built wheels to use that.
Further, they not that it can be used to e.g. describe other local
policies like 'I want hardened C builds' that are not inferrable from
context at all today.
But since this affects pip *and* wheel, we're bringing it here for discussion.
I'm in favour of it - it fits the existing wheel spec; its opt-in,
non-intrusive and solves a real, present problem. [just not the 'how
do we get these damn things into PyPI, yet].
-Rob
--
Robert Collins
Makes sense to me. Could be an environment variable or bdist_wheel option
or both.
On Tue, Jun 9, 2015 at 3:10 PM Robert Collins
So the pip wheel caching feature has promoted a few latent problems into the limelight.
One is that many wheels projects built are not actually as compatible as thought. https://github.com/pypa/pip/issues/2882 One is that wheels don't support writing datafiles outside their environment. https://github.com/pypa/pip/issues/2874 And the one I want to talk about is that we don't generate sufficiently good platform markers on Linux. https://github.com/pypa/pip/issues/2875
That bug describes the scenario of using a wheel cache shared across multiple Linux ABIs - e.g. LXC containers with a bind mounted home dir, or NFS on a cluster. pip 7's implicit wheel cache triggers this situation for people.
The OP has suggested that being able to control the platform tag that pip a) looks for and b) instructs wheel to use would let them avoid issues in this situation without having to solve the general problem of 'the universe of Linuxs can't agree on anything'.
e.g, in each context they could set a /etc/pip.conf containing [global] wheel_platform = contextname
and then pip would use that and instruct built wheels to use that.
Further, they not that it can be used to e.g. describe other local policies like 'I want hardened C builds' that are not inferrable from context at all today.
But since this affects pip *and* wheel, we're bringing it here for discussion.
I'm in favour of it - it fits the existing wheel spec; its opt-in, non-intrusive and solves a real, present problem. [just not the 'how do we get these damn things into PyPI, yet].
-Rob
-- Robert Collins
Distinguished Technologist HP Converged Cloud _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 9 June 2015 at 20:09, Robert Collins
I'm in favour of it - it fits the existing wheel spec; its opt-in, non-intrusive and solves a real, present problem. [just not the 'how do we get these damn things into PyPI, yet].
Agreed, it sounds like a reasonable approach. One thought - rather than (ab)using the platform tag, maybe an additional "variant" tag could be used (empty by default)? OTOH, I don't see an obvious disadvantage to just using a custom platform tag. Paul
On 10 June 2015 at 07:27, Paul Moore
On 9 June 2015 at 20:09, Robert Collins
wrote: I'm in favour of it - it fits the existing wheel spec; its opt-in, non-intrusive and solves a real, present problem. [just not the 'how do we get these damn things into PyPI, yet].
Agreed, it sounds like a reasonable approach.
One thought - rather than (ab)using the platform tag, maybe an additional "variant" tag could be used (empty by default)? OTOH, I don't see an obvious disadvantage to just using a custom platform tag.
I don't see a space for that in PEP 427; so that might be an option in
future but not compatible with wheel 1.0 AIUI.
-Rob
--
Robert Collins
On 9 June 2015 at 21:35, Robert Collins
I don't see a space for that in PEP 427; so that might be an option in future but not compatible with wheel 1.0 AIUI.
Good point. We can revisit this for Wheel 2.0, but practicality beats purity for now so I'm OK with using the platform tag on that basis. Paul
The platform tag is quite pure enough for this use case. On Tue, Jun 9, 2015, 4:50 PM Paul Moore
On 9 June 2015 at 21:35, Robert Collins
wrote: I don't see a space for that in PEP 427; so that might be an option in future but not compatible with wheel 1.0 AIUI.
Good point. We can revisit this for Wheel 2.0, but practicality beats purity for now so I'm OK with using the platform tag on that basis. Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 10 June 2015 at 10:05, Daniel Holth
The platform tag is quite pure enough for this use case.
Aye, +1 from me as well. Overriding the platform tag entirely also has the benefit of letting you categorically prevent the use of wheels from PyPI when you want to force local from-source builds. I've also been pondering this question a fair bit in the context of wanting to be able to define "RHEL 7 + derivatives" as a platform, and between not only full distro derivatives like CentOS and OEL, but also internal variations like RHEL Atomic, Red Hat Storage and RHEL-for-ARM, trying to *derive* the ABI from OS level info in a generic way gets close to impossible. An explicit platform override would make it possible to simply adopt "EPEL7" as the platform tag for such wheel files, since they'd be the Python level equivalent of the pre-built EPEL RPMs [1] Cheers, Nick. [1] https://fedoraproject.org/wiki/EPEL -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (4)
-
Daniel Holth
-
Nick Coghlan
-
Paul Moore
-
Robert Collins