additional paths in wheel
It's always been obvious that wheel would probably need additional paths besides the sysconfig ones, and there's been some discussion here recently. For the next version we should: 1. Add the autoconf dirs. https://www.gnu.org/prep/standards/html_node/Directory-Variables.html. "packagename-1.0/data/dvidir/" or any of the other autoconf paths would be valid, in addition to the existing distutils paths. (The autoconf paths are defined relative to a $prefix, in Python's case $prefix is usually the base of the virtualenv). 2. Replace WHEEL with wheel.json. wheel.json contains all the information from WHEEL but is json which is rather popular these days. wheel.json may contain custom paths with string Template() interpolation. { "paths": "name":"$prefix/mypath", "othername":"$bindir/etc", "thirdname": "$othername/subfolder" } (the sysconfig names, autoconf names, and custom path names can be interpolated here) 3. The extra paths are not very useful if you can't find them at runtime. Provide a mechanism for recording the actual paths used during installation (in wheel.json). The installer should record the installation scheme only when requested, in one or more of the formats a) an importable Python file b) a .json file somewhere in the installation c) a file inside the .dist-info directory itself. Of course this is only useful if you have a build system that would actually produce wheels with files under these extra paths. I don't have that bit.
Provide a mechanism for recording the actual paths used during installation (in wheel.json).
Well, there is already a list of paths used which is used during uninstallation (RECORD). It would make sense to record all the installed files there, and no reason to duplicate that elsewhere. In general, IMO importable (executable) Python files should be avoided for this kind of use case - JSON files in the .dist-info would be better. We see from the .pth file and setup.py examples that executable Python for these types of usages can be a bit of a double-edged sword. Possibly we should replace RECORD, etc. with JSON equivalents (to be neat and tidy). Regards, Vinay Sajip
On Sep 4, 2014, at 1:58 PM, Daniel Holth
wrote: It's always been obvious that wheel would probably need additional paths besides the sysconfig ones, and there's been some discussion here recently.
For the next version we should:
The next version of the Wheel spec?
1. Add the autoconf dirs. https://www.gnu.org/prep/standards/html_node/Directory-Variables.html. "packagename-1.0/data/dvidir/" or any of the other autoconf paths would be valid, in addition to the existing distutils paths. (The autoconf paths are defined relative to a $prefix, in Python's case $prefix is usually the base of the virtualenv).
Sounds plausible.
2. Replace WHEEL with wheel.json. wheel.json contains all the information from WHEEL but is json which is rather popular these days.
wheel.json may contain custom paths with string Template() interpolation.
{ "paths": "name":"$prefix/mypath", "othername":"$bindir/etc", "thirdname": "$othername/subfolder" }
(the sysconfig names, autoconf names, and custom path names can be interpolated here)
I don’t understand this paths stuff, what is it supposed to be doing? Also with JSON, the problem is the current tooling is now setup to handle a key: value WHEEL store, so we’ll need some sort of a migration path for old tools to know that this is a Wheel they can’t handle. It’s possible that it’s not worth it to do this. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Thu, Sep 4, 2014 at 2:55 PM, Donald Stufft
On Sep 4, 2014, at 1:58 PM, Daniel Holth
wrote: It's always been obvious that wheel would probably need additional paths besides the sysconfig ones, and there's been some discussion here recently.
For the next version we should:
The next version of the Wheel spec?
This would be Wheel 2. It would still include WHEEL with the file format version, older clients would be able to show the error. Why allow paths to be written in a .py? The .py would be purely declarative; it would allow a program to find its files without using any pkg_resources API. IMO it's important to allow programs to work without iterating over the metadata of all installed packages or without participating in the package management system at all. # possibly only the paths that were actually used... paths = { 'prefix':'/usr/local/', 'bindir':'/usr/local/bin' , ...} # OR PREFIX="/usr/local" BINDIR="/usr/local/bin" MANDIR="/usr/local/share/man/..."
1. Add the autoconf dirs. https://www.gnu.org/prep/standards/html_node/Directory-Variables.html. "packagename-1.0/data/dvidir/" or any of the other autoconf paths would be valid, in addition to the existing distutils paths. (The autoconf paths are defined relative to a $prefix, in Python's case $prefix is usually the base of the virtualenv).
Sounds plausible.
2. Replace WHEEL with wheel.json. wheel.json contains all the information from WHEEL but is json which is rather popular these days.
wheel.json may contain custom paths with string Template() interpolation.
{ "paths": "name":"$prefix/mypath", "othername":"$bindir/etc", "thirdname": "$othername/subfolder" }
(the sysconfig names, autoconf names, and custom path names can be interpolated here)
I don’t understand this paths stuff, what is it supposed to be doing
Instead of just having predefined paths in the packagename-1.0.data/[path] directories, the user could define additional values for "path" in the {"path" : "installation directory"} dictionary used by wheel installers. (I'm not 100% sure this proposed feature does not cause more problems than it solves...)
Also with JSON, the problem is the current tooling is now setup to handle a key: value WHEEL store, so we’ll need some sort of a migration path for old tools to know that this is a Wheel they can’t handle. It’s possible that it’s not worth it to do this.
They would just contain a stub WHEEL advertising version 2.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
For example, a wheel has these directories:
pyramid-1.4.data/scripts/prequest
pyramid-1.4.data/scripts/pshell
With custom paths it could also have
pyramid-1.4.data/spam/pharmacy.txt
and "paths" : { "spam" : "$datadir/pyramid/junkmail" }
pharmacy.txt would be installed into $datadir/pyramid/junkmail/pharmacy.txt
It would also be appealing to be able to interpolate the package name
and version into the "paths" dictionary...
On Thu, Sep 4, 2014 at 4:04 PM, Daniel Holth
On Thu, Sep 4, 2014 at 2:55 PM, Donald Stufft
wrote: On Sep 4, 2014, at 1:58 PM, Daniel Holth
wrote: It's always been obvious that wheel would probably need additional paths besides the sysconfig ones, and there's been some discussion here recently.
For the next version we should:
The next version of the Wheel spec?
This would be Wheel 2.
It would still include WHEEL with the file format version, older clients would be able to show the error.
Why allow paths to be written in a .py? The .py would be purely declarative; it would allow a program to find its files without using any pkg_resources API. IMO it's important to allow programs to work without iterating over the metadata of all installed packages or without participating in the package management system at all.
# possibly only the paths that were actually used... paths = { 'prefix':'/usr/local/', 'bindir':'/usr/local/bin' , ...} # OR PREFIX="/usr/local" BINDIR="/usr/local/bin" MANDIR="/usr/local/share/man/..."
1. Add the autoconf dirs. https://www.gnu.org/prep/standards/html_node/Directory-Variables.html. "packagename-1.0/data/dvidir/" or any of the other autoconf paths would be valid, in addition to the existing distutils paths. (The autoconf paths are defined relative to a $prefix, in Python's case $prefix is usually the base of the virtualenv).
Sounds plausible.
2. Replace WHEEL with wheel.json. wheel.json contains all the information from WHEEL but is json which is rather popular these days.
wheel.json may contain custom paths with string Template() interpolation.
{ "paths": "name":"$prefix/mypath", "othername":"$bindir/etc", "thirdname": "$othername/subfolder" }
(the sysconfig names, autoconf names, and custom path names can be interpolated here)
I don’t understand this paths stuff, what is it supposed to be doing
Instead of just having predefined paths in the packagename-1.0.data/[path] directories, the user could define additional values for "path" in the {"path" : "installation directory"} dictionary used by wheel installers. (I'm not 100% sure this proposed feature does not cause more problems than it solves...)
Also with JSON, the problem is the current tooling is now setup to handle a key: value WHEEL store, so we’ll need some sort of a migration path for old tools to know that this is a Wheel they can’t handle. It’s possible that it’s not worth it to do this.
They would just contain a stub WHEEL advertising version 2.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 4 September 2014 18:58, Daniel Holth
For the next version we should:
1. Add the autoconf dirs.
OK. Can we make it clear in the documentation that these are non-portable when used in distributions installed in the system Python? And can we have some documentation on how code should be written to find a file that's installed in "$libdir/foo" so that the code works for a system installation and a virtualenv installation?
2. Replace WHEEL with wheel.json. wheel.json contains all the information from WHEEL but is json which is rather popular these days.
Formats parseable by stdlib tools is a good thing, IMO. I'm not sure it's worth switching to JSON just for the sake of doing so - we need format stability as well, because tools like pip will still have to parse the older formats for some time yet. But you mention extending the format below - that may be a good reason to switch to the more general JSON format at the same time.
wheel.json may contain custom paths with string Template() interpolation.
{ "paths": "name":"$prefix/mypath", "othername":"$bindir/etc", "thirdname": "$othername/subfolder" }
(the sysconfig names, autoconf names, and custom path names can be interpolated here)
Not 100% sure what you're intending here.
3. The extra paths are not very useful if you can't find them at runtime. Provide a mechanism for recording the actual paths used during installation (in wheel.json). The installer should record the installation scheme only when requested, in one or more of the formats a) an importable Python file b) a .json file somewhere in the installation c) a file inside the .dist-info directory itself.
This is basically much like RECORD, as Vinay says, and should be a file inside the .dist-info directory for that reason. I still wish there was a *really* lightweight way of getting dist-info files at runtime, distlib and setuptools cover far more ground, but getting at the extra paths needs nothing more than get_distinfo_file('mydist', 'paths.json'), and a runtime dependency on distlib/setuptools just for that seems like a lot.
Of course this is only useful if you have a build system that would actually produce wheels with files under these extra paths. I don't have that bit.
By this, do you mean that it's not possible to get distutils/setuptools/bdist_wheel to support this, or just that extra work is needed to add support to those tools? Paul
On 4 September 2014 21:09, Daniel Holth
For example, a wheel has these directories:
pyramid-1.4.data/scripts/prequest pyramid-1.4.data/scripts/pshell
With custom paths it could also have
pyramid-1.4.data/spam/pharmacy.txt
and "paths" : { "spam" : "$datadir/pyramid/junkmail" }
pharmacy.txt would be installed into $datadir/pyramid/junkmail/pharmacy.txt
It would also be appealing to be able to interpolate the package name and version into the "paths" dictionary...
Ah, I see what you're trying to achieve. No idea how useful it would be in practice, most projects seem to get away without it, but maybe they shouldn't have to. But based on this, it's critical IMO that there's good docs on how to find these files at runtime, otherwise projects will hard-code for the simple case (installed in the system Python on Unix - I imagine it'll be the Unix developers who will be the typical users of this feature) and end up breaking in all other environments... Paul
Instead of just having predefined paths in the packagename-1.0.data/[path] directories, the user could define additional values for "path" in the {"path" : "installation directory"} dictionary used by wheel installers. (I'm not 100% sure this proposed feature does not cause more problems than it solves...)
well, as it is now, if "data_files" contains absolute paths, wheel silently produces broken whl packages. i.e. it produces packages that install your data_files relative to sys.prefix (not as absolute) so the proposed feature of custom paths, would at least offer a way to un-break those packages. another option for right now (and maybe in the long term too) though is to refuse to ever build wheels for these packages in the first place (see discussion in https://bitbucket.org/pypa/wheel/issue/92/bdist_wheel-makes-absolute-data_fi... )
Only the installer cares about RECORD. The application knows it's looking
for doom1.wad and it just needs to know what the $wadfiles path was - it
just needs the prefix, which is all the new option writes. That is why I
don't propose using record for this. We will not allow every file in the
dist to be relocated relative to every other.
On Sep 4, 2014 2:43 PM, "Vinay Sajip"
Provide a mechanism for recording the actual paths used during installation (in wheel.json).
Well, there is already a list of paths used which is used during uninstallation (RECORD). It would make sense to record all the installed files there, and no reason to duplicate that elsewhere.
In general, IMO importable (executable) Python files should be avoided for this kind of use case - JSON files in the .dist-info would be better. We see from the .pth file and setup.py examples that executable Python for these types of usages can be a bit of a double-edged sword.
Possibly we should replace RECORD, etc. with JSON equivalents (to be neat and tidy).
Regards,
Vinay Sajip
participants (5)
-
Daniel Holth
-
Donald Stufft
-
Marcus Smith
-
Paul Moore
-
Vinay Sajip