Platform specific destinations in wheel files?
With the Python 3.4 feature freeze behind us, I've started looking at doing a new update of the draft metadata 2.0 docs. Vaguely related to that are the recent discussions about being able to publish an FHS compliant daemon like cobblerd as a wheel file - using Python specific formats to define the layout of full applications, not just libraries. Currently, all of the paths defined in sysconfig schemes aim to be platform neutral. However, this means that some platform specific concepts can't easily be handled by the wheel format - the "data" scheme can sort of handle it, but it isn't particularly intuitive, and doesn't inherently reflect the platform dependent nature of the layout being defined. I'd generally been resisting the idea of supporting this (since I favour interoperating with system packaging tools where appropriate over attempting to replace them entirely), but in this case I'm starting to think it may be necessary to support these layouts in the next version of the wheel format in order to *also* support automated conversion of upstream projects to policy compliant system packages. As a reminder, here are the currently defined Python specific directories relevant to wheel files: platlib: directory for site-specific, platform-specific files. purelib: directory for site-specific, non-platform-specific files. include: directory for non-platform-specific header files. platinclude: directory for platform-specific header files. And these are the generic directories that aren't necessarily Python specific: scripts: directory for script files. data: directory for data files. This is still only a half-baked idea at this point, but I'm currently leaning towards keeping the "<name>.data" sysconfig subdirectories in the wheel format cross platform (and somewhat Python specific), and adding a new "<name>.app" subdirectory in parallel. A wheel that had content in "app" would be inherently platform specific - you wouldn't be permitted to use the new directory in a cross-platform wheel file. The defined subdirectories of app would also be platform specific. All POSIX systems would at least support the "fhs" subdirectory. For a system install, this would map to "/", for a virtualenv install it would map to the root of the virtualenv and for a user install it would map to "~/.local". I'm not sure what other subdirectories would be appropriate for Windows or Mac OS X, although I expect being able to install into Program Files and Application Data would be interesting for Windows apps, and into an application folder for Mac OS X. It's really the potential for FHS support that drives my interest in the idea, but if we're going to do something like this at all, it shouldn't be POSIX specific. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 21 December 2013 10:57, Nick Coghlan <ncoghlan@gmail.com> wrote:
And these are the generic directories that aren't necessarily Python specific:
scripts: directory for script files. data: directory for data files.
It's worth noting that "data", although in essence a platform neutral term, is in practice not very well defined. Projects as a result use it inconsistently, and usually based on their expectations from the platforms they are used to. One thing I have seen data being used for a lot is documentation - and not consistently - some projects use <data>/man, others use <data>/share/man, and others <data>/<projectname>. I'd suggest the following changes in this area: 1. Add a "docs" directory to hold documentation. We may need to have a debate on how this should be structured, and how to handle backward compatibility, but this is a genuine need that the current scheme does not cover. 2. Add some more explicit wording on how "data" should (and should not) be used. For instance, having FHS-style structures like share/man/man1 under "data" doesn't really help with cross-platform compatibility that much - it just provides somewhere to dump the data on Windows while still interoperating with standard tools on a POSIX system.
This is still only a half-baked idea at this point, but I'm currently leaning towards keeping the "<name>.data" sysconfig subdirectories in the wheel format cross platform (and somewhat Python specific), and adding a new "<name>.app" subdirectory in parallel.
A wheel that had content in "app" would be inherently platform specific - you wouldn't be permitted to use the new directory in a cross-platform wheel file. The defined subdirectories of app would also be platform specific.
It might be worth having the possibility of optional parts of "app". To use the docs example again, it would be easy to see projects putting Unix manpages in app, and so unintentinally making the distribution platform specific, when in practice it would be perfectly OK to simply omit the manpages on Windows. Other app files (for example config files in etc) *will* be essential for proper operation, of course.
All POSIX systems would at least support the "fhs" subdirectory. For a system install, this would map to "/", for a virtualenv install it would map to the root of the virtualenv and for a user install it would map to "~/.local".
I'm not sure what other subdirectories would be appropriate for Windows or Mac OS X, although I expect being able to install into Program Files and Application Data would be interesting for Windows apps, and into an application folder for Mac OS X.
I'm not sure Program Files is that useful - very few apps would install into *both* Program Files and an application area (site-packages). But Application Data might be useful. Another possibility would be the start menu - but that's not precisely a filesystem location, you'd need to put metadata into the "Start Menu" area of the wheel, and use the Windows API to create the start menu items at install time.
It's really the potential for FHS support that drives my interest in the idea, but if we're going to do something like this at all, it shouldn't be POSIX specific.
Agreed in principle. But in practice, do you expect the wheel spec to define exactly how each scheme under "app" is mapped? And do you expect every tool to implement those features? That implies a lot of work getting consensus and a change to the wheel spec every time someone thinks of a new platform-specific area that's worth supporting. Maybe this is something that should be handled by the post-install hook support. The "app" area could be an area that is reserved in the wheel for platform-specific data as you say, but there is no specified behaviour for tools to implement. Rather: 1. Tools must make the "app" area accessible in post-install hooks (how is yet to be defined, simplest answer just put it in a temporary location on disk, and tell the hooks the location of that location) 2. Tools should provide an extensibility mechanism for installing system-wide processors for app data (so that we don't need every wheel to ship the FHS-handling post-install hooks) 3. Standard processing for particular app areas (for example, app/fhs) could be defined independently of the wheel spec. One of the motivations I have here is to keep the install behaviour from getting too complex. We need to remember that pip is not the only wheel installer (there's distlib and wheel itself). So I don't want to mandate too much complexity into the install phase (for example, script wrappers aren't handled in wheel install, and I'm not sure what distlib does - so there's implementation differences creeping in right now). Paul
On 21 December 2013 21:40, Paul Moore <p.f.moore@gmail.com> wrote:
On 21 December 2013 10:57, Nick Coghlan <ncoghlan@gmail.com> wrote:
And these are the generic directories that aren't necessarily Python specific:
scripts: directory for script files. data: directory for data files.
It's worth noting that "data", although in essence a platform neutral term, is in practice not very well defined. Projects as a result use it inconsistently, and usually based on their expectations from the platforms they are used to. One thing I have seen data being used for a lot is documentation - and not consistently - some projects use <data>/man, others use <data>/share/man, and others <data>/<projectname>.
I'd suggest the following changes in this area:
1. Add a "docs" directory to hold documentation. We may need to have a debate on how this should be structured, and how to handle backward compatibility, but this is a genuine need that the current scheme does not cover.
Evolution of sysconfig as we go forward is indeed something for us to think about. It may be a case where we need to actually decouple the wheel spec from the implicit dependency on those definitions. The stdlib related paths don't make sense to use in a wheel file, so what probably makes sense is for us to make just purelib, platlib, include and platinclude depend on sysconfig (since they're installing to Python-specific locations and need the assistance from the standard library), while other paths like scripts and docs would be defined independently of sysconfig.
2. Add some more explicit wording on how "data" should (and should not) be used. For instance, having FHS-style structures like share/man/man1 under "data" doesn't really help with cross-platform compatibility that much - it just provides somewhere to dump the data on Windows while still interoperating with standard tools on a POSIX system.
I'd be interesting in reading such guidance, because I really have no idea... (I'd never even noticed the option until Daniel pointed it out to me recently)
This is still only a half-baked idea at this point, but I'm currently leaning towards keeping the "<name>.data" sysconfig subdirectories in the wheel format cross platform (and somewhat Python specific), and adding a new "<name>.app" subdirectory in parallel.
A wheel that had content in "app" would be inherently platform specific - you wouldn't be permitted to use the new directory in a cross-platform wheel file. The defined subdirectories of app would also be platform specific.
It might be worth having the possibility of optional parts of "app". To use the docs example again, it would be easy to see projects putting Unix manpages in app, and so unintentinally making the distribution platform specific, when in practice it would be perfectly OK to simply omit the manpages on Windows. Other app files (for example config files in etc) *will* be essential for proper operation, of course.
I think a separate docs path would be a better solution for that. There may be a couple of others we decide to support directly (like an admin_scripts path that maps to sbin under FHS and to the same place that scripts maps to on Windows).
All POSIX systems would at least support the "fhs" subdirectory. For a system install, this would map to "/", for a virtualenv install it would map to the root of the virtualenv and for a user install it would map to "~/.local".
I'm not sure what other subdirectories would be appropriate for Windows or Mac OS X, although I expect being able to install into Program Files and Application Data would be interesting for Windows apps, and into an application folder for Mac OS X.
I'm not sure Program Files is that useful - very few apps would install into *both* Program Files and an application area (site-packages). But Application Data might be useful. Another possibility would be the start menu - but that's not precisely a filesystem location, you'd need to put metadata into the "Start Menu" area of the wheel, and use the Windows API to create the start menu items at install time.
It's really the potential for FHS support that drives my interest in the idea, but if we're going to do something like this at all, it shouldn't be POSIX specific.
Agreed in principle. But in practice, do you expect the wheel spec to define exactly how each scheme under "app" is mapped? And do you expect every tool to implement those features? That implies a lot of work getting consensus and a change to the wheel spec every time someone thinks of a new platform-specific area that's worth supporting.
Maybe this is something that should be handled by the post-install hook support. The "app" area could be an area that is reserved in the wheel for platform-specific data as you say, but there is no specified behaviour for tools to implement. Rather:
1. Tools must make the "app" area accessible in post-install hooks (how is yet to be defined, simplest answer just put it in a temporary location on disk, and tell the hooks the location of that location) 2. Tools should provide an extensibility mechanism for installing system-wide processors for app data (so that we don't need every wheel to ship the FHS-handling post-install hooks) 3. Standard processing for particular app areas (for example, app/fhs) could be defined independently of the wheel spec.
One of the motivations I have here is to keep the install behaviour from getting too complex. We need to remember that pip is not the only wheel installer (there's distlib and wheel itself). So I don't want to mandate too much complexity into the install phase (for example, script wrappers aren't handled in wheel install, and I'm not sure what distlib does - so there's implementation differences creeping in right now).
Oh, I like that idea. Let me get the next draft of the metadata 2.0 PEPs cleaned up and published, and then we can discuss it further (I think it ties into nicely with some of the changes I've made in response to PJE's feedback on the last draft). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 21 December 2013 22:01, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 21 December 2013 21:40, Paul Moore <p.f.moore@gmail.com> wrote:
On 21 December 2013 10:57, Nick Coghlan <ncoghlan@gmail.com> wrote:
It's really the potential for FHS support that drives my interest in the idea, but if we're going to do something like this at all, it shouldn't be POSIX specific.
Agreed in principle. But in practice, do you expect the wheel spec to define exactly how each scheme under "app" is mapped? And do you expect every tool to implement those features? That implies a lot of work getting consensus and a change to the wheel spec every time someone thinks of a new platform-specific area that's worth supporting.
Maybe this is something that should be handled by the post-install hook support. The "app" area could be an area that is reserved in the wheel for platform-specific data as you say, but there is no specified behaviour for tools to implement. Rather:
1. Tools must make the "app" area accessible in post-install hooks (how is yet to be defined, simplest answer just put it in a temporary location on disk, and tell the hooks the location of that location) 2. Tools should provide an extensibility mechanism for installing system-wide processors for app data (so that we don't need every wheel to ship the FHS-handling post-install hooks) 3. Standard processing for particular app areas (for example, app/fhs) could be defined independently of the wheel spec.
One of the motivations I have here is to keep the install behaviour from getting too complex. We need to remember that pip is not the only wheel installer (there's distlib and wheel itself). So I don't want to mandate too much complexity into the install phase (for example, script wrappers aren't handled in wheel install, and I'm not sure what distlib does - so there's implementation differences creeping in right now).
Oh, I like that idea. Let me get the next draft of the metadata 2.0 PEPs cleaned up and published, and then we can discuss it further (I think it ties into nicely with some of the changes I've made in response to PJE's feedback on the last draft).
The revised proposal for the hooks mechanism is now up as part of the new standard metadata extensions PEP: http://www.python.org/dev/peps/pep-0459/#the-metadata-hooks-extension This currently just covers step 2 - there's no mechanism for giving such hooks access to files without first putting them in one of the standard locations. It's the first draft of that design though, so I expect it will need a few iterations before we're happy with it. For example, I think a slight tweak to the metadata_hooks extension could extend it to cover this case as well: support a "custom_path" option as a new kind of hook definition. So, for example, you would be able to trap the fhs path this way: "python.metadata_hooks": { "custom_paths": { "fhs": { "postinstall": "fedora_pip.plugins:install_fhs_files", } } } In this case, the signature of the postinstall hook would be slightly different from the one defined in the PEP for extensions and export_groups, since it would also need to include the path to a temporary directory containing the extracted files. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Dec 21, 2013 at 2:57 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
compliant daemon like cobblerd as a wheel file - using Python specific formats to define the layout of full applications, not just libraries.
I'd generally been resisting the idea of supporting this (since I favour interoperating with system packaging tools where appropriate over attempting to replace them entirely), but in this case I'm starting to think it may be necessary to support these layouts in the next version of the wheel format in order to *also* support automated conversion of upstream projects to policy compliant system packages.
hmm I tend to think, like you, that this isn't a problem wheel should solve, but can also see the advantages...for the moment though talking about how it would solve it may help clarify whether it should. adding a new "<name>.app" subdirectory in parallel.
A wheel that had content in "app" would be inherently platform specific - you wouldn't be permitted to use the new directory in a cross-platform wheel file. The defined subdirectories of app would also be platform specific.
is this necessary -- couldn't there be a way to provide the info in a cross-platform way, and have it mapped to the platform specific locations at install-time?
All POSIX systems would at least support the "fhs" subdirectory. For a system install, this would map to "/", for a virtualenv install it would map to the root of the virtualenv and for a user install it would map to "~/.local".
then you explicitly put in "bin", sbin", "share", whatever? This seems really klunky to me, and also forces platform dependence, and is fundamentally tied to how posix does things.... Maybe it's not possible, but I suggest that we could pre-define the locations that might be needed: executables (bin) system_executables (sbin) user_executables (./bin) documentation (doc) system_data_files (share ?) user_data_files (./share ) app_configuration (/etc/appname) user_app_configuration (./.app_name : ./Library ) .... This could end up being a pretty big list, but I think it could be finite. Then at install-time, the installer maps these to the appropriate place on the system. It's a little different application, but wxWidgets does this pretty successfully with wxStandardPaths. -Chris
I'm not sure what other subdirectories would be appropriate for Windows or Mac OS X, although I expect being able to install into Program Files and Application Data would be interesting for Windows apps, and into an application folder for Mac OS X.
It's really the potential for FHS support that drives my interest in the idea, but if we're going to do something like this at all, it shouldn't be POSIX specific.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 23 December 2013 20:53, Chris Barker <chris.barker@noaa.gov> wrote:
then you explicitly put in "bin", sbin", "share", whatever?
This seems really klunky to me, and also forces platform dependence, and is fundamentally tied to how posix does things....
Maybe it's not possible, but I suggest that we could pre-define the locations that might be needed:
Agreed. My biggest concern with this whole idea is that developers (typically POSIX developers, but it applies equally to all) will *think* they need something like sbin because they are used to the concept from their environment, and so write their wheel to use it and hence be platform specific. However, with a little thought (possibly hardly any thought in the case of sbin :-)) they could have chosen a more generic approach which makes their project available to users of other platforms. This has been a very common issue historically, and I'm very reluctant to add anything that makes it easier to limit usability of packages across platforms. Portable by default should be the principle. But I see there is a need, and I understand that practicality beats purity. Let's just make sure that it's more effort to use the platform-specific approach than not to, so it doesn't become an attractive nuisance. BTW, I like your idea of defining generic locations - but in the past efforts to do this have been doomed to death by bikeshedding and platform wars. So while I agree in principle, I fear it may not happen in practice :-( Paul
On Mon, Dec 23, 2013 at 4:22 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 23 December 2013 20:53, Chris Barker <chris.barker@noaa.gov> wrote:
then you explicitly put in "bin", sbin", "share", whatever?
This seems really klunky to me, and also forces platform dependence, and is fundamentally tied to how posix does things....
Maybe it's not possible, but I suggest that we could pre-define the locations that might be needed:
Agreed. My biggest concern with this whole idea is that developers (typically POSIX developers, but it applies equally to all) will *think* they need something like sbin because they are used to the concept from their environment, and so write their wheel to use it and hence be platform specific. However, with a little thought (possibly hardly any thought in the case of sbin :-)) they could have chosen a more generic approach which makes their project available to users of other platforms.
This has been a very common issue historically, and I'm very reluctant to add anything that makes it easier to limit usability of packages across platforms. Portable by default should be the principle.
But I see there is a need, and I understand that practicality beats purity. Let's just make sure that it's more effort to use the platform-specific approach than not to, so it doesn't become an attractive nuisance.
"Attractive nuisance" is the first phrase that comes to mind when I think of the FHS.
BTW, I like your idea of defining generic locations - but in the past efforts to do this have been doomed to death by bikeshedding and platform wars. So while I agree in principle, I fear it may not happen in practice :-(
Paul
The GNU autoconf paths are the obvious choice. It would be really easy to try adding these to the dict that wheel installers use to relocate stuff. http://www.gnu.org/prep/standards/html_node/Directory-Variables.html
On Mon, Dec 23, 2013 at 1:43 PM, Daniel Holth > Agreed. My biggest concern with this whole idea is that developers
(typically POSIX developers, but it applies equally to all) will *think* they need something like sbin because they are used to the concept from their environment, and so write their wheel to use it and hence be platform specific.
exactly. However, with a little thought (possibly
hardly any thought in the case of sbin :-)) they could have chosen a more generic approach which makes their project available to users of other platforms.
right, but setting the system up to allow a prefix and then hard-specify paths under that makes it impossible to do it cross-platform!
Portable by default should be the principle.
+1 ... and at least possible (without writing multiple platform-specific versions...)! ANother thought is: who should control where things are put? a) the package developer? b) the python system maintainer? I think clearly the answer is (b). i.e. we'll have reasonable defaults in the python.org builds, but if Anaconda or Canopy, or a BSD or a linux distro, or Raspberry Pi, or Micro..... could all define their own paths for standard locations. I think this is much better than the package maintainers fixing that. The GNU autoconf paths are the obvious choice. Certainly a good palce to start. One issue -- that (and the FHS) is built around the idea of a "prefix", where you can shift the whole pile to /usr or /usr/local, or /opt, or??? But that concept doesn't work on all platforms, so we should be careful about isolating it. -Chris
It would be really easy to try adding these to the dict that wheel installers use to relocate stuff. http://www.gnu.org/prep/standards/html_node/Directory-Variables.html
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 25 Dec 2013 04:14, "Chris Barker" <chris.barker@noaa.gov> wrote:
On Mon, Dec 23, 2013 at 1:43 PM, Daniel Holth > Agreed. My biggest
(typically POSIX developers, but it applies equally to all) will *think* they need something like sbin because they are used to the concept from their environment, and so write their wheel to use it and hence be platform specific.
exactly.
However, with a little thought (possibly
hardly any thought in the case of sbin :-)) they could have chosen a more generic approach which makes their project available to users of other platforms.
right, but setting the system up to allow a prefix and then hard-specify
concern with this whole idea is that developers paths under that makes it impossible to do it cross-platform!
Portable by default should be the principle.
+1 ... and at least possible (without writing multiple platform-specific
versions...)!
ANother thought is:
who should control where things are put?
a) the package developer? b) the python system maintainer?
I think clearly the answer is (b). i.e. we'll have reasonable defaults in
the python.org builds, but if Anaconda or Canopy, or a BSD or a linux distro, or Raspberry Pi, or Micro..... could all define their own paths for standard locations. I think this is much better than the package maintainers fixing that.
The GNU autoconf paths are the obvious choice.
Certainly a good palce to start.
One issue -- that (and the FHS) is built around the idea of a "prefix",
where you can shift the whole pile to /usr or /usr/local, or /opt, or???
But that concept doesn't work on all platforms, so we should be careful
about isolating it. Encapsulating that assumption is why I think the "gnu" nesting is justified. There are layout expectations inherent in the autoconf directory design that just don't make sense on Windows, so any package expecting them is going to be a bit quirky if installed on Windows. Cheers, Nick.
-Chris
It would be really easy to try adding these to the dict that wheel installers use to relocate stuff.
http://www.gnu.org/prep/standards/html_node/Directory-Variables.html
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Tue, Dec 24, 2013 at 2:28 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
But that concept doesn't work on all platforms, so we should be careful about isolating it.
Encapsulating that assumption is why I think the "gnu" nesting is justified. There are layout expectations inherent in the autoconf directory design that just don't make sense on Windows, so any package expecting them is going to be a bit quirky if installed on Windows.
I'm confused now as to what has been proposed or being discussed, or... I _thought_ that this thread started with a proposal that package authors would do something like specifying a file hierarchy for the stuff they are delivering: /bin/some_scripts /share/some_data /man/some_docs .... then at installation time, the python distro would decide where to copy all that. But this would beb worthless on non-gnu systems and would require: 1) package writer to write three or four versions 2) wheels to be platform-dependent unnecessarily. So my suggestion was to define the various locations where stuff may need to be installed at a higher level: place_to_put_top_level_scripts place_to_put_documentation place_to_app_data place_to_put_user_configuration_files place_to_put_user_system_configuration_files ..... Then the python distro would map these to actual paths at install time: gnu systems would map the gnu locations, Windows to Windows-appropriate locations, OS-X to OS-X locations, etc.... This wold also allow python distros like Anaconda or macports python, or ??? to do their own thing, which may not be putting everything in /usr/local, or ... That may be what you had in mind, but I got confused. -Chris
Cheers, Nick.
-Chris
It would be really easy to try adding these to the dict that wheel installers use to relocate stuff.
http://www.gnu.org/prep/standards/html_node/Directory-Variables.html
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 28 December 2013 06:02, Chris Barker <chris.barker@noaa.gov> wrote:
On Tue, Dec 24, 2013 at 2:28 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
But that concept doesn't work on all platforms, so we should be careful about isolating it.
Encapsulating that assumption is why I think the "gnu" nesting is justified. There are layout expectations inherent in the autoconf directory design that just don't make sense on Windows, so any package expecting them is going to be a bit quirky if installed on Windows.
I'm confused now as to what has been proposed or being discussed, or...
I _thought_ that this thread started with a proposal that package authors would do something like specifying a file hierarchy for the stuff they are delivering:
/bin/some_scripts /share/some_data /man/some_docs ....
then at installation time, the python distro would decide where to copy all that.
But this would beb worthless on non-gnu systems and would require: 1) package writer to write three or four versions 2) wheels to be platform-dependent unnecessarily.
So my suggestion was to define the various locations where stuff may need to be installed at a higher level:
place_to_put_top_level_scripts place_to_put_documentation place_to_app_data place_to_put_user_configuration_files place_to_put_user_system_configuration_files .....
Then the python distro would map these to actual paths at install time: gnu systems would map the gnu locations, Windows to Windows-appropriate locations, OS-X to OS-X locations, etc.... This wold also allow python distros like Anaconda or macports python, or ??? to do their own thing, which may not be putting everything in /usr/local, or ...
That may be what you had in mind, but I got confused.
There's no concrete proposal on the table at this point in time - just an acknowledgement that this is a use case the wheel format doesn't yet handle (sdists can handle it due to the way that setuptools handles the data_files option). However, any concrete proposal will likely use the GNU structure as the basis for "kinds of files" rather than inventing a new Python-specific naming scheme (or make an extremely compelling case for inventing our own custom scheme), and on Windows and Mac OS X, many of those will likely just map to dumping things in the app directory (or the application data directory in the case of Windows). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 28 December 2013 16:00, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 28 December 2013 06:02, Chris Barker <chris.barker@noaa.gov> wrote:
Then the python distro would map these to actual paths at install time: gnu systems would map the gnu locations, Windows to Windows-appropriate locations, OS-X to OS-X locations, etc.... This wold also allow python distros like Anaconda or macports python, or ??? to do their own thing, which may not be putting everything in /usr/local, or ...
That may be what you had in mind, but I got confused.
There's no concrete proposal on the table at this point in time - just an acknowledgement that this is a use case the wheel format doesn't yet handle (sdists can handle it due to the way that setuptools handles the data_files option).
However, any concrete proposal will likely use the GNU structure as the basis for "kinds of files" rather than inventing a new Python-specific naming scheme (or make an extremely compelling case for inventing our own custom scheme), and on Windows and Mac OS X, many of those will likely just map to dumping things in the app directory (or the application data directory in the case of Windows).
The other main thing to look at in terms of current state of the art is npm, for both structure and hooks. https://npmjs.org/doc/files/npm-folders.html https://npmjs.org/doc/files/package.json.html https://npmjs.org/doc/misc/npm-scripts.html npm is basically a full-blown platform packaging system (covering config, runtime scripts, etc), but with a heavily POSIX biased bent (even moreso than the Python ecosystem). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 28 December 2013 06:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
The other main thing to look at in terms of current state of the art is npm, for both structure and hooks.
Another place to look would be existing usage in distributions on PyPI. I know that Vinay has done some work on extracting build-time data from PyPI (the red-dove metadata repo) - I don't know if he has anything like this available. It's probably too much work to get if nobody has already done it as a side effect of something else, though... Paul.
data: directory for data files.
This is still only a half-baked idea at this point, but I'm currently leaning towards keeping the "<name>.data" sysconfig subdirectories in the wheel format cross platform (and somewhat Python specific), and adding a new "<name>.app" subdirectory in parallel.
It's important to point out that the distutils/setuptools implementation of installing setup "data_files" offers a loophole now to achieve the "app" notion (at least when installing from sdist). When "data_files" paths are absolute, it honors them as absolute, allowing you to install anything/anywhere, if you have privileges (otoh when paths are relative, they go into the sysconfig "data" path as you would expect) as noted, wheel installs have no such mechanism, because there's just "*.data" and it maps to sysconfig "data" in the pip wheel installer. Also note that bdist_wheel actually does the wrong thing now with absolute "data_files" paths by letting them end up relative to site-packages https://bitbucket.org/dholth/wheel/issue/92/bdist_wheel-makes-absolute-data_... ) but I agree we need something more than "data" for platform-specific data files. Marcus
On 27 December 2013 12:34, Marcus Smith <qwcode@gmail.com> wrote:
data: directory for data files.
This is still only a half-baked idea at this point, but I'm currently leaning towards keeping the "<name>.data" sysconfig subdirectories in the wheel format cross platform (and somewhat Python specific), and adding a new "<name>.app" subdirectory in parallel.
It's important to point out that the distutils/setuptools implementation of installing setup "data_files" offers a loophole now to achieve the "app" notion (at least when installing from sdist).
When "data_files" paths are absolute, it honors them as absolute, allowing you to install anything/anywhere, if you have privileges (otoh when paths are relative, they go into the sysconfig "data" path as you would expect)
as noted, wheel installs have no such mechanism, because there's just "*.data" and it maps to sysconfig "data" in the pip wheel installer. Also note that bdist_wheel actually does the wrong thing now with absolute "data_files" paths by letting them end up relative to site-packages https://bitbucket.org/dholth/wheel/issue/92/bdist_wheel-makes-absolute-data_...)
but I agree we need something more than "data" for platform-specific data files.
Yeah, I don't think it's an urgent problem, but I believe it's one we will want to solve eventually. If we do another packaging BoF in Montreal (and hopefully we'll have the numbers for it to make sense to do so), that could be a good opportunity to bounce a few ideas around without spamming the list with more incomplete designs like my initial attempt. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (5)
-
Chris Barker -
Daniel Holth -
Marcus Smith -
Nick Coghlan -
Paul Moore