"Absolute" paths in PEP 376 RECORD files
Just an idea... suppose that instead of using "real" absolute paths in the RECORD file for non-local files (scripts, data, etc) we changed the format to include a "prefix" field, containing something like LIBDIR, SHARE, SCRIPTS, etc., ala bdist_wininst internals? Also, we could include a separate (optional) PREFIXES file defining what those locations translated to at install time. Dumb bdists would still have their build paths there (or delete the file before packaging, or use real paths instead of the --root prefixed versions). Upside: relocation can be detected and handled, RECORD remains cross-platform, and bdists are ok. Downside: more complex API required to read/manipulate paths and delete files, since you need to be able to check that you have the right prefixes, and may have to ask the user for a prefix you don't recognize, if the default in PREFIXES doesn't match. Thoughts? (Also, as a special case, any file that's actually installed to LIBDIR or a subdirectory thereof (even if it's technically a "data" file or script), will probably need to be designated under LIBDIR prefix to prevent runtime breakage in the event any system package maintainers are tempted to use RECORD files as a way of forcing a HFS conformance. These are installation prefixes, *not* content types.)
Is there any reason why RECORD file can't be generated at runtime? Also, why should the RECORD file be generated at all by bdist* commands? A .deb file contains "data.tar.gz" that is simply extracted over "/". There is no need for RECORD inside a .deb file because it is implicit in the contents on data.tar.gz. -srid On Tue, 07 Jul 2009 13:30:05 -0700, P.J. Eby <pje@telecommunity.com> wrote:
Just an idea... suppose that instead of using "real" absolute paths in the RECORD file for non-local files (scripts, data, etc) we changed the format to include a "prefix" field, containing something like LIBDIR, SHARE, SCRIPTS, etc., ala bdist_wininst internals?
Also, we could include a separate (optional) PREFIXES file defining what those locations translated to at install time. Dumb bdists would still have their build paths there (or delete the file before packaging, or use real paths instead of the --root prefixed versions).
Upside: relocation can be detected and handled, RECORD remains cross-platform, and bdists are ok.
Downside: more complex API required to read/manipulate paths and delete files, since you need to be able to check that you have the right prefixes, and may have to ask the user for a prefix you don't recognize, if the default in PREFIXES doesn't match.
Thoughts?
(Also, as a special case, any file that's actually installed to LIBDIR or a subdirectory thereof (even if it's technically a "data" file or script), will probably need to be designated under LIBDIR prefix to prevent runtime breakage in the event any system package maintainers are tempted to use RECORD files as a way of forcing a HFS conformance. These are installation prefixes, *not* content types.)
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/sridharr%40activestate.com
At 06:51 PM 7/8/2009 -0700, Sridhar Ratnakumar wrote:
Is there any reason why RECORD file can't be generated at runtime?
Applications can be relocatable, yet require plugins and libraries to be upgraded, installed, uninstalled, etc. (Also, RECORD is cross-platform to allow cross-platform installation/testing of cross-platform packages; that's why I propose keeping the prefix information separate.)
Also, why should the RECORD file be generated at all by bdist* commands?
bdist commands that use "install --root" will get it as a side effect, so they don't have to do anything special. They needn't even generate the PREFIXES, if PREFIXES is also generated by "install --root", and omits the root from the prefixes written. In other words, most bdist tools wouldn't change. bdist_wininst and bdist_msi might need to regenerate PREFIXES at actual installation time, but AFAIK those are the only ones. (And the spec would allow for the possibility that a PREFIXES file can be incorrect.)
P.J. Eby wrote:
Also, why should the RECORD file be generated at all by bdist* commands?
bdist commands that use "install --root" will get it as a side effect, so they don't have to do anything special. They needn't even generate the PREFIXES, if PREFIXES is also generated by "install --root", and omits the root from the prefixes written. In other words, most bdist tools wouldn't change. bdist_wininst and bdist_msi might need to regenerate PREFIXES at actual installation time, but AFAIK those are the only ones. (And the spec would allow for the possibility that a PREFIXES file can be incorrect.)
I haven't seen any suggestions for handling non-relative paths that are cleaner than this. The distutils based list that Tarek posted also gives a good "starter set" of supported entries in the PREFIXES file: PURELIB=<path> PLATLIB=<path> HEADERS=<path> SCRIPTS=<path> DATA=<path> The question then would be whether relative paths should be handled directly in the RECORD file (as in the current PEP) or by writing a "." into PREFIXES to indicate relative paths. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nick Coghlan wrote:
P.J. Eby wrote:
Also, why should the RECORD file be generated at all by bdist* commands? bdist commands that use "install --root" will get it as a side effect, so they don't have to do anything special. They needn't even generate the PREFIXES, if PREFIXES is also generated by "install --root", and omits the root from the prefixes written. In other words, most bdist tools wouldn't change. bdist_wininst and bdist_msi might need to regenerate PREFIXES at actual installation time, but AFAIK those are the only ones. (And the spec would allow for the possibility that a PREFIXES file can be incorrect.)
I haven't seen any suggestions for handling non-relative paths that are cleaner than this. The distutils based list that Tarek posted also gives a good "starter set" of supported entries in the PREFIXES file:
PURELIB=<path> PLATLIB=<path> HEADERS=<path> SCRIPTS=<path> DATA=<path>
The question then would be whether relative paths should be handled directly in the RECORD file (as in the current PEP) or by writing a "." into PREFIXES to indicate relative paths.
Per discussion with Matthias Klose, it would be useful to split out the following prefixes for "special" kinds of data: - - docs - - i18n / locales Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKVjr6+gerLs4ltQ4RAqwTAKCoohHwkbON3Gg6gmJ4AmZUcPSRdQCgyeZi nBJVZQziPYnWN0cI3B2i4A0= =bSHL -----END PGP SIGNATURE-----
At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Nick Coghlan wrote:
P.J. Eby wrote:
Also, why should the RECORD file be generated at all by bdist* commands? bdist commands that use "install --root" will get it as a side effect, so they don't have to do anything special. They needn't even generate the PREFIXES, if PREFIXES is also generated by "install --root", and omits the root from the prefixes written. In other words, most bdist tools wouldn't change. bdist_wininst and bdist_msi might need to regenerate PREFIXES at actual installation time, but AFAIK those are the only ones. (And the spec would allow for the possibility that a PREFIXES file can be incorrect.)
I haven't seen any suggestions for handling non-relative paths that are cleaner than this. The distutils based list that Tarek posted also gives a good "starter set" of supported entries in the PREFIXES file:
PURELIB=<path> PLATLIB=<path> HEADERS=<path> SCRIPTS=<path> DATA=<path>
The question then would be whether relative paths should be handled directly in the RECORD file (as in the current PEP) or by writing a "." into PREFIXES to indicate relative paths.
Per discussion with Matthias Klose, it would be useful to split out the following prefixes for "special" kinds of data:
- - docs - - i18n / locales
Unfortunately, the distutils don't currently have a way to specify these. I think we'll need to use only the ones that are distutils install targets at first, but of course the format is extensible. OTOH, one supposed that at least the 'data_files' setup argument could be enhanced to allow a data type flag, that would then be used as a prefix name for those files, with a fallback to the DATA prefix if there isn't one specified. But that's something that will require tagging support in the install_data command. As it is, refactoring the distutils to track prefixes for the RECORD file is probably going to be... challenging. ;-)
At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote:
- - docs - - i18n / locales
On Jul 9, 2009, at 3:09 PM, P.J. Eby wrote:
Unfortunately, the distutils don't currently have a way to specify these.
There actually is a way to identify documentation files via setup.cfg, but I don't know if the RPM packagers (or others) use that information. -Fred -- Fred Drake <fdrake at acm.org>
On Thu, 09 Jul 2009 16:00:41 -0400, Fred Drake <fdrake@acm.org> wrote:
At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote:
- - docs - - i18n / locales
On Jul 9, 2009, at 3:09 PM, P.J. Eby wrote:
Unfortunately, the distutils don't currently have a way to specify these.
There actually is a way to identify documentation files via setup.cfg, but I don't know if the RPM packagers (or others) use that information.
Hi, Perphaps the RPM packagers dont use it but the Python Package Manager GUI (http://sourceforge.net/projects/pythonpkgmgr) does. With that, if there is any package documentation in a docs directory the user can click on it and display it. Regards David
On Thu, Jul 9, 2009 at 9:09 PM, P.J. Eby<pje@telecommunity.com> wrote:
At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Nick Coghlan wrote:
P.J. Eby wrote:
Also, why should the RECORD file be generated at all by bdist* commands? bdist commands that use "install --root" will get it as a side effect, so they don't have to do anything special. They needn't even generate the PREFIXES, if PREFIXES is also generated by "install --root", and omits the root from the prefixes written. In other words, most bdist tools wouldn't change. bdist_wininst and bdist_msi might need to regenerate PREFIXES at actual installation time, but AFAIK those are the only ones. (And the spec would allow for the possibility that a PREFIXES file can be incorrect.)
I haven't seen any suggestions for handling non-relative paths that are cleaner than this. The distutils based list that Tarek posted also gives a good "starter set" of supported entries in the PREFIXES file:
PURELIB=<path> PLATLIB=<path> HEADERS=<path> SCRIPTS=<path> DATA=<path>
The question then would be whether relative paths should be handled directly in the RECORD file (as in the current PEP) or by writing a "." into PREFIXES to indicate relative paths.
I like the idea of a separated PREFIXES file, then using variables in the RECORD file. While this doesn't change anything code-wise, we can keep that way two csv-browsable files.
Per discussion with Matthias Klose, it would be useful to split out the following prefixes for "special" kinds of data:
- - docs - - i18n / locales
Unfortunately, the distutils don't currently have a way to specify these. I think we'll need to use only the ones that are distutils install targets at first, but of course the format is extensible.
OTOH, one supposed that at least the 'data_files' setup argument could be enhanced to allow a data type flag, that would then be used as a prefix name for those files, with a fallback to the DATA prefix if there isn't one specified. But that's something that will require tagging support in the install_data command. As it is, refactoring the distutils to track prefixes for the RECORD file is probably going to be... challenging. ;-)
yes, if we want to provide extensibility, a new prefix that is used by a developer in its distribution will have to be provided for all platforms, so distutils can build the proper RECORD file by querying the path that corresponds to the platform it runs on. In other words, if a data type flag is provided in data_files, the list of absolute paths per key in distutils.command.install.INSTALL_SCHEMES should be provided with it. This could be done with a new 'extra_prefixes' argument for example, provided with the 'data_files', that would let distutils extend INSTALL_SCHEMES on the fly setup( ... data_files={'i18n': ['en.po', 'fr.po']}, extra_prefixes={'i18n': {'unix_prefix': '/somewhere', 'mac': '/Application/right/here'} ... ) But then data_files could maybe be left alone (that's specific to 'data' after all) and a new option created. In any case that would be a great solution to avoid 'wild absolute path' and be sure that the developer thinks about where each fil of his distrbution goes, depending on the plaftorm. While debian packagers and al might not use it to re-package python distributions in debian packages, I am sure they will like it very much because it helps identifiying what the developer wanted to do with these "data" files.
-On [20090710 15:32], Tarek Ziadé (ziade.tarek@gmail.com) wrote:
setup( ... data_files={'i18n': ['en.po', 'fr.po']}, extra_prefixes={'i18n': {'unix_prefix': '/somewhere', 'mac': '/Application/right/here'} ... )
As an aside: Where i18n above would sooner be called 'locale' rather than i18n, since i18n is only providing software functionality for supporting localization. So in that sense L10n would be better, but locale is quite a defacto standard in many projects. -- Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B In every colour there's the Light...
Tres Seaver wrote:
Nick Coghlan wrote:
The question then would be whether relative paths should be handled directly in the RECORD file (as in the current PEP) or by writing a "." into PREFIXES to indicate relative paths.
Per discussion with Matthias Klose, it would be useful to split out the following prefixes for "special" kinds of data:
- docs - i18n / locales
There are probably all sorts of other useful distinctions which could be made. However, that would be a topic for a different PEP - as Tarek pointed out, improving distutils is such a huge can of worms that trying to tackle all the issues in a single PEP would most likely just result in developer burnout rather than improvement. The basic PREFIXES discussion is at least somewhat relevant to PEP 376, since the distutils uninstaller needs *some* way for the bdist_* installers to tell it where the non-relative files ended up. Since we need a mechanism for it, it might as well be one that should help with the system package builders needs in the future. Actually adding new categories, while I agree it would be useful, isn't something that is needed to support uninstallation. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Fri, 10 Jul 2009 05:53:13 +1000, Nick Coghlan <ncoghlan@gmail.com> wrote:
There are probably all sorts of other useful distinctions which could be made. However, that would be a topic for a different PEP - as Tarek pointed out, improving distutils is such a huge can of worms that trying to tackle all the issues in a single PEP would most likely just result in developer burnout rather than improvement.
All it requires is a few word changes to the documentation to tell programmers to put your documentation in "docs"... If that is going to burn out the existing developers, it might suggest that it is time to get some fresh developers onboard.... Thats not even a programmatic fix... Regards David
David Lyon wrote:
All it requires is a few word changes to the documentation to tell programmers to put your documentation in "docs"...
There is no current distutils setting to indicate that a particular data file is actually documentation rather than something else - hence such a thing *isn't* just a matter of updating the documentation. If you're actually saying that "hey, that metadata shouldn't be in the code, it should just be a file naming convention" then that *in itself* is a design decision that needs to be discussed and agreed via a PEP (a 'standard' that nobody follows or has even agreed to follow is a useless standard).
If that is going to burn out the existing developers, it might suggest that it is time to get some fresh developers onboard....
No, tackling *all* of the distutils issues (supporting uninstallation, moving from setup.py to static metadata for simple cases, providing more fine-grained *programmatic* file categorisation, anything else I've missed) at once would probably burn out anyone that tried to do it. The coding in all this really isn't that hard. It's the gathering of feedback, assessment of use cases and the constant balancing act between retaining enough flexibility to allow for future extensions without a complete rebuild of the design without making the current design so complex as to be unusable. If it was just a matter of throwing some code in the standard library or some words in the documentation and declaring "this is how it shall be from now on" it would have been done years ago :P Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
participants (8)
-
David Lyon -
Fred Drake -
Jeroen Ruigrok van der Werven -
Nick Coghlan -
P.J. Eby -
Sridhar Ratnakumar -
Tarek Ziadé -
Tres Seaver