Re: [Python-Dev] PEP 376 - Open questions
At 11:30 PM 7/7/2009 +0200, Tarek Ziadé wrote:
On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby<pje@telecommunity.com> wrote:
At 03:23 PM 7/7/2009 +0200, Tarek Ziadé wrote:
When I started to work on this I didn't realize the gigantic amount of work and coordination it requires
No one expects the package inquisition. ;-)
Sorry, i've looked in the english dictionary but I don't get this one. what do you mean ?
It's a Monty Python reference, i.e. "no one expects the Spanish Inquisition". In the Spanish Inquisition sketch, a character is asked several questions he can't answer, and then complains that he wasn't expecting some kind of Spanish inquisition, at which point three scarlet-robed cardinals burst into the room and exclaim, "NO ONE expects the Spanish Inquisition!" So I punned on that in order to imply that nobody who takes on the job of packaging expects to be asked the kind of detailed, unanswerable questions that come with the territory of packaging, or to be subjected to torture by third parties merely for trying to do something useful. And, by further implication, I was thus expressing camaraderie with you on your initiation into the wonderful world of trying to satisfy everyone in Pythonland, and our shared experience in the area of "if I'd really known what I was getting into, I probably wouldn't have". ;-)
On Wed, Jul 8, 2009 at 7:09 AM, P.J. Eby<pje@telecommunity.com> wrote:
At 11:30 PM 7/7/2009 +0200, Tarek Ziadé wrote:
On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby<pje@telecommunity.com> wrote:
At 03:23 PM 7/7/2009 +0200, Tarek Ziadé wrote:
When I started to work on this I didn't realize the gigantic amount of work and coordination it requires
No one expects the package inquisition. ;-)
Sorry, i've looked in the english dictionary but I don't get this one. what do you mean ?
It's a Monty Python reference, i.e. "no one expects the Spanish Inquisition".
In the Spanish Inquisition sketch, a character is asked several questions he can't answer, and then complains that he wasn't expecting some kind of Spanish inquisition, at which point three scarlet-robed cardinals burst into the room and exclaim, "NO ONE expects the Spanish Inquisition!"
So I punned on that in order to imply that nobody who takes on the job of packaging expects to be asked the kind of detailed, unanswerable questions that come with the territory of packaging, or to be subjected to torture by third parties merely for trying to do something useful.
And, by further implication, I was thus expressing camaraderie with you on your initiation into the wonderful world of trying to satisfy everyone in Pythonland, and our shared experience in the area of "if I'd really known what I was getting into, I probably wouldn't have". ;-)
Hehe very nice, thanks for the explanation :) (thanks to the others too)
-- Tarek Ziadé | http://ziade.org
Tarek Ziadé wrote:
On Wed, Jul 8, 2009 at 7:09 AM, P.J. Eby<pje@telecommunity.com> wrote:
So I punned on that in order to imply that nobody who takes on the job of packaging expects to be asked the kind of detailed, unanswerable questions that come with the territory of packaging, or to be subjected to torture by third parties merely for trying to do something useful.
And, by further implication, I was thus expressing camaraderie with you on your initiation into the wonderful world of trying to satisfy everyone in Pythonland, and our shared experience in the area of "if I'd really known what I was getting into, I probably wouldn't have". ;-)
Hehe very nice, thanks for the explanation :)
(thanks to the others too)
I think is exchange actually a good reminder that the bar for acceptance of PEP 376 shouldn't be "Addresses every possible packaging issue we can come up with". That would be setting expectations far too high because packaging cross-platform is so messy. A more appropriate bar is probably: a) Better that what we have now; and b) Provides possible avenues for even more improvement in the future For example, I quite like the concept behind the various ideas for "location" or "prefix" definitions either in the RECORD file itself or in a separate PREFIXES file, since such approaches feeds directly in to part b) above. Once some kind prefix definition mechanism is in place in the metadata, it becomes possible to leverage it to categorise files for the benefit of system packagers while still allowing simple internal-use-only packages to just bundle everything together. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan <ncoghlan <at> gmail.com> writes:
I think is exchange actually a good reminder that the bar for acceptance of PEP 376 shouldn't be "Addresses every possible packaging issue we can come up with". That would be setting expectations far too high because packaging cross-platform is so messy.
+1
For example, I quite like the concept behind the various ideas for "location" or "prefix" definitions either in the RECORD file itself or in a separate PREFIXES file, since such approaches feeds directly in to part b) above.
I don't understand the point. Since RECORD is specific to a particular machine and installation (it is generated at install time), what would you gain by adding a fake genericity to its entries? Regards Antoine.
Antoine Pitrou wrote:
Nick Coghlan <ncoghlan <at> gmail.com> writes:
For example, I quite like the concept behind the various ideas for "location" or "prefix" definitions either in the RECORD file itself or in a separate PREFIXES file, since such approaches feeds directly in to part b) above.
I don't understand the point. Since RECORD is specific to a particular machine and installation (it is generated at install time), what would you gain by adding a fake genericity to its entries?
My understanding is that RECORD isn't generated at install time for the bdist_* commands -it's generated at the time the binary distribution is created. Hence the problems raised with the idea of having absolute paths in RECORD - if they were written directly into RECORD they would reflect where the files were when the distribution was created, not where they end up when it is installed. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
2009/7/8 Nick Coghlan <ncoghlan@gmail.com>:
Antoine Pitrou wrote:
Nick Coghlan <ncoghlan <at> gmail.com> writes:
For example, I quite like the concept behind the various ideas for "location" or "prefix" definitions either in the RECORD file itself or in a separate PREFIXES file, since such approaches feeds directly in to part b) above.
I don't understand the point. Since RECORD is specific to a particular machine and installation (it is generated at install time), what would you gain by adding a fake genericity to its entries?
My understanding is that RECORD isn't generated at install time for the bdist_* commands -it's generated at the time the binary distribution is created. Hence the problems raised with the idea of having absolute paths in RECORD - if they were written directly into RECORD they would reflect where the files were when the distribution was created, not where they end up when it is installed.
This is a key open question, IMHO. No-one has done any work on updating the core bdist_xxx commands to the new structure, and the PEP is silent on the matter, so it's still all speculation. The current "best guess" assumption is that, given that bdist_xxx uses a "setup.py install --root=xxx" incantation to generate the structure to package, the RECORD can be picked up from this. There are a number of issues with this assumption, though - it doesn't handle absolute locations well (there are existing bugs), paths may be affected by the environment at the time the installer is built, and it requires RECORD to be relocatable. An alternative solution would be for the bdist_xxx commands to ignore the RECORD file generated in the temp area, and build its own on the target machine when the installer is run. This is conceptuially far cleaner, it's in line with the implicit assumptions in the PEP, and it avoids a lot of the problems with path formats (the filenames in RECORD could be absolute local pathnames, and that's the end of the story). But - and it's a big but - this dumps RECORD file handling on the bdist_xxx command, and they all need to implement the same code, resulting in a lot of duplication and opportunity for inconsistency. I have no answer here. But I think we need feedback from the people who will ultimately be building bdist_xxx formats - Debian packagers, people wrting alternate RPM builders, etc. There's also Phillip commenting from the setuptools POV and MAL who has mentioned his build process. Bottom line - Is RECORD to be created on the target machine at install time, or must it be relocatable? Subsequent question - if it's to be relocatable, how will that work? Paul.
Paul Moore <p.f.moore <at> gmail.com> writes:
I have no answer here. But I think we need feedback from the people who will ultimately be building bdist_xxx formats - Debian packagers, people wrting alternate RPM builders, etc.
I think any bdist_xxx command which targets externally handled package formats (rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or dpkg, you will uninstall with the same tool, won't you?
Bottom line - Is RECORD to be created on the target machine at install time, or must it be relocatable?
This is quite an uninformed opinion, but a relocatable RECORD looks like a can of worms to me. As for creating RECORD at install time, I don't understand the argument about duplicate code. Distutils can provide useful primitives for generating RECORD, so that interested subcommands don't have a lot of additional work to do.
2009/7/8 Antoine Pitrou <solipsis@pitrou.net>:
Paul Moore <p.f.moore <at> gmail.com> writes:
I have no answer here. But I think we need feedback from the people who will ultimately be building bdist_xxx formats - Debian packagers, people wrting alternate RPM builders, etc.
I think any bdist_xxx command which targets externally handled package formats (rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or dpkg, you will uninstall with the same tool, won't you?
I thought the purpose of RECORD was (also) so that *other* installs could check who "owned" an existing file, before potentially overwriting it. That seems like it's the reasoning behind the uses() function. But the PEP is very light on practical applications, so it's hard to tell what the methods are intended to be used for.
Bottom line - Is RECORD to be created on the target machine at install time, or must it be relocatable?
This is quite an uninformed opinion, but a relocatable RECORD looks like a can of worms to me.
As for creating RECORD at install time, I don't understand the argument about duplicate code. Distutils can provide useful primitives for generating RECORD, so that interested subcommands don't have a lot of additional work to do.
The bdist_wininst code is in C, and the bdist_msi code is driven by the MSI data format. I don't know what scripting options exist for .deb and .rpm files. Different languages, so no code sharing seems possible. Maybe I'm wrong, and something could be done along the lines of a "pkgutil.generate_record_file()" function which could be run from a post-install script written in Python, but I don't know how plausible or robust that would be... Paul.
Antoine Pitrou wrote:
Paul Moore <p.f.moore <at> gmail.com> writes:
I have no answer here. But I think we need feedback from the people who will ultimately be building bdist_xxx formats - Debian packagers, people wrting alternate RPM builders, etc.
I think any bdist_xxx command which targets externally handled package formats (rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or dpkg, you will uninstall with the same tool, won't you?
I agree with this. For RPM's, there's a whole other database of what files were installed. Is it really the intent that a file will be managed by multiple different installers? That I can install with RPM but remove with some python-installer (or other) tool? That way lies madness. In fact, I see RECORD as an installer-specific detail that doesn't need to be standardized at all. For bdist_rpm (which I'm contemplating taking over, but no promises), the only thing that needs to be done with the metadata is be able to build a .spec file. Once the .spec file is generated, rpmbuild (or equivalent) will produce the .rpm files. I image that .deb's are similar, but I'm no expert. Which brings up the "static metadata" argument I was making at PyCon. I want there to be enough static metadata that the new bdist_rpm can read so that it produces a well-behaved .spec file. I need to know the intent of some of the files (which are documentation, which are config files, which are scripts, which are libraries, etc.) and not much else. I envision the new bdist_rpm as a standalone tool, not a distutils extension.
Bottom line - Is RECORD to be created on the target machine at install time, or must it be relocatable?
This is quite an uninformed opinion, but a relocatable RECORD looks like a can of worms to me.
Agreed.
As for creating RECORD at install time, I don't understand the argument about duplicate code. Distutils can provide useful primitives for generating RECORD, so that interested subcommands don't have a lot of additional work to do.
I envision a "distlib" that contains new support code that's not part of the current distutils morass.
At 01:13 PM 7/8/2009 +0100, Paul Moore wrote:
An alternative solution would be for the bdist_xxx commands to ignore the RECORD file generated in the temp area, and build its own on the target machine when the installer is run. This is conceptuially far cleaner, it's in line with the implicit assumptions in the PEP, and it avoids a lot of the problems with path formats (the filenames in RECORD could be absolute local pathnames, and that's the end of the story). But - and it's a big but - this dumps RECORD file handling on the bdist_xxx command, and they all need to implement the same code, resulting in a lot of duplication and opportunity for inconsistency.
The PREFIXES approach I suggested doesn't have these problems; the bdist installer only needs to write a PREFIXES file, and doesn't need to touch RECORD (which would be generated by the install --root). In fact, since PREFIXES is strictly advisory, it can be generated by the install --root and left alone at installation time. And, if it's generated without the --root prefix included, RPMs would have a correct PREFIXES file without further manipulation. Plus, the bdist_wininst .exe installer can easily write a PREFIXES file, since it has to know the prefixes in order to do an installation in the first place. (We still need to hash out an initial list of prefixes, of course.) At 08:59 AM 7/8/2009 -0400, Eric Smith wrote:
I agree with this. For RPM's, there's a whole other database of what files were installed. Is it really the intent that a file will be managed by multiple different installers? That I can install with RPM but remove with some python-installer (or other) tool? That way lies madness. In fact, I see RECORD as an installer-specific detail that doesn't need to be standardized at all.
This is a misunderstanding. The purpose is to let an *installer* (like easy_install) know that these files belong to some other installer, and not allow overwriting them. That's why there's also an INSTALLER file.
On Wed, Jul 8, 2009 at 5:11 PM, P.J. Eby<pje@telecommunity.com> wrote:
At 01:13 PM 7/8/2009 +0100, Paul Moore wrote:
An alternative solution would be for the bdist_xxx commands to ignore the RECORD file generated in the temp area, and build its own on the target machine when the installer is run. This is conceptuially far cleaner, it's in line with the implicit assumptions in the PEP, and it avoids a lot of the problems with path formats (the filenames in RECORD could be absolute local pathnames, and that's the end of the story). But - and it's a big but - this dumps RECORD file handling on the bdist_xxx command, and they all need to implement the same code, resulting in a lot of duplication and opportunity for inconsistency.
The PREFIXES approach I suggested doesn't have these problems; the bdist installer only needs to write a PREFIXES file, and doesn't need to touch RECORD (which would be generated by the install --root). In fact, since PREFIXES is strictly advisory, it can be generated by the install --root and left alone at installation time. And, if it's generated without the --root prefix included, RPMs would have a correct PREFIXES file without further manipulation. Plus, the bdist_wininst .exe installer can easily write a PREFIXES file, since it has to know the prefixes in order to do an installation in the first place.
(We still need to hash out an initial list of prefixes, of course.)
Instead of building this list of prefixes from scratch, let's use the ones that are *actually* used when the install command is called, have a look at the list I've suggested : http://mail.python.org/pipermail/python-dev/2009-July/090514.html And let's drop other absolute paths,
P.J. Eby wrote:
At 08:59 AM 7/8/2009 -0400, Eric Smith wrote:
I agree with this. For RPM's, there's a whole other database of what files were installed. Is it really the intent that a file will be managed by multiple different installers? That I can install with RPM but remove with some python-installer (or other) tool? That way lies madness. In fact, I see RECORD as an installer-specific detail that doesn't need to be standardized at all.
This is a misunderstanding. The purpose is to let an *installer* (like easy_install) know that these files belong to some other installer, and not allow overwriting them. That's why there's also an INSTALLER file.
I really don't see this use case. Supporting multiple installers for the same file (or even just trying to prevent two installers from writing the same file)? Wouldn't you be better off just saying an installer can't overwrite any existing file? Likewise, I don't see a use case for installing with one installer and uninstalling with another. Put those two together, and the mechanism that an installer uses to record what files it installed is a private implementation detail. Eric.
At 01:58 PM 7/8/2009 -0400, Eric Smith wrote:
I really don't see this use case. Supporting multiple installers for the same file (or even just trying to prevent two installers from writing the same file)? Wouldn't you be better off just saying an installer can't overwrite any existing file?
Likewise, I don't see a use case for installing with one installer and uninstalling with another.
Put those two together, and the mechanism that an installer uses to record what files it installed is a private implementation detail.
You'll have to ask Tarek about that. As I've said in another email, if I were doing this for setuptools I'd just go and do it; it's already on my backlog of things to do. The point of the PEP is to provide an uninstall facility for the distutils, and to expose those records via an API so that other tools can be built on top of it. Whether anything other than distutils-based tools will actually use the format directly is entirely irrelevant to the PEP's merits, since bdist_* tools and platform-specific distutils installers may need to read or write the format. Hence, a PEP is appropriate to describe a format and API for the distutils family of tools to operate on. The specific proposed format is based on an existing de facto standard for installed package metadata, so that the many existing packaging tools based on that other format and existing APIs can easily read or write this data, and so that the API can also be used as a way to detect the presence of already-installed distributions of software -- even ones installed other ways.
Hehe very nice, thanks for the explanation :)
(thanks to the others too)
I think is exchange actually a good reminder that the bar for acceptance of PEP 376 shouldn't be "Addresses every possible packaging issue we can come up with". That would be setting expectations far too high because packaging cross-platform is so messy.
OTOH, it *is* expected that the PEP explicitly lists aspects that it didn't resolve, so that any reader can find out whether something was discussed and rejected, or not discussed. Regards, Martin
participants (7)
-
"Martin v. Löwis" -
Antoine Pitrou -
Eric Smith -
Nick Coghlan -
P.J. Eby -
Paul Moore -
Tarek Ziadé