PEP 376 - RECORD file / Data files + pip feedback ?
Hi,
The part that still requires some work is how to handle files prefixes in the RECORD file.
Last time we said that we could benefit of having a PREFIXES file. Then Wolojda started to work on a much more complete solution to track data file locations : http://mail.python.org/pipermail/distutils-sig/2009-November/014424.html
But IIRC he's not able to work on this at this time. (CCing him in case..)
What I suggest is that we leave this new work for a future enhanced version of PEP 376, and try to finish a version that we would want to have in Python 2.7 (that's coming very soon), where RECORD contains what we want to make it possible to remove installed files + provide the query API.
As far as I am concerned, the RECORD thingie needs more live examples on various platforms (that's what I am working on now),
But since Pip has now a uninstaller for a few months, the question is: what do you Pip guys think about this RECORD file ? (cc'ing Ian as well - I know the other involved in that are listening here :))
Regards Tarek
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
hi Tarek,
Tarek Ziadé wrote:
The part that still requires some work is how to handle files prefixes in the RECORD file.
Last time we said that we could benefit of having a PREFIXES file. Then Wolojda started to work on a much more complete solution to track data file locations : http://mail.python.org/pipermail/distutils-sig/2009-November/014424.html
[snip]
But since Pip has now a uninstaller for a few months, the question is: what do you Pip guys think about this RECORD file ? (cc'ing Ian as well - I know the other involved in that are listening here :))
The prefix stuff in RECORD appears to be new since I last looked at PEP 376, and I don't like it.
If RECORD does not contain absolute paths, it becomes a lot trickier to use it for uninstallation. In particular if prefix placeholders are used, it's impossible to uninstall unless we can query for the actual prefix paths used at install time. And we don't have that unless we finish Wolodja's work.
If we do finish Wolodja's work, the PREFIXES file will necessarily contain absolute paths anyway. In which case the installation metadata is once again not relocatable.
So it seems to me that this idea of a relocatable installation is inherently incompatible with providing metadata that records the actual files installed on an actual system. Which is the metadata needed for uninstall.
Are relocatable installations really all that valuable anyway? What's the use-case? It seems to me that "move an existing installation" is something a higher-level tool should worry about (rewriting RECORD in that case).
I think we should remove all of the relative-path and prefix stuff from RECORD and go back to requiring absolute paths for all files. This keeps it simple and obvious, drastically reducing the surface area for bugs. And that makes it orthogonal to Wolodja's work instead of dependent on it, so we can go ahead with what we have for 2.7.
Carl
On Fri, Feb 5, 2010 at 6:18 PM, Carl Meyer carl@meyerloewen.net wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
hi Tarek,
Tarek Ziadé wrote:
The part that still requires some work is how to handle files prefixes in the RECORD file.
Last time we said that we could benefit of having a PREFIXES file. Then Wolojda started to work on a much more complete solution to track data file locations : http://mail.python.org/pipermail/distutils-sig/2009-November/014424.html
[snip]
But since Pip has now a uninstaller for a few months, the question is: what do you Pip guys think about this RECORD file ? (cc'ing Ian as well - I know the other involved in that are listening here :))
The prefix stuff in RECORD appears to be new since I last looked at PEP 376, and I don't like it.
If RECORD does not contain absolute paths, it becomes a lot trickier to use it for uninstallation. In particular if prefix placeholders are used, it's impossible to uninstall unless we can query for the actual prefix paths used at install time. And we don't have that unless we finish Wolodja's work.
This information is given by the location where the egg info resides. (sys.prefix/sys.exec_prefix-like)
[..]
Are relocatable installations really all that valuable anyway? What's the use-case? It seems to me that "move an existing installation" is something a higher-level tool should worry about (rewriting RECORD in that case).
I think we should remove all of the relative-path and prefix stuff from RECORD and go back to requiring absolute paths for all files. This keeps it simple and obvious, drastically reducing the surface area for bugs. And that makes it orthogonal to Wolodja's work instead of dependent on it, so we can go ahead with what we have for 2.7.
Keeping absolute paths will require extra work in all bdist command because those use a local "install --root" called to created an binary tree, then copied on the system target on installation.
IOW the RECORD file for them would be inaccurate at installation time. Now for applications, being relocatable would be a plus but, the real question I see is: how much pain each one of these solutions will be to implement.
I am not sure we want to change all the binary commands in 2.7...
Tarek
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Tarek,
I fully admit that I'm not cognizant of all the internal intricacies of distutils, in particular the bdist_* stuff, so take my thoughts with a grain of salt. I just have experience trying to make uninstall work in pip.
Tarek Ziadé wrote:
If RECORD does not contain absolute paths, it becomes a lot trickier to use it for uninstallation. In particular if prefix placeholders are used, it's impossible to uninstall unless we can query for the actual prefix paths used at install time. And we don't have that unless we finish Wolodja's work.
This information is given by the location where the egg info resides. (sys.prefix/sys.exec_prefix-like)
Presuming the API is now available in distutils for finding PREFIX and EXEC_PREFIX, given the location of the installed .egg-info, that might work. But how does this fit with "relocatable"? Doesn't this mean it's really just "relocatable as long as you're relocating to one of a few places where distutils expects an installed .egg-info to reside"?
But what about data files? What about all the myriad ways a setup.py author can munge installation schemes and put things wherever they want? I already went through the muck once trying to get uninstall working for the crazy mix of installation schemes we have currently; it's very unappealing to me to think that relying on complicated prefix-munging is the best we'll be able to do for uninstall even under PEP 376. I have my doubts that it will result in reliable uninstall.
Keeping absolute paths will require extra work in all bdist command because those use a local "install --root" called to created an binary tree, then copied on the system target on installation. IOW the RECORD file for them would be inaccurate at installation time.
That is an issue. I can think of two possible solutions, both of which seem preferable to me to making a mess out of RECORD:
1. Introduce a simple strip filter for adding paths to RECORD that, for the bdist_* commands, strips the .../build prefix from the paths.
2. Just don't do RECORD for bdist_* at all: if you use bdist_* you're declaring that you want the target package manager to manage install/uninstall for you. I have no idea if this meets the needs of all bdist_* use-cases, but naively it seems to me the whole reason for bdist_* is because you've got some other package management system already. Somebody in IRC just said bdist_wininst generates its own internal record anyway...
Now for applications, being relocatable would be a plus but, the real question I see is: how much pain each one of these solutions will be to implement.
I can see the attractiveness of a self-contained, relocatable installation for applications.
What if RECORD allowed either absolute paths or paths relative to the location of .egg-info, but nothing else? That would preserve the option for application authors to make relocatable self-contained installations, while keeping RECORD simple and useful for uninstall without relying on prefix-finding magic.
Carl
At 01:39 PM 2/5/2010 -0500, Carl Meyer wrote:
What if RECORD allowed either absolute paths or paths relative to the location of .egg-info, but nothing else? That would preserve the option for application authors to make relocatable self-contained installations, while keeping RECORD simple and useful for uninstall without relying on prefix-finding magic.
I'm confused about something here. There will still be a PREFIXES file, right? So you shouldn't need prefix-finding magic except in the case where something has been relocated, right?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
P.J. Eby wrote:
At 01:39 PM 2/5/2010 -0500, Carl Meyer wrote:
What if RECORD allowed either absolute paths or paths relative to the location of .egg-info, but nothing else? That would preserve the option for application authors to make relocatable self-contained installations, while keeping RECORD simple and useful for uninstall without relying on prefix-finding magic.
I'm confused about something here. There will still be a PREFIXES file, right? So you shouldn't need prefix-finding magic except in the case where something has been relocated, right?
Well, that's the third option; if someone has time to finish up the PREFIXES idea and get it working in time for 2.7. Tarek had said if Wolodja wasn't available to keep working on it, it might not get done quickly enough for 2.7 and might need to be postponed. So I'd been assuming we wouldn't have that... but if we do, that mitigates pretty much all my concerns.
Carl
On 2010-02-05, at 8:13 AM, Tarek Ziadé wrote:
Hi,
The part that still requires some work is how to handle files prefixes in the RECORD file.
Last time we said that we could benefit of having a PREFIXES file. Then Wolojda started to work on a much more complete solution to track data file locations : http://mail.python.org/pipermail/distutils-sig/2009-November/014424.html
But IIRC he's not able to work on this at this time. (CCing him in case..)
What I suggest is that we leave this new work for a future enhanced version of PEP 376, and try to finish a version that we would want to have in Python 2.7 (that's coming very soon), where RECORD contains what we want to make it possible to remove installed files + provide the query API.
As far as I am concerned, the RECORD thingie needs more live examples on various platforms (that's what I am working on now),
But since Pip has now a uninstaller for a few months, the question is: what do you Pip guys think about this RECORD file ? (cc'ing Ian as well - I know the other involved in that are listening here :))
PyPM stores file paths relative to sys.prefix in the installation database. The binary package file (*.pypm/data.tar.gz) contains installed files relative to sys.prefix - implying that files-outside-sys.prefix are not supported by PyPM.
Do we have a real-world example for installing uninstallable data files outside sys.prefix?
-srid
At 04:49 PM 2/5/2010 -0800, Sridhar Ratnakumar wrote:
Do we have a real-world example for installing uninstallable data files outside sys.prefix?
Scripts aren't always in sys.prefix, and some packages install fixed data (e.g. web pages, message catalogs, etc.) to /usr/share or some such.
Also make sure you're not forgetting e.g. shared hosting scenarios without a virtualenv, unless you're planning to make virtualenvs an officially-supported thing. In such cases, the paths involved won't necessarily fall into a single root.
Also don't forget sys.exec_prefix. (Isn't packaging fun?)
On 2010-02-05, at 4:57 PM, P.J. Eby wrote:
At 04:49 PM 2/5/2010 -0800, Sridhar Ratnakumar wrote:
Do we have a real-world example for installing uninstallable data files outside sys.prefix?
Scripts aren't always in sys.prefix,
Scripts *usually* go to sys.prefix/bin (or sys.prefix/Scripts or site.USER_BASE/Scripts). Are their packages that actually put them elsewhere?
and some packages install fixed data (e.g. web pages, message catalogs, etc.) to /usr/share or some such.
This seems like a possible case. Does this only happen with Linux specific packages (eg: GNOME apps)?
Also make sure you're not forgetting e.g. shared hosting scenarios without a virtualenv, unless you're planning to make virtualenvs an officially-supported thing. In such cases, the paths involved won't necessarily fall into a single root.
PyPM is an ActivePython tool which installs itself into, say, /opt/ActivePython-2.6 - a single root directory. So we never had to worry about other layouts. PyPM supports virtualenvs too, so "pypm -E /tmp/myenv install lxml" would install into the specified virtualenv.
Also don't forget sys.exec_prefix. (Isn't packaging fun?)
In ActivePython, sys.exec_prefix == sys.prefix.
-srid
At 06:10 PM 2/5/2010 -0800, Sridhar Ratnakumar wrote:
On 2010-02-05, at 4:57 PM, P.J. Eby wrote:
At 04:49 PM 2/5/2010 -0800, Sridhar Ratnakumar wrote:
Do we have a real-world example for installing uninstallable
data files outside sys.prefix?
Scripts aren't always in sys.prefix,
Scripts *usually* go to sys.prefix/bin (or sys.prefix/Scripts or site.USER_BASE/Scripts). Are their packages that actually put them elsewhere?
It's not the package that determines installation locations, it's the user's command line options and configuration file settings that determine the locations for libraries, scripts, and data files.
Also make sure you're not forgetting e.g. shared hosting
scenarios without a virtualenv, unless you're planning to make virtualenvs an officially-supported thing. In such cases, the paths involved won't necessarily fall into a single root.
PyPM is an ActivePython tool which installs itself into, say, /opt/ActivePython-2.6 - a single root directory. So we never had to worry about other layouts. PyPM supports virtualenvs too, so "pypm -E /tmp/myenv install lxml" would install into the specified virtualenv.
Also don't forget sys.exec_prefix. (Isn't packaging fun?)
In ActivePython, sys.exec_prefix == sys.prefix.
Yes, well, as I said, isn't packaging (in the wider Python world, where few such simplifications can be had) fun? ;-)
On Sat, Feb 6, 2010 at 1:33 PM, P.J. Eby pje@telecommunity.com wrote:
At 06:10 PM 2/5/2010 -0800, Sridhar Ratnakumar wrote:
On 2010-02-05, at 4:57 PM, P.J. Eby wrote:
At 04:49 PM 2/5/2010 -0800, Sridhar Ratnakumar wrote:
Do we have a real-world example for installing uninstallable data files outside sys.prefix?
Scripts aren't always in sys.prefix,
Scripts *usually* go to sys.prefix/bin (or sys.prefix/Scripts or site.USER_BASE/Scripts). Are their packages that actually put them elsewhere?
It's not the package that determines installation locations, it's the user's command line options and configuration file settings that determine the locations for libraries, scripts, and data files.
Also make sure you're not forgetting e.g. shared hosting scenarios without a virtualenv, unless you're planning to make virtualenvs an officially-supported thing. In such cases, the paths involved won't necessarily fall into a single root.
PyPM is an ActivePython tool which installs itself into, say, /opt/ActivePython-2.6 - a single root directory. So we never had to worry about other layouts. PyPM supports virtualenvs too, so "pypm -E /tmp/myenv install lxml" would install into the specified virtualenv.
Also don't forget sys.exec_prefix. (Isn't packaging fun?)
In ActivePython, sys.exec_prefix == sys.prefix.
Yes, well, as I said, isn't packaging (in the wider Python world, where few such simplifications can be had) fun? ;-)
May I remind of my recent suggestion of a build manifest ? It was precisely designed to solve this kind of issues, and a similar format is used by Cabal for Haskell, which supports a wide variety of install scenario as well. It supports relocation.
I am rather suspicious about attempts of designing such things from scratch without looking at how other packaging tools do similar things.
cheers,
David
On Sat, Feb 6, 2010 at 5:42 AM, David Cournapeau cournape@gmail.com wrote:
On Sat, Feb 6, 2010 at 1:33 PM, P.J. Eby pje@telecommunity.com wrote:
At 06:10 PM 2/5/2010 -0800, Sridhar Ratnakumar wrote:
On 2010-02-05, at 4:57 PM, P.J. Eby wrote:
At 04:49 PM 2/5/2010 -0800, Sridhar Ratnakumar wrote:
Do we have a real-world example for installing uninstallable data files outside sys.prefix?
Scripts aren't always in sys.prefix,
Scripts *usually* go to sys.prefix/bin (or sys.prefix/Scripts or site.USER_BASE/Scripts). Are their packages that actually put them elsewhere?
It's not the package that determines installation locations, it's the user's command line options and configuration file settings that determine the locations for libraries, scripts, and data files.
Also make sure you're not forgetting e.g. shared hosting scenarios without a virtualenv, unless you're planning to make virtualenvs an officially-supported thing. In such cases, the paths involved won't necessarily fall into a single root.
PyPM is an ActivePython tool which installs itself into, say, /opt/ActivePython-2.6 - a single root directory. So we never had to worry about other layouts. PyPM supports virtualenvs too, so "pypm -E /tmp/myenv install lxml" would install into the specified virtualenv.
Also don't forget sys.exec_prefix. (Isn't packaging fun?)
In ActivePython, sys.exec_prefix == sys.prefix.
Yes, well, as I said, isn't packaging (in the wider Python world, where few such simplifications can be had) fun? ;-)
May I remind of my recent suggestion of a build manifest ? It was precisely designed to solve this kind of issues, and a similar format is used by Cabal for Haskell, which supports a wide variety of install scenario as well. It supports relocation.
I am rather suspicious about attempts of designing such things from scratch without looking at how other packaging tools do similar things.
I still have that in mind. In fact one of the sprints task in Pycon could be to create a configure command for distutils (like what 4suite has) (I've pointed you to the repo where I am have started a prototype for this in a recent mail -- we sure could use your help/knowledge on this if you want to contribute)
But that's a lot of work that will not happen in 2.7. Here, the attempt is not to correct distutils design flaw about building things (e.g. the fact that install is also a build command in some way), but rather to find the simplest way to define the RECORD file, whichars is the latest missing bits to have a PEP that can be accepted for inclusion in PEP 376 --
Regards Tarek
On Sat, Feb 06, 2010 at 12:26:28PM +0100, Tarek Ziadé wrote:
I still have that in mind. In fact one of the sprints task in Pycon could be to create a configure command for distutils (like what 4suite has) (I've pointed you to the repo where I am have started a prototype for this in a recent mail -- we sure could use your help/knowledge on this if you want to contribute)
Either this was in private or I missed this. If possible could you share this pointer again? I'd be interested in looking at it too.
Cheers Floris
On Sat, Feb 6, 2010 at 8:26 PM, Tarek Ziadé ziade.tarek@gmail.com wrote:
On Sat, Feb 6, 2010 at 5:42 AM, David Cournapeau cournape@gmail.com wrote:
On Sat, Feb 6, 2010 at 1:33 PM, P.J. Eby pje@telecommunity.com wrote:
At 06:10 PM 2/5/2010 -0800, Sridhar Ratnakumar wrote:
On 2010-02-05, at 4:57 PM, P.J. Eby wrote:
At 04:49 PM 2/5/2010 -0800, Sridhar Ratnakumar wrote:
Do we have a real-world example for installing uninstallable data files outside sys.prefix?
Scripts aren't always in sys.prefix,
Scripts *usually* go to sys.prefix/bin (or sys.prefix/Scripts or site.USER_BASE/Scripts). Are their packages that actually put them elsewhere?
It's not the package that determines installation locations, it's the user's command line options and configuration file settings that determine the locations for libraries, scripts, and data files.
Also make sure you're not forgetting e.g. shared hosting scenarios without a virtualenv, unless you're planning to make virtualenvs an officially-supported thing. In such cases, the paths involved won't necessarily fall into a single root.
PyPM is an ActivePython tool which installs itself into, say, /opt/ActivePython-2.6 - a single root directory. So we never had to worry about other layouts. PyPM supports virtualenvs too, so "pypm -E /tmp/myenv install lxml" would install into the specified virtualenv.
Also don't forget sys.exec_prefix. (Isn't packaging fun?)
In ActivePython, sys.exec_prefix == sys.prefix.
Yes, well, as I said, isn't packaging (in the wider Python world, where few such simplifications can be had) fun? ;-)
May I remind of my recent suggestion of a build manifest ? It was precisely designed to solve this kind of issues, and a similar format is used by Cabal for Haskell, which supports a wide variety of install scenario as well. It supports relocation.
I am rather suspicious about attempts of designing such things from scratch without looking at how other packaging tools do similar things.
I still have that in mind. In fact one of the sprints task in Pycon could be to create a configure command for distutils
Could you expand on that ? I don't understand the link between configure and the topic at hand ?
But that's a lot of work that will not happen in 2.7. Here, the attempt is not to correct distutils design flaw about building things (e.g. the fact that install is also a build command in some way), but rather to find the simplest way to define the RECORD file
Maybe we are not speaking about the same thing. I was talking about this thread
http://mail.python.org/pipermail/distutils-sig/2009-December/014994.html
Having a workable specification for this should not require any heavy distutils refactoring. As it is mostly reimplementing a known and already-used design, it should not be a lot of work.
David
On Tue, Feb 9, 2010 at 11:20 AM, David Cournapeau cournape@gmail.com wrote: [..]
But that's a lot of work that will not happen in 2.7. Here, the attempt is not to correct distutils design flaw about building things (e.g. the fact that install is also a build command in some way), but rather to find the simplest way to define the RECORD file
Maybe we are not speaking about the same thing. I was talking about this thread
http://mail.python.org/pipermail/distutils-sig/2009-December/014994.html
Having a workable specification for this should not require any heavy distutils refactoring. As it is mostly reimplementing a known and already-used design, it should not be a lot of work.
ok, I thought you were mentioning the creation of a configure command, that would create a description file that can be used at build step.
The build manifest you are mentioning (as I said back then) seems like a good idea, and is complementary in some ways with what Wolojda has started.
Now, since it seems a bit similar to the RECORD file (a list of files + variables) , maybe you could describe the differences (keeping the target file, file groups) by taking an example of RECORD file from PEP 376, and rewriting it, so we can compare and understand it better,
Tarek
On Tue, Feb 9, 2010 at 8:00 PM, Tarek Ziadé ziade.tarek@gmail.com wrote:
Now, since it seems a bit similar to the RECORD file (a list of files
- variables) , maybe you could describe
the differences (keeping the target file, file groups) by taking an example of RECORD file from PEP 376, and rewriting it, so we can compare and understand it better,
Is there anything besides what is described in PEP 376 for the RECORD syntax ? It would also be useful for me to see the actual implementation for the syntax (or is everything described in PEP 376 already in the main distribute hg repo ?)
thanks,
David
On Sat, Feb 6, 2010 at 5:33 AM, P.J. Eby pje@telecommunity.com wrote:
At 06:10 PM 2/5/2010 -0800, Sridhar Ratnakumar wrote:
On 2010-02-05, at 4:57 PM, P.J. Eby wrote:
At 04:49 PM 2/5/2010 -0800, Sridhar Ratnakumar wrote:
Do we have a real-world example for installing uninstallable data files outside sys.prefix?
Scripts aren't always in sys.prefix,
Scripts *usually* go to sys.prefix/bin (or sys.prefix/Scripts or site.USER_BASE/Scripts). Are their packages that actually put them elsewhere?
It's not the package that determines installation locations, it's the user's command line options and configuration file settings that determine the locations for libraries, scripts, and data files.
IOW some paths in the PREFIXES file (or the RECORD file as long as we are able to find-and-replace paths in there) *need* to be updated at installation time no matter what we decide to do here.
So, it seems to me that we need to keep unexpanded paths [1] in the RECORD file and let the install command expand them using installation option . But we have to avoid this expansion when doing binary releases.
It seems feasible to me in both cases. And Wolojda could work later on adding more paths definitions and how to configure them through distutils options.
[1] See _INSTALL_SCHEME in http://svn.python.org/view/python/trunk/Lib/sysconfig.py?view=markup that can be read through API (expanded or unexpanded)
Regards Tarek
On Sat, Feb 06, 2010 at 11:16 +0100, Tarek Ziadé wrote:
Hi all,
sorry for joining in so late. I am pretty busy with my thesis and examinations, which is why I can't work on this right now.
I am still subscribed to python-distutils and read the mails regularly, so there is no need to CC me.
IOW some paths in the PREFIXES file (or the RECORD file as long as we are able to find-and-replace paths in there) *need* to be updated at installation time no matter what we decide to do here.
Exactly!
So, it seems to me that we need to keep unexpanded paths [1] in the RECORD file and let the install command expand them using installation option . But we have to avoid this expansion when doing binary releases.
I still think that a PREFIX file, that contains *all* prefixes which are configurable during installation right now, should make it to PEP 376 and to Python 2.7/3.2 . These prefixes could be kept unexpanded in the RECORD file and it would therefore only be necessary to change them within the PREFIX file if one wants to relocate an installation. (Is this really a common use case?)
The PEP lists $PREFIX and $EXEC_PREFIX, but the set of prefixes that are configurable during installation comprises:
Prefix Default value Command line option ----------------------------------------------------------- PREFIX sys.prefix --install-purelib EXEC_PREFIX sys.exec_prefix --install-platlib SCRIPTS_PREFIX $PREFIX/bin --install-scripts DATA_PREFIX $PREFIX/share --install-data
I think that it'll help Linux distribution developers *and* Python developers a lot if these prefixes where available during runtime or could be obtained by parsing the PREFIX file. This would mean the end of __file__ magic to find data files and Python packagers could adapt their tools to copy, for example, a configuration files from $DATA_PREFIX/foo.cfg to a location that complies with the FHS (i.e. /etc/foo.cfg).
It seems feasible to me in both cases. And Wolojda could work later on adding more paths definitions and how to configure them through distutils options.
Will do :-)
Yours sincerely
participants (8)
-
Carl Meyer -
Carl Meyer -
David Cournapeau -
Floris Bruynooghe -
P.J. Eby -
Sridhar Ratnakumar -
Tarek Ziadé -
Wolodja Wentland