On 25 April 2000, Harry Henry Gebel said:

It doesn't look like I will be done with bdist_rpm tonight, here is the patch for the now install options; they are both intended to be run by RPM or other packaging systems from within a .spec file (or equivalent) (although the RPM process may have been launched by another setup.py) Some things I would like different:

I'm not sure I entirely agree with the method this patch. To summarize, for those who didn't take 5 or 10 minutes to read the patch (and 3 months to understand what's going on in distutils.command.install ;-), Harry's patch adds this capability (as I understand it): * when you run "setup.py install --record-install", it installs as usual and writes the list of files installed to INSTALLED_FILES in the current directory * when you run "setup.py install --package-prefix=/tmp --record-install", then it installs to (eg.) /tmp/usr/lib/python1.5/site-packages, and writes the installed filenames -- minus the "/tmp/" prefix -- to INSTALLED_FILES It's the addition of another, somewhat artifical, installation scheme that bothers me about this. I'm also not keen on adding yet another type of "base directory" -- we already have "install_base", "prefix", "exec_prefix", and "home", and now you want to add "root" which is sometimes stuck in ahead of the "install_base". Ugh. Now, the goal is noble: do an installation that looks exactly like what the RPM you're creating will install, except it's not actually in the live Python installation. There's no obvious way to do this with Distutils as it stands, because if you supply --prefix, you have to supply the whole thing -- "/usr/local" or "/tmp/usr" or "/tmp/usr/local" or whatever you want. What you want is a "meta-prefix", which you called the "package prefix" or "root". [...some time passes...] OK, I think I've got it. I've generalized the notion of "configuration variables" to work on the command line, at least the command line of the "install" command. (Config variables will be very important when we start dealing with config files, but they're still useful on the command line. They're even used in the Distutils code: if anyone has looked at the INSTALL_SCHEMES dictionary in install.py, I was *not* smoking drugs, Perl or otherwise, when I wrote that code. They're called configuration variables, and I borrowed the shell/perl syntax because it works. So there.) Anyways, the idea is this: if you want to do a fake install prior to creating an RPM (or what-have-you), you can do this: ./setup.py -n install \ --prefix='/tmp$sys_prefix' --exec-prefix='/tmp$sys_exec_prefix' ...and that does more or less what you might expect. Update your Distutils to the latest CVS -- I'll check this change in shortly -- to see what I mean. Oh, damn! I just remembered that I've already sorta-kinda solved this problem, for the "bdist_dumb" command. There, I did it all in Python code, since there's no need to put shell command in a spec file that will be run later -- so I think the above trick might still be needed for building RPMs. Plus, it's just plain cool, so I think it'll stay unless/until someone thinks of a better way. BTW, the "install" command as I am about to check it in will spew lots of debugging output at you. This is a feature. Please read the output and see if you can grok the latest madness escaping from my fevered imagination. Wouldn't hurt to read the code -- 'finalize_options()' in install.py -- if that's your poison. Brain hurts... off to bed... Greg -- Greg Ward - just another /P(erl|ython)/ hacker gward@python.net http://starship.python.net/~gward/ MTV -- get off the air! -- Dead Kennedys

Ok, I'm jumping in late but, what the hell, here goes both feet. If I'm following the --package-prefix theme properly, it's very much in line with RPM's BUILDROOT, and install tree used for 2 very important reasons. First, the package builder need not be root to produce a binary package that installs in directories that only root can write to. Second, it provides for relocatable binary RPM's. I consider the first helpfull, the second mandatory. If I have several machines with different disk partitioning schemes, I want to be able to install the same RPM's on all of them, even if I have to relocate some modules (or even the entire Python tree) and install time. The manual way I've been doing packages (not RPM's, but Solaris pkgs and HP-UX depots), I have the pre- install scripts find the Python install location and automatically relocate to there. (User can override, of course.) Again, I haven't come up to speed yet, but the addition of a "temporary" install tree is far from artificial. Simply being able to test your packaging process without stepping on your production distribution is worth what little effort is required to support it. While I'm still catching up here: Has anyone stepped up to do either Solaris bdist-pkgtool or HP-UX bdist-depot? If not, it looks like a good way for me to start wading in. Mark Alexander mwa@gate.net On Tue, 25 Apr 2000, Greg Ward wrote:

Date: Tue, 25 Apr 2000 22:22:58 -0400 From: Greg Ward To: distutils-sig@python.org Subject: Re: [Distutils] Patch for new install options

On 25 April 2000, Harry Henry Gebel said:

...

It doesn't look like I will be done with bdist_rpm tonight, here is the patch for the now install options; they are both intended to be run by RPM or other packaging systems from within a .spec file (or equivalent) (although the RPM process may have been launched by another setup.py) Some things I would like different:

I'm not sure I entirely agree with the method this patch. To summarize, for those who didn't take 5 or 10 minutes to read the patch (and 3 months to understand what's going on in distutils.command.install ;-), Harry's patch adds this capability (as I understand it):

* when you run "setup.py install --record-install", it installs as usual and writes the list of files installed to INSTALLED_FILES in the current directory

* when you run "setup.py install --package-prefix=/tmp --record-install", then it installs to (eg.) /tmp/usr/lib/python1.5/site-packages, and writes the installed filenames -- minus the "/tmp/" prefix -- to INSTALLED_FILES

It's the addition of another, somewhat artifical, installation scheme that bothers me about this. I'm also not keen on adding yet another type of "base directory" -- we already have "install_base", "prefix", "exec_prefix", and "home", and now you want to add "root" which is sometimes stuck in ahead of the "install_base". Ugh.

Now, the goal is noble: do an installation that looks exactly like what the RPM you're creating will install, except it's not actually in the live Python installation. There's no obvious way to do this with Distutils as it stands, because if you supply --prefix, you have to supply the whole thing -- "/usr/local" or "/tmp/usr" or "/tmp/usr/local" or whatever you want. What you want is a "meta-prefix", which you called the "package prefix" or "root".

[...some time passes...]

OK, I think I've got it. I've generalized the notion of "configuration variables" to work on the command line, at least the command line of the "install" command. (Config variables will be very important when we start dealing with config files, but they're still useful on the command line. They're even used in the Distutils code: if anyone has looked at the INSTALL_SCHEMES dictionary in install.py, I was *not* smoking drugs, Perl or otherwise, when I wrote that code. They're called configuration variables, and I borrowed the shell/perl syntax because it works. So there.)

Anyways, the idea is this: if you want to do a fake install prior to creating an RPM (or what-have-you), you can do this:

./setup.py -n install \ --prefix='/tmp$sys_prefix' --exec-prefix='/tmp$sys_exec_prefix'

...and that does more or less what you might expect. Update your Distutils to the latest CVS -- I'll check this change in shortly -- to see what I mean.

Oh, damn! I just remembered that I've already sorta-kinda solved this problem, for the "bdist_dumb" command. There, I did it all in Python code, since there's no need to put shell command in a spec file that will be run later -- so I think the above trick might still be needed for building RPMs. Plus, it's just plain cool, so I think it'll stay unless/until someone thinks of a better way.

BTW, the "install" command as I am about to check it in will spew lots of debugging output at you. This is a feature. Please read the output and see if you can grok the latest madness escaping from my fevered imagination. Wouldn't hurt to read the code -- 'finalize_options()' in install.py -- if that's your poison.

Brain hurts... off to bed...

Greg -- Greg Ward - just another /P(erl|ython)/ hacker gward@python.net http://starship.python.net/~gward/ MTV -- get off the air! -- Dead Kennedys

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://www.python.org/mailman/listinfo/distutils-sig

On Tue, Apr 25, 2000 at 10:22:58PM -0400, Greg Ward wrote:

On 25 April 2000, Harry Henry Gebel said:

...

It doesn't look like I will be done with bdist_rpm tonight, here is the patch for the now install options; they are both intended to be run by RPM or other packaging systems from within a .spec file (or equivalent) (although the RPM process may have been launched by another setup.py) Some

* when you run "setup.py install --record-install", it installs as usual and writes the list of files installed to INSTALLED_FILES in the current directory

* when you run "setup.py install --package-prefix=/tmp --record-install", then it installs to (eg.) /tmp/usr/lib/python1.5/site-packages, and writes the installed filenames -- minus the "/tmp/" prefix -- to INSTALLED_FILES

It's the addition of another, somewhat artifical, installation scheme that bothers me about this. I'm also not keen on adding yet another type of "base directory" -- we already have "install_base", "prefix", "exec_prefix", and "home", and now you want to add "root" which is sometimes stuck in ahead of the "install_base". Ugh.

Now, the goal is noble: do an installation that looks exactly like what the RPM you're creating will install, except it's not actually in the live Python installation. There's no obvious way to do this with Distutils as it stands, because if you supply --prefix, you have to supply the whole thing -- "/usr/local" or "/tmp/usr" or "/tmp/usr/local" or whatever you want. What you want is a "meta-prefix", which you called the "package prefix" or "root".

It doesn't seem like an artificial scheme to me, it is a standard method used in many RPMs, in fact almost all RPMs install into a temporary tree in order to avoid the need for root access and to preserve a system's existing installed programs, the '--package-prefix' option provides a simple and direct method of doing this. And if you can fit all of the options on one screen you don't have too many :)

./setup.py -n install \ --prefix='/tmp$sys_prefix' --exec-prefix='/tmp$sys_exec_prefix'

While this works perfectly fine, my problems with it is that it is not pretty and it presupposes that someone reading the spec file has some knowledge of the internal operations of Distutils. It is even uglier as it will appear in an actual spec file (I'm not sure why the above is a dry run, so I left the -n off of this example): ./setup.py install --prefix=$RPM_BUILD_ROOT'$sys_prefix' \ --exec-prefix=$RPM_BUILD_ROOT'$sys_exec_prefix' Compare this with: ./setup.py install --package-prefix=$RPM_BUILD_ROOT Of course, if we assume that the generated spec file will never be seen by human eyes none of this matters, but spec files are looked at by many people, such as people who are making RPMs of patched distributions, people who are trying to learn how to use RPM and people who are simply curious about what is going when they build RPMs. I think in is important to present them with a simple and clear spec file. It is true that many (most) install scripts (and makefile install targets) require just these sorts of shenanigans (and worse) to use in RPM; but as the saying goes, if all those other guys jumped off the St. Georges Bridge ... Also, with this code if you ever changed the meaning of sys_prefix or sys_exec_prefix (or replaced them with something else) people with old source RPMs would no longer be able to build them on machines running a version of Distutils with the new conventions. While the meaning of these attributes may be unlikely to change, I think it is a bad idea in principle to display internals in this manner, and thereby lock yourself into certain ways of doing things. -- Harry Henry Gebel, Senior Developer, Landon House SBS West Dover Hundred, Delaware "Why do you look for the living among the dead? He is not here, but has risen." Luke 24:5 (NRSV)

[my proposed solution to the "fake install" problem]

./setup.py -n install \ --prefix='/tmp$sys_prefix' --exec-prefix='/tmp$sys_exec_prefix'

[Harry begs to differ]

While this works perfectly fine, my problems with it is that it is not pretty and it presupposes that someone reading the spec file has some knowledge of the internal operations of Distutils. It is even uglier as it will appear in an actual spec file (I'm not sure why the above is a dry run, so I left the -n off of this example):

./setup.py install --prefix=$RPM_BUILD_ROOT'$sys_prefix' \ --exec-prefix=$RPM_BUILD_ROOT'$sys_exec_prefix'

Compare this with:

./setup.py install --package-prefix=$RPM_BUILD_ROOT

Hmmm, you have a point. When generating built distributions, you really do want to sneak a new root directory into the installation paths. So how about this minor spelling change: ./setup.py install --root=$RPM_BUILD_ROOT and, instead of having a new artifical "unix_package_prefix" scheme (BTW, *that's* what I was calling artificial, not the idea of a --root or --package-prefix option), a little bit of code somewhere in 'finalize_options()' would do this: if self.root is not None: for attr in ('install_lib', 'install_scripts', ...): setattr (self, attr, os.path.join (self.root, getattr (self, attr))) This would have to be pretty late in the game: after expanding all config vars, after setting 'install_lib', and after 'handle_extra_path()'. So yeah, we would have yet another kind of "base directory", which still bothers me a little bit. However, it would only be needed for generating built distributions, and use of it would usually be buried inside one of the bdist_* commands... so I guess I can live with it. I'm going to have to think about how to fix/simplify the 'bdist_dumb' command now -- it does a lot of this stuff on its own, and it probably should be done by the "install" command. One useful side-effect is that this will fix the bug whereby .pth files don't get included in built distributions, since the mock install is done by the "install" command, as it should be. Yeah, OK, I'm starting to like this. Grudgingly. ;-) Greg PS. using "--prefix='/tmp$sys_prefix'" won't require knowledge of Distutils internals, since the configuration variables will be a documented part of the interface. I admit that it's still a tricky and specialized part of the interface though, and it would be best to restrict config vars to the config file if at all possible. It just seems bogus to *disallow* them from the command line when they might be useful, hence last night's hack. Err, patch. -- Greg Ward - programmer-at-large gward@python.net http://starship.python.net/~gward/ UFO's are for real: the Air Force doesn't exist.

On 25 April 2000, Mark W. Alexander said:

If I'm following the --package-prefix theme properly, it's very much in line with RPM's BUILDROOT, and install tree used for 2 very important reasons. First, the package builder need not be root to produce a binary package that installs in directories that only root can write to. Second, it provides for relocatable binary RPM's.

Yeah, I think Harry and you won, although I implemented it differently from Harry's --package-prefix patch. Try "install --root=/tmp" and see how it works for you. (It works for me)!

While I'm still catching up here: Has anyone stepped up to do either Solaris bdist-pkgtool or HP-UX bdist-depot? If not, it looks like a good way for me to start wading in.

No, please wade right in. You might want to wait a few days until I have fixed "bdist_dumb" to use the latest changes to "install", and until Harry and I agree on what "bdist_rpm" should look like. Or go ahead and start playing around, but be prepared to hack your "bdist_xxx" commands up when I change my mind about how things should look. Yeah, on second thought, start playing around... rearranging Python code is cheap. Greg -- Greg Ward - Unix geek gward@python.net http://starship.python.net/~gward/ And I wonder ... will Elvis take the place of Jesus in a thousand years? -- Dead Kennedys

On Wed, Apr 26, 2000 at 10:00:20PM -0400, Greg Ward wrote:

On 26 April 2000, I said:

...

./setup.py install --root=$RPM_BUILD_ROOT

OK, I've added this option to the install command. Works for me, but it will blow up on Windows and Mac OS because I'm not sure of the right way to forcibly set a new root directory on a path that might already be absolute. Probably just because my brain is running at a low ebb tonight. Anyways, take a look in distutils/util.py at 'change_root()' in the latest CVS if you can help me.

This is basically why --package-prefix didn't support Mac and Windows, because don't know much about Windows path conventions, or anything about Mac, so I thought it would be better to let someone who was familiar with those platforms fill it in. -- Harry Henry Gebel, Senior Developer, Landon House SBS West Dover Hundred, Delaware PyNcurses ncurses binding for Python: http://pyncurses.sourceforge.ne