configure/build/packaging problems
I picked up responsibility for the mailman rpm here at Red Hat a little while ago, just about the time of the 2.1 release. As a consequence I don't have a history with the prior releases, but I seem to have run into some build and packaging problems that I think were introduced in 2.1 and then carried over into 2.1.1.
I believe as of 2.1 it became a requirement that the mailman user and group exist before the configure script is run. As far as I can tell this is because the configure script looks up user ids and/or user names in the system passwd database when its building its set of substitution strings for the .in files.
One of the problems I've encountered is that production of the mailman RPM is done on machines where the set of users and groups are fixed for a variety of package production robustness reasons and I've lost the battle and have been told mailman must build without requiring the existence of a specific user. Whether one has sympathy for the rule or not is of little value, it is what it is. By the way, this problem just came to light as Red Hat users upgraded to Red Hat 9 which includes the mailman 2.1 release. In the interim I have produced a 2.1.1 rpm, but that is only available as a separate download. The problem was that some build machines had the user/group 'mailman' predefined and others did not, so some rpm's were good but others ended up with bogus configurations.
I've studied the configure script in some detail and I think I understand the areas that need fixing but before I do that I'd like to ask some questions to sanity check my thinking before I go down the wrong rat hole.
At the heart of the mailman build philosophy seems to be the notion that mailman will (must?) be built on the target system. This seems to be in opposition to packaging concepts where a software component is built on an unrelated build machine, the resulting files gathered into an installation package, and then that package is installed on the target. Given that packaging is a common distribution mechanism I assume the approach of requiring the package to build on the target was a deliberate choice for some reason. If so what was the reason? If there isn't a compelling reason for the approach are you open to receiving patches that make the configure/build step friendly to packaging requirements or does this violate some principle mailman is trying to maintain?
I've noticed that some of the checks and lookups done in configure can be controlled with with-permcheck & without-permcheck. But not all the lookups are under the control of this variable. My initial thoughts were to have ALL the local lookups under the control of this variable, if without-permcheck is set then NO local lookups during configure/build are performed, configuration values are used as supplied. Does this sound reasonable? Is this overloading this variable with meaning that was not intended?
I've also run into problems where the .py files cannot be compiled into .pyc files on build system during the build. The culprit is the paths.py file which is included by quite a few of the python files. The problem is that paths.py has a hardcoded pathname (generated by the configure step) that only exists in the post-installed target. It does not exist on the build machine. Therefore some of the compiles fail because some of the source python files include files via paths.py that don't exist (yet). This means the compilation needs to be postponed to post-installation on the target. My understanding is that the python interpreter will drop a .pyc file in the source directory the first time it compiles the .py file so in theory its like a lazy compilation. But this has a few problems too, the python process has to have permission to write in that directory and it violates some packaging requirements that every file belonging to a package exists and is known at both package install and package remove time. I haven't really looked into what is required to fix this problem yet but it seems like we have to get rid of the hardcoded pathnames in paths.py and if someone has a suggestion I'd be curious to hear. I'd really like to be able to compile the python files during build before producing the installation package.
John
On Mon, 2003-04-07 at 16:16, John Dennis wrote:
I picked up responsibility for the mailman rpm here at Red Hat a little while ago, just about the time of the 2.1 release. As a consequence I don't have a history with the prior releases, but I seem to have run into some build and packaging problems that I think were introduced in 2.1 and then carried over into 2.1.1.
Hi John. Say, were you at PyCon a while back? I don't remember if I met you there or not.
I believe as of 2.1 it became a requirement that the mailman user and group exist before the configure script is run. As far as I can tell this is because the configure script looks up user ids and/or user names in the system passwd database when its building its set of substitution strings for the .in files.
Yes, but there is a --without-permcheck option to configure that will bypass the check. In 2.1.1 you'll get a traceback from the configure script, which I think you can ignore. I'm about to check in a fix for that which should be part of 2.1.2.
- At the heart of the mailman build philosophy seems to be the notion that mailman will (must?) be built on the target system.
Traditionally this has been true, yes. I would definitely be willing to make Mailman easier to package, as long as we can continue to ensure adequate security.
I assume the approach of requiring the package to build on the target was a deliberate choice for some reason. If so what was the reason?
Much of the build process is historical. It works for me but I won't claim it's optimal.
If there isn't a compelling reason for the approach are you open to receiving patches that make the configure/build step friendly to packaging requirements or does this violate some principle mailman is trying to maintain?
Sure.
- I've noticed that some of the checks and lookups done in configure can be controlled with with-permcheck & without-permcheck. But not all the lookups are under the control of this variable. My initial thoughts were to have ALL the local lookups under the control of this variable, if without-permcheck is set then NO local lookups during configure/build are performed, configuration values are used as supplied. Does this sound reasonable? Is this overloading this variable with meaning that was not intended?
Which checks did you have in mind specifically? Mailman 2.1 encodes symbolic user and group names now, so as long as mailman:mailman exists on the target system (which I hope you agree is not unreasonable), everything should still work. The intent was definitely to have --without-permcheck support packagers building on systems without that group and user.
- I've also run into problems where the .py files cannot be compiled into .pyc files on build system during the build.
but it seems like we have to get rid of the hardcoded pathnames in paths.py and if someone has a suggestion I'd be curious to hear. I'd really like to be able to compile the python files during build before producing the installation package.
That's going to be a problem too, because you're in a catch-22. The whole point of the paths.py files is to add hardcoded paths to sys.path so all the other modules can be found. I can't really think of any other way of doing it.
Here's a suggestion though: you already /know/ what .pyc files will exist on the target machine after the compilation phase is complete, so can you add those to the rpm's list of files it knows about anyway? Then the post-install phase can do the compileall to build the pycs. You definitely don't want to leave things to lazy byte compilation.
I'm up for any other suggestions you might have to make 2.1 easier to package. I'm also up for any longer term (read: more disruptive ;) changes that might be appropriate for a 2.2 or 3.0 release. I'm leaning strongly toward adopting some of the Zope3 build mechanisms (i.e. largely setup.py based).
-Barry
participants (2)
-
Barry Warsaw -
John Dennis