Re: [Distutils] distlib updated - comments sought
The issue I had raised was attention to the needs of Linux packaging and
the Filesystem Hierarchy Standard, and apparently that is under
consideration.
The response I received (from "Daniel Holth"
Message: 3 Date: Fri, 5 Oct 2012 15:49:04 +0000 (UTC) From: Vinay Sajip
To: Distutils-Sig@Python.Org Subject: Re: [Distutils] distlib updated - comments sought
Stanley A. Klein
writes: I looked at the documentation and can't completely follow what you are doing, although I was told in another email that the issue I raised was being addressed in pkg_resources.
The goal of distlib is to provide a library of low-level functions relating to packaging and distribution of Python software, and it does not contain any installers (though it will contain low-level code which could be used by an installer). The distlib code aims to serve as an implementation of the various packaging PEPs, such that if higher level tools use distlib, they should be able to both save writing code (and thereby gain time) as well as achieve better interoperability with other packaging tools.
The goal of distlib is not to be a complete packaging solution (which packaging tried to be for Python 3.3, but it proved to be too ambitious a goal).
Which issue are you referring to, which you were told was addressed by pkg_resources? From the section starting "Here is what I need", I couldn't see how pkg_resources could fulfil those needs by itself.
Here is what I need: [snip] I can't completely see how I would do this using what is described in the distlib documentation.
What you need is an installer, and distlib doesn't contain functionality to do installation. I won't rehash the various discussions around the needs of Linux distro packagers, except to say that the PEPs have taken into account the needs of distro packagers and the ability to map various parts of a distribution to various target directories in a distro-specific manner.
Regards,
Vinay Sajip
On 5 October 2012 20:28, Stanley A. Klein
What I didn't notice in the distlib documentation were low level functions that would facilitate the allocation of Pypi files/directories to target files/directories to help in preparation of the rpm spec.
While I'm not particularly familiar with Linux filesystems and FHS, I believe the key piece of the puzzle is in the stdlib, the sysconfig module. The "PyPI files/directories" as you call them (actually, the Python standard locations) are not specific directories, but rather are given by logical "paths" - platlib, scripts, data, and so on. Mapping these to OS-dependent locations can be handled by the OS installer code (RPM, I guess, in the case of Red Hat and similar). Properly written Python packages will only use those standard names. I hope that helps. Paul.
On Fri, Oct 5, 2012 at 3:28 PM, Stanley A. Klein
The issue I had raised was attention to the needs of Linux packaging and the Filesystem Hierarchy Standard, and apparently that is under consideration.
The response I received (from "Daniel Holth"
) said in part "The FHS issues are on the list, which is why there are "resource categories" that can be installed wherever in new-packaging land." I assume that by an "installer" you mean the combination of package building (done by e.g., rpmbuild) and installation (done by e.g., rpm or yum).
The mapping of files from the Pypi structure to the target directories is done at package-build time and not usually at install-time. What a function like bdist_rpm does is to create an rpm spec and turn the package building function over to rpmbuild. The file mapping is defined in the rpm spec and takes place during the rpmbuild processing.
What I didn't notice in the distlib documentation were low level functions that would facilitate the allocation of Pypi files/directories to target files/directories to help in preparation of the rpm spec.
I don't think that is in distlib's problem domain at the moment; the information about which file should go where is not generally present in setup.py at all. It was part of the distutils2/packaging effort to define a setup.cfg [files] section. Similar to http://docs.python.org/distutils/setupscript.html#installing-additional-file... but with better definitions for the category names, and a config file to show where they should go. Bento uses the autoconf names http://cournape.github.com/Bento/html/tutorial.html#adding-data-files, http://cournape.github.com/Bento/html/reference/bento_format.html#available-..., e.g. prefix: install architecture-independent files eprefix: install architecture-dependent files bindir: user executables sbindir: system admin executables ... The next build system that's made, or Bento right now, will provide for finding those files again at runtime even if they have been flung to the four corners of the disk. The wheel spec should probably be updated to understand something about packagename-1.0.data/bindir etc. When you are build an RPM, install-time from the Python perspective would indeed be build-time from the RPM packager's perspective. Daniel
On 5 Oct 2012, at 20:28, Stanley A. Klein wrote:
I assume that by an "installer" you mean the combination of package building (done by e.g., rpmbuild) and installation (done by e.g., rpm or yum).
Hi, I think you've hit the jackpot hereā¦ Here few lesson learnt using rpm and python at fairly large scale (http://download.opensuse.org/repositories/home:/cavallo71:/opt-python-interp...) project of mine I wish could be of some help in thinking of a "packaging system". (I'm totally happy with the old-pain setup.py and I see no need to switch to anything else). * Installation != Configuration * running scripts at "install" time transforms the installer into a development process (I'm finger pointing the retarded dpkg debian way). Good rpm packages don't have *any* script at all. * Don't cross roles border * Files don't get created into the void: the act of "installing" shouldn't be no more than transferring file to a filesystem plus adding metadata information in a database eventually (the rpm database, the windows registry etc.). Adjusting paths/files during the install (msiexec /i *.msi, rpm -i *.rpm etc.) is not the right way. * I'm not expecting tar to download packages from internet, so rpm * Having dependency resolution (and trying to install packages within a setup.py of any sort) is outright retarded although seems to fit somebody agenda. There're already plenty of tools in the os to do that e.g.. the dependency resolution between rpms is handled at yum level with a tiny help from the rpm. I don't want that placed in the wrong place. BTW rpmbuild is a tool part of rpm, technically users (as sysadmin) shouldn't needed for managing packages.
On Sat, Oct 6, 2012 at 3:44 AM, Antonio Cavallo
Here few lesson learnt using rpm and python at fairly large scale (http://download.opensuse.org/repositories/home:/cavallo71:/opt-python-interp...) project of mine I wish could be of some help in thinking of a "packaging system". (I'm totally happy with the old-pain setup.py and I see no need to switch to anything else).
I agree that setup.py should be replaced and that Python packaging should be better in various ways, but I think the process of replacing the build system should take 5-10 years. I do not want to fork the 27 working dependencies of my simplest project by rewriting their setup.py and replacing any "import pkg_resources" with a different API. Instead, I will use binary packaging to forget about the build system at install time and to install much more quickly, and if setup.py is a problem, at least it's a problem less often (because 'install' is a much more common operation than 'build'). If it becomes necessary, I (or someone) will re-implement the most used parts of the pkg_resources API with a different back end. The strategy has nothing to do with liking or disliking the existing popular APIs. It is only because it takes less time than debugging and maintaining forks of my dependencies due to a subtly incompatible new build system. (If the new build system was not incompatible, it would not be able to improve on setuptools and distutils.) Eventually everyone will realize that we should be building with Bento, and we will get better APIs that do what pkg_resources does without confusing people with a very long single source file and the new systems will become preferred. In the meantime there is going to be more than one way to do it for a long time.
participants (4)
-
Antonio Cavallo
-
Daniel Holth
-
Paul Moore
-
Stanley A. Klein