Phillip J. Eby wrote:

At 08:42 AM 7/4/2006 +0200, Matthias Klose wrote:

...

Phillip J. Eby writes:

...

...

what about installing the .egg-info directory without version information when --single-version-externally-managed is used?

Note that the .pyc files will be built for a specific Python version; that's why the version number is there.

yes, but this shouldn't be necessary if you install into a path which is known to belong to a specific python version (i.e. /usr/lib/python2.4/site-pacakges)

Setuptools doesn't inspect the path it's installing to in order to know whether it happens to contain the Python version already. :) I don't see what problem you're trying to solve here.

The problem he's trying to solve is that the new Debian Python policy allows a single directory tree of .py files (not the .pyc and .pyo files) to serve all installed Python versions. The .pyc files get placed in a separate tree which is, obviously, versioned. I don't see what problem you're trying to solve by having the python version number in the .egg-info/ directory name. :) Is there any Python-version-specific info the in the .egg-info directory? How 'bout in the case when it is installed with --single-version-externally-managed command, which what we're talking about with Debian, anyway? Anyhow, this is also creating an issue for me with stdeb because I'm trying to get the generated source packages as close as possible to Debian policy. Cheers! Andrew

Phillip, thanks for your patient attempts to make this all clear. I appreciate it. Some more comments at the end... Phillip J. Eby wrote:

At 11:51 AM 7/21/2006 -0700, Andrew Straw wrote:

...

I don't see what problem you're trying to solve by having the python version number in the .egg-info/ directory name. :) Is there any Python-version-specific info the in the .egg-info directory?

Yes.

...

How 'bout in the case when it is installed with --single-version-externally-managed command, which what we're talking about with Debian, anyway?

Yes.

...

Anyhow, this is also creating an issue for me with stdeb because I'm trying to get the generated source packages as close as possible to Debian policy.

Unfortunately, Debian's policy -- especially the idea of mixing paths for multiple Python versions -- doesn't mesh well with reality. Python projects don't always install the same files for a given Python version, or the same content in the same files. Setuptools is only a relatively minor example of this; other Python projects do far more customization of the install process (by Python version and dependencies) than setuptools does of its own. Bluntly, mixing Python versions on the same path for *installed* packages (whether installed by the distutils or setuptools) borders on insanity.

(More precisely, it sounds like exactly the sort of thing to do if one wanted to create all sorts of problems for which one could then claim credit for fixing as part of a "quality assurance" effort, constantly patching upstream packages to work around the problems, thereby cementing in one's own mind the importance of doing QA work to fix those upstream authors' shoddy work that fails to live up to your distribution's "quality" standards. And yet somehow, nobody will ever seem to notice that it's the policy *itself* that's causing the problems, because of *course* those upstream developers know *nothing* about quality assurance... sigh. But that's a rant for another day.)

Anyway, in this case at least, the reason you're running into a problem is because there's an inherent problem here, not because setuptools is doing something it shouldn't be.

OK, I understand your point. Upon re-reading the new Debian python policy, I see that the idea of having .py files installed in one tree for multiple python versions is optional (albeit "strongly encouraged" as stated at http://www.debian.org/doc/packaging-manuals/python-policy/ap-packaging_tools...). So, I can personally continue to ignore this supposedly optional aspect of the new Debian Python policy, but I don't know for how long. For example, recent discussions on the debian-python list suggest to me that the use of this new policy is very strongly encouraged rather than merely "optional". Somebody please enlighten me if I'm wrong. I've said it before and I'll say it again -- I'm no Debian (Python) policy expert. So, to ask your advice, then, how would _you_ suggest stdeb install Python packages using the Debian system? Plain old "--single-version-externally-managed" without any .egg-info directory renaming? That would be simple enough for me...

Phillip J. Eby writes:

I read the entire policy you linked to, and I don't actually see many problems.

It seems to me that the single largest problem in that policy is that it clearly predates the existence of the distutils. It has no conception of a Python *project* or *distribution*, only modules and packages. It's therefore not surprising that it also doesn't encompass such issues as distribution metadata, package data, namespace packages, and the like. It also explains why the policy is so out-of-sync with e.g. PyPI. (I hesitate to see what would happen if somebody tries to package any of my Python projects such as SymbolType or ProxyTypes for Debian: they all are modules in the 'peak' package, but each is distributed as a separate project!)

The Python policy is just a sub-part of the Debian Policy [1]; the Debian Policy predates PyPi. You are missing the existing bits about i.e. distribution metadata, distributions, etc. [1] http://www.us.debian.org/doc/debian-policy/ I cannot find the term "project" in the distutils documentation. Any pointers? So yes, if peak is a rather complex setup, it might be worthful to have it as an example for a Debian package and to identify omissions in Debian packaging practices and distutils/setuptools.

From the extremely limited conceptual position that only modules and packages exist, it's also impossible to understand how you can have different versions of things for different versions of Python, because everything is defined in terms of the modules! However, project distributions are merely *composed* of modules, and what modules get installed (or what distribution metadata is generated) may depend on the Python version. The same project may also have different dependencies, depending on the Python version.

It is not the goal for Debian to ship every known Python version; we try to ship a minimal count of Python versions, which is current and works for most applications. We discourage shipping deliberate versions of python modules/packages. IIRC, for the past release of Debian sarge, we had two versions of pygtk and two versions of wxwidgets. So it seems that it's quiet possible to do that. I'm not sure what you mean by the generation of distribution metadata and different dependencies.

These concepts can't be well-understood from the perspective that only modules and packages exist, so until the policy's conceptual underpinning is expanded, it's going to continue to be difficult to squeeze square pegs into the policy's round holes.

agreed, but it cannot be as open as the possibilities of distutils/setuptools are. Python packages (in the Debian sense) still have to follow [1] and general decisions made by release management.

...

So, to ask your advice, then, how would _you_ suggest stdeb install Python packages using the Debian system? Plain old "--single-version-externally-managed" without any .egg-info directory renaming? That would be simple enough for me...

That is what I would recommend, yes. Installed, of course, to the standard Python directories, rather than to a shared one, since that would lead to conflicts when alternate versions exist.

I don't know enough about Debian to know what it does with package naming, but I do know that e.g. NetBSD package naming conventions include the Python version for which a project is packaged, such that you could have py22-foo, py23-foo, and py24-foo of the same project "foo".

That naming scheme was used until recently. As already said, it had several disadvantages for Debian: - manual interaction for each new package name entering the Debian archive - big bang transitions between unstable and testing releases. - higher space requirements for ftp archives, ftp mirrors, local installations, more band with usage - old packages left after package upgrades

IMO, any other way of doing it is an accident waiting to happen... or else a "job security" bonus for the people who write the tools and fix the problems. I guess it's always better to look at it from the bright side. ;)

Well, let's see what will happen and lets count the accidents then ;) Many problems that PyPI and setuptools try to solve are well addressed by existing packaging tools for Linux and *BSD distributions. It would be nice to see setuptools to use this infrastructure where available. Linux and *BSD distributions do integrate software well, PyPI and setuptools might do that job better for Python things, but just for Python things. I'm sure you can improve things from both sides. Matthias

At 12:08 AM 7/22/2006 +0200, Matthias Klose wrote:

...

I read the entire policy you linked to, and I don't actually see many

Phillip J. Eby writes: problems.

...

It seems to me that the single largest problem in that policy is that it clearly predates the existence of the distutils. It has no conception

of a

...

Python *project* or *distribution*, only modules and packages. It's therefore not surprising that it also doesn't encompass such issues as distribution metadata, package data, namespace packages, and the like. It also explains why the policy is so out-of-sync with e.g. PyPI. (I hesitate to see what would happen if somebody tries to package any of my Python projects such as SymbolType or ProxyTypes for Debian: they all are modules in the 'peak' package, but each is distributed as a separate project!)

The Python policy is just a sub-part of the Debian Policy [1]; the Debian Policy predates PyPi. You are missing the existing bits about i.e. distribution metadata, distributions, etc. [1] http://www.us.debian.org/doc/debian-policy/

I'm referring above to Python distribution metadata, not Debian's. That is, the distribution of a Python project, not a Linux distribution.

I cannot find the term "project" in the distutils documentation. Any pointers?

I use the term "project" to refer to the logical thing of which a distutils "distribution" is a physical manifestation. The distutils documentation confusingly uses the word "package" to refer both to what I'm calling a "project", and the notion of an individual Python package. You can tell this by close inspection of the distutils documentation, if you notice that there are many places where the configuration of a "package" (meaning #1) in fact can list multiple "packages" (meaning #2) for inclusion. (Many Python developers have previously commented on this naming ambiguity in the distutils.) I thus prefer to use (and promote the use of) the word "project" for meaning 1, in order to have better communication about what is actually going on. It is intuitive and does not confuse two different notions of "package".

So yes, if peak is a rather complex setup, it might be worthful to have it as an example for a Debian package and to identify omissions in Debian packaging practices and distutils/setuptools.

This very statement helps to illustrate the impedance mismatch and communication difficulty. You appear to be interpreting my statement that e.g. SymbolTypes and ProxyTypes contain modules in the "peak" package (meaning #2) to imply that peak is or should be a Debian package (meaning #1, or perhaps a new meaning #3!). But this would be the same as concluding that the various Java projects whose packages are contained in the org.apache.* namespace are a single "package" in this way, and should thus be combined into a Debian "org-apache" package -- split out of their respective jars and forcibly recombined! But as with the org.apache.* prefix, the peak.* prefix is merely a namespace that indicates an affiliation and prevents unintentional name conflicts. As the number of distributed Python projects increases (1400+ on PyPI of late), this kind of namespace management will become increasingly important. This is where the term "namespace package" comes from; it was coined several years ago by Zope to distinguish package-as-unit from package-as-namespace. The latter kind of package is still relatively uncommon, since there are relatively few large organizations distributing large projects as split distributions. Unbundling these large projects into smaller pieces is an increasing trend, however, as it allows smaller units to be spun off as sub-projects, each with its own release management and version lifecycle. PEAK and Zope as monolithic projects might contain some elements in alpha or beta releases that, considered in their own right, might be worthy of 1.x or 2.x stable version labelling. Lumping these in together with other components doesn't help anybody, but spinning them off as separate projects allows them to be reused. So far, I've spun off seven such packages from the monolithic PEAK distribution, and there will be more over time. These other packages live in the peak.* namespace, and the monolithic distribution depends on them, but it would not make sense to aggregate them all as one Debian package, since other packages may depend directly on them. SymbolTypes, ProxyTypes, and DecoratorTools are all likely to get used in other projects that would depend on them directly, but not necessarily require any other part of the peak.* namespace. And this, you see, is why I say that the Debian Python policy is based on a limited conceptual framework that doesn't mesh well with a distutils- or setuptools-based world. Mapping "1 Debian package = 1 Python package = 1 project" is inaccurate, because one project may contain multiple Python packages, and a single Python package can be spread across multiple projects. And it's not just PEAK and Zope doing it -- I discovered last year that there's an ll.* namespace package out there that uses an interesting quirk of the distutils installation system to implement a namespace package. It might actually predate Zope's coining of the term "namespace package" for all I know.

I'm not sure what you mean by the generation of distribution metadata and different dependencies.

The PKG-INFO format changed in some Python versions. The entry points that setuptools offers as commands depends on what Python version it's installed with. The dependencies of some projects depend on whether a needed thing is now bundled in Python. For example, there is a standalone "ctypes" project, that is bundled in Python 2.5. A project being installed for 2.5 would have no reason to declare a dependency on ctypes, and it would be entirely reasonable for a setup.py to contain something like: deps = [] if sys.version[:3]<"2.5": deps.append("ctypes>=0.9.6") ... setup( ... install_requires = deps, ... ) Similar code might decide to build alternate extensions, etc. The monolithic PEAK distribution for example includes its own version of the Python expat module, and builds it to replace Python's if it's being installed for Python 2.3 (whose expat interface doesn't include access to the current line number during regular parsing callbacks). (Note: it builds this backported extension as peak.util.pyexpat; it doesn't override the stdlib-supplied pyexpat!) Anyway, now that setuptools exists, the right way for me to have handled that would be to create a separate project, let's say "pyexpat-backport" that provides the 2.4 expat interface for Python 2.3, and then declare that as a dependency if I'm installing under Python 2.3 -- a Python-version-conditional dependency.

...

These concepts can't be well-understood from the perspective that only modules and packages exist, so until the policy's conceptual underpinning is expanded, it's going to continue to be difficult to squeeze square pegs into the policy's round holes.

agreed, but it cannot be as open as the possibilities of distutils/setuptools are. Python packages (in the Debian sense) still have to follow [1] and general decisions made by release management.

You'll have to clue me in as to which meaning of "package" you're using here. I personally try to use the following terms to be unambiguous: 1. "Project" - a thing that somebody distributes 2. "Python package" - something you can actually import! 3. "System package" - something that is installed with a system packaging tool, like a .rpm 4. "Distribution" - an embodiment of a particular release of a project As far as I can tell, Debian terminology conflates some of these terms. And so long as its vocabulary is thus restricted, there will be an impedance mismatch at the interface where people try to create tools to support mapping #1 and #4 on to Debian's #3.

Many problems that PyPI and setuptools try to solve are well addressed by existing packaging tools for Linux and *BSD distributions.

A similarity in solutions is not the same as similarity in problems. The goals of a system packaging tool and the goals of setuptools are quite different, and in some cases may actually be opposed. :) Setuptools' fundamental goal is to encourage reuse by lowering the transaction cost of depending on another developer's software. Not merely in the sense of lowering *distribution* or *installation* cost, but also enhancing the extensibility and interoperability of the projects themselves. Metadata and entry points facilitate creating *platforms* in Python, such as the joint TurboGears-CherryPy template plugin API. That API couldn't exist without something like setuptools; system packaging tools simply don't play in that space. Now, furthering setuptools' goals *does* require distribution and dependency management... but its "low transaction cost" goal means that it requires a *common* nomenclature for referencing projects. A nomenclature that varied from one packaging system to another would not lower transaction cost, since it would force a developer to learn the ever-changing and mutually incompatible naming conventions of every Linux and BSD variant. The only universal nomenclature available, therefore, was project names. The distutils built distributions using project names, and PyPI displayed project names. Hence, it was and is the right choice for Python to identify projects by those names. Distributions, however, that insist on deconstructing Python projects and creating nomenclature with no mapping to PyPI project names, simply create a policy barrier between those upstream projects and ready access by their users. It increases the transaction cost for providing software to Debian users -- and Debian of course ends up bearing those costs. The efforts of people like Andrew and Vincenzo to create tools that map PyPI projects into Debian packages are therefore in vain; Debian doesn't want to decrease transaction cost, which then leaves the tool developers confused, since their goal is to further reduce transaction costs. I myself was initially baffled by this resistance from Debian representatives, but now I simply accept it as a fact that Debian's goals differ from mine. I do think it's unfortunate, though, because other people seem to keep thinking that they will be able to write a conversion tool and solve a sociopolitical/conceptual problem with a technical solution. It just ain't gonna happen. :) (I don't mean it's unfortunate that Debian has different goals, I just mean it's unfortunate that this fact isn't immediately obvious to the people who keep beating their heads on this particular wall. You can't work to lower the impedance between PyPI and Debian, and still please Debian policy, because the policy itself is the source of the impedance.)

It would be nice to see setuptools to use this infrastructure where available.

The --single-version-externally-managed option exists so that setuptools can get out of system packaging tools' way. There's also extensive work that I did to make namespace packages play well with system packaging tools that don't allow more than one system package to provide the same file, although this required what some Pythoneers would consider a horrific abuse of Python's .pth file system. These things were done because people doing work for Debian asked for them, and if anybody asks nicely for other things that I can provide, I'll certainly do so. However, some things just aren't doable. I can't, for example, turn back the clock seven years and make the distutils go away, or even four years to make namespace packages go away, just because Debian policy doesn't grok those concepts yet, or refuses to acknowledge their validity. Even if I agreed with Debian on these points (and I don't), the Python community voted with its feet years ago, and Guido blessed all of them. The distutils were blessed for the stdlib in what, Python 2.1? Namespace packages were blessed for 2.3 (see the "pkgutil" module docs, although they use the term "logical package"). (Guido himself wrote that module, if I recall correctly.) Support for package data (data files found inside package directories) was added in 2.4, and .egg-info distribution metadata was blessed for 2.5. From the Python POV, most of this stuff is ancient history by now.

At 10:46 PM 7/21/2006 +0200, Matthias Klose wrote:

again, they are not mixed. I didn't see that many packages with different files between Python versions. could you give some examples?

The one that's most obvious to me is of course setuptools itself; its installation metadata changes between Python versions, to reflect differences in the distutils between Python versions. The most common case of a project shipping different modules are projects that depend on features that were part of the stdlib in a newer Python version, but must be bundled for an older Python version. e.g., the logging module, optparse/optik, doctest, the threading.local facility, etc. I seem to recall Zope releases doing this a lot.

- remove included third party modules and depend on the modules instead which are shipped in the distribution (removed three copies of pytz in the last months). Maybe it's the desire of upstream authors to provide a complete one-click-setup lacking a decent package management system for other platforms, but it should be an option, not the default.

Luckily, setuptools permits improvement in this area, and certainly I agree that the dependencies *should* be made separate. That's not always possible, however, if you are dealing with the existing base of Python projects.

- add exact meta information (dependencies) that allows upgrades, even partial ones, between distribution releases. It would be nice to use upstream meta information, without relying on a newly invented dependency tracking system which doesn't integrate well with existing packaging systems. I think the PEP trying to get this information into the meta-data was rejected as over-specified.

Actually, it was only over-specified on *syntax*. The *semantics* on the other hand, are badly *under*-specified. The version syntax was so narrowly-defined that *Python's own* version numbers wouldn't be considered conformant, but the meaning of the names was left entirely unspecified! I could say I wanted "Foo>1.2.7", but there was no definition of what "Foo" would mean. Regarding the rest of your comments (e.g. FHS), I think it would be counterproductive for me to respond in detail, because I do understand that these things that are meaningful within the Debian world-view. However, they are self-fulfilling prophecies in the sense that these forms of "QA" works primarily to ensure the need for more of the same "QA". If this is what is valued, then by all means, feel free to go for as much of it as you like. :) Where this leads to friction with outside work (like Andrew's stdeb) is that the Debian viewpoint doesn't encompass the idea that the proper division of Python modules is according to the projects that provide them. This is an impedance mismatch that means that any tool designed to help get Python projects into Debian (like easy_deb or stdeb or even bdist_deb) is never going to be both reasonably automated *and* reasonably policy-compliant, because the conceptual framework on which the policy is based cannot encompass the idea of projects. It seems that to Debian, the only software project is Debian itself, and anything else is merely a source of components to be broken down and reassembled in Debian's own image. This may or may not be what you want, of course, and I certainly don't object to people doing what they want. :) I'm bringing this up, however, in the hopes of saving more people from slaving away trying to build Python packaging tools for Debian based on a misconception that their work will ever be accepted (or even understood) in the scope of Debian's Python policy, which effectively precludes any useful mapping from PyPI to Debian.