[Python-Dev] Status of packaging in 3.3

[David Cournapeau]

On Fri, Jun 22, 2012 at 6:05 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Fri, Jun 22, 2012 at 10:01 AM, Donald Stufft <donald.stufft at gmail.com>
> wrote:
> > The idea i'm hoping for is to stop worrying about one implementation over
> > another and
> > hoping to create a common format that all the tools can agree upon and
> > create/install.
>
> Right, and this is where it encouraged me to see in the Bento docs
> that David had cribbed from RPM in this regard (although I don't
> believe he has cribbed *enough*).
>
> A packaging system really needs to cope with two very different levels
> of packaging:
> 1. Source distributions (e.g. SRPMs). To get from this to useful
> software requires developer tools.
> 2. "Binary" distributions (e.g. RPMs). To get from this to useful
> software mainly requires a "file copy" utility (well, that and an
> archive decompressor).
>
> An SRPM is *just* a SPEC file and source tarball. That's it. To get
> from that to an installed product, you have a bunch of additional
> "BuildRequires" dependencies, along with %build and %install scripts
> and a %files definition that define what will be packaged up and
> included in the binary RPM. The exact nature of the metadata format
> doesn't really matter, what matters is that it's a documented standard
> that multiple tools can read.
>
> An RPM includes files that actually get installed on the target
> system. An RPM can be arch specific (if they include built binary
> bits) or "noarch" if they're platform neutral.
>
> distutils really only plays at the SRPM level - there is no defined OS
> neutral RPM equivalent. That's why I brought up the bdist_simple
> discussion earlier in the thread - if we can agree on a standard
> bdist_simple format, then we can more cleanly decouple the "build"
> step from the "install" step.
>
> I think one of the key things to learn from the SPEC file format is
> the configuration language it used for the various build phases: sh
> (technically, any shell on the system, but almost everyone just uses
> the default system shell)
>
> This is why you can integrate whatever build system you like with it:
> so long as you can invoke the build from the shell, then you can use
> it to make your RPM.
>
> Now, there's an obvious problem with this: it's completely useless
> from a *cross-platform* building point of view. Isn't it a shame
> there's no language we could use that would let us invoke build
> systems in a cross platform way? Oh, wait...
>
> So here's some sheer pie-in-the-sky speculation. If people like
> elements of this idea enough to run with it, great. If not... oh well:
>
> - I believe the "egg" term has way too much negative baggage (courtesy
> of easy_install), and find the full term Distribution to be too easily
> confused with "Linux distribution". However, "Python dist" is
> unambiguous (since the more typical abbreviation for an aggregate
> distribution is "distro"). Thus, I attempt to systematically refer to
> the objects used to distribute Python software from developers to
> users as "dists". In practice, this terminology is already used in
> many places (distutils, sdist, bdist_msi, bdist_rpm, the .dist-info
> format in PEP 376 etc). Thus, Python software is distributed as dists
> (either sdists or bdists), which may in turn be converted to distro
> packages (e.g. SRPMs and RPMs) for deployment to particular
> environments.
>
> - I reject setup.cfg, as I believe ini-style configuration files are
> not appropriate for a metadata format that needs to include file
> listings and code fragments
>
> - I reject bento.info, as I think if we accept
> yet-another-custom-configuration-file-format into the standard library
> instead of just using YAML, we're even crazier than is already
> apparent
>

I agree having yet another format is a bit crazy, and am actually
considering changing bento.info to be a yaml. I initially did got toward a
cabal-like syntax instead for the following reasons:
  - lack of conditional (a must IMO, it is even more useful for cross
-platform stuff than it is for RPM only)
  - yaml becomes quite a bit verbose for some cases

I find JSON to be inappropriate because beyond the above issues, it does
not support comments, and it is significantly more verbose. That being
said, that's just syntax and what matters more is the features we allow:
  - I like the idea of categorizing like you did better than how it works
in bento, but I think one need to be able to create its own category as
well. A category is just a mapping from a name to an install directory (see
http://cournape.github.com/Bento/html/tutorial.html#installed-data-files-datafiles-section,
but we could find another syntax of course).
  - I don't find the distinction between source and build very useful in
the-yet-to-be-implemented description. Or maybe that's just a naming issue,
and it is just the same distinction as extra files vs installed files I
made in bento ? See next point
  - regarding build, I don't think we want to force people to implement
target locations there. I also don't see how you want to make it work for
built files (you don't know the name yet). Can you give an example of how
it would work for say extension and built doc ?
  - regarding hooks: I think it is simpler to have a single file which
contains all the hooks, if only to allow for easy communication between
hooks and code reuse between hooks. I don't see any drawback to using only
one file ?
  - Besides containing the file bits + metadata, I wonder if one should
allow additional fields, that maybe would be tool specific. In bento, there
are a couple of such additional fields that may not be very useful to
others.
  - do we want to allow for recursive dist.yaml ? This numpy.distutils
feature is used quite a bit, and I believe twisted has something similar.

David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120622/3e00b0f1/attachment.html>