[Catalog-sig] [Distutils] pre-PEP : Synthesis of previous threads, and irc talks + proposals

[Phillip J. Eby]

At 07:55 PM 10/1/2008 +0200, Tarek Ziadé wrote:
>On Wed, Oct 1, 2008 at 7:29 PM, Phillip J. Eby <pje at telecommunity.com> wrote:
> > I'm -1 on all of the above.  I think we need a standard for tools interop
> > (ala WSGI), not implementation tweaks for the existing tools.  I also think
> > that a concrete metadata format proposal is premature at this time; we've
> > barely begun to gather -- let alone specify -- our requirements for that
> > metadata.  (Essentially, only version dependencies have been discussed,
> > AFAICT.)
>
>What are the other important points we need to discuss at this point
>in your opinion ?

What information needs to be conveyed by a "build" tool to an 
"install" tool, and vice versa.

For example, an install tool needs to know what files are 
documentation, which are sample data, and what is part of the library 
proper, as well as what things are scripts (and in what language 
those scripts are written, e.g. Python, shell, batch, etc.).  Some 
install tools need to know about icons, menus, registry info, cron 
jobs, etc.   (These are perhaps more properly the domain of 
applications than libraries, but I'm going to assume that these 
things are in scope.)

The way that this information is communicated needs to be extensible, 
so that optional metadata for debs and msi's and rpm's and 
whathaveyou can be incorporated, without needing to modify the 
standard -- especially if the APIs for reading and writing this data 
are in the stdlib.

There needs to be a way for install tools to ask a source package to 
"configure" itself, possibly specifying options (and a way for it to 
find out what those options are, to be able to present them to the 
user).  Then there needs to be a way for the install tool to ask the 
package to build itself with the configured options, and a standard 
for how the build tool(s) communicate errors or other issues back to 
the install tools.

There needs to be a way, of course, for the package to specify what 
build tools it needs in order to be built, and for those to plug in 
to the (again stdlib-contained) build API.

There needs to be a better API for querying C configuration and 
compiler details, that's separate from the distutils "ccompiler" stuff.

Last, but not least, there needs to be a definition of core build and 
install tools to be both an example/reference implementation of the 
standard, and to provide the basics needed by the core.

I think I've mentioned all of these previously in the thread.  I also 
think that as a matter of technicalities, these things are not 
difficult to achieve...  but if it they are only achieved 
*technically*, then the result is a failure, not a success.

In order for the *real* goal to be achieved (i.e., a flourishing 
build/install system for Python), widespread participation and buy-in 
is required.  If the OS people or the big package people (e.g. Zope 
Corp., Enthought) or the distutils aficionados are left out, then the 
result won't get used.

I think the best way to ensure that nobody is left out, is to get 
them to participate in the design of a standard that ensures that 
*they* will be able to control their destiny, by creating their own 
build plugins and/or install tools...  or at least having a robust 
choice of alternatives.

We need a consensus "de jure" standard (ala WSGI), rather than just 
an ongoing "de facto" standard (ala distutils/setuptools), or we're 
not making any substantial progress, just handing the reins over to 
somebody else.


> >> Enforcements:
> >>    - a binary distribution cannot be uploaded if a source distrbution
> >> has not been previously provided for the version
> >
> > Note that this doesn't allow closed-source packages to be uploaded; thus it
> > would need to be a warning, rather than a requirement.
> >
>
>Right. do you agree it is something useful to do ?

Sure.


> >> New features:
> >>   - we should be able to download the metadata of a package without
> >> downloading the package
> >>   - PyPI should display the install and test dependencies in the UI
> >
> > It could only do this for specific binaries, since dependencies can be
> > dynamic.
> >
> >
>
>What dynamic means here ? the python module to static file process or more ?
>can you provide an example ?

Dependencies can be platform-specific as well as python-version 
specific.  If you want ctypes, you would depend on it in python 2.4 
but not in 2.5.  Similarly, if on some platform a certain library is 
required to implement a feature that is natively accessible on other 
platforms.  In these cases, you would have logic in setup.py to 
detect this and choose the appropriate dependencies.