[Distutils] Q about best practices now (or near future)

[Ronald Oussoren]

On 18 Jul, 2013, at 13:48, Oscar Benjamin <oscar.j.benjamin at gmail.com> wrote:

> On 17 July 2013 22:43, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> 
>> On 18 Jul 2013 01:46, "Daniel Holth" <dholth at gmail.com> wrote:
>>> 
>>> On Wed, Jul 17, 2013 at 11:12 AM, Brett Cannon <brett at python.org> wrote:
>>>> I'm going to be pushing an update to one of my projects to PyPI this
>>>> week
>>>> and so I figured I could use this opportunity to help with patches to
>>>> the
>>>> User Guide's packaging tutorial.
>>>> 
>>>> But to do that I wanted to ask what the current best practices are.
>>>> 
>>>> * Are we even close to suggesting wheels for source distributions?
>>> 
>>> No, wheels don't replace source distributions at all. They just let
>>> you install something without having to have whatever built the wheel
>>> from its sdist. It is currently nice to have them available.
>>> 
>>> I'd like to see an ambitious person begin uploading wheels that have
>>> no traditional sdist.
>> 
>> Argh, don't even suggest that. Such projects could never be included in a
>> Linux distribution - we need the original source to push into a trusted
>> build system.
> 
> What do you mean by this?
> 
> I interpret Daniel's comment as meaning that there's no setup.py in
> the sdist. And I think it's a great idea and that lots of others would
> be very happy to ditch the setup.py concept in favour of something
> entirely different from the distutils way of doing things.
> 
> In another thread you mentioned the idea that someone would build
> without using distutils/setuptools by using a setup.py that simply
> invokes an alternate build system that is build-required by the sdist.
> That's fine for simple cases but how many 'python setup.py <command>'s
> should the setup.py support?

I don't think that's clear at the moment. It could be as little as
"bdist_wheel", that could be enough to interface to get from an extracted
sdist to a wheel.

The current focus is on defining a common metadata format (the metadata 2.0
JSON files) and a binary distribution format, and that's enough to keep
the folks doing the actual work occupied for now.

In the long run we'll probably end up with something like this:

* Sources from a VCS (that is, project in the layout used by those doing development)

          |
     [tool specific]
          |
          V

* sdist archive (sources + metadata.json + ???, to be specified)

          |
   [to be specified interface]
          |
          V

* wheel archive

          |
     ["pip", PEP 376(?)]

* installed package


If I recall correctly the transformation from sdist to wheel is currently not 
specified because getting the last steps (binary distribution and installation) right 
is more important right now.  The exact format of an sdist, and the interface for specifying
how to build a wheel from an sdist is still open for discussion and experimentation. 

That is, what's the minimal tool that could be used to create wheels for distributions that contain
one or more python packages with dependency information? And what would be needed for
a more complex distribution with (optional) C extensions, data files, custom compilers, ...?

The initial interface to the build system could well be a setup.py file that the
build system will only invoke as "python setup.py bdist_wheel --bdist-dir=DIR" (with
build-time depedencies specified in the metdata file) because
that's easy to arrange for distutils/setuptools, and it should be easy enough to
provide a dummy setup.py file with just that interface for alternative build systems.

Ronald