[Distutils] Ensuring source availability for PyPI entries / PEP: Build system abstraction for pip/conda etc

Thu Feb 11 13:00:58 EST 2016

On 11.02.2016 17:48, Donald Stufft wrote:
> 
>> On Feb 11, 2016, at 11:08 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>>
>> Then why not fix distutils' sdist command to add the needed
>> information to PKG-INFO and rely on it ?
>>
>> Or perhaps add a new distutils command which outputs the needed
>> information as JSON file and fix the sdist command to call this
>> by default ?
>>
>> There are many ways to address such issues, but defining a new
>> standard for every issue we have instead of fixing the existing
>> distutils implementation is not the best way to approach this.
> 
> 
> The very nature of distutils (later inherited by setuptools) is the problem to
> be honest. The reason we're adding new standards and moving away from these
> systems is that fixing them is essentially fundamentally altering them.

Of course. We're doing that constantly in Python, so why not
in distutils too ?

> For instance, adding some new information to PKG-INFO or turning it into a json
> file doesn't address the fundamental problem with why we can't trust the
> metadata.

pip is running on the target platform, so why would that be
an issue ?

Right now it's using the egg_info command to generate the
meta data, so it's well possible to add a better command which
then outputs JSON for pip and other installers to use.

> The reason we can't trust the metadata is because setup.py means that
> the metadata can (and does!) change based on what system you're executing the
> setup.py on. Here's a common pattern:
> 
> 
>     import sys
> 
>     from setuptools import setup
> 
>     install_requires = []
> 
>     if sys.version_info[:2] < (2,7):
>         install_requires.append("argparse")
> 
>     setup(..., install_requires=install_requires, ...)
> 
> 
> Any static metadata that is generated by that setup.py is going to change based
> on what version of Python you're executing it under. This isn't something you
> can just sort of replace, the setup.py *is* the "source of truth" for this
> information and as long as it is, we can't trust a byproduct of executing that
> file.

Again, there's nothing stopping us from adding a new command
which then allows defining meta data in a platform independent
way.

The reason for the above code is that it's convenient to
write. If there were an interface to provide such requirements
in a platform dependent way, which is then also understood
by the setup() command, we could get people to use the
new interface.

> In addition, the idea that a singular build system is really the best fit for
> every situation is I think, fairly nieve. Even today we have multiple build
> systems (such as numpy.distutils) even though utilizing them is actually fairly
> painful. This speaks to me that the need is fairly strong since people are
> willing to put up with that pain in order to swap out distutils/setuptools for
> something else.

AFAIK, numpy.distutils is just a customized version of distutils,
not a completely new system. We have customized distutils too, since
it allows us to do things stock distutils doesn't support.

That's a great freedom to have.

distutils allows using different builds system already (and has
ever since it became part of the stdlib). You don't have to
use the stock distutils build_* command implementations. Each
of those can be overridden or replaced. It's also possible to
add completely new ones. Same for the binary distribution format
bdist_* commands.

The complete PEP could be implemented straight in distutils
as new build command, or we could make things easier for package
authors and simply provide dedicated build commands for the different
build tools, so that authors only have to configure build system
to use in the setup.cfg file.

> As far as whether setup.py or something else should be the integration point
> I don't think that either choice would be a bad one. However I prefer using
> something else for a few reasons:
> 
> * The setup.py interface is completely entangled with the implementation of
>   distutils and setuptools. We can't change anything about it because of it
>   being baked into the Python standard library.
> 
> * A script that is executed as part of the packaging of the project is
>   notoriously hard to test. The best layout if we make setup.py the integration
>   point is that the setup.py will be a small shim that will call into some
>   other bit of code to do it's work. At that point though, why bother with the
>   shim instead of just calling that code directly?
> 
> * Having the script be part of the project encourages small, project specific
>   one off hacks. These hacks have a tendency to be fragile and they regularly
>   break down. Having the build tool be packaged externally and pulled in tends
>   to encourage reusable code.
> 
> That's on top of the *other* problem, that we can't fundamentally change
> distutils in anything but Python 3.6 at this point. That's not a terribly
> great place to be in though, because packaging has network effects and nobody
> is going to drop support for 2.7 (209,453,609 downloads from PyPI since Jan 14)
> just to make packaging their library slightly nicer. Even 2.6 (154,27,345) has
> more of the marketshare of what is downloading from PyPI than any version of
> Python 3.x. Moving things out of the standard library is how we've been able to
> progress packaging as quickly and as universally as we have been in the past
> few years. I am very opposed to any plan of improvement that involves waiting
> around on 3.6 to be the minimum version of Python.

But isn't that the reason why setuptools is kept on PyPI instead
of being added to the stdlib ?

With setuptools providing a way to support several different
Python versions, we don't have the backwards compatibility issues
you are describing. New commands and support can be added to
new releases of setuptools and are then available for packages
and installers to use and rely on.

Yes, adding new features will have to be done with some care,
but that's no different to what we're doing with Python itself
as well. It's certainly possible.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 11 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________
2016-01-19: Released eGenix pyOpenSSL 0.13.13 ... http://egenix.com/go86

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/