[Distutils] build system abstraction PEP, take #2

[Nick Coghlan]

On 10 December 2015 at 07:17, Robert Collins <robertc at robertcollins.net> wrote:
> On 10 December 2015 at 10:07, Donald Stufft <donald at stufft.io> wrote:
>> If we consider the setuptools/distutils era to be “Make it Work”, then we’re now at “Make it Right”, making it fast can come later but sacrificing correctness for speed isn’t something I think we should be doing and so speed arguments (vs why it’s more correct to do X instead of Y) don’t matter much to me.
>
> Developer speed is a correctness issue: this took ages to get my head
> fully around, but at the heart of it, there's a very narrow window
> between in-flow and breaking-flow and the reason developers care so
> much about latency of local operations is staying in-flow.

+1

I think this is also the practical difference between things like unit
tests, functional tests and acceptance tests: it's not about who
writes them, or how they're written, but which kinds of activities
trigger them being run. The key timelines:

* local development cycle (checking what you're currently working on)
* per-change activities (checking suitability for your fellow contributors)
* per-release activities (checking suitability for end users)

How those activities are carried out and the exact timelines involved
varies greatly based on the kind of software you're writing (e.g.
safety critical software vs a language runtime vs your typical web
application), and the context you're writing it in (e.g. a large
organisation vs a start-up vs a personal side project), but it's the
first one that has an essentially human element to it: how long we're
able to wait for a build-and-test cycle before we get distracted and
start thinking about something other than the change we're working on.

The value of setup.py develop and pip -e is that they take the
sdist->wheel->install process out of the local development cycle (most
of the time), so the speed of that can be governed by your test suite
and your test runners ability to tailor the tests it runs to the code
you're changing, rather than the duration of your build and install
process.

Regarding the digression into needing to duplicate pip's wheel
installation logic in the setuptools abstract buildsystem shim, I
think that's a reasonable thing to do as an initial step. Armed with
those two use cases, we can then consider how best to factor that code
out into a shared library that pip vendors (perhaps the existing
packaging library used for PEP 440 version specifier support, perhaps
a new library)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia