[Distutils] command line versus python API for build system abstraction (was Re: build system abstraction PEP)

[Robert Collins]

On 11 November 2015 at 19:49, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 11 November 2015 at 16:19, Robert Collins <robertc at robertcollins.net> wrote:
...>> pip is going to be invoking a CLI *no matter what*. Thats a hard
>> requirement unless Python's very fundamental import behaviour changes.
>> Slapping a Python API on things is lipstick on a pig here IMO: we're
>> going to have to downgrade any richer interface; and by specifying the
>> actual LCD as the interface it is then amenable to direct exploration
>> by users without them having to reverse engineer an undocumented thunk
>> within pip.
>
> I'm not opposed to documenting how pip talks to its worker CLI - I
> just share Nathan's concerns about locking that down in a PEP vs
> keeping *that* CLI within pip's boundary of responsibilities, and
> having a documented Python interface used for invoking build systems.

I'm also very wary of something that would be an attractive nuisance.
I've seen nothing suggesting that a Python API would be anything but:
 - it won't be usable [it requires the glue to set up an isolated
context, which is buried in pip] in the general case
 - no matter what we do, pip can't benefit from it beyond the
subprocess interface pip needs, because pip *cannot* import and use
the build interface

tl;dr - I think making the case that the layer we define should be a
Python protocol rather than a subprocess protocol requires some really
strong evidence. We're *not* dealing with the same moving parts that
typical Python stuff requires.

> However, I've now realised that we're not constrained even if we start
> with the CLI interface, as there's still a migration path to a Python
> API based model:
>
> Now: documented CLI for invoking build systems
> Future: documented Python API for invoking build systems, default
> fallback invokes the documented CLI

Or we just issue an updated bootstrap schema, and there's no fallback
or anything needed.

> So the CLI documented in the PEP isn't *necessarily* going to be the
> one used by pip to communicate into the build environment - it may be
> invoked locally within the build environment.

No, it totally will be. Exactly as setup.py is today. Thats
deliberate: The *new* thing we're setting out to enable is abstract
build systems, not reengineering pip.

The future - sure, someone can write a new thing, and the necessary
capability we're building in to allow future changes will allow a new
PEP to slot in easily and take on that [non trivial and substantial
chunk of work]. (For instance, how do you do compiler and build system
specific options when you have a CLI to talk to pip with)?

-Rob


-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud